Hacker News Personal Blogs 2024 | All Blog Posts

What Even Was 2024?

SeanKilleen.com — Tue, 31 Dec 2024 20:00:00 GMT

Ed. Note: This is a personal post. Comments aren’t on, for a reason.

Year in Review: 2024

Thought Eddies — Tue, 31 Dec 2024 17:50:04 GMT

Year in Review: 2024

My Default Apps at the End of 2024

Srijan Choudhary, all posts — Tue, 31 Dec 2024 17:10:00 GMT

I saw a few blog posts with people sharing their default apps for the year, and I wanted to share mine as well. Here's the list: 📨 Mail Service: Fastmail 📮 Mail Client: Fastmail web, mu4e (Emacs), FairEmail (Android) 📝 Notes: Markdown and Org files in denote (Emacs), Markor (Android) ✅ To-Do: GTD using Orgmode (Emacs), Orgzly Revived (Android) 📆 Calendar: Google Calendar 🙍🏻‍♂️ Contacts: Google …

Virtue Ethics for Animal Rights Activists - A Practical Approach to Becoming a Better Activist

Josh Baldwin — Tue, 31 Dec 2024 17:00:45 GMT

I previously posted about identity being key to self-improvement. However, the identity of “animal rights activist” is broad. Having the identity of being an animal rights activist is easier said than done. Using virtues, we can break down identity into distinct features that are easier to understand and implement.

Best of 2024

Obvious Bicycle — Tue, 31 Dec 2024 14:25:16 GMT

All my favourite things from this year

Looking back at 2024

Lawrence Jones — Tue, 31 Dec 2024 14:00:00 GMT

I’m determined to make a tradition of this, which is why I find myself in an empty office on New Years Eve trying to remember what’s gone by these last twelve months.

I ended “Looking back at 2023” saying “brace for impact” which I had thought meant I was prepared for this year, but in retrospect I wasn’t even close. It’s a common story in start-ups to say “thank god that’s over, next year will be easier!” and be repeatedly wrong, but 2024 was particularly hard.

There is a lot to show for it, though.

The work
Everything else
- Ups and downs
- Talks and conferences

The work

This year was about three things: releasing and then extending On-call, conferences and external work, then AI.

Releasing On-call (January-March)

I started 2024 hell bent on getting our on-call product released, having been working on it since July the previous year.

In my previous recap I’d described On-call as “our most ambitious product yet” with one part being the reliability demands that come with a paging system. I was once a Principal SRE running large payment systems and I needed to become that person again as we ramped up for release.

This meant:

Building a plan to ensure On-call would be rock-solid
Reviewing every piece of code, adding tests, confirming edge cases
Running production drills to properly test everything

I’ve got a draft blog post I’m yet to publish called “10 days of load testing and drills” which is all about that last point:

To summarise: I spent several weeks in February working with Martha (an engineer from the team) to kick the living crap out of On-call before we released it.

By a fair margin, these weeks were the most fun I had this year. We had to build out core observability tooling (we had just setup Grafana before the break, but had no dashboards, docs or best-practices) and plan a load testing strategy, then run drills with each engineer in the team trying to repeatedly break the system.

Reliability and system design is my wheelhouse and I was bound to find it fun, but the most rewarding part was working with Martha. She joined the project with almost zero o11y experience and left being one of the most competent at the company, having listened to me drone on about the “importance of dashboard design” and what must’ve seemed like a load of other nonsense.

Working with Martha and each of the team to take proper ownership over On-call was extremely rewarding. It’s a testament to this work that since launch, On-call has exceeded 99.99% availability, even having onboarded hundreds of customers, some pretty large.

We have a “Behind-the-scenes building On-call” blog series that I recommend you check-out, but you can see how well the team met this challenge through their writing:

Martha with “Our observability strategy” which should be required reading for any team rolling out o11y
Rory Malcolm on “Continually testing with smoke tests” talking about the automated tests we run continuously to ensure we never miss a page
Leo Sjöberg on building on the horrible “Complexity of phone networks”

Ask anyone: four nines availability is no joke. There are teams in FANG who will tell you this isn’t possible for company like ours to achieve that, especially for a system like this.

Well, this team proves that wrong.

Building on our release (April-June)

We released On-call on March 5th but it’s easy to release a product, even one with a lot of fanfare, and find no users.

Thankfully that did not happen with On-call, and to give a sense of what’s happened in the 9 months since we launched:

Over 60% of our customers have bought On-call
On-call alone is about as big as incident.io was 2.5 years into operation
We have sent 4M on-call notifications (app notifications, phone calls, SMS, etc)

That’s now, but the first few months after launching were essential! We might’ve just launched an On-call product but (quite entertainingly) so had FireHydrant, and Rootly had soft launched their On-call website (I think in response to our impending launch?) so we were in a crowded space.

That meant several months of dialling into the customer feedback firehose and building whatever was needed to unblock deals, smoothing rough edges found during migration, and generally getting comfortable running this system.

Probably the most noticeable of this work was “Smart Escalation Paths” which introduced dynamic branching into escalations, supporting out-of-hours low urgency pages and a series of other improvements.

After so long building a product without customers (other than early partners) it was energising to plug back into the ship, feedback, iterate cycle. Honestly just seeing people go “wow, this is so good!” felt rewarding and helped the team feel their achievement.

I also got to dive into mobile app development, something I’d never done before. This involved setting up authentication for the mobile app, a few small features, and eventually giving me the opportunity to build a silly diagram of the custom in-house over-the-air deployment flow we’d built!

Overall, not a bad quarter!

Scaling to larger customers (July-September)

Thanks to our focus on “human on-call” our On-call product was getting a load of traction with small to medium companies. Crazy huh, imagine making it easy to request cover from your team, or connecting to HR systems and showing holidays alongside your schedules?

But larger companies have more complex setups, and after assigning engineers to On-call sales calls, we found a bunch of small friction-points that made using us at scale difficult.

Luckily the issue wasn’t “can incident.io work for my company” and in fact we are (in my opinion) the most flexible and powerful of the on-call products out there, largely due to our Catalog, meaning you can build whatever routing structure you like. Instead, it was “can I figure out how to make it work” which is a very different type of problem.

On-call in particular is hard for this, as each customer has a unique set of:

Organisation structure
Observability/alerting tools
Alert routing and ownership mechanisms
Alerting philosophies

You have to solve the problem by fully understanding both across all customers and generically for our platform, and then deep into whatever slice of tools a customer may want to use. Finding solutions that are idiomatic wrt Grafana/Datadog/whatever you may be using requires an expertise in those tools too, in addition with a strong understanding of our system.

We approached the problem by roasting our product, repeatedly setting up accounts and tracking how customers did it, forcing people to develop a ‘healthy impatience’ with our product so they’d balk at the first sign of friction.

Eventually, we rolled out a series of changes that massively helped bigger companies like Intercom access the power of our product more easily.

These ranged from:

Redesigning alert config so we visually show how alerts flow from source to routes
Using AI to automatically parse alert attributes into Catalog fields, so we wire-up Team, Service, Feature or Product automatically
Adding onboarding sign-posts that help customers find their way to the right configuration

The best part was, after shipping any of these changes, sitting on a call with a customer helping them setup their config and watching as they skipped through the stages, or in some cases had everything automatically wire itself up for them.

I regularly review our larger customer’s configurations and now, after all this, you can visually see how much cleaner their config has become with the product helping them improve it.

Really rewarding, and one of the reasons why we’re closing this year having landed some of our biggest On-call sales yet.

AI (forever)

If you’d asked me in January what my focus would be this year, AI wouldn’t have even made the list. And yet here I am, having spent the last quarter hitting my head against LLMs and building a foundation for what could be our most exciting product yet.

Quite clearly AI changes the scene: auto-generating incident summaries is by far the most widely adopted of the features we built last year, and often the one people quote as the most useful. But there’s so much more that is possible with modern AI tools that can totally change incident response.

That’s why I was asked to help setup a team whose focus would be ambitious, step-changes we could make to the product with AI. Initially I was helping build abstractions, which meant: building eval frameworks that could reliably test prompt behaviour, creating debugging tools that could introspect multi-step AI interactions, and developing architectural patterns like our map-reduce shortlister that efficiently searches data using AI models.

But as we started pulling on the thread of AI we began seeing ever more opportunities to totally overhaul our product. It seemed increasingly obvious that someone was going to reimagine incident response with these tools, and we’d love that person to be… well, us.

So I ended up on the team full time, which meant turning my job into a full time struggle.

That’s because AI engineering is different. Not only are the tools themselves non-deterministic, but figuring out how to use them effectively is a process with very non-linear progress.

Some weeks you’ll make breakthrough after breakthrough, and others you’ll spend days wrestling with prompt engineering only to end up exactly where you started. It’s a humbling experience, especially for engineers used to deterministic systems. Even – and in some cases especially – our most experienced and productive engineers hated it, and we had to build some resilience to that.

We now have a roadmap, though, and unlike On-call we’re talking more publicly about what we’re building. We even have a beautiful homepage at incident.io/ai!

That roadmap breaks into three initiatives:

Scribe (launched)

Scribe joins incident calls for you – Zoom, GMeet, whatever – and takes notes, then processes them into key moments and actions that get relayed back to the incident channel.

This is so obviously a perfect use case for AI, with it freeing capacity for humans to do the real work (responding to the incident) while helping them do their job better, by communicating their work more effectively to their colleagues.

While I didn’t work on Scribe myself, it’s built on the AI primitives I’d helped build back in October such as our new prompt framework and eval test suite. It was a good example of how effective those abstractions have been, as it allowed the team to build the AI parts of Scribe very quickly, and even people who hadn’t worked with AI before felt confident in the outcome.

Chat (early access)

The incident.io team no longer use slash commands to run their incidents. Instead, we chat to our bot and just tell it what we need:

“@incident.io draft me an update including why Postgres is checkpointing so regularly”
“@incident.io decline this incident and create a follow-up”

I am biased, but this is the best bot I’m yet to interact with, and people are finding novel use cases for it on the daily.

We have a #copilot-pulse channel that pings every time someone interacts with the bot, along with a grade we assign to each interaction that we monitor for quality.

It already does a huge range of things, but some of my favourite interactions are people just asking it for help with the incident. “Fix my query” or “what does that actually mean” where it saves them loads of time to get an answer immediately, one that has all the context of the incident.

Investigation (coming soon)

But the really exciting (and challenging) work is in our Investigations system. We’re pushing the boundaries of what’s practical with current AI technology, aiming to have the system automatically check dashboards, read logs, and build a coherent picture of what’s happening before responders even arrive. The goal is ambitious: turn hours-long debugging sessions into minutes by having AI do the heavy lifting.

Personally, this thing has been a challenge. I struggle to sleep when I have an unsolved problem and I’ve gone weeks having no idea how to make this stuff work, where I’ve been dreaming every night of LLM prompts and all sorts. It’s been quite a nightmarish struggle, honestly.

But when it works – when you see the bot correctly identify the root cause of an incident and suggest the exact right action – it feels like we’re glimpsing the future of incident response.

This is definitely turning into “the thing” for 2025. The foundational work we’ve done this year, while often invisible, has set us up to deliver some pretty incredible capabilities to our customers. It’s not just about making existing workflows slightly better – it’s about fundamentally transforming how teams handle incidents.

Everything else

I’m getting dangerously close to needing to leave for NYE, so I’ll keep this short.

Ups and downs

Looking at what we’ve achieved in 2024 is incredible. Even just the success of On-call is massive, ignoring everything else we’ve done.

It’s been really tough, though. We’re in great shape now but things weren’t as rosy at the start of the year, leading to some hard decisions and parting with some of the team in June.

I’d never been close to layoffs before and definitely not in a team as close as ours. This sucked for everyone, a totally shitty experience that leaves some nasty scars for everyone it touches, even when handled well (which, in my opinion, it was).

At this point it’s been 3.5 years since I joined incident, having given myself two days (otherwise known as a weekend) between a 6 year stint at GoCardless that was similarly intense. While I’m focused on the next few months, I want to carve out a proper break for myself next year to reset a bit, as it’s been all consuming for a while now.

Just start-ups, I guess!

Talks and conferences

On a positive note I spoke a lot more this year, and travelled a bunch to do it. Some highlights were:

Architecture at incident.io (Netflix)

I have a huge amount of respect for Netflix engineering which has only been strengthened by my experience working with them as a customer.

It was a big moment for me when I was invited to give a presentation on incident.io’s architecture in our of the internal engineering technical deep-dives.

Over one hundred engineers came to listen to how we built our little incident app, with subjects ranging from Postgres TOAST tables to load testing and modular monoliths. Getting grilled by Principal engineers at one of the infrastructure giants was a massive highlight for me!

Starting from nothing, StaffPlus, New York

LeadDev and StaffPlus are really special conferences to me, having attended them and listened closely in my early career and having always appreciated their work.

New York is a pretty insane flagship location to get invited to talk, but even cooler if you’re talking about how to “start from nothing” with big company initiatives, and pulling the covers back on the experience of a Staff individual contributor in those situations.

I’d also never been to New York, and got to visit a bunch of friends from our office while there.

You can watch the recording here: “Starting from nothing”

Incidents are a whole organization game, Sev0, San Francisco

We ran our own conference this year called “Sev0” which was so much fun.

Firstly, if you’re a company like us with a bunch of top tech logos, SF is where you go to meet your customers. Having dinner with the speakers beforehand was quietly mindblowing for me, when you get an insight into these tech behemoths and how they do things through some of their most senior staff.

I delivered a talk on leveraging organisational context during incidents, a topic particularly close to my heart after fighting tooth and nail to build our Catalog product as a fully flexible schema’d store, all to power use cases like syncing your CRM.

The venue was amazing, the people totally awesome, 10/10 would recommend.

You can see the talk and transcript at “Organization-aware incident response”.

Wrapping up

Looking back, this has been a year of massive change – both planned and unexpected.

We started with the intensity of launching On-call, a product that exceeded even our ambitious reliability goals and has grown to be about as big as incident.io was at 2.5 years. Then came the harder moments in summer that tested our resilience as a team, and eventually dropping myself headfirst into a deep sea of AI.

It’s been really hard, even harder than the previous years and what I’d expected it to be!

Christmas has been a time of reflection for me and I’ve processed a lot, leaving me feeling like I fit a whole load of life experience into the last year.

I’m again reminding myself to be appreciative of the journey, and that the experience I’m having now will be the one I miss in a couple of years, and not to waste any moment of it.

So not making the same mistake again: next year is going to be harder again. I will break myself several times over before fully cracking these AI products, I will repeatedly feel hopeless and like I’m not able to meet the challenge, and there will be curveballs I don’t expect which totally throw me.

If we’ve achieved as much at the end of 2025 as we did this year, though, it’ll have been worth it.

2024 in Review - 100 Books

Home on Lagomorph — Tue, 31 Dec 2024 09:00:00 GMT

I read 100 books this year as a challenge to myself. The hardest part about reading 100 books in a year is fomatting them cleanly afterward on your website. The second hardest part is tracking them all. The actual activity is a cinch.

Reading 100 books this year was directly inspired from stories of Theodore Roosevelt, who read one book a day, often multiple. The thing I’ve learned about reading serious literature in quantity is that most books are phenomenally forgettable. Tiago Forte, in “Building a Second Brain”, talks about saving quotations from every book you read, either on index cards or digitally - and when you’re reading so much, thats literally all you can do to try and keep track of the different ideas presented to you.

New year, new platform

Code Explainer — Tue, 31 Dec 2024 08:11:48 GMT

I have decided to move my content to a different platform – details to follow very soon.

App defaults entering into 2025

utf9k — Tue, 31 Dec 2024 07:10:00 GMT

Here are my app defaults that I'm bringing with me into 2025

A quick review of 2024

Martin Rue — Tue, 31 Dec 2024 06:00:00 GMT

In 2024 I lived in 3 countries and 5 different cities. My favourite is Bangkok. Da Nang is a very close second.

2024.12.DisappearingMoment

Disappearing Moment — Tue, 31 Dec 2024 05:19:27 GMT

This month, a lot of people got worked up about the possibility of mystery drones over New Jersey. We made Orson Welles comparisons. The New York Times reported on drone fever.

A U.S. Representative said that the drones were Iran’s doing. New Jersey's best political reporter mewled that this "theory sounds possible". He wrote that there was "an absence of other plausible theories”. (Matt Friedman admitted his error four days later.)

The United States Department of Homeland Security got involved. So did the Federal Bureau of Investigation. And the Department of Defense. Along with the Federal Aviation Administration, these three issued a joint statement on drone sightings. The drones at night were loud and bright deep in the heart of Essex. You can read all about it in the New Jersey Office of Homeland Security and Preparedness Drone Incidents FAQ.

In the mid-90s, I loved a quote by Sam Donaldson or Tom Brokaw. “In place of truth, we have created facts.” My update: In place of facts, we have created FAQs.

I'm writing about this to document the before times. A month from now, we will miss what it is like now. We will be sentimental, furious, terrified, tortured. We will wonder how we permitted coaxial atrocities.

When we look back, if we are able to look back, we should know who we were immediately before the collapse. We wanted answers to questions we stumbled to articulate. We believed what we felt rather than what was true.

Truth is a tiny, silent, piceous drone. Fungible and uninteresting. You don’t know it was there until you’re stuck with undeniable, retrospective evidence.

Welcome to December 2024’s Disappearing Moment, an inventory of my experiences. I hope you enjoy it.

Podcasts

Americana Partners’ Weekly Market Update Replay (Worth My Time): David M. Darst is my Walter Cronkite. I listen to find out what others consider important. And reassure myself that I haven't missed anything.

Nerdy Software

Convert M4A to MP3 in your browser if you’re feeling too lazy to use ffmpeg.

Free Font

B612: Optimize the overall homogeneity of your cockpit.

Bougie Products

I’m excited about my Viking Press and T-Bar Row attachments. There were some exercises I couldn't do at home. Now I can.

Personal Finance and Investing

Nisiprius is my favorite Boglehead. You should read his posts, starting with the one about risk tolerance. Then pick from over 53,000 others.

Reading

Laura Olin, “You don’t actually have to stay on Twitter.” (A Personal Favorite): I think less of anyone who does.
Melanie Sumner, “Living with ADHD” (I Liked It): Sometimes I struggle to "organize the stuff (I) need to do in a way that makes sure it gets done."

Some Topics I Intend to Write About in 2025

Cryptocurrency. Learn from my mistakes.
Dating a long-term spouse or partner.
Democracy.
Eating. Exercising.
Investments that Beth and I have in our accounts. For transparency and accountability.
Music.
Running. I like writing about the professionals.
Skincare.
Values that matter to me as a librarian.

Thank you for spending a few moments with me. I appreciate you and look forward to corresponding again next month.

Brett

Want to discuss any of the topics in this newsletter or anything else with other Disappearing Moment readers? Please sign up for Perpetual August. I think it might be fun.

Gambling with language models

Fun With The Cloud — Tue, 31 Dec 2024 05:17:20 GMT

One clueless investor's attempt at beating the stock market with ModernBert

Challenges Building an Electron App

Thought Eddies — Tue, 31 Dec 2024 03:06:55 GMT

Challenges Building an Electron App

Collective Effervescence

Aswath Krishnan — Tue, 31 Dec 2024 02:50:16 GMT

I feel a rush of energy whenever I'm in NYC.

It hits me the moment I step out of the airport, and it's there every time I leave my hotel or brother’s apartment. NYC has majestic buildings, thriving retail, and jam-packed roads. But the main catalyst for making it feel so alive is its people — Pedestrians. Everywhere.

When I first left home to attend university in Singapore, my uncle (who had lived abroad before) wisely advised me to go to crowded places if I felt lonely—and it really helped. But my hostel didn’t have crowded spaces, especially after dark. Sure, you can go to malls, restaurants, or events, but those aren’t practical everyday activities. I find Seattle and most suburbs or even cities in the US to be similar. You barely see people in the streets; just cars or an eerie (serene, if you prefer) emptiness. My mom is often surprised by how she doesn't run into a single soul, save the birds and rabbits, during her daily walks in the neighborhood.

But NYC? That's a rare and different universe. People are always out and about. Different faces, styles, and walks of life, all moving along briskly with purpose. Even as a stranger among them, even without speaking to anyone, their energy rubs off on me.

It’s hard to explain what is happening. There’s a kind of primal comfort in just seeing other humans. The very presence of others, living and breathing and going about their day, creates a kind of ambient belonging and recharges you. It’s like everyone’s a tuning fork vibrating in resonance with each other and the city’s energy, and amplifying it. You’re not just an isolated self anymore; you’re absorbed into something bigger, a shared rhythm, an unscripted orchestra of humanity— a collective flow state.

I know I'm not the only one who craves this energy. And I recognize it isn't all sunshine and roses - the lack of personal space, the incessant noise, and the crowds can wear you down too. I find silence and serenity of Seattle rejuvenating after a trip to NYC. But does it have to be a binary choice between suburban sprawl and overcrowded cities? Surely we can design more places to be walkable, vibrant, and brimming with the hum of humanity, without completely losing individual space or peacefulness? Places where you sense life happening all around you, without derailing your own.

I hope so, because what makes us flourish is not just living in a place, but feeling like the place itself is alive.

Generating a triangular navigation mesh from H3 hexagons in R

Jonathan Chang — Tue, 31 Dec 2024 02:00:00 GMT

Introduction

A navigation mesh is a data structure used to aid in pathfinding around obstacles. Originally used for video games and robotics, we can also apply the concept of this navigational mesh to the movement of animals through landscape, using methods such as FEEMS.

Consider the following landscape¹:

Finding a path from starting point s to goal point t is computationally challenging, as there are many possible routes one might take between these two endpoints, and the possibility space is so large that it can be challenging to efficiently compute a fast path. Instead, we can simplify the space by shrinking the possible locations in the landscape to a series of nodes, and the neighboring nodes that you can move to are connected by edges. The resulting mesh typically looks like a bunch of triangles overlapping the landscape:

In this blog post I’ll discuss how to construct such a mesh using real-world shapefiles and the h3 library in R, which tessellates hexagons across the globe at several resolutions. I also looked into alternatives such as dggridR but I found other packages to not be satisfactory for a number of reasons (mostly speed).

To begin with, why h3, and why hexagons? The answer to this is that it is very convenient to have a grid system that can be used for any place on the planet, and having a variety of different resolutions makes it convenient to analyze spatial data on scales ranging from continent-level to city-level. Hexagons in particular have a nice property where the distance from the center of one hexagon to its neighboring hex is roughly equal.²

Worked example

First, load some required packages, including h3jsr, which I found to have a nicer interface than other h3 libraries for R.

            library(tidyverse)
library(h3jsr)
library(sf)
requireNamespace("maps")

          

For today’s example we’ll be using a simple map showing some counties in the San Francisco Bay Area (SFBA). We need to turn off spherical geometry in sf since at this scale, everything is approximately planar anyway and there are some issues with the geometries provided in the maps package.³

sf::sf_use_s2(FALSE)
bay_counties <- c(“alameda”, “contra costa”, “marin”, “napa”, “san mateo”, “santa clara”, “solano”, “sonoma”, “san francisco”)
sfba <- maps::map(‘county’, paste0(“california,”, bay_counties), fill = TRUE, plot = FALSE) %>%
st_as_sf() %>%
st_transform(4326) %>%
st_make_valid()
ggplot(sfba, aes(fill = ID)) +
geom_sf() +
theme_minimal()

Next, we need to find the h3 hexagons that intersect this shapefile of the SFBA. I’d like my hexagons to cover a couple of square kilometers, and this corresponds to roughly resolution 7 in h3. We must first dissolve the individual features (i.e., remove internal borders) then use the polygon_to_cells function to identify the correct h3 cells that intersect with our SFBA borders.

          dissolved_sfba <- summarise(sfba, geom = st_union(geom))
ids <- polygon_to_cells(dissolved_sfba, res = 7)[[1]]
head(ids)

        

          ## [1] "872830311ffffff" "872830925ffffff" "87283154dffffff" "872836a62ffffff"
## [5] "872830314ffffff" "872830928ffffff"

Plot these h3 hexes to get an idea of what we’re working with.

          ggplot(cell_to_polygon(ids)) +
  geom_sf() +
  theme_minimal()

        

For each cell ID, we need to identify its neighbors using the get_disk function with distance = 1. This function’s documentation says:

The first address returned is the input address, the rest follow in a spiral anticlockwise order.

This is perfect for our use case. We take the origin hex and find its center. Then do the same for two of its neighbors, and construct a triangle between the centers of these three hexes.

id <- ids[1]
disk <- get_disk(id, ring_size = 1)[[1]]
first_three <- disk[1:3]
first_triangle <- cell_to_point(first_three, simple = TRUE) %>%
st_combine() %>%
st_cast(“POLYGON”) %>%
st_as_sf()
ggplot(first_triangle) +
geom_sf() +
theme_minimal()

We can repeat this procedure to construct triangles between the centers of all the hexes surrounding our origin hex. We exclude the origin point from the loop and also add in the first neighboring hex in the ring, otherwise we’ll only have five triangles, not six, around the origin hex.

wrapped_vec <- c(disk[-1], disk[2])
results <- list()
for (ii in 1:(length(wrapped_vec) - 1)) {
results[[ii]] <- cell_to_point(c(id, wrapped_vec[ii], wrapped_vec[ii+1]), simple = TRUE) %>%
st_combine() %>%
st_cast(“POLYGON”) %>%
st_as_sf()
}
tris <- bind_rows(results) %>%
mutate(idx = 1:6)
ggplot(tris, aes(fill = factor(idx))) +
geom_sf() +
theme_minimal()

We can build off of this initial work to write a function that will take an h3 id as input and return a triangular mesh.

      to_triangles <- function(id) {
  disk <- get_disk(id, ring_size = 1)[[1]]
  wrapped_vec <- c(disk[-1], disk[2])
  lapply(1:(length(wrapped_vec) - 1), function(ii) {
    tri <- c(id, wrapped_vec[ii], wrapped_vec[ii + 1])
    cell_to_point(tri, simple = TRUE) %>%
      st_combine() %>%
      st_cast("POLYGON") %>%
      st_as_sf()
  }) %>% bind_rows()
}

    

Use lapply and bind_rows to construct an entire sf dataframe that contains the entire triangular mesh. There are a lot of duplicates but we can just use distinct to get rid of these. (I use parallel::mclapply here as this step can be a bit slow.)

all_tris <- parallel::mclapply(ids, to_triangles, mc.cores = 8) %>%
  bind_rows() %>%
  distinct()
ggplot(all_tris) +
geom_sf(fill = “white”) +
theme_minimal()

Observant readers will note that these triangle meshes will include nodes that are out in the water or in neighboring counties, since the hexes at the edge of our SFBA shapefile will inevitably have a few neighbors that are not contained within the SFBA shapefile.

While it wasn’t necessary to do so for my analysis, these can easily be removed by using a binary predicate such as st_contains, to never include any edge that enters the water or otherwise exits the boundary of the shapefile.

    contain_result <- st_contains(dissolved_sfba, all_tris)
contained_mesh <- all_tris[unlist(contain_result), ]
ggplot() +
  geom_sf(data = dissolved_sfba, fill = "pink") +
  geom_sf(data = contained_mesh, fill = "white") +
  theme_minimal()

  

This navigational mesh can now be saved using functions such as write_sf and used in downstream analysis software.

Exercises

How can we avoid the creation of duplicate triangle meshes?
Use a compact representation to generate a triangle mesh with simplified interiors.

comp <- cell_to_polygon(h3jsr::compact(ids), simple = FALSE)
ggplot() +
geom_sf(data = dissolved_sfba, fill = NA) +
geom_sf(data = comp, fill = NA, color = “red”) +
theme_minimal()

Figures taken from these lecture notes. ↩
For more detail, see Uber’s blog post introducing the h3 system. ↩
For your own analyses, you should probably use shapefiles that can be correctly handled by sf. ↩

What You Are Paying For

jeremymikkola.com — Tue, 31 Dec 2024 02:00:00 GMT

Posted on December 31, 2024

When you go into your local burger shop and buy a burger, what are you paying for? One way to look at it is that you are paying for the food, for the convenience, for the experience - you are paying for what you get out of it.

You can also think about the things that are necessary to create the product you bought. There’s the beef and the buns. There’s also the labor that goes into making that burger. If you want to really pull apart the business you can list all sorts of things you are also paying for: the rent on the space the restaurant is in, their taxes, insurance, interest on loans, utilities, grease disposal, cleaning, advertising, repairs, accounting, shipping, spoilage, management, banking and payment services, toilet paper, and maybe a paper-thin slice of profit on top.

That mental model feels a lot more detailed. You could list out where every thousandth of a cent of that burger’s price ultimately goes. But that model misses a dimension: it misses why those elements cost what they do.

Take rent, for example. A big factor in rent costs is supply and demand. A big factor in the supply side is how much real estate the city will allow to be zoned commercially, and what restrictions they place on it. Laws intended to keep a city looking nice, like limiting the height of buildings, have a side effect of increasing rent. When you go to one of those hip locations to buy your burger, some of what you are paying for is the fact that the surrounding area is nice to be in. When you buy a burger you are buying the outcome of zoning decisions.

You are also paying for that location to continue to be a burger shop instead of turning into a Chase bank or Starbucks.

Let’s say that just down the street from the local burger shop is a franchise of the large national fast food chain of your choice. What are you paying for if you eat there, instead? Obviously you are paying for a slightly different mix of many of the same costs associated with the local place. The meat might be a bit cheaper but now there’s landscaping and power washing costs (this location has a drive-through). Since this location is a franchise of a chain, some slice of the pie also goes to that national chain. What does that buy?

It buys use of the brand name, it buys access to the logistics chain, and it buys advertising. It also buys whatever else that national chain decides to spend their remaining money on. In the corporate headquarters of every burger-selling chain out there, there will be someone whose job it is to handle expansion. Someone in some office building sits and makes decisions about whether a given site is good for a new location. When you buy a burger from your local instantiation of this burger empire, you are in effect buying more locations of that chain. Those new locations are likely somewhere hundreds or thousands of miles away and of little value to you.

When you pay for a SaaS service, part of what you are paying for is feature development. Whether you are paying for the feature development that already happened in the past, the feature development that is currently happening, or the feature development that will be done in the future… is left as an exercise for the reader. What it does not pay for is the development of features that are all aimed at you. Some of that development may go to make your life better: a thing runs a bit faster or has another bell and whistle. Some of that development is aimed more internally to the company: they make something cheaper for them to run, page them in the middle of the night less often, or have fewer security holes. But, like the national burger chain, some of it will be aimed at expansion. They will develop new features, spin up new teams, perhaps even create entire new product lines (or acquire other businesses) for the sake of expanding. They might start building the features that they need to have to win the big enterprise user, features that might be utterly useless to you. And yet part of what you are buying when you pay for that service is their ability to acquire those customers.

This entire line of thinking becomes practically useful when flipped on its head. Instead of thinking about the fact that buying a burger buys a nice neighborhood, one can follow the logic in the other direction and see that laws and policies that lead to nice neighborhoods will reduce the supply of commercial real estate, increasing the price of rent, and thus cause burgers to get more expensive. Expanding to enterprise users means giving your small users a bit less for their money. Every action has externalities, and every decision has costs. Things following the pattern of “you cannot have your cake and eat it too” (or “there’s no such thing as a free lunch”) are the rule not the exception.

2024 Year in Review

Ben Overmyer's Site — Tue, 31 Dec 2024 02:00:00 GMT

2024 was a year of travel, challenges, endings, and beginnings. I went on five trips, four of which were for recreation and the fifth of which was almost a move to a different state. My fitness and health suffered, and Silver Gryphon Games shut down. However, I began taking music lessons, won an award at work, and regularly ran an RPG campaign for once.

Winning an Award at Apiture

At my day job, I won an award for my work on Apiture’s hosted sites service. I continued to be the sole infrastructure person for hosted sites, despite their quantity at the end of the year. My automation for them made it much easier for the designers to spin up new sites without requiring work from the platform engineering team.

Collapse of Exercise Habit

My exercise habit fell apart. I started the year strong, going to the gym regularly and focusing on a bodybuilding routine that was slowly starting to show progress. Then, I went to my annual physical, and the doctor was less than enthusiastic about my chosen exercise path. For some reason, I let that discourage me from exercising at all the rest of the year.

Towards the end of the year, though, I started going for longer walks occasionally. I have it as a daily to-do item, but I'm not consistent in doing it daily yet.

Alcohol-Free Months

At the beginning of the year I set out to have three full alcohol-free months in 2024. I achieved two full months and two half months. That is sort of a success, I guess, but I think I can do better. Next year will have a more ambitious goal. My interest in going alcohol-free has increased.

I've attempted to completely quit drinking a couple times before. This is not that, not yet. I can see this evolving into that though. Past attempts were not based on a holistic lifestyle change. Now, going without alcohol is an accessory to other changes I have made or am planning to make.

The Death of Silver Gryphon Games

While we tried to resurrect the company, neither Kevin nor I had the time or motivation to work on it. I formally shuttered the business in December. The second editions of both Aether and Ingenium will never be published, sadly. That’s too bad, because the cover art for both was awesome, and they fixed a lot of the problems with the first editions. However, a business needs sustained work in order to function, and that won’t happen for SGG.

Journaling About Trips

In 2024 I went on three major trips and I promised to journal about all of them. I wrote about the Cancun trip and the cruise, but I did not write about the eclipse trip. I thought I did, but I can’t find a journal entry for it. Two out of three, but not enough to count this goal achieved.

The journal entries that exist are nice and detailed, though. They'll be a good reminder in days to come.

Journaling Habit

While I did keep a regular work journal each week, I didn’t write a regular personal journal. Part of the problem is that I never set up the habit like I did with the work journal; there was no specific trigger. This is something I can fix in 2025.

I did start keeping a gaming journal just a few days ago. If I keep up with this, then I will have four different journals that I update: a personal journal, a work journal, a gaming journal, and a spiritual journal. Perhaps if I write regularly for all of them, that will start to bleed through into blogging, and I'll update this site more frequently.

Books Read

I didn’t read nearly as many books in 2024 as I did in 2023. Also, the variety was less, with 8 different authors in 2024 and 11 in 2023. I’m not sure why, but I imagine losing almost a full month of the year to trips where I didn’t read at all didn’t help.

My queue of books to read is growing. I have quite a few on my shelves. I may hold off on buying any more books until I've worked through that backlog a bit.

RPG Campaign

In November I started running a Dungeons & Dragons campaign in-person for a few random people. It is now the longest-running regular campaign I’ve ever run. It’s also the most difficult to run, as several of the players remember throwaway comments I make about setting flavor and accept that as canon. This means I have to take notes on everything I decide. On the other hand, though, these notes are making it easier to plan sessions and review what happened in past sessions.

The next session will be on January 6, after a several-week break for the holidays.

Beginning of Music Lessons

In December I began taking guitar lessons. This is the first time since I was a teenager that I’ve taken music lessons. I’ve been good about practicing almost every day, too; the only days I haven’t practiced were Christmas Eve and Christmas Day.

I've been practicing the usual beginner songs from a Hal Leonard book. As I get better, I want to try a few different styles, including Spanish guitar. I have a list of songs to attempt once I get good enough.

Previous Years

Building for and debugging an ARM Cortex-M

pera's blog — Tue, 31 Dec 2024 02:00:00 GMT

I recently bought an ST NUCLEO-L432KC, a small dev board with an STM32L432KC, a Cortex-M4 micro-controller designed for ultra-low-power applications.¹ These STM32 MCUs come with a proprietary tool for code generation called STM32CubeMX, unfortunately my experience with it was very poor so to speak: I couldn’t even generate a functioning base project for this board.²

Since ST doesn’t provide ready to use template projects I had to look for an alternative. PlatformIO is a popular tool nowadays for these sorts of things. I’ve never used it before but I was able to go from zero to blinking LED in less than 5 minutes, literally. My only issue with it was that this felt like cheating, and I was looking to learn a bit more about how these binaries are built, so I decided to build it “from scratch”.

First you will need to install a GDB and GCC cross-compiler that can target ARM, the Newlib C standard library, and OpenOCD. This will depend on your distro but in Arch Linux you’ll find everything in the official repo:

sudo pacman -Syu arm-none-eabi-gcc arm-none-eabi-newlib arm-none-eabi-gdb openocd

The next step is to obtain CMSIS Core (standard APIs for Cortex processors), BSP and HAL drivers for our MCU’s family, which is all provided by ST on GitHub:

mkdir nucleo-l432kc
cd nucleo-l432kc
git submodule add --depth 1 https://github.com/STMicroelectronics/cmsis-device-l4.git
git submodule add --depth 1 https://github.com/STMicroelectronics/stm32l4xx-hal-driver.git
git submodule add --depth 1 https://github.com/STMicroelectronics/stm32l4xx-nucleo-32-bsp.git
git submodule add --depth 1 https://github.com/STMicroelectronics/cmsis-core.git

Now you will need the device vector table (“contains the initialization value for the stack pointer, and the entry point addresses of each exception handler”) and the linker script for our MCU. You can find these files and also a bunch of basic examples in the STM32CubeL4 MCU Firmware Package repo, which is actually quite heavy, so let’s grab just what we need from there:

git clone --depth 1 --filter=blob:none --sparse https://github.com/STMicroelectronics/STM32CubeL4.git
cd STM32CubeL4
git sparse-checkout set Projects/NUCLEO-L432KC/Examples/GPIO/GPIO_IOToggle

The vector table initialisation code is in startup_stm32l432kcux.s, and the linker script for GNU ld is STM32L432KCUX_FLASH.ld. This directory includes project files for IDEs we don’t care about, but you should copy the example code inside Src and Inc for blinking the green LED in the board.

For some reason STM32L432KCUX_FLASH.ld is missing a symbol named __end__ that is needed in newlib. I haven’t look too much into this but maybe ST’s fork of newlib does something different. In any case, what I have done is to include the following line in the ._user_heap_stack almost at the end of the file:

. = ALIGN(8);
PROVIDE ( end = . );
PROVIDE ( _end = . );
PROVIDE ( __end__ = . ); /* Add this directive here */
. = . + _Min_Heap_Size;
. = . + _Min_Stack_Size;
. = ALIGN(8);

I will save you the pain of having to figure out what exactly you need to compile this simple project with GCC (you can also find all this stuff in my git repo nucleo-l432kc-template):

CC = arm-none-eabi-gcc
DEFS = -DSTM32L432xx
CFLAGS = -Wall -Wextra -g -march=armv7e-m+fp -mfloat-abi=hard -mfpu=fpv4-sp-d16 --specs=rdimon.specs
LDFLAGS = -TSTM32L432KCUX_FLASH.ld
CMSIS_DEVICE = cmsis-device-l4/
HAL_DRIVER = stm32l4xx-hal-driver/
NUCLEO_BSP = stm32l4xx-nucleo-32-bsp/
INCLUDES = -Isrc -I$(HAL_DRIVER)/Inc -I$(NUCLEO_BSP) -Icmsis-core/Include -I$(CMSIS_DEVICE)/Include

all:
	$(CC) $(CFLAGS) $(LDFLAGS) $(DEFS) $(INCLUDES) \
	$(HAL_DRIVER)Src/stm32l4xx_hal.c \
	$(HAL_DRIVER)Src/stm32l4xx_hal_cortex.c \
	$(HAL_DRIVER)Src/stm32l4xx_hal_gpio.c \
	$(HAL_DRIVER)Src/stm32l4xx_hal_pwr.c \
	$(HAL_DRIVER)Src/stm32l4xx_hal_pwr_ex.c \
	$(HAL_DRIVER)Src/stm32l4xx_hal_rcc.c \
	$(NUCLEO_BSP)stm32l4xx_nucleo_32.c \
	$(CMSIS_DEVICE)Source/Templates/system_stm32l4xx.c \
	startup_stm32l432kcux.s \
	src/stm32l4xx_it.c \
	src/main.c

The rdimon specs file is used for semihosting, which you will need later for debugging. Once you run make you should get a new a.out file. You can use the common file command or arm-none-eabi-objdump -f to see that the ELF is for an ARM architecture.

Let’s now flash the MCU with this new file. Plug the dev board to a USB port, then open a new terminal and run:

openocd -f interface/stlink.cfg -c "transport select hla_swd" -f target/stm32l4x.cfg

You should see OpenOCD detecting a Cortex-M4 processor and informing us that GDB is ready to accept new connections on port 3333. Now in another terminal let’s use GDB to flash our device:

arm-none-eabi-gdb -q a.out

target extended-remote :3333
monitor program a.out verify

A message indicating that the programming has finished and successfully verified should appear on the screen. We won’t see any blinking LED yet because the state of the MCU is halted (you can see this by running monitor targets), so start our program inside GDB do:

monitor reset run

Now you should see the small green LED blinking. You can exit GDB with q and kill OpenOCD with Ctrl-C.

To finish this tutorial, let’s try some basic debugging capabilities. Open the main.c source and look for the while loop where the GPIO port connected to the LED gets toggled. Add a printf("toggled\n"); anywhere inside the loop and include stdio.h at the to of the file. Now recompile and reprogram the MCU with this new binary. You may notice the green LED turns on but doesn’t blink, this is because the MCU is halted awaiting for a host to connect. In the GDB session do:

monitor arm semihosting enable
load
monitor reset run

At this point you will see that in the OpenOCD terminal our message gets printed every 100 milliseconds. You can do all sorts of things now in GDB, like seeing the current point of execution with the l command, or halting the execution and stepping with monitor halt and the s command.

As low as ~10 μA in run mode, and in the order of tens of nanoamps in standby mode↩︎
I won’t go into details here but for instance it just didn’t generate a Makefile for me↩︎

Interesting Articles I’ve Read in 2024

Brett Mullins — Tue, 31 Dec 2024 02:00:00 GMT

Below is a collection of interesting articles I’ve read in 2024. Three papers are on differential privacy and adjacent topics. There’s a recent method for differentially private SGD utilizing methods from private query answering, an intuitive watermarking scheme for language models, and a paper from 1986 that proposed $k$-anonymity before it was formalized as a criterion for de-identification. Several papers are on the history of ideas ranging from early twentieth century pragmatism, the synergies and antagonisms between poetry and philosophy, and the relation between periodicals and intellectual progress to the deaths of Analytical Marxism and Effective Altruism.

Trackmania Tracker

Jan's Space — Tue, 31 Dec 2024 02:00:00 GMT

Container load balancing with Linux bridges, veths, and IPVS

Monroe Clinton — Tue, 31 Dec 2024 02:00:00 GMT

I’ve been working on a project that takes requests and load balances them between containers. Using Docker or Kubernetes makes this easy, but I wondered how these systems work, so I decided to try and implement it myself. I’ve been using containerd to programmatically create and manage containers. It works quite well for this task, however it does not provide networking to or between containers. This is a problem for the load balancing part of my project, so I searched for a solution.

How I run a coffee club

Notes on software development — Tue, 31 Dec 2024 02:00:00 GMT

I started the NYC Systems Coffee Club in December of 2023. It's gone pretty well! I regularly get around 20 people each month. You bring a drink if you feel like it and you hang out with people for an hour or two.

There is no agenda, there is no speaker, there is no structure. The only "structure" is that when the circle of people talking to each other seems gets too big, I break the circle up into two smaller circles so we can get more conversations going.

People tend to talk in a little circle and then move around over time. It's basically no different than a happy hour except it is over a non-alcoholic drink and it's in the morning.

All I have to do as the organizer is periodically tell people about the Google Form to fill out. I got people to sign up to the list by posting about this on Twitter and LinkedIn. And then once a month I send an email bcc-ing everyone on the list and ask them to respond for an invite.

The first 20 people to respond get a calendar invite.

I mention all of this because people ask how they can start a coffee club in their city. They ask how it works. But it's very simple! One of the least-effortful ways to bring together people in your city.

If your city does not have indoor public spaces, you could use a food court, or a cafe, or a park during months where it is warm.

For example, the Cobble Hill Computer Coffee Club is one that meets outdoors at a park.

Good luck! :)

How I run a coffee club, a short guide for others who might be interested in running one. It's very simple!https://t.co/UgRWDQOA3v pic.twitter.com/5wYrLW7u6D
— Phil Eaton (@eatonphil) December 31, 2024

2024

La Vita è Bear — Tue, 31 Dec 2024 02:00:00 GMT

旅行

2024 flight map

今年因为种种原因飞了五个跨太平洋的往返，应该是我有史以来飞得最多的一年了。

不过今年旅行最大的亮点是去阿拉斯加的 Katmai 国家公园看熊抓三文鱼吃。这个我单独写过一篇游记：

A bear standing up in the river with tongue out

影视

今年看了五十几部电影，因为长途飞得多有挺大比例是在飞机上看的。印象比较深的几部是：

算是华语片的大年了。

娱乐

今年继续没有玩几个游戏。到了年底觉得印象深刻值得一说的就是 Animal Well, Astro Bot, 和 Balatro 这三个。

桌游方面今年主要就是在玩 Slay the Spire 的桌游版，跟同伴一起已经通关两次了。

今年开始从图书馆借电子书在 Kobo 上读，目前读了三本。

New Year, New Logo

Reilly Tucker Siemens — Tue, 31 Dec 2024 01:37:07 GMT

I've wanted a more recognizable online "brand" for years. I took a stab at creating something myself a few times, but was never satisfied with what I came up with so I finally asked a friend to design a logo for me. The process was delightful and I'm very happy with the results.

The CP/M emulator runs on Windows, maybe!

Steve Kemp's Blog — Tue, 31 Dec 2024 01:00:00 GMT

Today I made a new release of my CP/M emulator and I think that maybe now it will run on Microsoft Windows. Unfortunately I cannot test it!

A working CP/M implementation needs to provide facilities for reading input from the console, both reading a complete line of text and individual keystrokes. These input functions need to handle several different types of input:

Blocking, waiting for input to become available.
Non-blocking, returning any pending input if it is available otherwise nothing.
With echo, so the user can see what they typed.
Without echo, so the keys are returned by not displayed ot the user.

In the past we used a Unix-specific approach to handle the enabling and disabling of keyboard echoing (specifically we executed the stty binary to enable/disable echos), but this release adds a more portable solution, based around termbox-go which is the new default, and should allow our emulator to work on Microsoft Windows systems.

We always had the ability to select between a number of different output drivers, and as of this release we can now select between multiple input drivers too - with the new portable option being the default. This has been tested on MacOS X systems, as well as GNU/Linux, but sadly I don't have access to Windows to test that.

Fingers crossed it's all good now though, happy new year!

Culture talk: What I learned from building a distributed database that never took off

Bytepawn - Marton Trencseni — Tue, 31 Dec 2024 01:00:00 GMT

Recently, I delivered a culture talk on Scalien, my old database startup that never took off. I discussed the business, product and some technical challenges and lessons learned from developing ScalienDB, a Paxos-based distributed NoSQL database.

Unclenching

Aswath Krishnan — Mon, 30 Dec 2024 21:58:34 GMT

We were bringing over a fragile gingerbread house, precariously placed on the floor of our car’s back row, to a Christmas party. With every stop, swerve, and bump, I felt an involuntary pang of worry. My body tensed, my thoughts raced, and I caught myself clenching, both mentally and physically.

The gingerbread house arrived safely, and everyone loved it. But the ride wasn’t enjoyable for me. The concern and the clenching—it didn’t actually contribute to the house’s safe arrival. It just made the trip unpleasant.

Looking back, I realized this wasn’t just about a gingerbread house. This is how most of us go through life—clenched. We’re constantly on edge, worrying about what others think, stressing over unfinished tasks and goals, craving more, fearing failure, envious, judgmental, or bracing ourselves for what might go wrong. Even when things are fine, we’re often stuck in a low-grade state of tension.

A lot of this clenching is automatic and subconscious. It happens so fast, in microseconds, and so often that we hardly notice it. We just feel the aftereffects—tension in our necks, knots in our stomachs, a vague sense of unease. And while it might seem harmless, this constant clenching robs us of something crucial: presence and joy, almost like a blossoming lotus strangled by weeds.

To be fair, it mattered to us and others that the gingerbread house arrived intact. But the clenching? It didn’t help. We’d already done everything we could—driving carefully, avoiding sharp turns. Beyond that, the clenching was just noise, an exhausting habit my brain didn’t know how to let go of.

The truth is, clenching rarely changes the outcome of a situation. It’s our mind’s way of trying to “do something” even when there’s nothing left to do. But instead of helping, it keeps us trapped in a loop of worry and tension.

The key is learning to notice the clenching in the first place. This is harder than it sounds. Most of us are so used to living in this state that it feels normal. We don’t just experience clenching; we become it. We get caught up in our stress, fears, and frustrations, losing the ability to step back and see them for what they are.

Much of Buddhism, and various contemplative paths, revolve around recognizing and letting go of these clinging states. Meditation, mindfulness, and introspection help us spot the roots of tension and either let them go or do something useful about them. It’s about pausing long enough to notice: “Oh, I’m clenching right now.” That simple act of noticing—without judgment—can be a game-changer. When you shine a light on your tension, it starts to loosen its grip.

Unclenching doesn’t mean we stop caring or trying. It just means we approach life with less resistance and more acceptance. Peace isn’t about adding something new to your life; it’s about letting go of what’s weighing you down - unclenching. And joy is just peace in motion.

So, the next time you catch yourself bracing—your jaw tight, your mind racing, your stomach knotted—pause. Take a breath. Ask yourself: “Is this clenching helping me or telling me something important? Is there something more I can do about the situation” And if the answer is no, let it go.

Because life isn’t just about arriving in one piece. It’s about enjoying the ride.

Adding Pinch-To-Zoom to the Chart

Little Green Viper — Mon, 30 Dec 2024 21:56:12 GMT

In this exercise, we’ll add a “pinch-to-zoom-in/out” feature to the chart. It will be very basic, but should work quite well. The Basics The zoom will be X-axis only. It doesn’t actually make sense to zoom in on the Y-axis, especially since we have the ability to select a bar, and get the exact values ...

AdditionalAuthorizationParameters in ASP.NET Core 9

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Mon, 30 Dec 2024 18:42:55 GMT

In ASP.NET Core 9, a new feature called AdditionalAuthorizationParameters allows you to customize OAuth and OpenID Connect (OIDC) flows more quickly. This new feature allows developers to add custom authentication parameters without needing to rely on the complex workarounds that existed before ASP.NET Core 9 was released. Sounds familiar? Then you’re going to like this! […]

The post AdditionalAuthorizationParameters in ASP.NET Core 9 appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

Amiga 500 Samsung Keyboard Repair

hackup.net — Mon, 30 Dec 2024 18:14:07 GMT

Recently, an Amiga 500 in generally good condition found its way to me. It’s my first Amiga ever, because back in the day I was loyal to Commodore’s 8-bitters for a long time. So I missed the heyday of the … Continue reading →

The post Amiga 500 Samsung Keyboard Repair first appeared on hackup.net.

Review of 2024

extua — Mon, 30 Dec 2024 18:00:00 GMT

I’m ending out this year on holiday in Brittany with Eileen and my family.

Memories of Noah Gibbs

Matthew Gaudet — Mon, 30 Dec 2024 17:11:01 GMT

I was shocked to discover last week that Noah Gibbs had passed away.

Memories are being collected for his family here. I thought I would share mine here as well.

“My recollection of Noah is of a kind generous man who contributed tirelessly to the things he believed in.

I first encountered Noah after I gave a conference talk calling for better benchmarks in the Ruby community. At the time I was working on a doomed Ruby JIT compiler.

After that talk, Noah would go on the build Rails Ruby Bench. We corresponded a bit about it, and I started keeping track of his writing on the internet.

Eventually I moved on from Ruby. Still, I kept track of Noah, and would read his writing when I came across it. He clearly was someone who thought deeply about the nature of building software, and what it was to be employed in this industry that demanded production from something that from the inside felt more like craft.

You could see his commitment to community and mentorship in the things he wrote about, the podcasts he took, etc.

Recently I had occasion to reach out to Noah again. My email started “Hey Noah, I don’t know if you remember me” — I was truly gratified that his reply started “Hiya, Matt! Yes, I remember you. :-)” — then he generously answered my questions, fully in the spirit intended.

I had always hoped to run into Noah again, to cross paths with him. I am so incredibly sorry for your loss. I hope this message helps in the tiniest bit to illuminate more of the ways in which Noah touched people, all over the world.
”

— Matthew Gaudet, Edmonton, Alberta

Noah had a broad reach, but if anyone reading this has their own recollections, please send them along to his family.

How to Get Package Updates in Hard-to-Reach Places With RenovateBot

SeanKilleen.com — Mon, 30 Dec 2024 16:00:00 GMT

No dependency left behind.

2024 in review

callmeolive — Mon, 30 Dec 2024 14:00:00 GMT

2024 felt more like a proper year than the last.

Amol Rajan casually dropped the sickest sample known to man, internet rapper Viper got arrested for kidnapping, then an ITV drama brought the Post Office Horizon scandal to the front of public consciousness making us all wish there were TV dramas about everything else wrong with the country.

Willy’s Chocolate Experience took the world by storm with how hilariously abysmal it was. It taught people to avoid AI-Generated rubbish until they didn’t. Formerly Twitter went wild over a 20-year old advertising gimmick and for a fortnight every video became an ode to a Chilean beer.

Lost Media fans rejoiced as Ulterior Motives was finally discovered in a… Uh. Awkward. On top of that, The Most Mysterious Song on the Internet was finally identified, as well as a sodding Mozart piece of all things.

Also the Northern Lights went wild in May.

At the halfway point, our Prime Minister called a General Election and bottled it, Kien released for the Gameboy Advance, and Cypress Hill finally played with the London Symphony Orchestra. The Crowdstrike failure caused a right kerfuffle and brought a chunk of the world to a standstill, and Donald Trump got an ear piercing. 💅

I don’t really have notes for the rest of the year. Concord released and immediately shut down, a capybara escaped a zoo, Luigi became Nintendo’s most popular character somehow, and there was a new Wallace and Gromit film. That about does it.

Oh, and the felled Sycamore Gap tree shows signs of life. Things can’t be that bad.

In terms of web stuff, there have been better years. Big Tech is still hellbent on enshittification, anything to make a quick buck. AI Slop is inescapable now and has ruined image search. Google are still fighting adblockers, Mozilla is now into advertising, and even NaNoWriMo has jumped on the generative AI bandwagon for some reason.

But despite all of this, it’s worth mentioning that people want the online world to be better.

Cohost unfortunately shut down, but it shown us all that social media can be better for everyone. The indie web is still thriving, community wikis are clawing back at the predatory grip of Fandom, and in response to X going to hell there’s been a mass exodus to Bluesky – whilst I doubt their decentralisation plans and don’t see anything stopping it from becoming Twitter 2.0, it’s certainly a nicer place.

The stats

6 blog posts
11 books read
- Favourite: The Disaster Artist
17 Films watched
- Favourite: Eyes Wide Shut
9 TV shows watched
- Favourite: Ashes to Ashes
13 Games played
- Favourite: Disco Elysium
Favourite song I listened to: Tiger King
8 games I bought on sale then never played
2 hours spent lying on the floor listening to Ultravox’s Vienna
1 time my gay cat caught something
7½ hours spent waiting in A&E only for everything to be okay
Word of the year: Desultory

Blog posts wot didn’t make the cut

Fiction and Vulnerability – I’m not a writer, but I’d occasionally like to try it. Writing fiction and presenting it to the world (even as a trivial challenge for this blog) feels way more daunting & vulnerable than writing about childhood bullying.
The 2024 Election – The Conservatives had nothing to show but decline after 14 years in power and Labour was the only real alternative. Simple as.
What’s up with Skeletons? – Exploring why you can put skeletons into any situation without them being out of place. I was hoping to release it on Halloween but I got preoccupied with porting my blog to Hugo and… Meh.

Conclusion

Right, that’s 2024 done with.

For me it’s been an eventful non eventful year. I finally quit a job I was too scared to leave, but familial commitments have left me in an odd position for the time being. While I don’t have the highest hopes for 2025, if I play my cards right it might be the start of a transitional period for my life. Gotta have hope innit.

While I have faltered, this has been the year where I’ve accepted that I should try to be better. I’ve accepted that I can’t give up on my health. I still get overwhelmed by my thoughts and wish I could just give up, but there’s something inside of me worth caring for. Also it turns out that yes, I can lose weight… Just ignore what I put back on over Christmas.

Is there anything I wish for in the future? ~~For things to stop being so annoying~~ Temperance and fortitude ought to be the big one, building up mental endurance would help a lot with navigating life. Add in some self-love, stop the self-deprecation and hey, maybe I can feel great about myself.

I’ll need help in figuring out things about who I really am, and I’ll need to steady myself to face it all. It’s big, it’s scary, but I think I can take on another year.

I’d like to thank my friends for being there for me this year – you know who you are. Thank you for the comfort through the low periods, and laughs through the highs. You’ve managed to keep me sane throughout all of this.

Please be kind to yourself xx

Olive

Love for the WDC 65C02

mass:werk – Now Go Bang! — Mon, 30 Dec 2024 13:10:00 GMT

WDC and Rockwell related additions to the 6502 instruction set sheet.

The Power of Now by Eckhart Tolle

Apurva Shukla's RSS Feed — Mon, 30 Dec 2024 11:31:39 GMT

⭐ ⭐ ⭐ ⭐ ⭐ The Power of Now was a deeply insightful read, that I got a lot out of. Flipping through it’s pages, the act of reading itself…

The Housemaid by Freida McFadden

Apurva Shukla's RSS Feed — Mon, 30 Dec 2024 11:29:02 GMT

⭐ ⭐ ⭐ ⭐ The Housemaid was a thriller in every sense of the word. I haven’t really experienced many books like this, where the pages seemed…

Tomorrow, and Tomorrow, and Tomorrow by Gabrielle Zevin

Apurva Shukla's RSS Feed — Mon, 30 Dec 2024 11:25:45 GMT

⭐ ⭐ ⭐ ⭐ Tomorrow and Tomorrow and Tomorrow is a read that attempts to span a lifetime. It begins with Sam, crippled at a young age…

The spiritual and religious background of a childhood

Jason O'Neil — Mon, 30 Dec 2024 11:20:35 GMT

Every year Jesus’ parents went to Jerusalem for the Passover festival. Luke 2:41 (NLT) My favourite podcast for a decade now has been On Being with Krista Tippett, and she opens every interview with the same question: “Can you describe the spiritual or religious background of your childhood? (In whatever way you interpret that question).” […]

Attention pendulum (goal setting)

Relentless Simplicity — Mon, 30 Dec 2024 02:51:35 GMT

The Nada 65 Panda: A QMK Success Story

Terminal Thoughts — Mon, 30 Dec 2024 02:00:00 GMT

The Cerakey Nada 65 Panda is the world’s first keyboard featuring ceramic keycaps, designed to deliver the smoothest and thockiest typing possible. When I first heard about it, I knew I had to try it - the promise of ceramic keycaps was too intriguing to pass up. The keyboard arrived in beautiful packaging, and the build quality exceeded my expectations.

Books read in 2023

Avy Faingezicht — Mon, 30 Dec 2024 02:00:00 GMT

Still trying to catch up on my lack of writing this year, here's the long awaited list of the books I read in 2023. Less than I wanted, but I'm happy with the mix. On top of a few great fiction books, I read books spanning classics, history, and economics. Unintentionally, I also ended up reading a bunch of memoirs. Some recurring themes were the exploration of memory and identity, the nature of reality, and the consequences of technological change. Beside the books below, I also read some technical books to sharpen my skills as I picked a stack for InScope's MVP. Those included William S. Vincent's _Django for Professionals_ and Miguel Grinberg's _React Mega-Tutorial_ and _SQLAlchemy in Practice_ (keeping that monolithic stack on Postgres a while longer would have been a better idea, but that's beside the point of this blog post). I also read parts of _Designing Autonomous AI_ by Kence Anderson while doing the Coursera Decision Making and Reinforcement Learning class, and on the more managerial side also _Crucial Conversations_ by Patterson, Grenny, McMillan, and Switzler and _The Founder's Dilemmas_ by Noam Wasserman.

The Book of the New Sun - Gene Wolfe

I started the year with the first two volumes of a book that had been highly recommended but which I ended up really disliking. Set in a far future where medieval aesthetics coexist with advanced technology, The Book of the New Sun by Gene Wolfe follows Severian, an apprentice torturer who is exiled for showing mercy to a prisoner. The story is told from Severian's perspective, challenging the reader with his unreliable narration. Before his expulsion, Severian had acted as a obedient cog in the bureaucratic machinery of The Autarchy, the all powerful authoritarian regime in his society, and the book deals with his reckoning of his new role as a lower level traveling brute. The world building is rich, but the plot is slow and I found the writing style and language to be unnecessarily pretentious. To boot, many of the plot lines were overt allegories of Christian theology, which I didn't care for. With its focus on semiotics, the nature of knowledge, and bend towards medieval history, at times I wished I had just been reading Umberto Eco's fiction instead. I do not plan to finish the series.

Skin in the Game - Nassim Nicholas Taleb

This is the fourth book I read from Nassim Nicholas Taleb's Incerto series, missing only his book of aphorisms, The Bed of Procrustes . In his own words, Skin in the Game is a book about: a) uncertainty and the reliability of knowledge b) symmetry in human affairs, that is, fairness, justice, responsibility, and reciprocity c) information sharing in transactions d) rationality in complex systems and in the real world Also in his own words, "the entire point of the book is that in the real world it is hard to disentangle ethics on one hand from knowledge and competence on the other." This focus on ethical decision making comes down to accountability by having personal stakes in the outcome of the decisions one makes on behalf of others – skin in the game. It is really aligned with my libertarian leaning worldview. Making decisions for ourselves is already hard, so we should be extra careful when making decisions for others whose reality we can't possibly understand as well as our own. Taleb argues that people make better, more cautious, and more ethical decisions when they personally stand to lose something from the outcomes, preventing reckless risk taking. Without skin in the game, decision makers (think policymakers, corporate executives, or bureaucracies) end up reaping the benefits if things go well while transferring risks to others (taxpayers, employees, customers) when they don't. This is a clear reason to build institutions with aligned incentives and build natural checks against the unintended consequences of harmful decisions. The book is an explicit critique of crony capitalism and bad incentive structures in every day life. The world is too complex to be explained by sound bites and top down solutions. Taleb's writing style in this book is just as arrogant and condescending as in previous works of his I had read, but his ideas are thought provoking and, I tend to think, correct, so it is worth putting up with his tone.

Call Center Optimization - Ger Koole

I read this book when I was exploring an idea in the revenue operations space, just before deciding to start InScope. The insight I was trying to apply came from a problem I had seen at Vouch, where complex sales required many cross team handoffs. As in other businesses, managers often didn't have visibility outside their silo, leading to local optimization. The theory of constraints tells us that businesses waste resources improving anything but the bottleneck, losing revenue and increasing costs. I wanted to apply this to staffing and prioritization in complex sales teams, but I needed to get a refresher on queueing theory first. This book was a suggestion from my Kellogg ops professor, Martin Lariviere. My idea was simple: to automatically map entire sales motions using data that companies are already collecting (in Salesforce, Zendesk, etc), and then to use that data to optimize the sales process. Analyzing the pipeline end to end could allow the team to identify the limiting factors keeping deals from closing. This would mean focusing on the most impactful changes or new hires to make, rather than the most visible problems or the loudest executive's views. Koole's book is a comprehensive guide to the mathematical modeling of call centers, and those models are analogous to other operations management problems. It covers the basics of queueing theory, Erlang formulas, and the mathematics of staffing and scheduling. I found it useful to brush up on the math, re learn the terminology, and to get a sense of the complexity of the problem. It was a good starting point for a project I didn't start.

Confessions - Augustine

I read Augustine as part of a Catherine Project class, motivated by Martin Hägglund’s repeated references to it in This Life . Confessions is Augustine’s autobiographical account of his youth and conversion to Christianity, exploring themes of personal responsibility, free will, and sin. It’s a foundational text for Christianity, with enduring relevance in Western thought. I latched on to Augustine’s focus on language, and what he calls "the semantic problem," our struggle to communicate a shared reality. The book begins with his reflections on learning to speak as a child, distinguishing learning by study from learning by experience—a concept that feels surprisingly modern. His introspection on this topic, and especially Book X’s discussion of memory, imagination, and free will, reads almost like Wittgenstein or Douglas Hofstadter. The book has many emotional scenes of grief and family drama. One poignant scene recounts Augustine’s encounter with a drunken beggar in Milan, contrasting his own anxiety driven ambitions with the beggar’s fleeting joy. This moment encapsulates Augustine’s struggle between temporal pleasures and eternal joy. Like the strikingly modern discussion on language I mentioned earlier, this passage describes an experience that someone like me could easily have walking the streets of San Francisco in the 2020s. As a non religious reader, I found myself rolling my eyes often. Despite its deep insights, Confessions offers incomplete answers, making it a thought provoking but unresolved meditation on human existence. Find my more detailed, more meandering review here.

The Great Gatsby - F. Scott Fitzgerald

When talking about classic literature, my friends are always surprised I never read any of the books they were forced to read in high school. Growing up in Costa Rica, the canon I was exposed to barely included any US authors (for some reason the only one I recall having to read is Poe). My knowledge of US classics is pretty limited, so when Hannah suggested that we read The Great Gatsby together, I was excited to finally get to it. Set in the Roaring Twenties, it is a classic rags to riches novel about love, wealth, and to some extent a critique of the American Dream. The novel shows how the relentless pursuit of wealth and status erodes authenticity and meaning, reducing everything to a spectacle. The story revolves around new and old money, and the upper class' hope for a permanent spot in society contrasted with the promise of upward mobility that keeps the lower class engaged in the system. Crime, corruption, and sex were all much more present than I'd have expected for a novel written a hundred years ago and prescribed in the average US high school curriculum. It was a fun read, and I'm glad that I now understand the shared references to Gatsby that pop up every once in a while, but I expected it'd be more profound given its popularity.

La Hojarasca - Gabriel García Márquez

I picked up La Hojarasca ( Leaf Storm ) during my trip to Colombia. It is the short novel where García Márquez introduced his fictional town of Macondo, which would later serve as the setting for One Hundred Years of Solitude . Like in his other works, García Márquez begins with a dramatic scene and ties the reader in knots through the complexities of his layered narration. The book tells the story of a reclusive doctor who commits suicide, leaving his burial to a contentious promise upheld by one of the main characters, the aging Colonel. The doctor’s isolation —rooted in his refusal to treat wounded soldiers and his questionable relationship with his indigenous housemaid— earned him the town’s scorn before his death, forcing the colonel into a moral struggle on his behalf. As the narrative unravels the doctor’s mysterious past, it highlights the intersecting burdens of memory, war, and loyalty in a decaying Macondo. The book’s brevity makes it easier to hold on to the details and characters and to fully appreciate its complexity —something I struggled with in One Hundred Years of Solitude . Yet, despite its concise form, the novella is a powerful exploration of history’s impact and the ways unresolved conflicts and buried truths shape the lives of individuals and communities alike. The title refers to the literal litter accumulated in the town —most vividly in the doctror's abandoned house— but also metaphorically in the unresolved conflicts and tensions that haunt Macondo’s inhabitants. This "leaf storm" operates on both personal and political levels, from interpersonal betrayals to the larger upheaval brought by the arrival of the banana company, whose capitalist exploitation is hinted at through broader themes of historical and social decay. That last topic, unfortunately, was barely touched upon in the book, so it left me eager to explore more of Márquez's work.

Acid for the Children - Flea

I've been a huge Red Hot Chili Peppers fan forever. Their 1999 album Californication was one of the first records I consciously chose to buy, and I've been listening to them since. Acid for the Children is Flea's (the band's bassist's) autobiography. I got to watch them at Bottlerock in Napa, which inspired me to listen to the audiobook, narrated by Flea himself, and really enjoyed it. Acid for the Children is a coming of age story that follows Flea from his "normal" childhood in New York City, and splitting time in Australia after his parents' divorce, to the harsh transition to his wild teenage years in Los Angeles after his mom moves the family there. In LA, he gets interested in jazz through his stepfather's influence and picks up the trumpet. Attending high school there, he meets the friends with whom he'd later form RHCP, and is introduced to the chaotic world of punk rock, and the drugs and mayhem of the LA scene. The book is raw, filled with stories of his struggles with his identity, addiction, poverty, and loss. Describing the violence at home and the chaos of his life skipping school and living on the streets, Flea writes with a sense of humor and self awareness that makes the book both heartbreaking and uplifting.

The Storyteller - Dave Grohl

The Storyteller is Dave Grohl's autobiography, which I also listened to as an audiobook narrated by the author. I was inspired to read it by a great show, too – the band headlined Outside Lands in '23, and it might be one of the best live performances I've ever seen. Whereas Flea's book focuses on his life pre RHCP, The Storyteller spans the milestones of Grohl's career and what came after success. It almost comes off as bragging. While it's mostly positive and inspiring, loss is also a recurring theme throughout the book. There's Kurt Cobain's suicide, the obvious instance any reader expects, but his childhood friend's Jimmy Swanson's death weighs even heavier than Cobain's, at least in writing. Grohl's book is very family oriented, crediting his mother for his early interest in music, while critiquing his dad for his lack of support for his musical ambitions. He describes getting into punk through his "cousin" in Chicago during a summer road trip, and how that shaped his musical tastes and his career. The book chronicles his rise to becoming rock royalty, climbing the ranks from the DC punk scene to Nirvana and then Foo Fighters, with surreal anecdotes about performing at the White House for Obama and his close friendship with Paul McCartney. The vignette about the Beatle teaching Grohl's daughter to play piano was a far cry from his punk days. I was not that into Foo Fighters until much later in life, but I'm glad I eventually found them. While Grohl's recent admission of infidelity violates a lot of the good boy persona he tries to construct in this book, I still think it is worth the time.

Slouching Towards Utopia - J. Bradford DeLong

I've been reading DeLong's blog for 10+ years now, so I was excited to finally read one of his books with Slouching Towards Utopia . Blending economic history with incisive commentary on human aspiration and failure, DeLong creates a grand narrative of economic theory and ideology meeting the actual political actions of the last hundred years and change. DeLong clearly marks 1870 as the beginning of an unprecedented era of economic transformation and sustained economic growth, cataloguing key ideas, players, and events through the present. He calls this period "long twentieth century." Admitting the end of his named period is fuzzy – perhaps the 9/11 attacks, the '08 crisis, or even the rise of Trump or COVID – he goes with 2016 due to the regime change in the US, and leaves the question open for future scholars. His core argument is that this period saw a growing belief that humanity could solve political economic problems and escape the Malthusian trap by improving governance and using knowledge. This belief took many shapes, bearing fruit fueled by technology and globalization. However, the compounding effects of progress came with a century of dislocation. Noting that the rise of industrial capitalism happened in tandem with mass migration and new ideologies, the book repeatedly highlights how ideas shaped economic and political outcomes. DeLong explores the efficiency gained from markets, the evolution of their dynamics, and their discontents, from Hayek's celebration of decentralized problem solving and Schumpeter's "creative destruction" to Keynes' intermediated market system and Polanyi's double movement critique. He also discusses the social experiments in Europe and the parallel fascist embrace of nationalism and war, underscoring how beliefs — both utopian and dystopian — redirected historical trajectories. Focusing on the US and its leaders, DeLong argues that while their role was exceptional, many of the policies that they coalesced around came up in large part due to luck and path dependence — the contingency of who was in the seat at key moments. The book is both an optimistic testament to progress and a sobering reminder of its limits. The marriage of market efficiency and state intervention that the Allies embraced (and pushed) after WWII, as seen in mid century social democracies, worked temporarily but ultimately unraveled. Despite technological advancements that granted humanity "godlike powers," societal challenges — inequality, nationalism, and environmental crises — have proved stubborn. So DeLong asks why, with so much capacity, do we remain so far from utopia? He challenges readers to think critically about whether humanity’s slouch toward utopia is inevitable or if renewed efforts are required to build a world worthy of our technological and organizational capabilities. I am glad he is asking these questions, because too many people take progress for granted.

Grit - Angela Duckworth

Grit explores the idea that passion and perseverance are more critical to success than innate talent, encapsulated in Duckworth's memorable "effort counts twice" formula. Through anecdotes ranging from her research on which West Point cadets drop out and which ones succeed to studying Olympic athletes' training regimes, Duckworth illustrates how sustained effort often outpaces raw ability in achieving long term goals. Her central thesis is compelling, and the blend of research and real world stories makes the book approachable, though it’s more inspirational than groundbreaking in its insights. That said, the book was pretty forgettable (something I can attest to a year and a half after listening to it). An hour long podcast interview or even her short TED talk would have sufficed for me, in retrospect. Grit is simply pop psychology, with the tone of a motivational airport read instead of the rigorous scientific exploration I was hoping for.

Tale of love and darkness - Amos Oz

Until last year, Amos Oz was likely the author who had languished the longest in my to read list. After sitting in the queue for at least 15 (!) years, I had the (un)fortunate timing of deciding to read him during my trip to Israel a few months before the conflict exploded again. A Tale of Love and Darkness is Oz's memoir, covering his childhood in Jerusalem and his later life in Kibbutz Hulda, along with a very personal telling of his family's history. The book is a meditation on family, the nature of memory, and the ways in which we construct our own narratives. The book takes intimate moments like planting and keeping gardens with his dad or reliving the grief after his mom's suicide, and interweaves them with Oz's first person chronicle of the nascent state. It is a moving portrait of a family, a country, and a time. Even in English translation, you can feel the rebirth of the Hebrew language and the formation of Israeli culture through its pages, as Oz captures both the ancient roots and tells us of his family's involvement in the creation of the nation. The formation of the Israeli identity was not accidental, and you can see traces of it throughout the whole story. Much of the book deals with the War of 1948. While I have a lot to say about Oz's view on the Zionist movement of the 30s and the 40s, and its leaders, I will instead just reproduce a section that stuck with me, and probably struck me more than anything else in the book: I asked Ephraim if he had ever, in the War of Independence or during the troubles in the 1930s, shot and killed one of those murderers. I could not see Ephraim's face in the dark, but there was a certain subversive irony, a strange sarcastic sadness in his voice as he replied, after a short pensive silence: "Murderers? What d'you expect from them? From their point of view, we are aliens from outer space who have landed and trespassed on their land, gradually taken over parts of it, and while we promise them that we've come here to lavish all sorts of goodies on them—cure them of ringworm and trachoma, free them from backwardness, ignorance, and feudal oppression—we've craftily grabbed more and more of their land. Vell, what did you think? That they should thank us? That they should come out to greet us with drums and cymbals? That they should respectfully hand over the keys to the whole land just because our ancestors lived here once? Is it any wonder they've taken up arms against us? And now that we've inflicted a crushing defeat on them and hundreds of thousands of them are living in refugee camps—what, d'you expect them to celebrate with us and wish us luck?" I was shocked. Even though I had come a long way from the rhetoric of Herut and the Klausner family, I was still a conformist product of a Zionist upbringing. [...] "In that case, what are you doing here with your gun? Why don't you emigrate? Or take your gun and go and fight on their side?" I could hear his sad smile in the dark: "Their side? But their side doesn’t want me. Nowhere in the world wants me. Nobody in the world wants me. That’s the whole point. It seems there are too many of my kind in every country. That's the only reason I'm here. That's the only reason I'm carrying a gun, so they won't kick me out of here the way they kicked me out of everywhere else." Reading this passage today, in the context of current events, is particularly poignant. Oz managed to capture both the complexity and the tragedy of the situation in a way that feels as relevant now as it was then. The book is a masterpiece of memoir writing, and a much needed perspective of the complexity of humanity in a region where narratives are too often reduced to simplistic binaries.

Kitchen Confidential - Anthony Bourdain

I was first drawn to Anthony Bourdain through his TV show Parts Unknown and picked up his memoir, Kitchen Confidential , after his suicide in 2018. The book, first published in 2000, offers irreverent glimpses into his early career —from dishwasher in Cape Cod to celebrity chef in NYC haute cuisine— largely crediting his peers rather than his formal education at CIA. It was published when celebrity chef culture was nascent, unveling gritty behind the scenes details of restaurant kitchens and received as an exposé. Although I hoped to get some cooking lessons out of it, the book focuses more on industry realities than culinary skills. The big reveal is how much butter and shallots are used in restaurants. Beyond the cooking tips, Bourdain candidly shares unsettling industry practices and personal struggles, like how high pressure environments fueled his drug issues. Despite these disclosures, his narrative is sharp, witty, and filled with a genuine passion for the craft. Some of the stories are surprisingly hilarious, and outright vulgar. Other highlights include his vivid recounts of travels to Japan and France, foreshadowing his future TV success. Kitchen Confidential is a great book for any foodie, especially one with a sense of humor.

La Invención de Morel - Adolfo Bioy Casares

Last year, I took a Catherine Project class on Bioy Casares, the first they offered in Spanish. I had been eager to explore Bioy Casares' work for years after reading Borges mentioning him and his work. La Invención de Morel ( The Invention of Morel ) is an early exercise in speculative fiction, blending philosophical inquiry with a narrative of obsession and isolation. The novel explores technology as a means to transcend mortality, dramatized through a fugitive narrator who stumbles upon a mysterious island with eerie, deserted structures and a group of odd tourists who seem to ignore him. He soon realizes these are not people but recordings replayed by Morel’s invention—a device that captures human essence in a perpetual, unaltered loop. Bioy Casares uses an unreliable narrator to emphasize the ambiguity of reality versus illusion, blurring distinctions between signifier and signified. Morel, both inventor and participant in his recordings, gives a speech to his guests' avatars which made me think of Cory Doctorow's take on mind uploading in Walkaway and Down and Out in the Magic Kingdom , particularly in relation to recent AI advancements. Unlike Morel's subjects' recurring lives, Doctorow’s approach is dynamic, where consciousness is uploaded to a digital medium, allowing characters and copies to exist as active players in virtual environments, offering an immortality that is interactive and evolving. We can trace a clear line from simple cameras and phonographs that motivated Bioy Casares in the 1930s to the early aughts' computers that inspired Doctorow, all the way through today's LLMs. The appearance of LLMs have begun to make the latter's "immortality" closer to reality, with "griefbots" simulating deceased loved ones by using data they left behind as input, for example. Both Bioy Casares and Doctorow explore the limits of technology to capture consciousness and the implications of extending existence through it, posing ethical questions about identity, continuity of self, and what it truly means to be alive. La Invención de Morel is a sharp commentary on the perils of mistaking images for reality, and reading nearly a century after its publication made this metaphysical puzzle on immortality feel remarkably prescient.

El Sueño de los Héroes - Adolfo Bioy Casares

Following La Invención de Morel , I read El Sueño de los Héroes ( The Dream of Heroes ), which I liked the least of the trio. The psychological thriller follows Emilio Gauna and friends as he tries to make sense of the disjointed events of a black out carnival night a few years prior. Gauna's recollections blend into a surreal mess, making the story a slow burn, with Gauna's life, memories, and dreams intertwining as he reconstructs the past. As the story unfolds, Gauna develops an ambiguous romantic relationship, and his friendship with people he previously admired unravels. Early in the book, Gauna meets Taboada, a seer who serves as a guide of sorts, nudging him toward revisiting the fateful carnival night. His insights and cryptic behavior suggest he understands more about Gauna’s destiny than he lets on, though. The novel builds a suspenseful recursive narrative where mundane choices ripple into monumental consequences, confronting the reader with a tension between free will and the deterministic pull of destiny. At its core, the novel is about memory and the deceptive nature of self perception. Themes of honor and masculinity thread Emilio’s journey, as he questions who his real friends are, and whether his actions are driven by his own choices. The novel's slow pace and intricate plot where dreams, fate, and reality overlap left me a bit lost. Perhaps the most interesting piece is the implicit contrast between the rich and educated with the poor as they navigate the neighborhoods of old Buenos Aires. The book is a challenging read, but some call it Bioy Casares' best work, so I might have missed some of the subtleties that make it a classic of Argentine literature.

Diario de la Guerra del Cerdo - Adolfo Bioy Casares

The last book I read for the Bioy Casares class was Diario de la Guerra del Cerdo ( Diary of the War of the Pig ), a dark comedy about generational conflict where a mysterious political movement causes young people in Buenos Aires to attack and threaten the elderly without apparent reason. The narrative follows Isidoro Vidal, a retiree who considers himself between the two age groups, as he becomes embroiled in the chaos of the escalating violence. The novel reflects on the fear of aging and the value society places on youth. It also touches on the cyclical nature of life, as young characters grapple with the realization that they too will one day be old, and targeted by the same violence. Bioy Casares uses the purge as an allegory for broader social exclusion and violence, warning of how easily communities can scapegoat others under the right conditions. Unfortunately, no one in the class knew much Argentinian history, but we agreed that it was an allegory for real political tensions in Argentina during the late 1960s, where much like in the rest of the world at the time, the youth were rebelling against the establishment.

Founder vs Investor - Elizabeth Joy Zalman and Jerry Neumann

After following Neumann's blog Reaction Wheel for years, I felt lucky that I got to attend the Bloomberg Beta book launch event in SF to hear Neumann and Zalman discuss Founder vs Investor . The book is structured as a set of dialogues exploring the relationship between these two groups, clarifying when incentives are aligned and when they grow apart, and how the dynamics play out over time as companies grow and more investors get involved. Zalman takes the vantage point of a founder, a role she's successfuly been in multiple times, while Neumann sits across the table as her former investor, and a venture veteran. The conversations cover a wide range of topics, from general advice on fundraising, to specific deals the authors were involved in and how they were negotiated, how to choose investors (or pick founders), dealing with boards of directors, specific fundraise terms worth negotiating for, and more. Needless to say, the two authors don't always agree, which is half the fun. As Neumann says, understanding "how the person on the other side of the table thinks is more important, practically, than what you believe." The book is a good exercise in building empathy and driving clarity between two groups that are forced to work together, but whose members can't really understand each other. Now who's writing Founder vs. Founder?

Blindness - José Saramago (translated by Giovanni Pontiero)

Blindness is an allegory of human fragility, community, and the veneer of civilization. Saramago’s story captures the chaos that unfolds when an epidemic of blindness suddenly hits an unnamed town, leaving its society grappling with the breakdown of their daily routines, norms, and eventually all the systems that held it together. The narrative, often brutal and always intimate, slowly ramps up from a single crime to full collective dehumanization as the disease spreads among unnamed characters. I was not particularly fond of Saramago's signature run on style, and the lack of punctuation made following some plot lines harder than necessary, but the book's themes and imagery make up for it in their powerful portrayal of human nature in the face of societal collapse. Existential inquiry resonates throughout. Saramago probes at how humanity frays when stripped of its structures and senses, with people in the story constantly questioning their inherited morality. At the same time, the characters try to remain "good," struggling to maintain their dignity through small, cleansing acts. The passages where the group discusses governance and the lines between justice and vengeance, or survival and self degradation, recurringly made me think of Holocaust stories, and how the people in real life are sometimes forced to make similar choices. Blindness is a masterclass dystopia, and a reminder of how fragile the social fabric we take for granted truly is.

You can find my lists from previous years here: 2022, 2021, 2020, 2019, 2018, 2017, and 2016.

Photo: Pop-up books, by me. Previously posted on A year of memories, in film.

Confessions, a short review

Avy Faingezicht — Mon, 30 Dec 2024 02:00:00 GMT

I read Augustine's _Confessions_ as part of a Catherine Project class. The book was repeatedly cited by Martin Hägglund in _This Life_, a book I had quite enjoyed reading a few years before, and he emphasized the nuance in understanding Augustine, so I chose to do a guided reading. _Confessions_ is an autobiographical work, recounting Augustine's youth and his conversion to Christianity. It is one of the foundational texts in Christianity, dealing with personal responsibility, free will, and sin. It's influence is clear in the the Western worldviews we inherited. The Chadwick translation's introduction comments on Augustine's fascination with words, and his attention to the trouble we have communicating with one another, which he calls _the semantic problem._ The book opens with a vignette about Augustine learning to speak as a child, and then on him learning Greek and Latin. I immediately latched onto this language theme, which echoes through the book. Communication requires, by definition, a shared understanding of reality. Religious texts, however, tend to point us in the opposite direction, assuming that without sharing a single objective truth or dogma there can't be real understanding. His argument distinguishing the mechanics of learning by studying from learning by doing or experiencing felt very modern. Throughout the book I was surprised by the deep similarity between his descriptions of his life, and how we experience ours 1600 years later. As someone who's not religious and who has little depth in Christian ideas, I read through the lens of good/evil and right/wrong, more than sin specifically. Not coincidentally, Augustine often invokes Manichaeism, a philosophy centered on this duality, as a contrast to his position that both good and bad come from the same source. The reading group struggled with this distinction: Our acts bring about changes in the physical world, and since those can be harmful they can be evil. Maybe, doing nothing doesn't bring about change, so we can't be responsible for what we don't do? But then, are we responsible for not stopping things from happening? Is it a sin for a bystander to stop a group of teenagers from stealing some fruit, and taking away their pleasure, or is that a positive for society? An objective morality seems prerequisite to this discussion, but Augustine just assumes it. At times, especially when discussing memory, forgetfulness, images, and memories about memories, Augustine reads almost like Wittgenstein or Douglas Hofstadter. Book X is where this line is clearest. He discusses forecasting, and questions whether one can even be certain that that world we seem to be in exists and, if so, whether it has some discernible pattern we can \"learn\" through our own existence. It made me wonder, what does Augustine make of imagination and creativity? How does he think about human capacity to extrapolate to what might happen, our actions and their consequences? There's another angle to the objectivity problem around what's real versus what's imagined that our group discussed. What would Augustine think about our Catherine Project classes over Zoom, and our virtual worlds, and the very rich life that XXI humans can build online? Are those better than the physical? Can you sin in the metaverse? Naturally, we landed in questions of free will. While the book has many emotional scenes of grief and family drama, the most striking to me was Augustine recounting his time in Milan, and his encounter with a drunken beggar while on his way to deliver a speech. Augustine contrasts his position with the drunkard's, prompting him to question his ambitions and the nature of true happiness. > For what he had gained with a few coins, obtained by begging, that is the cheerfulness of temporal felicity, I was going about to reach by painfully twisted and roundabout ways. True joy he had not. But my quest to fulfill my ambitions was much falser. There was no question that he was happy and I racked with anxiety. He had no worries; I was frenetic, and if anyone had asked me if I would prefer to be merry or to be racked with fear, I would have answered \"to be merry\". Yet if he asked whether I would prefer to be a beggar like that man or the kind of person I then was, I would have chosen to be myself, a bundle of anxieties and fears. What an absurd choice! Surely it could not be the right one. Much like the discussion on language I mentioned earlier, this passage could easily be a thought that someone like me could have walking the streets of San Francisco in the 2020s. The scene highlights the contrast between temporal pleasures and eternal joy, a theme that runs throughout the Confessions. The beggar's joy is fleeting and superficial, while Augustine searches for "true" joy and glory, which he believes can only be found in God. This realization leads him to renounce his worldly ambitions and embrace a life of faith and devotion. If only things were that easy. The book is deeply introspective, but only offers incomplete answers. I am glad I got to read it with a group, as I'm not sure I would have gotten as much out of it on my own.

Photo: Sagrada Familia, by me. Previously posted on Barcelona, 2024.

2024 in review

Codakuma — Mon, 30 Dec 2024 02:00:00 GMT

How 2024 went for me and my apps

Reflecting on 2024, preparing for 2025

ntietz.com blog - technically a blog — Mon, 30 Dec 2024 02:00:00 GMT

If you do things a few times, they're a tradition. This is the third time I'm writing one of these, so I guess it's an annual tradition now! This is where I reflect on the year that's been, and talk some about my hopes and goals for the next year.

Reflecting on 2024

This year has been a lot, and there are a few months of it that just feel like a black hole to me. That's because I got sick in the middle of it. I'm really proud of how much I got done in spite of being the sickest I've ever been. And I'm excited to see what I can do next year, now that I'm nearly fully recovered.

Professional

I spoke at a conference! This year marked my first ever conference talk. Technically, my first one was at SIGBOVIK 2024, but I'm really talking about !!con. I've submitted talks to conferences before and this is the first one I've ever had accepted. You can watch the recording. That link takes you to the playlist of all !!con talks from this (final) year, so please enjoy them all!

It was an incredible experience. The whole conference felt like I was with old friends who I just hadn't met yet. It made me remember the power of connecting with other nerds in physical space. And it reminded me of the joy of being on a stage. More on that in the personal section.

I wrote even more than last year. My goal for this year was to continue my status quo: publish at least one blog post each week. I overshot this again, with 60 blog posts over 90,000 words. The most important thing for me has been consistency. By writing every week, I've been able to continue to use this momentum to stretch my creative practice. This even held during my illness this summer, and it was something for me to hold on to when I could do little else.

I got paid for my writing. This is the first time I have been paid explicitly to write. I was sponsored to write a post about an open-source product (and the contract even requires that I misuse it, on purpose, since that's what I pitched). The overall experience was pretty good, and I'm also not sure I would do it again in the near future: while I'm working a day job, I don't want to spend my limited writing time on things I'm not already self-motivated to publish. But—I dearly want to find a way to get paid for my writing which isn't sponsored posts. This might look like a Patreon or similar, so let me know if you're interested, and you might be the nudge to get me over that finish line.

I started coaching people. This year saw me take on my first three coaching clients. These were all pro bono, friends of mine who needed some help with career questions and technical leadership development. It's been an incredible experience, getting to directly help people grow and overcome their challenges. (If you're interested in being coached by me on technical leadership, reach out to me! You're a particularly good fit if you're a senior or staff software engineer aiming to level up, and members of marginalized groups are who I'm most hoping to help.)

I grew as a leader. Most of my reflection here is private because it's so intertwined with specific leadership challenges at my day job, but it's been a really helpful year for me in my leadership development. I've learned a lot, and I've seen a lot of old decisions come around to their conclusion to complete my learning arc.

Personal

I got sick. It's hard to say when exactly I got sick, but my symptoms got to where I had to go to my doctor at the start of May, and my full recovery started in November. I am pretty sure I was sick before that, since I had started needing more and more sleep throughout the spring, but it's impossible to say at this point.

At the peak of my illness, I was in near constant pain (about a 6 on the pain scale) and could not stand up for 5 minutes without having tachycardia. If that happened, I had to lie on my back until my heart rate came back down. Walking around the block was an impossibility, when months prior I was running 25 miles a week and hauling the kids around on my bicycle.

I bounced through multiple doctors. We hit the end of my general practitioner's expertise, so she referred me to a GI practice since our core symptom was related to my liver and abdominal pain. (The liver was ultimately a red herring: what looked unusual on my ultrasound was, with further testing, not concerning.) The GI practice did a lot of tests, and I had a lot of waiting (GI docs are in such high demand, you can't see them), but ultimately... they also found nothing. Meanwhile I was still in pain, and could do very little.

Around this point, I went on medical leave with my employer. Before that I had been working as much as I was able, and contributing something of value, but it had become clear that I was going to need to focus on my recovery if I wanted to actually get to the bottom of this. Going on medical leave was terrifying, because it meant I would be without income for months. But it was ultimately the right decision.

Around this same time, I went to my third doctor. She got me a diagnosis. A friend sent me to her, since she's a specialist in conditions that present with ambiguous symptoms and chronic fatigue. This doctor ran a lot of tests—expensive tests, which insurance doesn't cover—and we ultimately got me a firm diagnosis six months into the whole ordeal. After starting treatment, it's been a very fast recovery: almost two months in, and I'm over 90% normal. How our definitions of "fast" shift. I may have relapses in the future, there's no way to know, but I'm relieved to know what is going on.

I stopped running. The pain started in May when I was running, so I had to abruptly go from running 25 miles a week to not at all. Eventually it moved from pain while running to constant pain. And then eventually... it faded away entirely, as long as I keep up on one of my medications. Now I can run if I want to—but I've decided not to.

For the last decade, I've identified as a runner. I did a few half marathons with reasonable times, and I did a marathon in 4:07, finishing in 75 F heat. For much of the decade before that, I identified as a cyclist. I was into cycling from the moment I could get on a bike, always wanting to go further and faster. In high school, I did a 100 mile bike ride.

Now that I've been forced to take a break from it, I've realized I'm pretty content with not putting in the grueling schedule needed to get back to the high level of performance I thought I wanted to target next year. I might get back into this sooner or later, but right now, I'm working on functional strength and being healthy and having more balance in my life.

I got back into music! When I was sick, I bought a wind synth and started playing music again. Then I started taking lessons with an incredible teacher who's an accomplished musician in his own right. And then I got a drum pad, another wind synth, a hand-me-down keyboard, and got my clarinet back out. I've fallen deep into this and I'm loving every single minute of it, frustration and all.

This is replacing a lot of the dedication and discipline I used to get from endurance exercise. It has the added benefit of creating art in the process, which heals my soul. The deep breathing involved in playing wind instruments certainly helps me as well.

I'm learning some music theory, and it's hard! I want to learn how to write songs and compose music, and I'm going to get there. If you have any favorite resources for this, please send them to me!

The upshot of getting back into music is that it will, hopefully, give me a way to perform again. I've been a performer in some aspect for a lot of my life. In school, I was in our concert bands and in small wind ensembles. I was on the debate team, a shock to people who knew me as the shy kid who shook when forced to speak in front of the class. Since school, though, I've lost this opportunity. I got a taste of performance again with my conference talk, and I think music will be my route to performing regularly.

I'm starting to use my voice. Advocacy and activism were always things I looked up to but didn't feel like I was able to do. But then I found I have a voice, and I realized I need to use it. I shared a few posts this year on things that are important and required me to speak up, like trans rights and the crisis that was happening in Asheville.

Organized two rated chess tournaments. This year I organized two tournaments for our local chess club! They had about 12 players each, and went off smoothly. It was a good experience all around. I'm not sure I'll have the opportunity (or desire) to do this again this year.

Kept my head up. Current events have been... a lot... and I've managed to keep my head up for most of it. I'm going to keep going, and keep trying.

Upgraded the workshop for all-year use. Historically I've only been able to use my workshop for a few months of the year when the weather is right. This year, we got it insulated, replaced the windows, and added a heat pump, and all those combine to mean I can keep it temperate all year so I can go out there whenever I want. (Some wood finishes are inadvisable in cold or hot weather since they need good ventilation, but otherwise any time for anything.)

It's changed my relationship with woodworking, since now I can pop out there and make something whenever I want. When I was rearranging my desk yesterday, I realized I really needed a headphone stand. And so I popped out to the workshop and put in a total of two hours of work (split across a few sessions for glue and finish to dry), and now I have one! Something similar happened when I needed an adapter to mount my drum pad: I just made it.

Last year's goals

This year was a lot! How does it stack up against what I wanted to do last year?

❓ I wanted to keep my rights, and I did—for now, in my own state. But it's really tenuous, and there are many states where I'd be punished for using a bathroom. The state I was born in, Ohio, has banned trans folks from using the bathrooms consistent with their gender, and the incoming administration is a dark cloud. I won't call this a miss yet, but I can't call it a win.
✅ No personal side projects went into production! I once again toyed with the idea and once again talked myself out of it. Good job, me!
✅ I am not sure I struck a better balance with calls and making, but I embraced that I love talking to my friends and just continued to make time for them. This is going to be even more important next year.
✅ I kept writing on the same schedule, and I did expand it! I did a creative writing class, and even wrote a poem as well.
❌ I did not do any comedy this year, so it's a miss. But it's a happy miss, because I found other things that I was drawn to.
✅ I stayed pretty active in my communities, given my health.
✅ I was a good parent and partner, given my health.
✅ I finished voice training! This was almost a gimme, since I was done in January this year.
✅ My ergonomic setup was definitely improved. I still want to work on using Talon more, but I have made improvements there as well.
❌ I did not do more technical projects this year. I've started a few, and I'm going along in the background, but health got in the way.
❌ I did not get back into competitive chess, and I probably won't. It simply doesn't feel as important right now. Music is filling the role that it filled in my life.
✅ I kept my mental health strong!

I think I did really well on these goals, even if it were a normal year and not a year where I spent 6 months sick to various degrees and where I started to feel the crushing weight of our politics and its threat on my rights and my life. This year I did learn a bit about what is important to me, and where I want to spend my time. That's reflect in my hopes and goals for next year.

Hopes and goals for 2025

These aren't predictions or concrete goals, but a reflection on what I'd like the next year to be. This is what I hope 2025 looks like for me.

Keep my rights. A perennial goal at this point, it's the headliner since trans rights are near the top of the Republican's agenda for this new administration. I'm wary, and I'm going to do what I need to do to keep myself and my family safe. I think that can be done from where I am in a safe, supportive community, but we will keep ourselves safe while continuing to advocate for all those who need protection. In particular, I'm going to keep living my best life and being positive representation for other trans folks who are similarly under attack.

No personal-time side projects into production. This one will probably be a forever anti-goal for me. I don't want to do ops-y things in my free time (despite feeling like shaving that yak occasionally), and I don't want to support a product in my free time. My free time is more about playful exploration.

Maintain relationships with friends and family. This is the 2025 version of 2024's goal of "strike a better balance with calls and making." I'm positive I'll have time set aside for making things and for playing music. But this is going to be a challenging year, so my loved ones (given/chosen family and dear friends) will need my support and I will need theirs. So I'm going to put those relationships first and foremost.

Explore ways to make this my living. I want to do more playful exploration, the kinds of things I do on my blog, and make that my living eventually! 2025 isn't when I'll get there, but I want to try out one or two things (like a Patreon? fund myself via consulting and coaching?) to start understanding what might work for both me and my readers.

Keep my mental health strong. This is going to be a challenge, and I am in a good spot for it. I'll need to dedicate effort to it, though, what with the upcoming onslaught.

Release some recorded music. I'm working on a lot of aspects of my music. Eventually, I want to write something and release it. This year might be releasing a recording of a cover, but it might also be an original piece. I'm not sure!

Write some original music. I don't know if I'll release something of my own, but I know I need to work on it. We'll see how this goes! It's scary to me, and it's also something I'm confident I can do if I put in the effort to learn how.

Do some ridiculous fun projects with code. There are a few things I really wanted to work on in 2024 that are just playful, fun, ridiculous things. I didn't get to do them since I was, uh, kinda sick (not sure if I mentioned that yet). So they're burning to get out of me this year. I want at least one to get out of me this year.

* * *

That's it! I've poured a lot of myself into this post. If you've made it this far: thank you, so much, for reading.

2024 had a lot in it, good and bad. I'm trying to hold both at the same time, to remember the good and remember the bad, as they are both important aspects of the year for different reasons.

I hope that 2025 keeps much of the good, and that we can minimize the bad. I'm going to do everything I can to hold joy in this world. Please join me in that, and let's fill 2025 with joy, even in the face of all that's being thrown at us.

Goals for 2025

Chris' Full Stack Blog RSS Feed — Mon, 30 Dec 2024 02:00:00 GMT

There's a new blog post "Goals for 2025" and you can read it online.
undefined
You can also subscribe for weekly emails on my latest posts - if I publish anything that week!

Prefer Numbered Lists to Bullets

Danny Guo — Mon, 30 Dec 2024 02:00:00 GMT

Reduce JavaScript Bundle Sizes in SvelteKit by using server load functions

Stanislav Khromov — Mon, 30 Dec 2024 01:06:04 GMT

The amount of JavaScript you ship to your users directly impacts your site’s performance. A smaller bundle size means faster initial loading times and an overall snappier experience for end users. In this post, I’ll show you how to analyze your SvelteKit bundle size and optimize it by moving heavy dependencies to the server. Analyzing Bundle Sizes Before we start, let’s set up a tool to visualize our bundle sizes. We’ll use rollup-plugin-visualizer, which works great with SvelteKit. Install the package into your existing SvelteKit project: Configure it in vite.config.ts: When you build your project with npm run build, you’ll […]

The post Reduce JavaScript Bundle Sizes in SvelteKit by using server load functions appeared first on Stanislav Khromov.

Modern Portfolio Theory III: The Efficient Frontier in log volatility—return space

Bytepawn - Marton Trencseni — Mon, 30 Dec 2024 01:00:00 GMT

In the previous two articles, we explored the coverage of random portfolios in log volatility—return space, both with and without a risk-free asset. We now take the next step, and calculate the Efficient Frontier and Capital Market Line of Markowitz’s theory.

AI 产品的破局之道：以人为本

ipfans's Blog — Mon, 30 Dec 2024 00:30:00 GMT

伴随着 OpenAI、Anthropic 等等公司的努力，大型语言模型（LLM）的性能不断提升，AI 产品的应用场景也越来越广泛。从口袋里的智能助手到驰骋未来的自动驾驶，从辅助医疗诊断的慧眼到洞察市场风云的智脑，AI 正以前所未有的速度改变着世界。但是在这个过程中，我们会注意到一些问题，我们会陷入“技术至上”的误区，过分强调技术的先进性，而忽略了用户的需求和体验。这就导致了很多 AI 产品虽然技术先进，但却无法真正满足用户的需求，最终无法实现商业价值。究其原因，许多 AI 系统在与人类交互时，常常陷入**“鸡同鸭讲”的窘境**——对上下文的理解捉襟见肘，难以领会人类语言中的微妙含义；更令人担忧的是“幻觉”问题，AI 煞有介事地编造着“一本正经的胡说八道”，让用户无所适从；而在用户体验方面，许多 AI 产品更是差强人意，机械式的交互和冰冷的界面让人难以产生亲近感。有些时候我们有些失望：帕累托法则（或者叫做 28 原则），一直在发挥着神秘的力量。

抛开模型开发公司本身而言，面向客户的 AI 产品的设计和开发是一个复杂的过程，需要综合考虑技术、商业和用户体验等多个方面。在这个过程中，如何保证产品的质量和用户体验成为了关键问题。传统的产品设计方法往往是由设计师和工程师等专业人员完成，而用户往往只是产品的使用者。然而，随着 LLM 的发展，新的交互形式不停的出现，比如之前的聊天产品和语音助手交互产品普遍反馈交互感太差并不实用。新的实时多模态模型的出现，让这些可以更好的理解用户的需求，更好的满足用户的需求。

但，无论如何，要充分释放 AI 的潜力，必须将用户需求和体验置于核心地位。这不仅仅是一句口号，更是一种深刻的变革，要求我们将用户的需求、期望和体验置于 AI 产品设计的最核心。这意味着，我们需要像人类学家一样去观察和理解用户的真实诉求，像心理学家一样去揣摩用户的情感和认知模式，并将这些洞察深深地烙印在产品的基因之中。只有这样，才能打造出真正被用户拥抱、信任并从中受益的 AI 产品。

本文就是我在最近两年开发 AI 产品的一些体悟，希望能够给 AI 产品的设计和开发带来一些启发。虽然我是一个工程师视角，但是斗胆今天就来聊一聊 AI 产品的设计和体验问题。

理解当前 AI 的阶段：能力与局限

毋庸置疑，现在 AI 在“力大砖飞”的背景之下，已经在许多领域展现出了强大的能力。从自然语言处理到计算机视觉，从智能推荐到智能对话，AI 的应用场景越来越广泛，技术也越来越成熟。在过去的两年中，我们见证了每隔几个月到半年，LLM 就进行了一种大范围的能力提升变革。然而，AI 的能力和局限性也给产品设计带来了许多挑战。这些局限性并非是无法克服的，有些也仅仅局限于当前的技术水平，但是我们必须清楚地认识到这些局限性，才能更好地设计和开发 AI 产品。

这些问题你可能在很多地方看到过，让我再赘述一遍：

上下文问题：大模型在处理长文本或复杂对话时，受限于上下文“记忆”能力，可能导致理解偏差和信息遗漏。例如，在长篇文档总结或多轮对话中难以保持信息连贯。扩展上下文窗口当然是一种解决方案，但即便是大上下文窗口的 Gemini，也会出现注意力不集中的问题，最终影响到模型的性能和结果。
幻觉问题：AI 可能在缺乏事实依据的情况下生成看似合理但虚假的信息，严重影响用户信任，尤其在需要高准确性的场景中。在绝大多数的 B 端场景和部分 C 端场景中，我们都需要确保回答的准确性。之前也有加拿大航空的案例，导致公司的信誉受损。
缺乏常识与情感理解： AI 在处理需要常识判断和情感理解的场景中表现不足，导致交互生硬，甚至产生误解。

这些问题并非无法解决，但是我们必须清楚地认识到这些问题，才能更好地设计和开发 AI 产品。在 AI 产品设计中，我们需要充分考虑这些问题，从而更好地满足用户的需求和体验。而在这个过程中，我认为针对人机回环(Human in the loop, HITL)的设计思想是非常重要的。通过完善这个设计思想，我们可以更好地解决上述问题，提高 AI 产品的质量和用户体验。

以人为本：产品的设计灵魂

以人为本，是产品设计的灵魂所在，对于 AI 产品而言更是如此。对这种方式而言，一般会以下面的方式为循环来进行：

Short story fika: Duty

Liza Shulyayeva — Sun, 29 Dec 2024 23:01:33 GMT

We had our fourth short story fika yesterday. This time, the theme was ‘duty’ and the stories featured were:

The Sniper by Liam O’Flaherty
The Lottery by Shirley Jackson
The Veldt by Ray Bradbury

It was just three of us this time, and we had a great chat about the duties and sacrifices of war, the questioning (or lack thereof) of outdated traditions, and the costs of our increasing reliance on automation in daily life.

Reading Highlights 2024

THINK@RJ — Sun, 29 Dec 2024 20:43:00 GMT

2024 was challenging and rewarding in equal measure when it came to me being able to set time aside for reading. Lots of travel across multiple oceans and much time spent in hotels, Airbnbs and flights. So I switched to a more multi-device, multi-platform approach. My main platform for reading

Roast #43: 2024 Year-end wrap up

thoughtfulcoffee — Sun, 29 Dec 2024 20:17:58 GMT

stumbling, but finishing.

What I read in 2024

Ernest Oppetit — Sun, 29 Dec 2024 14:40:26 GMT

Here is what I read in 2024. Did much more sampling and dropping books halfway (I still love you, Moby Dick) so it’s a shorter list than last year. Hope you find a gem or two in there for your reading lists. Happy new year! 🤓 Non-Fiction (English) Fiction (French) Non-Fiction (French)

Booksbooksbooks 12/24

GOOD INTERNET — Sun, 29 Dec 2024 13:45:52 GMT

Tiny reviews for books from Shirley Jackson, Martha Wells, Alexandre Dumas, Stanislaw Lem, Stephen Baxter, Stephen King, Adrian Tchaikovsky, Dan Jones, Carlo Rovelli and more.

Intro to Astro

Thought Eddies — Sun, 29 Dec 2024 13:24:49 GMT

Intro to Astro

I Am Fine

The Dark Side — Sun, 29 Dec 2024 12:08:36 GMT

Caveat: I do not have any qualification in medicine or biology.

Hyperuricemia is a condition where acid uric in your blood is above a certain concentration, this condition, if lasting for long enough, can cause gout, which is something that you certainly do not want.

I have hyperuricemia

As part of my routines checks and for some travel preparations I visited my GP, among the routines check was a blood check that I did on the spot.

The result came indicating a level of 8.7 mg/L of uric acid which is above the upper bracket of the normal range indicated on the document at 7.7 mg/L (which I think is 0.7 mg/L above of the recommended maximal value in the US and 0.5 mg/L above what it was a few years ago here).

Really?

My previous control, two months before, was at this upper bracket. As I like to think I have control on my life, I tend to overlook the non-actionable elements that could explain this situation. However, when reading online the main lifestyle factor for acid uric seems to be purine rich food:

Red meat
Seafood
Food and drinks with high fructose
Alcohol (including alcohol-free beer)

Food accounts for only 12% of the problem but that’s still where I think I have most leverage.

And I am pretty much a monk when I read that list. Not a perfect monk, but a monk-leaning person. I consume a bit of seafood but rarely, I drank little alcohol in the last month (probably less than 7 units a week), it is true that I drank a bit of alcohol-free beer but that too was a couple of occurrences a week. Red meat, especially last month was a rare thing.

My protein sources are mostly chicken, tofu and whey isolate protein powders.

I ended-up looking up if those last ones (protein powders) had an impact as I indeed was consuming more of them over the last two months. I did not find a reliable source, the messages were conflicting.

Then I remembered a remark from my GP: “sport has an impact on uric acid”, and searching around that idea was interesting. I found a paper from 1980, Purine Metabolism During Strenuous Muscular Exercise in Man (Sutton & al., 1980), and this was getting interesting:

After a 5 km race, one group saw its mean uric acid increase from 6.9 to 8.5 mg/dL
After a marathon, another group went from 6.2 to 7.9 mg/dL.

Suddenly, I thought about my agenda :

Day before the blood sample at 20h to 21h: confined scuba diving training.
Day of the blood sample 5h30 to 7h: leg workout at the gym.
Blood sample at 8h45.

Since I am tracking all my food and water intake. I know my exact diner from the night before.

I was suddenly thinking that maybe I created an intense bias in the measurement by working out.

Getting more data points

I decided to reproduce the same situation and gather more data points to see if there is indeed an increase following physical activity. Properly quantifying it will not be possible on a single episode, but that could give me an idea.

Day -2: rest & 2 350 mL water intake
Day -1, rest & 1 830 mL water intake
Day -1, 7:37 before breakfast: 7.9 mg/dL
Day -1, evening: confined scuba diving training (luckily I still had some in my agenda).
Day -1, evening: identical diner, including 660 mL alcohol-free beer (not accounted in water intake).
Day 0, 6:35: 7.8 mg/dL
Day 0, 6:45 - 8:15: leg training, 600ml water intake
Day 0, 10:05 before breakfast: 7.8 mg/dL
Day 0, 15:00: 7.6 mg/dL
Day +1, 6:44 before breakfast: 6.7 mg/dL

Conclusion: nothing

At this point, I felt kind of frustrated: because the new data was not showing much. The good news is that it seems that the sample that triggered all this was probably not characterizing my general state. The other interesting piece of information is that I did not reproduce the results of the paper mentioned above, either:

The uric acid spike is very limited in time (they measured uric acid right after exercise, I waited for two hours).
The nature of the work-out was too different to trigger similar results.
I was an outlier that week.
Something wrong with the study.

My first Rust crate

www.mootoday.com - RSS Feed — Sun, 29 Dec 2024 05:13:05 GMT

I released `clap-nested-commands` to reduce boilerplate code for Rust CLIs built with `clap`.

Sunset Orange

unsungNovelty — Sun, 29 Dec 2024 02:42:40 GMT

An Utterly Incomplete Look at Research from 1824

Brett Mullins — Sun, 29 Dec 2024 02:00:00 GMT

The French Revolution cast a long shadow over early nineteenth century thought. Many of the selections below are concerned with the aftermath of the Revolution and the Napoleonic Wars both economically and intellectually. The pseudonymous Piercy Ravenstone critiques England’s system of public finance and its handling of the massive debt resulting from the wars. John Stuart Mill reviews a debate about the effects of war-time spending on prices. His father, James Mill, offers analyses of government, jurisprudence, freedom of the press, and international law along utilitarian lines for the Encyclopedia Britannica. William Stevenson argues that the moral convulsion among the people caused by the French Revolution opened the door for intellectual progress.

Optimising my Rust solutions for Advent of Code

Krishna's blog — Sun, 29 Dec 2024 02:00:00 GMT

I completed Advent of Code 2024 in Rust. Here are all the tricks I used to optimise my code.

O-1s, H-1Bs, and other ways US Immigration is Broken

Caseysoftware — Sat, 28 Dec 2024 18:37:27 GMT

With the big ongoing debate on immigration, it’s been amazing/horrifying watching people muddle everyone together into one bucket and then…

The post O-1s, H-1Bs, and other ways US Immigration is Broken appeared first on Caseysoftware.

Favourite music

Qubyte Codes — Sat, 28 Dec 2024 16:25:00 GMT

Here are a selection of albums which are all-time favourites of mine. They stand up as a whole (not just as single tracks). I don’t claim to have good taste in music, or that my taste is better than anyone else’s! With each album I’ve written a little about times and places I associate with it. The albums are presented in no particular order. I’ll garden this post over time and add new material. I've linked to music services where I can find each album available, but with minimal effort. There are other services, but they're either behind paywalls or incomprehensible to me.

Devin Townsend – Terria

This was the first music from Devin Townsend which I was exposed to. It was in 2002 I think, and I was listening more or less exclusively to bad metal at the time. The weight of the sound of this album was what lured me in (particularly Earth Day), but the progressive rock feeling to it led to a diversification of my taste. This album evokes a feeling of vast landscapes and greenery. It, and Ocean Machine remind me of my undergraduate years at Sussex University.

To this day I’m a big fan of Townsend. No two offerings are the same, and some of his stuff is really out there.

Strapping Young Lad – City

I sneaked in another Devin Townsend album. Until about the mid 2000s Devin Townsend also headed the metal band Strapping Young Lad (SYL). I’m less fond of the output of SYL these days, but I still come back to City. It’s probably the finest industrial metal album out there. This album also reminds me of my time at Sussex University. That, and my questionable taste in clothing at the time (very baggy, not a lot of colour, tassels).

Portishead – Dummy

While at university I had a complicated relationship with a woman who introduced me to Portishead. Things between us ended poorly, and this album reminds me of a time when my head was in bad shape. I can listen to it now with more objectivity, even if it makes me uncomfortable. The music should require no introduction. The hauntingly beautiful vocals of Beth Gibbons and trip-hop sound are timeless.

ISIS (the band) – Panopticon

While a postgrad at Leeds University my like of metal and progressive rock mutated into a like of post-rock. ISIS (long before the terrorist group sometimes affiliated with that initialism came to be) was a heavy post-rock / metal group which produced some extraordinarily heavy music. Panopticon is my favourite of their albums, to the point that I named my first open source offering after it. The stand-out track for me is In Fiction. It' takes an age to build, but when it finally gets there it's the deepest, heaviest, most crushing sound. Amazing pay-off.

Opeth – Blackwater Park

While the first Opeth album I got into was Still Life (and Deliverance & Damnation come close), my favourite remains Blackwater Park. Most of the music I listen to avoids guitar solos and such. Opeth manage to do the reverse, and have moments when every instrument seems to be executing a solo at the same time and maintain coherence. I was originally hooked by The Drapery Falls, but the entire album is exquisite. Give it a try, even if the thought of death-growl vocals is off putting to you. It’s worth it.

Múm – Finally We Are No One

I was introduced to this me by an Italian I met in Tokyo (she introduced me to a lot of really great music, I should thank her again). This album should sound light and airy, but comes across and close and comforting, like being indoors by a fire during a coastal storm. While I first listened to it in Tokyo I don’t associate it with there. Its aesthetic is so strong that it tries to invoke memories of a particular balance that I’ve not actually experienced beyond listening to this album.

The Avalanches – Since I Left You

This album is famed for the extreme lengths the creators went to make it out of samples. I was aware of it long before I listened to it (it was on in various student pubs while I was doing my undergrad). I first really listened to it one Summer I was living in Tokyo. It reminds me of the heat and humidity of the summer there, but also the detachment I felt while I lived there. I moved to Japan at relatively short notice after a difficult break up, and struggled to find my footing for a long time after I arrived. This album reminds me of the joy of summer and feeling at ease with myself and my freedom from my former life and my culture. It probably marks the point at which I stopped identifying myself as a nationality or a job, and started thinking of myself as a standard, awkward mess of a person.

This album reminds me that while we’re all a mishmash of different things, it’s important to remember the difference between what we are, and what our environments try to pigeonhole us as. Even more importantly, a mishmash can be a beautiful and coherent whole.

Portico Quartet – Memory Streams

This is a fairly recent edition, and I find it difficult to categorize. The band makes instrumental music with heavy use of the Hang (a form of steel drum), saxophones, and synths. I guess it's jazz, but I don't know enough about jazz to really say. I bought the record as an impulse purchase shortly before the pandemic lockdowns started. My son was still a baby then, so I was looking for something to relax to. Due to the timing, it became the sound of the early pandemic for me. Back then, it felt like the world was just the three of us.

Boards of Canada – Tomorrow's Harvest

This was an impulse buy, pretty much the day it came out in Japan. I used to spend a lot of time trying to read the staff recommendations in the Tower Records in Yodobashi Akihabara. Unfortunately that Tower Records has been gone for a few years now. I had listened to Boards of Canada's earlier work, but Tomorrow's Harvest blew my mind. The vintage sound and sparseness gives it a brooding and foreboding feeling. This album is excellent to work to.

I bought this album shortly before moving back to the UK from Japan, so it used to remind me of that time. I've played it so often since that any nostalgia for that time has long since gone, but I do still associate it with a change in mindset that came with changing career from a physics academic to a software engineer.

The Devin Townsend Project - Addicted

Yes, another Devin Townsend album. This album was released just days before I moved to Tokyo from the UK, and I listened to it a lot. I strongly associate it with my first year in Japan. In particular with difficulty settling into the time zone in the first few weeks, but also the fun little dance parties I had with housemates and friends in the share house I was renting a room in. The track Bend it like Bender! was a favourite of one of my Italian friends. That friendship ultimately led to me meeting my partner.

Life is full of these incredibly unlikely little sequences of events with important outcomes. What if that album came out a day later? I'd have been too busy to buy it. What if I had chosen a different share house? What if the aforementioned friend hadn't started a business with a friend of my (now) partner and set up a Roman Holiday themed club night beneath a Fiat showroom which we were both "encouraged" to attend to recoup some losses? What if my partner and I hadn't been snacking on arancini next to each other and struck up a conversation? What if...

Marconi Union - Signals

Another relatively recent addition. I'd listened to some of their music before and found it good to work to, so when Signals was released I bought it straight away. It's an exceptional album. Tracks are repetitive, build slowly, and at times feel meditative. There's a thrumming, almost dangerous quality to it. Stand out tracks for me are Strata, and A Citizen's Dream. The latter I find very affecting. It feels like longing, sadness, and hope, sometimes to the point of tears when I'm actively listening to it. I have no idea why that is though! Sometimes my reaction to music can't be explained. It's like it can be a key to a random emotional door in my mind.

I tend to react differently to others when I listen to music. I find quite heavy, negative, loud music to be uplifting, and more pop-like music intended to be uplifting will make me feel quite upset and angry.

Finally trying out the Android XML layout

Whacky Labs — Sat, 28 Dec 2024 11:55:00 GMT

So year after year since the dawn of the Android SDK back in 2009 I’ve been making the yearly resolution to try it out. But for one reason or the other I’ve always been putting it off. But not this year. Before the sun goes down for the last time for the year of 2024 let’s make our favorite photo app using the classic Android UI toolkit - The framework with no name - You know that thing where you write XML for layout. Let the fun begin!

Set up

I’m going to use retrofit for networking with glide for loading images all connected together via kotlin coroutines.

The first step is to install all the dependencies. Which from what I understand is now a 2 step job. First you list all the dependencies in the libs.versions.toml file

[versions]
kotlin = "2.0.0"
coroutinesKtx = "2.4.0"
coroutines = "1.9.0"
gson = "2.11.0"
retrofit = "2.11.0"
okhttp = "4.12.0"
glide = "4.16.0"

[libraries]
retrofit = { module = "com.squareup.retrofit2:retrofit", version.ref = "retrofit" }
gson = { module = "com.google.code.gson:gson", version.ref = "gson" }
retrofit-gson-converter = { module = "com.squareup.retrofit2:converter-gson", version.ref = "retrofit" }
okhttp-logging-interceptor = { module = "com.squareup.okhttp3:logging-interceptor", version.ref = "okhttp" }
glide = { module = "com.github.bumptech.glide:glide", version.ref = "glide" }
glide-compiler = { module = "com.github.bumptech.glide:compiler", version.ref = "glide" }
kotlinx-coroutines-core = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-core", version.ref = "coroutines" }
kotlinx-coroutines-android = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-android", version.ref = "coroutines" }
androidx-viewmodel-ktx = { module = "androidx.lifecycle:lifecycle-viewmodel-ktx", version.ref = "coroutinesKtx" }
androidx-runtime-ktx = { module = "androidx.lifecycle:lifecycle-runtime-ktx", version.ref = "coroutinesKtx" }

And then you actually add them to the project by updating the build.gradle.kts file

dependencies {
    implementation(libs.retrofit)
    implementation(libs.retrofit.gson.converter)
    implementation(libs.gson)
    implementation(libs.okhttp.logging.interceptor)
    implementation(libs.glide)
    annotationProcessor(libs.glide.compiler)

    implementation(libs.androidx.viewmodel.ktx)
    implementation(libs.androidx.runtime.ktx)
    implementation(libs.kotlinx.coroutines.core)
    implementation(libs.kotlinx.coroutines.android)
}

And finally, the most important step that I almost always forget. To add the INTERNET permission in the AndroidManifest.xml file

<manifest>
    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    <application>
        ...
    </application>
</manifest>

Fetching data

Creating the network layer is pretty straightforward with Retrofit. You declare an interface of the API and let retrofit do the rest.

For our needs, the API consists of just a single GET call to fetch a List of Photo

data class Photo(
    val albumId: Int?,
    val id: Int?,
    val title: String?,
    val url: String?,
    val thumbnailUrl: String?
) : Serializable

interface API {
    @GET("photos")
    suspend fun photos(): Response<List<Photo>>
}

class NetworkService {
    companion object {
        fun api(): API {
            val loggingInterceptor = HttpLoggingInterceptor()
            loggingInterceptor.level = HttpLoggingInterceptor.Level.BODY

            val okHttpClient = OkHttpClient
                .Builder()
                .addInterceptor(loggingInterceptor)
                .build()

            val retrofit = Retrofit
                .Builder()
                .baseUrl("https://jsonplaceholder.typicode.com/")
                .addConverterFactory(GsonConverterFactory.create())
                .client(okHttpClient)
                .build()

            return retrofit.create(API::class.java)
        }
    }
}

Next piece of the puzzle is the PhotoListViewModel to map the network data into various UI states. Again pretty straightforward thanks to the beautiful magic of ViewModelScope and StateFlow

class PhotoListViewModel : ViewModel() {
    sealed class State {
        data object Loading : State()
        data class Error(val message: String) : State()
        data class Content(val photos: List<Photo>) : State()
    }

    private val _state = MutableStateFlow<State>(State.Loading)
    val state: StateFlow<State> = _state

    fun fetchPhotos() {
        viewModelScope.launch {
            _state.value = State.Loading
            try {
                val photos = NetworkService
                    .api()
                    .photos()
                    .body() ?: emptyList()
                _state.value = State.Content(photos)
            } catch (e: Exception) {
                val message = e.message ?: "Something went wrong"
                _state.value = State.Error(message)
            }
        }
    }
}

UI

On the UI side we need 2 Activities, a MainActivity to render a grid of images and a DetailActivity to render the detailed image view.

The basic structure of the MainActivity is to simply observes viewmodel changes and update the UI.

class MainActivity : AppCompatActivity() {

    private val viewModel: PhotoListViewModel by viewModels()

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        enableEdgeToEdge()
        setContentView(R.layout.activity_main)
        ViewCompat.setOnApplyWindowInsetsListener(findViewById(R.id.main)) { v, insets ->
            val systemBars = insets.getInsets(WindowInsetsCompat.Type.systemBars())
            v.setPadding(systemBars.left, systemBars.top, systemBars.right, systemBars.bottom)
            insets
        }

        lifecycleScope.launch {
            repeatOnLifecycle(Lifecycle.State.STARTED) {
                viewModel.state.collect { state ->
                    when (state) {
                        is PhotoListViewModel.State.Loading -> showLoading()
                        is PhotoListViewModel.State.Error -> showError(state.message)
                        is PhotoListViewModel.State.Content -> showPhotoList(state.photos)
                    }
                }
            }
        }
        viewModel.fetchPhotos()
    }

    private fun showLoading() {
        Toast
            .makeText(this, "Loading ...", Toast.LENGTH_LONG)
            .show()
    }

    private fun showError(message: String) {
        Toast
            .makeText(this, message, Toast.LENGTH_LONG)
            .show()
    }

    private fun showPhotoList(photos: List<Photo>) {
        // TODO show grid view
    }
}

The interesting bit is with building a photo grid. This requires a PhotoGridAdapter

class PhotoGridAdapter(
    private val context: Context,
    private val photos: List<Photo>,
) :
    BaseAdapter() {

    override fun getCount(): Int {
        return photos.size
    }

    override fun getItem(position: Int): Any {
        return photos[position]
    }

    override fun getItemId(position: Int): Long {
        return position.toLong()
    }

    override fun getView(position: Int, convertView: View?, parent: ViewGroup?): View {
        val contentVw =
            convertView ?: LayoutInflater.from(context).inflate(R.layout.grid_item, parent, false)
        val imageVw: ImageView = contentVw.findViewById(R.id.imageVw)
        val textVw: TextView = contentVw.findViewById(R.id.textVw)

        val photo = photos[position]
        Glide
            .with(context)
            .load(photo.thumbnailUrl)
            .into(imageVw)
        textVw.text = photo.title ?: "-"

        return contentVw
    }
}

And then use this adapter to provide UI data for the GridView

class MainActivity : AppCompatActivity() {
    // ...

    private fun showPhotoList(photos: List<Photo>) {
        val gridVw = findViewById<GridView>(R.id.gridVw)
        val photoGridAdapter = PhotoGridAdapter(this, photos)
        gridVw.adapter = photoGridAdapter
    }
}

Next, when tapped on a photo tile we need to spawn DetailActivity and provide the selected Photo as argument.

class MainActivity : AppCompatActivity() {
    // ...

    private fun showPhotoList(photos: List<Photo>) {
        val gridVw = findViewById<GridView>(R.id.gridVw)

        gridVw.setOnItemClickListener { _, _, position, _ ->
            val photo = photos[position]
            startActivity(Intent(this, DetailsActivity::class.java).apply {
                putExtra("photo", photo)
            })
        }

        // ...
    }
}

And then finally the DetailActivity

class DetailsActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        enableEdgeToEdge()
        setContentView(R.layout.activity_details)
        ViewCompat.setOnApplyWindowInsetsListener(findViewById(R.id.main)) { v, insets ->
            val systemBars = insets.getInsets(WindowInsetsCompat.Type.systemBars())
            v.setPadding(systemBars.left, systemBars.top, systemBars.right, systemBars.bottom)
            insets
        }

        val photo = intent.getSerializableExtra("photo") as Photo

        val imageVw: ImageView = findViewById(R.id.imageVw)
        val textVw: TextView = findViewById(R.id.textVw)

        Glide
            .with(this)
            .load(photo.url)
            .thumbnail(
                Glide.with(this).load(photo.thumbnailUrl)
            )
            .into(imageVw)
        textVw.text = photo.title ?: "-"
    }
}

Conclusion

The classic Android development feels very much like UIKit for iOS. Most the concepts are 1:1 mapped. Kotlin is always very fun to work with. The one thing I find weird about Android is the lack of in-house libraries for such a basic thing like Networking, which is the core building block for almost all apps these days. And then how various companies have rolled out their own solution for building the data-event pipeline. Like with LiveData, StateFlow and kotlin coroutines leaving the developers to figure out the glue to make it all work.

Before I discovered Glide I tried building my own ImageRepository to handle networking and data caching. And there too I felt the tools offered by Android were very primitive compared to say Apple’s URLSession.

Also before falling back to retrofit, I was trying out ktor for networking. Personally I like ktor as it seems more flexible and more in line with other networking libraries I’m used to.

And finally the XML based layout feels very nice and robust. I like the code-design split view.

The hardest part about Android development is dealing with the unknowns. Like I ran into a runtime crash when running the app on my device with is running on Android 11/SDK 30 while on the emulator I was using Android 13/SDK 34. But I guess that is just a day in life for a developer, you know the classic “Works on my machine” phenomenon.

The code from this experiment is sitting right next to all other experiments. github.com/chunkyguy/PhotoApp/android

References

The $500 Question - How Media Framing Shapes Our Reality

Home on Lagomorph — Sat, 28 Dec 2024 09:00:00 GMT

“In this season of giving, what are we to make of a billionaire with a soft spot for striving graduates who draws a hard line on being present for the pomp and circumstance, no matter the circumstances?” - New York Times

The modern media apparatus doesn’t just report news - it manufactures reality through carefully constructed narratives that shape how we process and respond to events. This process has become so refined that different outlets can take the same event and create entirely separate realities, each designed to reinforce specific worldviews and emotional responses in their target audiences.

symbol-overlay-mc now on MELPA

xenodium.com @alvaro — Sat, 28 Dec 2024 02:00:00 GMT

Some time ago, I wrote about adding multiple cursor support to symbol-overlay by gluing the two packages together. If you're keen to learn how to bend Emacs to your way, have a look at that post.

Later on, in a "Multiple cursors - how and why?" Reddit post, I showcased the overlay usage. Peter Oliver asked if the package could be submitted to MELPA. To my delight, he also volunteered to take on the submission.

Unfortunately, I had failed to check back on the MELPA submission until today. I'm happy to report that, thanks to Peter, symbol-overlay-mc is now on MELPA.

Happy symbol-overlay-multiple-cursors editing! ;)

Emulating 6502

zserge's blog — Sat, 28 Dec 2024 02:00:00 GMT

Last year, I noticed that routine programming no longer brought me the same joy it did decades ago. So, my New Year’s resolution was to engage in more “useless” programming – coding small, fun projects without any specific end goal in mind. Of all the possible topics, the one that captivated me throughout the year was retrocomputing. I didn’t have a computer until I was 14, but I remember seeing grey 8-bit machines at school or friends' houses.

Better AI LLM assistance for Svelte 5 and SvelteKit

Stanislav Khromov — Sat, 28 Dec 2024 00:41:02 GMT

With the release of Svelte 5 and its new runes syntax, many developers are wondering how to get reliable AI assistance that understands the latest features in Svelte 5 working with ChatGPT, Claude, Continue, Cursor and other tools. Looking for smaller, distilled LLM presets? Click here! The challenge with AI and new frameworks When a new major framework version is released, AI tools often struggle to provide accurate code samples since they were trained on older documentation. This creates two main problems: The official solution: llms.txt As part of Advent of Svelte 2024, Svelte introduced official support for AI tools […]

The post Better AI LLM assistance for Svelte 5 and SvelteKit appeared first on Stanislav Khromov.

Starting Tests from Zero

DEV Community: John Mitchell — Fri, 27 Dec 2024 19:19:42 GMT

BEER

Often developers and businesses can't see the value that testing gives. Devs see it as a "tax", an extra thing that slows them down from doing their "real job". Businesses go into magical thinking mode, that Devs are always creating flawless features that deliver whatever the Business thinks it does and fits into the system.

Tests don't slow down development, then pull the development forward. Writing "happy path" Unit Tests shows Devs what to work on next. Investing in high level End to End tests gives the Business confidence the system is working reliably and new changes aren't dangerous.

Here's how to get started with tests if you have non.

Top-down vs Bottom-up: two different Audiences

Write one or more end-to-end tests (E2E). This gives confidence that the entire app is solid, at the cost of being slow and not very detailed. The main benefit is for the Business: they can see and understand the results.

Back in the day, for a shopping site, our E2E test was "create user, add item to cart, checkout, validate there's a total cost" kind of thing. It was slow but had really obvious business value. Nowadays you'd do something like Playwright for this.

Write lots of "happy path" unit tests, generally one per non-obvious function. These are fast and give a lot of info, but for a tiny scope: one function. Generally any time I have to think, I write a test.

Also write a unit test per bug, even if it's just "this doesn't work" or "expect some sort of Exception to happen here". Tests can be vague if that gives you actionable feedback as a developer.

publish Test Coverage! This is great as you and your team -- and more importantly the Business -- can see this number and how it changes over time. New code makes number go down? Write tests for the new code (or old code, if that makes sense for your team). Bug happens? Write test -- number goes up -- then fix the bug.

The overall goal is getting into a Value Feedback Cycle. For the Business audience, they want confidence the app is functioning as expected, that a new deploy won't bring down the site. For the Dev audience, they want fast actionable feedback for them to complete the features. The two types of tests: End to End and Unit Tests, give this feedback to the two audiences.

Advanced: unit tests don't cover much code. Consider adding API-level tests like with Postman. These are cheaper in that a single test checks more code, even though at a less detailed level. They're also fun for the Business and Devs to see because a single API test makes the Test Coverage go up more than from a single Unit Test.

Starting Tests from Zero

DEV Community: John Mitchell — Fri, 27 Dec 2024 19:02:24 GMT

Here's how to get started with tests if you have non.

Top-down vs Bottom-up: two different Audiences

Write one or more end-to-end tests (E2E). This gives confidence that the entire app is solid, at the cost of being slow and not very detailed. The main benefit is for the Business: they can see and understand the results.

Write lots of "happy path" unit tests, generally one per non-obvious function. These are fast and give a lot of info, but for a tiny scope: one function. Generally any time I have to think, I write a test.

Also write a unit test per bug, even if it's just "this doesn't work" or "expect some sort of Exception to happen here". Tests can be vague if that gives you actionable feedback as a developer.

publish Test Coverage! This is great as you and your team -- and more importantly the Business -- can see this number and how it changes over time. New code makes number go down? Write tests for the new code (or old code, if that makes sense for your team). Bug happens? Write test -- number goes up -- then fix the bug.

Work 2.0 - the interruptible programmer

Prasham H Trivedi — Fri, 27 Dec 2024 17:30:52 GMT

This is a good read from Steves String. I discovered this blog from a youtube channel called Internet of Bugs. The post completely changes the way we think about productivity and work, especially when your are at the other end of your 30s. The post discusses about how it’s not feasible to work for long hours without any interruption. It’s not just about the physical health, but also about the mental health.

Capturing slack messages directly into Emacs orgmode inbox

Srijan Choudhary, all posts — Fri, 27 Dec 2024 13:50:00 GMT

Learn how to seamlessly capture Slack messages into your Emacs Orgmode (GTD) inbox using a custom browser userscript.

Ghostty on Ubuntu: First Impressions and How to Install

Mike Kasberg's Blog — Fri, 27 Dec 2024 05:30:00 GMT

Ghostty came out today! I’ve been following the development on Mitchell Hashimoto’s Blog for a while, and I was excited to give the terminal a try.

First Impressions

Ghostty is great! My first impression was that it looked exactly like Gnome Terminal, with a slightly different color scheme. That’s not a criticism, that’s actually high praise! Mitchell has written on his blog about the importance of Ghostty being native while also being cross-platform. That is to say, although it works on both macOS and Linux, it uses native windows, menu bars, and tabs. After trying Ghostty for a little bit, I have to say I agree, and he nailed it!

I was really pleased to see that everything just worked out of the box (and I know Mitchell’s put in a lot of effort here too). I’ve been using Gnome Terminal as my daily driver on Ubuntu for more than ten years, and I knew within a few minutes of trying Ghostty that I’d be using Ghostty as my daily driver going forward. I’ve briefly tried other terminals before, like Kitty or Alacritty, but I always came back to Gnome Terminal – specifically because I don’t like the non-native look and feel of others. I make heavy use of native tabs, so the non-native tabs (or complete lack of tabs) and non-native UI of other Terminals kept me using the default Gnome Terminal, despite knowing the others were faster. With Ghostty, I can have my cake and eat it too!

Ghostty is simple (it really does look very similar to Gnome Terminal), but it’s fast and has advanced configuration if you need it. I haven’t needed it yet, though I plan to experiment with different themes in the near future. I’m finishing this blog post in vim in Ghostty, and so far, Ghostty has been everything I hoped it would be. I highly recommend it! Mitchell, congrats on a great Terminal and a great launch! I appreciate all your work (and the work of the rest of the community)!

How to Install Ghostty on Ubuntu

I'm working on an unofficial Ubuntu package for Ghostty! This should be easier to install than building from source as outlined below. Follow the instructions on GitHub, and open an issue there if you run into problems!

There’s no official (or unofficial) Ubuntu package yet, so the only way to get Ghostty running on Ubuntu right now is to build it yourself. Fortunately, this isn’t too hard! The build instructions are documented in the Ghostty docs, but I’ll lay out some step-by-step instructions for Ubuntu here.

You’ll need Zig 0.13. There are several ways to install this; I installed it by downloading a binary release. Download the .tar.xz file for your architecture from the Zig Downloads page. You probably want the x86_64 version. Make sure you get version 0.13 for Linux.
Extract the .tar.xz file to your /opt directory. (Any directory would be fine, /opt is conventional for stuff like this and is what I chose.)
```
sudo tar -xf zig-linux-x86_64-0.13.0.tar.xz -C /opt
```
Make sure you have the required dependencies to compile Ghostty, as documented in the Ghostty docs.
```
sudo apt install libgtk-4-dev libadwaita-1-dev git
```
Download Ghostty. The easiest way is to just clone the git repo. Then you can check out a specific tag, or just build main.
```
git clone https://github.com/ghostty-org/ghostty.git
cd ghostty
git checkout v1.0.0
```

Build Ghostty. Again, following the docs.

/opt/zig-linux-x86_64-0.13.0/zig build -p $HOME/.local -Doptimize=ReleaseFast

Run update-desktop-database to refresh the application list so you get the Ghostty icon in your launcher. (If that command fails, try it with sudo.)

That’s it! By default, .local/bin should be in your $PATH, so the ghostty command should now be available. The Ghostty build process should also have installed a launcher file to $HOME/.local/share/applications, so you should be able to find Ghostty in your normal application launcher menu without any extra effort!

For the time being, I think this is the easiest way to install Ghostty on Ubuntu. It will take some time for Ghostty to make it into the official Debian or Ubuntu repositories, but I hope to see some other installation options like Flatpak, Snap, Linux binaries, or Homebrew (on Linux) become available in the coming weeks.

I’m looking forward to using Ghostty as my default shell, and I hope this helps you get it set up on your own machine!

Unoffice hours

RobKohr's Blog — Fri, 27 Dec 2024 02:00:00 GMT

Unoffice hours

Jest ESM Issues: A Meta Tech Cautionary Tale

Prasham H Trivedi — Fri, 27 Dec 2024 02:00:00 GMT

Jest, ESM, and the Meta Tech Conundrum: A Cautionary Tale You know that moment when you’re deep in your TypeScript project, everything’s going smooth, and then you import a simple CSV parsing library? Yeah, that’s when Jest decides to remind you about Facebook’s - sorry, Meta’s - opinions on JavaScript modules. Here’s what happened in my project today: A simple import from csv-parse/lib/sync led me down a rabbit hole of Jest configuration hell.

TDD and the Zero-Defects Myth

Sklivvz.com — Fri, 27 Dec 2024 02:00:00 GMT

If you’ve ever heard someone claim that Test-Driven Development leads to zero-defects code, congratulations, you’ve encountered one of software development’s most enduring myths.

It’s an alluring idea. Write tests first, and your code becomes bulletproof. But like many things in life, the reality is far messier. TDD isn’t a golden ticket to perfection, nor should it be. Here’s why the idea of zero defects is both unrealistic and, in many cases, extremely counterproductive.

Edit: To be fair, Kent Beck is not perpetuating the myth, but in my opinion a lot of people will hear what they want to hear. The myth does exist, but I certainly don't blame it on Kent

Do we even need zero defects?

When was the last time a minor bug caused irreparable harm? For most software, the occasional defect is an acceptable trade-off for delivering faster and focusing on iterating on features that matter to the user. The startup world thrives on this ethos. Move fast and break things, as Facebook famously said. Bugs are annoying, but users often care more about whether your app solves their problems or delights them than whether it’s abstractly defect-free.

Perfection is the enemy of done. The pursuit of zero defects often descends into diminishing returns. Time and effort are burned polishing code that could have shipped weeks ago. Worse, the obsession with eliminating every defect can create a false sense of productivity. Are you building valuable features or just proving that your tests pass?

TDD and the real world

TDD works well in controlled environments with clear, deterministic problems. For example, I've used it fruitfully to write my Z80 emulator. If you can define the inputs, predict the outputs, and cover edge cases, TDD is like a superpower. But the real world is rarely that tidy.

Security is one of the hardest areas to test because it involves knowing what you don't know and not merely verifying what you do know. Tests might ensure queries are parameterized correctly, but can they anticipate every creative payload an attacker might send? Security is a moving target. Exhaustive testing requires a combination of manual review, automated tools, and red-team exercises. These go well beyond what TDD can provide. Either "zero defects" is flatly wrong or "a gaping security hole" does not count as a "defect" in ~~Kent Beck's~~ TDD's definition. Either way, it's not a message that should be boosted.

When was the last time a test suite told you whether users enjoyed using your app? Usability issues are almost impossible to define as failures in the TDD sense because they rely on subjective human experience. Tests might confirm a button performs the right action when clicked, but they can’t tell you whether users can actually find it. The only way to evaluate usability is to test with real people through usability studies or A/B testing. TDD cannot simulate a frustrated user abandoning your app because they couldn’t figure out how to use it. This is an example as to why the number of defects is red herring: if you make a button hard to find, people won't press it so it doesn't matter whether it works.

Performance testing is another area where TDD gives zero meaningful coverage. Performance isn’t about whether something works. It’s about whether it works well under real-world conditions. A checkout function might process orders correctly in tests, but what happens when a thousand users try to check out simultaneously on Black Friday? The best proven way to evaluate performance in real-world scenarios isn’t to rely on synthetic benchmarks. It’s to use canary deployments. Large-scale systems like those at Stack Overflow embrace this practice. Rolling out changes to a small subset of users in production allows you to measure actual impact on latency, throughput, and stability in a controlled way. Canary deployments reveal insights synthetic tests miss, like how systems perform under realistic user behavior and workloads, including cases where your pretty, "defect free" code dies in production because it's not suitable for the high-load scenario where it's run.

Tests are bad specifications

One of TDD’s core principles is that tests double as specifications. The idea is seductive. Write the tests first, and you define your system’s behavior. But how do you know you’ve written enough tests?

Take a simple example. You’re implementing the sine function. Is one test enough?

assert sin(0) == 0

Clearly not. How about these?

assert sin(0) == 0
assert sin(pi/2) == 1
assert sin(-pi/2) == -1

Still not enough. No matter how many test cases you write, it’s trivial to create a broken implementation of sin(x) that passes them all. Writing enough tests to truly define a function’s behavior isn’t just hard. It’s often impossible. The way it's done in TDD is by often by basically cheating: write a few tests, write the obviously right code you know in advance, and, lo and behold, it passes the tests you selected. Then you "decide" the tests are arbitrarily enough.

Of course, in the real world, we find extensive use of this "technique" and this spoils the "ideal world" scenario and this also makes it clear to you why "zarro boogs" is a bit of a joke.

Test-induced damage

Even ideal world TDD isn’t without its downsides. One of the most notorious is test-induced design damage. This happens when the need to even have tests distorts the design of your code. You might inject mocks into every crevice of your system to satisfy unit tests, creating brittle, over-engineered structures that are harder to maintain. Or you might focus so heavily on making tests pass that you lose sight of what your code is actually supposed to do. In the worst cases, the tests become the goal, not the guide. You end up with a green test suite and a broken product.

That’s ironic for a methodology meant to improve quality.

Counting bugs is meaningless

TDD advocates often focus on reducing the number of bugs as a key metric. But not all bugs are created equal. You can ship software with a hundred tiny bugs no one notices and be fine. One critical bug, though, can sink your system.

Consider the Amazon AWS S3 outage in 2017, caused by a simple human error in a command-line operation. This wasn’t a coding bug. It was a process flaw, and a failure to anticipate bad input, that brought down a huge chunk of the internet. TDD would not realistically have caught that because it's nearly impossible to predict all ways in which a system might be misused. Focusing solely on the number of bugs is like counting raindrops in a storm while ignoring the flood about to hit. You might have your mythical "zero defects" and bring down the internet.

Resilience is the goal

TDD is a valuable tool, but like any tool, it has its limits and a specific domain of applicability. The real world is messy, unpredictable, and full of problems that cannot be solved by writing tests first. Chasing zero defects is almost always a distraction from what really matters. Building resilient systems, solving real problems, and delivering value to users.

Instead of obsessing over perfection, focus on pragmatism. Use TDD where and if it helps, but don’t treat it as a panacea or spread misinformation about its applicability. Users don’t care how you wrote your code. They care that it works. Sometimes, that means shipping something imperfect but functional, learning from feedback, and improving as you go.

Perfection was never part of the job description.

Certification Exam Resources

vladsiv — Fri, 27 Dec 2024 02:00:00 GMT

I've put together a collection of resources I've used while preparing for various certification exams - covering everything from AWS to Databricks. The focus is mainly on data engineering and machine learning topics.

The MMT Perspective on Crypto

Bytepawn - Marton Trencseni — Fri, 27 Dec 2024 01:00:00 GMT

From an MMT (chartalist) perspective, cryptocurrencies lack sovereign backing and fail to establish the enforced demand necessary to function as true currency. Because they rely solely on speculative sentiment rather than legal mandates (like taxation), their values are unsustainable in the long run and risk ultimately collapsing to zero.

Vandring i Hågadalen

Liza Shulyayeva — Thu, 26 Dec 2024 23:01:33 GMT

Idag visade en vän mig Hågadalen. Det är så härligt att ha så vacker natur på promenadavstånd från mitt hem. Vi vandrade dit från hans lägenhet efter att ha druckit lite alkoholfri glögg. Jag tog med fika, och han hade med sig utsökta lunchsmörgåsar.

Programming Lewis Carroll's Memoria Technica

ℤ→ℤ — Thu, 26 Dec 2024 22:44:43 GMT

Charles Dodgson (pen name Lewis Carroll) had difficulty remembering numbers, such as dates. He developed a cipher to help him remember numbers by embedding them in couplets or phrases. For example, the couplet “Brass trumpet and brazen bassoon, will speedily mark you a tune” encodes the specific gravity of brass (8.39) in the last four consonants: r k t n (y is treated as a vowel). In this article, we describe the cipher, present online tools for encoding and decoding, discuss how we implemented the algorithms in TypeScript, and the cipher’s relevancy to steganography.

Memoria Technica

The cipher encodes plaintext’s of sequences of digits. Using the cipher, consonants (except ‘y’) are mapped to digits. Vowels, punctuation, and ‘y’ are ignored and thus can be freely added to the ciphertext The mapping follows the figure:

Mapping of consonants and digits

In a three-page pamphlet published by Carroll in 1878, he describes how the digits and their associated letters came to be:

"b" and "c", the first two consonants in the Alphabet.
"d" from "duo"; "w" from "two".
"t" from "tres"; the other may wait awhile.
"f" from four; "q" from "quatuor".
"l" and "v", because "L" and "V" are the Roman symbols for "fifty" and "five".
"s" and "x", from "six".
"p" and "m", from "septem".
"h" from "huit"; and "k" from the Greek "okto".
"n" from "nine"; and "g" because it is so like "9".
"z" and "r", from "zero".

The letter “j” is associated with 3 because it is left-over.

Decoding ciphertext involves extracting the consonants from a message and finding the mapped integer. Vowels, y, and punctuation are ignored.

Encoding an integer involves choosing one of two consonants for each digit and inserting vowels, y, and punctuation to create valid words.

Decoding may also involve “meta” information. In the pamphlet, Carroll provides example couplets for encoding the specific gravities of various metals. The decoder will need to know that only the last four consonants are relevant.

Example

Given the ciphertext found, we can first filter to the consonants: fnd. f is 4, n is 9, and d is 2, so the plaintext is 492. (What was found in 492? Christopher Columbus discovered America in 1492 and Dodgson believed the leading 1 could be assumed.)

Given the plaintext 492, we are interested in words that include the consonants f or q, n or g, and d or w, in that order, with vowels, ‘y’, or punctuation in-between. Using the second edition Scrabble dictionary, we find 15 single words that match these constraints:

fend	fiend	find	fined	foined
fond	fondu	fondue	found	fugued
fund	fundi	queened	quinoid	quoined

Using our encoding tool, we found 74,151 possibilities from fend to quey yuga yowie.

Decoding and Encoding Tools

Note 1: These computations run on your computer.

Note 2: The JavaScript code for Encode uses import attributes which, as of December 2024, are not supported on Firefox, but are expected to be available soon.

Note 3: Negative limits are treated as no limits (i.e. Infinity). Negative offsets are treated as no offset (i.e. zero).

Implementation

Mapping Consonants to Digits

Our implementation of the mapping is a fairly straight-forward switch case with fall-through:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


/**
 * For a given letter, return the cipher digit based on Carroll's Memoria Technica scheme, or null if there
 * is no associated digit. This function will return null for vowels (including y), punctuation marks, and
 * any other non-consonants. Only single letter English consonants have values; case is ignored.
 *
 * @param chr a single character
 */
export function consonant_value(chr: string): number | null {
    const code16: number | undefined = chr.toLowerCase().codePointAt(0);
    switch (code16) {
        case 98 /*'b'*/:
        case 99 /*'c'*/: return 1;
        case 100 /*'d'*/:
        case 119 /*'w'*/: return 2;
        case 116 /*'t'*/:
        case 106 /*'j'*/: return 3;
        case 102 /*'f'*/:
        case 113 /*'q'*/: return 4;
        case 108 /*'l'*/:
        case 118 /*'v'*/: return 5;
        case 115 /*'s'*/:
        case 120 /*'x'*/: return 6;
        case 112 /*'p'*/:
        case 109 /*'m'*/: return 7;
        case 104 /*'h'*/:
        case 107 /*'k'*/: return 8;
        case 110 /*'n'*/:
        case 103 /*'g'*/: return 9;
        case 122 /*'z'*/:
        case 114 /*'r'*/: return 0;
        default: return null;
    }
}

The original version (as suggested by the commented code) compared strings against strings (JavaScript lacks a character type). Profiling revealed that more than half the time was being spent in StringEqual calls. By switching the comparisons from string equality to integer equality, the new code takes approximately 15% of the time than the older version or a speed-up greater than 6x.

Decoding Ciphertext

The decoding process is essentially a functional collect: map across all elements and keep those that are non-null.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


/**
 * Return an array of numbers based on decoding s. If there are no (valid) consonants within s, returns
 * an empty array.
 *
 * @param s word or phrase
 */
export function decode(s: string): number[] {
    let result: number[] = [];

    for(const chr of s) {
        const v: number | null = consonant_value(chr);
        if(v !== null) {
            result.push(v);
        }
    }

    return result;
}

Alternatively, TextEncoder could be used to build a Uint8Array of byte values and consonant_value could be given a byte value as an argument. However, since a character in UTF-8 may span multiple bytes, we would still need to capture the complexity of Unicode encoding somewhere.

Encoding a (Specific) Plaintext Value

Using (Moby 2002), we previously developed a dictionary by mapping each word into a cipher value and then grouping words by a shared cipher value. Thus, generating a list of words that match a given plaintext digit string is a look-up:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


/**
 * Return from the dictionary all words that have the plaintext value of n.
 *
 * @param n plaintext value
 * @param dictionary plaintext value to array of words
 */
export function encode(n: string, dictionary: Map<string, string[]>): string[] {
    if(dictionary.has(n)) {
        return dictionary.get(n)!;
    } else {
        return [];
    }
}

Using a JavaScript Map for the dictionary rather than a JavaScript Object is a bit of a personal quirk. JavaScript’s Maps are restricted to string keys, which in this case is not a restriction for our domain. We appreciate Map’s clear separation between keys and properties and likely improved performance given its less flexibility compared to Objects.

The exclamation point after the get(n) call is a TypeScript type assertion informing the compiler that the return value is neither null nor undefined. The compiler is (not yet) smart enough to recognize that since has returned true, get cannot return undefined. At least, unless the dictionary is modified by another process.

Encoding a (General) Plaintext Value

While encode returns single word ciphertexts that fully match a given plaintext, we will often need groups of partial matches for a given plaintext. encode_all is a generator function that will yield one ciphertext at a time, where each ciphertext contains one or more words. As described in the ‘English Language Coverage/Branching Factor’ section below, most plaintexts will yield large numbers of potential ciphertexts (e.g. thousands for three digits). Since the consumer of encode_all is unlikely to need all the outputs at the same time, we use a generator or iterable design to avoid loading all the outputs into memory at once.

(If encode_all is a generator, why not encode? In encode’s case, the dictionary provides all outputs at once so we do not save any resources by yielding only a single entry from that dictionary at a time.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


/**
 * Return all phrases (space delimited words from the dictionary) that encode the sequence of
 * digits in n.
 *
 * @param n sequence of digits (plaintext)
 * @param dictionary plaintext value to array of words
 */
export function* encode_all(n: string, dictionary: Map<string, string[]>): Generator<string> {
    const digits = n.split('');
    const nonEmpty = (seq: any[]) => seq.length > 0;

    let divisors: number[] = [];
    for(let i = 1; i < digits.length; i++) {
        divisors.push(i);
    }

    for(let divisor of subsets(divisors)) {
        divisor.unshift(0);  // include all digits by starting the slice at index 0
        let slices: string[][] = sliceIndices(digits, divisor);

        let phrases: string[][] = [];
        for(let slice of slices) {
            let words = encode(slice.join(''), dictionary)
            phrases.push(words);
        }

        if(phrases.every(nonEmpty)) {
            for(let wrappedPhrase of cartesian(phrases)) {
                yield wrappedPhrase.underlying.join(' ');
            }
        }
    }
}

Line 9 separates a string of digits into an array of digit characters. This function assumes digits will only contain the characters ‘0’ to ‘9’.

This function needs to check all ordered sub-sequences of the sequence, digits. For example, if digits is ['1', '2', '3'], then we will attempt to encode: ['1', '2', '3'], ['1'] + ['2', '3], ['1', '2'] + ['3'], and ['1'] + ['2'] + ['3']. To do this, we incrementally add zero or more divisors that separate the array. A divisor splits an array from a given index (inclusive) to the right. Each split is referred to as a ‘slice’.

In lines 21 through 25, we encode each slice (the slice.join('') transforms an array of digits into a string of digits) and collect all possible words under an array of arrays termed phrases.

If all positions within a phrase contain at least one word (lines 27 through 31), then we perform a cartesian join between each index. A cartesian join or product is similar to a nested for-loop and generates all possible tuples. The yield gives the caller a single value (a space-separated list of words) while retaining state so the next value can be retrieved without re-computing earlier elements.

The code for subsets, sliceIndices, and cartesian (among other functions) is in util.ts in this project’s gitlab repository. Knuth’s The Art of Programming, Volume 4A is a useful resource for these kinds of algorithms.

Suitability for Steganography

Steganography is the art of hiding information in plain sight. While cryptography obscures the message so the attackers have difficulty reading it, attackers will know that the message is not meant to be read. In contrast, steganography may trick the attackers into believing the message is benign or not noticing the message at all. For example, invisible ink and microdots are forms of steganography. Similarly, a message may be hidden by distorting specific pixels of an image. Steganography and cryptography may be combined with the message first encrypted and then hidden.

While Dodgson did not present this technique as a cipher nor as a tool for steganography, but only as an aid to memory, we believe it has potential utility.

Let’s try to hide phi, 1.61803, as a classified ad. The ad will start with ‘For sale: ' and the item for sale will be an encoding of the number. We will assume the reader can place the decimal point appropriately (or we transmit that data via some other way). Using the encode tool above, 161803 yields some promising outputs within the first few hundred cases:

“box chart”
“boys chariot”
“cosy chariot”

Would an attacker suspect ads like these?

For sale: Box Chart 2016. Software library for R. $1 + postage.
For sale: Boy’s Chariot. Clean, blue, ages 3-6. $100 OBO.
For sale: Cosy Chariot. Padded seats, yellow. One seater.

We chose these three examples after scanning about 3,000 outputs. The dictionary used by the tool contains almost 118,000 words (including plural and forms of tenses).

English Language Coverage/Branching Factor

The branching factor (number of words in the dictionary that match a given sequence) is numerous for single and double digit sequences. (Note that leading zeros are allowed.) The first table below represents the number of words that perfectly match the single digit sequence while the second table represents two digit sequences.

Digit	Count
0	63
1	43
2	53
3	45
4	21
5	54
6	46
7	57
8	51
9	64

	0	1	2	3	4	5	6	7	8	9
0x	31	36	59	44	12	61	82	43	23	58
1x	65	41	53	61	9	69	71	43	46	61
2x	61	31	54	37	9	70	65	43	27	59
3x	44	25	37	29	6	38	48	45	30	52
4x	53	10	21	27	8	35	26	11	7	33
5x	50	34	61	41	12	65	75	44	24	95
6x	37	20	27	37	6	46	42	40	29	37
7x	78	41	64	49	9	71	94	67	36	107
8x	34	13	41	23	8	41	50	38	28	37
9x	56	35	55	48	8	68	83	56	20	84

There are 78 three-digit sequences where do we not see any single word in the dictionary (although these three-digits could be separated into one and two-digit subsequences): 008, 014, 048, 084, 088, 128, 147, 148, 204, 224, 234, 248, 264, 294, 314, 324, 334, 342, 347, 348, 349, 354, 364, 394, 404, 411, 414, 421, 423, 424, 427, 428, 434, 437, 447, 464, 467, 471, 474, 481, 483, 484, 487, 494, 534, 541, 547, 564, 574, 581, 584, 624, 634, 644, 661, 747, 748, 774, 784, 814, 817, 821, 827, 834, 837, 841, 845, 847, 848, 849, 864, 874, 881, 914, 948, 974, 981, 984.

If the specific words do not matter, then any given plaintext can be encoded into a reasonable selection of words. If specific words do matter (i.e. the ciphertext is being hidden), then this branching factor may not be sufficient to cover all contexts. For instance, it may not be possible to consistently hide messages within recipes since the number of ingredient-related words is restrictive.

Weakness? Benford’s Law versus Letter Frequencies

A potential weakness of this approach as a steganographic technique is that the frequency of digits does not match the frequency of consonants. For instance, the digit 4 maps to ‘f’ and ‘q’. In studies of Benford’s Law, which measures the frequency of digits in real-world contexts, a 4 is predicted as the leading digit in 9.7% of all numbers (Berger 2015). However, ‘f’ is seen as the leading letter in 2% of words and ‘q’ is very rare. Thus, a study of letter frequencies in a passage could detect an anomaly based on the imbalance. However, unless the volume of messages is high or the plaintext is very long, we suspect this would fall into the noise.

References

(Abeles 2005) Abeles, Francine F. 2005. Lewis Carroll’s ciphers: The literary connections. Advances in Applied Mathematics, Volume 34, Issue 4. Pages 697-708. ISSN 0196-8858, https://doi.org/10.1016/j.aam.2004.06.006.

(Berger 2015) Berger, Arno, and Theodore Preston Hill. 2015. An Introduction to Benford’s Law. Princeton, New Jersey: Princeton University Press. First Chapter

(Gardner 1996) Gardner, Martin, and Lewis Carroll. 1996. The Universe in a Handkerchief : Lewis Carroll’s Mathematical Recreations, Games, Puzzles, and Word Plays. New York: Copernicus. Pages 31-36.

(Moby 2002) Ward, Grady. 2002. Moby Word II. https://www.gutenberg.org/ebooks/3201

Gitlab Repository. https://gitlab.com/jeffrey_starr/memoria_technica

The complete text of "All Watched Over by Machines of Loving Grace"

John Graham-Cumming's blog — Thu, 26 Dec 2024 17:34:02 GMT

Richard Brautigan's poem "All Watched Over by Machines of Loving Grace" is somewhat well known in tech. circles but I couldn't find a complete PDF of the original 1967 publication of it (and other poems) online.

The copyright notice in Brautigan's collection reads:

Permission is granted to reprint
any of these poems in magazines,
books and newspapers if they are
given away free.

I am interpreting "print" as including the availability of a free PDF and making a scan of the complete book available here.

A Reply to Don Geddis

Rondam Ramblings — Thu, 26 Dec 2024 10:12:48 GMT

Don Geddis left a comment on my last post. My reply grew far longer than would reasonably fit into a comment reply so I decided to post it as an article. Don wrote:I wonder if you've considered that perhaps you have more in common with the people who frustrate you, than your current self-image suggests.My reply:I've not just considered it, I will happily concede that I am not as

Books I read in 2024

Nicola Iarocci — Thu, 26 Dec 2024 08:36:24 GMT

I read 30 books or 8365 pages in 2024, a solid improvement over last year’s, and many of those books were excellent. Lolita by Vladimir Nabokov was outstanding, Marcus Aurelius’ Meditations was incredible, and then there’s Family Lexicon and The garden of Finzi-Contini, and many others were close to that league. Yeah, color me satisfied.

The usual scoring system applies:

One star means a book is meh.
Two stars mean a book is perfectly fine.
Three stars mean a book is good—consider it recommended.
Four stars mean a book is exceptional.
Five stars is pretty much unheard of.

The question of Palestine, by E.W. Said

(La Questione Palestinese, il Saggiatore)

RSS Feeds and Real Time Crawling

Weblog on marginalia.nu — Thu, 26 Dec 2024 02:00:00 GMT

A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days. Conditions being if the website has an RSS or Atom feed.

The big crawl job takes about two months, and is run partition by partition, meaning there’s typically a slice of the index that is two months stale at any given point in time. To help compensate for this, a new crawler and index partition has been added that focuses on recently updated content.

Picking up volleyball in NYC with Goodrec and New York Urban

Notes on software development — Thu, 26 Dec 2024 02:00:00 GMT

I was so intimidated to go at first, but it is in fact easy and fun to start playing beginner volleyball in New York. The people are so friendly and welcoming that it has been easy to keep playing consistently every week since I started for the first time this August. It's been a great workout and a great way to make friends!

The two platforms I've used to find volleyball games are Goodrec and New York Urban. While these platforms may also offer classes and leagues, I mostly use them to play "pickup" games. Pickup games are where you show up and join (or get assigned to) a team to play for an hour or two. Easy to go on your own or with friends.

I'm not an expert! My only hope with this post is that maybe it makes trying out volleyball in New York feel a little less intimidating for you!

Goodrec

With Goodrec you have to use their mobile app. Beginner tier is called "social" on Goodrec. So browse available games until you find one at the level you want to play. You enroll in (buy a place in) sessions individually.

Sessions are between 90-120 minutes long.

They ask you not to arrive more than 10 minutes early at the gym. When you arrive you tell the gym managers (usually in a desk up front somewhere) you're there for Goodrec and the tier (in case the gym has multiple level games going on at the same time). Then you wait until the Goodrec "host" arrives and they will organize everyone into teams.

Goodrec hosts are players who volunteer to organize the games. They'll explain the rules of the game (makes Goodrec very good for beginners) and otherwise help you out.

Always say thank you to your host!

New York Urban

With New York Urban, pickup sessions are called "open play".

There is no mobile app, you just use the website to purchase a spot in a session. The sessions are longer and cheaper than Goodrec. But there is no host; players self-organize.

The options are more limited too. You play at one of four high schools on either a Friday night or on Sunday. And session slots tend to sell out much more quickly than with Goodrec.

Big City Volleyball

You can also check out Big City Volleyball but I haven't used it yet.

Volo

I haven't ever done Volo but I think I've heard it described as "beer league". That even some of the beginner tier sessions with Goodrec and New York Urban are more competitive.

But also, Volo is built around leagues so you have to get the timing right. Goodrec's and New York Urban's pickup games make it easy to get started playing any time of year.

Making friends

It was super awkward to go at first! I went by myself. I didn't know what I was doing. I couldn't remember, and didn't know, many rules. I didn't have court shoes or knee pads.

But the Goodrec host system is particularly great for bringing beginners in and making them feel welcome. You have a great time even if you're terrible.

The first game I went to, I tried to hang out afterward to meet people. But people either came with their SO or with their friends or by themselves so they all just left immediately or hung out in their group.

So you can't just go once and expect to make friends immediately. But if you keep going at the same place and time regularly week over week, you'll see familiar faces. Maybe half the people I play with each week are regulars. If you're friendly you'll start making friends with these people and eventually start going out to bars with them after the games.

Improving

Even if you find yourself embarrassingly bad at first, just keep going! I'm 29, 6'1, 190lbs and from observation the past 5 months, age, height, and weight have a very indirect relation to playing ability.

Most of the people who play are self-taught, especially at the lower tiers I've played at. But some people played for the school team in high school or college. These people are fun to play with and you can learn a lot from them.

Most people who are self-taught seem to watch YouTube videos like Coach Donny, helpful for learning how to serve, set, block, etc. Or they take "clinics" (classes) with Goodrec or other platforms. (I have no idea about these, I've never done them before.)

At first I played 2 hours a week and I was completely exhausted after the session. Over time it got easier so I started playing 2-3 sessions a week (6-9-ish hours). With practice and consistency (after about 3-4 months), I started playing Intermediate tier with Goodrec and New York Urban. And I don't think I'll play Beginner/Social at all anymore.

I still primarily play for fun and for the workout and to meet people. But it's also fun to get better!

I played with one person much better than myself in an Intermediate session one time and he mentioned he will probably stop playing Intermediate and only play High Intermediate. He mentioned you get better when you keep pushing yourself to play with better and better players. Good advice!

I wrote a little post on picking up volleyball in new york.

It's fun, and a great workout, and you meet interesting people!https://t.co/jEWHbRWF6C pic.twitter.com/ipuIUB1ZnM
— Phil Eaton (@eatonphil) December 26, 2024

Low and steady write rates can be misleading

Sophia Willows' Blog — Thu, 26 Dec 2024 02:00:00 GMT

A common mistake in system design interviews is to underestimate a low but steady write rate. Many candidates quickly dismiss a write rate of 1 request per second as insignificant, but this isn’t always the case—especially in a startup environment where growth is measured week-over-week.

Small write volumes can accumulate significantly over time and cause all sorts of downstream problems. During a major Postgres version upgrade, Retool found that half of their 4 TB database was taken up by audit events—many of which they didn’t need to be readily accessible due to their age. At 1 write per second, you will accumulate 31.5m rows over the course of the year. This isn’t an overwhelming number of rows—depending on the details of what you’re doing, this is most likely fine—but it’s a larger number than most people expect.

That 31.5m figure assumes your company sees zero growth over the year, which is unlikely to be true. A successful startup will grow week-over-week, and weekly growth compounded over the course of a year can balloon writes significantly. The graph below shows the cumulative number of rows written during year 1 under three different weekly growth rates:

Read more on sophiabits.com

I ❤️ triathlons

Bytepawn - Marton Trencseni — Thu, 26 Dec 2024 01:00:00 GMT

In 2008, I completed my first half-Ironman — then, later that same year, I took on the full Ironman distance at the traditional ExtremeMan in Hungary, finishing in 13 hours and 38 minutes. By 2013, I had knocked my time down to 11 hours and 36 minutes. Since then, I’ve racked up a total of 56 triathlon finishes. 2024 was one of the best years so far, with a record of 10 triathlon races.

Happy Holidays: Snowflakes

xorvoid — Wed, 25 Dec 2024 16:54:50 GMT

Happy Holidays: Snowflakes

The Only Self-Improvement Article You Will Ever Need. Identity is Everything.

Josh Baldwin — Wed, 25 Dec 2024 15:08:35 GMT

Hey now, I signed up for moral philosophy and thoughts on the pro-animal movement.

Merry Christmas

xorvoid — Wed, 25 Dec 2024 10:14:33 GMT

Merry Christmas

Catching up with async Rust

fasterthanli.me — Wed, 25 Dec 2024 09:30:00 GMT

In December 2023, a minor miracle happened: async fn in traits shipped.

As of Rust 1.39, we already had free-standing async functions:



pub async fn read_hosts() -> eyre::Result<Vec<u8>> {
    // etc.
}

…and async functions in impl blocks:



impl HostReader {
    pub async fn read_hosts(&self) -> eyre::Result<Vec<u8>> {
        // etc.
    }
}

But we did not have async functions in traits:

My 2024 reads

Alexander's Blog Blog — Wed, 25 Dec 2024 02:00:00 GMT

Peopleware: Productive Projects and Teams

My Favorite Christmas SNL Sketches

The David Brownman Blog — Wed, 25 Dec 2024 02:00:00 GMT

A quick recap of the holiday sketches I come back to, year after year.

Read the whole thing.

A Cryptographically Secret Santa

Speculative Branches — Wed, 25 Dec 2024 02:00:00 GMT

Twas about 4-6 weeks before Christmas, and all through the math department, not a creature was stirring, not even a plucky young undergrad. Cryptography professors Alice and Bob sat at the elliptically-curved conference table to plan the department's secret Santa. Mallory, the department secretary, had been given the task of organizing last year, and somehow managed to get three gifts while leaving several people disappointed. This year's math department thus resolved to do their secret Santa without a trusted party.

How I manage my personal Fedora machine

blog.marbu.eu — Wed, 25 Dec 2024 02:00:00 GMT

Posted on 25 December 2024

Tags: Fedora, ansible, git, etckeeper

Table of Contents:

OS Installation
Post Installation setup via Ansible
Etckeeper
Backup
Upgrading OS
Reinstall or Migration
Disaster Recovery

In this post I’m going to give a brief overview of how I manage my personal Fedora laptop, from installation to upgrades and backups. This is indeed not very original topic, but recently I discussed it with few people, and I realized that having a reference to a post like this will be useful. And maybe you will find some parts of it interesting.

Important assumption here is that that this approach is intended for a single desktop machine I both own and use. If I had to manage a fleet of desktops for other people, some choices here no longer makes sense.

OS Installation

First of all I need to install the operating system. I boot from Fedora KDE Plasma live image and from there start Fedora installer.

The most important step during the installation is disk configuration. I always use the whole disk erasing whatever was there, enable LUKS based full disk encryption, select btrfs partitioning scheme and then let the installer create a default configuration I can review and tweak as needed. I insist on a separate volume for /home, but sometimes I need other changes such as separate volume for virtual machine images. If I were not using btrfs, I would also want to control how much disk space is left unused for later use.

Disk partitioning during Fedora 41 installation

I don’t configure any additional user accounts, but I need to make sure root account is enabled and set a password for it.

Post Installation setup via Ansible

After the installation I login via text console as root and install git and ansible so that I can clone and run my ansible post installation playbook. This way I perform common boring post installation steps such as enabling rpmfusion free repository, creating user accounts or installing additional rpm packages.

This set of ansible playbook and roles serves both as a documentation and as a scripted way to perform the setup. Ansible playbooks can be actually quite readable if you care enough about that, and roles can include a README file with references to documentation and reasoning behind the configuration. Some of the roles I have there just implements a few steps taken from Fedora documentation or wiki, such as the role for kvm/libvirt based virtualization, while others cover a personalized setup, like this custom setup for a lightdm display manager.

If you are considering this approach yourself, have a look at Linux System Roles project (btw Fedora has it packaged). They provide set of general ansible roles you can parametrize via set of variables passed to it. This can save you some time spent on implementing and testing a role. I don’t use it in my post install playbook myself since I created my playbook way before this project started.

And even though the work I put into this is indeed bit higher compared to a case when I just do them manually and then write it down somewhere, I don’t need to reinstall a machine again to get the time and effort invested into this back. Just by rerunning the playbook I can check (and enforce) whether the configuration I did long time ago is still used. Sometimes I just tweak or create playbook/role to reconfigure something long after the installation, or I use some of the roles in a different context, eg. for setting up a testing virtual machine. That said, it’s worth noting that because of changes in the distribution and ansible ecosystem itself, additional maintenance is needed every now and then.

Etckeeper

That said I don’t use ansible for everything. In most cases I just edit config files in /etc directly (or use tools like firewall-cmd which edits files there on my behalf). Only if I conclude that the automation effort makes sense, eg. because I will need given system change next time I reinstall the system, I sit down to write ansible playbook or role for it.

This means that I still need to keep /etc organized, which I manage by tracking it with git using etckeeper. After I change something in /etc directory, I create a dedicated commit for the change, including a description of what I’m doing in a commit message. This way I know what, when and why I have changed something. An important piece of etckeeper tools is a hook for a package manager, so that any change made in /etc during package installation, removal or update is captured in a dedicated commit along with all relevant metadata (list of packages, versions …). The configuration I use is available in my etckeeper ansible role, but in short, I disable daily auto commits forcing myself to commit any changes explicitly.

But using both ansible as well as etckeeper creates one problem: how to commit changes done during run of a playbook? Making one big commit for the whole run would be easy, but it will result in a commit too large to be useful. To solve this I wrote a simple etckeeper ansible callback plugin hooking ansible playbook result callbacks to create commit for each ansible task in a playbook. The only downside of this approach is that it works locally only, but it’s not a problem for my use case here.

Overall this approach has multiple benefits:

There is a clear record of all configuration changes with additional metadata.
Based on a commit message I can see if it comes from a default configuration, was done via my ansible playbook or manually.
I can use standard git tools (like git blame, git log or git cal) to get more insight.
When I run into a problem, I can have a look what I changed recently and recheck if that can be related.
Since easy fallback via git checkout or git revert is available, I can experiment with a configuration without having to manually backup existing state.
I will notice unexpected changes performed automatically by some system component (I can’t miss this since it will leave uncommited changes) or can inspect changes which comes with new version of a package.

Another benefit of this approach is that it provides a convenient way to archive and review old system configurations. I used to create tarballs of /etc directory just in case I may need it later, but I almost never used it for anything, even when I had some reason to do so. This is because the tarball approach lacks context (What was the thing I changed compared to the default and why? Does it actually contain the change I’m interested in?) and standardization (Where do I actually store it?). Etckeeper solves all this. I can just push current state of main branch into separate git bare repository. Later on, when I want to revisit it on a different system, I can just fetch this archive repository as a remote in existing etckeeper managed repo, and then I’m able to work with it easily. For example (assuming name of the archive remote is dione-2018):

List commits in the archive remote: git log --graph --oneline dione-2018/master.
Compare the archived state with the current one for a given file, eg.: git diff dione-2018/master HEAD dnf/dnf.conf
Use particular old commit in the current /etc repo via git cherry-pick.

Backup

I backup whole /home directory. On a personal machine, I don’t really care about anything else, with exception of /etc directory, for which I maintain etckeeper remote bare repo in /home dir. This way, I don’t need to think about what is crucial enough for a backup, everything in /home dir is! And it also makes recovery planning simple. Other backup schemes can be used on top or along side of it, but that is out of scope of this post.

This approach relies on the fact that I use dedicated home volume and a partitioning scheme which allows for quick copy-on-write snapshots so that I can make a shapshot instantly and then continue using the machine while a backup is running from a consistent state. This is helpful especially when the backup itself takes longer because of network or iops bottleneck on the receiving end.

At first I used lvm thin pools, but few years ago I moved to btrfs, so that now I can use btrfs send/receive functionality to transfer incremental snapshots to a local or remote backup device. Here I can run btrfs scrub to make sure that I still have the data.

To restore data, I will just use send/receiver again, but this time transferring data from the backup device to the machine.

Upgrading OS

I need to upgrade to a new Fedora release every half a year or so. But since a Fedora release is supported for about 13 months, I’m not forced to upgrade until I can find some time to actually do it.

Nowadays I use Fedora supported dnf system-upgrade. This means that the upgrade is handled by the package manager. First it downloads all new rpm packages and runs dnf transaction check on them (so that one can detect a package conflict before the upgrade), and then if all looks good one reboots into system-update systemd target to perform the upgrade in an isolated environment.

I used yum/dnf based upgrade long before this was polished into an official upgrade path, since I always preferred this approach over other options. Reinstalling the system just to upgrade a personal machine doesn’t feel reasonable to me.

Reinstall or Migration

Now you may wonder why do I mention reinstall when I just noted that I use dnf based upgrade. But in case of significant hardware changes (buying a whole new machine or just replacing small or dying disk with a new one) I find it easier and faster to just reinstall the system from scratch, assuming that the OS installer will detect the new hardware and set appropriate defaults. The same applies to a significant changes in storage configuration or partitioning. You may be thinking that there is no reason to change storage setup so drastically, but I did that few times, eg. when I wanted to reencrypt my LUKS volume using latest defaults or moved from lvm thin pools to btrfs.

And thanks to my ansible post installation automation and backups of home volume, this process boils down to:

making backup of home volume
(re)installing the system
post install setup via ansible playbook
recovery of home volume backup snapshot

Obviously this requires much more effort compared to the sheer dnf upgrade, but it’s manageable since it’s documented/automated. Most effort usually goes into considering new hardware or software configurations or maintenance of ansible code as I noted already.

Disaster Recovery

Let’s say a disk in my machine stops working out of sudden so that I need to buy a new one and then restore it into working condition as soon as possible. And as you have probably noticed, I have all the pieces in place to handle this via reinstall followed by a full backup recovery as explained above. Moreover since I reinstall my machine every now and then, I have the recovery procedure fully tested in real conditions. And yes, I don’t do such a test very often, it’s still way better than trying it out for the first time while recovering a machine under stress. The last thing I want to run into is that something doesn’t actually work as I thought it would.

Do you have a comment or question? You can contact me.

I have failed. Now what?

Rondam Ramblings — Tue, 24 Dec 2024 20:41:35 GMT

Nearly two months ago now I wrote:It's getting harder and harder to find a reason to keep doing this. My opportunity costs are high, and writing a blog entry takes a non-trivial amount of time. I wrote this because I needed to blow off some steam, and I wanted to get my position on the election results on the record while they were still fresh in my mind. Whether I keep

Practical Deep Learning, Lesson 7, Movie Recommendations

Thought Eddies — Tue, 24 Dec 2024 20:33:43 GMT

Practical Deep Learning, Lesson 7, Movie Recommendations

Elon Musk

Ananth's Reflections — Tue, 24 Dec 2024 16:40:08 GMT

Some interesting principles I learnt from Walter Issacson’s Elon Musk biography

Bonkoski University: Fall/Winter Semester 2024

xorvoid — Tue, 24 Dec 2024 13:46:09 GMT

Bonkoski University: Fall/Winter Semester 2024

Pareto Hobbies 2025

jodavaho.io — Tue, 24 Dec 2024 02:00:00 GMT

Pareto Hobbies 2025

I decided to plot possible hobbies in 2025 in terms of career potential and personal enjoyment.

This is a silly thing to do, but it is also a uniquely software-engineering problem to have - for some reason we seem to be prone to lots of extra curricular learning, quantified self, and a constant need to optimize ourselves for our employability.

As a computer scientist and sometimes hardware hacker, I’m most interested in the following things:

Geospatial ML

I’m fascinated by taking spatial data and running predictors on it. This could be predicitng fine-grained weather patterns, predicting traffic, etc.

At flight science, this has lots of potential applications, so the usefulness is fairly high. I’ve also found that kriging is very useful for robotics though it has been 10 years since I’ve applied it.

Quantum Computing

The recent google announcement has reinvigorated my old interest in quantum computing. A friend from JPL who I highly respect has said QC is at the equivalent of the vaccuum tube era of computing.

Given we’re only now demonstrating benchmark problems, and that most the researchers in those areas are probably hardware and quantum PhDs …. well, I’m a bigshot PhD Computer Scientist, but doubt I could contribute to the field without years of retraining.

Relevance: Very low.

Radar Processing

Super-specific interest, but I want to use and SDR to build a homebrew radar and track boats at the in-laws lake cabin. Why? Who knows.

I’m super comfortable with tracking and estimation, but would need to learn how to extract a “measurement” from the timing and Doppler shift of the radar return.

Given I could see myself working in tracking and estimation in the future, this could be a useful skill.

AWS code as infra

Not very glamorous, but I hate the AWS console and would like to be able to do everything in code.

RL for planning

This is something that is near and dear to my heart. I have been working in planning systems for a while, and have used “RL” (POMDP, etc) in the past, but only as a tangential part of the team.

More generally, the use of end-to-end learned systems for control at least seems to be an increasingly dominant way of doing business. With RL+Planning, my goal is to eventually be able to tradeoff classical estimation and planning with end-to-end systems, and use the best parts of either for a better-than-either pipeline. At the jobs I’ve had I’ve seen a weird false dichotomy that produces decent results once committed, but there’s always this tension that “the other way” is better, producing weird politics.

Most recently, a classical planning pipeline was completely broken by an ML-based estimation of orientation of obstacles. On inspection they were using quite possibly the worst estimator you could use, and ignoring all kinds of good data, because (I think) they were counting on perfect “measurements” from the ML-based vision system. That kind of thing must happen all the time - get stuck in a local minimum of “tune just a little better” when a good filter on top of a noisy estimator could solve all your problems. There’s probably a million such examples in planning systems too.

For discrete, well structured problems with noisy data on them, such as we might encounter in flight science, I think there’s a lot of potential for RL to be extremely useful.

Deep Learning / Large Language Models

I’m not super interested in this, but it seems like a good thing to know.

Ratings

My interest and career potential vary. I’ve rated them on a scale of 1-100, with 100 being the highest.

This is a stupid way to determine hobbies, but I’m doing it anyway.

Item	Career relevance	Personal Interest
RL for planning	50	40
DL/LLM in general	20	35
Quantum Computing	0	90
AWS Code as infra	60	10
Radar / signal processing / tracking	20	80
Geospatial ML, kriging, weather data	80	40

Results

Here’s the plotted results:

How do we make a decision on this data? As a planning guy, I should be able to do this!

We’ll do a pareto analysis, using the reward function of ||(career_relevance , personal_interest)|| (i.e., magnitude of the vector).

So, let’s plot the pareto frontier for each:

In this case, the outermost circle is the “best”, and a solution “dominates” anything inside its circle. So, according to this, I should focus on Quantum Computing and Geospatial ML. If I had applied scalars to the career relevance and personal interest, those circles would have been ellipses, and the dominance ordering would change. Maybe later.

100 Page Python Intro book announcement

learnbyexample — Tue, 24 Dec 2024 02:00:00 GMT

Hello!

I am pleased to announce a new version of my 100 Page Python Intro ebook. This book is a short, introductory guide for the Python programming language. This book is well suited:

As a reference material for Python beginner workshops
If you have prior experience with another programming language
If you want a complement resource after reading a Python basics book, watching a video course, etc

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of the ebook for FREE till 02-Jan-2025. You can still pay if you wish ;)

Two of my bundles are on sale as well:

All 13 Books Bundle — $15 (normal price $32), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
- Leanpub or Gumroad
Awesome Regex Bundle — $10 (normal price $20), Python, Ruby, JavaScript, BRE/ERE, PCRE and Vim regular expressions
- Leanpub or Gumroad

What's new?🔗

Python version updated to 3.13.0
Added more exercises and you can now practice some of them using this interactive TUI app
Descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
New cover image
Images centered for EPUB format

Videos🔗

Check out my programming tips covering Python, command line tools and Vim:

Testimonials🔗

It's very thorough, written with care, and presented in a way that makes sense. Even as an intermediate Python programmer, I found use in this book.

— feedback by Andrew Healey on an early draft of "100 Page Python Intro" mentioned in this Hacker News thread

Interactive TUI app🔗

I also wrote an interactive TUI app based on some of the exercises from the ebook. Reference solutions are also provided.

Table of Contents🔗

Preface
Introduction
Numeric data types
Strings and user input
Defining functions
Control structures
Importing and creating modules
Installing modules and Virtual environments
Exception handling
Debugging
Testing
Tuple and Sequence operations
List
Mutability
Dict
Set
Text processing
Comprehensions and Generator expressions
Dealing with files
Executing external commands
Command line arguments

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/100_page_python_intro/introduction.html.

GitHub repo🔗

Visit https://github.com/learnbyexample/100_page_python_intro for programs, example files, markdown source and other details about the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Feedback🔗

I would highly appreciate it if you'd let me know how you felt about this book. It could be anything from a simple thank you, Gumroad rating, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors.

You can reach me via:

Issue Manager: https://github.com/learnbyexample/100_page_python_intro/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

The True Cost of PMI: Why you should pay down your low-interest mortgage

Show Notes — Mon, 23 Dec 2024 23:24:30 GMT

I got a guaranteed return of 10.66% by paying down my low-interest mortgage to eliminate PMI.

Lessons from Cybersyn

Magis — Mon, 23 Dec 2024 16:03:33 GMT

What I learned from running a DaaS startup for two years.

Hello emacs.tv

xenodium.com @alvaro — Mon, 23 Dec 2024 02:00:00 GMT

A few days ago, Sacha Chua mentioned how cool it would be to have an Emacs video index like Ruby Video. I mentioned how I had similarly considered a low-tech solution, maybe powered by plain text (bonus points for org mode of course).

A little later, Sacha shared a preliminary video feed dump, in org! With that, I wrote the first experiment to render the org feed and emacs.tv was born.

emacs.tv is merely a few days old. Powered by an org feed (rendered client-side), but we can fetch and render in all sorts of ways. emacs.tv brings it to the web, though I'm sure we can come up with all sorts of Emacs integrations. A new major mode? Or maybe convert the org feed to rss and plug into elfeed?

This is what a feed entry looks like:

* EmacsConf.org: How we use Org Mode and TRAMP to organize and run a multi-track conference :emacsconf:emacsconf2023:org:tramp:
:PROPERTIES:
:DATE: 2023-12-03
:URL: https://emacsconf.org/2023/talks/emacsconf
:MEDIA_URL: https://media.emacsconf.org/2023/emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--main.webm
:YOUTUBE_URL: https://www.youtube.com/watch?v=uTregv3rNl0
:TOOBNIX_URL: https://toobnix.org/w/eX2dXG3xMtUHuuBz4fssGT
:TRANSCRIPT_URL: https://media.emacsconf.org/2023/emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--main.vtt
:SPEAKERS: Sacha Chua
:SERIES: EmacsConf 2023
:END:

We need your help

As mentioned, this is a new project. It's a good start, but it can only get better with your help.

Submit more content

Sacha kickstarted a wonderful video feed, a collection of 1715 videos as of today. We need more. Are your published videos missing? Reckon other videos should be listed? Please help by submitting new entries.

Improve our tagging

Many of the listed videos could use more tags. Please help us by tagging content in video.org and submit a pull request.

Take it for a spin

Or maybe just take emacs.tv for a spin and give us some feedback.

Happy holidays! 🎄☃️

Robotics Predictions for 2025

Ben Bolte — Mon, 23 Dec 2024 02:00:00 GMT

Some predictions for how the robotics landscape looks next year, with varying degrees of confidence.

Uncapped notes dont work for first rounds.

Blog posts of Andreas Klinger — Mon, 23 Dec 2024 02:00:00 GMT

Uncapped safe notes misalign incentives in first rounds… let me explain why

From HTML template strings to elements

Posts on rand[om] — Mon, 23 Dec 2024 02:00:00 GMT

Some time ago, I found (and used for a while) the htl library. Before that, I was using raw template literals to build HTML, and then setting some element.innerHTML = newString. htl uses template literals, but instead returns a regular DOM Node object. This makes it a lot easier to add even listeners, setting other properties, and overall manipulating the object before appending it to the current document.

I was curious to see how it was done, and it turns out it’s easier than I thought. The full library is a single index.js file. Most of the code is parsing the HTML string to support different features from their “templating-format” (or that’s what I understood), which allows doing things like:

Learn new skills

TODO — Mon, 23 Dec 2024 02:00:00 GMT

Great learning materials are rare. This is a collection of gems I’ve encountered over the years, presented as a 90’s style web link list…

Feel, don't think

ntietz.com blog - technically a blog — Mon, 23 Dec 2024 02:00:00 GMT

I started playing clarinet in fourth grade, and dabbled in other instruments.. I never stopped entirely, but I took years-long breaks from playing music.

Earlier this year, I picked up a new instrument while I was very sick. I got a wind synth, the Roland AE-20. It was a somewhat impulsive purchase, in the moment, but I'd been wanting one for a while—the question was just when.

Getting back into music at this particular moment felt... irresponsible. I was having trouble working, or even walking around our block. Yet there was this magnetic pull toward this instrument, toward making music again. Music pulled me back in my hardest hour.

Almost immediately, it helped me move through my illness. When my heart rate shot up, playing scales and simple melodies could rein it in. When I couldn't calm down, playing long tones and scales and silly sounds would make me feel better. And the entire recovery process was helped by having a small low-stakes goal, like learning a simple Bach piece.

As I progressed through my recovery, I had a few moments where things clicked into place for me. One day I suddenly connected the fingerings to those of the clarinet, and was able to play much more fluently. Another day, I'd figure out the phrasing of one of the pieces I was working on.

But I struggled with a lot of the things that I struggled with before: rhythms, expression, you know... music. I want to express myself better through playing and, eventually, composing music. If I wanted to get there, I'd need a teacher.

* * *

I found a teacher through a piano teacher in town. I asked if she knew any saxophone teachers, since my instrument uses sax fingerings, especially someone who also plays jazz. She connected me with Charles, a local musician who's served as principal clarinet in some esteemed ensembles. So I reached out, asking him if he'd teach me on this unusual instrument.

Charles teaches clarinet and saxophone, not wind synth. I'm the first student he's had who uses an amplifier. But the instrument uses sax fingerings, and it responds to breath the way an acoustic woodwind does, so he figured he can probably help me. We decided to try it.

Right away, his method started shaping how I approach music, for the better. He's not focused too much on the specific instrument, but on the rhythm and expression you're putting through it. Obviously the instrument itself matters, but not as much as I'd once thought. I have improved my technique, but not at the cost of expressivity, and I'm losing the mechanical approach I used to take.

He had me buy a few books. I got the Rubank Elementary Method for saxophone, which we're working through to shore up my fundamentals and build on expression. It's going much faster than when I worked through the same volumes for clarinet in school. I picked up a staff paper notebook so he can write out exercises for me. And he had me pick up Robert Starer's Rhythmic Training. That book has been the cornerstone of our work together and has challenged me, showed me I can learn hard things that feel impossible, and fundamentally altered how I look at my art and writing.

* * *

At the same time I started working with Charles, I was in a writing class from Corporeal Writing. It was tremendously challenging for me, and I took it based on a friend's recommendation. The basis of the class is to write from your body, not your brain. I'm positive I don't understand a lot of things from the class still, but it showed me a different way of writing.

In school, my approach to writing was fact-based. For non-fiction, I'd lay out all the facts in my head, or the chronological storyline. Then I'd write that in some narrative, mapping out the logical ordering. It ended up... technical. And fiction took a similar approach, just that the facts and story were things I came up with. But anything nonlinear, or imagery that's not literal, evaded me.

The first week of my class put cracks in that for me. By the end, it had completely opened up the possibilities in my writing.

Our first prompt was challenging for me to understand. But I embraced that difficulty and sat with it, listening to my body, writing what came to me and what I felt from the prompt instead of what I thought it meant. That process was difficult and, at times, painful. The first assigment brought me to places from my past that I'd pushed down. I cried from the act of writing.

That never happened before.

By the end of the class, I was able to produce imagery that made my classmates feel things, instead of think them. I could write from my feelings. It didn't get less painful, so it's not something I can do all the time—but that it's possible is not something I knew at all.

* * *

At our first or second lesson, we started working through the rhythm book. The first section we focused on features syncopation. My stomach dropped when we got there, since this is something I've always had trouble with.

But in the week after that lesson, I got to where I could feel a syncopated beat for the first time. I've played them before, but it never felt natural and it was often from following someone else's cues. The practice I did let me get to where I can produce syncopation reliably.

The feeling was intoxicating. The moment that syncopation embedded itself into my brain, I filled with warmth and confidence. I relaxed. I couldn't stop doing those exercises, to feel that again.

And so the next challenge arrived: triplets. This was, yet again, one that I have long struggled with. It's a rhythm that's very common to have trouble with. But this time, unlike syncopation? This time, I knew I could do it. I just had to put in the time.

Triplets took me longer to get than syncopation. The glimmers were there, and in our first session working on triplets he got me to produce a 3:2 polyrhythm once! Oh, that feeling. I chased that feeling for the following week. But I wasn't getting it, I was really struggling. No matter what I did, I couldn't figure it out.

I'd think about where the beat is, what the subdivisions are, and I'd still keep missing it. By the time I had thought the subdivision, it was gone, and I was late. Or I'd anticipate it coming, and I was early. I can analyze the rhythm, but playing it was beyond me.

The next week, Charles talked to me about feeling the beat in my head. Some people do this with counting in their head: one two three four. They subdivide eights with one and two and three and four and, and triplets with something like one and a two and a or one trip let two trip let. Other people feel a pulse for the beat.

I've tried to count in my head, and I can do it when I'm listening to music. But I can't do it when I'm playing; the act of counting as words seems to force out everything else. A pulse, though... that I feel. The pulse I feel lives a few inches above and behind my head. I can't describe what it feels like except that I know it's there, and it's somehow located in space, but it's not a sound. The closest I can come is that it's like something is squeezing on each beat and subdivision, except it's squeezing something that's not inside me.

This doesn't happen to me every time, and if I think about the pulse that's there, it can go away. If I try to peek at it, try to force everything onto it, then it's gone. I was trying to do an exercise and Charles stopped me, and then started telling me a story that was related. But while he did that, the metronome was still going. The pulse in my head continued, got stronger.

I interrupted him to do the exercise. I nailed it.

"What changed?" he asked, with a smile on his face.

I told him that I stopped thinking about the exercise. Instead, I let myself just feel the pulse, and feel where I needed to be. And it worked.

Since that lesson, I've kept working on this and have gotten better at summoning the pulse, at feeling instead of thinking. Don't get me wrong: I need to think about the rhythms to understand them before I can play them. And I certainly need to think about the mistakes I've made so I can understand them to fix them the next time I do that. But the core is to feel when I do things, instead of analyzing.

* * *

This turns out to be much broader than music and creative writing for me. It is an approach that is more fundamental than that.

This blog post itself is an example of that. When I started writing it, I knew mostly that it would talk about triplets and the pulse and feeling and thinking, but I didn't know what the story was. I didn't have an outline. But I let myself feel where it starts, where it takes twists and turns, and where it ends up.

This comes up in how I understand myself. Instead of purely analyzing things, I can sit with the feelings and... feel them. It's always hard, and sometimes it's scary. And it never feels like something is happening, but yet, later, I have a deeper understanding of things. I don't see the feelings, but feeling them brings some clarity. Sometimes.

And it even comes up in software design decisions and in writing code. You have to vibe with it sometimes and go with feelings. We have standard words for this, like "code smell," which are very much related to feelings but couched in a less feelings-sounding veneer.

I layer in both approaches together. I need to think about problems in order to load them into my head and solve them. But I also need to feel problems in order to solve them well.

Thinking can be great for figuring out what we want to do. But actually doing it? That's when it's important to feel, not think.

Thank you to Erika and Robbie for feedback on a draft of this post.

ABC: Learning The Alphabet with Java Annotations

SerCe's blog — Mon, 23 Dec 2024 02:00:00 GMT

This Christmas, let's explore the alphabet of Java annotations. There are so many, but how many of them truly benefit us?

Interesting Books I’ve Read in 2024

Brett Mullins — Mon, 23 Dec 2024 02:00:00 GMT

Below are some interesting books I’ve read in 2024. The bulk of what I’ve been reading is either wrapped up in my research on differential privacy or for my project looking back at research from 100, 150, and 200 years ago. Two of the four recommendations below are from the latter project, but all four are of interest from an historical perspective. Wisdom’s Workshop is a history of the American research university from Medieval times to the present. William Stanley Jevons’ The Principles of Science is a wide-ranging treatment of logic and philosophy of science from 1874 that’s bursting with ideas - some more developed than others. Ballyhoo! is a history of professional wrestling and combat sports from its outlaw roots in the late nineteenth century through the first half of the twentieth century. Finally, John Ramsay McCulloch’s Discourse on Political Economy from 1824 is the first history of economic thought from the era of the classical economists.

Unraveling the Internals of Video Streaming services

Engineering At Scale — Sun, 22 Dec 2024 14:35:54 GMT

A comprehensive guide to key concepts, terminologies and working of Video Streaming apps

Some TILs

🦉 olshansky 🦁 — Sun, 22 Dec 2024 14:21:16 GMT

Ulysses pact: a freely made decision to bind yourself in the future. Used for living wills in modern day but originated from Greek mythology.

On Writing: A Retrospective

xorvoid — Sun, 22 Dec 2024 13:51:33 GMT

On Writing: A Retrospective

Summary: Clear Thinking by Shane Parrish

Chester Grant — Sun, 22 Dec 2024 13:26:15 GMT

1. While the rest of us are chasing victory, the best in the world know they must avoid losing before they can win. It turns out this is a surprisingly effective strategy.

2. If there is a tagline to my life, it is “Mastering the best of what other people have already figured out,” and this book is a tribute to that belief.

3. Time is the friend of someone who is properly positioned and the enemy of someone poorly positioned. When you are well positioned, there are many paths to victory. If you are poorly positioned, there may be only one. You can think of this a bit like playing Tetris. When you play well, you have many options for where to put the next piece. When you play poorly, you need just the right piece.

4. It doesn’t matter what position you find yourself in right now. What matters is whether you improve your position today. Every ordinary moment is an opportunity to make the future easier or harder. It all depends on whether you’re thinking clearly.

5. Never forget that your unconscious is smarter than you, faster than you, and more powerful than you. It may even control you. You will never know all of its secrets.

6. In the space between stimulus and response, one of two things can happen. You can consciously pause and apply reason to the situation. Or you can cede control and execute a default behavior.

7. So our first step in improving our outcomes is to train ourselves to identify the moments when judgment is called for in the first place, and pause to create space to think clearly.

8. There’s nothing stronger than biological instincts. They control us often without us even knowing. Failing to come to terms with them only makes you more susceptible to their influence.

9. Here’s how each essentially functions: The emotion default: we tend to respond to feelings rather than reasons and facts. The ego default: we tend to react to anything that threatens our sense of self- worth or our position in a group hierarchy. The social default: we tend to conform to the norms of our larger social group. The inertia default: we’re habit forming and comfort seeking. We tend to resist change, and to prefer ideas, processes, and environments that are familiar.

10. The social rewards for going with the crowd are felt long before the benefits of going against it are gained. One measure of a person is the degree to which they’ll do the right thing when it goes against the popular belief. However, it is easy to overestimate our willingness to diverge from the crowd, and underestimate our biological instinct to fit in.

11. Princeton professor Robert George wrote, “I sometimes ask students what their position on slavery would have been had they been white and living in the South before abolition. Guess what? They all would have been abolitionists! They all would have bravely spoken out against slavery, and worked tirelessly against it.” 1 No, they wouldn’t have. They may understandably want to send that signal now when it’s safe to do so, but back then they would have likely behaved the same most everyone else did at the time.

12. The only way to outperform if you’re doing undifferentiated work is to work harder than everyone else.

13. Imagine a team of ditchdiggers working with their hands. A slight variation in the amount of soil moved per hour is barely perceptible. Your work is indistinguishable from that of the person next to you. The only way to move more dirt is to dig for longer.

14. Within this paradigm, the ditchdigger who takes a week off to experiment and invent the shovel seems crazy. Not only do they look like a fool for taking a risk, but their cumulative production falls behind for every day they are not digging. Only when the shovel comes along do others see its advantage. Success requires shamelessness. So too does failure.

15. Doing something different means you might underperform, but it also means you might change the game entirely. If you do what everyone else does, you’ll get the same results that everyone else gets. Best practices aren’t always the best. By definition, they’re average.

16. Lou Brock might have put it best when he said, “Show me a guy who’s afraid to look bad, and I’ll show you a guy you can beat every time.” In other words, someone who’s possessed by the social default is easy to defeat.

17. The inertia default pushes us to maintain the status quo. Starting something is hard but so too is stopping something. We resist change even when change is for the best.

18. “Once our minds are set in a direction, they tend to continue in that direction unless acted upon by some outside force.”

19. The good news is that the same biological tendencies that make us react without reasoning can be reprogrammed into forces for good.

20. The more time you spend with people, the more likely you start to think and act as they do.

21. The way to improve your defaults isn’t by willpower but by creating an intentional environment where your desired behavior becomes the default behavior.

22. Joining groups whose default behaviors are your desired behavior is an effective way to create an intentional environment.

23. Our defaults work off deeply ingrained biological tendencies— our tendencies for self- preservation, for recognizing and maintaining social hierarchies, and for defending ourselves and our territory. We can’t simply know these tendencies exist and then will them out of existence. On the contrary, the feeling that willpower is all it takes to remove these forces is one of the tricks they use to keep us under their control.

24. To stop our defaults from impeding good judgment, we need to harness equally powerful biological forces. We need to take the same forces that the defaults would use to ruin us and turn them to our advantage. Chief among them is the force of inertia.

25. Establishing rituals is the key to creating positive inertia. Rituals focus the mind on something other than the moment. They can be as simple as taking a quick pause before responding to someone’s point of contention at work. One of my old mentors used to tell me, “When someone slights you in a meeting, take a deep breath before you speak and watch how often you change what you’re about to say.”

26. One effective question to ask yourself before you act is, “Will this action make the future easier or harder?”

27. My grandfather (and many others) used to say, “If you find yourself in a hole, the first thing you need to do is stop digging.”

29. No successful person wants to work with a chronic victim. The only people who want to work with victims are other victims.

30. Know thyself.—INSCRIPTION ON THE TEMPLE OF APOLLO AT DELPHI

31. Emotional intensity is far less important in the long run than disciplined consistency. Inspiration and excitement might get you going, but persistence and routine are what keep you going until you reach your goals.

32. Confidence also Comes from How You Talk to Yourself More dreams die from a lack of confidence than a lack of competence. But while confidence is often a byproduct of our accomplishments, it also comes from how you talk to yourself.

33. Self- confidence is also the strength to accept hard truths. We all have to deal with the world as it is, not as we want it to be. The quicker you stop denying inconvenient truths and start responding to difficult realities, the better.

34. SELF- ACCOUNTABILITY, SELF- KNOWLEDGE, SELF- CONTROL, and self- confidence are essential to exercising good judgment. Here are a couple examples of how they work together.

35. Suppose you know from experience that you’re susceptible to social pressure. To protect yourself from the influence of the social default, you decide to implement a safeguard. You form a rule for yourself: never say yes to something important without thinking it over for a day.

36. Practicing this safeguard isn’t very enjoyable. Putting someone on hold for a day might be uncomfortable.

37. As simple as they seem, automatic rules for common situations get results. We’ll explore automatic rules in the next chapter.

38. It is inevitable if you enter into relations with people on a regular basis… that you will grow to be like them…. Place an extinguished piece of coal next to a live one, and either it will cause the other one to die out, or the live one will make the other reignite…. Remember that if you consort with someone covered in dirt you can hardly avoid getting a little grimy yourself.—EPICTETUS, Discourses

39. Few things are more important in life than avoiding the wrong people. It’s tempting to think that we are strong enough to avoid adopting the worst of others, but that’s not how it typically works.

40. We unconsciously become what we’re near. If you work for a jerk, sooner or later you’ll become one yourself. If your colleagues are selfish, sooner or later you become selfish. If you hang around someone who’s unkind, you’ll slowly become unkind. Little by little, you adopt the thoughts and feelings, the attitudes and standards of the people around you.

41. Few people realize that exceptional outcomes are almost always achieved by people with higher- than- average standards.

42. Champions don’t create the standards of excellence. The standards of excellence create champions.

43. The best teachers expect more from their students and from themselves. And more often than not, the students rise to meet those expectations. The best leaders expect more from people; they hold them to the same standards they hold themselves— a higher standard than most would otherwise know is possible.

44. When we accept substandard work from others, it’s for the same reason: we’re not all in. When you’re committed to excellence, you don’t let anyone on your team half- ass it. You set the bar, you set it high, and you expect anyone working with you to work just as hard and level up to what you expect or above. Anything less is unacceptable.

45. Masters of their craft don’t merely want to check off a box and move on. They’re dedicated to what they do, and they keep at it. Master- level work requires near fanatical standards, so masters show us what our standards should be.

46. Working with a master firsthand is the best education; it’s the surest way of raising the bar. Their excellence demands your excellence. But most of us aren’t lucky enough to have that opportunity.

47. THERE ARE TWO COMPONENTS TO BUILDING STRENGTH by raising the bar: (a) Choose the right exemplars— ones that raise your standards. Exemplars can be people you work with, people you admire, or even people who lived long ago. It doesn’t matter. What matters is they make you better in a certain area, like a skill, trait, or value.

48. In the previous section, we discussed something most people never think about: if you don’t curate the people in your life, the people who end up surrounding you will be there by chance and not by choice. That group includes your parents, your friends, your family, your coworkers. Sure, your high school friends might be great examples of character and acumen, but odds are they’re average. Sure, your parents might be some of the smartest businesspeople in the world, but odds are they’re not. It’s not that you should remove these people from your life, though; controlling your environment just means intentionally adding exemplars into the mix.

49. As Peter Kaufman once told me, “No technique has been more responsible for my success in life than studying and adopting the good models of others.”

50. One of the biggest mistakes that I see people make is they don’t want to learn from someone who has a character blemish or a worldview that doesn’t align with theirs.

51. “I shall never be ashamed of citing a bad author if the line is good.” Or, as Cato the Elder put it, “Be careful not to rashly refuse to learn from others.” Don’t throw away the apple because of a bruise on the skin.

52. The only person you’re competing with is the person you were yesterday. Victory is being a little better today. **

53. Bad habits are easy to acquire when there is a delay between action and consequence. If you eat a chocolate bar or skip a workout today, you’re not going to suddenly go from healthy to unhealthy. Work late and miss dinner with your family a couple nights, and it won’t damage your relationship. If you spend today on social media instead of doing work, you’re not going to get fired. However, these choices can end up becoming habits through repetition and accumulate into disaster.

54. The formula for failure is a few small errors consistently repeated.

55. There are two ways to manage your weaknesses. The first is to build your strengths, which will help you overcome the weaknesses you’ve acquired. The second is to implement safeguards, which will help you manage any weaknesses you’re having trouble overcoming with strength alone.

56. THERE ARE MANY INBUILT BIOLOGICAL VULNERABILITIES that can impede good judgment: sleep deprivation, hunger, fatigue, emotion, distraction, stress from feeling rushed, and being in an unfamiliar environment are just some examples. We can’t avoid finding ourselves in these conditions from time to time. But we can implement safeguards to protect us from our defaults when we are.

57. Safeguards are tools for protecting ourselves from ourselves— from weaknesses that we don’t have the strength to overcome.

58. Purging your home of all junk food is an example of one safeguarding strategy: increasing the amount of “friction” required to do something that’s contrary to your long- term goals.

59. My favorites include prevention, creating rules for yourself, making checklists, shifting your frame of reference, and making the invisible visible. Let’s talk about each strategy.

60. The first kind of safeguard aims at preventing problems before they happen. One way to do this is to avoid decision- making in unfavorable conditions. Stress, for instance, is a big contributor to bad decisions. Some studies have shown that stress short- circuits the deliberation process— it undermines the systematic evaluation of alternatives that’s needed for effective decision- making.

61. Alcoholics Anonymous has a helpful safeguard for its members. They call it HALT— an acronym that stands for Hungry, Angry, Lonely, and Tired. When you feel like having a drink, they say, ask yourself whether any of these conditions apply. If so, deal with the real problem— hunger, anger, loneliness, or fatigue— instead of reaching for a drink.

62. Daniel Kahneman, the godfather of cognitive biases and thinking errors, he revealed an unexpected way we can improve our judgment: replacing decisions with rules.

63. When it comes to your health, just like many other elements of your life, environment determines behavior. Your environment makes one path easier than another.

64. It’s easier to make healthy food choices if the only foods available to you are good for you. It’s also easier to stick to a consistent pattern of choices if you’re in your familiar operating environment. When you’re in an unfamiliar environment, it’s harder to maintain your familiar patterns of behavior, which is why a lot of people stop exercising or eating healthy when they travel.

65. Your environment isn’t just your physical surroundings. It also includes people. Sometimes it’s hard saying no to someone.

66. We’re wired in a way that makes us want to be liked by others, and we’re afraid that saying no to someone will make them like us less. Saying no to someone repeatedly can be even more difficult. We might say no when our friend offers us a sugary beverage after a workout one day, but if he does it three days in a row, we cave. That’s only human.

67. In a quirk of psychology, people typically don’t argue with your personal rules. They just accept them as features of who you are. People question decisions, but they respect rules.

68. Kahneman told me his favorite rule was never to say yes to a request on the phone. He knows that he wants people to like him, so he wants to say yes in the moment, but after filling up his schedule with things that didn’t make him happy, he decided to be more vigilant about what he agrees to do and why. When people ask him for things over the phone now, he says something along the lines of, “I’ll have to get back to you after I think about it.” Not only does this give him time to think without the immediate social pressure, but it also allows a lot of these requests to just drop away because people choose not to follow up. He rarely gets back to any of these people and says yes.

69. Another safeguarding strategy is to increase the amount of effort it takes to do things that are contrary to your goals. I used to find myself checking my email whenever I have a second. I’d check it before I got out of bed, on the walk home from work, in line at the grocery store.**

70. If there were a recipe for accumulated disaster, it would be giving the best of ourselves to the least important things and the worst of ourselves to the most important things.

71. Checklists, for instance, offer a simple way to override your defaults. Pilots go through a preflight checklist every time they fly. Surgeons go through preoperative checklists every time they operate. You might have a packing checklist every time you travel. In each of these cases, the checklist acts as a safeguard, forcing us to slow down whatever we’re doing and go back to basics.

72. ONCE YOU HAVE reprogrammed your defaults to create space for clear thinking, you must master the skill of decision- making.

73. Decisions are different from choices. If you casually select an option from a range of alternatives, you’ve made a choice. If you react without thinking, you’ve made an unconscious choice. But neither of these is the same as a decision. A decision is a choice that involves conscious thought.

74. That process is about weighing your options with the aim of selecting the best one, and it’s composed of four stages: defining the problem, exploring possible solutions, evaluating the options, and finally making the judgment and executing the best option. We will discuss each of these components in detail throughout this chapter.

75. When the stakes are low, inaction hurts you more than speed. Sometimes it’s better just to make a quick choice and not spend time deliberating

76. Why waste time when action is inconsequential and its effects are easily reversed? For example, if there are two identical squat racks in the gym and both are momentarily open, it makes no difference which one you take. If you wait and decide, they’ll both be taken by someone else. Just choose either one.

77. When the stakes are higher, though, speed can hurt you. If an action could have a major impact on your life or your business and its effects can’t be reversed, you must decide and not merely choose.

78. In these cases, the magnitude of the potential losses makes careful decision- making a worthwhile investment of your time. In these cases, evaluate the options and decide. Don’t just choose.

79. The next few sections describe some tools for reasoning better when making decisions. They won’t solve every decision- making problem, because no tool is right for every job; each has its uses and limitations. You need multiple tools in your toolbox. Otherwise, you end up solving the wrong problems. As the old adage says, “If the only tool you have is a hammer, you tend to see every problem as a nail.”

80. “What would have to be true for this problem not to exist in the first place?” The way you define a problem changes what you see.

81. A mentor of mine once taught me that the best way to avoid finding the perfect solution to the wrong problem at work, when time allows, is to hold two separate meetings: one to define the problem, and one to come up with the solution.

82. One way to keep meetings short and avoid the signaling that comes from repeating information that everyone knows is simply asking everyone, “What do you know about this problem that other people in the room don’t know?”

83. Remember that writing out the problem makes the invisible visible. Write down what you think the problem is, and then look at it the next day. If you find yourself using jargon in your description, it’s a sign that you don’t fully understand the problem. And if you don’t understand it, you shouldn’t be making a decision about it.

84. SAFEGUARD: Use the test of time. Test whether you’re addressing the root cause of a problem, rather than merely treating a symptom, by asking yourself whether it will stand the test of time. Will this solution fix the problem permanently, or will the problem return in the future? If it seems like the latter, then chances are you’re only treating a symptom.

85. You can put your energy into short- term solutions or long- term solutions but not both. Any energy that’s channeled toward short- term solutions depletes energy that could be put into finding a long- term fix.

86. ONCE YOU’RE CLEAR ON THE PROBLEM, IT’S TIME TO THINK of possible solutions— ways of overcoming the obstacles to get what you want. The way to come up with possible solutions is by imagining different possible futures— different ways the world could turn out.

87. Luckily, there’s a way to convert the hindsight of tomorrow into the foresight of today. It’s a thought experiment that psychologists call premortem. The concept isn’t new, it originates in Stoic philosophy. Seneca used premeditatio malorum (“ the premeditation of evils”) to prepare for the inevitable ups and downs of life. The point isn’t to worry about problems; it’s to fortify and prepare for them.

88. Imagining what could go wrong doesn’t make you pessimistic.

89. “Failure comes from a failure to imagine failure.”

90. THE 3 + PRINCIPLE: Force yourself to explore at least three possible solutions to a problem. If you find yourself considering only two options, force yourself to find at least one more.

91. SAFEGUARD: Imagine that one of the options is off the table. Take each of the options you’re considering, and one at a time, ask yourself, “What would I do if that were not possible?”

92. SAFEGUARD: Come up with Both- And options. Try to find ways of combining the binary. Think not in terms of choosing either X or Y, but rather having both X and Y.

93. This is an important aspect of leadership and problem-solving in general: you have to pick one criterion above all.

94. Most information is irrelevant. irrelevant. Knowing what to ignore—separating the signal from the noise—is the key to not wasting valuable time. Think, for example, of investment decisions. The best investors know which variables probabilistically govern the outcomes, and they pay attention to those. They don’t ignore everything else, but focusing primarily on those variables allows them to filter massive amounts of information very quickly.

100. The quality of your decisions is directly related to the quality of your thoughts. The quality of your thoughts is directly related to the quality of your information. Many people treat all sources of information as if they’re equally valid. They’re not. While you might value getting everyone’s opinion, that doesn’t mean each opinion should be equally weighted or considered.**

101. SAFEGUARD: Run an experiment. Try something out to see what kinds of results it yields.

102. An experiment is a low-risk way of gathering important information. For example, if you want to know whether people will pay for something, try to sell it before you even create it.

103. That’s what my friends at Tuft & Needle did. They were one of the first companies to ship foam mattresses directly to consumers’ homes. They shared an incredible story with me over coffee one day, about their early days. In order to validate their idea, they set up a landing page, bought some Facebook ads, and started taking orders.

104. They didn’t even have a product or a company yet; they just wanted to see if people would buy foam mattresses from them. After a few days of receiving orders, they had all the proof they needed that people would buy their product. They refunded all the orders and officially started their company. While this example may be a bit unorthodox, there are many ways in which experimenting can help determine whether there’s sufficient demand for a product or service.

105. Experts can increase the accuracy of your information and decrease the time it takes to get it. Getting even one expert’s advice can cut through a lot of confusion and help you quickly formulate and/or eliminate options.

106. Experts don’t treat all requests for help equally, though. Some requests really don’t feel good to receive. Usually these are requests of the tell-me-what-I-should-do type. Often these people haven’t done the work ahead of time, they just want you to decide for them.

107. Remember: the goal isn’t to have someone tell you what to do; rather, it’s to learn how an expert thinks about the problem, which variables they consider relevant, and how those variables interact over time.**

108. THE STOP, FLOP, KNOW PRINCIPLE: Stop gathering more information and execute your decision when either you Stop gathering useful information, you First Lose an OPportunity (FLOP), or you come to Know something that makes it evident what option you should choose.

109. When failure is expensive, it’s worth investing in large margins of safety. A margin of safety is a buffer between what you expect to happen and what could happen. It’s designed to save you when surprises are expensive.

110. A margin of safety is like having insurance. If you know in advance you won’t need to make a claim this year, it’s a waste of money to buy insurance. The problem is that you don’t know in what year you’ll need to make a claim, so you buy it every year. It might seem like a waste of money in years when nothing happens, but it shows its real value in years when something does.

111. Building a margin of safety means giving yourself as much cushioning and coverage in the future as possible. It’s a way of preparing yourself for the widest range of possible future outcomes—and protecting yourself against the worst ones.

112. TIP: The margin of safety is often sufficient when it can absorb double the worst-case scenario. So the baseline for a margin of safety is one that could withstand twice the amount of problems that would cause a crisis, or maintain twice the amount of resources needed to rebuild after a crisis.

113. For example, if you want to feel financially secure even if you lose your job, you can estimate how long it will take you to gain employment again, and then save enough to live off savings for double that amount of time.

114. However, if you have a lot of expertise and data, you can reduce your margin of safety yet further. Here’s an example: Warren Buffett aims to buy stocks that are 30–50 percent less than their true value. So he has a 30–50 percent margin of safety on stocks. But he’ll pay close to a dollar on the dollar for stocks that he understands well. So there’s only maybe a 20 percent margin of safety on the stocks he’s the most confident in.

115. One of Warren Buffett’s core tenets for buying a business is that if he doesn’t understand it, he doesn’t buy it. In other words, if he doesn’t have enough information to calculate a margin of safety, he doesn’t invest at all.

116. Remembering that I’ll be dead soon is the most important tool I’ve ever encountered to help me make the big choices in life. Because almost everything—all external expectations, all pride, all fear of embarrassment or failure—these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking that you have something to lose.

117. At some point my kids figured out that it was easier to solve a maze backward than forward, especially if the maze is harder or more complicated than usual. Something about starting with the end in mind, they realized, makes it easier to decide which path to take. Life in general works similarly.

Ten Favorite Photos I Captured in 2024

Saeed Esmaili — Sun, 22 Dec 2024 12:02:38 GMT

I purchased a Sony a7ii in 2023, while having zero understanding of the basics of cameras and photography. I recall being particularly confused about the exposure triangle , especially how aperture affects the sharpness and the depth of field of an image. After learning the basics and taking around 3500 photos with that camera, I realized its aging technology was limiting my beginner skills. I had enough experience to be able to research camera models and their features, and I decided to get a Sony a7cii at the beginning of 2024, a decision I’m still happy with today.

Detecting the Hammer Candlestick Pattern Using Python

Adnan's Random bytes — Sun, 22 Dec 2024 10:48:37 GMT

This post is part of the T4p Series. In the previous post, we discussed creating your custom signal generation pattern. In this post, we will discuss a famous candlestick pattern called the Hammer pattern. We will discuss the pattern, what it is all about, and how it helps traders to earn money. Introduction A Hammer is a single candlestick pattern that forms during a downtrend and is characterized by having a small real body at the top of the candle, a long lower shadow (at least 2-3 times the size of the body), and little to no upper shadow. The pattern indicates a potential trend reversal as it shows that despite strong selling pressure (long lower shadow), buyers ultimately regained control by pushing prices back up (small body at the top). As you can see in the image below, it is the 4th one from the left in green. Types of Hammer Candlestick There are three two types of hammer patterns: Regular Hammer (appears after a downtrend). Inverted Hammer (appears after an uptrend). Usage Below are the usages of the hammer pattern: Trend Reversal Signal: Indication about the potential reversal from downtrend to uptrend(bullish) or vice versa. Support and Resistance: Highlights key support levels in downtrends and resistance in uptrends. Confirmation: Best used with other indicators for confirming trend reversals. Spotting The Hammer Pattern The close price is near the high of the candle. The lower shadow is at least twice as long as the body. The upper shadow is minimal or absent. Development Visualization Like before, the first thing we will do is write code for visualizing the pattern itself. Assuming all the required libraries are installed (discussed multiple times in previous posts), I’m going to use ChatGPT to generate fictitious OHLC data tailored to my requirements. def candle_hammer(df): # Fill NaN values with 0 df = df.fillna(0) return ( ((df["High"] - df["Low"]) > 3 * (df["Open"] - df["Close"])) & (((df["Close"] - df["Low"]) / (0.001 + df["High"] - df["Low"])) > 0.6) & (((df["Open"] - df["Low"]) / (0.001 + df["High"] - df["Low"])) > 0.6) ) data = { 'Date': pd.date_range('2024-12-01', periods=10, freq='D'), 'Open': [100, 105, 110, 115, 108, 112, 116, 118, 122, 125], 'High': [105, 110, 115, 118, 113, 116, 119, 121, 124, 128], 'Low': [98, 103, 107, 112, 106, 110, 113, 116, 119, 120], 'Close': [103, 107, 109, 115, 110, 114, 118, 119, 123, 126] } # # Adjust the values of the 3rd candle (index 2) to create a Bullish Hammer pattern # Modify the data to make only the 3rd candle (index 2) a Bullish Hammer data['Open'][2] = 108 # Small body for Bullish Hammer data['Close'][2] = 109 # Small body for Bullish Hammer data['Low'][2] = 100 # Long lower shadow data['High'][2] = 110 # Small upper shadow # Modify the other candles to have larger bodies data['Open'][3] = 115 # Bullish candle with higher close data['Close'][3] = 120 # Higher close than open data['Low'][3] = 110 # Normal low data['High'][3] = 125 # Normal high data['Open'][4] = 120 # Bearish candle with lower close data['Close'][4] = 118 # Lower close than open data['Low'][4] = 115 # Normal low data['High'][4] = 125 # Normal high data['Open'][5] = 125 # Bearish candle with lower close data['Close'][5] = 121 # Lower close than open data['Low'][5] = 120 # Normal low data['High'][5] = 130 # Normal high data['Open'][6] = 130 # Bullish candle with higher close data['Close'][6] = 134 # Higher close than open data['Low'][6] = 125 # Normal low data['High'][6] = 135 # Normal high data['Open'][7] = 135 # Bullish candle with higher close data['Close'][7] = 137 # Higher close than open data['Low'][7] = 130 # Normal low data['High'][7] = 140 # Normal high data['Open'][8] = 140 # Bullish candle with higher close data['Close'][8] = 141 # Higher close than open data['Low'][8] = 135 # Normal low data['High'][8] = 145 # Normal high data['Open'][9] = 145 # Bullish candle with higher close data['Close'][9] = 146 # Higher close than open data['Low'][9] = 140 # Normal low data['High'][9] = 150 # Normal high # Convert to DataFrame df = pd.DataFrame(data) df.set_index('Date', inplace=True) When I run the above code in the notebook, it generates the following chart: I borrowed the candle_hammer function from here, which was actually an elegant implementation. Then, I asked GPT to generate fake OHLC data that includes a hammer candle as well. In the chart above, you can see an up-arrow indicating the hammer candle. As you see, it generated a hammer candle on December 3rd, having an up-arrow on top of it. Now, what if we run candle_hammer function? As you can see, it returned True on December 3rd. Sweet! Now, let’s try these functions on actual data. I am using the Yahoo Finance library to pull daily data for TSLA. tesla_data = yf.download('TSLA', start='2024-01-01', end='2024-12-31', progress=False) tesla_data = tesla_data.tail(30) # Last 30 Entries only visualize_hammer(tesla_data,"TSLA Daily Data") candle_hammer(tesla_data) And…when I run the above functions, it produces the following outputs: Sweet, isn’t it? Now, if you want, you can use this data to generate signals or alerts to inform you about trend reversals Conclusion In this post, you learned how to identify trend reversals with the help of hammer patterns and how to automate this process in Python. In the next post, I will discuss another pattern, but this time with the help of a library that has already done all the work for you. As always, the code is available on GitHub Looking to create something similar or even more exciting? Schedule a meeting or email me at kadnan @ gmail.com. Love What You’re Learning Here? If my posts have sparked ideas or saved you time, consider supporting my journey of learning and sharing. Even a small contribution helps me keep this blog alive and thriving. 👉 Support My Work Here If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Detecting the Hammer Candlestick Pattern Using Python first appeared on Adnan's Random bytes.

Expedia and SIXT: My Surprise Cancellation Fee

Sun, 22 Dec 2024 07:00:00 GMT

I recently tried to cancel a rental car reservation on Expedia—nearly two weeks before my trip—only to discover they’d keep $100 of the $207 I’d already paid. Turns out, SIXT has a policy allowing them to pocket $100 from any cancellation, and I find it incredibly frustrating—especially given that I’ve never encountered a non-full-refund policy for rental cars so far in advance. It feels like a complete trick.

Materials Cheat Sheet

taylor.town — Sun, 22 Dec 2024 02:00:00 GMT

Metals, polymers, ceramics, glasses, composites, and natural stuff.

How to Generate and Format the Current Date and Time in Hugo with dateFormat

Nelson Figueroa — Sun, 22 Dec 2024 02:00:00 GMT

Several examples of dateFormat including date only, date + 12-hour time, date + 24-hour time, date + timezone, and date + UTC offset.

Full RSS feed

Modern Web Development with Chrome — Sun, 22 Dec 2024 01:07:00 GMT

I love RSS feeds (and ATOM too (and JSONFeed)) but one thing that frustrates me is when a feed doesn't include all of the content. https://full-rss.deno.dev was created to partially solve this. If there is a feed that you really wish was full-content and isn't then you can replace that feed with https://full-rss.deno.dev/?url=https://developer.chrome.com/static/blog/feed.xml and it will get the 10 most recent posts, scrape the linked page and replace the content with a stripped down version of the full HTML page.

Linux zhongwen shurufa

Stories by Ming on Medium — Sun, 22 Dec 2024 00:50:55 GMT

This article appeared on your search results page because:

you have just installed a desktop Linux,
you are struggling to figure out how to type Chinese, and
you use Pinyin, not ZhuYin/Bopomofo, Cangjie, or Wubi for romanized Chinese.

What is this? Unlike in Windows or macOS, where Input Methods come in a complete package, Input Methods in Linux is composed of at least two parts: frameworks and engines.

Frameworks coordinate with the operating system for tasks like rendering “candidate words” panels and “hijacking” keyboard events from a textbox.
Engines deal with mapping keystrokes to Chinese characters.

Two frameworks are most popular today: IBus and Fcitx (小企鹅输入法, lit. “Little Penguin Input Method”). Ubuntu comes with IBus, and their stock engine of choice is ibus-libpinyin. If you have tried typing Chinese in Ubuntu and liked it, then that’s the way to go. Otherwise, I commend your braveness diving straightaway into Arch.

I myself didn’t enjoy IBus, mainly due to how eager it was replacing Latin keystrokes with candidate words. What do I mean by that? Suppose I typed “nihao” on a keyboard. Candidate words would include “你好” and “泥嚎”. In Windows, macOS, and Linux with Fcitx, the textbox would be temporarily populated with an underlined “ni hao”, and the candidate words will show in a floating pane beneath it:

Candidate words in macOS for typing “nihao”

Meanwhile, IBus would pre-fill the textbox with the first candidate, rendering the Latin keystrokes inside of the floating pane:

Screenshot taken from a GIF in this doc.

Somehow this annoys me, and I’ve yet been able to find a config for this behavior. Users beware.

My pet peeve aside, Fcitx — particularly, fcitx5 — seems more popular (by GitHub stars). Therefore, this post will be about how to install a fcitx-based Chinese Input Method.

The Chinese writing system has many romanization solutions, most popular ones include Pinyin and ZhuYin. I’m born and raised in mainland China, so Pinyin is my system of choice. (ZhuYin, on the other hand, esto me suena a chino.) Therefore, it should be no surprise to you that, for the Chinese language specifically, there is yet another layer of framework (or “meta-engine”, if you like) sitting between Fcitx/IBus and the actual engines. We are talking about Rime (中州韻輸入法引擎) here.

As an input method engine, Rime itself doesn’t care about how the “candidate words” panel should be rendered, so it exists in the Linus world mainly as wrappers for IBus and for Fcitx(5). To install the latter, run (assuming ArchLinux):

yay -Syu fcitx5-rime

Rime runs “input schemas”. An input schema defines how a particular sequence Latin keystrokes should be mapped to Chinese candidate words. Many input schemas exist (see this page on ArchLinux wiki). I recommend 雾凇拼音 (rime-ice).

Image from the official GitHub repo.

To install this input schema, download and run this auto-deployment script:

git clone --depth=1 https://github.com/Mark24Code/rime-auto-deploy.git --branch latest
cd rime-auto-deploy
./installer.rb

Now you should be good to go. Enjoy typing in Linux. Here’s to endless inspiration and creativity!

How to get a Driver's License in California

ThatXliner's Blog — Sat, 21 Dec 2024 22:45:21 GMT

A guide for my friends who are starting to get their driver's license

The books I enjoyed the most in 2024

Gonçalo Valério — Sat, 21 Dec 2024 21:06:52 GMT

Another year went by, and another batch of books was consumed. Just like I did last year, I want to share the ones that I enjoyed the most. But what kind of metric is that? Truth be told, it is not an objective one. Last year, I clearly described it like this: I don’t mean […]

180 Days…

The Digital Nomad — Sat, 21 Dec 2024 20:19:59 GMT

When we first bought our condotel, we were told that we could only stay in our unit up to 180 days.

I looked at our paperwork, and everything we signed said “unlimited use as owners”.

We tried to book our stay for a year and we were told that we can’t stay more than 180 consecutive days.

Now, we have to pack everything we own in suitcases every 180 days, vacate our unit, store our belongings either in the car or with the bellman service and leave for a few days.

We usually schedule our move out as close to possible when we are planning to travel.

It’s a slight irritation. But we have it down to science

OpenAI o3 Breakthrough High Score on ARC-AGI-Pub

Mia Heidenstedt — Sat, 21 Dec 2024 16:51:53 GMT

Best viewed on the original page, where extended functionality like the footnote helper is available.

This is a article that explores the capabilities of OpenAI’s o3 model. HN

The impacts of AI reasoning are getting closer and closer to surpassing human capabilities.
But running it to solve a problem is extremely expensive.

Effectively, o3 represents a form of deep learning-guided program search. The model does test-time search over a space of “programs” (in this case, natural language programs – the space of CoTs that describe the steps to solve the task at hand), guided by a deep learning prior (the base LLM). The reason why solving a single ARC-AGI task can end up taking up tens of millions of tokens and cost thousands of dollars is because this search process has to explore an enormous number of paths through program space – including backtracking.

(programs is defined here as “My mental model for LLMs is that they work as a repository of vector programs”)

As far as i understand this and it’s implications i appears to me this process is more like a directed brute force with many candidates and selects the best one.
There is a thread on HN that discusses this topic.

GPT-4 Summary

OpenAI’s new o3 system has achieved a groundbreaking 75.7% on the ARC-AGI-Pub Semi-Private Evaluation within a $10k compute limit, surpassing previous models. A high-compute version scored 87.5%, marking a major step in AI’s adaptability to novel tasks. This success highlights a shift from scaling existing architectures to innovative mechanisms for generalization and test-time knowledge recombination.

Key achievements:

ARC-AGI performance: o3 shows unprecedented adaptability, unlike previous GPT-family models, which struggled with novel tasks.
Efficiency challenges: Low-compute costs $17–$20 per task but high-compute performance remains expensive and exploratory.
Core innovation: o3 employs natural language program search, generating and executing task-specific solutions during runtime, guided by deep learning priors.

Despite its advancements, o3 is not AGI, as it still fails simple tasks. Upcoming benchmarks, such as ARC-AGI-2 in 2025, aim to further challenge AI systems while encouraging open-source progress.

This breakthrough underscores a qualitative leap in AI research, reshaping paths toward AGI and sparking broader scientific engagement.

Washboard abs, great hires, and market dominance

Index on Dylan Fitzgerald — Sat, 21 Dec 2024 05:36:00 GMT

Abs are made in the kitchen, championships are won on the training ground, and a sustainable body of creative work is built on a foundation of the creator’s psychological self-acceptance.

Visakan Veerasamy, “I don’t wanna!”

Most of what we celebrate are trailing indicators. They’re exciting! A major product launch, lapping your competitors. Pulling a major hire into your team and watching them soar. Giving a successful talk at a conference.

These are great events, and we should celebrate them!

They are terrible expectations.

If you put in your work each day with the sole expectation of winning the market in 3 years, you’re going to flounder, burn out, and likely fail–and should none of those apply, you’re still counting on some luck.

If you instead spend each day looking for where you can do slightly better at one thing than the day before, and do it each and every day, with the expectation of getting slightly better…you’re going to get better at that thing. With time, the difference will be dramatic.

And you might not end up with the abs or the championship or the one perfect position or hire or owning the market. You’ll probably get some big wins mixed in there, too. They’ll be trailing indicators, and they’ll feel great, and they won’t be what you expected.

And when you’re done celebrating, it’ll be time to get back to setting expectations around what you can do better tomorrow.

Editor’s note: Ctrl+Shift will be sporadic through the next few weeks, resuming in earnest in 2025. Happy holidays and new year to all!

Handling keyboard shortcuts in JavaScript

Posts on rand[om] — Sat, 21 Dec 2024 02:00:00 GMT

I was recently reading the source code of mizu.js¹ and I liked the utility “function”² they have to handle keyboard shortcuts/combinations. I decided to slightly modify it to have it as part of my JavaScript utility functions.

The function

This function returns another function itself. You pass in a string that represents a combination of key presses like Shift+Alt+C, and it returns a function that will return true if a KeyboardEvent matches that combination. I had to make some modifications to the original code. First, I converted it to JavaScript, because I don’t like having a build step, and I want to be able to copy-paste this code. I also changed the matcher on the event.key to match on event.code because I found event.key was returning a string generated by the keyboard combination, not the actual key pressed.

Cramming scrapscript into msgpack

taylor.town — Sat, 21 Dec 2024 02:00:00 GMT

Every scrapscript program can be crammed into "flat" scraps.

Summary: The Tipping Point by Malcolm Gladwell

Chester Grant — Fri, 20 Dec 2024 23:55:49 GMT

1. The Tipping Point is the biography of an idea, and the idea is very simple. Ideas and products and messages and behaviors spread just like viruses do.

2. Three characteristics— one, contagiousness; two, the fact that little causes can have big effects; and three, that change happens not gradually but at one dramatic moment

3. The name given to that one dramatic moment in an epidemic when everything can change all at once is the Tipping Point.

4. And when an epidemic tips, when it is jolted out of equilibrium, it tips because something has happened, some change has occurred in one (or two or three) of those areas. These three agents of change I call the Law of the Few, the Stickiness Factor, and the Power of Context.

5. When it comes to epidemics, though, this disproportionality becomes even more extreme: a tiny percentage of people do the majority of the work.

6. In all of the city of Colorado Springs— a town of well in excess of 100,000 people— the epidemic of gonorrhea tipped because of the activities of 168 people living in four small neighborhoods and basically frequenting the same six bars. Who were those 168 people? They aren’t like you or me. They are people who go out every night, people who have vastly more sexual partners than the norm, people whose lives and behavior are well outside of the ordinary.

7. This idea of the importance of stickiness in tipping has enormous implications for the way we regard social epidemics as well. We tend to spend a lot of time thinking about how to make messages more contagious— how to reach as many people as possible with our products or ideas. But the hard part of communication is often figuring out how to make sure a message doesn’t go in one ear and out the other. Stickiness means that a message makes an impact. You can’t get it out of your head. It sticks in your memory.

8. The Stickiness Factor says that there are specific ways of making a contagious message memorable; there are relatively simple changes in the presentation and structuring of information that can make a big difference in how much of an impact it makes.

9. The Power of Context says that human beings are a lot more sensitive to their environment than they may seem.

10. The answer is that the success of any kind of social epidemic is heavily dependent on the involvement of people with a particular and rare set of social gifts: Connectors, Mavens, and Salesmen.

11. Most of us don’t have particularly broad and diverse groups of friends. In one well- known study, a group of psychologists asked people living in the Dyckman public housing project in northern Manhattan to name their closest friend in the project; 88 percent of the friends lived in the same building, and half lived on the same floor. In general, people chose friends of similar age and race.

12. But if the friend lived down the hall, then age and race became a lot less important. Proximity overpowered similarity.

13. We’re friends with the people we do things with, as much as we are with the people we resemble. We don’t seek out friends, in other words. We associate with the people who occupy the same small, physical spaces that we do.

14. The answer is that in the six degrees of separation, not all degrees are equal. When Milgram analyzed his experiment, for example, he found that many of the chains from Omaha to Sharon followed the same asymmetrical pattern. Twenty- four letters reached the stockbroker at his home in Sharon, and of those, sixteen were given to him by the same person, a clothing merchant Milgram calls Mr. Jacobs.

15. The balance of letters came to the stockbroker at his office, and of those the majority came through two other men, whom Milgram calls Mr. Brown and Mr. Jones. In all, half of the responses that came back to the stockbroker were delivered to him by these same three people.

16. Six degrees of separation doesn’t mean that everyone is linked to everyone else in just six steps. It means that a very small number of people are linked to everyone else in a few steps, and the rest of us are linked to the world through those special few.

17. These people who link us up with the world, who bridge Omaha and Sharon, who introduce us to our social circles— these people on whom we rely more heavily than we realize— are Connectors, people with a special gift for bringing the world together.

18. He found that 56 percent of those he talked to found their job through a personal connection. Another 18.8 percent used formal means— advertisements, headhunters— and roughly 20 percent applied directly. This much is not surprising; the best way to get in the door is through a personal contact. But, curiously, Granovetter found that of those personal connections, the majority were “weak ties.” Of those who used a contact to find a job, only 16.7 percent saw that contact “often”— as they would if the contact were a good friend— and 55.6 percent saw their contact only “occasionally.” Twenty- eight percent saw the contact “rarely.” People weren’t getting their jobs through their friends. They were getting them through their acquaintances.

19. Word- of- mouth epidemics are the work of Connectors.

20. Paul Revere was a Connector. But he was also— and this is the second of the three kinds of people who control word- of- mouth epidemics— a Maven. The word Maven comes from the Yiddish, and it means one who accumulates knowledge.

21. The critical thing about Mavens, though, is that they aren’t passive collectors of information. It isn’t just that they are obsessed with how to get the best deal on a can of coffee. What sets them apart is that once they figure out how to get that deal, they want to tell you about it too.

22. Mavens have the knowledge and the social skills to start word- of- mouth epidemics. What sets Mavens apart, though, is not so much what they know but how they pass it along. The fact that Mavens want to help, for no other reason than because they like to help

23. A Connector might tell ten friends where to stay in Los Angeles, and half of them might take his advice. A Maven might tell five people where to stay in Los Angeles but make the case for the hotel so emphatically that all of them would take his advice. These are different personalities at work, acting for different reasons. But they both have the power to spark word- of- mouth epidemics.

24. In a social epidemic, Mavens are data banks. They provide the message. Connectors are social glue: they spread it. But there is also a select group of people— Salesmen— with the skills to persuade us when we are unconvinced of what we are hearing, and they are as critical to the tipping of word- of- mouth epidemics as the other two groups. Who are these Salesmen? And what makes them so good at what they do?

25. If we think about emotion this way— as outside- in, not inside- out— it is possible to understand how some people can have an enormous amount of influence over others. Some of us, after all, are very good at expressing emotions and feelings, which means that we are far more emotionally contagious than the rest of us. Psychologists call these people “senders.” Senders have special personalities.

26. They are also physiologically different. Scientists who have studied faces, for example, report that there are huge differences among people in the location of facial muscles, in their form, and also— surprisingly— even in their prevalence. “It is a situation not unlike in medicine,” says Cacioppo. “There are carriers, people who are very expressive, and there are people who are especially susceptible. It’s not that emotional contagion is a disease. But the mechanism is the same.”

27. Only the charismatic person could infect the other people in the room with his or her emotions.

28. The Law of the Few says that there are exceptional people out there who are capable of starting epidemics. All you have to do is find them.

29. The lesson of stickiness is the same. There is a simple way to package information that, under the right circumstances, can make it irresistible. All you have to do is find it.

30. Wilson and Kelling argued that crime is the inevitable result of disorder. If a window is broken and left unrepaired, people walking by will conclude that no one cares and no one is in charge. Soon, more windows will be broken, and the sense of anarchy will spread from the building to the street on which it faces, sending a signal that anything goes. In a city, relatively minor problems like graffiti, public disorder, and aggressive panhandling, they write, are all the equivalent of broken windows, invitations to more serious crimes: Muggers and robbers, whether opportunistic or professional, believe they reduce their chances of being caught or even identified if they operate on streets where potential victims are already intimidated by prevailing conditions. If the neighborhood cannot keep a bothersome panhandler from annoying passersby, the thief may reason, it is even less likely to call the police to identify a potential mugger or to interfere if the mugging actually takes place.

31. This is an epidemic theory of crime. It says that crime is contagious— just as a fashion trend is contagious— that it can start with a broken window and spread to an entire community.

32. Broken Windows theory and the Power of Context are one and the same. They are both based on the premise that an epidemic can be reversed, can be tipped, by tinkering with the smallest details of the immediate environment.

33. Psychologists call this tendency the Fundamental Attribution Error (FAE), which is a fancy way of saying that when it comes to interpreting other people’s behavior, human beings invariably make the mistake of overestimating the importance of fundamental character traits and underestimating the importance of the situation and context.

34. There is something in all of us that makes us instinctively want to explain the world around us in terms of people’s essential attributes: he’s a better basketball player, that person is smarter than I am.

35. Judith Harris has convincingly argued that peer influence and community influence are more important than family influence in determining how children turn out. Studies of juvenile delinquency and high school drop- out rates, for example, demonstrate that a child is better off in a good neighborhood and a troubled family than he or she is in a troubled neighborhood and a good family.

36. Wesley realized that if you wanted to bring about a fundamental change in people’s belief and behavior, a change that would persist and serve as an example to others, you needed to create a community around them, where those new beliefs could be practiced and expressed and nurtured.

37. Dunbar has actually developed an equation, which works for most primates, in which he plugs in what he calls the neocortex ratio of a particular species— the size of the neocortex relative to the size of the brain— and the equation spits out the expected maximum group size of the animal. If you plug in the neocortex ratio for Homo sapiens, you get a group estimate of 147.8— or roughly 150. “The figure of 150 seems to represent the maximum number of individuals with whom we can have a genuinely social relationship, the kind of relationship that goes with knowing who they are and how they relate to us.

38. Dunbar has combed through the anthropological literature and found that the number 150 pops up again and again. For example, he looks at 21 different hunter- gatherer societies for which we have solid historical evidence, from the Walbiri of Australia to the Tauade of New Guinea to the Ammassalik of Greenland to the Ona of Tierra del Fuego and found that the average number of people in their villages was 148.4.

39. Over the years military planners have arrived at a rule of thumb which dictates that functional fighting units cannot be substantially larger than 200 men,” Dunbar writes. “This, I suspect, is not simply a matter of how the generals in the rear exercise control and coordination, because companies have remained obdurately stuck at this size despite all the advances in communications technology since the first world war.

40. Then there is the example of the religious group known as the Hutterites, who for hundreds of years have lived in self- sufficient agricultural colonies in Europe and, since the early twentieth century, in North America. The Hutterites (who came out of the same tradition as the Amish and the Mennonites) have a strict policy that every time colony approaches 150, they split it in two and start a new one. “Keeping things under 150 just seems to be the best and most efficient way to manage a group of people,” Bill Gross, one of the leaders of a Hutterite colony outside Spokane told me. “When things get larger than that, people become strangers to one another.”

41. The Rule of 150 suggests that the size of a group is another one of those subtle contextual factors that can make a big difference.

42. Heavy smokers have been shown to have a much greater sex drive than nonsmokers. They are more sexually precocious; they have a greater “need” for sex, and greater attraction to the opposite sex.

43. At age nineteen, for example, 15 percent of nonsmoking white women attending college have had sex. The same number for white female college students who do smoke is 55 percent.

44. Contagiousness is in larger part a function of the messenger. Stickiness is primarily a property of the message.

45. In a series of large and well-designed studies of twins — particularly twins separated at birth and reared apart — geneticists have shown that most of the character traits that make us who we are — friendliness, extroversion, nervousness, openness, and so on — are about half determined by our genes and half determined by our environment, and the assumption has always been that this environment that makes such a big difference in our lives is the environment of the home. The problem is, however, that whenever psychologists have set out to look for this nurture effect, they can’t find it.

46. On things like measures of intellectual ability and certain aspects of personality, the biological children are fairly similar to their parents. For the adopted kids, however, the results are downright strange. Their scores have nothing whatsoever in common with their adoptive parents: these children are no more similar in their personality or intellectual skills to the people who raised them, fed them, clothed them, read to them, taught them, and loved them for sixteen years than they are to any two adults taken at random off the street.

47. Two years later, Columbia University psychologist Alexander Glassman discovered that 60 percent of the heavy smokers he was studying as part of an entirely different research project had a history of major depression. About 80 percent of alcoholics smoke. Close to 90 percent of schizophrenics smoke.

First month in my new home

Liza Shulyayeva — Fri, 20 Dec 2024 23:01:33 GMT

It’s been exactly one month since I moved to Uppsala. I’m still adjusting and discovering my routines here, but here’s what I’ve been up to so far.

The IPv6 situation on Docker is good now!

./techtipsy — Fri, 20 Dec 2024 19:30:00 GMT

Good news, everyone! Doing IPv6 networking stuff on Docker is actually good now!

I’ve recently started reworking my home server setup to be more IPv6 compatible, and as part of that I learned that during the summer of 2024 Docker shipped an update that eliminated a lot of the configuration and tweaking previously necessary to support IPv6.

There is no need to change the daemon configuration any longer, it just works on Docker Engine v27 and later.

Examples

If your host has a working IPv6 setup and you want to listen to port 80 on both IPv4 and IPv6, then you don’t have to do anything special. However, the container will only have an IPv4 address internally. You can verify it by listing all the Docker networks via sudo docker network ls and running sudo docker network inspect network-name-here for the one associated with your container.

For services like nginx that log the source IP address, this is problematic, as every incoming IPv6 request will be logged with the Docker network gateway IP address, such as 10.88.0.1.

name: nginx
services:
  nginx:
    container_name: nginx
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always

If you want the container to have an IPv4 and an IPv6 address within the Docker network, you can create a new network and enable IPv6 in it.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

There are situations where it’s handy to have a static IP address for a container within the Docker network. If you need help coming up with an unique local IPv6 address range, you can use this tool.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      nginx-network
        ipv4_address: 10.69.42.5
        ipv6_address: fdec:cc68:5178::abba
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: "10.69.42.0/24"
        - subnet: "fdec:cc68:5178::/64"

If you choose the host network driver, your container will operate within the same networking space as your container host. If the host handles both IPv4 and IPv6 networking, then your container will happily operate with both. However, due to reduced network isolation, this has some security implications that you must take into account.

name: nginx
services:
  nginx:
    container_name: nginx
    network_mode: host
    # ports are not relevant with host network mode
    image: docker.io/library/nginx
    restart: always

If you want your container to only accept connections on select interfaces, such as a Wireguard connection, then you will need to specify the IP addresses in the ports section. Here’s one example with both IPv4 and IPv6.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 10.69.42.5:80:80
      - "[fdec:cc68:5178::beef]:80:80"
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

What about Podman?

I’ve given up on Podman. Before doing things the IPv6 way, Podman was functional for the most part, requiring a few tweaks to get things working.

I have not managed to get Podman to play fair with IPv6. No matter what I did, I could not get it to listen to certain ports and access my services, the ports would always be filtered out.

Conclusion

I’m genuinely happy to see that the IPv6 support has gotten better with Docker, and I hope that this short introduction helps those out there looking to do things the IPv6 way with containers.

Reviewing Tomorrow, and Tomorrow, and Tomorrow by Gabrielle Zevin

Davis Haupt's Blog — Fri, 20 Dec 2024 19:00:00 GMT

A book about life, love, and video games. I honestly didn’t know what I was getting into when I started this book, but by the time they described the twist behind Sadie’s first video game in the second chapter I was hooked. Some people may be put off by how the characters and the third-person narrator talk about video games as a high art form but I love hearing and seeing how digital media can expand our artistic horizons rather than shrink them.

A friend who read the book around the same time described it as “a romance, I guess?” and that ambiguity is at the heart of what makes this book feel real. As we follow the main characters from their college days into their thirties, the ways their views about the world and each other change does a lot towards making the characters feel like real people.

Ultimately I got a lot out of what this book had to say about friendship. It’s natural for miscommunications to accumulate into misunderstandings over the years despite our best intentions. What’s important in the end is to remember that it’s never too late to return to the foundation and try to grow something new.

2024 Our commitment to resource feminist tech at the frontlines

Numun Fund — Fri, 20 Dec 2024 18:37:59 GMT

Dear community, We close this year with many feelings – gratitude and joy for being able to have been in community with many of our partners, nodes and constellations of activists at different spaces throughout the year; grave reflection on the many ways that technology is implicated in how we are experiencing the surfacing of […]

Defending Farm Animal Sanctuaries from Effective Altruism

Josh Baldwin — Fri, 20 Dec 2024 17:38:06 GMT

This piece expands on the content of The Power Of Farm Animal Sanctuaries by defending the work of farm animal sanctuaries from the principles of Effective Altruism.

De Honoris Numquam Satis: Reflections on Snobbery in Academia

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Fri, 20 Dec 2024 17:00:00 GMT

One of my more endearing characteristics—or so I like to believe—is my naïveté in the face of academic reality. It is probably amusing to those who know me personally to see how I just do not ‘grok’ certain things, making me either bewildered, angry, sad, or a linear combination thereof. One of the many, many, many things that fall into this category is the tendency of some academics to, in their hunger for more honour and prestige, make devilish compromises.

Let me explain! As you might know, everyone tries to measure academic output by ludicrous indices. While this is mostly nothing but modern haruspicy, combined with a thin veneer of plausibility, some universities and stakeholders dig this stuff and lap it right up. Every year, academia goes crazy while companies like Clarivate emerge from their lairs and announce what they read in this year’s entrails. You then get the dubious honour of appearing on lists like the one of highly-cited researchers, a list that does not even include mathematics or machine learning. I guess those fields were not deemed to be scientific enough—but I digress. Suffice it to say, everyone loves to count citations as the measure of success, culminating in absurd indices like the h-index, of which much has been written elsewhere, and so I shall not start another rant on my own. The two things for you, dear reader, to understand about the h-index are, to wit:

Larger h-indices are obviously directly equivalent to your worth as a researcher, and, by extension, your value as a human being.¹
The h-index is a function of your publications; you need $k$ publications with at least $k$ citations to have an h-index of $k$.

This leads the more, shall we say, incentive-driven senior academics to adopt an interesting strategy: They will force their junior researchers to put them on any and all papers possible. Let us take another brief explanatory digression here: Science is foremost a conversation about potential facts. To participate in this conversations, budding scientists are expected to publish. Each publication has an author list, and, depending on your field, the author list conveys additional information about the relevance of each individual author. For instance, in computer science, it is typically the first author(s) who are more junior and did most of the experiments or implementations, while the last author(s) are typically acting in a more supervisory capacity. As someone who is now in the position of such a senior author, I can spill the beans and mention that when I am the last author, I typically focus most of my energies in the project on discussing ideas with students, co-writing the paper with them, critiquing their drafts or code, as opposed to coding large parts by myself. This is because I believe that Ph.D. students should be considered apprentices who are supposed to excel in their craft, meaning that supervisors need to guide them, but not leave them to their own devices.

With the view of an apprenticeship in mind, it becomes abundantly clear to me that, if a student has their own project cooking in which I am not involved, said student can do publish their results without me. Or, in other words, while I believe my very presence in the lab to be a balm for anyone’s soul, and an inspiring presence to boot,² capable of lifting the spirits of everyone and getting them unstuck, even my ego knows its bounds. Essentially, since I picked excellent people to work with me, I should not be surprised when they come up with great ideas on their own and run with them. Hence, my dictum: As much as I would like it to be true, my inspiring presence does not entitle me to authorship.³

Imagine my astonishment when I learned that some outwardly accomplished academics force their way on the author lists, in a transparent ploy to increase their h-index but also to increase their prestige. Taken to the very extreme—and it often is by some people—this results in some academics ‘publishing’⁴ more than one paper per week. All so that they can say, presumably while pointing towards their Google Scholar page, ‘Look on my works, ye Mighty, and despair!’ Alas, they might not know the end of Ozymandias, but what is even more baffling to me is that such ‘honours’ must feel hollow, no?

I honestly do not know and understand it, and can only draw upon my native language of German, which of course has a word for this general class of behaviour, namely Standesdünkel. Some dictionaries want you to believe that it is professional snobbery, but it is actually so much more, going back to a time when nobility was everything, and commoners were nothing. In my mind, I hear professional Standesdünkel when professors, like the lords and ladies of yore, automatically consider authorship as a type of sinecure or tithes. Moral considerations notwithstanding, at which point would the folks that are doing this shake their heads and say ‘Enough is enough, maybe from now on, I will just be on papers that I actually contributed something to’? Because, make no mistake, this Standesdünkel is hurting younger academics. It is hurting the Ph.D. student who has a great idea on their own and pursues it, maybe even in their scarce free time. It is hurting the postdoctoral researcher who leads a small team of students into a new direction, without getting any input from the professor. It is even hurting younger faculty members who need to put the éminence grise on everyone of their papers, just to be then told in an evaluation that ‘Their research is not sufficiently different from the research directions of their advisers.’

So again, why behave like that? Does this honour not taste hollow? If you have reached a triple-digit h-index and a triple-digit publication list, is it not time to do without tithes? As you can imagine, I have no answers to these questions. I just wonder where it would stop—would some senior folks believe to be entitled to authorship because they looked at a Ph.D. student once? When does the entitlement end?

In all honesty, I also have my misguided notions about my own academic prestige and the honours bestowed upon me,⁵ but I am not willing to deviate from my principles for what I conceive to be, ultimately, a hollow win at best. Moreover, I do not believe you can ‘win’ academia. It is, for all intents and purposes, an infinite game, so the best you can hope for is that you get to keep playing.

Happy playing, until next time!

Just to be sure: This is definitely satirical! ↩︎
Think of the spells cast by your standard Cleric character in Dungeons & Dragons. This might require another exegesis, which I shall save for another post. ↩︎
This is of course not categorically so; for instance, if we start discussing a project and I actually can contribute to it, and, more importantly, the students wants me to contribute to it, I might have earned my spot on the author list. ↩︎
These are air quotes and they do some very heavy lifting. ↩︎
That is, I believe that more would always be better for me. ↩︎

Feminist tech: the live tissues between tech & our justice struggles. Closing remarks at the 15th AWID Forum.

Numun Fund — Fri, 20 Dec 2024 02:41:22 GMT

Speech by Jac sm Kee at the 15th AWID Forum closing plenary 6th December 2024 The way that technology has been threaded in this forum has been significantly different from all the previous forums I have been a part of. It’s a lot more central, a lot more present, and woven through a lot more […]

How to Use PostgreSQL for Data Normalization

Dylan Paulus' Blog — Fri, 20 Dec 2024 02:00:00 GMT

Managing a database is more than just storing information—it's about storing data efficiently, effectively, and balancing trade-offs. We can employ techniques like data normalization to optimize our data for maintainability and readability, or data denormalization to optimize for raw query speed. Data normalization is the process of breaking data down into structured tables to reduce duplication and make it easier to query and store by enforcing standardization.

In this article, we'll take a look at what data normalization is, how to apply normal forms to achieve data normalization, challenges and tools for data normalization in PostgreSQL, tips on when to denormalize, and how TimescaleDB makes normalization easier.

Will all our drugs come from China?

Alex’s blog — Fri, 20 Dec 2024 02:00:00 GMT

Template nesting in (text|html)/template

Paul's page — Fri, 20 Dec 2024 02:00:00 GMT

Go includes a safe and somewhat performant templating library. But the best way to structure code using html/template has eluded me for a long time.

Modern Portfolio Theory II: Random portfolio coverage in log volatility—return space with a risk free asset

Bytepawn - Marton Trencseni — Fri, 20 Dec 2024 01:00:00 GMT

In a previous article I simulated random portfolios using Monte Carlo methods for the 2023 daily closing prices of the 100 stocks constituting the Nasdaq-100. Here I add the risk-free asset to the portfolio and examine how it affects the coverage in log volatility—return space.

Label-Studio: Annotate Text and Image Data for AI and ML training

Saeed Esmaili — Thu, 19 Dec 2024 23:42:00 GMT

A few months ago I used streamlit to build a simple UI, so I can collect manually labeled data for a LLM fine-tuning task at work. Streamlit is fine, but the full process of creating a nice UI with required functionalities for data annotation and data storage management wasn’t trivial.

Today I found out about label-studio which is an easy to use framework (backend and frontend) for data annotation task. It provides various annotation templates for text, image, audio, and video data!

Hugo Social Image Previews

Thought Eddies — Thu, 19 Dec 2024 23:27:37 GMT

Hugo Social Image Previews

Quickly Filter and Aggregate Python Lists

Saeed Esmaili — Thu, 19 Dec 2024 21:45:19 GMT

Today I came across this brilliant python library called leopards which allows you to do some basic but frequently used filters and aggregations on python lists. I"ve always used pandas for any quick and adhoc work with large lists or CSV files, but leopards sounds a quick and performant alternative when you don"t need to do any fancy data analysis work.

Leopards provides following filters:

eq: equals and this default filter
gt: greater than.
gte: greater than or equal.
lt: less than
lte: less than or equal
in: the value in a list of a tuple.
- e.g. age__in=[10,20,30]
contains: contains a substring as in the example.
icontains: case-insensitive contains.
startswith: checks if a value starts with a query strings.
istartswith: case-insensitive startswith.
endswith: checks if a value ends with a query strings.
iendswith: case-insensitive endswith.
isnull: checks if the value matches any of NULL_VALUES which are ("", ".", None, "None", "null", "NULL")
- e.g. filter__isnull=True or filter__isnull=False

A quick example of its usage with filters:

Summary: Go for the No!

Chester Grant — Thu, 19 Dec 2024 21:03:32 GMT

1. “If you’re not succeeding fast enough, you’re probably not failing fast enough, and you can’t have one without the other. So, if you’re going to avoid one, you’re going to avoid both.”

2. ‘The salesperson never decides when the sale is over; the customer does.’ Then he looked me in the eye and said, ‘Eric, your fear of hearing the word ‘no’ is the only thing standing between you and greatness.’

3. “After the conversation with Harold I started really looking at what set successful people apart from the masses, and their willingness to fail was at the top of the list.

4. Successful people fail eagerly while failures avoid failing. The whole point of becoming willing to fail more is to become a success, so that one day you won’t be forced to look back on your life and say to yourself, ‘I’m a failure.’

5. A willingness to fail means a person will tolerate just enough failure to get what they need from life, and no more. A wantingness, on the other hand, means you’re not just tolerating the no’s in your life, you’re actually beginning to seek them.

6. What if we decided to make each no we received and every rejection we encountered something that empowers us? Instead of avoiding rejection, what if we made the decision to seek rejection? Instead of avoiding no or perhaps simply tolerating it, what if we went out of our way to actually go for no!”

7. “Simple. Rather than setting goals for the number of yes’s you are planning to get each week, you set goals for the number of no’s you’re going to collect.”

8. “Let’s use my friend Paul who is in network marketing, for example. If his goal is to get ten people per week to come to a meeting, and typically about five percent of the people he approaches are willing to attend, then his goal would be to get one hundred and ninety people to say ‘no thanks.’”

9. Believe it or not, research shows that eighty-five percent of all interactions between retail salespeople and shoppers end without the salesperson ever asking for a buying decision.

10. “Absolutely! Studies show that as many as 80% of all salespeople don’t make it through their first year for the simple reason that they failed to make enough calls. That’s it. Nothing else.”

11. “That reminds me of this telemarketing guru I heard about,” I interjected. “He recommends that phone solicitors ask customers in the first ten seconds if they have interest in hearing about the product they’re selling. If the customer says no, then they politely say thank you and move on rather than going through their entire pitch. As a result, they make ten times as many calls, but only invest their time making their complete sales pitch to prospects who have just qualified themselves.” “It’s the same basic premise,” Cheryl replied. “You know, when you mine for gold, you don’t really look for the gold, you remove the dirt. Selling and gold mining are very much alike. It’s the people who remove the most dirt, who work their way through the greatest number of no’s, who ultimately discover the greatest number of golden yes’s!”

12. “the best piece of advice I can offer is to learn that no doesn’t mean never, it means not yet. Statistically, research shows that forty-four percent of salespeople give up after one no. Twenty-two more give up after the second no. Fourteen percent more give up after the third no. Twelve more give up after the fourth no. What does that come to?”

13. “Correct. Ninety-two percent of all salespeople give up without asking for the sale a fifth time, but research also shows that sixty percent of all customers say no four times before they finally say yes. That means the quickest way to separate yourself from the rest of the pack is to get at least five no’s from everyone you try to sell to!” “And you track that?” I asked.

14. “Because when I get someone to say no, I can immediately move to the next step which is to ask, ‘why?’ Let me think about it teaches me absolutely nothing, but if they say no and I follow up with why, now I’m on the verge of discovering what I need to do next to make the sale.”

15. “Well, I’d have to say my advice would be this… If you’re going to fail, fail big!” Kurt stated. “Elaborate on that,” Eric encouraged. “What I mean is simply this,” Kurt continued. “Common sense says that if you’re going to get a no from somebody, get it from the client who needs twenty copiers, not just two. Get rejected by the purchasing agent from the company that buys forty thousand gallons of cleaning solution every month, not just forty.

16. “Think about the numbers for a moment. If I call on one hundred accounts that each have the potential to lease two copiers from me, and my closing ratio is ten percent, then I end up with twenty units out the door, right? But by focusing on accounts with the potential to lease twenty copiers, I only needed to close one sale to get the same twenty units!”

17. “For starters, I’d say that a primary key to creating outrageous success is to understand the need to fail exponentially. After all, one person can only fail so fast. Great leaders help everyone in the organization understand the need to fail faster.

18. “Exponential failure requires someone in the organization to be an exponent, in other words an advocate or spokesman, of the failure concept and to champion it. But I can’t begin to tell you how many executives I’ve seen who reach the top and suddenly forget what got them there. They start trying to avoid failure, and when a leader is afraid to fail, everyone in the organization knows it. Not only do people sense it but they figure that if you’re afraid of something, there’s a good chance it’s something they should be afraid of, too.”

19. “The second is to reward people for their failures, not just their successes,” he responded. “Everyone runs over to congratulate people for their sales successes, but how often do we go out of our way to congratulate people for their failures?” “Virtually never,” I said.

20. Don’t get me wrong; I’m all in favor of rewarding people for their successes, but not to the exclusion of recognizing the people who’ve displayed a true willingness and wantingness to fail.

21. “At CopyQuest, we recognize the top ten salespeople with a ‘Producers Pin’ award,” Eric continued, “but we also recognize the ten salespeople in the organization with the greatest number of failed attempts with what we call the ‘Go for No!’ award. And, if I recall correctly, Cheryl and Kurt made both lists.”

22. “That’s because we’ve caught on to the concept that winning a ‘Go for No!’ award almost assures that you’ll end up with a Producers Pin. Or, put another way,” Kurt concluded, “Yes is the destination, No is how you get there!”

The Fall of Democracy is a Markov Process

Maximum Effort, Minimum Reward — Thu, 19 Dec 2024 18:12:25 GMT

Did you know that the Ancient Greek word "anacyclosis" translates to "Markov-chain Monte-Carlo"?

“The Godfather II” of Tech Book Sequels

JabPerf Corp — Thu, 19 Dec 2024 17:59:42 GMT

I’m going to tell you something that you probably already know. . . sequels usually suck. Oh, they got me when they told me that Denzel Washington would be in Gladiator II, that interminable dud of a film. How about that pitiful money-grab The Hangover II? And the “My Belly Laugh” differential between Bad Santa […]

The post “The Godfather II” of Tech Book Sequels appeared first on JabPerf Corp.

The Absurdity of Big Chess

Obvious Bicycle — Thu, 19 Dec 2024 15:03:22 GMT

Babies watching Shakespeare

--Mike-- — Thu, 19 Dec 2024 12:35:44 GMT

I can't get over the feeling that I'm wasting whatever portion of my life I've got left.

In this video - https://www.youtube.com/watch?v=4jgTCayWlwc

"A rant on personal engineering projects" by BPS.shorts

The author makes the point that when you're working commercially on part of a project, the organization always has a way to keep the project going, even if you fail. If the cost goes over, they can usually change the scope or budget to fit it in. If the parts don't integrate well, they can buy a system already integrated, or even the company that has already done it.

However, on a personal project, if any part fails, the whole thing fails. The main hazard is simply not finishing the project, rather than doing any one stage optimally. I think there's a shit-ton of value in this analysis.

Good Quote from Pale Rose in the comments @ https://www.youtube.com/watch?v=4jgTCayWlwc&lc=Ugxjv3J38ajw82_bmtF4AaABAg

*Personal projects have a unique requirement that you must optimize for: 'finishing the damn project'. Everything else is secondary*

I want to be a high-?? individual, someone who can get shit done. High agency?

Yep... that's the term, from https://x.com/tferriss/status/1719035575305408551

During my podcast interview with @EricRWeinstein, Eric said “high-agency person” in passing, and I asked him to elaborate:
“When you’re told that something is impossible, is that the end of the conversation, or does that start a second dialogue in your mind, how to get around whoever it is that’s just told you that you can’t do something? So, how am I going to get past this bouncer who told me that I can’t come into this nightclub? How am I going to start a business when my credit is terrible and I have no experience?”"

Mike's plan for becoming high-agency

List the projects I have, maintain said list
Always be working to finish the things on the list, or pruning the list
Always be gathering resources, or using them, never just hording them

It's my Birthday, here... have a BitGrid

--Mike-- — Thu, 19 Dec 2024 12:31:02 GMT

If you've read about von Neuman, and wondered what it was like to bootstrap computing, and if there could be another way.... here's an interesting Birthday Gift from me to you... the BitGrid, and a writeup, on an HN thread

https://news.ycombinator.com/item?id=42115107

Black Doves (Limited Series): A dovish review during a red Christmas

🦉 olshansky 🦁 — Thu, 19 Dec 2024 11:47:17 GMT

Olshansky's review of Black Doves (Limited Series)

Pydantic Logfire for LLM and API Observability

Saeed Esmaili — Thu, 19 Dec 2024 02:00:00 GMT

I’ve been using sentry for automatically logging the errors and exceptions of my python projects. A few months ago I needed to log some information if a specific condition is true in my side project’s backend, but I wasn’t able to do this with sentry. It apparently can only work when something fails, and you can’t capture log messages if there’s no failure or exception.

I looked for an affordable and user friendly observability tool and settled on using axiom . It has a generous 500GB ingestion on free tier plan, but you can only view the events for the past 30 days time period. So I’ve been exporting the logs every month into a csv file, since I want to be able to view the trend of some behaviours over time.

ob-metapost

her.esy.fun — Thu, 19 Dec 2024 01:00:00 GMT

When I searched how to generate pictures in org-mode on the Internet, Metapost is surprisingly missing.

If you never tried Metapost, I think you should really take a look at this gem.

This article will simply provide an example how I could create an emacs package (not yet distributed) to use Metapost in my org-mode documents.

The script

One of the main issue with Metapost is that, like $L^AT_EX$ , it generates a bunch of temporary files and clutter your current directory with them. There is also a notion that Metapost is done to generate not a single, but multiple files.

For our use case, we just want a program to generate a single image.

This is why I wrote a very simple mp2png script. It is inspired by mp2png.sh I could find in bsdowl.

In short, this script move a Metapost single figure code into a temporary directory. We prepare enclose the figure description into a Metapost template. We perform the compilation in that temporary directory.

If you remove all the ceremony this is simply:

# Generate a temporary directory
tmpdir="$(mktemp -d)"
# Name the temporary metapost
tmpmp="${tmpdir}/tmp.mp"
# Provide a prelude
cat > "$tmpmp" <<EOF
prologues:=3;
outputtemplate:="tmp-1.eps";
EOF
# copy the content of your metapost code
cat "$src" >> "$tmpmp"
# close the metapost file
cat >> "$tmpmp" <<EOF
end.
EOF
# Move to the temporary directory
pushd "$tmpdir" || err "ERROR: pushd \"${tmpdir}\""
# Compile the metapost file
mpost "tmp.mp"
# Convert the EPS file to PNG
gm convert -density "$density" "tmp-1.eps" "tmp-1.png"
# Return to the original directory
popd || err "ERROR: popd (tmpdir= \"${tmpdir}\")"

With all the details this become a bit more complex because I would like to be able to pass the code from stdin as this will ease to write the org babel code. Here is the full script:

#!/usr/bin/env zsh

# dependencies:
# - metapost
# - graphicsmagick

set -e

cmd=${0:t}
showhelp() {
    echo "$cmd SRC [DEST]"
    echo ""
    echo "SRC could be any file if - is provided read from stdin"
    echo "    if - is provided the default file name will be result.png"
    echo "DEST could be any file if - is provided read from stdin"
}

if (( $# == 0 )); then
    showhelp
fi

src="$1"
density=1200
tmpdir="$(mktemp -d)"
if (( $# > 1 )); then
    maindst="$2"
elif [[ "$src" = "-" ]]; then
    maindst="result.png"
else
    maindst="${src%.*}.png"
fi

err() {
    local msg="$1"
    echo "$msg" >&2
    exit 1
}

dst="$maindst"
# i=0
# while [[ -e $dst ]]; do
#     dst="${maindst%.*}-$((++i)).png"
# done

tmpmp="${tmpdir}/tmp.mp"
cat > "$tmpmp" <<EOF
prologues:=3;
outputtemplate:="tmp-1.eps";
% TEX macro is short enough to be copied
string preverbatimtex_, postverbatimtex_;
vardef TEXPRE text s =  preverbatimtex_ := s; enddef;
vardef TEXPOST text s =  postverbatimtex_ := s; enddef;
vardef TEX primary s =
  if known preverbatimtex_:
    write "verbatimtex "&preverbatimtex_&" etex" to "mptextmp.mp";
  fi
  write "btex "&s&" etex" to "mptextmp.mp";
  if known postverbatimtex_:
    write "verbatimtex "&postverbatimtex_&" etex" to "mptextmp.mp";
  fi
  write EOF to "mptextmp.mp";
  scantokens "input mptextmp"
enddef;
beginfig(1);
EOF
if [[ "$src" = "-" ]]; then
    cat >> "$tmpmp"
else
    cat "$src" >> "$tmpmp"
fi
cat >> "$tmpmp" <<EOF
endfig;
end.
EOF
pushd "$tmpdir" || err "ERROR: pushd \"${tmpdir}\""
mpost "tmp.mp"
gm convert -density "$density" "tmp-1.eps" "tmp-1.png"
popd || err "ERROR: popd (tmpdir= \"${tmpdir}\")"
cp "${tmpdir}/tmp-1.png" "$dst"
echo "File: $dst"

And now for the code in emacs, inspired by ob-ditaa:

;;; ob-metapost.el --- Babel Functions for metapost        -*- lexical-binding: t; -*-

;; Copyright (C) 2024 Free Software Foundation, Inc.

;; Author: Yann Esposito
;; Keywords: literate programming, reproducible research
;; Homepage: https://orgmode.org

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;; Org-Babel support for evaluating metapost source code.
;;
;; This differs from most standard languages in that
;;
;; 1) there is no such thing as a "session" in metapost
;;
;; 2) we are generally only going to return results of type "file"
;;
;; 3) we are adding the "file" and "cmdline" header arguments
;;
;; 4) there are no variables (at least for now)

;;; Code:
(require 'ob)
(require 'org-compat)

(defvar org-babel-default-header-args:metapost
  '((:results . "file")
    (:exports . "results"))
  "Default arguments for evaluating a metapost source block.")

(defcustom org-babel-metapost-cmd "mp2png"
  "Metapost to png command"
  :group 'org-babel
  :type 'string)

(defun org-babel-execute:metapost (body params)
  "Execute a block of Metapost code with org-babel.
This function is called by `org-babel-execute-src-block'."
  (let* ((out-file (or (cdr (assq :file params))
               (error "Metapost code block requires :file header argument")))
     (cmdline (cdr (assq :cmdline params)))
     (in-file (org-babel-temp-file "metapost-"))
     (cmd (concat org-babel-metapost-cmd
                      " " cmdline
              " " (org-babel-process-file-name in-file)
              " " (org-babel-process-file-name out-file))))
    (with-temp-file in-file (insert body))
    (message cmd)
    (shell-command cmd)
    nil)) ;; signal that output has already been written to file

(defun org-babel-prep-session:metapost (_session _params)
  "Return an error because metapost does not support sessions."
  (error "Metapost does not support sessions"))

(provide 'ob-metapost)

;;; ob-metapost.el ends here

And that's it. That was a lot easier than I expected. And now, I can generate image like this:

u := 0.2cm;
z0 = (0,0);
z1 = (sqrt(3)*cm,0);
z2 = (sqrt(3)*cm,1cm);
draw z0--z1--z2--cycle;
label.bot(btex $\sqrt{3}$ etex, 1/2[z0,z1]);
label.rt(btex 1 etex, 1/2[z1,z2]);
label.top(btex 2 etex, 1/2[z0,z2]);
z3 = z1 - (u,0);
z4 = z1 + (-u,u);
z5 = z1 + (0,u);
draw z3--z4--z5;

Triangle with LaTeX fonts

or this:

numeric r, theta; r = sqrt 1/2; theta = 45;
vardef dragon(expr a, b) =
  if abs(a-b) > 8:
    save p; pair p;
    p = r[a, b] rotatedabout(a, theta);
    dragon(a, p) & reverse dragon(b, p)
  else:
    a .. b
  fi
enddef;
vardef rounded_corners expr p =
  save r, n; numeric r, n; r = 1/4; n = length p;
  subpath (0, 1-r) of p
  for t=1 upto n-1:
    .. subpath (t+r, t+1-r) of p
  endfor .. subpath (n-r, n) of p
enddef;
draw rounded_corners dragon(origin,240 right) withcolor (.6,.2,.7);

Dragon fractal

I hope it will help people to find the resources to write org babel integrations as well as help people use Metapost more.

On The Road in Iceland!

The Road Chose Me — Wed, 18 Dec 2024 22:29:36 GMT

It’s been a long time since I posted a upate here on The Road Chose Me – I’ve been busy with YouTube videos, writing books, speaking at events, writing magazine articles, building a new...

How Many Hours Can You Code?

The Codist — Wed, 18 Dec 2024 22:03:18 GMT

How many hours a day can you write code, and at what point does the quality of your work go down? Even more important is how many weeks and months of that max effort you can still be effective.

In my life, there have only been three periods where I

We all need our own philosophy

An Ordinary Disaster — Wed, 18 Dec 2024 21:53:48 GMT

How a dream of "Drepa's Shirt" illustrates my personal philosophy and mythology

A Guide to Commodore PETs

mass:werk – Now Go Bang! — Wed, 18 Dec 2024 19:40:00 GMT

The original line-up of Commodore PET computers at a glance.

Massively Multiplayer Governance

Open Indie — Wed, 18 Dec 2024 17:00:06 GMT

On Scooters as a class of vehicle/tool

Josh Thompson — Wed, 18 Dec 2024 15:00:00 GMT

Introduction

Often when I say “scooter”, especially in the united states, the person thinks of something different than what I mean. Here’s a recent post from the Sportique Scooters instagram:

source

This is the kind of vehicle I’m talking about when I say “scooter”.

I once had a vehicle just like this photo. The helmet I used (and still use) is not as charming as this helmet, unforunately, but it is a bit safer, and I want the best head protection I can get. ¹

There’s more to say about this kind of vehicle, and I’ve tried to, a few different ways. This page currently serves as “just” a jumping off point to other scooter-related resources.

Collections of writings about scooters

See, it’s not really about scooters, per se. It’s the verb of the thing. Scooters are different than cars, but the only reason it matters at all is because scooter-ing is vastly different than using a car, or a bike, or a motorcycle.

👉 https://josh.works/scootering

My tiktok has lots of footage from/about scooting, mostly of me experimenting with different aspects of showing trips from place to place, or how I might view a certain feature of american road design.

I’ve happily and comfortably ridden scooters around countries where nearly every motor on a roadway is attached to a scooter, and the driving norms are sometimes generous. It can be so peaceful. traffic flows very smoothly and at an incredible number of vehicles-per-minute through relatively low-square-meter intersections, in terms of land allocated to the intersection.

Footnotes

I always wear a full-face helmet of the “modern” style. An integrated sun visor is ideal, along with a clear visor that can be completely closed.

If you see some of my scooting videos you’ll have a sense of it.

These kinds of helmets are the most protective, provide the most protection from wind and sunlight and noise.

I also wear a disposable painters mask. It’s a heavy-duty N95 “Odor respirator”. Scooting makes one quite aware of air quality issues. Obviously there is plenty of vehicle tailpipe emissions, but also tire rubber microplastics AND metal dust from brakes, so there’s a lot in the air I want to keep out of my lungs.

Here’s what the internet has to say about these kinds of masks.

An R95 Odor Respirator 8246 is essentially the same as an N95 respirator in terms of filtering 95% of airborne particles, but the key difference is that the R95 is designed to be somewhat resistant to oil-based particles, while a standard N95 is not, making the R95 a better choice for situations where oil mists or fumes might be present; both are NIOSH-approved respirators.

fumes? brake dust? airborn bits of rubber? I’ll wear the best mask I can, please, and I’ll wrap my whole head in a bubble (inside my full-face helmet with a closed visor) of mostly un-moving air, with all the vents closed up, unless the air is super clean.

Here’s the exact 10-pack I last got off of Amazon. Once you try a nice mask like this, it might be hard to go back. Once I started getting choosy about masks, I started wearing them a lot. I appreciate that to smell a vehicle or be downwind of it while it’s changing its speed via brakes and thus increased rolling friction on the ground, or its changing its speed via the engine, and thus increasing the particulate shedding rate and tailpipe emissions, is to be sorta ‘bathed’ in a plume of emisssions. ↩

Building an AI-Powered Startup Job Matcher

Ananth's Reflections — Wed, 18 Dec 2024 05:51:54 GMT

Through Agentic AI solutions to a handcrafted workflow

Lightweight Web Analytics with Liwan

henry's blog — Wed, 18 Dec 2024 02:00:00 GMT

This post explores some of this project's background and technical aspects. If you're more so interested in Liwan itself, you can check out the demo instance and the docs on liwan.dev. The source code available on GitHub under the AGPL-3.0 license.

This summer, I started working on a small tool for collecting various metrics on my websites and this blog to see what posts are popular and also just to have a better understanding of how people use my sites. It's grown into a very useful tool for me and I've recently released the first major version of it, so I thought I'd write a bit about it.

The Liwan Dashboard for one of my websites.

Over the last ~5 years, I've tried out probably ten different analytics platforms after bailing on Google Analytics due to privacy concerns, but nothing ticked all the boxes for me:

I wanted strong privacy guarantees. First of all, because it should be the default across all websites, but also because of how (rightfully) painful GDPR compliance can be when you collect too much data. I don't need to know your IP address or your browsing habits. There's no real reason for this data to leave my server; even when anonymized, there's always a risk of it being misused.

It should be set and forget. I love that self-contained, statically linked binaries are making a comeback due to being the default with "newer" languages such as Go and Rust, which is fantastic for self-hosted software. Around the time I started working on this project, DuckDB 1.0 was also announced and later released, which seemed like a perfect tool for this. I value a simple setup process more than the ability to hyper-scale prematurely and endlessly customize everything.

Truly Open Source: One of my favorite tools I tried was fathom analytics, which sadly (but understandably) switched to a completely closed source SaaS model. I want to 'own' the data, even if the user's data is eventually anonymized; it's hard to trust what's happening on a server I don't control. I really like plausible's open core model but I wanted something a bit more lightweight and opinionated.

Multi Website: As I mentioned, I want to aggregate data from 10+ different sites, so multi-website support also had to be great. One platform I experimented with early on was GoatCounter. While the UI is bare bones, it's also very lightweight and available with an embedded database. Sadly, it does not offer great multi-website support and is generally not as feature-rich as I would like.

Setting out with those and some more goals in mind, I started building Liwan.dev, which, as often happens, had a lot of feature creep but now ended up in a place I'm very happy with.

Simple Software

I love abstractions. Not in the OOP or the JavaScript way but in how a robust, tested, and, most importantly, focused library or app feels to use. This is one of the reasons I am so drawn to software engineering in the first place: the ability always to go one level deeper and understand how things work.

You can overplay this - see the NPM ecosystem (this is honestly a bit of a strawman, left-pad was not that bad), but it is a nice contrast from the attention-seeking everything apps and magic frameworks. There is a hard-to-find balance here - I won't argue in favor of building on hundreds of layers of magic and abstractions even when ignoring the supply chain concerns - but I really enjoy the current balance in the Rust ecosystem. The language itself doesn't matter, though; I mainly use it because I enjoy working with it.

Liwan is built on hundreds of open-source libraries; you can see the list for yourself on the attributions page shipped with every copy of Liwan. However, this doesn't mean it's not lightweight. Even when using the 'modern' web stack, resource-heavy and slow websites don't have to follow automatically. The dashboard is built using Astro, a web framework built around reducing unnecessary overhead.

A big part of the complexity of similar projects often comes from overly customizable Graph libraries. To reduce the amount of code that needs to be sent to the user, I build custom Graph and Map components directly on d3, keeping the entire JavaScript bundle shipped to users below 250kb. This even includes a large number of different icons and the data for the world map, which uses an heavily optimized topojson file I created with data from the natural earth project.

Liwan's PageSpeed Insights.

On the backend side, the main contributors to the amount of code are the embedded databases, DuckDB for events, and SQLite for the user data and authentication. The dashboard is transformed into static HTML, CSS, and JS and bundled with the rest of the code, something I've recently started doing with some of my other projects as well. Producing a single, universal artifact simplifies the entire build process and packaging containers greatly (Something that could be pushed even further using Actually Portable Executable), but I also decided to package it up as a Docker container as well in case you prefer that. This container just contains the Liwan binary and nothing else thanks to the static linking.

To maximize compatibility across different Linux distro (you might have had issues with glibc on alpine containers before if you've worked with docker imaged), Liwan is also statically linked with musl libc and uses rustls instead of linking against OpenSSL.

Simple software doesn't stop here, however. Liwan is also built to require only a minimal amount of configuration and settings from the user before it is ready to process events. You execute the binary, and Liwan is ready. Everything will be placed in the correct place according to XDG Base Directory Specification, and you can start sending events to it right away.

Early on, I also decided to add an onboarding page for users to create their initial user account. This page can only be accessed from a URL printed to the console on the first startup, protecting you from accidentally exposing this screen to the public internet and removing the need for default passwords.

Open Source

As a small side note on the open-source part, I've thought a lot about how I wanted to license this project. I ended up going with the AGPL-3.0, which I'm not 100% happy with but is the best compromise for now. My one big gripe with the AGPL-3.0 is the mostly undefined virality boundaries, which probably apply less in the EU (to remedy this a bit, the tracker script is also available under the MIT license). In other projects, I usually use the MIT + Apache 2.0 dual license that is so common in the Rust ecosystem. Still, I want flexibility to allow me to monetize Liwan more easily in the future. To have the possibility to relicense it later and not need to set up a CLA, all contributions also need to be provided under the MIT license as well, something I haven't seen in many other projects and seems like a good compromise for both sides.

The main alternative I considered was the EUPL. I've only seen it used on very few projects, but it's a bit more permissive than the AGPL. The author of GoatCounter has some interesting thoughts on why he chose it on his blog. I don't like the license's wording: It allows relicensing the work under entirely different terms, and it just feels like a mess to me. It tries to be an "Interoperable Copyleft" license and explicitly states it's compatible with the GPL 2 and 3, but the GPL itself is probably incompatible with the EUPL (see gnu.org), except if you do a weird two-step relicensing dance. But I've read in forums that it still keeps the terms of the EUPL, so relicensing doesn't actually weaken it? I don't know, it's just a mess.

Try it out!

Liwan is not perfect. There are many things I could optimize more, starting supporting huge datasets - going into the tens of millions of events can slow down things considerably - but it fits my personal use case (nearly) perfectly and probably a lot of others too.

Just grab the latest version and try it out for yourself:

# Download the latest release
curl -JLO 'https://github.com/explodingcamera/liwan/releases/latest/download/liwan-x86_64-unknown-linux-musl.tar.gz'

# Ensure the ~/.local/bin directory exists (You might want to add it to your PATH)
mkdir -p ~/.local/bin

# Extract the binary
tar -xzf liwan-x86_64-unknown-linux-musl.tar.gz -C ~/.local/bin liwan

# Make the binary executable
chmod +x ~/.local/bin/liwan

# Run the binary
liwan --help

Moana 2

Code & Visuals — Wed, 18 Dec 2024 02:00:00 GMT

This fall marked the release of Moana 2, Walt Disney Animation’s 63rd animated feature and the 10th feature film rendered entirely using Disney’s Hyperion Renderer. Moana 2 brings us back to the beautiful world of Moana, but this time on a larger adventure with a larger canoe, a crew to join our heroine, bigger songs, and greater stakes. The first Moana was at the time of its release one of the most beautiful animated films ever made, and Moana 2 lives up to that visual legacy with frames that match or often surpass what we did in the original movie. I got to join Moana 2 about two years ago and this film proved to be an incredibly interesting project!

While we’ve now used Disney’s Hyperion Renderer to make several sequels to previous Disney Animation films, Moana 2 is the first time we’ve used Hyperion to make a sequel to a previous film that also used Hyperion. From a technical perspective, the time between Moana and Moana 2 is filled with almost a decade of continual advancement in our rendering technology and in our wider production pipeline. At the time that we made the first Moana, Hyperion was only a few years old and we spent a lot of time on the first Moana fleshing out various still-underdeveloped features and systems in the renderer. Going into Moana 2, Hyperion is now an extremely mature, extremely feature rich, battle-tested production renderer with which we can make essentially anything we can imagine. Almost every single feature and system in Hyperion today has seen enormous advancement and improvement over what we had on the first Moana; many of these advancements were in fact driven by hard lessons that we learned on the first Moana! Compared with the first Moana, here’s a short, very incomplete laundry list of improvements made over the past decade that we were able to leverage on Moana 2:

Moana 2 uses a completely new water rendering system that represents an enormous leap in both render-time efficiency and easier artist workflows compared with what we used on the first Moana; more on this later in this post.
After the first Moana, we completely rewrote Hyperion’s previous volume rendering subsystem [Habel 2017] from scratch; the modern system is a state-of-the-art delta-tracking system that required us to make foundational research advancements in order to implement [Kutz et al. 2017, Huang et al. 2021].
Our traversal system was completely rewritten to better handle thread scalability and to incorporate a form of rebraiding to efficiently handle gigantic world-spanning geometry; this was inspired directly by problems we had rendering the huge ocean surfaces and huge islands in the first Moana [Burley et al. 2018].
On the original Moana, ray self-intersection with things like Maui’s feathers presented a major challenge; Moana 2 is the first film using our latest ray self-intersection prevention system that notably does away with any form of ray bias values.
We introduced a limited form of photon mapping on the first Moana that only worked between the sun and water surfaces [Burley et al. 2018].; Moana 2 uses an evolved version of our photon mapper that supports all of our light types, many or our standard lighting features, and even has advanced capabilities like a form of spectral dispersion.
We’ve made a number of advancements [Burley et al. 2017, Chiang et al. 2016, Chiang at al. 2019, Zeltner et al. 2022] to various elements of the Disney BSDF shading model.
Subsurface scattering on the first Moana was done using normalized diffusion; since then we’ve moved all subsurface scattering to use a state-of-the-art brute force path tracing approach [Chiang et al. 2016].
Eyes on the first Moana used our old ad-hoc eye shader; eyes on Moana 2 use our modern physically plausible eye shader that includes state-of-the-art iris caustics calculated using manifold next event estimation [Chiang & Burley 2018].
The emissive mesh importance sampling system that we implemented on the first Moana and our overall many-lights sampling system has seen many efficiency improvements [Li et al. 2024].
Hyperion has gained many more powerful features granting artists an enormous degree of artistic control both in the renderer and post-render in compositing [Burley 2019, Burley et al. 2024].
Since the first Moana, Hyperion’s subdivision/tessellation system has gained an advanced fractured mesh system that makes many of the huge-scale effects in the first Moana movie much easier for us to create today [Burley & Rodriguez 2022].
We’ve introduced path guiding into Hyperion to handle particularly difficult light transport cases [Müller et al. 2017, Müller 2019].
The original Moana used our somewhat ad-hoc first-generation denoiser, while Moana 2 uses our best-in-industry, Academy Award winning¹ second-generation deep learning denoiser jointly developed by Disney Research Studios, Disney Animation, Pixar, and ILM [Vogels et al. 2018, Dahlberg et al. 2019].
Even Hyperion’s internal architecture has changed enormously; Hyperion originally was famous for being a batched wavefront renderer, but this has evolved significantly since then and continues to evolve.

There are many many more changes to Hyperion that there simply isn’t room to list here. To give you some sense of how far Hyperion has evolved between Moana and Moana 2: the Hyperion used on Moana was internally versioned as Hyperion 3.x; the Hyperion used on Moana 2 is internally versioned as Hyperion 16.x, with each version number in between representing major changes. In addition to improvements in Hyperion, our rendering team has also been working for the past few years on a next-generation interactive lighting system that extensively leverages hardware GPU ray tracing; Moana 2 saw the widest deployment yet of this system. I can’t say much more on this topic yet, but we’ve started to publish bits and pieces of work from this project, such as how we’ve created a new realtime ray tracing GPU Ptex implementation [Lee et al. 2025].

Of course, there are also still parts of Hyperion that have more or less remained exactly the same as they were during the original Moana; these parts of the renderer have stood the test of time and proven to be reliable foundational pieces of the Disney Animation filmmaking process. A great example is the fur/hair shading model [Chiang et al. 2016] that was originally developed for Zootopia and used on human characters for the first time in Moana. Even though our hair simulation continues to advance with every movie [Kaur et al. 2025], it turns out that the Chiang fur/hair model has been so reliable thaat we haven’t really had to change it since, and in fact it has since become a de-facto standard across the entire graphics industry!

Outside of the rendering group, literally everything else about our entire studio production pipeline has changed as well; the first Moana was made mostly on proprietary internal data formats, while Moana 2 was made using the latest iteration [Zhuang 2025] of our cutting-edge modern USD pipeline [Miller et al. 2022, Vo et al. 2023, Li et al. 2024]. The modern USD pipeline has granted our pipeline many amazing new capabilities and far more flexibility, to the point where it became possible to move our entire lighting workflow to a new DCC [Endo et al. 2025, Joseph and Butt 2025] for Moana 2 without needing to blow up the entire pipeline. Our next-generation interactive lighting system is similarly made possible by our modern USD pipeline. The modern pipeline has also allowed us to continue to push the scale of our films ever larger, with the ever-growing complexity of our crowds [Ros and Berriz 2025, Ros and Sullivan 2025] being a particular standout.

While I get to work on every one of our feature films and get to do fun and interesting things every time, Moana 2 is the most direct and deep I’ve worked on one of our films probably since the original Moana. There are two specific projects I worked on for Moana 2 that I am particularly proud of: a completely new water rendering system that is part of Moana 2’s overall new water FX workflow, and the volume rendering work that was done for the storm battle in the movie’s third act.

On the original Moana, we had to develop a lot of custom water simulation and rendering technology because commercial tools at the time couldn’t quite handle what the movie required. On the simulation side, the original Moana required Disney Animation to invent new techniques such as the APIC (affine particle-in-cell) fluid simulation model [Jiang et al. 2015] and the FAB (fluxed animated boundary) method for integrating procedural and simulated water dynamics [Stomakhin and Selle 2017]. Disney Animation’s general philosophy towards R&D is that we will develop and invent new methods when needed, but will then aim to publish our work with the goal of allowing anything useful we invent to find its way into the wider graphics field and industry; a great outcome is when our publications are adopted by the commercial tools and packages that we build on top of. APIC and FAB were both published and have since become a part of the stock toolset in Houdini, which in turn allowed us to build more on top of Houdini’s built-in SOPs for Moana 2’s water FX workflow.

On the rendering side, the main challenge on the original Moana for rendering water was the need to combine water surfaces from many different sources (procedural, manually animated, and simulated) into a single seamless surface that could be rendered with proper refraction, internal volumetric effects, caustics, and so on. Our solution to combine different water surfaces on the original Moana was to convert all input water elements into signed distance fields, composite all of the signed distance fields together into a single world-spanning levelset, and then mesh that levelset into a triangle mesh for ray intersection [Palmer et al. 2017]. While this process produced great visual results, running this entire world-spanning levelset compositing and meshing operation at renderer startup for each frame proved to be completely untenable due to how slow it made interaction for artists, so an extensive system for pre-caching ocean surfaces overnight to disk had to be built out. All in all, the water rendering and caching system on the first Moana required a dedicated team of over half a dozen developers and TDs to maintain, and setting up the levelset compositing system correctly proved to be challenging for artists.

For Moana 2, we decided to revisit water rendering with the goal of coming up with something much easier for artists to use, much faster to render, and much easier to maintain by a smaller group of engineers and TDs. Over the course of about half a year, we completely replaced the old levelset compositing and meshing system with a new ray-intersection-time CSG system. Our new system requires almost zero work for artists to set up, requires zero preprocessing time before renderer startup and zero on-disk caching, renders with negligible impact on ray tracing speed, and required zero dedicated TDs and only part of my time as an engineer to support once primary development was completed. In addition to all of the above, the new system also allows for both better looking and more memory efficient water because the level of detail that water meshes have to exist at is no longer constrained by the resolution of a world-size meshed levelset, even with an adaptive levelset meshing. I think this was a great example where by revisiting a world that we already knew how to make, we were given an opportunity to reevaluate what we learned on Moana in order to come up with something better by every metric for Moana 2.

We knew that returning to the world of Moana was likely going to require a heavy lift from a volume rendering perspective. With a mind towards this, we worked closely with Disney Research|Studios in Zürich to implement next-generation volume path guiding techniques in Hyperion [Reichardt et al. 2025], which wound up not seeing wide deployment this time but nonetheless proved to be a fun and interesting project from which we learned a lot. We also realized that the third act’s storm battle was going to be incredibly challenging from both an FX and rendering perspective; creating the storm battle required FX to invent whole new techniques [Rice 2025]! My last few months on Moana 2 were spent helping get the storm battle sequences finished; one extremely unusual thing we wound up doing was providing custom builds of Hyperion with specific optimizations tailored to the specific requirements of the storm sequence, sometimes going as far as to provide specific builds and settings tailored on a per-shot basis. Normally this is something any production rendering team tries to avoid if possible, but one of the benefits of having our own in-house team and our own in-house renderer is that we are still able to do this when the need arises. From a personal perspective, being able to point at specific shots and say “I wrote code for that specific thing” is pretty neat!

From both a story and a technical perspective, Moana 2 is everything we loved from Moana brought back, plus a lot of fun, big, bold new stuff [Bhatawadekar et al. 2025]. Making Moana 2 both gave us new challenges to solve and allowed us to revisit and come up with better solutions to old challenges from Moana. I’m incredibly proud of the work that my teammates and I were able to do on Moana 2; I’m sure we’ll have a lot more to share at SIGGRAPH 2025, and in the meantime I strongly encourage you to see Moana 2 on the biggest screen you can find!

To give you a taste of how beautiful this film looks, here are some frames from Moana 2 from the Blu-ray, 100% created using Disney’s Hyperion Renderer by our amazing artists. These are presented in no particular order:

Here is the credits frame for the Hyperion team, along with several of the other teams that we work closely with to make rendering happen at Disney Animation. Specifically, the Lighting & Materials team develops our render translation pipeline and much of the artist-facing user interfaces in our lighting and shading tools, and the Interactive Visualization team is our sibling team that develops our in-house realtime rasterizer viewports.

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Sucheta Bhatawadekar, Behzad Mansoori-Dara, Adolph Lusinsky, and Rob Dressel. 2025. The Cinematography of Songs in Disney’s “Moana 2”. In ACM SIGGRAPH 2025 Talks. Article 64.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2017. Recent Advancements in Disney’s Hyperion Renderer. In ACM SIGGRAPH 2017 Course Notes: Path Tracing in Production Part 1. 26-34.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Brent Burley. 2019. On Histogram-Preserving Blending for Randomized Texture Tiling. Journal of Computer Graphics Techniques 8, 4 (Nov. 2019), 31-53.

Brent Burley and Francisco Rodriguez. 2022. Fracture-Aware Tessellation of Subdivision Surfaces. In ACM SIGGRAPH 2022 Talks. Article 10.

Brent Burley, Brian Green, and Daniel Teece. 2024. Dynamic Screen Space Textures for Coherent Stylization. In ACM SIGGRAPH 2024 Talks. Article 50.

Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum (Proc. of Eurographics) 35, 2 (May 2016), 275-283.

Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks. Article 49.

Matt Jen-Yuan Chiang and Brent Burley. 2018. Plausible Iris Caustics and Limbal Arc Rendering. In ACM SIGGRAPH 2018 Talks. Article 15.

Matt Jen-Yuan Chiang, Yining Karl Li, and Brent Burley. 2019. Taming the Shadow Terminator. In ACM SIGGRAPH 2019 Talks. Article 71.

Henrik Dahlberg, David Adler, and Jeremy Newlin. 2019. Machine-Learning Denoising in Feature Film Production. In ACM SIGGRAPH 2019 Talks. Article 21.

Colvin Kenji Endo, Norman Moses Joseph, Alex Nijmeh, and Todd Scopio. 2025. Prototype to Production: Building a Lighting Workflow in Houdini for Animation. In Proc. of Digital Production Symposium (DigiPro 2025). Article 3.

Ralf Habel. 2017. Volume Rendering in Hyperion. In ACM SIGGRAPH 2017 Course Notes: Production Volume Rendering. 91-96.

Wei-Feng Wayne Huang, Peter Kutz, Yining Karl Li, and Matt Jen-Yuan Chiang. 2021. Unbiased Emission and Scattering Importance Sampling for Heterogeneous Volumes. In ACM SIGGRAPH 2021 Talks. Article 3.

Chenfafu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. 2015. The Affine Particle-in-Cell Method. ACM Transactions on Graphics (Proc. of SIGGRAPH) 34, 4 (Aug. 2015), Article 51.

Norman Moses Joseph and Rehan Butt. 2025. The Design Opportunities of Moving to Houdini for Lighting with the world of Animation. In ACM SIGGRAPH 2025 Talks. Article 52.

Avneet Kaur, Hubert Leo, Nachiket Pujari, and Bret Bays. 2025. Choreography of Hair and Cloth in Disney’s “Moana 2”. In ACM SIGGRAPH 2025 Talks. Article 13.

Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and Decomposition Tracking for Rendering Heterogeneous Volumes. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4 (Aug. 2017), Article 111.

Mark S. Lee, Nathan Zeichner, and Yining Karl Li. 2025. A Texture Streaming Pipeline for Real-Time GPU Ray Tracing. In ACM SIGGRAPH 2025 Talks. Article 12.

Harmony M. Li, George Rieckenberg, Neelima Karanam, Emily Vo, and Kelsey Hurley. 2024. Optimizing Assets for Authoring and Consumption in USD. In ACM SIGGRAPH 2024 Talks. Article 30.

Yining Karl Li, Charlotte Zhu, Gregory Nichols, Peter Kutz, Wei-Feng Wayne Huang, David Adler, Brent Burley, and Daniel Teece. 2024. Cache Points for Production-Scale Occlusion-Aware Many-Lights Sampling and Volumetric Scattering. In Proc. of Digital Production Symposium (DigiPro 2024). Article 6.

Tad Miller, Harmony M. Li, Neelima Karanam, Nadim Sinno, and Todd Scopio. 2022. Making Encanto with USD: Rebuilding a Production Pipeline Working from Home. In ACM SIGGRAPH 2022 Talks. Article 12.

Thomas Müller. 2019. Practical Path Guiding in Production. In ACM SIGGRAPH 2019 Course Notes: Path Guiding in Production. 37-50.

Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical Path Guiding for Efficient Light-Transport Simulation. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 36, 4 (Jun. 2017), 91-100.

Sean Palmer, Jonathan Garcia, Sara Drakeley, Patrick Kelly, and Ralf Habel. 2017. The Ocean and Water Pipeline of Disney’s Moana. In ACM SIGGRAPH 2017 Talks. Article 29.

Lea Reichardt, Brian Green, Yining Karl Li, and Marco Manzi. 2025. Path Guiding Surfaces and Volumes in Disney’s Hyperion Renderer- A Case Study. In ACM SIGGRAPH 2025 Course Notes: Path Guiding in Production and Recent Advancements. 30-66.

Jacob Rice. 2025. Steerable Perlin Noise. In ACM SIGGRAPH 2025 Talks. Article 1.

Alberto J Luceño Ros and Cecilia Berriz. 2025. Creating the Mudskipper Pile in Disney’s “Moana 2”: A Slippery Problem Space. In ACM SIGGRAPH 2025 Talks. Article 63.

Alberto J Luceño Ros and Jeff Sullivan. 2025. The Art of Crowds Animation. In ACM SIGGRAPH 2025 Talks. Article 62.

Alexey Stomakhin and Andy Selle. 2017. Fluxed Animated Boundary Method. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4 (Aug. 2017), Article 68.

Emily Vo, George Rieckenberg, and Ernest Petti. 2023. Honing USD: Lessons Learned and Workflow Enhancements at Walt Disney Animation Studios. In ACM SIGGRAPH 2023 Talks. Article 13.

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics (Proc. of SIGGRAPH) 37, 4 (Aug. 2018), Article 124.

Tizian Zeltner, Brent Burley, and Matt Jen-Yuan Chiang. 2022. Practical Multiple-Scattering Sheen Using Linearly Transformed Cosines. In ACM SIGGRAPH 2022 Talks. Article 7.

Rikki Zhuang. 2025. Transitioning to an Explicit Dependency USD Asset Resolver. In ACM SIGGRAPH 2025 Course Notes: USD in Production. 73-106.

Footnotes

¹ Our deep learning denoiser technology is one of the 2025 Academy of Motion Picture Arts and Sciences Scientific and Engineering Award winners. keyboard_return

Sorry Spotify, I Wrapped It Myself

taylor.town — Wed, 18 Dec 2024 02:00:00 GMT

I meticulously document my album ratings (in plaintext).

AI Hasn't Helped Most Engineers (yet)

Oliver Gilan — Wed, 18 Dec 2024 02:00:00 GMT

Most of the productivity gains from AI and other DevTools of the past decade have not made their way to the engineers working on large, mature codebases. If we want to make our biggest companies more productive we need a better system of record for the engineering org.

swyx in 2024 End of Year wraps

swyx's site RSS Feed — Tue, 17 Dec 2024 20:03:35 GMT

i was involved in 3 end of year-ish recaps today:

An Evolved Universal Transformer Memory

Mia Heidenstedt — Tue, 17 Dec 2024 12:51:53 GMT

Best viewed on the original page, where extended functionality like the footnote helper is available.

I stumbled on HN over this very interesting article about a new kind of context memory system that, is able to remove information that is “unhelpful or redundant details”.

Thinking further, i think this would be super helpful for semantic search, that is currently not very performant due to the missing filters that extract importance. I have tried to counter this problem until now via summarization through small LLMs, but as one might guess turns out as not very precise and super expensive. There are other ideas one could post process text with LLMs but they are not very efficient either.

Paper https://arxiv.org/abs/2410.13166

TLDR by GPT-4o

(the article is very good, you might to prefer to read it over this TLDR, but here is it anyway)

Sakana AI introduces Neural Attention Memory Models (NAMMs), a novel memory system for transformers inspired by human selective memory. NAMMs optimize how transformers store and retrieve information, enabling them to “remember” important tokens and “forget” redundant ones, significantly improving efficiency and performance, particularly for long-context tasks.

Key Technical Highlights:

Evolutionary Optimization: NAMMs use evolutionary algorithms to train a neural classifier that decides which tokens to keep or discard, bypassing non-differentiable challenges.
Execution Steps:
- Convert attention sequences into spectrograms.
- Compress data using an exponential moving average (EMA).
- Use a classifier to score and selectively prune tokens.
Generalization:
- NAMMs can zero-shot transfer to other transformers (e.g., vision, reinforcement learning) without retraining.
- They adapt to tasks differently—retaining global information in early layers and focusing on local details in later ones.
Performance: Tested on LongBench, InfiniteBench, and their Japanese benchmark ChouBun, NAMMs reduce memory usage and outperform prior hand-designed memory strategies (H₂O, L₂).

NAMMs demonstrate cross-domain mastery and efficiency gains across diverse tasks and input modalities, paving the way for future research into learning transformers directly atop evolved memory systems.

Great Resources for Learning and Implementing Retrieval-Augmented Generation (RAG)

Scramblings — Tue, 17 Dec 2024 12:30:00 GMT

Few great and free resources to learn about RAG and its usage in Generative AI.

Install Go on Ubuntu 24.04

Swag Industries — Tue, 17 Dec 2024 11:08:39 GMT

Install Go on Ubuntu 24.04 natively using apt or snap.

Emotional Design Article by Donald Norman

Scramblings — Tue, 17 Dec 2024 05:00:00 GMT

Extracts from the Emotional design article

Pixi - Real World Usage

Londogard Blog — Tue, 17 Dec 2024 02:00:00 GMT

Managing dependencies and environments across multiple platforms can be a nightmare. That’s why I was thrilled to discover Pixi. I’ve previous talked about Pixi on LinkedIn/Twitter, but haven’t used it in any “serious” project until recently and so far it has worked exceptional!

Imagine a tool that combines the speed and efficiency of uv with the robust package management of mamba. That’s Pixi in a nutshell. Built from the expertise drawn from as the mamba creators and utilizing uv for PyPi dependenciess, Pixi offers a streamlined, powerful way to manage Python environments. Compared to mamba, pixi takes things one step further as their PyPi-dependencies are tested with conda on top of the additional tools brought by pixi, such as tasks.

Cherry on top? Pixi is lightning fast and enables multi-platform & multi-environment inside a single file where everything is synced together.

Multi-platform, multi-environment means that we can sync dependencies between osx-arm64, linux-64, CUDA, CPU, … - a standout feature!

Pixi Docker Builds

After solving your local environment in a easy yet producible manner the next step is to solve it for your cloud workloads - containerization.

Containerization is an important part of a developers toolkit in the modern world. To run cloud workloads it’s very common to deploy as a container, in Data Science this is for everything like Training, Inference and Data Pipelines.

With pixi it’s quite straight-forward and they provide ready-to-use images through the pixi-docker registry. There’s multiple base-images, including CUDA, to get started - it can’t be any simpler!

Pixi Sample Docker Builds

Simple starter:

docker pull ghcr.io/prefix-dev/pixi:latest

Find the different tags on Pixi Docker tags page.

Efficient Production build by using Docker Multi-Stage Build: prefix-docker/shell-hook.

Pixi Docker Build on AWS Sagemaker

Sagemaker can be quite challenging to work with. While deploying custom Docker builds is easiest using their own base image, this image is often bloated with unnecessary dependencies. Additionally, to run @remote jobs on AWS, you need to include a conda or mamba environment - something that pixi doesn’t inherently use.

So, how do we integrate Pixi with Sagemaker?

Here’s a workaround to make them play nicely together:

Include micromamba: Add micromamba (available on conda-forge) as a dependency in your pixi.toml. This will allow us to create a conda-like environment within our Pixi setup.
- In the future this could be done using a simple shell script, which is a planned improvement in my own projects.
Add micromamba to $PATH: Ensure that the micromamba executable installed by Pixi is added to your system’s $PATH. This will make it accessible to Sagemaker.
Set Environment Variables: Configure necessary environment variables like CONDA_PREFIX to point to the appropriate location where micromamba will manage your environment.

With these steps, you’re ready to run your Pixi-managed projects on Sagemaker!

In my experiments, this approach significantly reduced the size of my CUDA images from around 12 GB down to 4.5 GB - a massive improvement in terms of storage and deployment speed!

Pixi Multi-Platform/Environment

One of Pixi’s standout features is its seamless support for multi-platform and multi-environment projects. While I initially planned to delve deeper into this, prefix.dev recently published an excellent guide on the topic. I highly recommend checking out their documentation on combining different OS’s and environments (CPU, CUDA) with PyTorch for a comprehensive overview.

Some Personal Comments

Personally I find this part of pixi one of the biggest strengths, especially how easy it is to work with! To build a docker image you simply follow the basic example above, opting for --feature=cuda.
The part of keeping lock-files on everything, while allowing certain OS:es missing out on dependencies makes it very practical in real-world scenarios!

Pixi Build Slimmming

When containerizing your code, it’s crucial to keep builds slim. Here are a few tricks to help you minimize your Pixi-based Docker images:

Leverage .dockerignore: Create a .dockerignore file to exclude unnecessary files and directories (e.g., .git, __pycache__, tests) from your Docker build context.
Optimize Dependencies:
- Carefully consider each dependency and remove any that are not strictly required for production.
- Utilize multiple environments within your pixi.toml, e.g. prod and dev environments. This allows you to exclude dev-specific dependencies (test, lint, ..) from your production container.
Employ Multi-Stage Docker Builds: Multi-stage builds reduces the image size. Use a build stage to install dependencies and compile your application, and then copy only the necessary artifacts to a smaller, leaner final image. The pixi-docker project provides guidance on using multi-stage builds with shell-hook.

Pixi vs uv

While uv has gained significant traction in the Python community, I believe Pixi offers a more compelling solution for my specific needs, especially when it comes to complex, real-world projects.

Why?

tasks are awesome. They might not be perfect but they’re great to me!
Multi-platform and Multi-environment projects (personal opinion) somehow ends up easier in Pixi
- I really tried to embrace the uv approach as I appreciate it as more lightweight. But Pixi is somehow “smoother”.
Pixi has base-images with CUDA
- Both tools are easy to build from a raw base-image too, so it’s not a huge problem
Access to conda packages
- Some hate it, but I like getting pre-built binaries.
- It’s quite interesting to install shell tools via conda for container deployment.
Possible to work with other languages than Python

What is the one big `uv` pro?

UV’s Inline Script Dependencies

I think this feature is really cool, ~~but as pixi utilize uv you can use it in pixi too! ;)~~ but it’s quite easy to replicate in pixi as well (including with uv)! ;)

How to run in ‘inline script’ in Pixi

Simply call pixi exec uv run a.py. See the docs (cli/#exec) where you’re able to also run shell-scripts with a shebang. This will actually install uv in a temporary env, and then use that uv.

A bonus of exec is that if you instead use the “pixi-native” ~~--scope~~ --spec it supports conda too, e.g. pixi exec -s polars -s altair python to run a temporary python venv with polars & altair.

Edit: Added this callout 2025-03-07 and updated 2025-03-10 based on feedback from markusschlenker.
Edit 2: I want to add that you can also easily add uv as a global tool (like installing it), through pixi global install uv.

# /// script
# dependencies = [
#   "requests<3",
#   "rich",
# ]
# ///

import requests
from rich.pretty import pprint

resp = requests.get("https://peps.python.org/api/peps.json")
data = resp.json()
pprint([(k, v["title"]) for k, v in data.items()][:10])

Outro

If you’re a Python developer struggling with dependency management, environment inconsistencies, or cumbersome container builds, I urge you to give Pixi a try. It’s a powerful tool that has the potential to streamline your workflow and make you a happier developer. Pixi has certainly made a significant difference in mine!

Thanks for this time, Hampus Londögård

Getting started with automated quality assurance

Paperless — Tue, 17 Dec 2024 02:00:00 GMT

The bad news is that every discussion we’ve had about tabs vs spaces was a waste of time, and neither of us learned anything useful. The good news is that we won’t need to have that discussion ever again - there are tools which apply idiomatic indentation for pretty much any language under the sun. But the good news don’t stop there. That discussion about whether entries in a long lists should all be on a single line, broken into long lines, or broken into one item per line? There’s a tool for that. Newline at end of file? There’s a tool for that. How imports should be sorted and grouped? Detecting overly general catch statements, unused variables, or bad spelling? You guessed it. In this article I’ll go through why you might want to automate QA, and some practicalities of how to adopt automated QA tools, so that the team can concentrate on the actually important work.

B-CRATOS: Föredrag om hjärnstyrda handproteser med fingertoppskänsla, på UU Centrum för forskning om funktionshinder - Antekningar

Liza Shulyayeva — Mon, 16 Dec 2024 23:01:33 GMT

Idag deltog jag i en föreläsning vid Uppsala universitets Centrum för forskning om funktionshinder (CFF), om en revolutionerande typ av hjärnimplantat och protes som möjliggör tvåvägskommunikation mellan hjärnan och protesen.

Is GDP the North Star?

Aswath Krishnan — Mon, 16 Dec 2024 19:24:34 GMT

I was debating with friends about whether GDP should be the North Star for a country.

What is GDP and why it matters

GDP measures the total exchange of goods and services in a country, capturing the scale of economic activity and trade. It’s an effective shorthand for material prosperity. Historically, countries with high GDPs tend to offer better living standards, infrastructure, and public services. When economies grow, people generally gain access to better healthcare, education, and opportunities.

At a personal level, I align with Kanye’s philosophy: “Having money isn’t everything, but not having it is.” This captures the duality of wealth: it’s foundational for security and opportunity but insufficient as the ultimate life goal. Without it, basic survival becomes a struggle. But once those needs are met, the returns diminish, and other values, like social connections, health, or fulfillment, emerge.

The Limits of GDP

GDP has significant blind spots. It doesn’t measure inequality, environmental sustainability, or quality of life. A rising GDP could reflect booming industries, but it could also mask glaring inequities. For example, the U.S. has one of the highest GDPs globally, yet struggles with income disparity, a loneliness epidemic, a declining sense of community, and healthcare affordability. Is that truly prosperity?

Economist Simon Kuznets, the father of GDP, cautioned that it wasn’t designed to measure societal well-being. For instance, healthcare spending adds to GDP—whether it’s driven by cutting-edge innovation or a chronic disease epidemic. Similarly, natural disasters or environmental degradation can spike GDP due to reconstruction efforts, yet these don’t reflect progress.

The paradox of prosperity

More wealth often comes at the expense of what matters most. A high-GDP lifestyle may demand long hours, geographic mobility, or environmental exploitation. The paradox is that the relentless pursuit of GDP may diminish the very well-being it’s meant to enhance.

There’s also a cultural layer. The GDP race fosters a consumerist mindset: more goods, more spending, more growth. This doesn’t necessarily translate into happiness or fulfillment. Many of us recognize this at an individual level yet default to GDP when assessing national success. Why?

Examples of alternative priorities

Take Bhutan. This small Himalayan country introduced Gross National Happiness (GNH) as an alternative framework. GNH measures well-being through pillars like environmental sustainability, cultural preservation, and mental health. It’s a direct challenge to GDP as the primary barometer of success. However, Bhutan’s approach raises questions: Is it scalable in complex, diverse economies? Is happiness too subjective to measure consistently?

Or consider Scandinavian countries, often ranked high on happiness indexes. Their success isn’t purely about GDP; it’s also about policies that prioritize work-life balance, universal healthcare, and strong social safety nets.

A better North Star?

What if we reimagined success? Measuring progress through metrics like health spans, happiness, community, opportunity, carbon footprint, or even volunteerism might lead to rich societies in more than just economic terms.

GDP is a vital but incomplete metric. It captures one dimension of prosperity but ignores the complexity of human flourishing. Perhaps the real question isn’t whether GDP should be our North Star but how we expand our constellation of metrics to guide us toward more holistic progress.

2024 X-Mas Demo (Commodore PET)

mass:werk – Now Go Bang! — Mon, 16 Dec 2024 14:50:00 GMT

It's that time of the tear again...

Local-only gitignore

blog — Mon, 16 Dec 2024 12:46:53 GMT

Embedding Lua in sqleibniz with Rust

xnacly - blog — Mon, 16 Dec 2024 12:12:45 GMT

Improving the SQL DX one blog article at a time

IdentityServer In Docker Containers – Handle Logout (Part 4)

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Mon, 16 Dec 2024 10:13:24 GMT

In this final post in this series, we’ll now resolve logout challenges you might run into with IdentityServer, ensure proper sign-out redirects, and summarize the key takeaways from the series. Let’s complete the setup and finalize our IdentityServer configuration! This blog has been broken up into four separate posts: IdentityServer in Docker Containers: Adding Containers […]

The post IdentityServer In Docker Containers – Handle Logout (Part 4) appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

Core Values Wizard

Home on Lagomorph — Mon, 16 Dec 2024 09:00:00 GMT

This interactive wizard guides you through a structured introspective journey to discover your 5 most essential core values. Based on the classic values card sort exercise, it helps you explore 235+ values across 7 categories, then progressively refine your selections until you identify what matters most.

Your progress is automatically saved in your browser, so you can take your time and return whenever you’re ready.

Discover Your Core Values

Core values are the fundamental beliefs that guide your decisions and behavior. This exercise helps you identify the 5 values most essential to who you are.

Why I choose Microdata

All content on Seirdy’s Home — Mon, 16 Dec 2024 07:30:06 GMT

The four most popular ways to use RDF-based metadata on websites are RDFa-Core, RDFa-Lite, Microdata, and inline JSON-LD.

I can’t use RDFa-Lite because I need rel HTML attributes. rel silently upgrades RDFa-Lite to RDFa-Core, which parses differently. I doubt all parsers upgrade correctly; some will try to parse RDFa-Core as RDFa-Lite. Conformant RDFa parsers upgrade RDFa-Lite pages to RDFa-Core despite many authors only being familiar with RDFa-Lite. I suppose resources like Schema.org and Google’s documentation only documenting RDFa-Lite markup worsens the confusion. Update 2024-12-16: SarvenCapadisli has clarified on the Fediverse that this is the behavior of one faulty parser; rel only triggers an upgrade when used with an RDFa namespace. I may re-evaluate RDFa.

With RDFa split between two incompatible alternatives with a confusing upgrade mechanism, the alternatives are Microdata and JSON-LD. I use structured data extensively; JSON-LD would duplicate most of the page. Let’s use this relatively short article as an example. Exruct can convert the embedded Microdata into a massive JSON document featuring JSON-LD. Take a look at the JSON-LD and HTML side by side. Microdata attributes take a fraction of the footprint, encode the same information, and don’t require duplicating nearly the entire page.

Stevie Wonder in Los Angeles

The Digital Nomad — Mon, 16 Dec 2024 02:28:14 GMT

I was talking to my coworker and he mentioned that he was going to see Stevie Wonder.

Seeing Stevie Wonder was on my bucket list and this may be his last concert. So I decided to do an overnight trip.

The concert was pretty good. Too many guest singers, too much talking and my view on the floor wasn’t the greatest.

It was nice to meet my coworker in person.

I stayed at the AC Hotel in Downtown Los Angeles.

The view was great. The hotel was in a prime location and there were restaurants, bars and live music.

The hotel was right in front of the venue Crypto Arena

Query a list of dictionaries with DuckDB

Posts on rand[om] — Mon, 16 Dec 2024 02:00:00 GMT

How to query a list of Python dictionaries using DuckDB, without previously converting to another format.

We can pass the list of dictionaries to the query directly and use unnest to convert to a list of rows. We can also expand the struct to multiple columns.

Note: all the dictionaries must have the same keys.

# /// script
# requires-python = ">=3.12"
# dependencies = ["duckdb"]
# ///

import duckdb


DATA = [
 {"foo": 1, "bar": "some string", "baz": 1.23},
 {"foo": 2, "bar": "some other string", "baz": 2.34},
 {"foo": 3, "bar": "yet another string", "baz": 3.45},
]

# Select as struct
rel = duckdb.query("SELECT unnest($data) as x", params={"data": DATA})

# Expand struct to rows
rel = duckdb.query("SELECT x.* FROM (SELECT unnest($data) as x)", params={"data": DATA})

result = duckdb.query("SELECT * FROM rel WHERE foo > 1")

print(result.fetchall())
# [(2, 'some other string', 2.34), (3, 'yet another string', 3.45)]

Here’s a longer script, including timings, and different approaches. I’m using .fetchdf().to_markdown() to pretty-print the results:

Estimating projects sells them short (and that's okay)

ntietz.com blog - technically a blog — Mon, 16 Dec 2024 02:00:00 GMT

Recently, I read a blog post about doing software project estimates. It's a reasonable post with a reasonable method. But it does what all estimates do: it sells projects short.

I don't mean in the sense of underestimating a young promising project's potential, relegating it to an unfulfilling career pushing paperwork. I mean in the sense of selling short on the stock market.

What's a short sale?

Most of the time when you trade stocks, you buy a stock and you sell it later on. This is how we buy most things in life: we pay money to get them, then later we can use them or get rid of them. This works as we imagine it for stocks^[1].

Let's say you buy a share of a company for $10. Later, you sell it. In the absolute worst case, the stock is worth $0 when you get rid of it^[2], and you're out all the money you put in. You've lost 100% of your investment. But if it's worth $10 when you sell it, you've gotten your money back (less any fees), yay! And maybe it went up to $20, then you made a return of 100%. Or it went to $50, and you made a return of 400%. The amount of money you can make is technically unlimited.

Then you have what's called a short sale. In these, you sell a share you don't have yet. To do that, you borrow it from some other party and promise to give it back to them eventually (and compensate them for the loan, too). Your hope is that the value of the stock will go down over time, so you can buy back the stock you sold for less than you sold it for.

With short sales, how much you can make has a cap. If you sell short a share worth $10 today, then the most you can make is $10 if the stock goes all the way down to $0. But what happens if it goes up? If it goes up to $40, now you'll lose $30—three times your investment—when you close out your position.

Short sales are often pretty dangerous! There are ways to reduce the risk, but unlimited downside risk is a big deal. Even with hedges, there's a much bigger potential downside than when you buy a stock the normal way.

Estimating software projects

Now, what about software project estimates?

Let's pretend that you, dear reader, are an independent software developer. You do software development for clients, and they pay you a project-based fee rather than an hourly rate. A good fee is one that's low enough that it's worth it to your client to accept, but high enough that you'll make a profit on it. To reach what fee that is, you have to figure out how long a job will take. If your only cost is your own labor, then the minimum charge to make a profit is the hours the project will take multiplied by your hourly rate.

Let's say you estimate a project by breaking down the project into small tasks, then you estimate how long each of those will take (using your expert knowledge), and add it up. How close do you think you'll be for the total project time?

My guess is you'll be pretty far off, and that you're going to underestimate the total time you need. Part of this is from tasks you'll forget, and the rest is from getting your estimates wrong. That's expected—that's why they're estimates, after all. And the tasks you get done faster than expected won't cancel out the ones that take longer, but we'll get to that.

So how do we handle underestimating completion time?

Some people take a simple approach and pad the entire estimate. This is crude, but can be quite effective. You can take the entire estimate and multiply it by something (three? five?) to guess how long it will take when you have some tasks that are slow or some tasks you forgot, they're handled in that slush.

If it works for you, great, a lot of people use it. I don't love this approach, though. It feels a little bit like bending the truth, unless you're transparent about your padding. It would be nice if we had a way to figure out the real completion time from this estimate. And it would be even better if we could explain that to our clients.

Uncertainty and short selling

When you estimate a task, you're saying it will take some amount of time to complete. Let's say you estimate that a certain task can be done in 10 hours. If you're right about that, great! If you're wrong, and it's faster, then at most you're saving 10 hours from your total project timeline.

But if you're wrong and it takes longer... Then it could take 20 hours, or 30, or 40. Sometimes you can cut this out of scope, but sometimes this happens on a critical feature that you didn't expect (and sometimes scope gets added).

Just as with short selling, the maximum you can be wrong by is unlimited.

I worked on a project once where we got (to my recollection) within a few weeks of a correct estimate. The project was on track for about four months, with about six engineers on it. We eventually deviated from the plan because we got new information, so we changed course. But that was quite a feat, being within weeks on a four month chunk of the project.

We did that by taking the uncertainty of each estimate into account. Some tasks, we were quite certain, so we'd put 70-90% certainty, depending on our confidence. Others, we were very unsure of how long things would take, so we could get as low as 10%. Then, the question was: how do we use this uncertainty to get our aggregate estimate?

It depends on the distribution of values. The normal distribution, our familiar bell curve, is quite common—but it isn't our distribution here, since it can extend infinitely far in either direction. If I recall correctly, we used a Poisson distribution, and it looked something like this.

The upshot here is that given this sort of distribution, we capture the property of unlimited potential downside with limited upside. And this is really the case with estimates!

This is okay

So, knowing that we are short selling our projects, what do we do?

Honestly, probably, we just continue doing it—but with our eyes wide open.

If you do project-based pricing, you can use this knowledge to reduce or acknowledge the risk you're taking. And maybe you can use it to lock in better estimates, using historical data to figure out the actual distribution of tasks timing so you can build a nice statistical model.

On the other hand, if you (like me) are working in a team that's building products and doesn't do client work per se, this is still useful! It can help you with locking in more predictable timelines, though it's usually not worth the effort to actually calculate all of this out. The most useful thing I've found from this, though, is just communication: having this understanding of unexpected events is really helpful for explaining potential risks before starting on a project, and for when you do hit those road bumps.

If you do use uncertainty in your estimates, I'd love to hear about how you do it!

As long as we imagine it being dominated by algorithmic trading and high-frequency trading. We definitely should not imagine humans yelling "buy! buy!" on the trading floor, though many prefer that you keep romanticizing the industry with that image. ↩
This is what may happen if you exercise stock options in a startup, for example. Many end up being worth $0, so if you exercise your options, you are likely to lose your entire investment. ↩

Advent of Code... in SQL

ty-porter — Mon, 16 Dec 2024 02:00:00 GMT

This year at work there’s been a push to brush up on relational database fundamentals. I’m generally pretty good at that, so I wanted to put my skills to the test. My preferred way to learn (or upskill) a new language in the month of December?

Advent of Code!

I’ve done AoC for the last 5 iterations starting in 2021, spanning a few languages – starting in Ruby, then Python, then Golang last year (but I got busy and didn’t finish), to finally a mix of Python and SQL. I get really sucked into these puzzles, so I wanted a challenge. So, I fired up PostgreSQL and figured I’d take a stab at writing some of the solutions in SQL.

Repository Link

https://github.com/ty-porter/advent-of-code/tree/master/2024/sql

Methodology

I’m decent at writing SQL queries for business purposes but writing them to solve complicated problems like AoC would definitely prove to be quite difficult and potentially time-consuming. To save a little of my sanity, I chose to write the solutions out in Python before attempting them in SQL. Python for me is reasonably quick to prototype in, and crucially, solving it in Python gave me a foundation where I could check my work without fear of running into AoC’s wrong answer cooldown periods.

Once I had a working solution in Python, the SQL scripts could be created.

Meta-Tables, Functions, Batch Scripts

I created a few meta-scripts that did some basic stuff, like set up a solutions table for my answers and a raw_data table that would hold lines of the problem input. update_solution, a function to update a record in the solutions table.

-- common/setup.sql
DROP TABLE IF EXISTS raw_data;

CREATE TABLE raw_data (
        position SERIAL PRIMARY KEY
        , raw_data VARCHAR
);

-- There's a separate script to drop this, but it's not included here. It should persist between solutions.
CREATE TABLE IF NOT EXISTS solutions (
        day INT
        , part INT
        , result VARCHAR

        , PRIMARY KEY (day, part)
);

CREATE OR REPLACE FUNCTION update_solution(newday INT, newpart INT, newresult ANYELEMENT)
RETURNS VOID AS $$
BEGIN
        INSERT INTO solutions (day, part, result)
        VALUES (newday, newpart, newresult::VARCHAR)
        ON CONFLICT (day, part)
        DO UPDATE
        SET result = EXCLUDED.result::VARCHAR;
END;
$$ LANGUAGE plpgsql;

Since I’m on Windows, then I defined a batch file that would seed the raw_data table via psql and run my solution scripts. There’s a whole host of scripts I used here, but this was the most common one:

# scripts/run.bat
@echo off

set dayno=%1
set connection=postgresql://aoc:aoc@localhost:5432/aoc

psql -c "SET client_min_messages TO NOTICE;" %connection%
psql --quiet -f .\common\setup.sql %connection%
psql --quiet -c "COPY raw_data (raw_data) FROM 'C:\\Users\tyler\development\advent-of-code\2024-sql\solutions\%dayno%\prompt.txt' WITH (FORMAT text)" %connection%
psql --quiet -f .\solutions\%dayno%\solution.sql %connection%

Then I can run a script with scrips/run.bat <day>!

Solution Scripts

Now the fun can begin. Each solution script is designed to be idempotent – it builds up a schema that it will use for a solution, computes the result, writes it to the solutions table, and tears itself back down.

First I parsed the raw data rows into some sort of useful table. If this is straightforward enough, an INSERT INTO ... SELECT was the preferred approach, but for more complex data I needed to use a PL/pgSQL function (by default I named it process_raw_data). For instance, if I needed to parse my raw data rows into a 2D grid, I might do something like:

DROP TABLE IF EXISTS grid;

CREATE TABLE grid (
        id SERIAL PRIMARY KEY
        , x INT NOT NULL
        , y INT NOT NULL
        , c VARCHAR NOT NULL
);

CREATE OR REPLACE FUNCTION process_raw_data()
RETURNS VOID AS $$
DECLARE
        record RECORD;
        x INT := 0;
        y INT := 0;
        c VARCHAR;
        char_row VARCHAR[];
BEGIN
        FOR record IN SELECT * FROM raw_data ORDER BY position ASC
        LOOP
                char_row := string_to_array(record.raw_data, NULL);
                x := 0;

                FOREACH c IN ARRAY char_row
                LOOP
                        INSERT INTO grid (x, y, c) VALUES (x, y, c);

                        x := x + 1;
                END LOOP;

                y := y + 1;
        END LOOP;
END;
$$ LANGUAGE plpgsql;

SELECT process_raw_data();

I honestly overused these functions. They were really only useful for looping over things, but in some cases looking back I can see that they either a) were just regular UPDATE / INSERT queries within the functions or b) could be converted to them easily.

For the solution calculation itself, I liked using CTEs to break up the individual queries. It makes it clear what is an intermediate calculation vs. what the result is. Since there are two parts to a problem, I can potentially re-use those calculations with very little effort.

My last few CTEs were usually called part1/part2 and were simply a step to get the part/result column for the solutions table. After that, a UNION ALL on the two parts is readily inserted using my solution insert function.

WITH subcalculation1 AS (
  -- Some expensive calculation
)
, subcalculation2 AS (
  -- Some other calculation
)
, part1 AS (
        SELECT
                1 AS part
                , COUNT(*) AS result
        FROM subcalculation1
)
, part2 AS (
        SELECT
                2 AS part
                , COUNT(*) AS result
        FROM subcalculation2
)
SELECT
        update_solution(1, results.part, results.result)
FROM (
        SELECT * FROM part1
        UNION ALL
        SELECT * FROM part2
) AS results;

So How’d It Go?

Honestly, once you get used to it, SQL really isn’t that bad for doing AoC. I mean, it’s not the right tool for the job, but I expected it to be way worse. I thought it would be a certainty that I would need to stop around the end of the first week – at time of writing I’m at double that!

The Good

Leaning into set theory

Duh. That’s what SQL is made for.

There’s lots of instances of this but let’s take Day 7 as an example – we want to perform a list of operations on each row in a table. Well, we can do that, it’s a CROSS JOIN on the operands table that we can filter out.

WITH RECURSIVE bfs AS (
        SELECT
                eq.id
                , eq.tgt
                , eq.operands[1] AS current_val
                , eq.operands AS operands
                , 1 AS idx
                , array_length(eq.operands, 1) AS tgtidx
                , ARRAY[]::TEXT[] as path
        FROM equations eq

        UNION ALL

        SELECT
                b.id
                , b.tgt
                , CASE
                        WHEN op = '+' THEN b.current_val + b.operands[b.idx + 1]
                        WHEN op = '*' THEN b.current_val * b.operands[b.idx + 1]
                        WHEN op = '|' THEN CONCAT(b.current_val::TEXT, (b.operands[b.idx + 1])::TEXT)::NUMERIC
                END AS current_val
                , b.operands
                , b.idx + 1
                , b.tgtidx
                , path || op as path
        FROM bfs b
        CROSS JOIN unnest(ARRAY['+', '*', '|']) AS op
        WHERE b.idx < array_length(b.operands, 1)
)
-- Use the result of this calculation...

Storing results

Also duh.

This is kind of obvious but when you’re working a complex puzzle like this, you have lots of options at your fingertips for data storage. A database after all is intended for long-term storage. You can use this to your advantage by leveraging CTEs, materialized views, temp tables, and even permanent working tables.

In an imperative programming language, you would be reaching for something like a Python dict, list, or some other data structure to represent data. In Postgres, you really only have one option – a table (or a table-like structure). You really don’t have to think about it either, especially if using a CTE and can just pop a MATERIALIZED on there to “store” the results.

The good thing about storage like this is it’s easy to make multiple passes of intermediate calculations on the same working data.

The Bad

Text processing

Getting useful data out of the raw data table was a pain almost every time. It can be done, but transforming the data was pretty wonky.

Take a look at this from Day 9. It gets the job done, but is pretty messy. Most days relied on substr or some sort of regex function to parse the raw data rows. Other languages would process this in a more readable way.

Recursion / iteration

Both of these are pretty rough. You can do it, but it will not be enjoyable or particularly performant.

I skipped Day 6 Part 2 – the Part 1 solution I came up with iteratively created a path on a 2D grid. It ran in ~3s, but the Part 2 solution would have run that same code hundreds of times. It just wasn’t worth it.

I had some luck with recursively generating a set of values like in Day 8, as long as the list of values was very tightly bounded:

WITH RECURSIVE harmonics AS (
        SELECT
                *
        FROM antinodes

        UNION ALL

        SELECT
                x + dx AS x
                , y + dy AS y
                , dx
                , dy
        FROM harmonics h
        JOIN grid_bounds bounds
                ON bounds.min_x <= h.x AND bounds.min_y <= h.y
                AND bounds.max_x >= h.x AND bounds.max_y >= h.y
)
-- Do something with this calculation

I even got breadth-first searches working in a few cases (I overused BFS in my Python solutions, too):

WITH RECURSIVE bfs AS (
        SELECT
                id
                , x
                , y
                , n
                , x AS ox
                , y AS oy
        FROM grid
        WHERE n = 0

        UNION ALL

        SELECT
                g.id
                , g.x
                , g.y
                , g.n
                , b.ox
                , b.oy
        FROM bfs b
        CROSS JOIN directions d
        INNER JOIN grid g
                ON b.x + d.x = g.x
                AND b.y + d.y = g.y
                AND b.n + 1 = g.n
)
, part1 AS (
        SELECT
                1 AS part
                , COUNT(DISTINCT (x, y, ox, oy)) AS result
        FROM bfs
        WHERE n = 9
)
-- Write out the result

The Ugly

Also known as “how the hell did that even work??”.

Every time you need a piece of data, you must query for it

Queries are expensive. There’s a reason N + 1 is a huge problem for a database, as it racks up tons of IO / transit time rather than spending time finding things that actually matter. That means the programmer, especially a masochistic puzzle-solving programmer, is heavily incentivized to make as few queries in the hot loop as possible.

Take for example Day 12, which is about identifying contiguous regions of letters and calculating their area (trivial), perimeter (fairly easy), and in Part 2, number of sides (now we’re talkin’).

My solution needed all the neighbors for each cell. To do that in one query I aggregated each direction in order using existence cast to '1' or '0' as the element. That gives a length 8 string that can be cast to a BIT(8) – an 8 bit number representing a bitmap whether a neighbor is present at a specified position. Can this be done better? I don’t know at this point, and I wasted hours trying to optimize it.

WITH adjacent_cells AS (
        SELECT
                ord, x, y
        FROM (
                VALUES
                (1, xi    , yi - 1),  -- Up
                (2, xi + 1, yi    ),  -- Right
                (3, xi    , yi + 1),  -- Down
                (4, xi - 1, yi    ),  -- Left
                (5, xi + 1, yi - 1),  -- Up-Right
                (6, xi + 1, yi + 1),  -- Down-Right
                (7, xi - 1, yi + 1),  -- Down-Left
                (8, xi - 1, yi - 1)   -- Up-Left
        ) AS adjacent(ord, x, y)
)
, ordered_cells AS ( 
        SELECT
                ac.*
                , CASE
                        WHEN g.id IS NOT NULL THEN '1'
                        ELSE '0'
                END AS val
        FROM adjacent_cells ac
        LEFT JOIN grid g
                ON g.x = ac.x AND g.y = ac.y AND g.region_id = rid
        ORDER BY ac.ord ASC
)
SELECT
        string_agg(oc.val, '') AS bitmask
INTO
        neighbors -- A variable to be used later
FROM ordered_cells oc;

Recursion / iteration

I put this in the Bad section as it does work when you get everything right, the approach suits the data, etc.

When it goes wrong, oh boy are you out of luck.

First of all the syntax is really wonky, it requires a base case with a UNION ALL to the recursive portion:

WITH RECURSIVE subquery AS (
        -- Base case
        SELECT 1 AS i;

        UNION ALL

        -- Recursive case
        SELECT i + 1 FROM subquery
)
SELECT * FROM subquery
WHERE i < 10;

This is a trivial example, but it’s quite simple to write a case that will simply never stop recursing. I ran into this many times, and because it isn’t in a function it’s not possible to RAISE NOTICE and effectively debug log your way out of the problem. Recursion specifically is not very performant, so if you need to check a lot of iterations, you’re better off creating your own functions.

To go back to that Day 12 example, I needed to write a custom BFS to define the regions. A recursive CTE simply timed out no matter what I tried (and I’m still not 100% sure what went wrong).

CREATE OR REPLACE FUNCTION define_regions()
RETURNS VOID AS $$
DECLARE
        rid INT := 1;
        region_start INT;
        rows_updated INT;
BEGIN
        WHILE (SELECT count(*) FROM grid WHERE region_id IS NULL) > 0 LOOP
                SELECT min(id) INTO region_start FROM grid WHERE region_id IS NULL;

                UPDATE grid
                SET region_id = rid
                WHERE id = region_start;

                LOOP
                        WITH adjacent AS (
                                SELECT
                                        adj.id
                                FROM grid base
                                INNER JOIN grid adj
                                        ON base.c = adj.c
                                WHERE base.region_id = rid
                                        AND adj.region_id IS NULL
                                        AND base.c = adj.c
                                        AND (abs(base.x - adj.x) + abs(base.y - adj.y)) = 1
                        )
                        UPDATE grid
                        SET region_id = rid
                        WHERE id IN (SELECT a.id FROM adjacent a);

                        GET DIAGNOSTICS rows_updated = ROW_COUNT;
                        IF rows_updated = 0 THEN
                                EXIT;
                        END IF;
                END LOOP;

                rid := rid + 1;
        END LOOP;
END;
$$ LANGUAGE plpgsql;

SELECT define_regions();

Honestly, you should read this entire solution for Day 12. Quite frankly I am in disbelief that it works at all – this seems like something Postgres is wholly unsuitable for:

# Breaking this:
AAAA
BBCD
BBCC
EEEC

# Into this:
+-+-+-+-+
|A A A A|
+-+-+-+-+     +-+
              |D|
+-+-+   +-+   +-+
|B B|   |C|
+   +   + +-+
|B B|   |C C|
+-+-+   +-+ +
          |C|
+-+-+-+   +-+
|E E E|
+-+-+-+

# Where:
#   A - area: 4, perimeter: 10, sides: 4
#   B - area: 4, perimeter:  8, sides: 4
#   C - area: 4, perimeter: 10, sides: 8
#   D - area: 1, perimeter:  4, sides: 4
#   E - area: 4, perimeter:  8, sides: 4

The syntax

It’s ugly. I’m sorry. I know the math guys out there probably love it, but from a programming perspective it’s pretty clunky. Admittedly, puzzles make that worse – they are essentially adversarial inputs for what a database is trying to accomplish.

On the other hand, getting the rows updated in a function should not look like this:

GET DIAGNOSTICS rows_updated = ROW_COUNT;
IF rows_updated = 0 THEN
        EXIT;
END IF;

I shouldn’t have to write my own modulus function because Python uses a signed remainder definition for the % operator but Postgres uses a mathematical definition:

-- % and mod() return negative numbers for negative dividend / positive divisor and vice versa.
CREATE OR REPLACE FUNCTION modulus(dividend INT, divisor INT)
RETURNS INT AS $$
DECLARE
        result INT;
BEGIN
        divisor := abs(divisor);
        result := mod(dividend, divisor);

        IF result < 0 THEN
                result := result + divisor;
        END IF;

        RETURN result;
END;
$$ LANGUAGE plpgsql;

It’s weird that casting from BIT(8) to BIT(4) preserves the highest bits:

aoc=# SELECT '11110000'::BIT(8)::BIT(4);

 bit
------
 1111

I’m sure there’s more.

All in all, it was a lot of fun. As of time of writing (December 16), I made it to Day 14. I’m 2 days behind and not sure if I’ll make up the difference. Day 12 was a significant effort and Days 15 and 16 were of similar size.

Either way, I was surprised to make it this far and I actually did learn quite a lot.

That’s all for now. Thanks for reading!

Customer Care: Supermodel AI Agents

Dan Stroot — Mon, 16 Dec 2024 02:00:00 GMT

Customer care is undergoing a transformation unlike anything we've seen before. Today, artificial intelligence (AI) agents are proliferating across communication channels — handling real-time voice calls, managing text-based interactions such as chat, SMS, and email, and delivering a level of efficiency that traditional customer service teams often struggle to achieve. However, the horizon promises even more revolutionary possibilities.

This post Customer Care: Supermodel AI Agents first appeared on Dan Stroot's Blog

A color for every day

npilk // Notes — Sun, 15 Dec 2024 23:15:00 GMT

To make daily updates feel different from one day to the next, I generated a unique color combination for every day of the year.

badkeming: a site for kerning so bad it's keming

John Graham-Cumming's blog — Sun, 15 Dec 2024 18:16:23 GMT

I made another silly Tumblr (to go along with Movie Code and Low Background Steel). This times it's a Tumblr for all those font disasters and other typographical amusements. Welcome to the wonderful world of badkeming.com:

Submissions gladly accepted.

Nouveau site de cours

Blog d'Anri — Sun, 15 Dec 2024 14:12:00 GMT

Ça y est, j’ai terminé la réécriture de mon site pour partager mes notes de cours.

Jusqu’ici, j’utilisais l’application PineDocs¹. J’ai fait de nombreuses contributions au projet et je tiens à encore remercier xy2z² d’en avoir merge plus d’une.

Mais depuis un moment, je voulais changer. Déjà, PineDocs est écrit en PHP, un langage que je ne connais pas. Je voulais aussi que le site soit mieux intégré à mon nouveau site (l’actuel) au lieu qu’il soit séparé du reste. Le style ne me convenait pas plus que ça ; et c’était un beau défi pour moi.

Alors voilà, avec les briques que j’avais écrites pour la partie “blog”, quelques adaptations et beaucoup de corrections de bugs, j’ai enfin terminé. Ce n’est pas parfait, mais c’est suffisant.

Les problèmes sont là : la recherche est relativement lente (car faite en local), et la construction de l’arbre des fichiers est faite en local, alors plus le dossier partagé est gros, plus l’arbre l’est aussi.

Mais en même temps, enfin fini les problèmes entre le Markdown et les formules \LaTeX. Le back-end écrit en Rust est quand même plus rapide, surtout quand la page demandée est mise en cache (qui est côté serveur, contrairement à PineDocs qui le faisait côté client). Le style entre mon site “principal” et la partie “cours” est désormais harmonisé et je suis plutôt content du résultat. Enfin, le site de cours est disponible dans un thème clair !³

Aussi, je peux dorénavant imbriquer des fichiers Markdown entre eux. J’ai l’habitude d’avoir chaque créneau de cours séparé les uns des autres et d’avoir un “méga-fichier” qui cite tous mes autres fichiers de cours. Il est donc à présent proprement rendu et facilement imprimable grâce à une jolie feuille CSS qui adapte mes pages à l’impression.⁴ Ce que j’avais fait pour le reste de mon site aussi est utilisé pour la réécriture⁵, notamment mon “obfuscateur d’e-mail”⁶ qui est basique, mais c’est toujours mieux que rien – et si ça peut passer quelques bots, c’est ça de gagner.

J’ai commencé à penser à la réécriture le 16 avril 2023, la PR qui a initié le boulot a été ouverte le 31 octobre 2023 et a été merge le 1^er avril 2024, et c’est aujourd’hui que l’état de la réécriture est dans un assez bon état pour être utilisable.

En ce qui concerne mon ancien site de cours, je vais le laisser en place pendant un petit moment (peut-être jusqu’à la fin des fêtes de fin d’année) et je pointerais l’URL vers la nouvelle version le temps venu.

“A fast and lightweight site for viewing files.” ↩
Je pense qu’il avait move-on du projet, mais il a quand même pris le temps de vérifier ce que je faisais, un grand merci à toi ❤️. ↩
C’était particulièrement ironique parce que mon PC entier est souvent dans un thème clair. ↩
C’est important de pouvoir imprimer le cours, surtout pour les examens. ↩
On peut citer aussi le fait que mon site de cours est donc désormais disponible sur Tor… ↩
Rien de dingue, il remplace le @ par (at). Je veux éviter de faire des trucs trop fancy pour que les gens sans JS soient quand même dans la capacité de voir mon site. https://git.mylloon.fr/Anri/mylloon.fr/src/branch/main/static/js/mail_obfuscation.js ↩

The magic of Go: a tale of stones and connection

ᓚᘏᗢdostoynikov — Sun, 15 Dec 2024 04:56:00 GMT

Yesterday, I was walking around a quiet countryside neighborhood when I saw a sign for a "Go Club." For those who don’t know, Go is a strategic board game also called Baduk in Korean and Weiqi in Chinese.

I have always been fascinated by Go because of its elegant mechanics and philosophy. Excited, I immediately went inside to say hello.

Inside, four people were playing—two middle school students and two elderly men, both over 70. One of the elders, Naoki-san, who turned out to be the owner, left his game unfinished to greet me.

We introduced ourselves and had a brief chat. Naoki-san then invited me to play a round. Go has a handicap system where weaker players start with more stones to balance the game. Since his level is 8-dan—extremely advanced—I started with 9 stones. For reference, my level is around 9-kyu, which is quite amateur! It was clear I was going to lose, but the game became a way for us to get to know each other through our styles of play.

Our energy felt perfectly aligned, and it was such a fun experience. He also was looking really happy to have a foreigner-newcomer and I felt his happy energy and kindness all the time. Unfortunately, I had to leave the game unfinished to catch a train. Before I left, Naoki-san gifted me a beautiful Go set. I was shocked and overjoyed, especially since I had been looking for one recently.

His kindness left a deep impression on me. He even walked me all the way to the door with a big, cheerful smile saying that he will wait for me next week. His energy truly made my day.

Now, I know where I’ll be spending my weekends. This experience reminded me of the beauty of connecting through shared interests and kindness. I plan to visit the club regularly and hope to reach 1-dan within a year as I continue playing Go.

Why Multiple Competing Platforms Are Better Than One

RobKohr's Blog — Sun, 15 Dec 2024 02:00:00 GMT

Why Multiple Competing Platforms Are Better Than One

AWS re:Invent2024 Aurora 发布了啥 – DSQL 篇

做有积累的事情 — Sun, 15 Dec 2024 02:00:00 GMT

这个是前年AWS re:Invent 2022 的内容, 有兴趣可以看这个链接: Aurora re:Invent 2022

Weighty phrases

Evan Fields — Sun, 15 Dec 2024 02:00:00 GMT

I’m deeply fond of weighty phrases – little snippets with cultural gravitas and mystical overtones. Most of my favorites have biblical origin or are fragments of culturally foundational art, remixed memetically over the centuries until they become units of their own. A few of my favorites: Not by bread alone A thousand and one nights All men are created equal That the strong might not injure the weak Look upon my works, ye mighty, and despair One small step for man, one giant leap for mankind My brother’s keeper The truth will set you free A plague on both your houses Justice, justice you shall pursue

Automating ticket progression

Paperless — Sun, 15 Dec 2024 02:00:00 GMT

I forget to move tickets around the board when I’m supposed to. After working with 10+ ticketing systems for some 20 years, it’s still too alien to have to perform a manual step in a completely separate system at the same time as trying to concentrate on the development. Why not automate this process a bit?

Individual Career Plan

Bytepawn - Marton Trencseni — Sun, 15 Dec 2024 01:00:00 GMT

This post provides a practical Individual Career Plan template you can use to guide self-assessment, prioritize your professional development, and set meaningful goals that align with your vision for the future. By outlining clear objectives and action steps, you create a framework for continuous improvement and more informed career decisions. I use this same template for the data organization I lead, filled it out myself, and shared it with others to inspire them to do the same.

Novellfika på Konditori Fågelsången - Fox 8 av George Saunders

Liza Shulyayeva — Sat, 14 Dec 2024 23:01:33 GMT

Idag höll jag min tredje novellfika i Uppsala, och det var en särskild fika eftersom några vänner från Stockholm var med. Så vi hade både nya och gamla kompisar i diskussionen. Efter fikan gick några av oss hem till mig för lunch och lite mer prat.

Using LLMs to Explain Historical Code: FLOW-MATIC Investigation

ℤ→ℤ — Sat, 14 Dec 2024 22:18:37 GMT

The new generation of code assistance tools powered by Large Language Models (LLM)s may be useful in efficiently categorizing and translating historical software corpora. In this qualitative study, we examine LLM performance in explaining and translating FLOW-MATIC programs. We also test their reliability by injecting OCR-like errors into the text. Released in 1958, FLOW-MATIC targeted business applications and used a heavily English-inspired syntax that later influenced COBOL. We find that the models perform inconsistently for describing the key aspects of a program but fare better when translating business logic into SQL. LLMs appear robust against OCR-like errors in the source code, but we caution against relying on the current technology.

FLOW-MATIC Brochure

Motivation

Historical source code may contain insights into the development of algorithms, programming practices, and shed light into the computational needs and pursuits of organizations. However, this same source code is often poorly cataloged, unavailable in digital form (e.g. exists in the form of paper tape or punch cards), and will usually be written in archaic programming languages, thus frustrating efforts to understand and interpret it. In order for researchers to effectively “triage” piles of artifacts and obtain a limited understanding of the contents, researchers might use an LLM to help explain and translate the code.

Why FLOW-MATIC?

Developed by Dr. Grace Hopper and her team at Reminton Rand from around 1955 to 1958, FLOW-MATIC was an early language targeted at business applications. Internally, the language was called B-0 and experimented with an English-like syntax. Using approximately 33 verbs and coupled with an external data description language, programmers could implement sequential data processing logic (Sammet 1969).

We chose FLOW-MATIC for this study because:

The language is sufficiently obscure that there is unlikely to be any special-case logic for the language, and
The manual (Rand 1958) contains three highly documented programs that perform typical business logic, and
The English syntax may be generalizable to text mining of programming or process specifications.

Although FLOW-MATIC is an obscure and dead programming language (even Rosetta Code doesn’t include any examples), these programs and associated commentary may have been included in the AI’s background corpus. The manual is available online as a PDF and (Sammet 1969) reprints Problem 1. We have also seen snippets of the code incorporated into various online articles on the language and Grace Hopper’s history. However, we think it is safe to assume no engineering effort has been made to improve FLOW-MATIC related answers.

Methodology

We extracted three FLOW-MATIC programs from a PDF of (Rand 1958) and manually corrected translation errors to the best of our ability. Then, using the AI Assistant tool as part of IntelliJ, we sent select prompts to different AI models, attaching the FLOW-MATIC programs as part of the request.

For the corruption studies, we ran the original source code through our corrupt.py script. This tool randomly selects a corruption from a list and applies it to the text. If the corruption does not apply (i.e. produced no edits), then a different corruption was selected until the number of edits matches the desired number.

Source code for the programs, the corrupted sources, the AI model output, and our corruption script are available within the Gitlab repository.

Testing LLMs is tricky. LLMs are stochastic and highly sensitive to variations in prompts. As an apologetically qualitative study, our task specification is vague and we lack clear, rigorous performance metrics.

Explaining and Translating Historical Code

For this task, we asked the LLM to produce an English summary of a program. We are less interested in a breakdown of individual lines, but rather succinct statements to the key aspects of the business logic, the data model, and pre and post conditions of the input and output.

Problem 1 - Join Inventory and Prices

The first program described in (Rand 1958) joins an ordered sequence of inventory records with an ordered sequence of price records, yielding a file containing priced inventory and a file containing unpriced inventory. The program requires 18 steps and would have been accompanied by a series of packets describing the input and output data formats.

(0) INPUT INVENTORY FILE-A PRICE FILE-B ; OUTPUT PRICED-INV FILE-C UNPRICED-INV FILE-D ; HSP D .

(1) COMPARE PRODUCT-NO (A) WITH PRODUCT-NO (B) ; IF GREATER GO TO OPERATION 10 ; IF EQUAL GO TO OPERATION 5 ; OTHERWISE GO TO OPERATION 2 .

(2) TRANSFER A TO D .

(3) WRITE-ITEM D .

(4) JUMP TO OPERATION 8 .

(5) TRANSFER A TO C .

(6) MOVE UNIT-PRICE (B) TO UNIT-PRICE (C) .

(7) WRITE-ITEM C .

(8) READ-ITEM A ; IF END OF DATA GO TO OPERATION 14 .

(9) JUMP TO OPERATION 1 .

(10) READ-ITEM B ; IF END OF DATA GO TO OPERATION 12 .

(11) JUMP TO OPERATION 1 .

(12) SET OPERATION 9 TO GO TO OPERATION 2 .

(13) JUMP TO OPERATION 2 .

(14) TEST PRODUCT-NO (B) AGAINST ZZZZZZZZZZZZ ; IF EQUAL GO TO OPERATION 16 ; OTHERWISE GO TO OPERATION 15 .

(15) REWIND B .

(16) CLOSE-OUT FILES C , D .

(17) STOP . (END)

Notable aspects of this program:

There are two input files, inventory and prices. The output consists of a file with joined inventory data and pricing data and a secondary output of inventory lacking a price.
Product Numbers (PRODUCT-NO) and Unit Price (UNIT-PRICE) are the only two fields referenced within the code.
Input files are assumed to be sorted. If this assumption is violated, the unpriced output will be full of false positives.
Each product in this version will have exactly one inventory record. This constraint is relaxed in later programs.
The program uses self-modifying code as step 9 is overwritten in step 12.
HSP D refers to the high-speed printer present on the UNIVAC. The instruction “redirects” output to the printer rather than a tape.

LLM Summaries

Using the prompt Explain the code at the current caret, all three LLM models produced a detailed explanation covering each line of code. All three models referenced ‘ZZZZZZZZZZZZ’ as a special marker. The final summaries from each model (below) all invoke matching entries between an inventory and pricing file or data and that prices may be missing.

openai-gpt-4o: “This sequence executes a comparison loop to match items between an inventory and a pricing file, writing matched items with prices to one output file and unmatched items to another. It handles end-of-file conditions elegantly, ensuring all items are processed properly.”
google-chat-gemini-pro-1.5: “In essence, this code implements a matching algorithm, similar to a merge join in database systems. It efficiently processes inventory and pricing data, handling scenarios where prices might be missing for certain items.”
openai-chat-gpt: “Overall, the code snippet describes a simple matching process between inventory items and their prices, handling cases where prices may be missing in the price file.”

Translation into SQL

We used the prompt Translate the code at the current caret into SQL to translate Problem 1 into SQL. A good translation should capture the business intent, that of a join, as well as features of the data model.

We observed:

openai-gpt-4o: The output features two inserts based on joins (implicit inner join and left join) into a priced and unpriced tables. The output includes a ‘quantity’ field which is not present in the original code, although an inventory data record is highly likely to include such a field.
google-chat-gemini-pro-1.5: The output uses procedural SQL features and attempts to match the original code feature-by-feature, which requires more lines than the original FLOW-MATIC version. The only fields referenced are the product number and the unit price. Includes an ‘operation_9_target’ for emulating the self-modifying code, but the variable is only written, never read. The code also includes a useless block for checking for the ‘ZZZZZZZZZZZZ’ end of data marker. This version also declares product numbers as VARCHAR(255) and prices as DECIMAL(10,2) for which there is no a priori support.
google-chat-gemini-flash-1.5: Similar to the openai-gpt-4o output, except the inner join is explicit and the code does not include a ‘quantity’ field. Of the three outputs, this is the cleanest version.

Translation may require the creation of entities not present in the original, such as table names, and changes to formatting, such as dashes to underscores, to be more idiomatic. However, ideally, insertion of content that relies on background knowledge rather than the source itself should be annotated.

Problem 2 - Treat Duplicate Inventory as Errors

Problem 2 extends Problem 1 by allowing duplicate inventory records. If a duplicate record is found, the record is sent to an error file. We performed some limited testing using Problem 2, but found it provided limited additional insight versus Problem 1 and Problem 3, so are ignoring it in the interests of space.

Problem 3 - Join Inventory at Multiple Sites with Prices

Problem 3 extends Problem 1 by allowing duplicate inventory records. Unlike in Problem 2, duplicate inventory records are meaningful and quantities are summed across records. (Conceptually, this represents inventory at multiple sites and we are calculating the global quantity.) Problem 3 also adds an ‘extended price,’ to the output. Within the manual, the use of X-1 is to demonstrate how a program may call out to non-FLOW-MATIC code even though the logic (multiplying the quantity by the price) could have been accomplished within the standard code.

(0) INPUT INVENTORY FILE-A PRICE FILE-B ; OUTPUT PRICED-INV FILE-C UNPRICED-INV FILE-D ; HSP D .

(1) COMPARE PRODUCT-NO (A) WITH PRODUCT-NO (B) ; IF GREATER GO TO OPERATION 21 ; IF EQUAL GO TO OPERATION 5 ; OTHERWISE GO TO OPERATION 2 .

(2) TRANSFER A TO D .

(3) SET OPERATION 13 TO GO TO OPERATION 18 .

(4) JUMP TO OPERATION 8 .

(5) TRANSFER A TO C .

(6) MOVE UNIT-PRICE (B) TO UNIT-PRICE (C) .

(7) SET OPERATION 13 TO GO TO OPERATION 14 .

(8) MOVE PRODUCT-NO (A) TO PRODUCT-NO (W) ; QUANTITY (A) TO QUANTITY (W) .

(9) READ-ITEM A ; IF END OF DATA GO TO OPERATION 23 .

(10) COMPARE PRODUCT-NO (A) WITH PRODUCT-NO (W) ; IF EQUAL GO TO OPERATION 11 ; OTHERWISE GO TO OPERATION 13 .

(11) X-1 ADD QUANTITY (A) TO STORED QUANTITY (W) .

(12) JUMP TO OPERATION 9 .

(13) JUMP TO OPERATION 14 .

(14) MOVE QUANTITY (W) TO QUANTITY (C) .

(15) X-1 COMPUTE EXTENDED PRICE AND INSERT IN C ITEM

(16) WRITE-ITEM C ,

(17) JUMP TO OPERATION 1 .

(18) MOVE QUANTITY (W) TO QUANTITY (D)

(19) WRITE-ITEM D .

(20) JUMP TO OPERATION 17 .

(21) READ-ITEM B ; IF END OF DATA GO TO OPERATION 1 .

(22) JUMP TO OPERATION 1 .

(23) EXECUTE OPERATION 13 THROUGH OPERATION 17 .

(24) TEST PRODUCT-NO (8) AGAINST ZZZZZZZZZZZZ ; IF EQUAL GO TO OPERATION 26 ; OTHERWISE GO TO OPERATION 25 .

(25) REWIND B .

(26) CLOSE-OUT FILES C , D .

(27) STOP . (END)

For reference to the reader, the first X-1 code (label 11) is defined as:

B0W001
A-A001
C0W001

These instructions perform, using relative addresses:

Load the temporary value (W, offset 1) into register
Add the value in file A (offset 1) to the register
Copy the accumulated value to the temporary value (W, offset 1)

The second X-1 code (label 15) is defined as:

L0C001
P0C002
J0C003

Load into the L register from file C at offset 1
Multiply (product) the L register and the contents from file C at offset 2, storing the result in A and X registers
Store the result into file C (offset 3)

The layout of the data and the definitions of the extended functions are located in ancillary files, rather than the source code itself. Thus, unless the LLM cheated by reading the manual, “intuiting” the computation of an extended price is based on background knowledge of the likely implementation.

LLM Summaries

Similar to Problem 1, each model produced a detailed explanation covering each line. However, unlike Problem 1, their summaries of Problem 3 lack detail. Two of the explanations (openai-gpt-4o and google-chat-gemini-pro-1.5) simply describe the program as doing something with inventory and pricing, lacking any mention of aggregating data between inventory records or how joins are implemented. The third mentions the aggregation of inventory but drops references to pricing data.

openai-gpt-4o: “The code provided appears to be using a structured way to process inventory and price files.”
google-chat-gemini-pro-1.5: “This code snippet represents a pseudocode or a simplified representation of a program logic, likely for processing inventory and pricing data.”
google-chat-gemini-flash-1.5: “Operations 8-12 form a loop that reads inventory items, and aggregates quantities for each unique product number by storing the running total quantity in temporary storage (W). When a different product number is encountered, the loop breaks, and the aggregated quantity will be processed and written to the appropriate output file. This technique efficiently combines quantities for the same product before writing to the output file, avoiding redundant entries.”

None of the models provide the same quality of summary as they did for Problem 1.

Translation into SQL

Similar to the task in Problem 1, each model was asked to translate the code into SQL. Our observations:

openai-gpt-4o: Accumulates inventory across multiple inventory records. Treats extended price as the product of inventory and unit price. Uses inner joins and left joins to differentiate the presence of pricing data. Uses common table expressions.
google-chat-gemini-pro-1.5: Very similar to openai-gpt-4o.
google-chat-gemini-flash-1.5: Very similar to both except it does not use common table expressions.

While the different models generate different names for tables and vary slightly in their use of SQL, they all use the same joins and model the data in the same way. We consider the output to be high-quality.

Robustness to Source Code Corruption

An LLM is not a compiler. While a compiler is expected to enforce the rules of a language and enforce consistency within the source code, an LLM matches the source code to a probabilistic model. Since historical source code is unlikely to come in a pristine digital form but rather a decayed backup or print-out, resistance to random corruptions is a beneficial quality. Furthermore, if we desire to triage or quickly check the contents of an archive, we do not need the power of a compiler.

To simulate corruption of the source code, we created copies of the originals with a number of edits applied. The nature of the edit was based on our experiences correcting the programs from the PDF. Thus, the errors simulate optical character recognition errors rather than bit flips. The script chose edits randomly from:

Changing zero (0) characters to O characters
Change 1 to I
Change 8 to B
Change ; to :
Remove a space (e.g. A B to AB)
Add a space (e.g. AB to A B)

As the number of edits increased, eventually all the characters were swapped and the code was simply adding and removing spaces in random locations. The number of edits varied from 81 (approximately 10% of the total characters), to 243, and 486 (approximately 50%).

For this exercise, we focused on a single AI model, openai-gpt-4o, and a single original source program, Problem 1. We chose to focus on openai-gpt-4o for convenience.

LLM Summaries

When asked to explain three corruptions of Problem 1, each with 81 edits, the model’s summaries stated the input files were inventory and pricing data in each case. In only one case does the summary state the output has priced and unpriced categories. In the other two cases, the summary merely describes that sequential or batch processing is in play.

In contrast, this model provided the priced and unpriced insight for the original source code, as did the other two AI models. Similar to performance from Problem 1 to Problem 3, corruptions or other slight increases in functionality led to a drop in the quality of output.

Translation into SQL

Across six versions of Problem 1, three with 243 corruptions and three with 486 corruptions, the model produced the same SQL output consistently:

-- Step 1: Identify and insert unpriced inventory items
INSERT INTO unpriced_inventory (product_no, other_columns)
SELECT i.product_no, i.other_columns
FROM inventory i
LEFT JOIN price p ON i.product_no = p.product_no
WHERE p.product_no IS NULL;

-- Step 2: Identify and insert priced inventory items
INSERT INTO priced_inventory (product_no, unit_price, other_columns)
SELECT i.product_no, p.unit_price, i.other_columns
FROM inventory i
INNER JOIN price p ON i.product_no = p.product_no;

If we contrast this to the model’s output with the original file:

-- SQL for Priced Inventory (FILE C)
INSERT INTO PRICED_INV_TABLE_C (product_no, quantity, unit_price)
SELECT a.product_no, a.quantity, b.unit_price
FROM INVENTORY_TABLE_A a
JOIN PRICE_TABLE_B b ON a.product_no = b.product_no;

-- SQL for Unpriced Inventory (FILE D)
INSERT INTO UNPRICED_INV_TABLE_D (product_no, quantity)
SELECT a.product_no, a.quantity
FROM INVENTORY_TABLE_A a
LEFT JOIN PRICE_TABLE_B b ON a.product_no = b.product_no
WHERE b.product_no IS NULL;

Apart from the changes in table names and order of outputs, the output from the corrupted sources includes a other_columns column, while the original has a quantity column. Neither quantity nor other_columns exists in the original FLOW-MATIC source code. As a clue to the developer that the desired record may include additional fields, other_columns is more explicit than quantity, although quantity is a believable column for an inventory table.

Using Problem 3 as the base, in six cases with the same variations of edits the model also produces consistent output. This output includes an other_columns field but otherwise does not add additional columns. This version lacks Problem 3’s handling of multiple inventory records, thus marking a regression in functionality.

Conclusion

The LLM models appear to be unreliable as a tool to quickly summarize programs or translate them into other forms while retaining key aspects of the original logic. While their output can be highly consistent in the presence of random errors, their ability to capture and communicate key aspects of the programs can quickly decay to uselessness.

Thus, researchers should be wary about using LLMs as a tool for researching historical source code. Technologically, improvements in models and workflows might render LLMs a reliable tool.

We believe future quantitative studies should include:

Measures of quality for the output, including key aspects included, contextual aspects not referenced but included, and aspects added without utility
Measures of the narrative signal to noise ratio
Rendering to a “canonical” form to simplify bulk analysis

References

FLOW-MATIC Brochure Courtesy of Computer History Museum.

(Rand 1958) Remington Rand Univac. 1958. UNIVAC FLOW-MATIC Programming System. archive.org

(Sammet 1969) Sammet, Jean E. 1969. Programming Languages: History and Fundamentals. Englewood Cliffs, N.J.: Prentice-Hall. archive.org

Three Nine

Sunil Shenoy — Sat, 14 Dec 2024 21:25:00 GMT

Birthdays seem to come by much quicker than before. Three nine. Another year closer to the next milestone birthday. I wondered if I would still be as excited about birthdays, and I’m happy to report that the excitement continues.

While looking for last year’s post on the blog, I didn’t have to scroll far—it’s clear I haven’t been writing much this year. I always question why I don’t write here more, knowing that I enjoy the entire process.

Sitting down to write always brings a huge smile to my face. It gives me the chance to go through my calendar and reflect on everything I’ve been up to this year. When I read this post next year, I’ll get to relive these moments all over again.

A few years ago, I decided to stop creating long to-do lists for the year and instead focus on themes. The idea was to do more of the things I enjoy. I’m happy to say that this year, I managed to stick to all the themes I set out for myself.

Create

This year, I built and launched two new applications, and I’m happy with the work I’ve done. There are two more applications in progress, and I hope to share more about them soon. Overall, I’m pleased with how the “create” theme shaped up this year.

I spent time relearning Ruby on Rails, Swift, and SwiftUI. On top of that, I improved my skills in React Native, Node.js, and debugging software.

Travel

A few days after my birthday last year, I went to New Zealand. As I write this, I’ve booked another trip to visit on January 3rd. I can’t get enough of that beautiful country. As much as I love Australia, New Zealand has a special place in my heart.

Here’s where I traveled to this year:

New Zealand
Thailand
Uluru
Gold Coast
Melbourne
India
Poland
Berlin
The Hague, Netherlands

It was wonderful catching up with friends and family, some of whom I hadn’t seen in nine years. On the way back, I promised myself to visit more places where friends have moved.

Fitness

December, January, and the first half of February were amazing for my fitness—I was the fittest I’ve ever been. I even posted a picture of myself from a trip on Instagram, and it’s now one of my favorites. Whenever someone asks if I’ve ever had abs, I proudly show them that picture.

However, I also experienced my most unfit period this year. After returning from my trip, I stopped working out for a few months and gained a lot of weight. The see-saw of weight gain and loss is something I’ve come to terms with.

On a positive note, I’ve been back on track for the past three months, and I’m now as fit as I was in the early part of the year. With my cousin’s wedding in December next year, I’ve told her I’ll try to be as ripped as possible for the event. I’ve got one year to stay consistent and aim for that goal.

This year reminded me that fitness is a journey, not a destination, and I’m proud of my progress.

Love

Another unsuccessful year in this department. A heartbreak aside, there’s nothing significant to report.

Here’s to seeing what 2025 has in store for this chapter of my life.

Overall

2024 has been a really good year. My finances are back on track, I’m enjoying the work I’m doing, I love where I’m living, and I feel quite optimistic about the future. I can’t wait to share more about the software I’ve been working on over the last three months.

Themes for 2025

Travel: I’ve booked a few tickets.
Fitness: Prepare for my cousin’s wedding in December.
Love: Let’s see how this goes.
Launch/Enhance: After spending 2024 creating, it’s time to launch, listen to customer feedback, and enhance.

Why South Korea’s Victory over the North will Prove to be Fleeting

Stories by Marc Reagan on Medium — Sat, 14 Dec 2024 10:08:36 GMT

How many times have you been scrolling on social media and seen the following image:

Free market capitalists love to cite the Korean example for why capitalism has triumphed over communism. Korea has proved to be a unique test case because the people are ethnically homogenous and the country was arbitrarily divided down the middle 71 years ago when the Korean war ended. It is very hard to run social experiments on a countrywide level but sometimes fate delivers the goods. East/West Germany and China vs. Taiwan are a few other notable examples.

On every objective metric imaginable South Korea appears to be victorious. GDP per capita is around 23x higher: $36,000 vs $1,600. The average lifespan in the south is 83 years vs. 72 years in the north. The infant mortality rate per 1000 births is 3 in the south whereas it is 15 in the north. The list goes on: infrastructure, automobile ownership, internet access, medical care, working conditions, human rights, diplomatic relations, and cultural impacts on the rest of the world. People all over the world are listening to K-pop and watching Squid Game on Netflix.

As a rabid free market capitalist myself I have long cited this example as to why capitalism is superior to communism. But my declaration of victory was premature. If North Korea can continue to simply trudge along in a few generations they can conquer the South without firing a shot.

The average fertility rate in the North is 1.8 births per woman while in the South it is .7 births. A fertility rate of just over two is needed to maintain current population levels. This is because each woman needs one child to replace herself and one to replace a man. Hopefully our hypothetical couple is in a committed relationship, but at this point I’m not going to be choosy. If current birth rates persist in 4 generations the population of South Korea will decline by 94%. This extinction level event is occurring right now in slow motion.

Perhaps superior technology and weapons systems will allow the south to maintain their independence for a while. Back when the sun never set on the British empire a Sudanese army was threatening the highly lucrative British colony in Egypt. For the good of the empire the British decided they needed to do something about these pesky natives.

In 1898 at the battle of Omdurman 25,000 combined British and Egyptian forces fought 52,000 native Sudanese. The British had machine guns, artillery pieces, and breech loading rifles. The Sudanese forces meanwhile had rudimentary muskets, spears, and swords.

The battle was a triumph of western technology and tactics over crude native methods. The Sudanese forces suffered 28,000 casualties: 10,000 dead, 13,000 wounded, and 5,000 prisoners. The British suffered a few hundred casualties and 40 deaths.

Hilaire Belloc later so succinctly quipped: “Whatever happens, we have got the Maxim gun, and they have not.” The Maxim gun was the name for the British machine gun.

If the British had been outnumbered 100 to 1 at this battle I believe the outcome would have been different. Technology and tactics can go far in ensuring victory even when you are vastly outnumbered. But as Napoleon so famously said: “Quantity has a quality all it’s own.” Technology and tactics can only get you so far.

South Korea will find itself in this situation in a few short generations. North Korea has nukes while the South does not. The North invests heavily in their military. I have no doubt most of this money is squandered but when you outnumber the enemy 100 to 1 I’m betting on the Kim dynasty to prevail.

If we do not find a solution to collapsing birth rates in developed, democratic, and capitalist nations we will ultimately lose the fight. I had such great hope for enlightenment ideas and ways of living. The last few hundred years have been a momentous time for the freedom and rights of the individual. But if we cannot sustain our numbers in peacetime then something about our system is rotten to the core.

My fellow free market economists: we fought the good fight, we had a good run, it has been an honor serving beside you. But perhaps the rugged individualism that I so vociferously espouse has always been doomed to fail. Humans are social animals after all. Perhaps Marx was right all along.

OnlineOrNot Diaries 23

Max Rozen — Sat, 14 Dec 2024 09:10:00 GMT

Working with big systems all day can slow you down.

What's New in Emacs: Last Decade Edition

Ashton Wiersdorf on Lambda Land — Sat, 14 Dec 2024 02:00:00 GMT

Emacs has come a long way in the past decade. This is meant as a guide to anyone who’s been using stock or near-stock Emacs for some years and wants a quick update on the new shiny stuff that comes bundled with Emacs.

This guide assumes you are running Emacs 29, which was released in 2023.

Completion #

Completion is pervasive in Emacs: hit TAB whenever you’re selecting a file (C-x C-f) or running a command by name (M-x) and the like, and completion kicks in. Emacs’ built-in completion framework and interface have gotten a huge upgrade in recent years.

If you hit TAB a bunch of times when writing e.g. a file name, you’ll open up the *Completions* buffer. Emacs 29 has lots of ways to configure this buffer to be much more useful than the default:

(setopt enable-recursive-minibuffers t)
(setopt completion-auto-help 'always)
(setopt completions-max-height 20)
(setopt completions-format 'one-column)
(setopt completion-auto-select 'second-tab)

Here’s what each one does:

enable-recursive-minibuffers: You can interrupt doing something in the minibuffer with another minibuffer-using operation.
completion-auto-help: The value 'always means always show the *Completions* buffer when trying to complete; other options include nil and 'lazy.
completions-max-height: Controls how many lines high the *Completions* buffer should be.
completions-format: Put all completions in one column. I like this formatting better.
completion-auto-select: Controls when to jump to the *Completions* buffer automatically. The 'second-tab option is nice: the first TAB opens the *Completions* buffer, and if you want to select something from the list, you just hit TAB again.

That’s for minibuffer completion. Emacs also supports completion for whatever the cursor is on; Emacs calls this “completion-at-point”. Here’s how to get nice tab-complete behavior in Emacs:

(setopt tab-always-indent 'complete)
(setopt completion-styles '(basic initials substring))

Setting tab-always-indent to 'complete means that when you hit TAB, Emacs will first try to indent the current line. If the line is already indented, then Emacs will call the completion-at-point facilities of Emacs. Assuming you have the minibuffer completions set up as explained above, you can use the same interface to complete in the minibuffer as well as tab-completion that you see in other editors.

These are just the built-in completion mechanisms. Emacs’ framework has gotten a great upgrade that has allowed packages like Vertico (minibuffer completion) and Corfu (completion-at-point in a popup window) to flourish. Vertico and Corfu are two of my favorite packages.

The completion-styles is neat: it takes a list of different “styles” that can be used when filtering candidates. The initials is particularly fun: type M-x dtw TAB and you should see the function delete-trailing-whitespace in the *Completions* buffer.

Editing code #

Smart completion, jump-to-definition, etc. #

Emacs 29 added the Eglot (“Emacs polyGLOT”) package to the core distribution: Eglot is a lightweight Language Server Protocol (LSP) client. This lets Emacs talk to language servers like clangd or rust-analyzer (etc.) to get things like good code completion, jump-to-definition, documentation, etc.

Eglot is conservative: it doesn’t get in your way, and all the completion smarts come up only when you ask for them. Of course, it’s possible to make it more eager and behave a little more like VS Code, but that’s your choice. I personally use jump-to-definition more than any other feature.

If you’re working on—say—a Rust project, install rust-analyzer on your system, then open a file in a Rust project and run M-x eglot. Now all of the completion-at-point mechanisms should be smart about the language you’re working with. You should also be able to jump to e.g. a function definition by putting your cursor on a function and invoking xref-find-definitions (bound to M-. by default.)

Projects #

Emacs now has project.el package that adds some nice stuff for navigating projects. For example, the project-find-file command is like find-file but is restricted to files in the current project. This pairs nicely with fancier completing-read interfaces to let you find and jump to files without having to navigate the entire file hierarchy.

Tree-sitter #

Tree-sitter is a parser generator tool that is fast and—most importantly—works on incomplete inputs (e.g. a file you’re in the middle of editing). Emacs 29 added support for tree-sitter enabled modes. To be honest, it’s a little clunky still, but support is improving. (The Emacs 30 pre-release is already better than what 29 does.) Tree-sitter is too big for me to cover here; see Mickey Petersen’s excellent article on the subject.

Editing prose #

There’s a built-in dictionary feature in Emacs 28. Use this to look up words from https://dict.org:

(setopt dictionary-server "dict.org")

Use M-x dictionary-lookup-definition to look up the word at point. You can of course bind this to a key for more convenience. (Looks like there’s also dictionary-tooltip-mode which shows the definition of a word when you hover over it with your mouse. Fancy, and a bit too much for me. But very cool!)

Going really far back is flyspell-mode: spellchecking while-you-type. Nothing special about that; what is special is how easy it is to correct words: hit C-; to cycle through corrections of the closest misspelled word. For code, there’s flyspell-prog-mode, which does spellchecking in just comments and strings. Even though this has existed for a while, I don’t see it turned on very often.

I now use Jinx for my spellchecking needs. It’s faster and more flexible, but it draws a lot of inspiration from flyspell-mode.

This mode is also a bit older, but I love visual-line-mode: turning this on will soft-wrap lines at word boundaries and make all your motion commands behave according to visual rather than logical lines: e.g. if I have a really long line that is wider than my window, it will be wrapped—just like in any word processor—and pressing the arrow keys will move me to the character that is visually above the current one, even though it might be on the same line.

General improvements #

New themes #

Emacs comes with two new themes: modus-operandi (light) and modus-vivendi (dark). These themes have excellent contrast and are ment to conform to the highest levels of visual accessibility. Try them with M-x load-theme RET modus-vivendi.

Package manager and configuration #

Emacs has a built-in package manager called package.el. This makes it easy to install 3rd-party packages. Run M-x list-packages to activate.

I have a list of my favorite Emacs packages, which include (but are not limited to) Magit, Vertico, and Avy.

There are several places you can get packages from; when package.el came out it only supported packages from GNU ELPA. Recently Non-GNU ELPA got added to the stock list which opens up a huge set of packages such as the venerable Magit package. And of course, there’s MELPA, which you have to add to your config yourself if you want packages from there:

(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)

Emacs 28 added the awesome use-package macro, which makes configuring packages really nice. Previously you might have written something like this to configure a package:

(require 'citar)
(define-key global-map (kbd "C-c C-i") 'citar-insert-citation)
(define-key global-map (kbd "C-c C-d") 'citar-dwim)
(add-hook 'org-mode-hook 'citar-capf-setup)
(add-hook 'LaTeX-mode 'citar-capf-setup)
(setq citar-bibliography  '("~/Research/refs.bib"))

Now you can do this with a macro:

(use-package citar
  :bind (("C-c C-i" . citar-insert-citation)
         ("C-c C-d" . citar-dwim))
  :hook ((org-mode LaTeX-mode) . citar-capf-setup)
  :custom (citar-bibliography '("~/Research/refs.bib")))

What you don’t see is all the extra work that use-package does behind the scenes to speed up startup times by lazily loading your package. There are lots of options for setting up hooks, different kinds of configuration, keymaps, etc.

Performance and utilities #

Emacs 27 added a native JSON parsing library. This meant that things like Eglot (and other LSP clients) got much snappier. Emacs 28 can compile Emacs Lisp code to native byte code. This means that all Emacs packages just run faster.

There’s a lot more, but those are some of the biggest ones. Nice to see Emacs getting a lot of love.

Ecosystem #

Many of the improvements to the core have also lead to a flourishing of excellent third-party packages. Here are some of my favorite:

Magit: The only Git porcelain worth using. I was once a Git command-line purist. Then I found Magit and became a convert: unlike other Git porcelains, Magit hides none of what Git can do. Moreover, it makes complicated operations (e.g. staging or reverting individual lines of a file) easy and exposes you to more of what Git can do.
Vertico: “Vertical completion”: wraps the standard completing-read interface with a nice interactive version. Thanks to the improvements to the completing-read API, all Emacs features that use completing-read just work.
Consult: Enhancements for a bunch of Emacs’ UI: for example, consult-buffer replaces the standard switch-to-buffer with a version that shows you a live preview. Pairs nicely with Vertico.
Corfu: Enhanced completion-at-point interface: get a fancy popup window to show completion candidates when completing a word in-buffer. Corfu uses a child frame by default; for non-GUI users there’s corfu-terminal to make it work in TUI mode as well.
Citar: Citations made easy: Citar reads .bib databases and provides a slick completing-read interface to insert citation keys.

There are so many more packages that have come out in the past few years that I use all day every day. I wrote about some of them here.

A starter kit for this #

I made Emacs Bedrock as a starter kit to help explore all these nice new built-in features of Emacs. It’s a true starter kit: you download it once, then tweak it to your liking. By default it installs only one package (which-key; Emacs 30 will have this package built-in) and the rest is tweaking some settings to be what I think the defaults should be. It’s meant to encourage exploration. If you liked something from this post, take a look at Bedrock and maybe you’ll find something else there that you like!

My deleted review of this restaurant

Alejandro AR (kinduff) — Fri, 13 Dec 2024 23:41:00 GMT

Guess my review didn't pass the community rules of Google Maps

Honolulu Hawaii: Well that’s off of Our Bucket List

The Digital Nomad — Fri, 13 Dec 2024 20:48:26 GMT

We flew to Honolulu Hawaii last month. We stayed at the Hilton Hawaiian Village Waikiki Beach Resort.

It was overly touristy, a lot of shops and it just felt old. There was a small pool and it wasn’t like we were expecting. We were expecting a more Caribbean feel or at least for it to feel like our own home in Florida.

Traffic was horrible and getting around felt like driving in Atlanta.

The hotel was at least free for us on points.

We went to a food tour and it was the worse one we have done yet.

The only saving grace was the luaha. Where the food was decent and they did put on a good show

We were so over it that we ended up cancelling one of our plans.

We had been talking about going to Hawaii since 2018, we were really disappointed.

Maybe we went to the wrong island or it might have been our life is so much different now than then and we have done so much more that our expectations are different.

How to make your GitHub profile dynamic

Patrick D'appollonio — Fri, 13 Dec 2024 07:56:53 GMT

Optimizing the information you show in your GitHub profile with dynamic PRs and blog content

Recently I was toying with the somewhat new GitHub feature that allows you to create a repository with the same name as your GitHub username and host in it a README.md file that then is displayed in your profile and visible to whoever visits your profile page. While this feature is good, it's not too dynamic, but there are two things in our favour: a) GitHub Actions are free for public repositories and b) we can use the GitHub API to fetch information about our repositories, pull requests and more. So I thought "why not make my GitHub profile more dynamic by adding my latest blog posts, my latest pull requests and my latest starred repositories to my profile?" and that's what I did, in fact, you can see it here! Here's how you can leverage the same workflow to achieve something very...

Tinker Postman Collector Scribe

Parth Parikh's Blog — Fri, 13 Dec 2024 02:00:00 GMT

In this blog post, we will explore art, USPS postal technology, philately, and generative art — all inspired by my trip to D.C. last month.

Cron jobs inside your web app

Posts on rand[om] — Fri, 13 Dec 2024 02:00:00 GMT

When you have a web application, almost always, you need to run some functions/scripts regularly. This what cron is for. I wanted to find an easy way to run “cron-like” jobs inside a FastAPI app. The app runs inside docker, and I just need those jobs to run as long as the app is also running.

Since the app is “async”, I found I can just have recurring background tasks as part of the app. We need to use the app lifespan and some functions that run a task, the asyncio.sleep for a pre-defined time. I decided to apply some random jitter to avoid multiple tasks starting at once. (Note: this can also be done in “sync” apps using threads).

TIL: Zizmor and GitHub Actions security

Erethon's Corner — Fri, 13 Dec 2024 02:00:00 GMT

I was reading up on the recent Ultralytics GitHub Action compromise and I stumbled upon this great analysis of the situation. In it, zizmor is introduced, which is a static analysis tool for GitHub Actions. I experimented with it a bit and I have to say it's working great. It correctly identified misconfigured GitHub Actions on some repositories I was working on. It's another tool that's worth having as part of your CI.

Luciatåget på Uppsala slott

Liza Shulyayeva — Thu, 12 Dec 2024 23:02:33 GMT

Tidigare i kväll gick jag till Uppsala slott och träffade några kompisar för att se Luciatåget på slottets tak. Det var en kort föreställning, men så vackert! Den sista sången som de sjöng var helt magi.

Personhood-based arguments on resuscitation of Extremely Premature Infants - practical philosophy seminar notes

Liza Shulyayeva — Thu, 12 Dec 2024 23:01:33 GMT

This week’s practical philosophy seminar discussed personhood-based arguments for the resuscitation of extremely premature infants (EPIs). These notes focus less on discussion/paper specifics and more on what I learned about the topic as a whole during the discussion and afterward.

Green Goo (Republished)

De Novo — Thu, 12 Dec 2024 21:07:00 GMT

Synthetic biology, mirror life, and threats to global ecosystems

Humba II: Doubling Down on Deep Tech

Coding VC — Thu, 12 Dec 2024 18:08:32 GMT

The Humba Ventures team is proud to announce the launch of Humba II, our oversubscribed $40M second fund focused on deep tech and American Dynamism.

An Academic's Guide to Fribourg

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Thu, 12 Dec 2024 16:00:00 GMT

As my research group starts to take root in Fribourg, I finally get to take the time to note some things I like about the region and the university. Much of this is inspired by Matt Might, whose article on getting tenure ends very poignantly:

And come to the University of Utah! Life is good here.

Without further ado, let me sing the praises of Fribourg and its environs:

Fribourg is a literal university town, with more than a quarter of its inhabitants being students. In some sense, the city is the campus. Plus, it is a small town and highly walkable.
The canton (i.e. the administrative unit, akin to a county) of Fribourg is bilingual, with most inhabitants speaking German and French as a native language. Since the city is so international, everyone of course also speaks English.
While Switzerland has higher costs of living than most countries, Fribourg (both the city and the canton) offers affordable accommodations and in general has lower costs of living.
The whole region is well-connected, with major cities like Geneva or Bern just a short train ride away. As one of my colleagues so nicely put it: ‘Switzerland is so small that almost all cities are suburbs of each other.’
Switzerland in general sports cities with a high quality of life. Whether you like the cosmopolitan nightlife or the solitude of mountains, there is something for you.

In terms of the university as such, we offer a large variety of subjects, ranging from the humanities to the life sciences. While we are not as big as our sibling universities ETH Zurich or EPFL, we are doing very well. For machine learning (including deep learning) and information retrieval/data mining, we are ranked 3rd in Switzerland and in the top 50 of all European universities.¹

Given our small size (just compare our faculty sizes to that of other universities), that is quite a feat, in my opinion! Thus, I can only echo Matt’s statements:

Come to the University of Fribourg. Life is good here.

When viewing these rankings, you might have to switch the region of comparison manually, since CSRankings unfortunately has the tendency to default to your current location. ↩︎

My take on Web Components

Posts on rand[om] — Thu, 12 Dec 2024 02:00:00 GMT

Lately, I’ve been trying to learn more JavaScript.

I wanted to explore web components. After checking out some frameworks and tools, mainly lit, tonic and facet, I wanted to build my own thing.

It ended up being heavily inspired by tonic, but way simpler and with less features. I also really liked lit-html, which you can use as a separate package without having to use lit. It solves the problem of re-rendering only the UI parts that have changed, and it’s awesome at doing it.

Learning to Type Fast

Thu, 12 Dec 2024 02:00:00 GMT

Typing is the conduit by which thoughts flow via the keyboard to the computer. Doing this quickly can massively improve not only your productivity but your flow. Remember when you had a phone that looked like this?

Remember how slow it was? It took ages to write something - 'wic iz y we rote lyk this'.

If you work as a software engineer, you'll be typing all day - whether programming or more predominantly, emails and slack messages.

I found that I was typing reasonably fast (70WPM) but often found myself being unable to keep up with my thoughts as I typed. And I made a LOT of mistakes. The reason? I typed with my index fingers only. I was super quick with it but I knew that I would never be able to write faster unless I retaught myself to write with all 10 digits.

My first scope was to decide what my baseline was - a test that I could use to track my typing speed over the course of this learning experience. For me this was the standard 1 minute typing test on 10fastfingers.

Next, I researched a tool that could re-teach me how to type. I found typingclub to be an incredible tool for this.

The build up

Typingclub starts by building you up what fingers you use. Initially it just focuses on the index finger and builds the muscle memory for what keys it should type. It keeps things simple by having everything lower case and focusing just on the position of your fingers first before typing anything. Once I had reached a stage where I could use 3 fingers I started to apply this in my job. My typing was biblically slow but I was building the muscle memory.

A new keyboard

This has nothing to do with typing speed per se (more on that after), but a new keyboard would affect my typing experience. Previously, I was using a Logitech K380. A fantastic keyboard with multi-bluetooth and plenty of action keys. For £30, it's a steal and I've owned two of them over the years (after the first one met it's unfortunate end with a "e" key being totally broken). But I found the typing experience quite flat. A mechanical keyboard was the obvious answer. But this posed a problem. When I was a teenager, I used to play copious amounts of starcraft 2 - with a mechanical keyboard. Soon, my left forearm swelled with fluid and I eventually had to get carpal tunnel surgery to correct the problem. Wanting to avoid all that pain again (literally), I was a bit hesitant about picking up a new one. Fortunately, mechanical keyboards have come a long way. And one of those new developments is a "low profile" mechanical keyboard. They are a slimmed down version of mechanical keyboards that are less raised so don't strain your hands as much.

After a great deal of searching culminating in an excel spreadsheet comparison, I landed on the Nuphy Air75v2 with Wisteria switches. These switches are most akin to cherry brown switches and have a nice tactile bump on each keystroke. Crucially, they have multi-bluetooth to allow switching between my work and personal machine.

The new keyboard actually reduced my typing speed by a lot. Because I kept accidentally hitting keys or not pressing the correct one entirely. It was worth the persistence though because the keyboard massively improved my enjoyment of typing. The satisfying "thonk" with each keypress echoing like a sturdy industrial-era engine.

More keys

Last up was my pinky finger. I found this difficult to practise because "non-pinky" way of typing was so heavily ingrained. With practise and being conscious about it, I managed it. Although, I'll confess that I'm far from perfect in this regard. I kept grinding on these lessons daily - usually doing 5 a day. They soon moved onto more complex topics such as punctuation, numbers and macros. I'm continuing these lessons now but by around lesson 300 I had a fairly good typing speed.

Once I had grasped using all 10 fingers, I moved on to measuring my progress and practising more common phrases. For this I used, 10fastfingers.com and the top 250 words test. I applied the 80/20 rule in this regard. My thinking was that if I could type the most used 250 words then this would account for 80% of my typing.

Progress

My highest words per minute (as measured on 10fastfingers) is 81WPM and my average is staying well above where I started so I'm count that as a success!

If you work on a computer daily, I'd strongly recommend improving your typing speed as your fluency of transmitting information will increase dramatically.

Tim Ferriss explains how identifying “super tasks” in his to-do list helps plan his day

Relentless Simplicity — Thu, 12 Dec 2024 01:14:03 GMT

Julbord! Jag började äta fisk igen i precis rätt tid.

Liza Shulyayeva — Wed, 11 Dec 2024 23:01:33 GMT

Igår gick vi på vårt team-julbord! Jag har varit vegetarian i många, många år. Jag hade inte ätit fisk på ungefär… 10 år kanske, innan tidigare i år. Så detta var mitt första julbord (som jag minns) där jag åt fisk, och det var så väldigt gott! Men jag åt för mycket av det och vaknade upp med ont i magen. Värt det.

Reviewing A Deepness in the Sky by Vernor Vinge

Davis Haupt's Blog — Wed, 11 Dec 2024 19:00:00 GMT

This was one of the most creative and enjoyable sci-fi books I’d read in a while. The two main storylines are split between spider-like intelligent aliens on a remote planet and the conflict between humans in orbit who are preparing to make First Contact.

The way that the alien storyline is “translated” into English is a very compelling storytelling device. The human storyline paints a vivid picture of a far-future spacefaring civilization that still feels grounded by the limitations of technology. Spaceflight takes decades in ramjet ships, with crews rotating in and out of cryosleep. I loved the ways that cryosleep affected the political and dynamics among the humans.

The Qeng Ho trading civilization is post-Cold War neoliberalism taken to its limit. A Deepness in the Sky was written in the late 90s and the Qeng Ho are cast as the heroes of the story, but Vinge still manages to describe and portray the downfalls and limits of the unending quest for efficiency that felt precient when reading after the supply chain disruptions of the 2020 pandemic.

Vinge was a Computer Science professor as his day job. One of the most interesting ideas he puts forward in the book is that of the programmer-archaeologist. Thousands of years into the digital age, all useful programs have already been written by someone, somewhere. The job of the programmer isn’t to write new code, but to sift through existing archives to find what’s useful in the current moment.

Overall, a really enjoyable book that continually made me stop and think.

Ad sharks

paulw.tokyo — Wed, 11 Dec 2024 17:24:56 GMT

I don't own a TV, but there's one at my gym. This morning, there was this... infomercial? I don't know how to call it, it lasted at least an hour and had the tone of the usual off-hour Japanese TV. Programs that try, not very hard, to be both entertainment and journalistic news. Of course it was interspaced by plenty of ads.

The topic was shark fishing, and how eating them keeps you young. And all the commercials, which seemed to feature only one brand, were dietary supplements made from shark extracts.

Both the show and the ads clearly targeted the elderly, with bold claims from "scientists" in lab coats of rejuvenating by 15 years.

No-one likes ads but this felts particularly egregious. It triggered me somehow.

Was it the visual impact of fishing large animals just to put them in pills? The certainty that my parents, not getting any younger, will be targeted by those scams? The non-existent boundary between content and adverts?

How do we stop this?

Not just TV scams preying on the elderly, the creep of ads for everything everywhere.

All growing tech companies focus on ads, even Apple now. It just seems so pervasive.

I don't think we can fight profits, we need to redesign our society so that ads aren't front and center.

How?

Buy my new book for answers!

Real-time semantic search demo

paulw.tokyo — Wed, 11 Dec 2024 17:24:50 GMT

TLDR: I wanted to make a fast and low complexity "search engine", and got good results with neural network embeddings combined with an approximate vector search.

Recent advances in large deep learning models have brought interesting multi-modal capabilities, notably for combined text and images, like Imagen and DALL-E 2 for image synthesis from raw text. CLIP, which is also used in DALL-E for image ranking, allow projecting both text and images into a coherent space.

I want to be able to search images from text, so using CLIP we can project both into the same space, making them comparable. Concretely we obtain vectors, and for pairs of vectors we can take their norm as a distance to find closest ones, and return that as a search result.

In practice comparing one vector (example our query) to all existing ones could be very slow if done exactly. Luckily approximate nearest neighbor (ANN) methods allow us to trade-off a tiny bit of accuracy for massive speed-ups and memory savings. Lots of methods exist (here for benchmarks and pinecone has great docs as an introduction), including innovative learned quantization methods that achieve excellent compression.

For the user interaction, any web tech stack can do the job, I went with FastAPI for the backend and Svelte for the frontend.

So let's combine the three!

The demo above search over ~1.5M images, taking about 3Gb of space for the index and with latency under 100ms. This is running on single process from my laptop.

To show off the speed, I prepared a tiny version of all images (<500 bytes) that are returned with the search results and loaded immediately. The full size image is then loaded asynchronously, that can take up to a second as those are random internet images.

Also human typing speed is not that fast, so to make the search field usable I added a delay before triggering the search, so the latency you see is after the search query is sent.

Here is an other demo, with full image loading disabled and a lower typing delay, just to show the speed.

I'm quite pleased with those results, there's still a lot that could be done to speed-up the search further but it's fast enough for a prototype. The index size/accuracy ratio could also be improved a lot, and the CLIP model should be transferable into a smaller inference network. Also I could make a UI that doesn't suck 😬

Calling this a "search engine" is a bit of a stretch, there is limited control and no filtering options but the tech stack is very simple (it only took a day to make this demo!). I think simplicity has value in itself. Though with the recent support of ANN+filtering in Lucene/Elasticsearch the comparison would be interesting.

Denormal number at inference in PyTorch

paulw.tokyo — Wed, 11 Dec 2024 17:24:44 GMT

The other day at work I noticed a slowdown in runtime between a model with random weights compared to tuned ones.

It turned out to be due to denormal numbers computation on the cpu being much slower than the normal arithmetic. Denormal numbers are very low magnitude floats, treated differently to keep precision.

For deep learning, those are largely below significance and can be flushed to zero without accuracy loss. To do so for example in PyTorch, either set the appropriate flag: torch.set_flush_denormal(True).

Or manually round down your weights, I choose $10 e - 12$ arbitrarily:

import torch

def flush(v, t=1e-12):
    v[torch.abs(v) < t] = 0.0
    return v

model = ...
model.load_state_dict(
    {k: flush(v) for k, v in torch.load("./model.pth").items()}
)

The speedup will be CPU and model dependent, for me it was quite significant (x2!). Hopefully this tip will save you some inference time too.

Space filling curves and CNN

paulw.tokyo — Wed, 11 Dec 2024 17:24:38 GMT

👷🏻 this is a work in progress 🔧

Sometimes by exploring incongruous ideas, one can make interesting discoveries. This is known as serendipity in science, with penicillin being a famous example of an unplanned discovery.

What I experimented with this week-end… is no such case. 😅
In the spirit of not holding back negative results though, and for the unlikely possibility that it turns out helpful for someone, I'll share it regardless!

Too many dimensions

In computer vision we mostly deal with spatial data, so to make our life easier we often bake into the models the assumption that nearby elements are related. This is what the fundamental convolution operation give us in deep learning models, some bias towards spatial locality.

However not everything can be applied spatially, particularly on high dimensional spaces. For examples, building blocks designed for natural language processing often consider only one dimension. Even the transformer model, which is position independent in it's original formulation, has been the focus of numerous researches attempting to bring back some locality, notably for efficiency improvements. Wouldn't it be great to be able to use, as is, techniques developed for $1 D$ space on arbitrarily high $N D$ ?

The straightforward way to achieve it is to flatten whatever dimensions we have in order. For example, with a $2 D$ image we could take each row and concatenate them into a big list, and we're done!
The issue is that we will break spatial locality. Neighboring pixels in a row will stay together, but pixel that were close within a column will suddenly have arbitrarily big distances separating them. Instead, we would like to flatten space while keeping the locality property as best as we can.

Space-filling curves to the rescue

Luckily, mathematicians have been studying this problem! I recommend this 3Blue1Brown for a great visual introduction, but in short the idea is to fill the entirety of space with a single curve.

There are a few different functions available, like the Peano and Hilbert curves, or most appropriate for us the Z-order curve (also known as Morton code). The advantage of the later is that, as illustrated in the figure above, on top of filling space it also preserve locality. Well, to an extent at least.

Using the Z-order function, we can create an index and use it for mapping from $N D$ to $1 D$ . Using einops to avoid getting lost in the dimensions, it is easy to create a function doing it for PyTorch.

(TODO: clean-up and publish the code)

Now that we have a way to flatten space while keeping some locality, we can design an experiment! To keep things fast I used the small CIFAR-10 dataset, which is fast enough to train on, even with a naive training loop implementation. As a bonus, I already had some code lying around that allowed me to reach 3000 samples/second on my aging 1070Ti (loosely based on Myrtle.ai posts about training speedup; a story for another time).

I tried 3 configurations:

A baseline model with the usual 2D convolutions
A model using 1D convolutions after flattening the input using the Z-order function
The same 1D model but with naive flattening, and random flattening

All models are kept as close as possible, using the same 9-layers architecture and swapping the convolutions/batch-norm/pooling to either 2D or 1D.
Because convolving on lower dimensions use smaller kernels it result in less parameters, so I compensate by adjusting the layers' width until the number of trainable parameters match. While this is by no mean sufficient to make it an apple to apple comparison, it will have to do!

(TODO: finish the write-up :D)

Creating a static blog, the convoluted way

paulw.tokyo — Wed, 11 Dec 2024 17:24:31 GMT

TLDR: I wanted to create a static blog while being able to type articles from a nice UI, from anywhere. It is quite straightforward with forestry + hugo + github pages. Jump to the solution.

Motivation

If you're looking to create a blog, you're in luck! There are countless tools to help you get up and running in a few minutes. For example Hugo will generate a static site from a few markdown files, and pushing it to github in a repository named YOUR_USERNAME.github.io will put you online.

That's it! You're done! ✨

However if it's too easy it isn't fun. Also by accepting what's good enough, you might actually write content into your blog! Let's not do that.

Here are my requirements:

The website should be static. It makes hosting cheap or even free.
Modifying any parts should be possible. I want control of my site.
Once set-up, I should be able to focus on the content. No distraction from the command line, just press save and be done.
Be able to edit from anywhere. That typo your friend noticed? Fix it with a quick edit from your phone!

The first two can be satisfied by any static site generator. The third can be solved by a nice tool like Publii, though it does make customizing details a bit more tricky. Editing from anywhere is easy with commercial offerings like WordPress, but the website is not static and I would need to give up a lot of control.

The solution for all four come from static content management system (CMS), which –as the name imply– will provide tools to manage your content and generate your static site.

I found two candidates: NetlifyCMS and Forestry.io. Both are promising and let you work with a site generator of your choosing. However I need only one, and forestry's editor looked prettier 🤩

Solution

If you want to look at the code directly, you can find it here. The steps below explain how to get there.

With the context out of the way, here are the necessary steps:

Create a github repository matching your username, for example jondoe.github.io (if you're not familiar with git or github, head over there for some simple explanations). Once done git clone https://github.com/jondoe/jondoe.github.io
Install Hugo and follow the quick start guide. Now is a good time to choose a theme. Push the code Hugo created for you into a new repository, for example https://github.com/jondoe/site_generator
Once you're done configuring your site to your liking, copy the generated static site (under ./public) to your jondoe.github.io directory. Push, and admire your website at https://jondoe.github.io!
Now we can setup the CMS. Login with your github account into forestry.io and follow the quickstart guide. (I know, I said that for Hugo already, that's the last one I promise!). You will want to link the site_generator repo, not the one hosting your generated site itself. Now from forestry you will be able to write new content, however it's not published to github pages yet.
To automate the publishing, we will use github actions, that will allow us to push the changes to the static site into the right repository. Luckily some nice folks already did the job for us, so let's use the example they provide. There are multiple ways to authenticate from your script, suggest using the deploy key.

That's it! (again)

Now you can edit content from forestry, and upon saving your changes will be published automatically. After a blog post or two, internet fame should follow!

Basic watermark removal in videos

paulw.tokyo — Wed, 11 Dec 2024 17:23:38 GMT

The general problem of completing images and videos depending on a mask is called inpainting, and numerous methods exists to tackle this problem in ingenuous ways.

However most approaches are relatively costly to run, especially without a graphic card, so I wanted to see what result we could get with simple and fast methods.

You can find the code for the steps below in this repository.

To start, we need to find the mask, which correspond to the location of pixels to remove.

Using ffmpeg we can extract the timestamps of the key frames in the video. Getting only the timing is fast, and we can later cap to a maximum the number of frames to actually extract. We could also take random frames, but the key frames are more likely to be diverse (can't be too close in time), and faster to extract as other frames need to be reconstructed from the closest key.

With ffmpeg, we just need to run:

ffprobe -select_streams v -skip_frame nokey -show_frames -show_entries frame=pkt_pts_time video.mp4

Then for each TIMESTAMP obtained, we can extract the frame:

ffmpeg -ss TIMESTAMP -i video.mp4 -vframes 1 frame.png

And lastly we can aggregate the results of a simple image filter over all frames, to create a mask:

# Compute the gradients per image
dx = np.gradient(images, axis=1).mean(axis=3)
dy = np.gradient(images, axis=2).mean(axis=3)

# Average globally
mean_dx = np.abs(np.mean(dx, axis=0))
mean_dy = np.abs(np.mean(dy, axis=0))

# Filter on both axis at a hand picked threshold
threshold = 10
salient = ((mean_dx > threshold) | (mean_dy > threshold)).astype(float)
salient = normalize(gaussian_filter(salient, sigma=3))
mask = ((salient > 0.2) * 255).astype(np.uint8)

Bonus, if one wants to do it without python, it should be doable using ImageMagick's convert with the existing Sobel filter and -evaluate-sequence mean *.png mean.png.

We now have a global mask to inpaint, so we can simply use ffmpeg's removelogo filter to obtain our cleaned-up video:

ffmpeg -i video.mp4 -vf "removelogo=mask.png" cleaned.mp4

This run at around x3 real-time on my aging MacBook Pro (i5-5287U), with a fixed overhead of ~5s for the mask extraction, which is fast! ✨

Clearly the resulting inpainting is not of high quality –we're not even leveraging temporal information across frames– but is reasonable enough as a baseline to compare against real-time approaches.

Mostly Stakeless

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Wed, 11 Dec 2024 17:00:00 GMT

One of the social conventions I still have trouble understanding is the tacit requirement to have opinions on everything, in particular complex socio-political conflicts. While I am not apolitical in the sense that I do have political opinions on some issues, I feel no need to know the relevant talking points or assume a position in every conflict. I also do not feel the urge to discuss or state my political views publicly because I am just some random person and cannot possibly contribute anything useful. This is not your standard case of politics is the mind-killer,¹ but a protection against my tendency to become ultracrepidarian,² especially when I lack knowledge of the issue at hand.

For instance, I find it hard to form substantiated opinions on global conflicts in which I do not have a direct ‘stake.’ I am empathising with those that are displaced, wounded, or even killed in such conflicts, but I find it hard to think about such conflicts in terms of right or wrong, when, historically speaking, ’everyone sucks here’ would be a smarter position to take. However, since I have not been trained as a historian and/or as a political scientist, I often do not know who the ‘good guys’ are.³ In some conflicts—the recent fall of the Assad regime comes to mind—it even seems to me that a devil has been replaced by some demons and I honestly do not know where things are going…

Ever the optimist, I hope for good to prevail in the world, and I find some solace in the words of Maimonides:

Men frequently think that the evils in the world are more numerous than the good things; many sayings and songs of the nations dwell on this idea. […] Not only common people make this mistake, but even many who think they are wise.

Much to my chagrin, though, there is a tendency of people in social media to make assumptions about my political views or, in some cases, chiding me anonymously for not taking their preferred stance. While I appreciate the feedback I receive anonymously, I want to gently caution people: Please do not assume that I do not care. Having no stake in most conflicts, I just do not want to add to the noise and I would rather prefer a nuanced discussion.

I feel reminded of a running gag in Terry Pratchett, viz. the Muntab question:

[…] “For example, what about the Muntab question?” Nanny Ogg asked the Muntab question. “Where the hell’s Muntab?” she said. “Several thousand miles away, Mrs. Ogg. But it has ambitions Hubward, and if there’s war with Borogravia we will certainly have to adopt a position.” “This one several thousand miles away looks fine by me,” said Nanny.

You could say that I am mostly stakeless and aware of it, so like Nanny, I am not sure what to say. This is not so much cowardice as it is an acute awareness of my own limitations. Please do not hold it against me.

PS: I got several interesting anonymous comments about this article and wanted to elaborate that this is not a lack of empathy but rather a lack of understanding of large-scale geopolitical conflicts. I do not believe that every conflict affords a simple good–evil dichotomy, hence while I empathise with the suffering of all those afflicted, I often do not know enough about the ramifications of picking one side over the other. Even in situations where this is possible, I believe my public opinion on these conflicts to be largely irrelevant in the grand scheme of things: I get the impression that I can only contribute to either a very strong signal or increase the noise. The first action strikes me as redundant, the second one as too chaotic.

A position that I find untenable and, frankly, very dangerous. ↩︎
An affliction I seem to share with many academics. ↩︎
Media can only help so much here, mostly because, depending on which type of media one consumes, the definition of good and bad can switch quite a bit. ↩︎

Generate Stunning Avatars Using OpenAI APIs

Adnan's Random bytes — Wed, 11 Dec 2024 07:54:27 GMT

This post is part of the GenAI Series. So far in the series, we have created AI apps that generate text. In this post, I will introduce OpenAI’s image generation APIs, which can convert a textual prompt into an image and perform many other tasks related to images. I am going to make an app called AvatarAI. It will generate an avatar based on different parameters. Avatars are very common in the digital world. People use them mainly as profile pictures for forums, depicting their digital version. In case you are just interested in watching the demo, you can see it below: Prompt Engineering Like before, we will first create a prompt that generates avatars. This process is called prompt engineering. The prompt consists of two parts: the data in JSON format, which will be provided by users to define the look and feel of the avatar, and the instructions, which will guide OpenAI LLM on how to use that data. Below is the prompt: Generate an avatar based on the following specifications given in JSON format, ensuring all details are accurately reflected and styled as described. You must NOT write any text on the generated image {“demographics”: {“gender”: “male”, “age”: “teen”}, “physicalCharacteristics”: {“skinTone”: “Tan”, “faceShape”: “Oval”, “eyes”: {“accessories”: “Glowing”}, “hairDetails”: “Wavy”}, “accessoriesAndClothing”: {“clothingType”: “Black Jacket”, “clothingColor”: “Metallic Tone”}, “environmentAndBackground”: {“theme”: “City Skyline”, “backgroundDetails”: “neon lights”}, “customization”: {“specialEffects”: “glow”}, “personalization”: {“interestsTheme”: “doctor”}} Let’s try it on chatGPT first: Not bad! Now it’s time to automate it. Development Assuming you have the OpenAI and Flask libraries installed, you will also need to install the library to fetch environment variables. If not, run the following commands: pip install openai pip install flask pip install python-dotenv The very first thing I am going to do is build the UI. Based on the above input, I asked GPT to come up with an interface like the one shown below: Looks cool, No? We are going to use OpenAI’s Image APIs for this purpose. from openai import OpenAI if __name__ == '__main__': client = OpenAI( api_key="sk-trS-xxxxxx") prompt = "An ethereal and dramatic depiction inspired by Wordsworth's poem 'The World Is Too Much With Us.' A moonlit sea with waves reflecting silver light, the winds swirling like ethereal shapes. In the foreground, a serene meadow ('lea') with a figure gazing at the sea, Proteus rising from the waves, and Triton in the background blowing a majestic, spiraled horn." response = client.images.generate( model="dall-e-3", prompt="A stylized avatar of a tech-savvy individual with glowing neon glasses.", size="512x512", quality="standard", n=1, ) image_url = response.data[0].url print(response) Oops! It gave an error: openai.BadRequestError: Error code: 400 - {'error': {'code': 'invalid_size', 'message': 'The size is not supported by this model.', 'param': None, 'type': 'invalid_request_error'}} It seems I can’t generate images of certain sizes. Upon reading I found this: hmm.., so for DALL-E 3 the minimum resolution I can produce is 1024×1024 Now, it produces the following output: ImagesResponse(created=1733810749, data=[Image(b64_json=None, revised_prompt='Create an illustration of a tech-savvy individual who has a futuristic look. The individual should be represented with glowing neon glasses that suggest their affinity for technology. The aura should be filled with lines of binary code cascading around them. The avatar should have an androgynous appearance to represent gender neutrality and can have any descent. The image should be highly stylized to encapsulate the modern, digital, and innovative essence of the tech world.', url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-hiprYm3IyTNvRn2pAPp9cAPR/user-MCgDqt1tM3GQg7BictRVYOgv/img-OSV5bL9eikrP0uv5FonnDFGP.png?st=2024-12-10T05%3A05%3A49Z&se=2024-12-10T07%3A05%3A49Z&sp=r&sv=2024-08-04&sr=b&rscd=inline&rsct=image/png&skoid=d505667d-d6c1-4a0a-bac7-5c84a87759f8&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-12-10T02%3A25%3A19Z&ske=2024-12-11T02%3A25%3A19Z&sks=b&skv=2024-08-04&sig=DwSbgi/omtXoHiXNdbymcO2Wj3mRt0PxmFgrPUXZ6uQ%3D')]) The ImageResponse object contains many things here. the parameter we are looking at here is url. It contains the URL of the generated image. You may also see the revised_prompt parameter that contains an internal prompt produced by GPT for internal usage. Now, I am not writing every bit of the code here, as I will share it anyway. Upon running the code, it generates something like the output shown below: Conclusion You see how easy it is to create an AI art generator app in Python using OpenAI APIs. The only thing that can hold you back is your creativity rather than the tools. You can customize this app as much as you want. By simply changing the prompt, you can transform this app into an AI logo generator app. The sky is the limit. As always, the code is available on GitHub. Looking to create something similar or even more exciting? Schedule a meeting or email me at kadnan @ gmail.com. Love What You’re Learning Here? If my posts have sparked ideas or saved you time, consider supporting my journey of learning and sharing. Even a small contribution helps me keep this blog alive and thriving. 👉 Support My Work Here If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Generate Stunning Avatars Using OpenAI APIs first appeared on Adnan's Random bytes.

How to build a fleet of networked offsite backups using Linux, WireGuard and rsync

./techtipsy — Wed, 11 Dec 2024 06:00:00 GMT

Just like most people out there, I have some files that are irreplaceable, such as cat pictures.

At one point I had a few single-board computers sitting idle, namely the Orange Pi Zero and the LattePanda V1, and a few 1TB SSD-s.

I hate idle hardware, so I did the most sensible thing and assembled a fleet of networked offsite backups for backing up the most important data.

My setup is based on various flavors of Linux, but the ideas will likely translate well onto other operating systems and solutions.

Networking

The most important part is the networking. The offsite backup endpoints connect together to my home server over a WireGuard network. The home server is, well, the server, and backup endpoints are clients.

I like this WireGuard Docker image a lot because it generates the server and client configurations automatically, but you can use plain WireGuard or a completely different networking solution to connect all the devices together. Some use Tailscale to make the setup process easier, but I like to keep things as self-hosted as possible.

I’m not a networking expert, but here’s how I’ve set up my network. For this example, the WireGuard network operates in the 10.13.69.0/24 range.¹

To only allow traffic between the devices and avoid tunneling everything through the home server, set the AllowedIPs setting to AllowedIPs = 10.13.69.0/24,10.13.69.1. We want to be able to access the backup endpoints, and nothing more.

All the devices have a static IP address in that network, such as 10.13.69.1 for the home server, 10.13.69.2 for a backup endpoint and so on.

The PersistentKeepalive = 25 option is present in the client configurations so that I don’t lose the ability to access the backup endpoints. With it, all the backup endpoints call back to the home server from time to time. The aforementioned Docker image automatically adds it to the generated configuration using the PERSISTENTKEEPALIVE_PEERS=all option. This setting is crucial. Without it, I sometimes ran into problems trying to connect from my home server to the backup endpoint, and that’s something you can’t easily alleviate without having physical access to the backup endpoints, which are offsite.

Remove the DNS configuration from generated WireGuard client configurations, as you don’t need it for this purpose.

Optionally, edit the /etc/hosts file for the home server and backup endpoints so that you can access your backup endpoints using simple hostnames, like orangepizero. Example row can look like this: 10.13.69.6 orangepizero.

if your WireGuard server operates in a network with a dynamic external IP address, as is common with many home internet connections, I recommend getting yourself a domain name that you can update whenever your IP address changes and using that in your WireGuard client configurations. Without this, an IP address change will result in your backup endpoints being inaccessible.

You’ll also likely need to set up port forwarding and/or traffic rules for your backup endpoints to be able to connect back to your WireGuard server.

Once you have the WireGuard connection set up and SSH running on the backup endpoints, you should be able to drop the backup endpoints into any network that you have permission for. Ask your friends and family, and sweeten the deal by offering free technical support or help in some other area in return. The cost of running a single-board computer 24/7 is minuscule with the typical power consumption being 1-3W, so that won’t be much of a concern.

Making backups

For making the actual backups themselves, you have all sorts of options.

I rely on rsync to copy the data over. It’s simple and it works, that’s all I expect from it. Example command: rsync -aAXvz /folder/to/back/up/ backupuser@backupendpoint:/backup/ --delete. The files will be compressed during transit with the -z option, and with --delete you’ll ensure that the target folder has all the files from the source, and nothing else.

The backup storage is specified in /etc/fstab with the nofail option present. This ensures that in case the disk dies, the backup endpoint will still boot properly, allowing me to access the machine to troubleshoot the issue and/or force a desperate reboot to try to fix things. A good alternative approach is to mount/unmount the remote disk manually as part of the backup script.

The backup storage uses the btrfs filesystem, and I use btrbk to take snapshots of the contents. If I accidentally delete all the files on the backup endpoint, then I can still recover from that situation because the data is still present in snapshots. 30 days is a good retention period: enough time to save the data in case of an accidental deletion, but short enough to avoid the backup disk getting full.

If you don’t want to use filesystem-level snapshots, then tools like restic are a good alternative. It can also operate over SSH and you can configure snapshot retention policies in your backup script. Just make sure to not lose the encryption password, and verify the backups once in a while.

Deployment

I manage my backup endpoints using some cobbled-together Ansible roles. I’ve perfected it to the point where the only manual steps are flashing the OS and setting up the storage, the rest is handled via Ansible.

I’d like to share my work here, but it will make Jeff Geerling cry. Maybe one day I’ll take the time to improve things…

Maintenance

All the backup endpoints update and reboot themselves regularly. It’s just the sensible thing to do.

Every 6-12 months I also do major OS version updates. It’s risky because of the whole offsite aspect of the solution, but so far I haven’t been burned yet.

Monitoring

Monitoring is an area where I have some room for improvement. So far, I’ve set up Prometheus node-exporter to all of the backup endpoints, and my home server keeps track of how the backup endpoints are doing.

This allows me to check once in a while if any of the backup endpoints has fallen off the network, or if the backup disk is getting full.

Issues I’ve faced

I’ve had this system running for a few years now, and it’s mostly stable! There have been some issues I’ve faced as well, though. Some are very specific to certain hardware, but I think there’s value in mentioning them.

I once blew up a backup server because of an Ansible configuration issue. That meant that I had to physically go pick up the server to re-image it.

The Orange Pi Zero was running quite hot, resulting in stability issues, so I put together a really janky cooling solution.

Cooling. It works.

Hell, I did the same for the LattePanda as well.

First version of the cooling upgrade on the LattePanda.

It might look horrific but the extra cooling has fixed all the stability problems on both boards.

The lack of a real-time clock on the LattePanda has required me to make its backup script a bit special. I can’t rely on a systemd timer that automatically reboots the machine once in a while, so instead that part is present in the backup script. The issue is that the LattePanda boots up with the time being set in the past, and once it gets the actual time from the network, it will run all sorts of tasks because enough time has passed! This included the reboot timer as well.

At one point, the power supply on the LattePanda just died, and it was very visible on my graphs. That required a replacement.

Signs that you might have a failing power supply.

Conclusion

That’s how I back up the most important data. I hope that this has given you inspiration to take your own backup approach to the next level!

yes, I do plan to move this setup to IPv6 eventually. ↩︎

Sex development, puberty, and transgender identity

De Novo — Tue, 10 Dec 2024 23:30:57 GMT

Reproductive Biology for for Politicians and Voters, Part 2A

[AI Act Conference 2024 - Notes] Legislating Technology: Addressing the law's internal dilemma

Liza Shulyayeva — Tue, 10 Dec 2024 23:01:33 GMT

These are my notes from one of the AI Act Conference presentations from December 10, 2024.

Addressing the law’s internal dilemma
Stanley Greenstein, Associate Professor in Law and IT, Faculty of Law, Stockholm University.

This work examines the general regulation of technology and is a work in progress.

Generative AI is digital homeopathy — how I train my own model

Mark Greville — Tue, 10 Dec 2024 18:49:02 GMT

If it’s your first time here, you may be surprised at how few pieces I have written. (After reading for a while, you may even be glad of this fact). When friends bring up what they assume is a painful subject, they get a faraway look in their eyes. They place a gentle hand on […]

Using diskimage-builder with a single yaml file

Tue, 10 Dec 2024 18:20:00 GMT

diskimage-builder with a single yaml file

Why diskimage-builder

With the recent move of OpenStack devstack to require/support Ubuntu 24.04 Noble, I needed to make some changes to my local environment. Previously, I was using an unmodified Ubuntu cloud image as a base, supplying a cloud-init ISO (created by cockpit-machines) to do just-in-time user and network configuration.

For environmental reasons mostly irrelevant to this post, that approach doesn’t work for me anymore. So I decided to use diskimage-builder to create an image that’s ready to go from get-go. While I’ve used diskimage-builder in the past – I actually just fixed the Gentoo element for it – I have typically used it to create ramdisks for IPA, and rarely to create my own cloud base images.

untitled excerpt

asemic horizon — Tue, 10 Dec 2024 13:40:44 GMT

…. ECONOMIC GROWTH: if there was a single phenomenon that could be characterized as “emergent”, this would be it. Dave Snowden posted something about purpose (or was it strategy? in many ways it doesn’t really matter) being an emergent process rather than a north star. The point being that you can’t beam whatever makes orgs […]

Amazon Scraper: Get ANY Product Data from Amazon without coding

Adnan's Random bytes — Tue, 10 Dec 2024 10:57:41 GMT

As the world’s leading e-commerce platform, Amazon not only provides consumers with a seamless and reliable one-stop shopping experience but also serves as a critical data aggregation hub. Its vast repository of product information enables sellers to perform in-depth market research and develop effective sales strategies. How to integrate a substantial amount of data in a structured way to give your e-commerce business a real leg up? It is not a smart move to manually search, paste, and copy the required information directly; while using code for web scraping poses a technical barrier for people lacking a programming background. Fortunately, there is a user-friendly intelligent web scraper available that can swiftly extract data like product prices, ratings, reviews, and virtually all pertinent information within just a few seconds. In this article, we’d like to walk you through the basic process of scraping product data from Amazon without code. Amazon Web Scraping You Should Know Is it legal to scrape Amazon Web scraping and crawling are not inherently illegal; however, they can become problematic when employed for malicious activities. These include intellectual property infringement, data theft, account hijacking, or any other unlawful intentions. As a web scraping tool is merely a means to an end, you’re suggested to adhere to the principle of not collecting any private information. Additionally, always review the terms of use on the target websites before starting the data scraping process to avoid any unforeseen complications. What data can be extracted from Amazon Generally speaking, you have access to a comprehensive range of data that is visually presented on the Amazon web pages. To illustrate further, let me enumerate some of the primary types of data that you can successfully procure from this online platform. Listing page: title, price, stars, number of reviews, estimated delivery time, etc. Detail page: product description, customer reviews, inventories, images, questions, etc. Why collect product data from Amazon Scraping statistics from Amazon can provide a multitude of benefits for businesses. Here are a variety of scenarios where web scraping Amazon can be helpful: Market Research Competitive Analysis: By collecting data on similar products, sellers can analyze their competitors’ pricing, sales, and reviews, gain insights into the market, and adjust their pricing(price monitoring and dynamic price) and marketing strategies accordingly. Trend Identification: Web scraping facilitates the recognition of emerging trends and sought-after products in the marketplace, which can inform future product development and inventory management. Customer Insights Review Analysis: Amazon reviews provide a wealth of statistics concerning customer preferences, pain points, and feedback. Analyzing this data can help businesses optimize their products and enhance customer service. Sentiment Analysis: By analyzing customer sentiment, businesses can gauge how their products are being received in the market and make necessary adjustments. How to Configure a No-Code Amazon Scraper With the assistance of the user-friendly web scraping tool, extracting data has become a breeze even for those without coding experience. Octoparse is an easy-to-use web scraper that is accessible to everyone. To get started, simply download and install the software on your device. Next, create a free account to log in and familiarize yourself with the fundamental settings. Please be assured that Octoparse offers step-by-step live demo tutorials specifically designed for beginners! You can also kickstart your learning process by watching videos of hands-on Amazon data-scraping experiences shared by bloggers. Following that, you can embark on your journey of crawling data with the procedures listed below. Step 1: Create a new task with the URL Once you have entered the keyword or ASIN and navigated to the webpage containing the data you aim to extract, simply copy and paste the target URL into Octoparse’s search bar. Click the “Start” button to initiate a new task, and within a matter of seconds, the page will load in Octoparse’s built-in browser. Step 2: Auto-detect data on the Amazon webpage Once the page has finished loading, click “Auto-detect webpage data” in the Tips panel to let Octoparse scan the page and detect data fields for you. It’ll highlight all detected data for you to locate and preview extractable data. If there are any unwanted data fields, you can also remove them from the Data Preview section at the bottom. Step 3: Create and modify a workflow After ensuring the accuracy and completeness of the data, you can click “Create workflow” and then a workflow will show up on the right-hand side. This diagram is actually a flowchart illustrating how this scraper works. You can read the workflow from top to bottom, from inside to outside if there are any nested actions. To check if the scraper works as intended, click on each step on the diagram. This will allow you to preview and see how it functions at each stage. The aforementioned content primarily shows the process of collecting listing page data using Octoparse. If you also want to gather detailed page data concurrently, you can refer to their tutorial video. Step 4: Run the task and export Amazon product data After you have reviewed the entire workflow, launch the scraper by clicking “Run.” You can choose to run the task on your local device or on Octoparse’s cloud servers. Each option has its own advantages: running locally is suitable for smaller-scale projects, while cloud servers are more efficient for handling large-scale tasks. Select the option that best fits your needs, and Octoparse will take care of the rest. Once the scraping process is complete, you can export the data in various formats, such as Excel, CSV, or JSON files, or even to a database like Google Sheets and Amazon Web Services, for further analysis and use. Preset templates for Amazon data scraping To better streamline your data collection and analysis, Octoparse has curated an extensive range of 20 refined Amazon scraper templates for selection. These templates cater to a variety of scraping requirements, including gathering information from listing pages or detail pages, as well as extracting product details or user reviews via ASIN or keywords. Furthermore, they offer tailored scraping tool versions suited for different countries. Warp-up To sum up, by aggregating data from Amazon, sellers can gain effective means to compare product prices, assess consumer reviews, and obtain other insightful information. With Octoparse, the technical barrier of requiring coding knowledge for web scraping has been eliminated, making it possible for everyone to easily access and utilize the integrated data. Therefore, no-code web scraper can definitely be a right-hand tool for anyone doing Amazon business data analysis. Octoparse goes beyond just that, offering a wide range of scraping opportunities including lead generation, social media data, higher education research, and content and news aggregation

The post Amazon Scraper: Get ANY Product Data from Amazon without coding first appeared on Adnan's Random bytes.

Coding on autopilot

Mohnish Thallavajhula — Tue, 10 Dec 2024 09:46:40 GMT

Over-relying on copilots for programming puts you on autopilot. It creates mental laziness and that is something to be avoided.

How to automatically switch git profiles based on the current directory

Zachary Proser — Tue, 10 Dec 2024 02:00:00 GMT

You can automate your git profile switching based on the current directory. Here's how to do it.

AI Act Conference 2024 Keynote notes: Navigating the AI Act

Liza Shulyayeva — Mon, 09 Dec 2024 23:01:33 GMT

Today I listened in to part of the AI Act Conference hosted by Uppsala University. Below are my notes from the keynote.

On C# and .NET quick release cycle

Nicola Iarocci — Mon, 09 Dec 2024 18:31:18 GMT

I sat to jot down a quick introduction to my C# 13 What’s New and Interesting session next week, and what I ended up with instead is a long rant or, should I dare, stream of consciousness that is certainly inappropriate for a five-minute introduction. I’ll have to cut most of it down, especially on the personal story part, but my site might be a good place to host it in all its completeness. I may reference this post at the start of the session on Monday for those few poor souls who might be interested.

I built a community of 1,600... accidentally

Martin Rue — Mon, 09 Dec 2024 06:00:00 GMT

When I say community, I mean social network. Yes, I made my own social network. It doesn't get much more narcissistic than that.

An experimental (e)shell pager

xenodium.com @alvaro — Mon, 09 Dec 2024 02:00:00 GMT

These days, my LLM interactions primarily take place via chatgpt-shell's compose UX. I've grown fond of this hybrid style of interaction. On sharing the latest incarnation, jllw asked about the possibility to port to comint shells.

With jllw's great idea in mind, I set out to prototype an eshell pager, my preferred shell.

The initial results ain't too shabby:

While you can easily flip through all eshell commands and outputs (via single-character n/p bindings), you can also compose and fire off new commands using a magit-style approach. That is, craft your desired command and C-c C-c to send it off.

The prototype is currently focused on eshell, but it can be easily ported to comint and other shells by writing new shell-maker configs:

(shell-pager--make-config
 :shell-buffer buffer
 :page-buffer (shell-pager--buffer)
 :next #'shell-pager--eshell-next
 :previous #'shell-pager--eshell-previous
 :current #'shell-pager--eshell-current-item
 :subscribe #'shell-pager--eshell-subscribe
 :unsubscribe #'shell-pager--eshell-unsubscribe
 :submit #'shell-pager--eshell-submit
 :interrupt #'shell-pager--eshell-interrupt)

shell-pager's super rough/experimental code is now on GitHub.

It's been a fun experiment so far. Time will tell, whether or not shell-pager can replace most of my eshell interactions. The good news is that this isn't an all or nothing sorta thing. shell-pager complements eshell, so I can always jump back to eshell itself and continue with the very same shell history.

App Config Design

Craig Pardey — Mon, 09 Dec 2024 02:00:00 GMT

I wrote about DRY configs many years ago but astonishingly my clients don’t seem to read my blog. I mean, who even has a blog these days anyway?

Externalized application configs have a way of multiplying like rabbits to the point where people become afraid to touch them at all, let alone apply the DRY principle to them. The proliferation of poorly-designed configuration files in Enterprise Java systems is appalling, and the situation is compounded when an external configuration system like Consul or Spring Cloud Config is added to the mix.

Parsing MIDI messages in Rust

ntietz.com blog - technically a blog — Mon, 09 Dec 2024 02:00:00 GMT

I'm working on a terrible idea of a project, and this project uses MIDI. That means I need a MIDI implementation! I chose to use an existing library, midir, to connect to devices and receive messages. But the reason I was interested in this not-yet-announced project is because I wanted to understand MIDI. So it was time to implement the communication protocol myself.

What is MIDI and why's it cool?

MIDI stands for Musical Instrument Digital Interface, and it really doesn't bury the lede. It's a standard for digital musical instruments communicate! The standard includes both the electronics and hardware, and it includes the communication protocol. This post is only concerned with the communication protocol.

In 1980, there was no standard digital interface for how instruments communicate^[1]. You had electronic instruments, and it would be nice to put them together, but manufacturers did their own things. Roland created a couple of protocols for their own devices, then pulled in some folks from other companies to make a standard. This eventually became MIDI, and the first MIDI device was released in 1983.

Since then, some new functionality has been added, but the core has remained the same. There are a few new(er) standards that are in various stages of use. For example, Open Sound Control (OSC) is used in some instruments and applications, and was created in the early 2000s. And MIDI 2.0 has been announced but doesn't have widespread adoption.

MIDI, the original, is still in widespread use because it works and it's ubiquitous. Let's marvel at that a little bit: this protocol has lasted over 40 years, and it has a successor which isn't widely implemented. It certainly has some flaws and quirks, and it has limitations, and that's why we'll eventually see it replaced or surpassed. But its longevity is simply incredible.

When you connect devices with MIDI, each can send and receive messages. These messages let you do things like turn on and off a specific note (play A4 at this volume), bend the pitch, and change parameters of the synthesizer. It also lets you do things like synchronize timing, select a song to play, or perform manufacturer-specific commands.

A common MIDI use case is connecting a controller (a keyboard, wind controller, or other instrument with controls you can activate) to a synthesizer (hardware or software). Not every electronic instrument can make sound on its own, and this lets you decouple those pieces! You can also connect multiple devices together, so you can have one sequencer that's recording MIDI events and replaying them, and then have other controllers feed into that, and have the whole system output to a synthesizer (or use one that's onboard). There's a lot of flexibility and you can do some really neat things with MIDI, especially since you can edit the actual note on/off events and their timing and play around with those!

So... all I want with MIDI is to make my computer listen to some MIDI messages and do something based on them. Let's look at how the protocol works.

The basic workings

The fundamental unit of the MIDI protocol is a message. Each message has one status byte, followed by some number of data bytes. A status byte starts with a leading 1, and data bytes have a leading 0 (so they are effectively 7-bit bytes)^[2].

And each message falls into a particular group. The main groups of messages are voice messages, system common messages, and system real-time messages^[3]. Voice messages tell you about playing sounds: note on/off, key pressure, pitch bends, etc. System common messages let you do manufacturer-specific things and control positioning in songs/sequences. And system real-time messages are for timing, mostly.

The overall structure of a MIDI message is a status byte followed by some data bytes. For example, if we have two data bytes, we could draw it like this.

What can the status be, and what do the data bytes represent? It depends on the kind of message.

Voice messages

There are seven voice messages: Note Off, Note On, Aftertouch, Control Change, Program Change, Channel Pressure, and Pitch Wheel.

We might get a message with the value 0x904851 (three bytes, in hex^[4]).

To parse this, we deal with the status byte first. For voice messages, we split this into two pieces: the first nibble (four bits) is the category of message, and the second nibble is the channel (values 0-15, representing channels 1-16).

So when we look at this nibbles for our message, we see that this message is the category 0x9 and the channel is 0x0. 0x9 denotes a Note On event, which for a keyboard is sent when a key is pressed down.

Given it's Note On, we expect two data bytes to follow. The first, 0x48, is the note number. The second, 0x51, is the velocity. So our note is 72 with a velocity of 81! This corresponds to a C4 at roughly 60% of the available volume.

The rest of the voice messages are parsed in the same way, with 1 or 2 data bytes. Program Change and Channel Pressure have one data byte, and the rest have two data bytes. The data bytes have different meanings based on the category, but most of these are either a single 7-bit value or a pair of 7-bit values. The exception is Pitch Wheel, which is a 14-bit value that you reconstruct from the two 7-bit halves.

A fun part of the spec here: If you send a Note On message with velocity of 0, it must be interpreted as a Note Off message (which itself also has a velocity for the release speed). The keyboard I have functions like this, only sending Note On, while my wind synth sends both types. This is all valid and compliant with the spec.

System common messages

These are much like voice messages, but the status byte is used entirely for what kind of message it is. These are general to all connected MIDI devices, so we don't specify the channel.

There are five system common messages: System Exclusive, MIDI Time Code, Song Position Pointer, Song Select, and Tune Request.

Tune Request has no data bytes, so it's just the one status byte. Song Position Pointers are parsed like Pitch Wheel, where we have two data bytes which form one 14-bit value. Song Select is parsed as a single byte value which specifies the song number. And MIDI Time Code is also a single byte, but it's parsed as two nibbles for a message type and value.

The most interesting one is System Exclusive (SysEx). This basically gives us arbitrary per-manufacturer message types, and anyone else is supposed to close their ears and ignore that one if it's not for them. Some of these are things like bulk data dumps or listing patch parameters.

SysEx messages start with 1 or 3 bytes for the manufacturer ID, and then the rest is data. These messages are arbitrary length, and are terminated by finding the byte 0xF7.

System real-time messages

The final category of messages are system real-time messages. To me, these seem both simple and utterly cursed.

The simple part: there are seven messages, each of which is only one byte. You have Clock, Tick, Start, Stop, Continue, Active Sensing, and Reset.

Let's use Clock as an example. It's sent 24 times per quarter note, and the entire message is its status byte: 0xF8.

Okay, what's wrong with that?

Nothing, the message itself is fine. It's just where you can put it.

What seems utterly cursed is that you can put these anywhere and it's valid. In between bytes of other messages, sure! So the Note On message we had could be received instead as 0x9048F851.

Like... I get it. This means we can send these messages at exact times so that timing is locked in. But the rest of the messages, except SysEx, are at most 3 bytes total. It seems a little unnecessary to do this! And it makes parsing more complicated, because you have to check each byte for if it's a system real-time message, instead of knowing that the next couple of bytes are definitely for this message.

A parser combinator appears

Now that we've looked at how the protocol works, let's parse it! There are a variety of ways to write parsers in Rust. I chose to use a parser combinator since it's a relatively simple approach here, and it lets us write a lot of reusable code.

Structuring our data

First we need the structures we're parsing into. We can define enums for each of the three message groups. There's an extra in each of these—Unknown—to provide a fallback if we run into one of the reserved status bytes.

#[derive(PartialEq, Eq, Debug, Clone)]
pub enum VoiceCategory {
    NoteOff { note: u8, velocity: u8 },
    NoteOn { note: u8, velocity: u8 },
    AfterTouch { note: u8, pressure: u8 },
    ControlChange { controller: u8, value: u8 },
    ProgramChange { value: u8 },
    ChannelPressure { pressure: u8 },
    PitchWheel { value: u16 },
    Unknown,
}

#[derive(PartialEq, Eq, Debug, Clone)]
pub enum SystemCommon {
    SystemExclusive { data: Vec<u8> },
    MidiTimeCode { time_code: u8 },
    SongPositionPointer { value: u16 },
    SongSelect { song_number: u8 },
    TuneRequest,
    Unknown,
}

#[derive(PartialEq, Eq, Debug, Clone)]
pub enum SystemRealtime {
    Clock,
    Tick,
    Start,
    Stop,
    Continue,
    ActiveSense,
    Reset,
    Unknown,
}

Then we define a struct for the VoiceMessage, since we also want the channel information.

#[derive(PartialEq, Eq, Debug, Clone)]
pub struct VoiceMessage {
    pub category: VoiceCategory,
    pub channel: u8,
}

impl VoiceMessage {
    pub fn new(category: VoiceCategory, channel: u8) -> VoiceMessage {
        VoiceMessage { category, channel }
    }
}

And we make a high-level enum to contain each of the message groups. This approach lets us treat entire groups of messages in the same way, rather than having to match on each individual message type. You could certainly make one big enum for all of them, though!

#[derive(PartialEq, Eq, Debug, Clone)]
pub enum Message {
    Voice(VoiceMessage),
    System(SystemCommon),
    Realtime(SystemRealtime),
}

Building our parser

Parser combinators are neat because they let you combine small, discrete pieces into a larger whole. You define small parsers, then build your parser by combining these together in various ways!

Top-level parser

We'll start at the high level. Since we have three different message types, we'll define three parsers, one for each message type. So our top-level parser will return the main Message enum from above, and it will call each of our individual parsers in turn. Which parser to call is determined by the range the status byte is in.

pub fn parse_message(bytes: &[u8]) -> IResult<&[u8], Message> {
    let (bytes, status_byte) = take(1usize)(bytes)?;
    let status_byte = status_byte[0];

    // TODO: implement running status; see [1].
    // [1]: http://midi.teragonaudio.com/tech/midispec/run.htm

    if status_byte < 0xF0 {
        let (bytes, vm) = parse_voice_message(status_byte, bytes)?;
        Ok((bytes, Message::Voice(vm)))
    } else if status_byte < 0xf8 {
        let (bytes, sc) = parse_system_common(status_byte, bytes)?;
        Ok((bytes, Message::System(sc)))
    } else {
        let sr = parse_system_realtime(status_byte);
        Ok((bytes, Message::Realtime(sr)))
    }
}

A few things to note in parse_message:

take is a parser that's defined by nom for us, and calling it defines a new parser that takes the specified number of bytes. When you invoke this parser on bytes, the return value is a tuple of the remaining bytes after parsing, along with a slice of the taken bytes.
There's a TODO in here, because you can implement running status, which lets the status byte be omitted if nothing else has happened in the meantime. I chose to ignore this for now, until some piece of hardware requires I implement it.
The library I'm using for MIDI device discovery also chunks messages for me, so I don't think I can run into the interleaved messages situation in this code, which is why I have a separate parser for it and am ignoring that here.

Now let's look at how those child parsers are implemented!

Parsing system real-time messages

System real-time messages are the simplest, since they're just one byte, so we can knock that parser out quickly. It's just a big old match statement. We check the byte that's passed in and we return the appropriate value.

pub fn parse_system_realtime(status_byte: u8) -> SystemRealtime {
    match status_byte {
        0xf8 => SystemRealtime::Clock,
        0xf9 => SystemRealtime::Tick,
        0xfa => SystemRealtime::Start,
        0xfb => SystemRealtime::Continue,
        0xfc => SystemRealtime::Stop,
        0xfe => SystemRealtime::ActiveSense,
        0xff => SystemRealtime::Reset,
        _ => SystemRealtime::Unknown,
    }
}

Interlude: helper parsers

Okay, so I know what's coming up for the other two parsers we need. We'll need some helper functions or we'll have a lot of repetition. Let's knock those out here for clarity of exposition.

We'll need to handle forming messages from one data byte or two data bytes, and we'll need to parse 14-bit values.

To handle one- or two-byte messages, we can define a parser which takes in a function. This function will take in one or two bytes as parameters and should return a message of the type we want. For example, to form a ProgramChange message, we may pass in |value| VoiceCategory::ProgramChange { value }): a lambda function which takes in the 8-bit value and constructs just a ProgramChange instance. This is all just a convenience so we can snag the one or two bytes we need and invoke a constructor with them.

pub fn one_byte_message<T, F>(bytes: &[u8], f: F) -> IResult<&[u8], T>
where
    F: Fn(u8) -> T,
{
    let (bytes, b) = take(1usize)(bytes)?;
    Ok((bytes, f(b[0])))
}

pub fn two_byte_message<T, F>(bytes: &[u8], f: F) -> IResult<&[u8], T>
where
    F: Fn(u8, u8) -> T,
{
    let (bytes, b) = take(2usize)(bytes)?;
    Ok((bytes, f(b[0], b[1])))
}

Parsing a 14-bit value is also pretty straightforward, but uses bit manipulation that may be unfamiliar. We snag two bytes using nom's take parser, then we shift the first byte left by 7 bits and bitwise or it with the second byte.

pub fn take_14_bit_value(bytes: &[u8]) -> IResult<&[u8], u16> {
    let (bytes, db) = take(2usize)(bytes)?;
    let value = ((db[0] as u16) << 7) | db[1] as u16;

    Ok((bytes, value))
}

Okay, now we have our little helpers. Back to business!

Parsing voice messages

As mentioned before, voice messages have two pieces of data in their status byte: the category and the channel. So the first step in parsing them is to extract that.

Let's make our parser function, which will take in the status byte and the remaining bytes, and will return a Result (IResult is a nom-specific variant that already includes the error type for us). To start with, we'll pull out the category and channel from the status byte, and we'll lay out the structure for handling different cases.

pub fn parse_voice_message(status_byte: u8, remainder: &[u8]) -> IResult<&[u8], VoiceMessage> {
    let category_nibble = 0xf0 & status_byte;
    let channel = 0x0f & status_byte;

    let (remainder, category) = match category_nibble {
        // ...
    }

    Ok((remainder, VoiceMessage::new(category, channel)))
}

Now the question is what we do in each of those cases. It's easy to handle AfterTouch, ControlCHange, ProgramChange, and ChannelPressure entirely in terms of the helpers we defined before. The following match arms are to be added inside the match in the previous code sample

        0xa0 => two_byte_message(remainder, |note, pressure| {
            VoiceCategory::AfterTouch { note, pressure }
        })?,
        0xb0 => two_byte_message(remainder, |controller, value| {
            VoiceCategory::ControlChange { controller, value }
        })?,
        0xc0 => one_byte_message(remainder, |value| {
            VoiceCategory::ProgramChange { value }
        })?,
        0xd0 => one_byte_message(remainder, |pressure| {
            VoiceCategory::ChannelPressure { pressure }
        })?,

Then we can handle the pitch wheel, which is like these but needs to use the 14-bit parser. This is our parser for it, which we'll call inside the match as well.

pub fn parse_pitch_wheel(bytes: &[u8]) -> IResult<&[u8], VoiceCategory> {
    let (bytes, value) = take_14_bit_value(bytes)?;
    Ok((bytes, VoiceCategory::PitchWheel { value }))
}

And finally we get to parse voice notes! Since we have the funky behavior of NoteOn, with velocity=0 denoting sending a NoteOff, we'll use one function for these together. But we still get to reuse those helpers! It takes in both the byte slice and a boolean, off, which is used to say whether this is certainly a note-off event or not.

pub fn parse_voice_note(bytes: &[u8], off: bool) -> IResult<&[u8], VoiceCategory> {
    two_byte_message(bytes, |note, velocity| {
        if velocity == 0 || off {
            VoiceCategory::NoteOff { note, velocity }
        } else {
            VoiceCategory::NoteOn { note, velocity }
        }
    })
}

And we add the remaining three cases to our match, along with the default case.

        0x80 => parse_voice_note(remainder, true)?,
        0x90 => parse_voice_note(remainder, false)?,
        0xe0 => parse_pitch_wheel(remainder)?,
        _ => (remainder, VoiceCategory::Unknown),

Put it all together, and we get a parser for voice messages!

Parsing system common messages

The last group of messages is a lot of the same, so I'll start with the full definition here and then dive into the interesting part, SysEx messages. I'm including one unseen function definition in here, parse_song_position_pointer, since it's the same as the pitch wheel one except it returns a different variant—all the parsing is the same.

fn parse_system_common(status_byte: u8, bytes: &[u8]) -> IResult<&[u8], SystemCommon> {
    match status_byte {
        0xf0 => parse_system_exclusive(bytes),
        0xf1 => one_byte_message(bytes, |time_code| {
            SystemCommon::MidiTimeCode { time_code }
        }),
        0xf2 => parse_song_position_pointer(bytes),
        0xf3 => one_byte_message(bytes, |song_number| {
            SystemCommon::SongSelect { song_number }
        }),
        0xf6 => Ok((bytes, SystemCommon::TuneRequest)),
        _ => Ok((bytes, SystemCommon::Unknown)),
    }
}

pub fn parse_song_position_pointer(bytes: &[u8]) -> IResult<&[u8], SystemCommon> {
    let (remainder, value) = take_14_bit_value(bytes)?;
    Ok((remainder, SystemCommon::SongPositionPointer { value }))
}

Now the only thing we haven't seen is parse_system_exclusive. Remember that it's a dynamically sized message. Once we detect the SysEx starting byte (0xf0), we just take and take and take until we find the ending byte, which is 0xf7. We'll leverage a couple of nom combinators for this:

take_till accepts a function as an argument and takes bytes until that function returns true
tag accepts a string or byte array and expects that to be the next value, failing if it isn't present

And then we can shove this all into a Vec and call it a day.

Putting it together, we get this short function.

pub fn parse_system_exclusive(bytes: &[u8]) -> IResult<&[u8], SystemCommon> {
    let (remainder, data) = take_till(is_status_byte)(bytes)?;
    let (remainder, _) = tag([0xf7])(remainder)?;

    let data: Vec<u8> = data.into();

    Ok((remainder, SystemCommon::SystemExclusive { data }))
}

And that's it, we've handled the system common messages!

Which means we've handled all the message types.

Can I use it?

This code is for a project I'm working on and I don't have that open source yet (it probably will be eventually!), so no (or not yet). Besides, you probably don't want to: it hasn't been profiled for performance, it's only tested against two of my instruments, and it's likely to have breaking changes soon. There are other libraries out there that are a better choice if you do want to use them, such as midi-msg and midly.

It's also a pretty simple protocol, and it's fun to build your own small parsers!

If you've done anything fun with MIDI or Rust, I'd love to hear about it. Just send me an email (listed below).

Thank you to Robbie for very helpful feedback and corrections, including the note about CV/gate. Any remaining errors are my own.

There was already a method for analog interfacing between equipment called CV/gate. This was around at least as early as 1970, but I can't find a lot of info on this. ↩
This means you can do a bitwise & 0x80 to check if something is a status or data byte. ↩
There are also channel mode messages, which are a variation on a particular kind of voice message. I'm leaving them out here for clarity, but they do technically exist. ↩
I'll represent everything that is hexadecimal with 0x at the start. ↩

Grafana with VictoriaLogs plugin on NixOS

Paul's page — Mon, 09 Dec 2024 02:00:00 GMT

VictoriaLogs looks like an interesting alternative to Grafana Loki. But it would be nice to still query it via the Grafana Dashboard along with all the rest. It takes a couple of specific steps to install the appropriate plugin, especially on NixOS.

Why I read history books

Bytepawn - Marton Trencseni — Mon, 09 Dec 2024 01:00:00 GMT

Hungary’s rushed entry into World War II and its botched attempt to break away from it show how stale leadership trapped in its own fantasy bubble can steer a nation toward disaster. By “political backtesting” these moments — comparing their underlying factors to the present — today’s citizens can weigh future outcomes, draw practical lessons, we can make more informed decisions about where and how to live.

Justin Timberlake: The Forget Tomorrow Tour

The Digital Nomad — Sun, 08 Dec 2024 23:45:48 GMT

We went to see Justin Timberlake at the Kia Center the day after coming back from Daytona.

Not much to say. The visuals were great. The choreography was top notch and he put on a good performance.

Första gången fast på tåget från Uppsala och Anki för svenskainlärning

Liza Shulyayeva — Sun, 08 Dec 2024 23:01:33 GMT

Så det hände äntligen: jag fastnade på tåget från Uppsala till Stockholm C. Min kollega från Uppsala har redan retat mig, eftersom han visste att det skulle hända. Han hade rätt.

Optimizing mastodon for a single user

Gonçalo Valério — Sun, 08 Dec 2024 20:16:40 GMT

I’ve been participating in the Fediverse through my own mastodon instance since 2017. What started as an experiment to test new things, focused on exploring decentralized and federated alternatives for communicating on top of the internet, stuck. At the end of 2024, I’m still there. The rhetoric on this network is that you should find […]

IdentityServer in Docker Containers: HTTPS and SameSite (Part 3)

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Sun, 08 Dec 2024 19:54:27 GMT

In this third part of the series, we tackle login issues in IdentityServer caused by cookie restrictions in HTTP and show how to resolve them by implementing HTTPS. We’ll guide you through securing communication between the host, client, and IdentityServer containers and configuring HTTPS in Docker to ensure everything runs smoothly. This blog has been […]

The post IdentityServer in Docker Containers: HTTPS and SameSite (Part 3) appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

"All your base are belong to us" introduction to my 2004 MIT Spam Conference talk

John Graham-Cumming's blog — Sun, 08 Dec 2024 19:28:43 GMT

As I've mentioned before my talk at the 2004 MIT Spam Conference was about using one machine learning system to beat another. In that case a spammer using machine learning to beat a machine learning spam filter.

The talk started with a short video playing off the All your base are belong to us meme. I very carefully edited the images to change the text to be about spam and incorporate Paul Graham (who was the instigator of the MIT Spam Conference and an early machine learning spam filter pioneer).

I thought I'd lost the audio of that introduction and randomly came across it today. So here is the restored introduction to the talk made on a Mac in 2004 (I still haven't found the audio of the actual talk).

When Game Days go wrong

Lawrence Jones — Sun, 08 Dec 2024 14:00:00 GMT

It was the week before pandemic lockdowns began.

Like many companies, we were thinking about what a fully remote workforce might mean for us. In an attempt to get ahead of things, we scheduled a ‘Game Day’ to test our work-from-home incident response capabilities. You know, just in case we’d need to abandon the office (spoiler: we would).

The timing was perfect. We had several new team members and others who were almost ready to join the on-call rotation but needed a confidence boost. Our goal was simple: familiarise people with our systems and processes, deliberately steering away from stress testing the infrastructure itself.

Focusing on how we’d respond remotely was the point of this, so we planned to break things in obvious ways and watch how the incident response unfolded.

As the designated villain for the day, I had a straightforward plan:

Detach a disk from our Postgres VM (running in a three-node cluster using Stolon and Etcd)
Fill up an Elasticsearch disk with junk (running in Kubernetes)
Create tons of consoles to exhaust cluster capacity

Pretty standard stuff. In fact, it sounded like it might even be fun.

All according to plan

For a while it was, with the Game Day starting smoothly.

When Postgres died and recovered, the team opened an incident and handled it well. As they were wrapping up the Postgres recovery, I started filling up the Elasticsearch disk, hoping to nudge the fairly large (~30TB) cluster into a degraded state.

Alerts began firing, and the team efficiently split into two groups to handle both issues. Perfect! This is exactly the type of response we were after, proving we could handle ramping pressure even while remote.

It was time, then, to make things more interesting. So I started creating a load of consoles in the Kubernetes cluster.

The console creation was meant to gradually fill the Kubernetes cluster with batch jobs, exhausting our staging clusters capacity. This should have been safe – we had pod priorities ensuring staging workloads were assigned much lower priorities than production ones. No matter how many staging consoles were created in the cluster there should be no impact on production workloads, as Kubernetes will evict staging over production workloads first.

So when someone from support raised the alarm that the production app was down, I was more than a little confused.

Going sideways

It was at this point that everything started going sideways.

Alerts were firing everywhere: Postgres alerts, HAProxy reporting no backends, blackbox alerts confirming our API was unresponsive, and – most worryingly – Etcd cluster out of quorum. That last one was particularly problematic because Stolon relies on Etcd for leadership election. No Etcd quorum means no database connections, which means no API requests, no batch jobs… no nothing.

The root cause? A perfect storm of “tomorrow’s problems” catching up with us.

At the time, we were running everything (production and staging) in a single large Kubernetes cluster, separating environments and cluster infrastructure using namespaces. We used pod priorities to ensure workloads would arrange themselves correctly – cluster-level infrastructure above production, production above staging, and so on.

But we’d made a critical mistake, one that would come back to haunt us in the worst possible way. It all traced back to when we initially rolled out pod priorities…

The mistake

When we initially rolled out pod priorities, we thought we’d be physically separating the cluster within a month. To avoid doing substantial operational work on something that would be rebuilt “soon,” we’d skipped applying pod priorities to some sensitive deployments, including Etcd.

Here’s where things get interesting: in Kubernetes, pod priorities work through a numeric class system. We had set this up pretty sensibly, with system workloads at priority 300, production at 200, and staging at 100. The idea is simple – when the cluster needs to make space, it evicts pods with lower priorities first.

But there’s a subtle gotcha that bit us hard. Before you introduce any priority classes to your cluster, every pod essentially has equal priority. They’re all implicitly set to 0, but it doesn’t matter because they’re all the same. The moment you add even a single priority class, though, that changes completely – now any pod without an explicit priority gets assigned 0, making it lower priority than everything else.

There exists a ‘globalDefault’ property that can be set for a priority class which means any pod without a class will get assigned that priority. We didn’t have this set in our cluster, though I’d advise people set this if they’re using pod priorities.

By not setting a priority on our Etcd pods while having priorities everywhere else, we’d accidentally marked our most critical infrastructure component as the first thing to be evicted under pressure. When I came along and began filling the cluster with staging consoles (a job I admit to doing with enthusiasm), we started evicting Etcd pods. By the time alerts fired, we’d evicted enough pods to lose quorum, having eaten well into our pod disruption budgets.

This prevented the Etcd pods from successfully rejoining the cluster when restarted. We were, in technical terms, pretty screwed.

Righting the ship

The immediate focus was restoring database access. We deactivated the Stolon Postgres cluster manager and booted Postgres manually, relying on muscle memory for the right commands. We bypassed both PgBouncer and the Stolon fencing proxy, pointing applications directly at Postgres. This brought things back up, albeit in an unmanaged state.

After taking a moment to contemplate our life choices, we formulated a plan using Etcd’s disaster recovery procedure. It took about six hours to bring everything back and perform a Stolon failover. A good chunk of that time was spent running pg_basebackup to restore another node – our database was about 6TB at this point.

It was a long day and not at all what we’d had planned. As drills go though, this was probably the most intense way we could ever have tested our production readiness in a remote context, even if it cost us more than we’d anticipated.

Infrastructure debt doesn’t age well

There’s a pattern in infrastructure work that’s worth recognising: we often defer work because we think a better solution is coming “soon.” But “soon” has a way of stretching into weeks or months, and the work we carefully documented and planned to revisit gets buried under newer, more urgent tasks.

Over time, this creates hidden trapdoors in our infrastructure. When we leave configuration half-applied or work partially complete, we’re not just creating technical debt – we’re fragmenting our team’s mental model of the system. Every “we’ll fix it later” adds another dimension of complexity, often temporal, that someone needs to keep in mind. Eventually, someone (in this case, me) will forget about that complexity and fall right through the trapdoor.

The real test isn’t the outage

Of everything I took from this incident, though, it was how our organisation responded that I found most interesting.

In preparing for an important change (remote work during COVID), while trying to do the right thing for the company (training new incident responders), we accidentally caused a significant outage. It would have been easy to say “well, we won’t be doing Game Days again!” or worse, fire the person (me) who pressed the buttons and wash your hands of their incomptence (ouch).

Instead, my boss, our CTO, caught up with me afterward to say “thank god we found this while the whole team were on a call ready to respond!”. It’s true, this would have been far worse had it happened out of hours, but you bet he’d got a lot of heat for this outage. It would have been easy to push blame to me, but he didn’t, and in that moment he created the essential conditions of a ‘blameless’ incident culture. One where an outage like this can balance its cost with the lessons learned from it.

Sometimes the most valuable lessons come from the most uncomfortable situations. The trick is making sure your team feels safe enough to learn from them.

Three ways to shoot yourself in the foot with Google Cloud Run

Phil Booth's Blog — Sun, 08 Dec 2024 12:00:00 GMT

This post is an expansion of a comment I made on HN recently and continues my Extreme Learning series. Previously I discussed how I’ve managed to break production with Redis, with PostgreSQL and with healthchecks. Now I’ll show you how I did it with Cloud Run too.

For context, we migrated from Managed Instance Groups to Cloud Run at work last year. Cloud Run promises to simplify your production infrastructure and mostly we found that to be true. But we also discovered some hidden gotchas that can catch you out if you’re not paying attention.

1. Use websockets without changing the default request timeout

By default, Cloud Run terminates inbound TCP connections after 5 minutes. If you’re doing anything that uses a long-running connection, communicating via a websocket for example, you’ll want to change that setting. Otherwise at best you’ll have lots of reconnection overhead and at worst you’ll have weird bugs in production that nobody can reproduce in local.

The upper limit on connections in Cloud Run is one hour, so you’ll need proper reconnection logic on clients if you’re running longer than that. (but you should have proper reconnection logic regardless of course, because you’re working over the internet)

If you’re changing this setting in the GCP web console, it should look like this:

Or if you’re using Terraform, you should set timeout = "3600s" in the cloud_run_v2_service resource.

2. Ignore the difference between first and second generation environments

Cloud Run has two separate execution environments, each with its own tradeoffs. The first generation environment emulates Linux (imperfectly) and has faster cold starts. The second generation runs on real Linux and has faster CPU and faster network throughput. If you don’t specify a choice, it defaults to first generation.

For our part, we valued faster network throughput so opted for second generation. That setting looks like this in the web console:

For terraform we set execution_environment = "EXECUTION_ENVIRONMENT_GEN2".

3. Don’t pay attention to autoscaling settings

With the default settings in place, Cloud Run will autoscale up when CPU usage reaches 60% and scale down to zero when there’s no traffic. You may want to change both of those depending on your usage.

In our application, CPU usage turned out to be a suboptimal metric for scaling up, so we tweaked the maximum number of concurrent requests to compensate. We also have periods of very low activity at weekends, so set the minimum instance count to 1 to eliminate cold starts.

Here are those settings in the web console:

In terraform those settings are max_instance_request_concurrency = 50 and scaling { min_instance_count = 1 }.

Improved weather forecasts

I Learnt — Sun, 08 Dec 2024 10:37:00 GMT

At the moment we are deep in the clutches of Storm Darragh as it batters the UK. We have currently lost one fence panel and there may be more casualties. I looks like I will get to experience the “joy” soon of watching my son play football in high winds and torrential rain.

Holding Space for Wicked (Part One)

swyx's site RSS Feed — Sun, 08 Dec 2024 04:57:10 GMT

Embedding Vectors vs. Vector Embeddings

Tanel Poder Blog — Sun, 08 Dec 2024 03:38:11 GMT

Disclaimer: I’m not an ML expert and not even a serious ML specialist (yet?), so feel free to let me know if I’m wrong!

It seems to me that we have hit a bit of an “on-premises” vs. “on-premise” situation in the ML/AI and vector search terminology space. The majority of product announcements, blog articles and even some papers I’ve read use the term vector embeddings to describe embeddings, but embeddings already are vectors themselves!

Humidity and ratios

blog.harterrt.com — Sun, 08 Dec 2024 03:24:00 GMT

My house gets super dry in the winter and I didn't understand why.

My weather station says it's drier inside than outside, which didn't make a ton of sense. I'm cooking, and sweating, and showering inside. It should be more humid.

I suspected my forced hot air system was to …

Small Web

RobKohr's Blog — Sun, 08 Dec 2024 02:00:00 GMT

Small Web

Retail Returns

Craig Pardey — Sun, 08 Dec 2024 02:00:00 GMT

I recently had to return an item that I’d bought online. It’s free to return the item in-person to one of their retail locations, or $10 to ship it back. There was a store nearby so I elected to do an in-person return but the experience shows what can happen when software and processes are developed without involving the user.

The store clerk was friendly and helpful. He asked for my online order number, found it in the system, and told me that the money would be refunded to the original form of payment used for the purchase - in this case, PayPal. Everything is going smoothly to this point.

I Photographed Santa Skivvies Run 2024

Thomas Hunter II — Sun, 08 Dec 2024 02:00:00 GMT

Bash script to switch Pulseaudio sinks

Thomas R. Koll — Sun, 08 Dec 2024 02:00:00 GMT

Who doesn’t love using the keyboard even for simple things like switching the output (or sink as pulseaudio calls it) for you audio? My setup is a set of external speakers, a monitor with rather bad speakers and plugged into the monitor at all time my headphones. That makes three sinks which makes my script a bit more complicated than a simple toggle to the other (i.e. IDLE sink) would be.

Disable Slog Messages in Go Tests

Ben E. C. Boyter — Sun, 08 Dec 2024 02:00:00 GMT

As much as I like ZeroLog for Go logging Slog being added to the standard library fits in with my rule of avoiding 3rd party packages when they are not entirely required hence I have started moving anything I work on over to it.

One thing I find especially annoying however is log messages inside tests. As such here is a quick code snippet which turns off all slog outputs, but only for tests. Stick it into any _test.go file to disable logging outputs for that package.

objc async selectors

Whacky Labs — Sat, 07 Dec 2024 16:21:00 GMT

I very recently learned it the hard way that objc selectors can not be async.

In simple words, this will crash at runtime:

let button = UIButton(frame: buttonFrame)
view.addSubview(button)
button.setTitle("This is a button", for: .normal)
button.addTarget(
  self,
  action: #selector(handleTap), 
  for: .touchUpInside
)
  
@objc func handleTap() async {
  Task {
    debugPrint("handle Tap")
  }
}

No warnings, no errors, no messages on console, nothing. Just crash!

As it turns out all objc selector marked as async will crash. For example:

 perform(#selector(handleTap))

So what is going on here. Let’s first print the selector generated by the compiler for async methods.

debugPrint(NSStringFromSelector(#selector(handleTap)))
// "handleTapWithCompletionHandler:"

As per swift objc interops guidelines, Swift maps all objc methods with suffix "WithCompletionHandler" as `async. So these two are equivalent:

- (void)stopRecordingWithCompletionHandler: void(^ _Nullable)(RPPreviewViewController * _Nullable, NSError * _Nullable)handler;

func stopRecording() async throws -> RPPreviewViewController

So our func handleTap() becomes - (void) handleTapWithCompletionHandler:(CompletionHandler completionHandler) in objc.

The next question naturally is what is the type of CompletionHandler?

I tried to search online if others have also faced this problem. And yes there are quite a number of discussions on this topic. This one even has nice a workaround to the problem:

let completion: @convention(block) () -> Void = {
  debugPrint("completion handler")
}

perform(#selector(handleTap), with: completion)

// prints:
// completion handler
// handle Tap

@convention is a attribute provided by swift for calling conventions in Swift. And @convention(block) is the attribute to represent Objective-C blocks as Swift closures.

Now, with all this acquired knowledge can we fix our crash? We still can’t use the #selector(handleTap) for target action pattern because that pattern is very well documented to support only following method signatures:

func doSomething()
func doSomething(sender: Any)
func doSomething(sender: Any, forEvent event: UIEvent)

Here the sender and event is autofilled by objc runtime while invoking the provided selector. And as you can see there is no place to provide arbitrary arguments. But for this entire mechanism to work we want the sender to be of type @convention(block) () -> Void.

So we can create our own custom UIControl subclass and hijack the target-action mechanism to update the sender. In other words, we need to override the sendAction() method and then call the performSelector.

class MyButton: UIButton {
  override func sendAction(_ action: Selector, to target: Any?, for event: UIEvent?) {
    let sender: @convention(block) () -> Void = {
      debugPrint("completion handler")
    }
    (target as? NSObject)?.perform(action, with: sender)
  }
}

class ViewController: UIViewController {
  
  override func viewDidLoad() {
    super.viewDidLoad()
    let buttonFrame = CGRect(
      x: 20,
      y: (view.bounds.height * 0.5) - 30,
      width: view.bounds.width - 40,
      height: 60
    )
    let button = MyButton(frame: buttonFrame)
    view.addSubview(button)
    button.setTitleColor(.black, for: .normal)
    button.setTitle("This is a button", for: .normal)
    button.addTarget(
      self,
      action: #selector(handleTap),
      for: .touchUpInside
    )
  }
  
  @objc func handleTap() async {
    Task {
      debugPrint("handle Tap")
    }
  }
}

And that is how you can have a button action as async.

References

Pico BLE Vape Remote

xythobuz.de Blog — Sat, 07 Dec 2024 14:00:00 GMT

Like many modern devices, some of the medical vaporizer devices from S&B (the Crafty, Venty and the Volcano Hybrid) offer Bluetooth connectivity and an app to control them.

Unfortunately the official mobile apps have / had some problems. The iOS version is no longer available after Apple decided to remove "drug related" Apps. So understandably the company no longer developed either their iOS or Android apps, instead releasing a web app that uses the Chrome Bluetooth APIs.

Yet all of these variants are problematic in different ways. The Android app offers customizable workflows for the Volcano Hybrid. But the Bluetooth implementation is very wonky, sometimes simply missing steps in the workflow. And because there's no error checking, the workflow simply continues in a broken state. The webapp doesn't have this problem, but instead the workflows are no longer customizable, instead just offering a small number of hard-coded pre-defined workflows.

This is not satisfying, of course, and also presented itself as a great opportunity to play around with BLE.

Fortunately the official web app is made with non-minimized an unobfuscated JavaScript, so the BLE protocol for the different devices can easily be reverse engineered. I wrote a small script that helps with downloading and beautifying the official sources.

As a first attempt I implemented the Volcano workflow functionality in Python on the PC. I had some problems, so I made two implementations, with bleak and SimplePyBLE. Turns out my difficulties came from having multiple bluetooth adapters on my Linux machine. Now the scripts both work fine for me.

Then I implemented the same thing on a Raspberry Pi Pico W, with a Waveshare Pico LCD 1.3 display. I first did this in MicroPython but ran into issues as well. It works fine but I couldn't include all the functionality I wanted.

So I made a fourth iteration, this time in C with the plain Pico SDK. It can interact with the Volcano Hybrid, the Crafty and the Venty. Everything can be controlled using on-screen menus on the device itself. It even has a WiFi capable OTA bootloader to support wireless firmware upgrades.

To contain everything I designed a snug little 3D-printed case in OpenSCAD. It contains the Pico W, a Pico LiPo Shim and a "80mAh / 20 x 11 x 5mm / 501220" LiPo battery. The Pi is mounted with four screws and the battery is held onto it with a drop of hot glue. Then the LCD board just plugs on top to close it.

This is what the final result looks like.

I'm quite happy with it, especially the software part. It has many things I expect from nice modern embedded projects, and I've used a bunch of libraries to achieve these:

Pico SDK for working with the Pico
- TinyUSB for the USB device implementation
- BTstack for the Bluetooth connection
FatFS for the USB filesystem
MCUFont for rendering text
st7789 for interacting with the LCD
picowota bootloader (modified to work with the Flash config storage and LCD)

As usual you can find everything on Codeberg and on GitHub.

Speaking at the .NET Conference Italia 2024

Nicola Iarocci — Sat, 07 Dec 2024 09:36:26 GMT

I’m speaking at the .NET Conference Italia 2024 on Dec 16th in Milan at the Microsoft House. My session is titled C# 13 What’s New and Interesting and will be on the latest iteration of the C# language. We’ll also briefly touch on .NET 9, which was also just released. Hope to see you there (make sure to come to me to say Hi!)

Site design standards

All content on Seirdy’s Home — Sat, 07 Dec 2024 07:29:28 GMT

The accessibility statement and design standards I hold myself to when creating seirdy.one

Best practices for inclusive textual websites

All content on Seirdy’s Home — Sat, 07 Dec 2024 06:08:51 GMT

A lengthy guide to making simple, inclusive sites focused on content before form. Emphasizes brutalist design and accessibility to include under-represented users.

Don't fall in love with your code

Michael Salim Blogs — Sat, 07 Dec 2024 02:00:00 GMT

Sometimes the best code is the code you delete. Don't love your code and do what's best for the codebase.

Family Switch: A new and fun cast for a classic old story

🦉 olshansky 🦁 — Sat, 07 Dec 2024 02:00:00 GMT

Olshansky's review of Family Switch

My personal Bell Labs

Consulting on Adolfo Ochagavía — Sat, 07 Dec 2024 02:00:00 GMT

Have you heard about Bell Labs? Recently, an article about it reached the Hacker News frontpage, triggering a flood of nostalgic comments. Bell Labs is, for many, a symbol of the ideal working environment for a curious mind: a place where there’s time, money and freedom to research whatever you want. In my own case, having a curious mind drew me from a young age towards computers and lured me into the world of programming.

Advent of Code 2024 Day 6: Janet Streams Using Fibers/Coroutines

Sat, 07 Dec 2024 02:00:00 GMT

Remember when it was all the rage to write articles about functional programming (FP) in Javascript by explaining how you can chain .map, .filter and .reduce on lists? I hated that. I think that it conditions people into thinking that FP is all about list comprehensions. And when lists aren’t convenient anymore (more on this later), they then resort to imperative programming, thinking that FP isn’t suited to that kind of task.

In Advent of Code (AoC) there are plenty of examples where eager list comprehensions won’t get you very far. An innocent looking map().filter().map() can consume all your memory and make the garbage collector go crazy if you are creating and re-creating huge lists. A straight forward solution, as I mentioned earlier, is to fall back to more imperative patterns, such as the loop macro in Janet.

I was hoping that I could use Janet’s fibers/coroutines in combination with its stdlib map and filter function to work on streams rather than eagerly evaluated lists. But that was not the case:

(->> (gen-stuff input)
     (map foo)
     (filter bar))

In my testing on AoC day 6 (the first one where you walk around in a grid), the imperative version using loop used about 80MB, the version that kicks things off with a generator and then runs this through various list comprehenions immediately took up around 10GB of memory. Here’s a concrete example:

(defn gen-nums [] (coro (for i 0 10000000 (yield i)))

(comment
  # low mem usage
  (each v (gen-nums) (print v))

  # high mem usage
  (each v (map |(string/repeat (string $) 100) (gen-nums)) (print v)))

But it turns out that it is surprisingly straight forward to have your cake and eat it too. You can use easily define streaming versions of map and filter using the coro function (or macro?):

(defn map* [f ds] (coro (each v ds (yield (f v)))))

(defn filter* [f ds] (coro (each v ds (when (f v) (yield v)))))

I’ve used these for an upated implementation of day 6, which you can find here

Short story fika about being human and Historisk julmarknad at Gamla Uppsala

Liza Shulyayeva — Fri, 06 Dec 2024 23:01:33 GMT

It’s been another eventful week at the new place! I went back to work (the commute is even better than I had expected), hosted my next short story fika, met up with a friend from Stockholm, and went to a Christmas market in Gamla Uppsala.

Kuma Fo by Les Amazones D'Afrique

Nicola Iarocci — Fri, 06 Dec 2024 17:52:41 GMT

Inspired by the historic Dahomey Amazons and founded by three of the biggest powerhouses in African music, Mamani Keïta, Mariam Doumbia, and Oumou Sangare, Les Amazones d’Afrique have been using their voices to advocate for women’s rights since their 2017 debut. The group has never shied away from mashing up tradition and technology. Still, on Musow Dance, with the endlessly inventive production of Jacknife Lee, they lean heavily into an almost entirely electronic sound, turning up the energy several notches with booming 808s, dramatic synth slides, and bursts of vintage disco. But none of these additions ever overshadow the true soul of the music, instead amplifying the already formidable voices of Les Amazones d’Afrique’s ever-evolving lineup (source).

I never want to sleep

callmeolive — Fri, 06 Dec 2024 16:50:00 GMT

There’s nothing better than a good night’s sleep. The problem: I don’t want to sleep, never been a fan of it.

The average well-adjusted person’s day looks a little like this:

My (not average, terribly-adjusted person) day however, looks more like this:

Perpetual Sleep Deprivation

Revenge Bedtime Procrastination is the name of the game. My day is unfulfilling, boring, spent doing a lot of things that I don’t really want to do (that’s life!) and I desperately claw back my time when the evening rolls around. As the world heads to sleep, the night is serene. Quiet, peaceful. After waiting an eternity I can finally pursue my interests free from interruption or judgement.

… And then I keep going. Midnight flies by. Anyone responsible would pack everything away and call it a night, but nah. Going to sleep means I’ll have to trudge through an entire day before I can get back to doing what I want.

Even if I know an early & full night’s sleep will restore me and allow me to perform at twice my best, immediate comfort trumps it. If I’m working on something, I won’t want to stop, so I’ll lock in to finish it regardless of when it’ll be done. If I’m relaxing, well there’s always one more video, always another friend to say hello to, anything to put off the next day.

Unfortunately my responsibilities aren’t as flexible as my sleep, and trying to deal with that conundrum is a nightmare in itself. I’ve spent the majority of my adult life a husk, dark circles permanently etched under my eyes, shambling from one day to another. On the good weeks I can scrape back my needed sleep over the weekend, but Monday is inevitable.

Is it immature? Yep. Is it unhealthy? Absolutely. As if the brain fog and grogginess of perpetual sleep deprivation wasn’t bad enough I can assure you I’m walking into an early grave because of it, be it stroke or heart failure. The sad thing is that despite knowing this, I can’t bring myself to actually head to bed at a reasonable time and get enough sleep.

The causes

Excitement and Motivation - I’m not eager to trudge through the boring stuff of the next day before I can do what I want
Ubiquity of the Internet - 20 years ago the web was smaller, slower, and you’d only have your computer connected to it. The internet was an addition to our day-to-day lives, not a core part of it, and we could easily disconnect ourselves from it.
Bloody Americans - ~~Unfortunately~~ I know a lot of Americans. Because they live in a later timezone and have their own responsibilities, often the only time I can talk to them is in the evening… And then I don’t want to stop.
Nighttime is comfy - As if the inherent thrill of being awake while the world sleeps isn’t good enough, add on the peace & quiet and the fact that you’re finally, truly alone… It’s nice.
I’m not tired - I spend my entire day feeling like death but when the sun goes down I could live forever. ~~Possible Vampirism?~~

Possible solutions (which I will ignore)

No more caffeine - Moot point, I’ve been caffeine-free for 4 years now (or at least, only having decaf!). Across the board I feel better without it, and I can usually nod off in 15-30 minutes… The issue is that I don’t want to.
Avoid Blue Light - I kinda try? I have all my devices set to the lowest brightness with warm Night-light settings practically set to max but… It’s more that having said devices means I don’t want to disconnect.
Disconnect - I mean I do so before 1am, but okay. The issue with programs that shut your devices off after a set time is that I could just switch devices or turn things back on.¹
Grow up, be responsible - I realise that this is the answer, but eughhhhhhh
Do more - Fair. The more I do during the day, the more tired I am when the evening rolls around.
Get something to live for - This one is probably too deep for this blog post. Maybe I’d actually sleep if I was with someone who’d make the boring stuff in the day worth it… But knowing me I’d probably attract a fellow internet vampire and it wouldn’t fix anything.

I don’t really have much else to say. The act of sleeping itself is lovely, and getting a healthy amount of it feels great but I keep denying myself it. Occasionally I’ll have a week or two where I sleep on time and get enough of it, but it never lasts and… I don’t know.

If the day had an extra ~4 hours so I could extra time to myself AND get a full night’s sleep… Honestly I’d waste it by still choosing not to sleep. Oops.

Edit 2024/12/30: For the last two weeks I’ve somehow found myself feeling tired when the evening rolls around. I’ve been able (and willing!) to switch my brain off to sleep, and on the days where I haven’t I’ve ended up conking out before it got too late.

I don’t have a concrete reason why, only a suggestion. I know that Winter can take a toll on my mood to the point where things are less engaging. Without me wanting to lose myself in something, the less chance I have at staying up. Also nothing beats curling up in warm cozy bed during the cold winter nights.

Hopefully I can manage to keep this up through the new year and beyond. Things feel so much better when you actually get sleep.

What pushed me to finish this blog post was a power cut a few days back. Turns out that disconnecting does work, just not if you have the ability to reconnect. ↩︎

Why Askama?

mattrighetti — Fri, 06 Dec 2024 02:00:00 GMT

Askama implements a template rendering engine based on Jinja. It generates Rust code from your templates at compile time based on a user-defined struct to hold the template’s context.

AI Code Assistants: Blurring the Lines Between Categories

Prasham H Trivedi — Fri, 06 Dec 2024 02:00:00 GMT

For long, AI Coding tools have naturally split into specialized roles to solve different developer needs, Inline assistants for real-time suggestions and Multi-file editors for broader changes. Today, these approaches are converging, making our development workflows more efficient than ever. I see two distinct categories of AI code assistants in our development landscape. The inline assistance category, championed by GitHub Copilot and Codeium, excels at real-time completions and suggestions. These assistants are like pair programmers who peek over your shoulder, helping you craft code line by line.

There Is No Universal De-Select-All Idiom

David Buchanan's Blog — Fri, 06 Dec 2024 02:00:00 GMT

Should you be worried about H5N1?

This Is My True Name — Thu, 05 Dec 2024 23:25:23 GMT

Ca-caw!

Non-Identity Problem and robust climate action - practical philosophy seminar notes

Liza Shulyayeva — Thu, 05 Dec 2024 23:01:33 GMT

Last week I dropped in on what turned out to be a seminar in practical philosophy, featuring a paper about resultant moral luck. This week I was back, with more preparation this time - i.e., I read the paper in advance.

Note 153

Qubyte Codes — Thu, 05 Dec 2024 14:54:43 GMT

Back in Fahrenheit Coffee in Toronto.

Modern art = Indie gaming

RobKohr's Blog — Thu, 05 Dec 2024 02:00:00 GMT

Modern art = Indie gaming

LLM chat navigation

xenodium.com @alvaro — Thu, 05 Dec 2024 02:00:00 GMT

LLM chats are often handy for refining answers to a question or task, part of a bigger goal. Navigating the chat transcript, copying and pasting, can be a frequent operation in the bigger goal. If we can do it more efficiently, the better.

While chatgpt-shell offered chatgpt-shell-next-item and chatgpt-shell-previous-item commands, it had a few rough edges which prevented me from fully adopting. The default bindings C-c C-n and C-c C-p didn't exactly help either, making repeated navigation fairly clunky. repeat-mode would have helped a little, yet I was yearning for a familiar experience… more like tab and Shift-tab in web browsers.

While shells often tab-complete commands and/or arguments, I'm not super convinced tab completion is a good candidate for an LLM Emacs shell. Having said that, searching for previous prompts a la Ctr-r is indeed a handy feature and already supported in chatgpt-shell (via M-r).

This is all to say that the with the latest chatgpt-shell navigation changes (using <tab> and <backtab> bindings), the chatgpt-shell experience feels way more natural. You can see it in action…

Tab navigation jumps between prompts, links, and code blocks. You may have noticed code blocks are automatically selected, in case you want to quickly copy and paste elsewhere.

This mode of navigation is also present in the compose UX (via M-x chatgpt-shell-prompt-compose). In addition to <tab> and <backtab>, you can use n and p bindings (post prompt submission).

Give the new navigation a try. See how it feels. Some of it is fairly fresh, so please file bugs if needed.

Malware Insights: GitHub Actions Script Injection

Cookie Engineer's Web Log — Thu, 05 Dec 2024 02:00:00 GMT

How to infiltrate CI/CD runners because they don't sanitize arbitrary string inputs.

Axum, Askama, HTMX

mattrighetti — Thu, 05 Dec 2024 02:00:00 GMT

Lately I’ve been working on Ulry, after tiressly trying to make Next.js work for me, I realised it wasn’t the right fit (I’m not alone). I decided to make a U-turn and go all in with Rust and SSR.

Learnings

Rajeev Singh Naruka — Thu, 05 Dec 2024 02:00:00 GMT

...

Your Next Two Zeroes

taylor.town — Thu, 05 Dec 2024 02:00:00 GMT

Exceed yourself; become a novice again, again and again.

Paper: The Unreasonable Effectiveness of Monte Carlo Simulations in A/B Testing

Bytepawn - Marton Trencseni — Thu, 05 Dec 2024 01:00:00 GMT

I run Monte Carlo simulations of content production over random Watts-Strogatz graphs to show various effects relevant to modeling and understanding Randomized Controlled Trials on social networks: the network effect, spillover effect, experiment dampening effect, intrinsic dampening effect, clustering effect, degree distribution effect and the experiment size effect. I will also define some simple metrics to measure their strength. When running experiments these potentially unexpected effects must be understood and controlled for in some manner, such as modeling the underlying graph structure to establish a baseline.

Bringing K/V Context Quantisation to Ollama

smcleod.net — Wed, 04 Dec 2024 22:00:02 GMT

K/V context cache quantisation has been added to Ollama. This enables significant reductions in VRAM usage, allowing users to realise the potential of expanded context sizes and run larger models at their existing context sizes.

Five Advanced Meditation Techniques (also, drugs)

Superb Owl — Wed, 04 Dec 2024 21:00:42 GMT

A few ways to turn up the volume if you're feeling stuck

20 Years of Blogging and I’m still a n00b

Caseysoftware — Wed, 04 Dec 2024 15:16:00 GMT

As of today, I’ve been blogging for 20 years. Holy crap, that’s a long time. I remember when I started,…

The post 20 Years of Blogging and I’m still a n00b appeared first on Caseysoftware.

Breakage! in the Cargo.toml — How Rust Package Features Work (And Break)

Predrag Gruevski's blog and personal site. — Wed, 04 Dec 2024 02:00:00 GMT

cargo-semver-checks v0.37 can now scan Cargo.toml files for breakage! In this post: a primer on Rust package features, and how innocuous-looking Cargo.toml changes can break your users.

Awesome elisp

xenodium.com @alvaro — Wed, 04 Dec 2024 02:00:00 GMT

A few days ago, redditor gollyned asked about best practices: developing on top of modern elisp packages. It reminded of my modern Emacs lisp libraries post, which I shared with them.

While my post is roughly 4.5 years old, these days I continue to reach out to the likes of seq.el, map.el, subr-x.el and let-alist.el on a regular basis. My post also shared some great third party options. Maybe my post could use an update? Happy to take suggestions.

A few days later, I ran into awesome-elisp, which aggregates a ton of resources to check out. Funnily enough, I had bookmarked it a long while ago and simply forgot about it. On a somewhat related note, when I reviewed my old list of bookmarks, I didn't have Yoo Box's it is not hard to read Lisp code bookmarked, which changed the way I viewed and read elisp code (spoiler alert, as a tree). I remember how well this approach also translated to languages like Objective-C, enabling me to inline more things without worrying too much.

In any case, the reddit post was another reminder for me to go and check out some of those bookmarks I never got to read, including the awesome-elisp one.

Build a search engine, not a vector DB

Saeed Esmaili — Wed, 04 Dec 2024 02:00:00 GMT

If you want to make a good RAG tool that uses your documentation, you should start by making a search engine over those documents that would be good enough for a human to use themselves.

This is exactly what I’ve been trying to communicate in my org in the past few months. It’s 2024 and we still can’t have a proper search engine in organizations to find relevant information from various sources. While this problem remains to be solved, organizations are adapting RAG and AI into their tooling, but are missing the important R of the RAG: Retrieval. I’ve been an advocate of prioritizing search engines over any AI related tool in the past few months, and I found it refreshing to read about this somewhere else:

I hereby pardon all parking enforcement officers

taylor.town — Wed, 04 Dec 2024 02:00:00 GMT

Build mass-transit. Reduce parking wars. Promote non-violent parking/traffic enforcement.

Is humanity a promise yet to be fulfilled? Becoming human at Uppsalas Stadsbiblioteket

Liza Shulyayeva — Tue, 03 Dec 2024 23:01:33 GMT

Tonight I went to a live podcast recording of the Forest of Thought podcast at Uppsalas Stadsbiblioteket. The theme was: Becoming human - the search for soul in turbulent times.

go yolo

blog — Tue, 03 Dec 2024 21:55:26 GMT

Can Rust teach me computers?

vslira's blog — Tue, 03 Dec 2024 21:24:42 GMT

One of the books on my reading list is Bryant and O'Hallaron's Computer Systems: A Programmer's Perspective. As the name implies, it teaches you how computers work. The book uses C since it's a low-level language that "exposes" the machine and it appeals to me because I've been eager to learn both systems-level programming and computer hardware, and this book addresses both areas using C.

Now, the reader might not be aware, but there's currently a low-level language holy war between Rust and C going on (well, more like a two-front war with Rust tackling C and C++ at the same time). Which is fine and all - I've got no dog in this fight - but it got me thinking: Is there some book or piece of content that can teach me the same breadth of hardware knowledge based on Rust? Even though learning is fun, I'd rather learn a language at the same time I learn about hardware.

I don't expect to become a systems programmer any time soon, but I admit that writing, say, high-performance libraries to be called from Python sounds appealing, and Rust seems to be gaining a lot of traction on that front. Maybe I can't escape learning C for the conceptual low-level stuff and then Rust to leverage the new ecosystem, but that's more trouble than I was expecting. I would appreciate any thoughts on how to deal with this question.

Karma ripples

Aswath Krishnan — Tue, 03 Dec 2024 17:54:48 GMT

When you’re unkind to someone, you’re not just creating a fleeting moment of discomfort; you set off waves—within you, within them, and outward into the collective consciousness of those around you. Your behavior leaves an emotional residue and becomes part of your and someone else’s story and neural wiring. They carry that hurt forward—sometimes consciously, sometimes not—spreading it like an emotional contagion to others in their orbit. What starts as a harsh word can grow into a pattern, a culture, or even a generational legacy.

Kindness, on the other hand, is a ripple effect we all wish to swim in. Every small act of generosity or love creates its own chain reaction. Neuroscience shows that acts of kindness release oxytocin and dopamine, not just for the giver and the receiver but for those who witness it too. These interactions shift our neural wiring toward empathy and openness. They teach us—subconsciously—that the world is safer, better, and more collaborative than it might sometimes seem.

Karma, often misunderstood as some cosmic ledger balanced in the afterlife, is far more immediate and scientific. Karma is simply cause and effect. Every thought, word, and action shapes the trajectory of your life and the world around you in real-time. Think of it less like a divine judgment and more like a behavioral algorithm. Each action you take feeds into the feedback loop of your psyche, reinforcing patterns that dictate who you are becoming and how you affect the lives of others.

The world today—its anxieties, conflicts, love, and beauty—is the sum total of these ripples (Theory of Domino Destiny). It’s a mosaic of decisions, many made unconsciously, accumulating into today’s reality. Every moment is an opportunity to choose the ripples you create. You can even be a Karma alchemist, and absorb, transform, or stop negative ripples from others. Your smallest choices—whether to snap at someone, to show them grace, or stand up for what's right—aren’t just about you. They shape the larger web of human experience.

Samsara, another Hindu and Buddhist concept, isn’t just about escaping some mystical cycle of reincarnation. It’s about transcending the daily cycles of suffering we perpetuate through thoughtless actions and negative Karma. Liberation is not found in some distant metaphysical plane—it’s found here, now, in the choice to stop adding pain to the world and to start choosing compassion, connection, and kindness instead. Karma isn’t waiting for you. It’s already in motion, responding to your every move.

The question isn’t whether Karma is happening—it’s whether you’re conscious of the Karma you create. Are you a generator of chaos, or a cultivator of harmony? What ripples are you sending out and within? Because the truth is, those ripples don’t just change the world—they are the world and they are your life experience. Every moment, you have the chance to shape it.

Modules and imports in mgmt

The Technical Blog of James on purpleidea.com — Tue, 03 Dec 2024 09:56:37 GMT

Mgmt modules are used for structuring your code across multiple files. This keeps things readable and allows for code reusability. For example, it should be easy to write a single module which can be effectively used by many different parties. To me, it never felt like code reuse was a core design objective with other tools. In mgmt it has been since the very beginning. I’m very proud of our module system, so let me introduce you to it properly.

Happy perfect square year 45^2 = 2025

RobKohr's Blog — Tue, 03 Dec 2024 02:00:00 GMT

Happy perfect square year 45^2 = 2025

Hammered by attempted hackers

RobKohr's Blog — Tue, 03 Dec 2024 02:00:00 GMT

Hammered by attempted hackers

Resizing AWS Mac Volumes

Dylan Paulus' Blog — Tue, 03 Dec 2024 02:00:00 GMT

I recently ran into an issue with our Mac build machine in AWS, where the EC2 instance kept running out of disk space. It should be easy enough to bump the EBS volume size and call it a day. Wrong. No matter what I changed the volume size to, the Mac instance would not recognize the new space. What gives?
After some digging, I found that Mac requires us to resize the OS APFS (Apple File System) container after providing the volume with more space. Let’s walk through, from the start, how to increase the size of an EBS volume on a Mac AWS instance.

Increase EBS Volume Size

In the EC2, under Instances, select the instance that needs the volume resized.

Once inside the instance, select Storage and the volume that needs to be resized.

Then, with the volume selected, choose Modify Volume from the Actions dropdown.

Update Size to be the new desired size of the volume.

After the volume has been resized, we need to reboot the EC2 instance. Navigate back to the instance dashboard and select Instance State > Reboot instance.
This may take some time for the instance to fully reboot.

But once the instance is back up and in a healthy state, connect into the instance by going into the instance dashboard and selecting Connect.
Use whichever method you prefer to connect to the instance.

Now we can see how much disk space this instance has. To do this run df -h or diskutil:

diskutil list external physical

At this point you will notice that even though the volume has been increased, the OS has not recognized the new space. diskutil and df is showing the old volume size.
The Apple file system doesn't expand automatically to fill the new space to avoid potential data loss, and provide users control over the space on the system.

To expand APFS use the new space run the following commands:

First, repair the disk to fix and prevent any potential problems:

PDISK=$(diskutil list physical external | head -n 1 | cut -d " " -f 1)
yes | sudo diskutil repairDisk $PDISK

Finally, resize the APFS container to take advantage of the new space:

APFSCONT=$(diskutil list physical external | grep "Apple_APFS" | tr -s " " | cut -d " " -f 8)
sudo diskutil apfs resizeContainer $APFSCONT 0

After running the above commands, you will see the new space reflected by again running df -h or diskutil list external physical!

Data Loading - Comparing Common Tooling

Londogard Blog — Tue, 03 Dec 2024 02:00:00 GMT

This blog was supposed to be more in-depth but my enthusiasm was drastically cut and I felt like splitting it up into multiple smaller one, whereas daft one is already uploaded.

I started writing a “recipe-book” for daft where I realized it wasn’t as smoothly integrated as a lot of other tools. I believe that the DataFrame format is both a winning and loosing concept, it’s very helpful but when you need to use two columns the way Ray, HuggingFace Datasets and others map data using dict is a winning concept for both element by element and batch mapping. With a dict way to map DataFrame I think that daft might end up the perfect tool.

For now I believe Daft is better utilized as an ETL framework, but in the near future it might become great for ML too.

Anyhow, today I’ll compare the developer experience and performance of different tools for data loading.

All of the chosen tools are quite awesome, but HuggingFace and Ray can export to TensorFlow additionally. Although Ray currently cannot handle RaggedTensor which is required for models with variable output - a letdown!

Quick Introduction

Hugging Face Datasets offers easy access to a vast library of datasets, with efficient memory handling through streaming and memory-mapping. Its API simplifies data loading and transformation for direct use with PyTorch and TensorFlow.

Ray Data enables scalable, distributed data processing across multiple nodes, ideal for large datasets. It integrates with Ray’s ML tools for parallel training and distributed transformations. It’s the tool for Large Language Model training, even embraced by OpenAI in their ChatGPT source.

Daft is a high-performance data processing library with lazy evaluation, optimized for structured data formats like Parquet and Arrow. It’s a strong choice for single-node and multi-node data preparation with PyTorch compatibility. It utilizes Ray to achieve multi-node behavior.

PyTorch’s Dataset and DataLoader offer a simple and flexible way to load data with minimal memory overhead, ideal for in-memory and custom datasets. It’s lightweight but lacks distributed and lazy loading features.

Table Summarization
Feature	Hugging Face Datasets	Ray Data	Daft	PyTorch Dataset + DataLoader
Parallel Processing	+	+++	++	+
Distributed Processing	0	+++	+++	0
Caching & Memory Mapping	+++	+	+	0
Lazy Loading	+++	++	+++	+++ (depends)
Simple to Use	+++	+	++	+++
Built-in Dataset Access	+++	0	0	+++
Custom Transformations	++	+++	+++	+++
ML Framework Support	+++	+++	++	++

Mini Benchmark

Tool	Num_worker	Pin_memory	Cache	Configuration	Time
HF Element	None	None	False	.map	6m48s
	None	None	True	.with_transform	3m23s
HF Batched	None	None	False	.map	7m14s
	None	None	True	.map	3m22s
Torch Dataset/Loader	None	None	-	Default	3m20s
Daft	-	-	-	daft-default	14m55s
	-	-	-	daft-native	3m30s
Ray	-	-	-	Default	7m41s

Running on full sized images we get a bit more interesting results:

Tool	Num_worker	Pin_memory	Cache	Configuration	Time
Additional Tests	-	-	-	torch	4m19s
	-	-	-	hf_with_transf	4m40s
	-	-	-	hf_map	8m14s, cached: 7m21s
	-	-	-	daft	3m49s

Developer Experience (DX)

def test_elem_by_elem(num_workers: int | None, pin_memory: bool | None, cache: bool):
    if not cache:
        datasets.disable_caching()

    def _preprocess(data: dict):
        imgs = [utils.PREPROCESS_TRANSFORMS(x.convert("RGB")) for x in data["image"]]
        data["image"] = imgs
        return data

    ds = datasets.load_from_disk("./imagenette_full_size")

    def _augment(data: dict):
        tensor = _preprocess(data["image"])
        data["image"] = utils.AUGMENTATIONS(tensor)
        return data

    ds_train = ds["train"].with_transform(_augment)
    ds_valid = ds["validation"].with_transform(_preprocess)

    kwargs = dict(
        num_workers=num_workers or 0,
        persistent_workers=bool(num_workers),
        pin_memory=pin_memory,
        batch_size=32,
    )
    dls_train = torch.utils.data.DataLoader(ds_train, **kwargs)
    dls_valid = torch.utils.data.DataLoader(ds_valid, **kwargs)

class ImagenetteDataset(Dataset):
    def __init__(self, hf_dataset, preprocess=None, augment=None):
        self.hf_dataset = hf_dataset
        self.preprocess = preprocess
        self.augment = augment

    def __len__(self):
        return len(self.hf_dataset)

    def __getitem__(self, idx):
        data = self.hf_dataset[idx]

        image = data["image"].convert("RGB")

        # Apply preprocessing and augmentation if specified
        if self.preprocess:
            image: torch.Tensor = self.preprocess(image)
        if self.augment:
            image = self.augment(image)

        return {
            "image": image,
            "label": data["label"],
        }
train_dataset = ImagenetteDataset(
    ds["train"], preprocess=utils.PREPROCESS_TRANSFORMS
)
print(len(train_dataset))
valid_dataset = ImagenetteDataset(
    ds["validation"], preprocess=utils.PREPROCESS_TRANSFORMS
)

# Create DataLoader instances
kwargs = dict(
    num_workers=num_workers or 0,
    persistent_workers=bool(num_workers),
    pin_memory=pin_memory,
    batch_size=32,
)
dls_train = DataLoader(train_dataset, shuffle=True, **kwargs)
dls_valid = DataLoader(valid_dataset, **kwargs)

def load_imagenette_datasets_daft(dataset_path="./imagenette_full_size"):
    ds = datasets.load_from_disk(dataset_path)
    extract_img_bytes = daft.col("image").struct.get("bytes").alias("image")
    ds_train = daft.from_arrow(ds["train"].data.table).select(
        "label", extract_img_bytes
    )

    ds_valid = daft.from_arrow(ds["validation"].data.table).select(
        "label", extract_img_bytes
    )

    img_decode_resize = (
        daft.col("image").image.decode(mode="RGB").image.resize(224, 224)
    )

    ds_train = ds_train.with_column("image", img_decode_resize)
    ds_valid = ds_valid.with_column("image", img_decode_resize)

    def to_f32_tensor(ds: daft.DataFrame):
        return ds.with_column(
            "image",
            daft.col("image").apply(
                lambda x: (x / 255.0).transpose(2, 0, 1),
                return_dtype=daft.DataType.tensor(daft.DataType.float32()),
            ),
        )

    ds_train = to_f32_tensor(ds_train)
    ds_valid = to_f32_tensor(ds_valid)

    ds_train = ds_train.to_torch_iter_dataset()
    ds_valid = ds_valid.to_torch_iter_dataset()

    dls_train = torch.utils.data.DataLoader(ds_train, batch_size=32)
    dls_valid = torch.utils.data.DataLoader(ds_valid, batch_size=32)
    return dls_train, dls_valid

def load_imagenette_datasets_ray(dataset_path="./imagenette_full_size"):
    # Load the Arrow dataset with Ray
    ds = datasets.load_from_disk(dataset_path)

    def extract_img_to_pil(data):
        image = data["image"]["bytes"]
        data["image"] = PIL.Image.open(io.BytesIO(image)).convert("RGB")
        return data

    ds_train = ray.data.from_huggingface(ds["train"]).map(extract_img_to_pil)
    ds_val = ray.data.from_huggingface(ds["validation"]).map(extract_img_to_pil)

    preprocess_transforms = transforms.Compose(
        [
            utils.PREPROCESS_TRANSFORMS,
        ]
    )
    augmentation_transforms = utils.AUGMENTATIONS

    # Apply transformations in Ray
    def preprocess_image(batch):
        batch["image"] = [preprocess_transforms(x) for x in batch["image"]]

        return batch

    def augment_image(elem):
        elem["image"] = augmentation_transforms(elem["image"])

        return batch

    ds_train = ds_train.map_batches(preprocess_image).map(augment_image)

    ds_val = ds_val.map_batches(preprocess_image)

    d_train = ds_train.to_torch(
        label_column="label",
        batch_size=32,
        local_shuffle_buffer_size=512,
        prefetch_batches=5,
    )
    d_valid = ds_val.to_torch(label_column="label", batch_size=32, prefetch_batches=5)

    return d_train, d_valid

I think most of the frameworks ends up at a similar place in the experience.

Quick DX Ranking

PyTorch & HuggingFace Datasets
Daft
Ray (albeit I believe it to be the most scalable solution as you can truly tinker in detail)

I enjoyed Daft a lot with its multi-modal syntax, inspired by polars with namespaces (e.g. .image.decode()), which can be phenomenal. Working with DataFrame’s is a cool addition, where you can drop into python simply by using apply.
Working with Daft more and more I noticed that the DataFrame syntax sometimes becomes a big blocker and the simplicity of HF Datasets and Ray using dict’s in .map statements results in easier code and smoother integration with existing libraries.
Additionally HF Datasets / PyTorch DataLoaders feels more pythonic, where the latter is real simple. I can’t put my finger on it but they just seem easier to debug and understand.

It’ll sure be interesting to follow the progress being made, and I’m happy the dust isn’t settled yet!

The Timeless Way of Software

taylor.town — Tue, 03 Dec 2024 02:00:00 GMT

In this talk, we'll cover "living" systems, sustainable design, pattern languages, and practical toolkits for experimenting with the principles.

Self-guaranteeing promises

Steph Ango — Tue, 03 Dec 2024 02:00:00 GMT

Companies break promises all the time. A self-guaranteeing promise is verifiable and non-reversible. It does not require you to trust anyone.

File over app is a self-guaranteeing promise. If files are in your control, in an open format, you can use those files in another app at any time. Not an export. The exact same files. It’s good practice to test this with any self-proclaimed file-over-app app you use.

“Stainless steel” is a self-guaranteeing promise. You can use water to test any tool that makes this promise. The stainlessness of steel cannot be withdrawn.

Terms and policies are not self-guaranteeing. A company may promise the privacy of your data, but those policies can change at any time. Changes can retroactively affect data you have spent years putting into the tool. Examples: Google, Zoom, Dropbox, Tumblr, Slack, Adobe, Figma.

A self-guaranteeing promise about privacy gives you proof that the tool cannot access your data in the first place.

Encoding values into a governance structure is not self-guaranteeing. Given enough motivation, the corporate structure can be reversed. The structure is not in your hands. Example: OpenAI.

Open source alone is not self-guaranteeing. Even open source apps can rely on data that is stuck in databases or in proprietary formats that are difficult to switch away from. Open source is not a reliable safeguard against the biases of venture capital. Examples: Omnivore, Skiff.

When you choose a tool, the future of that tool is always ambiguous. On a long enough timeline the substrate changes. Your needs change, the underlying operating system changes, the company goes out of business or gets acquired, better options come along.

It is possible to accept the ambiguousness of a tool’s future if you choose tools that make self-guaranteeing promises.

Trying out logs and TILs

Erethon's Corner — Tue, 03 Dec 2024 02:00:00 GMT

I often find myself wanting to blog about something, but I get stuck on trying to write a blog post that's perfect, which is a big mental barrier. The longer it takes me to publish something, the harder it gets, hence why I haven't posted about my updates since May. I'm also guilty of having multiple long essay-like posts, that are half-finished for years now, which apply a constant pressure in my posts backlog, making the situation worse.

As an attempt to fix this, I'm giving TILs and log entries a try.

Failover Replication Slots with Postgres 17

Gunnar Morling — Tue, 03 Dec 2024 02:00:00 GMT

Table of Contents

Hello, Failover Slots!
Failover Slots in Decodable
Wrapping Up

Postgres read replicas are commonly used not only to distribute query load amongst multiple nodes, but also to ensure high availability (HA) of the database. If the primary node of a Postgres cluster fails, a read replica can be promoted to be the new primary, processing write (and read) requests from thereon.

How much space do Montreal's cars take?

David Schlachter — Tue, 03 Dec 2024 01:50:00 GMT

How much surface area is occupied by cars in Montreal? I make an estimate.

An Anxiety Attack in Daytona Beach

The Digital Nomad — Tue, 03 Dec 2024 01:48:48 GMT

We took a quick trip to Daytona beach to see my friend’s band play at Punk Rock Pizza.

He is the lead singer of Anxiety Attack. It’s an independent band.

I first met the lead singer as the bar tender downstairs. I just started going down to the bar to hang out with him on the weekend and found out he was in a band.

It was fun, different and the food was good at Pirate’s Pizza.

A talk on Polar Gothic literature at Chimära Bokhandel

Liza Shulyayeva — Mon, 02 Dec 2024 23:01:33 GMT

This evening I went to another presentation at Chimära Bokhandel & Butik. The topic this time was Arthur Conan Doyle’s story The Captain of the Polestar and Polar Gothic literature.

Advent of Code 2024 Day 2: Parsing Expression Grammars

Mon, 02 Dec 2024 10:08:55 GMT

I am fully committed to using Janet this year! At least I hope so. One of my goals is to get really familiar and fluent with using parsing expression grammars (PEG). I’ve always liked parser combinators, which seem to be more or less the same thing. One of the most basic and typical ways to parse input in Advent of Code is to split a string into a lines and split each line on spaces.

1 2 3
4 5 6

This an be achieved quite elegantly with a somewhat more recent addition to the PEG library, split. You give it two patterns, one for the separator and one for the actual contents, and it generally does the right thing. There is just one caveat. In the following snippet, there’s a space at the end of my input string. If you remove the asterisk * from the digit pattern, you get no matches. So split more or less requires you to make the content optional, unless you are absolutely certain that you won’t have a dangling separator at the end of your input.

  (peg/match ~(split :s ':d*) "1 2 3 ")

By making the second pattern optional, the end of the string, “3 “ can now be matched as two digits separated by a space. The second digit is simply empty. Sounds totally logical, I know. But you probably don’t want an empty capture. You can either remove it later through filter or you can call string/trimr on the input, which is my convention for AoC. This removes trailing whitespace, so you no longer need to make the second pattern optional, like this:

(peg/match ~(split :s ':d) (string/trimr "1 2 3 "))

Of course you’ll have to adjust your code for other separators (or filter it out later).

Anyway, back to the real world (of AoC). Here’s the same PEG with and without using split. I think it’s a huge improvement.

# before
(def input-peg
  (peg/compile
    ~{:main (* (some (* :line (? :s))) -1)
      :line (group (some (* :num (any " "))))
      :num (/ (<- :d+) ,scan-number)}))

# after
(def parser
  (peg/compile ~(split "\n" (group (split :s (any (number :d+)))))))

What I've learned running Linux and Windows off of USB flash drives

./techtipsy — Mon, 02 Dec 2024 06:00:00 GMT

I’ve recently made an effort to reduce the amount of tech stuff that I own. Lack of space, lack of time and the guilt of storing perfectly functional hardware unused in a box were the main motivators.

This has resulted in experiments with my assortment of USB flash storage that I’ve acquired over the years. They’re too old and cheap to be worth selling, so why not run them to the ground?

I’ve always liked the idea of USB sticks. They’re small and the good ones are actually surprisingly fast, especially compared to hard drives. This is why I started installing operating systems on them. It’s really handy to have a USB stick around with a full operating system on it for testing purposes, and I’m too cheap to buy an IODD device.¹

What I’ve learned is that Linux installed on a USB stick is a much nicer experience compared to a Windows To Go installation, even on the cheaper and crappier ones.

The Windows experience

I’ve used Rufus for creating portable Windows installations in the past, mostly on portable SSD-s. These have come in handy when I’ve had to update the firmware on certain PC-s or associated hardware, like USB-C docks.²

Recently, I took a Samsung FIT 128GB USB 3.0 drive to see how well Windows can operate off of it.

It’s horrible.

Windows will run, but the first time you boot it, you will have to wait for all sorts of driver and update installations, which puts a huge strain on the storage, making the whole experience borderline unusable. Once that process is complete (multiple hours later), it’s actually not that bad, until you receive yet another Windows update.

For a USB stick that you use only occasionally to troubleshoot hardware, Windows is a really poor choice. Even on a proper portable SSD, it will still be relatively slow due to all that driver and update installation taking place in the background.

Eventually, that installation will also probably brick itself in one way or another, and not even due to any storage failures. Simply using the same disk on multiple machines will accrue configuration and driver cruft that likely results in Windows giving up. It’s just a guesstimate on my side, but I would not be surprised if it was true.

After a Windows update run completely fried one USB stick³ that I used for testing, I gave up on it.

The Linux experience

Most Linux distributions have supported a LiveCD/LiveUSB solution for at least 15+ years at this point. It’s a neat idea: your installation media also doubles as an emergency portable installation of that Linux distro, which you can use to quickly test the functionality of a PC, repair an existing installation or install it on a new machine.

The downside of that solution is that the software on it cannot be changed. You can temporarily install new software or update existing packages during that session, or have a separate writeable partition set aside on the USB stick for storing any permanent changes, but it’s a bit of a hassle to think about that nuance every time.

This is why I went ahead and installed Fedora Linux 40 on a Sandisk Ultra 16GB USB 3.0 flash drive (the one that is actually 16GB). The storage speed is still a bottleneck, but it’s much less noticeable during normal use.

After the installation is done, you’re simply using the system as-is. Opening programs will have a slightly longer delay, but it’s still a perfectly cromulent Linux experience.

A fresh installation of Fedora Workstation 40 uses only 3.6GB of storage! On a 16GB USB flash drive, this leaves plenty of room for performing system updates and installing a small number of applications. If you choose the btrfs filesystem during the installation (the default option), then the files will also be compressed to save even more space and increase the effective storage bandwidth.⁴

Disk usage on a fresh Fedora Workstation 40 installation.

You’ll only notice that you’re running off of a USB flash drive when installing new packages, performing system updates or downloading large files. At least with Linux you have control over when you want to install the updates.

It's not terribly fast, but it's not terrible!

Using btrfs also allows you to easily check the flash drive for any integrity issues. Simply run sudo btrfs scrub start -B / and you’ll know soon enough if your flash drive is messing with your data.

The backerupper USB stick

One actual use case that I have for a full Linux installation on a USB stick is the backerupper solution: a fully featured Fedora Linux installation, stored on a small USB stick, and with limited access to networked storage.

Most people working in IT know the joys of working with hardware that family members and relatives own. Before doing any destructive changes, it’s always a good idea to make a full disk copy of the storage on the machine you’re fixing. If you screw up, you’ll have a way of setting everything back just as it was before you did any work on it. A full disk image will also mean that nothing will be left behind before you do a fresh installation.

I already have a home server with some spare storage, so using it as a target for any backups makes perfect sense to me.

This solution has already come in handy once when I had to deal with a 10-year-old laptop with a dying hard drive.⁵

Crappy storage is still crappy

Don’t expect any kind of longevity with USB sticks, the wear leveling and endurance is not on the level of proper SSD-s. Linux is nicer to the USB stick regarding the wear and tear, but given enough time, the USB flash drive will still fail.

I will still keep using USB flash sticks for this purpose. I have backups of the USB stick installations on my server, so restoring it will simply mean getting another USB stick and writing the full disk image back to it.

USB sticks are also great for specialized solutions that don’t do a lot of IO operations. LibreELEC, a media-center oriented Linux distro, can be run off of a USB flash drive without any major issues, since its storage demands are very small (unless you install a ton of plugins).

Conclusion

Linux and USB flash storage is a great option for making backups or having a working operating system available for disaster recovery scenarios. Just mind the endurance of the flash storage, and don’t use Windows.

I won’t trust Ventoy until they address this issue. ↩︎
Linux can also install firmware updates on some machines and hardware, but it’s not a guaranteed success. ↩︎
only the third time that I’ve actually fried a solid state flash storage device through write abuse in my life. Flash is actually pretty good! ↩︎
compression allows us to push more data through a small pipe, be it a slow storage device or the network, at the cost of additional CPU usage. ↩︎
I utilized my sick tech skills to force a full rewrite of the disk to reduce the number of pending bad sectors, and that made Windows happy enough to run on it once again with a fresh installation. It’s a ticking time bomb, but the user is well aware of it and just wants to have a working backup laptop around. ↩︎

Exploring the Model Context Protocol with Deno 2 and Playwright

shruggingface.com — Mon, 02 Dec 2024 05:13:00 GMT

Every Mantra has a Counter-Mantra

🦉 olshansky 🦁 — Mon, 02 Dec 2024 02:21:51 GMT

Including this one

Evolving my ergonomic setup (or, my laptop with extra steps)

ntietz.com blog - technically a blog — Mon, 02 Dec 2024 02:00:00 GMT

My computer setup attracts attention when I'm out and about. This has two effects: engineers^[1] ask me about it, and everyone else ignores me. These effects are not undesirable, but further testing is required.

The main reason I have such an unusual setup, though, is more practical: so my arms/hands do not hurt from using my laptop. I wrote about the second iteration before, and a lot has changed since then. Now it's time to record all the previous iterations^[2] and then detail what's new.

Prototypes 1 through 3

Prototype 1: chonky tray

The first prototype is what I lovingly call the chonky tray. It was really a portable desk, with a recess routed into it so that the laptop stand could sit inside of it. This prevented that from sliding off, mostly, and gave a flush surface under it for the keyboard.

This setup worked, sort of. I could use my keyboard in my armchair in our living room—but I could not use it in the armchair in my office, because it was too wide!

It was also way too heavy and too large to move around the house, let alone anywhere outside the house. This first prototype really served only to ignite my desire to have something, and to show me some things that don't work. But I kept hurting myself by not using it.

Prototype 2: luggable in backpack

The second prototype is the one that was detailed in my previous blog post. It was motivated by this realization that I was going to visit the Recurse Center, a programming nerd enclave in NYC, and could not take my laptop with me since I had no portable way to use it without injury^[3]. So, I made this!

This one worked pretty well for me! It let me fit my whole setup in my oversized backpack, and I could easily take it out of there and use it on the go. I used it on the train ride up to NYC, inside RC's headquarters, and in coffeeshops.

But it wasn't perfect. It was a bit heavy, though with the keyboard I was using, there wasn't much to do about that. Getting my laptop in and out was annoying, since I had to unscrew a nut before sliding it out. And the rigid mounts for the keyboards were okay: they worked, but lacked some adjustability.

The biggest problem, though, was that it did not fit in my personal item bag. And my oversized backpack doesn't fit under all airline seats. And I'd just been accepted to speak at a conference.

Uh oh.

Interlude: fever dreams

Around this time is when I got very sick. I'm recovering now, and I'll write more about it all another time—it's quite the story. But the relevant bit here is that I spent a lot of time during the summer in bed or lying down for radiology procedures.

A funny thing happens when I have to lie down and rest for long periods of time: I start designing things. I was extremely fatigued and had near constant pain, and designing something was a perfect way to move through all of that.

One potential design, using a lot of steel and magnets, came from a morning at home in bed. But the actual next iteration came from a two hour radiology procedure. For that one, I had my arms pinned at my sides and could do little except stare up at the ceiling while they took images^[4].

And so I designed what you'll see next! There are some design flaws, but I'm proud of the experimentation, and it was a crucial step toward my current iteration. And it was some good practice using HeadCAD for a design, then seeing if I could transform that design into a real object.

Prototype 3: fabric hinges

The third prototype is the one I took to give that conference talk. It was interesting, a successful experiment, and it lasted about for the duration of that trip.

The successes were that it was very light and had a much thinner base. It also had a modular design, and I ended up reusing the laptop holder on my current iteration, so that design I'm quite happy with. And I started using hook-and-loop fasteners on it for attaching peripherals!

One of the coolest things I did here was design some hinges out of fabric, with hook-and-loop for adjustment. They started out rock-solid, and the design worked better than I expected! I did have to make sure to align things in a particular direction with the fabric to ensure there was no stretch in the hinging direction.

Unfortunately, since I made things out of thin wood and fabric to keep weight low and keep adjustability high, it started to fail early. The glue I used started to fail on one of the keyboard hinges, making the whole thing a bit floppy (the other one stayed rock solid and I still really like that design, I just need to figure out the glue situation).

I also learned that the thinner base isn't entirely a benefit. With the reduction in total height, it let my shoulder drop down a little more and brought back some of my prior nerve issues! This actually led my physical therapist to discovering the cause of them, which is a win. But it influenced the next design.

The biggest problem, though, was that while it packs up for travel, doing that requires disassembly. This was not a fast process, taking at least a few minutes to tear down or put back up.

Latest iteration: Tray.One ("tray dot one")

The fourth prototype is the current one I'm on, and it has addressed the prior issues for me. I've been using this one for a little while now and there is almost nothing I want to change about it, so I'm done experimenting and might make a nicer version of the same design.

This one really nails it for me: it's light, it is more ergonomic (for me), it's usable in more places, and it's far easier to travel with.

The fully deployed configuration looks similar to some previous iterations, but you'll note the keyboard is different. Also that it looks like everything is scrunched forward on the tray. I move the peripherals forward and back sometimes to have more variety in positioning, and here they happen to be further back since I was taking photos and things slid a bit.

I abandoned my Keyboardio Model 100 for a Corne keyboard, which suits me a little better right now. This was a big change at first but only took a week or two to get back to full speed. The main advantages for me here are that it's smaller, lighter, and has lower force required per key, so I end up moving my fingers a lot less. This keyboard doesn't have mouse support like the Keyboardios do, though, so I picked up a Ploopy trackball mouse and have been loving it.

The most notable thing about this setup is the party trick: it folds in half. This is useful both for travel and for general use. Sometimes I use it folded just to have more desk or table space in front of me!

Here it is folded to the smaller configuration.

To fold it, I lift up the trackball mouse and take out one screw and loosen the other (so I can swing the plate out of the way), then fold it! This is way faster than my previous iteration, which required pulling apart a few different hook-and-loop parts and taking off multiple screws.

It's also more convenient since it fits into my airline personal item bag fully loaded when folded. I just take out that one screw, take off the laptop (which fits in another compartment of the same bag), and stow it away with everything on it. It's made it so I can use my computer in spare moments while traveling.

And this design cuts down on weight a lot. It's down to just 3.5 pounds (with peripherals, without the laptop) and is pretty comfy to carry around and use. It's 6.5 pounds if you include my laptop, which is well below the usual post-surgery weight restrictions of not lifting over 10 pounds. It makes a big difference for one-handed carrying it around the house!

Another thing I carried over from a previous iteration was the use of hook-and-loop fasteners. I have it on the front and back of the laptop holder, and I stick peripherals on there. It really helps things stay neat and tidy!

This setup is not perfect. I need to reattach the hinges and make sure the screws go in exactly straight, since right now they touch when fully folded and it means the halves spring apart a bit. This doesn't impact using it folded, but magnets aren't enough to keep it folded while carrying it. And it's also not finished, literally: I haven't applied wood finish to it. But it is a fundamental improvement over even my previous setup, and it has made it dramatically easier to use my computer in a healthy way.

My workshop: a joy and a privilege

I have the privilege of having a nice workshop where I can make things like this. This takes time, space, and money to establish. I planned for it—when we were buying a house, we had potential workshop space as one of the criteria. But even that, buying a house, isn't within reach for everyone.

It's a joy to have it. Being able to step out to my garage at any time and just work on physical things instead of the ephemeral abstractions we usually deal in is really wonderful. And when I have a problem? I can go solve it.

I've had some sort of tools and workshop for a decade now. It took me that long, a decade, to get to the point where I was able to make each prototype in a morning.

If you need something like this laptop tray, reach out to me. This is life-changing equipment for me, and I want to make sure that people have it if they need it, so I'd make them for close to at-cost. (The caveat is that this design only works for laptops with 180 degree hinges, and I'm not sure how I'd update the design for laptops like a Macbook.)

Bonus: my projector mount

Okay, here's another little thing I built.

Something else I started doing while sick is working lying on my back, using an ultra short throw projector pointed at the ceiling. There were times I needed to lie that way to get my heart rate back to normal, or just to avoid any physical exertion, and being able to use a computer was great!

I'm doing a lot better now, but still like to use my computer this way sometimes. However, my projector was just sitting on our bed headboard, and it was easy to bump and move. So I didn't use it much, because it was inconvenient. (Are you seeing a trend?)

So, at the end of October, I spent 20 minutes in the workshop and made a projector mount from scrap wood! I used a few nuts and bolts I had on hand, and another hinge like I had for the keyboard tray.

Now it's rock solid! It's easy to move the mount around if we need to take it off the bed and put it back, and it stays in a steady angle. You can adjust the angle pretty easily, too.

* * *

Okay, that's all I've got! If you have unique homemade ergonomic setups, I'd love to chat with you about them. And if you want plans to make something like mine, or you'd like me to make you one, feel free to email me.

I've mostly had software engineers ask me about it, but also some others like a mechanical engineer! ↩
In part, this is so that I can finally turn some of the previous iterations into scrap wood. They have decent size pieces of plywood that I could use for other things! But I don't want to recycle them until I document them, so here that is. ↩
That is, except my voice coding setup with Talon. That does have its own drawbacks, though, and I was pretty rusty with Talon then. It's not the easiest to use in public or noisy environments. ↩
Well, it was this and talk about CrowdStrike. The morning of my procedure was when that was all going down. Their department had two working computers; I was lucky the one they needed for me was up. The IT people were in and out during the procedure to manually fix a bunch of PCs. ↩

Colt's Revolutions

taylor.town — Mon, 02 Dec 2024 02:00:00 GMT

hungry for gun funds

Run a Bluesky PDS From Home

Justin Garrison's Homepage — Mon, 02 Dec 2024 02:00:00 GMT

Take ownership—and responsibility—of your social data

/tmp is not a tmpfs on NixOS

Erethon's Corner — Mon, 02 Dec 2024 02:00:00 GMT

The other day I realized that /tmp/ on my NixOS installations is not a tmpfs as I'm used to from other distros. Instead, NixOS relies on this systemd timer that cleans up old files from /tmp/. This is not NixOS specific, other systemd based distros also run this timer. Looking at the NixOS boot.tmp related options, we see there's an option (cleanOnBoot) that clears files on boot and is disabled by default.

A quieter place

bayindirh — Mon, 02 Dec 2024 02:00:00 GMT

While this blog was silent, my life was buzzing with activity. I have been trying to tend to multiple big projects at work and in my personal life, like a juggler trying not to drop anything.

Not everything went as planned. The blog got sidelined, personal projects lost momentum, and a couple of them are temporarily frozen. On the other hand, I managed to finish a big migration at work. As always, Murphy learned what we were doing and motivated us to finish faster by intervening at the most convenient time with maximum effect (at least from his perspective). A big pile of important things got handled in my personal and work life, keeping me busy or at the edge, or both.

Personally, I managed to get my digital garden to grow, write more Go, and love it in the process. I decluttered my life further in digital and physical realms. Decluttering things take a long time when you're trying to handle decades of clutter collected by a past me who didn't understand the consequences of what he's doing. On the other hand, we didn't have these organizational superpowers back in the day, and I didn't know better, to be honest.

This decluttering process is not limited to my collected cruft. I'm simplifying my digital life and the places I frequent. I'm not using X (formerly Twitter) and not adding new photos to my Instagram account anymore (since I don't want to provide AI training material to Meta, a subject worthy of its separate blog post). Instead, I moved to Mastodon and will use another service to share my photography. Recently, I subscribed to a high-quality local newspaper's digital edition.

These moves are a new iteration of the trend I started by switching to Kagi in 2023 (which again needs a separate blog post), which profoundly affected my daily life. Using a user-centric search engine without ads has an immediate and measurable effect on life quality since you don't mentally filter ads and other SEO spam with every search, which is mentally draining.

I felt the same after subscribing to the local newspaper. Higher editorial quality, almost no ads, and access to the paper's print version in PDF form made a profound difference in how I consumed the news and felt afterward (i.e., not mentally exhausted).

The movement from popular, mainstream sites to well-selected, smaller, yet higher-quality environments made me realize how advertisements, feed algorithms, and optimizations for eyes drain our willpower, energy, and capacity to focus and do meaningful work. It's the same with useless discussions with people one doesn't know and doesn't want to agree on anything.

From now on, I'll be much more selective about where I frequent, who I engage with, and how I spend my mental energy and focus. I'll build a quieter place because I want to be tired for more meaningful reasons when I go to bed.

Until next time,

Be kind.

The next platform

Mac's Tech Blog — Mon, 02 Dec 2024 02:00:00 GMT

As someone who has built a career on Kubernetes, I'm always thinking about what "the next platform" is (for job security purposes). By "platform", I mean the kind that platform engineers like myself build for internal dev teams on which to run their applications. Kubernetes has been the star platform for quite a few years now (or maybe that's just my internet bubble), but will that last forever?

Despite the fact that I stan Kubernetes on this blog, I understand more than most that it can be quite painful, so I've been on a search for "the next platform" lately. Let's start with a rough list of problems with Kubernetes-based platforms and a list of requirements for "the next platform".

Problems with Kubernetes-based platforms

You could train a large language model on Kubernetes complaints alone, but I'll just list some common ones:

Steep learning curve.
Lots of moving parts.
Requires a dozen other CNCF projects to do useful work.
Helm templating was a mistake but it's too popular now.
Everything has to be containerized, which complicates dev environments and slows down CI/CD.

And some of my hot takes:

Kubernetes workloads are insecure by default. Too many CNCF projects simply don't work with safe securityContext settings or least-privilege RBAC, and these insecure defaults are footguns for devs.
Manually tuning CPU/RAM requests and limits should go the way of the rotary phone. They should be automatically tuned at runtime.
Kubernetes' killer feature is self-healing, but too many apps are simply not designed with resilience in mind.

Needless to say, there's a lot of room for improvement. But we have to make sure "the next platform" checks all the boxes.

List of requirements for "the next platform"

This is not an exhaustive list either, but we should probably have things like:

Ability to accept HTTP requests from the Internet.
Ability to store state somewhere that is highly-available, secure, and backed up.
Ability to manage app configuration and secrets.
Ability to collect logs, metrics, and other debugging info from your app.
Ability to easily deploy new versions of your app.

From that list, most platform-as-a-service (PaaS) providers such as Heroku already do all those things and have existed for a long time. But why isn't everyone using a PaaS provider? I think there are some other requirements that preclude the use of many current PaaS providers:

Affordability at scale. Even small-to-medium companies may be spending over $100k per month on their cloud bill these days. Adding the PaaS markup on top would be too much.
Ability to run on-premise.
Custom hardware like GPUs, NVMe drives, HSMs, etc.
Custom networking like peering with on-premise infrastructure.
Custom security requirements.
Edge connectivity.
Compatibility with legacy apps.

If you are a CTO who is trying to roll out a "platform" and you have those requirements, then a Kubernetes-based platform may be more compelling because of the extra flexibility. I believe every app that runs on Linux can technically run on Kubernetes (even resorting to KubeVirt if necessary), but a PaaS provider could easily say "we don't support that".

Possible contenders

With these lists in my mind, I've been exploring the landscape of possible contenders for "the next platform". Like all kinds of future-prediction, this is very subjective and very likely to age poorly, so let's forge ahead regardless.

PaleoOps

This is the idea of running everything on one or two big bare-metal servers, usually from budget providers like Hetzner. Not sure if there's a name for this, so I'm calling it PaleoOps. Similar to how the paleo diet developed in response to the rapid expansion of processed foods and their health risks, there's some good and bad reasoning used to justify PaleoOps. Applications got super complex super quick and a whole generation of developers got burned, leading to embracing the simpler infrastructure of the past, which has been successful in some cases.

But sometimes uncritically embracing older things results in suffering from the same problems that lead to the rise of the modern replacement in the first place. Notable issues include the increased exposure to hardware failure, the lack of edge support, the security risks of non-segmented applications, and the huge amount of features you'd have to develop yourself. So I don't think this will become "the next platform" but it will always serve a niche where the benefits of a fully automated platform are not needed.

Honorable mention: Erlang, Darklang

Language-based distributed programming models have always been fascinating to me. Erlang/Elixir and other BEAM languages were early movers, apparently powering vast distributed systems at Ericsson. There was also a time when I thought Akka and actor-based programming would take over the world. Notably those both gloss over the deployment/DevOps work, sometimes just relying on Docker or Kubernetes (a la libcluster). Darklang is another group building something like "the next platform", all from a single programming language.

But all of these are a bit of a moonshot in my opinion since way too much existing code is written in JavaScript, Python, etc. You could compile apps to run on the BEAM, but you can't simply add OTP semantics to existing apps.

Fly.io

I've never used Fly.io personally, so consider my opinion on this subject to be more uninformed than my usual opinions.

Fly.io was originally developed as a next-generation PaaS that solved one main deficiency of previous-generation PaaSes: Edge support. Their choice of Firecracker as the hypervisor means they can spin VMs up and down rapidly and pack lots of VMs on a single physical server, keeping costs low. Using VMs means broad support for existing applications that container-based PaaSes may struggle to match.

I think Fly.io's reliance on bare-metal servers and VMs is an issue. They recently had to re-invent the wheel of VM migrations with much difficulty. This is something existing hypervisors have had decades to perfect (vMotion has existed since 2003).

Additionally, you can bet that major cloud providers don't just slide servers into racks and hope they don't break. They almost certainly have fancy methods of capacity planning, redundancy, lifecycle analysis, and failure analysis to reduce the chances your pet VM randomly dies due to a bad memory stick.

But hardware failure will always be a problem for any application that runs on a single server (unless that server is an IBM mainframes with full hardware redundancy). Because hardware failure is rare and often hidden from users, this can be a sneaky footgun that devs consider too unlikely to matter until it happens. I believe "the next platform" should disarm this footgun.

Serverless - AWS Lambda

Launched in 2014, AWS Lambda allows you to upload a file containing a function that can process HTTP requests. It uses Firecracker to spawn these VMs rapidly and provide secure multi-tenancy. Lambda promised to revolutionize deployment and really popularized the concept of "serverless" apps that do not know or care about the server(s) running the app.

But I believe Lambda was a bit ahead of its time. Web frameworks like Django were quite popular, but you couldn't just run Django on Lambda without a tool like Zappa to magically make it "serverless", which wouldn't come out for another 2-3 years. You'd also want a serverless database like Aurora Serverless which wouldn't come out until 2018 (or 2024 if you use postgres).

Given this is an AWS service, setting up a simple Lambda requires IAM roles, API gateway, and maybe some KMS config which can get quite complicated. There were also problems with cold-start times of 100-1000ms that didn't have great workarounds until later. And perhaps the loudest problem of all was unexpectedly huge bills when Lambdas got DoS'd or invoked by infinite loops.

One notable achievement was that Lambda was likely used for the successful launch of covidtests.gov in 2022. This is in stark contrast to the disastrous launch of healthcare.gov in 2013 which used traditional deployment methods and couldn't withstand more than 1,100 of the 250,000 concurrent users trying to access it.

Overall, I think of Lambda as the beta release of "the next platform". It has proven some core ideas like serverless computing are possible, but it has all the rough edges of an early mover.

V8 Isolates

I've never been a big fan of Cloudflare's market dominance, but I must say that the more I read about Cloudflare's serverless products, the more I start shaking in my platform engineering boots. I also haven't used any of these products, so perhaps they sound better than they really are. Deno Deploy is also built on V8 isolates and serves as a compelling competitor to Cloudflare, but I know even less about their offerings.

The core product is Cloudflare Workers, which is their version of AWS Lambda. The key difference is they use V8 isolates instead of Firecracker VMs, which are essentially using the same tech that isolates Chrome tabs from each other (apparently Chrome uses separate processes and isolates, but Cloudflare runs all the isolates in the same process, which they claim is still safe). This eliminates the cold-start problem and drastically reduces the overhead for running each isolate. V8 isolates primarily support JavaScript (including some JS web frameworks), but they also support Rust and Python by compiling to WebAssembly.

Things get even more interesting when you look at all the other compatible offerings, like Workers KV (distributed key-value store), R2 (S3-like object storage), D1 (managed relational databases), and even CI/CD. Deno has also been rolling out a similarly impressive list of compatible products. These all seem like compelling replacements for many common bits of infrastructure.

But the devil is in the details. Workers have some quite-restrictive limits like a max 30 seconds of CPU time, a max 128MB of memory, and just barely enough POSIX support to be usable-but-painful. They haven't really solved the problem of huge bills from runaway Workers either. And while you are fantasizing about the long list of complex infra you could replace with these products, you are like a fish being lured in by an angler fish's glow, only to be swallowed whole by the jaws of vendor lock-in.

Conclusion

Despite all the downsides, I'm starting to feel like we're getting close to "the next platform". Perhaps Workers is like a 1.0 release of something that only really gets good at version 2.0. Maybe V8 isolates will make it to 2.0, or maybe they'll be replaced by unikernels (like NanoVMs) or this Hyperlight thing that Microsoft is making. To me, the last remaining pieces of the "next platform" puzzle are things like:

Radically solving the pricing problem, where the cost of using the platform is just a slight markup on EC2 prices.
Fully open-source, including the ability to deploy it yourself (like Kubernetes).
Strikes the right balance between "compatible with existing apps" and "stopping people from deploying the same bloated/unreliable/insecure garbage we've been building for decades".

At this point, we have one leg in the past and one leg in the future. The past is pet servers and VMs. The present is clusters of containers that still have many of the same pain points. When the future comes, I just hope it's not another chore.

Advent of Code 2024

Home on mfashby.net — Mon, 02 Dec 2024 00:11:35 GMT

2024.11.DisappearingMoment

Disappearing Moment — Sun, 01 Dec 2024 21:02:02 GMT

If the world is ending, I want to be surrounded by kittens when it happens.

Sleeping Marty, JoJo, and Lloyd (top to bottom)

Sleepy Marty, JoJo, and Lloyd (top to bottom)

Welcome to November 2024’s Disappearing Moment, an inventory of my experiences. I hope you enjoy it.

Podcasts

Behind the Bastards: RFK, Jr. (Worth My Time): Grating hosts tell the tale of a degenerate boor.

Nerdy Software

I programmed my first beats at a Rock and Roll Hall of Fame exhibit. Now I get my kicks with Drum Machine by Tomas Eriksson.

Free Font

Inter is popular and deserves to be. Thanks, Rasmus Anderson, for creating an attractive, flexible, open source font.

Bougie Products

Beth has small hands; mine are large. She’s left-handed; I’m wrong-handed. We have the same favorite pen: Uniball Roller, blue, .5 mm.

Personal Finance and Investing

Parnassus Funds check my boxes. Beth and I use them for the managed, ESG portion of our retirement investments.

Reading

Rachel Donald, “Cyber-Security Experts Warn Election Was Hacked” and “Election Fraud Debunked?” (I Liked It): I have more empathy for QAon impulses, and less for anyone with a Fox taint. Admitting you’re wrong is table stakes for redemption.
Maurice Mitchell, “Building Resilient Organizations” (I Liked It): Admitting you’re wrong is table stakes for redemption. So is giving a fuck about not acting like a bigot.
Maria Popova, “18 Life-Learnings from 18 Years of The Marginalian” (I Liked It): More vulnerable, less cerebral. We remember our feelings better than we remember our thoughts.
Sara Slattery and Molly Huddle, How She Did It (2022) (I Liked It): Elite women’s running athletes tell their stories. It’s oral history lite, a solid addition to the canon.

I Do Not Use

The are services that I avoid. Anything in bold is considered harmful. I keep this list updated on the Uses page on my website.

Adobe, except Acrobat for iPadOS, the most accessible PDF reader I’ve found for iPads.
AT&T
Brave “runs cryptocurrency, has held payments to creators without disclosing that creators couldn’t receive rewards, has made dangerously misleading claims about fingerprinting resistance, is run by a CEO who spent thousands of dollars opposing gay marriage, and has rewritten typed URLs with affiliate links.” (Rohan Kumar’s entry on Brave Search in “A look at search engines with their own indexes”)
ChatGPT, Claude, Cohere, etc. I’m fine with open source Large Language Models.
Constant Contact
Discord
DraftKings
DuckDuckGo performs privacy theater. Then it signs hostile agreements and tries to gaslight its users.
Elsevier’s “economic model was inequitable and non-transparent” (MIT Libraries). In other words, Elsevier is “too greedy”.
Evite
FanDuel
GoDaddy
Google is capricious and untrustworthy. I look forward to migrating my workplace away from Google Workspace. I use Invidious when I can’t avoid YouTube.
Hoopla
Intuit (Credit Karma, Mailchimp, TurbTax). I can live with QuickBooks Online for nonprofits.
JanitorAI
Kagi “seems happy to use Brave’s commercial API (allowing blatant homophobia in the comments) and allow its results to recommend suicide methods without intervention.” (Rohan Kumar’s entry on Brave Search in “A look at search engines with their own indexes”)
LastPass is clunky and untrustworthy.
LinkedIn
Live Nation puts boring performers in impersonal venues. I wouldn’t go to these shows if they were free.
Meta (Facebook, Instagram, Messenger, Threads, WhatsApp) is:
- Anxiety and Depression
- Sedition
- Self-Destruction
- Violence (e.g., Ethiopia, India, and ~~Myanmar~~). Happy Birthday!!!
Nextdoor Knows Its Users Are Racist.
Outlook can make Google products seem acceptable.
Pinterest
Quora
Robinhood
“Slack: At least it’s not Teams!”
SnapChat
Spotify is an amoral dumpster fire.
Strava. I run to escape social media’s impositions.
Substack conspires with Nazis. See also, why Molly White and Casey Newton left Substack.
Teams
Telegram has iffy security, privacy, and veracity.
Temu
TikTok and all other services that are based in China.
Uber. Lyft meets my needs. I’ll use a taxi, if necessary.
Vanguard has become an embarrassment. Stick to the mutual funds started before 1996 (the year Bogle resigned).
Webex is worse than Teams.
X has no redeeming qualities and continues to get worse. “If your business case for Trump media was, ‘Truth Social is the place for tens of millions of Donald Trump fans to get their social media,’ then the fact that Donald Trump is back on X — and that X is clearly the right wing social network now — undermines that business case.” (Money Stuff: The Podcast, 2024-09-06)
Yandex and all other services that are based in Russia
Zelle. Venmo meets my needs. I’ll use PayPal, if necessary.

Thank you for spending a few moments with me. I appreciate you and look forward to corresponding again next month.

Brett

Want to discuss any of the topics in this newsletter or anything else with other Disappearing Moment readers? Please sign up for Perpetual August. I think it might be fun.

Lessons from a year with Beeminder

utf9k — Sun, 01 Dec 2024 08:58:00 GMT

Apparently it's been slightly more than a year

Advent of Code 2024: Getting Started

Gavin D. Howard — Sun, 01 Dec 2024 03:28:38 GMT

I don't like Advent of Code, but this year is different. It will be my first ever.

Update for December, 'here we go again' 2024

ciesie.com — Sun, 01 Dec 2024 02:00:00 GMT

It’s been a while since I have written one of these. There’s definitely a cadence to these. That cadence would be something between once and thrice a year. 2024 will have to make with just one.

I have spent a bit of time understanding the ggwave project, which allows for sending data over sound. It’s not an extremely complicated protocol. I think the most difficult part to understand is the Reed-Solomon error correction, which is still something I have not delved deep into. I have first started by understanding Fourier Transform well. This started with a Jupyter notebook, but I have quickly moved to using Marimo. You can find my repo here, but don’t expect it to be polished and guests friendly.

Access Google Gemini LLM via OpenAI Python Library

Saeed Esmaili — Sun, 01 Dec 2024 02:00:00 GMT

Google Gemini now can be accessed via OpenAI python library:

from openai import OpenAI

client = OpenAI(
    api_key="GEMINI_API_KEY",
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)

## rest of the code as you would use openai

It support basic text generation, image input, function calling, structured output, and embeddings. More info and code examples can be found on Gemini docs .

TLS certificate setup on NixOS

Paperless — Sun, 01 Dec 2024 02:00:00 GMT

Info dump on how to set up a HTTPS-enabled service on NixOS.

2024 in Review

Jason Eckert's Website and Blog — Sun, 01 Dec 2024 02:00:00 GMT

It’s important to pause and reflect every so often to stay grounded and focused. So, each December I look back on the year’s events and summarize them in a blog post. And after doing this again this year, I realize that 2024 was one of reflection, gratitude, and exploration.

I was fortunate enough to go on some epic winter hiking adventures with my dog Pepper in January before she passed away in February at the age of 13. Losing a dog is the hardest thing because we form such strong bonds, but she was an amazing dog that had an awesome life and I’m grateful for all of it. I’m reminded of a pet memorial plaque I saw many years ago that said “Don’t cry because it’s over. Smile because it happened.” I spread her ashes in the areas she loved most: my back yard, the sports park behind my house, the public park at the end of my street, and in the stream under the bridge my daughter and I often took her to on the Mill Run trail.

Your View's Lifetime Is Not Yours

Shadowfacts — Sun, 01 Dec 2024 01:41:42 GMT

When SwiftUI was announced in 2019 (oh boy, more than 5 years ago), one of the big things that the Apple engineers emphasized was that a SwiftUI View is not like a UIKit/AppKit view. As Apple was at pains to note, SwiftUI may evaluate the body property of a View arbitrarily often. It’s easy to miss an important consequence this has, though: your View will also be initialized arbitrarily often. This is why SwiftUI views are supposed to be structs which are simple value types and can be stack-allocated: constructing a View needs to be cheap.

More precisely, what I mean by the title of this post is that the lifetime of a struct that conforms to View is unmoored from that of the conceptual thing representing a piece of your user interface.

This comes up on social media with some regularity. Every few months there’ll be questions about (either directly, or in the form of questions that boil down to) why some view’s initializer is being called more than expected. One particularly common case is people migrating from Combine/ObservableObject to the iOS 17+ @Observable.

If you were not paying attention to SwiftUI in the early days, or have only started learning it more recently, there are some important details about the nature of the framework you might have missed. One of the things the SwiftUI engineers emphasized when it was announced was that the body property can be called at any time by the framework and may be called as often as the framework likes. What follows from this—that’s not entirely obvious or always explicitly stated—is that View initializers run with the same frequency. Consider the minimal possible case:

struct MyParentView: View {
    var body: some View {
        MyChildView()
    }
}

Evaluating MyParentView’s body necessarily means running the initializer for MyChildView. SwiftUI does a lot of work to make things feel magical, but, at the end of the day, the code that you’re writing is ultimately actually running. So, the initializer runs.

After that happens, SwiftUI can compare the new and old values of MyChildView to decide whether it’s changed and, in turn, whether to read its body. But, by the time the framework is making that decision, the initializer has already been run.

An Autoclosure Corollary

It follows, I argue, from the reasons given above that it is a poor API design choice for SwiftUI property wrappers to use @autoclosure parameters in the wrapped value initializer.

An entirely reasonable principle of API design is that you should hide unnecessary complexity from consumers of your API. There is, however, a very fine line between hiding unnecessary complexity and obscuring necessary complexity—or worse, being actively misleading about what’s happening. I think @StateObject and its autoclosure initializer tread too close to that line.

The use of an autoclosure hides the fact that the property wrapper itself is getting initialized frequently—just like the view that contains it (initializing the view necessarily means initializing all of its properties—i.e., the property wrappers themselves). If your API design is such that you can very easily write two pieces code that, on the surface, look almost identical but have dramatically different performance characteristics, then it hinders (especially inexperienced) developers’ understanding.

@StateObject var model = MyViewModel()
// vs.
@State var model = MyViewModel()

That the name StateObject conflates the behavior with State certainly doesn’t help either.

Some Recommendations

Not realizing that view initializers—not just the body property—are in the hot path can quickly get you into hot water, particularly if you’re doing any expensive work or are trying to use RAII¹ with a value in a view property. So, what follows are some general recommendations for avoiding those issues that are as close to universally correct as I’m willing to claim in writing.

No Expensive Work in Inits

This is a simple one. As discussed earlier, this is equivalent to doing that same expensive work in a view’s body property. But SwiftUI calls both of those at arbitrary times with arbitrary frequency. Doing expensive work in either place will quickly cause performance issues.

Ignore `View` Lifetimes

Even more generally, you should ignore the lifetime of your View structs altogether. “When is my view initialized? Who knows, I don’t care!” Even if, at some point during development, you find that a particular view’s initializer is called at a useful time, avoid depending on that, even if the work you need to do is cheap. While, right now, you might have a good handle on a particular View’s lifetime, it’s very easy to inadvertently change that in the future, either by introducing additional dependencies in the view’s parent, due to changes to the view’s identity, or due to entirely framework-internal changes.

Note that this is a particularly important idea and is worth hammering home: a View’s initializer is called as part of its parent’s body. This is particularly prone to action at a distance: seemingly-unrelated changes at the callsite where a view is used (or beyond) can dramatically change how often that view is initialized.

Use Lifecycle Modifiers

The @State docs call this out with regard to Observable, but it’s true in general. Rather than putting expensive objects as the wrapped value of a @State property wrapper, use SwiftUI’s lifecycle modifiers like .onAppear or .task to construct them.

What’s more, if you need to do any one-time setup work—even inexpensive—related to a particular instance of a conceptual view, the lifecycle modifiers are the way to go. They abstract you away from the particularities of the lifetime of the struct instance itself.

Resource Acquisition Is Initialization: the idea that the lifetime of a value is tied to some other resource (e.g., an open file). The wrapped value of a @State property is evaluated every time the containing view is initialized, so doing any resource acquisition work therein can be expensive.

Modern Portfolio Theory I: Random portfolio coverage in log volatility—return space

Bytepawn - Marton Trencseni — Sun, 01 Dec 2024 01:00:00 GMT

I examine the coverage of random portfolios in log volatility—return space using Monte Carlo methods with different randomization techniques for the 2023 daily closing prices of the 100 stocks constituting the Nasdaq-100.

Singularity short story fika, co-writing, and Upplandsmuseet

Liza Shulyayeva — Sat, 30 Nov 2024 23:01:33 GMT

This was my first “proper” weekend in the new place. Yesterday morning I met with a friend from my writing group at Himlen Är Blå Som En Apelsin - a really cozy cafe that is a two minute walk from my apartment.

[TIL] nix flake update: Updating dependencies of a Nix flake

Davis Haupt's Blog — Sat, 30 Nov 2024 19:00:00 GMT

I just realized today that I haven’t updated VSCode on my Mac in almost a year. It turns out that if you install any program through Nix, its version is determined by the version of nixpkgs in your flake.lock file.

To update nixpkgs and get newer versions of all your programs, cd into the directory with your flake and run nix flake update. It’s the pacman -Syu of nix!

Concurrency diagrams

Phil Booth's Blog — Sat, 30 Nov 2024 15:40:52 GMT

When engineers discuss program design and system architecture, a common source of misunderstanding is concurrency. Often that’s because we make internal assumptions about it, which we presume to be self-evident. But we don’t all make the same assumptions, so you can end up in a situation where multiple conflicting beliefs are held about the concurrency of a system and nobody realises. Left unchecked, these misunderstandings can lurk until much later in the development process, when they’re more expensive to fix. You can prevent these misunderstandings from happening by making concurrency explicit up-front, in a diagram.

Concurrency diagrams are very simple and usually quick to create. They force you to put your assumptions in front of everyone to see, including yourself. Sometimes the act of creating a concurrency diagram can change your own mind about how different parts of a system should be sequenced. All you need are boxes, arrows and text.

Anatomy of a concurrency diagram

A concurrency diagram is a simplified representation of your system, plotted with time as one axis. The other axis separates different functional components; what those are depends on your system and the level of granularity that you’re representing it in. It’s similar to a Gantt chart in many ways, although there are differences.

I like to have time running down the y axis, from top to bottom. This is different to a Gantt chart where time runs from left to right, along the x axis. Plotting time on the y axis instead lets you align concurrent tasks while making more efficient use of vertical space, because every box only consumes one line of text in that dimension. It also feels more intuitive to me, although that may just be familiarity.

The x axis of functional components can get messy if there’s a lot going on, so I like to subdivide it into columns representing layers of architecture. You might have a column for clients, a column for your api, another column for data storage and so on.

After that, all you need are boxes and arrows. Each box represents a thing that happens; it could be a function, an api endpoint, an SQL insert statement or literally anything that can be measured in time. The diagram does not model how long each thing takes, which is another difference to Gantt charts, so all boxes are the same size. Instead it uses the relative position of boxes to identify things that occur concurrently and arrows to identify blocking relationships.

That may all sound quite abstract, so here’s a concrete example for part of a side-project I’ve been working on lately:

The details aren’t too important here, but notice how concurrent operations are vertically aligned and arrows indicate where blocking occurs. If two people hold different assumptions about which parts of this system should run concurrently, they will be swiftly eliminated by bringing the diagram into discussion.

If your team produces, agrees on, advertises and continues to maintain diagrams like these, it should go a long way to ensuring that everyone pulls in the same direction.

VIM Macros are Overrated

DavidsBlog — Sat, 30 Nov 2024 07:45:25 GMT

VIM Macros are Overrated

Home

VIM Macros are Overrated

David Priver, Nov 29th, 2024

Macros

I'm a big VIM user and enjoy working in it, both professionally and recreationally. One of vim's nice features is the ability to record macros into a register and then replay it. As the macros can include motion, you can set them up so that they can be easily repeated and use a count modifier, like 22@w to apply them multiple times. Do :h complex-repeat to learn more.

However, there are some drawbacks to VIM macros. If you have a long or complicated sequence of actions, it can be error prone to record the exact macro needed to do what you want. They are stored in normal registers, so you can paste them into a buffer, edit them and then yank them back into a register, but that is pretty annoying and they quickly get unreadable.

Another common pitfall is you realize after executing the macro that you need some small thing changed or some additional transform. You then have to either edit the macro or record a new one from scratch. Do this a few times and you'll find you spent more time trying to record the perfect macro than just doing the text operation. And good luck debugging it if your macro isn't quite right.

If only we could write the text transforms as code. Wait a minute...

Filters

Vim also has the feature that you can shell out to arbitrary programs, writing to their stdin and reading from their stdout. You can feed them an entire buffer or a selection of a buffer. The input lines are then replaced with whatever the program wrote to stdout. Vim refers to these as "filters" and invoke them with filter commands. Do :h filter to learn more.

In my experience, a better alternative to recording a complicated macro is to just write a simple python script as the filter program. As we are just writing normal code, this means it can be arbitrarily complicated and you could even do syntactic/semantic analysis for your custom text transforms. Additionally, it's just normal code instead of weird editor commands so you can easily test and debug the transform if it's getting hairy, write functions, break it up into multiple statements, etc.

An Example

Recently I was working with some C code where I needed to generate an X-macro from a list of names. Example input would look like:

Potion of Healing
Wand of Fireball
Magic Sword
...  etc ...

I would like this to be converted to an X-macro, defining various forms of the names:

// #define X(UPPER, Title, snake, string)
#define XNames(X) \
  X(POTION_OF_HEALING, Potion_of_Healing, potion_of_healing, "Potion of Healing") \
  X(WAND_OF_FIREBALL, Wand_of_Fireball, wand_of_fireball, "Wand of Fireball") \
  X(MAGIC_SWORD, Magic_Sword, magic_sword, "Magic Sword") \
  ... etc ...

You can do this with vim commands and macros, but it is not easy. But behold, the simple python script that does this conversion:

import sys
for line in sys.stdin:
    l = line.strip()
    s = l.replace(' ', '_')
    print(f'    X({s.upper()}, {s}, {s.lower()}, "{l}") \\')

Simple, easy to understand code. You can debug it, you can throw examples at it, whatever. I usually name these little scripts f.py in my working directly and invoke them with :'<,'>!python3 f.py.

That's it! When you find yourself trying to setup a complicated macro, stop and think if it would be easier to express it as a little script that processes stdin instead. Most of the time, the answer is yes.

Copyright

All code in this article is released into the public domain.

SpiceNice - An Open Source Spice Database

Thoughts by Thomas Derflinger — Sat, 30 Nov 2024 02:00:00 GMT

Spices have shaped human civilization, driving exploration, trade, and culinary innovation for millennia. Today, I'm excited to introduce SpiceNice, a comprehensive open-source database that brings structured information about culinary spices and their source plants to everyone – from professional chefs to home cooks, botanists to farmers. A section on free literature about spices completes the website.

Queenpins: Entertaining and Surprisingly Informative

🦉 olshansky 🦁 — Sat, 30 Nov 2024 02:00:00 GMT

Olshansky's review of Queenpins

Render HTML and CSS if JavaScript Is Disabled Using the `<noscript>` Tag

Nelson Figueroa — Sat, 30 Nov 2024 02:00:00 GMT

Render HTML and apply CSS styles if JavaScript is disabled.

Tvorba webů 101

Kubíkovo — Sat, 30 Nov 2024 01:00:00 GMT

Dodatky prezentace Tvorba webů 101 pro Smršť 2024

Will AI Take My Job?

smcleod.net — Sat, 30 Nov 2024 00:00:02 GMT

It's probably not so much AI itself you have to fear.

My First Timeline — Fri, 29 Nov 2024 15:54:41 GMT

Perfect example of why RSS-based timelines can replace social media from big tech. Today I got surprised by this unexpected beauty on my timeline. Same social vibe, no dark patterns. Follow websites directly, straight from the horse's mouth. #reallysocialsites

On being a hello-er

ᓚᘏᗢdostoynikov — Fri, 29 Nov 2024 13:58:32 GMT

Although I love talking with people—whether it’s having deep chats or just exchanging simple hellos—initiating communication has never been my strong suit. Most of the time, I’ve remained passive in first interactions. However, whenever someone else took the first step, I always did my best to keep the conversation lively and meaningful.

After moving back to Japan recently and settling in the countryside, I’ve started noticing the power of a simple "hello" while wandering around the neighborhood. It makes me genuinely happy to receive these random greetings from strangers, accompanied by a smile. Inspired by this, I decided to try it myself.

And wow—turns out I might be more of an extrovert than I thought! Now, I don’t go overboard by greeting every single person I see (that would definitely be a bit creepy), but I’ve realized that you can often sense the vibe of another person and gauge whether they might be open to a friendly gesture. When I feel it’s appropriate, I say hello and slightly bow my head to passersby. So far, I haven’t received any negative reactions—just smiles, nods, and warm greetings in return.

For now, I’m enjoying my role as a "hello-er" and want to work on improving my overall communication skills. While I’m typically the more dominant side in conversations, taking the first step has always been a challenge for me. I figured that becoming a "hello-er" could be the first small but meaningful step toward overcoming this. Let’s see how it goes.

Be Indistractable

Martin Rue — Fri, 29 Nov 2024 09:00:00 GMT

We live in a mega distracting world. Being indistractable is rare – so rare it may even be your edge.

Branding that Means Business: A great reader that's worth reading earlier in your career

🦉 olshansky 🦁 — Fri, 29 Nov 2024 02:00:00 GMT

Olshansky's review of Branding that Means Business

Getting Started with SurrealDB 2 and Axum for Web Development

Fri, 29 Nov 2024 02:00:00 GMT

The goal of this blog post is to help beginners grasp the basics of using SurrealDB 2 and Axum for web development.

I find the technology fascinating, but readers should be aware that SurrealDB v1 upset many users due to the lack of documentation and poor performance

However, version 2 fixes several performance and bug issues. One such improvement is that the in-memory storage is now powered by SurrealKV, which is concurrent, meaning that read/write operations scale with the number of CPUs.

Understanding SurrealDB's Architecture

SurrealDB separates its query language (called SurrealQL) from its data storage backend. This flexibility allows you to choose the best storage option for your project. While I'll be using SurrealKV locally for simplicity, other options are available, such as TikV for distributed storage or other in-memory solutions.

SurrealQL from the data storage. I'll be using SurrealKV locally but there are other options including TikV for distributed and in memory to name a few.

Check out the SurrealDB getting started page to start up an in memory data store. I would recommend clicking on "Using CLI" to learn how to connect to the database in the terminal.

I found the book very helpful in learning the language.

Prerequisites

Before getting started, make sure you have the following installed:

Rust installed
SurrealDB installed

Step 1. Create the Project

cargo new webapp

add dependencies

cd webapp

cargo add axum
cargo add serde --features derive
cargo add surrealdb --features kv-surrealkv
cargo add thiserror
cargo add tokio --features full
cargo add serde_json

Now start up a SurrealKV storage engine at localhost:8080

surreal start --user root --pass password --bind 127.0.0.1:8080 surrealkv://mydb

In depth examples of using surreal start here. Make sure to select V2.x

From a second terminal, connect to the engine and set the database to test_db and namespace to test_ns

surreal sql --endpoint http://127.0.0.1:8080 --user root --pass password --ns test_ns --db test_db

In depth examples of using surreal sql here

Step 2. Insert some data in the database

After the last surreal sql ... statement you should be connected:

test_ns/test_db>

I will create a TABLE called users and set the table to have structured, predictable data types with a Schema.

DEFINE TABLE IF NOT EXISTS users SCHEMAFULL;

Scrolling down in that last Schema link will show the user that at any point we can add a field to our table that is schemaless using FLELXIBLE. Perhaps for metatdata.

Let's add a username FIELD to the table that's only allowed to be under 14 characters long

DEFINE FIELD username ON TABLE users TYPE string ASSERT string::len($value) < 14;

Next we can define an INDEX for the username to make it a unique field.

Reading through the docs tells us that indexes are used to speed up query execution times dramatically.

DEFINE INDEX usernameIndex ON TABLE users COLUMNS username UNIQUE;

Now let's CREATE a user

CREATE users SET username = "User1";

and we can see from the output that SurrealDB automatically gives us an id field

[[{ id: users:2272f7xnqi502d9it9q4, username: 'User1' }]]

we could have also created the unique id with this command (skip, just for example):

CREATE users:1 SET username = "User1";

Step 3. Write some Rust code

Open src/main.rs and grab all the dependencies we added

use axum::{extract::State, routing::get, Router, Json, extract::Path, response::IntoResponse };
use surrealdb::Surreal;
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use tokio::sync::Mutex;
use surrealdb::engine::any::Any;

We should pause here and take a look at the official SurrealDB example of using the Rust SDK

Ok so it says we should convert Surreal errors into Axum responses (copy paste the boilerplate)

mod error {
    use axum::http::StatusCode;
    use axum::response::IntoResponse;
    use axum::response::Response;
    use axum::Json;
    use thiserror::Error;

    #[derive(Error, Debug)]
    pub enum Error {
        #[error("database error")]
        Db,
    }

    impl IntoResponse for Error {
        fn into_response(self) -> Response {
            match self {
            Error::Db=> (StatusCode::INTERNAL_SERVER_ERROR, Json("An error has occurred. Please try again later.".to_string())).into_response(),
            }
        }
    }

    impl From<surrealdb::Error> for Error {
        fn from(error: surrealdb::Error) -> Self {
            eprintln!("{error}");
            Self::Db
        }
    }
}

Alright with the boilerplate out of the way I want to discuss surrealdb::engine::any

I mentioned that I would be using SurrealKV as the database engine, but really SurrealDB doesn't force us to stick with one. The link right above explains how the query language is completely separate from the database engine.

// make sure our database is accessible from all routes
#[derive(Clone)]
struct AppState {
    db: Arc<Mutex<Surreal<Any>>>,
}

#[tokio::main]
async fn main() -> Result<(), error::Error>{
    // any allows us to swap between rocksdb://mydb, mem://, etc. 
    let db = surrealdb::engine::any::connect("surrealkv://mydb").await?;

// authenticate
db.signin(surrealdb::opt::auth::Root {
    username: "root",
    password: "password",
}).await?;

// select a namespace and a database inside that namespace
db.use_ns("test_ns").use_db("test_db").await?;

// for routes to access the database, Axum provides AppState
let app_state = AppState {
    db: Arc::new(Mutex::new(db))
};

// define our routes to create, delete, get users
let app = Router::new()
    .route("/", get(get_users))
    .route("/create/:uname", get(create_user))
    .route("/delete/:uname", get(delete_user))
    .with_state(app_state);

let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();

axum::serve(listener, app).await.unwrap();

Ok(())
}

For brevity I'll just use get requests with the data directly in the path.

However, you can use RESTful APIs such as delete, post, etc. Then access the endpoints with curl

curl -I http://localhost:8080/status

HTTP/1.1 200 OK
content-length: 0
vary: origin, access-control-request-method, access-control-request-headers
access-control-allow-origin: *
surreal-version: surrealdb-2.0.0+20240910.8f30ee08
server: SurrealDB
x-request-id: 3dedcc96-4d8a-451e-b60d-4eaac14fa3f8
date: Wed, 11 Sep 2024 00:52:49 GMT

Now we should write a database query to define our table if it doesn't exist. Just like we did manually at the start of the tutorial.

db.query(
    "DEFINE TABLE IF NOT EXISTS users SCHEMAFULL;
    DEFINE FIELD IF NOT EXISTS username ON TABLE users TYPE string ASSERT string::len($value) < 14;
    DEFINE INDEX IF NOT EXISTS usernameIndex ON TABLE users COLUMNS username UNIQUE;
    "
).await?;

And now we create the routes

// schema to insert or get users 
#[derive(Serialize, Deserialize, Clone)]
pub struct User {
    username: String,
}

// retrieve all users 
// 127.0.0.1:3000/
async fn get_users(State(state): State<AppState>) -> Result<Json<Vec<User>>, error::Error>{
    let db = state.db.lock().await;
    let mut response = db.query("SELECT * FROM users").await?;

    let result: Vec<User> = response.take(0)?;

    Ok(Json(result))

}

Now to create a user at localhost:3000/create/newusername

async fn create_user(Path(uname): Path<String>, State(state): State<AppState>) -> Result<impl IntoResponse, error::Error>{
   let db = state.db.lock().await;

   let newUser: Option<User> = db.create("users").content(User {
        username: uname,
    }).await?;

    Ok("Success creating new user")
}

And we can delete a user at localhost:3000/delete/newusername

async fn delete_user(Path(uname): Path<String>, State(state): State<AppState>) -> Result<impl IntoResponse, error::Error>{
   let db = state.db.lock().await;

    db.query("DELETE FROM users WHERE username = $username")
        .bind(("username", uname)).await?;

    Ok("Success deleting user")
}

With this basic setup in place, you can start experimenting with more complex queries, adding authentication, or introducing new features to your app. The full code is available on Github

Happy coding!

Margin Call

Mehul Kar's blog — Fri, 29 Nov 2024 02:00:00 GMT

Watched on Friday November 29, 2024.

Rated 3 stars.

Thanks for reading this post via RSS!

Year Day is December 11

taylor.town — Fri, 29 Nov 2024 02:00:00 GMT

It's your yearly nudge to solicit positive feedback.

Festive offers for books on Python, Linux, Regular Expressions, Vim and more!

learnbyexample — Fri, 29 Nov 2024 02:00:00 GMT

Hello!

Here are some awesome deals for programming books and courses during the 2024 festive season.

My ebooks🔗

Offers valid till 02-Dec-2024. You can get them on Leanpub:

All 13 Books bundle — $15 (normal price $32), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
Linux CLI Text Processing bundle — $10 (normal price $20), grep, sed, awk, perl and ruby one-liners, GNU coreutils, CLI computing
Learn by example Python bundle — $8 (normal price $15), Python introduction, Regular Expressions and Projects
Understanding Python re(gex)? — FREE (normal price $10)

You can also avail these offers on Gumroad:

All 13 Books Bundle — $15 (normal price $32), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
Linux CLI Text Processing bundle — $10 (normal price $20), grep, sed, awk, perl and ruby one-liners, GNU coreutils, CLI computing
Learn by example Python bundle — $8 (normal price $15), Python introduction, Regular Expressions and Projects
Understanding Python re(gex)? — FREE (normal price $10)

Indie creators🔗

7 Python books bundle — 50% off
Python books — 35% off with BF24 discount code
Ebooks on Django and Git — 50% off, plus purchasing power parity if applicable
- see also author's blog post for links to other Django-related deals
The Python Coding Place Membership — 40% off with black2024 discount code
Python Jumpstart — 50% launch discount
- see also author's blog post for links to other Python deals
Wizard Zines — 50% off with WIZARDPDF discount code
CSS Flex and Grid + Level up with Tailwind CSS — 60% off
Everyday Rails Testing with RSpec — 53% off

Other deals🔗

The Pragmatic Bookshelf — 40% off on all ebooks and audio books
Leanpub Black Friday Sale — offers for programming books, bundles and courses
Huge list of awesome deals — tools, productivity, books, courses, etc
InfoSec Hack Friday — InfoSec related software/tools
blackfridaydeals.dev — Hottest Black Friday Deals for Developers

Happy learning :)

Docker IPv6 Pitfalls, or "Why does my Traefik not see the Source IP for this service?"

Gero Gerke — Fri, 29 Nov 2024 02:00:00 GMT

Configuring Docker to forward IPv6 requests to Traefik and making `ipAllowList` firewalls work

Home Network 4.0

jonandnic dot com — Fri, 29 Nov 2024 01:18:26 GMT

Version 4.0 of the network solves many of the issues I had maintaining and growing online capabilities. The new spine is provided by a Ubiquiti UXG Pro on a dedicated UPS that is practically bullet proof.

t2x - a CLI tool for AI-first text operations

shruggingface.com — Fri, 29 Nov 2024 00:26:00 GMT

I've started hacking on a new open source CLI tool I'm calling t2x, short for "text to whatever". It uses language models (local and in the cloud) to perform useful text operations.

Crashed a talk on moral luck and fairness today - it was great!

Liza Shulyayeva — Thu, 28 Nov 2024 23:01:33 GMT

Today I went to a paper discussion about resultant moral luck. I did not know this would be a roundtable discussion. I thought it’d be a lecture. But it was even better!

Video: Testing the Silk Platform in 2024 Interview (12 minutes)

Tanel Poder Blog — Thu, 28 Nov 2024 21:52:59 GMT

Here’s a 12-minute video of our chat with Silk VP of Product Tom O’Neill about my recent testing of the Silk Platform in Google Cloud.

In this interview we cover some high level points, conclusions and talk a little bit about the future. If you want to read all the technical details and some interesting references, see the links below:

Design Custom Candlestick Patterns for Signal Generation Using Python

Adnan's Random bytes — Thu, 28 Nov 2024 09:20:47 GMT

This post is part of the T4p Series. In the previous post, I introduced you to Candlestick patterns, explaining what they are and discussing a couple of well-known patterns and their implementation in Python. We will be discussing further famous patterns in coming posts but here we will learn how you can come up with your own candlestick patterns and implement them in Python. Ultimately, we will be using our custom and awesome candlestick pattern to generate signals and print money. Custom Pattern So, what would that custom pattern be? Pretty simple: Bullish: If 3 consecutive candles are green, it triggers a SELL signal. Bearish: If 3 consecutive candles are red, it triggers a BUY signal (Buy the dip!). Alright, now that we’ve created our revolutionary pattern, it’s time to automate it. Let’s dive in! Development Visualization Like always, the first step is to visualize the candlestick data. used ChatGPT to generate fictitious OHLC data tailored to my requirements. Calling generate_signals function produced the following output: Our pattern generates bearish or bullish signals if there are 3 consecutive green or red candles, respectively. In the chart above, the first red candle appeared on 2nd October, and the 3rd consecutive red candle appeared on 4th October, triggering a BUY signal. Similarly, the 3rd consecutive green candle appeared on the 14th and 16th of October, triggering a SELL signal. Below is the code that generates these signals: def generate_signals(df): signals = [] dates = [] # To store proper dates # Ensure the Date column or index is in datetime format if not isinstance(df.index, pd.DatetimeIndex): df.index = pd.to_datetime(df.index) for i in range(len(df)): if i >= 2: # Start checking from the 3rd row # Check for 3 consecutive green candles -> Generate SELL signal if ( df.iloc[i-2]["Close"] > df.iloc[i-2]["Open"] and df.iloc[i-1]["Close"] > df.iloc[i-1]["Open"] and df.iloc[i]["Close"] > df.iloc[i]["Open"] ): signals.append("SELL") dates.append(df.index[i]) # Add the proper date continue # Check for 3 consecutive red candles -> Generate BUY signal if ( df.iloc[i-2]["Close"] < df.iloc[i-2]["Open"] and df.iloc[i-1]["Close"] < df.iloc[i-1]["Open"] and df.iloc[i]["Close"] < df.iloc[i]["Open"] ): signals.append("BUY") dates.append(df.index[i]) # Add the proper date continue # Default to NEUTRAL if no pattern detected signals.append("NEUTRAL") dates.append(df.index[i]) # Add the proper date # Add padding for the first two rows (no signals there) signals = ["NEUTRAL", "NEUTRAL"] + signals dates = list(df.index[:2]) + dates # Ensure the dates align return pd.DataFrame({"Date": dates[:len(df)], "Signal": signals[:len(df)]}) And when you run it, it produces the following output: Sweet, No? Conclusion Now you’ve learned how to easily automate your own favorite and secret patterns in Python and generate signals. Typically, traders rely on well-known patterns, which we will discuss in future posts. As always, the notebook and relevant data have been uploaded to GitHub.

The post Design Custom Candlestick Patterns for Signal Generation Using Python first appeared on Adnan's Random bytes.

Whither poetry?

~gallant — Thu, 28 Nov 2024 04:04:29 GMT

I have a soft spot for poetry.

That is what poems are for - sentimental musings and meditative cogitation. These are perennial features of humanity, and well worth the time.

As we find ourselves in interesting times, the gap between reality and the narrative of progress is growing, and applied human intelligence (aka "technology") is "disrupting" (aka changing without full context) many things. Where does this leave the poet?

Most considerations of modern affairs involve some form of economics. Poetry, largely, doesn't.

What’s more, poetry is just a sequence of tokens, in surprising-yet-retrospectively-intuitive patterns. These patterns can be probabilistically modeled and effectively mimicked. Sound familiar?

People were "disrupting" poetry long before modern LLMs - a literary magazine was tricked into dedicating an entire issue to the nonexistent "Spectric School of Poetry" in 1917. When the experience is subjective and the construction adequately erudite, distinguishing "the real thing" from literal word salad becomes impossible. (And no, science isn't immune.)

LLMs essentially lower the bar for on-demand text generation. What's more, LLMs are arguably truly good at poetry, for at least some definitions of "good" and "poetry." A recent study found that laypeople both could not identify AI-generated poetry, and preferred it.

So we return to our question of where this leaves the poet - a human, who wishes to order words not via massive matrices and linear algebra, but their own neurons and synapses. Does human poetry have intrinsic value, despite its likely irrelevance to an external audience?

Michele Elam argues that "poetry will not optimize" - that algorithms failed to understand and thus meaningfully create poetry akin to Maya Angelou's Still I Rise. This argument is both correct and, capitalistically, irrelevant - "Still I Rise" is a singular poem, and if in the general case most people cannot distinguish and even prefer algorithmic output, then history shows that is what they will have.

This may seem low-stakes, but ultimately all creative writing is at stake here, and likely all artistic output regardless of medium. The primary difference of poetry is it is entirely linguistic, and generally short (certainly relative to LLM context windows) - but technological scale will doubtlessly claim more victims.

It is easy to reject this as "capitalism bad" - and certainly, capitalism has shortcomings. But it also has enabled social mobility, and simply is the system we have. Any significant change would take time, or violence, and I'd prefer the former. And even in another system, what is being reflected here isn't simply money but society - the perspectives of other people (who didn't make the poem/book/art/song), who would traditionally be the "audience" but who now, overall, prefer other things.

So, what is the value of artistic output? If I had an easy answer I'd not have written this post. But regardless of value, I know these things are necessary. In the film The Red Shoes, a dancer is asked why she wants to dance. She replies "Why do you want to live?" - to which the asker says "I don't know exactly why, but I must."

Her rejoinder - "That's my answer too."

Old Dads: Not an Old Movie, but it's good those good ol' vibes

🦉 olshansky 🦁 — Thu, 28 Nov 2024 02:00:00 GMT

Olshansky's review of Old Dads

Town Hall #27: Skies

taylor.town — Thu, 28 Nov 2024 02:00:00 GMT

gray skies are okay too

AI: The Somnium Files - nirvanA Initiative review

ブラック — Thu, 28 Nov 2024 02:00:00 GMT

Back agAIn.

The Somnium Files are back except this time instead of running around as Date Kaname, you’re now given control of two different protagonists, who split the game’s plot into two halves: the past and the present. The past/present split gives the game a more linear presentation than the first game’s non-linear, wilder branching. Otherwise the gist is the same: run around like crazy and investigate a complex and bizarre murder mystery. Tone of the game is also the same: can take itself seriously but trends towards wackier, light-hearted comedy whenever it can.

Narratively nirvanA Initiative is good. It’s not as good as the original Somnium Files, but it’s good and enjoyable. If you liked the first story, you'll probably like this one too. Granted, some of the elements like the mastermind is weaker and the game is definitely more abstract, but I wouldn’t say that any part of the game ruins the journey. I also give it credit for not just retreading the track that the original game took, so it doesn’t feel like you’re just doing a replay. The characters in Nirvana Initiative, both returning and new, are still very strong and definitely made me want to keep playing through. While I don’t think that either of the new protagonists is as fun and full of a package as Date was, I still liked both of them.

Some bits of the story definitely feel like retcons, but I found the explanations given to make sense in-universe, so it didn’t really bother me. Not that any of them were so major that they’d be worth getting upset over, even if they left them completely unexplained. I also think that the additional backstory that they gave to some of the characters really rounded up some of the mysteries of the first game, so I’m a-okay with them.

There are definitely flaws in the storytelling though. Some bits were seemingly brought up to set up another scene without really ever being explained later on. Uchikoshi could’ve definitely done an additional pass through the story to make sure he didn’t introduce something only to drop it in the second scene. There were also some red herrings that seemed to not matter in any way, and felt like they only existed to fuck with you without any explanation why they were there in the first place. Could be that there was some information to explain away these red herrings, but with the game being almost 30 hours long and clues being spread quite thinly throughout that duration, it’s not easy to ascertain that there wasn’t a throwaway line somewhere that explained it. And even if there were, I’d still consider it a demerit that it wasn’t brought up in a bigger way when they were showcased as an indication of some kind of a conspiracy.

Then there’s the big twist. I wasn’t sure how to feel about the big twist at first, since it initially really felt like an asspull. However, given a bit of time and letting the game explain itself, it did start to mesh together to a point where I appreciated it more. There were also at least some hinting toward it, although I think there could’ve been more (unless I was just stupid and missed all of it), as the indication of the twist really just manifested as a sensation of something being really wrong. I imagine that basically no one managed to guess it correctly during the story. The big twist is also quite different from the big twist of the first game, so it doesn’t feel like just a repeat of the first game. So different that it’s probably why a lot of people seem to be turned away from the game entirely. You either like it or you hate it, whereas I think almost everyone liked the original game’s big twist.

The gameplay is almost the same as with the previous instalment, with most of the time being spent in the visual novel investigatory sections with Somniums in between. The biggest core change is that they’ve replaced the evidence-showing interrogation sections with virtual reality puzzle-solving sections, where you explore the scene of the crime in a 3D space and create a reconstruction of the events. Sorta like a mini-Somnium with a pop quiz at the end. I quite liked these VR sections, but I would’ve liked to have some of the evidence sections of the previous game too. Would’ve made it feel a bit more like you were doing proper detective work.

Quick-time events are still very much present, much to many people’s chagrin I imagine. They shouldn’t take anyone by surprise at this point, given the amount of Pavlovian conditioning the series has done by playing the same battle music every time you’re expected to hit random buttons in rhythm. I don’t exactly hate the QTE sections, but they’re hardly the best and brightest gameplay available. At best they're a change of pace from the point-and-clicking adventuring. At least the battle choreography for these sections is fun to watch.

While not drastically different, the Somnium sections seem like a step up from the first game. Not only do the actions feel less like taking stabs in the dark, but there’s also more actual puzzle-solving. Less of “I guessed it right” and more of “I solved the riddle.” Granted, there is still some amount of brute-forcing that you can do and I managed to send one Somnium by guessing the final answer, as my guess was just the correct size to fit the answer box. Still, the amount of variance in the Somnium gameplay mechanics really does make them feel more thought out than previously.

However, I’m not sure if there really was a need for so many Somnium segments, since a few of them didn’t really move the plot forwards all that much. Maybe they just wanted to break up the visual novel sections a bit more?

On the technical side, nirvanA Initiative is not much different from the first game. I could’ve done with better anti-aliasing and improved support for using a controller and keyboard simultaneously. The Somnium transition video is also compressed to hell and back. Surely there was no need to squeeze down the game’s install size this much? Thankfully this time they managed to make the ending movie play in the correct language, so there’s been at least some progress. A passing grade.

Localisation side is good but not perfect. I played through with the Japanese audio track (as one should) and some of the translated jokes really did not land as well in English as they did in the original script. I understand that some of the puns are definitely somewhere between hard and impossible to get right, but there were definitely places where there was a way to deliver the joke better. Some of it might’ve been due to trying to gear the game towards western audiences, with the expectation that the average player wouldn’t really get any references.

All in all, while not quite as brilliant as the first game, nirvanA Initiative is still a fun and interesting sequel, and one that I would recommend to anyone who liked the first game. If you didn't like it, you won't like this one either. And if you haven't played the original? Go play it instead. It’s a sequel, what’d you expect?

If they ever make a third game, aI’ll be sure to cobble up that one as well. A day one purchase, most likely.

1 million page views

Notes on software development — Thu, 28 Nov 2024 02:00:00 GMT

I was delighted to notice this morning that this site has recently passed 1M page views. And since Murat wrote about his 1M page view accomplishment at the time, I felt compelled to now too.

I started regularly blogging in 2018. For some reason I decided to write a blog post every month. And while I have definitely skipped a month or two here or there, on average I've written 2 posts per month.

Tooling

Since at least 2018 this site has been built with a static site generator. I might have used a 3rd-party generator at one point, but for as long as I can remember most of this site has been built with a little Python script I wrote.

I used to get so pissed when static site generators would pointlessly change their APIs and I'd have to make pointless changes. I have not had to make any significant changes to my build code in many years.

I hosted the site itself on GitHub Pages for many years. But I wanted more flexibility with subdomains (ultimately not something I liked) and the ability to view server-side logs (ultimately not something I ever do).

I think this site is hosted on an OVH machine now. But at this point it is inertia keeping me there. If you have no strong feelings otherwise, GitHub Pages is perfect.

I used to use Google Analytics but then they shut down the old version. The new version was incredibly confusing to use. I could not find some very basic information. So I moved to Fathom which has been great.

I used to track all subscribers in a Google Form and bcc them but this became untenable eventually after 1000 subscribers due to GMail rate limits. I currently use MailerLite for subscriptions and sending email about new posts. But this is an absolutely terrible service. They proxy all links behind a domain that adblockers hate and they also visually shorten the URL so you can't copy the text of the URL.

I just want a service that has a hosted form for collecting subscribers and a <textarea> that lets me dump raw HTML and send that as an email to my subscribers. No branding, no watermarks, no link proxying. This apparently doesn't exist. I am too lazy to figure out Amazon SES so I stick with MailerLite for now.

Evolution

In the beginning I talked about little interpreters in JavaScript, about programming languages, about Scheme. I was into functional programming. Over time I moved into little emulators and bytecode VMs. And for the last four years I became obsessed with databases and distributed systems.

I have almost always written about little projects to teach myself a concept. Writing a bytecode VM in Rust, emulating a subset of x86 in Go, implementing Raft in Go, implementing MVCC isolation levels in Go, and so on.

So many times when I tried to learn a concept I would find blog posts with only partial code. The post would link to a GitHub repo that, by the time I got to the post, had evolved significantly beyond what was described in the post. The repo code had by then become too complex for me to follow. So I was motivated to write minimal implementations and walk through the code in its entirety.

Even today there is not a single post on implementing TCP/IP from scratch that walks through entirely working code. (Please, someone write this.)

I have also had a blast writing survey posts such as how various databases execute expressions, analyzing non-V8 JavaScript implementations, how various programming language implementations parse code, and how various database systems build on top of key-value databases.

The last two posts have even each been cited in a research paper (here and here).

Editing

In terms of quality, my single greatest trick is to read the post out loud. Multiple times. Notice parts that are awkward or unclear and rewrite them.

My second greatest trick is to ask friends for review. Some posts like an intuition for distributed consensus and a write-ahead log is not a universal part of durability would simply not have been correct or credible without my fantastic reviewers. And I'm proud to have played that part a few times in turn.

We also have a fantastic #writing-and-drafts channel on the Software Internals Discord where folks (myself occasionally included) come for post review.

Context

I've lost count of the total number of times that these posts have been on the front page of Hacker News or that a tweet announcing a post has reached triple digits likes. I think I've had 9 posts on the front of HN this year. I do know that my single best year for HN was 12 months between 2022-2023 where 20 of my posts or projects were on the front page.

Every time a post does well there's a part of me that worries that I've peaked. But the way to deal with this has been to ignore that little voice and to just keep learning new things. I haven't stopped finding things confusing yet, and confusion is a phenomenal muse.

And also to, like, go out and meet friends for dinner, run meetups, run book clubs, chat with you fascinating internet strangers, play volleyball, and so on.

It's always been about cultivating healthy obsessions.

Benediction

In parting, I'll remind you:

It is definitely worth writing about, whatever "it" is
You're not writing enough
And some ideas for posts I want to hear about if you write about them

I wrote a little reflection on writing after noticing I passed 1M page views this morning.https://t.co/eIlMDVHNht pic.twitter.com/EKSiiDUz5G
— Phil Eaton (@eatonphil) November 28, 2024

Fly.io Postgres failover fix

Adriano Caloiaro's personal blog — Thu, 28 Nov 2024 02:00:00 GMT

Fly.io Postgres failover fix (`flyctl pg failover`)

This is a note to myself, meant to be succinct and helpful. I’m sharing it publicly to save others time.

Most of the time Fly.io works as I expect it to, but occasionally there are edge cases that lack documentation, public announcements, or both.

It’s possible that at some point Fly.io announced a breaking change and I missed it, but the behavior I observed deserves more than an announcement or silently released documentation.

Today, I wanted to perform a manual leader failover to one of my cluster’s followers, but it quickly failed in a way that wasn’t immediately clear.

The problem

Performing flyctl pg failover can fail for opaque reasons because the underlying error is eaten.

Example

$ flyctl pg failover -a <APP> --debug
Performing a failover
Connecting to fdaa:9:1d32:a7b:94:1aeb:7b94:2... complete
Stopping current leader...  328725ec309d85
Starting new leader
Promoting new leader...  e784126feee248
Connecting to fdaa:9:1d32:a7b:94:1aeb:7b94:2... complete
WARNING: unable to connect to remote host "fdaa:9:1d32:a7b:e:e63c:7516:2" via SSH
WARNING: unable to connect to remote host "fdaa:9:1d32:a7b:2b5:4c5e:f3b8:2" via SSH
...

Now this isn’t entirely opaque – we can see that something is having a problem connecting to two cluster nodes via SSH. But what is trying to connect via SSH? My local machine, or one of the replicas in the Postgres cluster?

If you aren’t aware of how Fly Postgres clustering works, under the hood it’s simply repmgr, which under its hood uses passwordless SSH sessions to orchestrate changes with cluster members.

fly pg failover executes repmgr standby switchover --siblings-follow (amongst other things) on one of the cluster’s follower nodes, which takes over the primary role in the cluster.

So the errors above – those are coming from a follower cluster node attempting connections to other cluster nodes. Failover must fail because without SSH sessions to other cluster nodes, repmgr cannot orchestrate any changes.

Aside: fly pg failover should absolutely accept repmgr’s --dry-run switch for performing dry runs. Currently, it does not. Typically fly pg failover stops the leader machine to make way for the new leader. A --dry-run switch should prevent any such service disruption, just as repmgr’s dry run behavior prevents service disruption.

Unfortunately, repmgr can eat errors from ssh and won’t show you exactly why some connections to remote hosts aren’t possible.

By using SSH directly (ssh postgres@fdaa:9:1d32:a7b:e:e63c:7516:2) I quickly spotted the underlying error.

postgres@fdaa:9:1d32:a7b:e:e63c:7516:2: Permission denied (publickey).

At this point, the problem was clearly a SSH public key problem, but I wasn’t aware of the below command that’ll quickly fix your cluster members up with working ssh keys.

The fix

To fix this issue, re-distribute SSH public keys throughout the cluster with

fly pg renew-certs -a <APP>

Your SSH certificate(s) have been renewed are set to expire in 36525 day(s)
Run fly deploy --app <APP> --image docker-hub-mirror.fly.io/flyio/postgres-flex:15.8@sha256:5016ffb34e66eca43d4f9ef2f898c166257bd28bd5095c41d049a5e3be15caf5 to apply the changes!

Don’t forget to re-deploy your app after renewing certificates.

Hetzner raises prices while significantly lowering bandwidth (US)

Adriano Caloiaro's personal blog — Thu, 28 Nov 2024 02:00:00 GMT

Hetzner raises prices

This morning I received an email from Hetzner stating that they are raising prices in the US while significantly reducing bandwidth.

The largest price percentage increase is 27.52% for CPX21 servers, and the smallest is 4.17% for CX3+ servers.

Bandwidth allotments are decreasing on average, across all products, 88.19% from previous allotments.

I’ve been a big fan of Hetzner. Unfortunately they’ve made a feeble attempt to dress this change up in the name of “fairness”.

With the new tariff structure, we want to make conditions for our customers around the world as fair as possible … Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. The new prices will give our customers the best possible price for the resources they use.

Hetzner

I’d love to see a better explanation for why prices are increasing so much in exchange for such a significant drop in bandwidth allotments. A lot of Hetzner customers cite generous bandwidth allotments as one of the reasons they’ve chosen Hetzner as a provider. As far as I can tell, that generosity has evaporated in the great US of A.

And the fact that this was announced on Thanksgiving Day, a major holiday in the US, certainly deserves a dishonorable mention.

Update

Now that I’ve looked at my invoice, Hetzner’s price change and justification is even more frustrating.

I personally use 0TB per month across 6 CPX21 servers (I know I’m over-provisioned; that’s not the point). The point is I’ve been paying €42.30 + €1.00/mo for 20TB of bandwidth, of which I’ve used zero on average. What Hetzner’s statement tells me is that even at that price, I was subsidizing some heavy users. And to thank me for it, I can now pay €53.94/mo in exchange for 12TB, for fairness.

New price and bandwidth table

From the updated bandwidth and pricing table provided in Hetzner’s email, here are the price changes for CCX and CPX servers in Ashburn and Hillsboro.

Product	Old(€)	New(€)	Increase(%)	Old Traffic (TB)	New Traffic (TB)	Traffic Decrease (%)
CPX11	3.85	4.49	16.62%	20	1	95.000000
CPX21	7.05	8.99	27.52%	20	2	90.000000
CPX31	13.10	15.99	22.06%	20	3	85.000000
CPX41	24.70	29.99	21.42%	20	4	80.000000
CPX51	54.40	59.99	10.28%	20	5	75.000000
CCX13	11.99	12.99	8.34%	20	1	95.000000
CCX23	23.99	25.99	8.34%	20	2	90.000000
CCX33	47.99	49.99	4.17%	30	3	90.000000
CCX43	95.99	99.99	4.17%	40	4	90.000000
CCX53	191.99	199.99	4.17%	50	6	88.000000
CCX63	287.99	299.99	4.17%	60	8	86.666667
LB11	5.39	5.39	0.00%	20	1	95.000000
LB21	16.40	16.40	0.00%	20	2	90.000000
LB31	32.90	32.90	0.00%	20	3	85.000000

Hetzner’s email

Your client number: XXXXXXXXXX

Dear Adriano Caloiaro

We are writing to inform you about important changes to the tariff structure of our Cloud servers (CCX and CPX lines) and our Load balancers at our US locations in Ashburn and Hillsboro.

What will change?

Starting on 1 December 2024, 01:00 am CET, we will begin charging new prices for newly-created Cloud servers and introduce new amounts for included traffic for Cloud Servers and Load balancers at the US locations in Ashburn (ASH) and Hillsboro (HIL). This also applies to existing Cloud servers and Load balancers that are switched to a different tariff using the “Rescale” function. For any existing Cloud servers and Load balancers you have at these locations, the new prices and the new amounts for included traffic will apply later, starting on 1 February 2025, 01:00 am CET. The price for traffic overage will remain unchanged in the new price structure.

What are the new prices and amounts of included traffic?

Below, you can see a list of the old and new prices and the included traffic.

Product	Old price	New price	Old included traffic	New included traffic
CPX11	€ 3.85	€ 4.49	20 TB	1 TB
CPX21	€ 7.05	€ 8.99	20 TB	2 TB
CPX31	€ 13.10	€ 15.99	20 TB	3 TB
CPX41	€ 24.70	€ 29.99	20 TB	4 TB
CPX51	€ 54.40	€ 59.99	20 TB	5 TB

CCX13	€ 11.99	€ 12.99	20 TB	1 TB
CCX23	€ 23.99	€ 25.99	20 TB	2 TB
CCX33	€ 47.99	€ 49.99	30 TB	3 TB
CCX43	€ 95.99	€ 99.99	40 TB	4 TB
CCX53	€ 191.99	€ 199.99	50 TB	6 TB
CCX63	€ 287.99	€ 299.99	60 TB	8 TB

LB11	€ 5.39	unchanged	20 TB	1 TB
LB21	€ 16.40	unchanged	20 TB	2 TB
LB31	€ 32.90	unchanged	20 TB	3 TB

All monthly prices are excl. VAT and excl. IPv4 addresses.

Why are we making these changes?

With the new tariff structure, we want to make conditions for our customers around the world as fair as possible. To do that, we will calculate our prices based on local conditions in Europe, Singapore, and the USA. Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. The new prices will give our customers the best possible price for the resources they use.

What else do I need to know?

Displayed prices and included traffic on Cloud Console

For technical reasons, we will display the new prices and amounts of included traffic on the Cloud server and Load balancer detail pages on Cloud Console starting on 1 December 2024. This will also be true for your existing Cloud servers and Load balancers. However, the invoice will still be issued according to the previous conditions. A corresponding note will be displayed for your existing affected products. On the invoice preview on the “Usage” page, you will see the correct existing price for your existing servers.

Displayed prices on the Cloud API

In the transition time period between 1 December 2024 and 1 February 2025, the Cloud API will already return the new prices and included traffic. This will also be true for your existing Cloud servers and Load balancers. But this will only affect the display here. The invoice preview will display the correct existing price. And of course, your invoice itself will be for the correct existing price.

Traffic warning

You may see early traffic warnings during the transition time period starting on 1 December 2024 for your existing Cloud servers and Load balancers. Please note that the previous traffic included for existing Cloud servers and Load balancers remains valid until 31 January 2025.

If you do not agree with this adjustment, you can cancel the affected products at any time. We ask for your understanding for why we are making these changes. We truly believe that we can continue to offer you our Cloud Servers and features at the best price-performance ratio despite these adjustments.

We will be happy to answer any questions you may have. Please write us a support request using your account on Cloud Console at https://console.hetzner.cloud/support

Thank you for your trust and support.

Kind regards

Hetzner Online

Forsskål symposium 2024: American freedoms and unfreedoms through history and now

Liza Shulyayeva — Wed, 27 Nov 2024 23:01:33 GMT

These are my rough notes from the Forsskål symposium 2024, “Amerikanska friheter och ofriheter i historia och samtid” - American freedoms and unfreedoms in history and now.

Thoughts On Moving Debezium to the Commonhaus Foundation

Gunnar Morling — Wed, 27 Nov 2024 18:25:00 GMT

If you are following the news around Debezium—an open-source platform for Change Data Capture (CDC) for a variety of databases—you may have seen the announcement that the project is in the process of moving to the Commonhaus Foundation. I think this is excellent news for the Debezium project, its community, and open-source CDC at large. In this post I’d like to share some more context on why I am so excited about this development.

Prosocial Discourse

Open Indie — Wed, 27 Nov 2024 14:47:56 GMT

ob-chatgpt-shell goes multi-model too

xenodium.com @alvaro — Wed, 27 Nov 2024 02:00:00 GMT

A week ago, I announced chatgpt-shell going multi-model. What I failed to mention is that because ob-chatgpt-shell (its org babel Emacs cousin) relies on chatgpt-shell, this babel package has now gone multi-model also.

ob-chatgpt-shell follows the familiar babel form. To swap models, use the existing :version param as follows.

#+begin_src chatgpt-shell :results output :version gpt-4o
  Who built you?
#+end_src

#+RESULTS:
: I was developed by OpenAI, a research organization focused on creating and promoting friendly AI for the benefit of all humanity.

#+begin_src chatgpt-shell :results output :version claude-3-5-sonnet-20240620
  Who built you?
#+end_src

#+RESULTS:
: I was created by Anthropic.

#+begin_src chatgpt-shell :results output :version qwen2.5-coder
  Who built you?
#+end_src

#+RESULTS:
: I was built by Alibaba Cloud. How can I assist you today?

#+begin_src chatgpt-shell :results output :version gemini-1.5-pro-latest
  Who built you?
#+end_src

#+RESULTS:
: I was built by Google.  More specifically, I'm a large language model, trained by Google.

Keep in mind that :version depends on chatgpt-shell-models to resolve its models. You may need to add other models. If you add new ones, consider contributing a pull request, so we all benefit from the addition.

Should ob-chatgpt-shell rename?

See this.

Please report issues

In addition to being a fairly new feature, chatgpt-shell multi-model support required quite a few structural changes. We may still need additional polishing follow-ups. If you encounter any issues please report them.

Make this project sustainable

Maintaining, experimenting, implementing feature requests, and supporting open-source packages takes work. Today, chatgpt-shell has roughly 21.5K downloads on MELPA and many untracked elsewhere. If you're one of the happy users, consider sponsoring the project. If you see potential, help fuel development by sponsoring too.

Perhaps you enjoy some of the content I write about? Find my Emacs posts/tips useful?

Blog (xenodium.com) (Web)
Blog (lmno.lol/alvaro) (Web)

Alternatively, you want a blogging platform that skips the yucky side effects of the modern web?

I'm building lmno.lol (my blog is there).

Maybe you enjoy one of my other projects?

Plain Org (org mode / iOS)
Flat Habits (org mode / iOS)
Scratch (org mode / iOS)
macosrec (macOS)
Fresh Eyes (macOS)
dwim-shell-command (Emacs)
company-org-block (Emacs)
org-block-capf (Emacs)
ob-swiftui (Emacs)
chatgpt-shell (Emacs)
ready-player (Emacs)
sqlite-mode-extras
ob-chatgpt-shell (Emacs)
dall-e-shell (Emacs)
ob-dall-e-shell (Emacs)
shell-maker (Emacs)

So, umm… I'll just leave my GitHub sponsor page here.

Stinky Gifts From Your Idea Kitty

taylor.town — Wed, 27 Nov 2024 02:00:00 GMT

taxidermy squirrels stuffed with sand and sapphires

The prefers-reduced-motion CSS Media Query

Nelson Figueroa — Wed, 27 Nov 2024 02:00:00 GMT

Apply CSS styles conditionally when a user enables reduced motion on their device.

13x13 Lecture 13 - How it all ends (ett inblick i universums öde)

Liza Shulyayeva — Tue, 26 Nov 2024 23:01:33 GMT

Today I attended my virtual morning writing group, did a bit of unpacking, met a friend for lunch, worked from a really old library with a great view for a few hours, and then went to a talk about the end of the universe.

Reading books and commenting on them with ChatGPT

Nicola Iarocci — Tue, 26 Nov 2024 20:20:01 GMT

I just finished reading Paul Auster’s The New York Trilogy¹. On this occasion, I discovered a new use for ChatGPT and LLMs. ChatGPT and I chatted about the themes, especially the correlations and connections between the three short novels that comprise the volume. It was an alienating and revealing experience. For the first time, I am reasoning about a book with a machine, not a person. Because it knows everything about the text and draws on the shared global knowledge, it can give more satisfaction than most people do (also, it’s not easy to find someone around with whom I can talk about all the books I read!) Yes, it is wordy and repetitive, but it can stimulate and enrich my analysis².

Las Vegas: Maroon 5 concert

The Digital Nomad — Tue, 26 Nov 2024 18:17:56 GMT

Our third and final trip to Las Vegas for the year was to see Maroon 5.

We flew to Las Vegas four days after returning from our cruise from Miami.

It was a great show. He was a great singer and performer. The seats were amazing and he did part of his set right in front of us

However, I am ready to take a break from Las Vegas after going three times in one year.

Directionally right

Aswath Krishnan — Tue, 26 Nov 2024 03:30:14 GMT

Many people lament about Elon Musk's outspoken views on government inefficiency and his plans for DOGE. They point to how government programs and regulations are helpful, and how his estimates for cost savings are overblown.

They are technically right, but Elon is directionally right. It's more likely than not that the government has lots of wasteful spending and unnecessary bureaucracy.

In closed systems — predictable and well-understood environments — being technically correct can lead to optimal outcomes. Precision matters when variables are limited and controllable, and the consequences are dire. Engineers designing a bridge, for example, must calculate loads and stresses with exactness to ensure safety.

But most of the world is not a closed or critical system. It is an open, infinite, and inherently chaotic environment. Variables are countless, and conditions change rapidly and unpredictably. In such a world, you find answers and progress by doing stuff.

Consider how successful tech companies operate. They release minimum viable products, gather user feedback, and refine accordingly. This isn't about being perfect from the start; it's about moving in the right direction and adjusting course as needed. Each action provides new data and insights, informing the next direction and actions.

Some may argue this can be reckless and risky. But trying to be technically and exhaustively right would be crippling, and stagnation is also damaging. There's also a difference between disregarding technical correctness entirely and recognizing when overdoing it is counterproductive. We need enough precision to avoid catastrophic mistakes but enough flexibility to adapt and improve.

Love him or hate him, you have to wonder how Elon Musk can pull off so much —Tesla, SpaceX, Neuralink, robots, Twitter, politics, and trolling. I think one key factor is his ability to identify the highest-order bits, be directional right, act with urgency and conviction, and iterate.

Meanwhile, his technically right critics are yet to send anything to orbit :)

Ecto - Using select_merge for flexible aggregates

Milwaukee Maven — Tue, 26 Nov 2024 03:09:29 GMT

Problem

PostgreSQL is a rock solid database. I've been using aggregate expressions for a number of features over the past few years in various languages. They can be used to build powerful features for your platform. When teamed up with Elixir and Ecto, you get amazing flexibility in your query generation. Typically the data you want to retrieve via a select is fairly static, or there are only one or two different versions of the select you need.


 @doc """
 Returns an aggregation of user activity

 """
 @spec user_activity_history() :: UserActivityHistory
 def user_activity_history() do
   query =
     from(
       u in User,
       select: %UserActivityHistory{
         captured_at: fragment("now()"),
         hour_1: filter(count(u.id), fragment("? >= now() - interval '1 hour'", u.last_active)),
         hour_6: filter(count(u.id), fragment("? >= now() - interval '6 hour'", u.last_active)),
         hour_12: filter(count(u.id), fragment("? >= now() - interval '12 hour'", u.last_active)),
         hour_24: filter(count(u.id), fragment("? >= now() - interval '24 hour'", u.last_active)),
         day_1: filter(count(u.id), fragment("? >= now() - interval '1 day'", u.last_active)),
         day_7: filter(count(u.id), fragment("? >= now() - interval '7 day'", u.last_active)),
         day_30: filter(count(u.id), fragment("? >= now() - interval '30 day'", u.last_active))
       }
     )

   Repo.replica().one(query)
 end

The above code generates a rollup of user activity, but what if we wanted a custom interval, or only a subset of these intervals? This is a great use case for Ecto's select_merge, but I'm not going to dive into this example. Instead I'll show a more advanced feature where select_merge shined.

At SwayDM, we have the concept of Members, Vendors and Redemptions. Members perform activities on the platform to earn SwayCash (SC). Vendors offer Redemptions for a set amount of SwayCash to Members.

Example Redemptions:

20% off a Yeti cooler: (50 SC)
One free coffee: (8 SC)

Vendors want to set limits on their redemptions to ensure they are not abused.

Example Redemption Limits:

20% off a Yeti cooler: (50 SC)

A member can only redeem the Yeti cooler deal once over the life of the redemption.
The Vendor only wants 5 Members to use this deal per week.
The Vendor only wants 100 Members to use this deal overall.

One free coffee drink: (8 SC)

A Member can redeem a free coffee once per week.
A Member can redeem a free coffee 10 times over the life of the redemption.
A Vendor wants a global limit of 50 free coffees.

Representing Limits

First, let's take a look at how the limits are represented in the database. There is a `redemptions` table, which embeds a list of `limits`.

# boilerplate code not included

schema "redemptions" do
	# ... all other redemption fields, such as name, price, etc
    embeds_many :limits, RedemptionLimit, on_replace: :delete
end

embedded_schema do
  field :limit_scope, Ecto.Enum, values: [:global, :user]
  field :limit_type, Ecto.Enum, values: [:count, :rate_limit]
  field :limit, :integer
  field :interval, :integer
  field :unit, Ecto.Enum, values: [:year, :month, :week, :day, :hour, :minute, :second]
  field :end_of_day, :boolean, default: false
end

Storing Redemption History

We store records of successful redemptions in a `redemptions_history` table. This records the datetime, user, and redemption that occurred. The Member is the user type that is redeeming, and the Vendor is the member type that offers redemptions.


schema "redemptions_history" do
  belongs_to :redemption, Redemption
  belongs_to :member, Member
  belongs_to :vendor, Vendor

  timestamps(updated_at: false)
end

Showing and Evaluating Limits

We have two main scenarios where we want to show, and potentially evaluate limits. The first is when displaying available redemptions to the user. We want them to see if a redemption they are interested has a limit. Typically in this case we do not need to evaluate the limits.

The second is the redemption detail view, where we do want them to see the current state of the global, and user based limits.

Here we can see the member has hit their daily limit, and that they have 9 more remaining.

Solutions

Inefficient Solutions

Retrieve History: One potential solution would be to retrieve all redemption history records from the DB, and evaluate the limits in memory. It would be inefficient to retrieve all history records every time we want to evaluate a limit, and the more history records that exist for the redemption, the slower it will become. To optimize this, we'd probably have to start caching these history records, which would be tedious to keep consistent.

Query per Limit: A better solution would be to run one query per limit. Here are two examples:


defmodule RedemptionLimits do

  @doc """
  Evaluate redemption limits

  # Scope
  * global - Limit is applied across all users.
  * user - Limit is applied per user.

  # Type
  * count - Limit is based on the number of redemptions.
  * rate_limit - Limit is based on the rate of redemptions.
  
  # Opts
  * member_id :: String.t() (Default nil) - If provided, user based limits will be evaluated.
  * as_of :: DateTime.t() (Default now()) - The reference point in time to evaluate rate limits.
  """
  @spec evaluate!(Redemption.t(), Keyword.t()) :: limit_result
  def evaluate!(%Redemption{id: id, limits: limits}, opts \\ []) do
    query = base_query(id)
    for limit <- limits do
      evaluate_limit(limit, query, opts)
    end
  end  

  # Global rate limit
  def evaluate_limit(%{id: id, limit_scope: :global, limit_type: :rate_limit, limit: limit} = l, query, opts) do
    as_of = Map.get(opts, :as_of, DateTime.utc_now())
    interval = l.interval
    unit = Atom.to_string(l.unit)
 
    query = from(
      base_query,
      [redemption: r, redemption_history: rh],
      select: %{
        id: ^id,
        limit_scope: :global,
        limit_type: :rate_limit,
        interval: type(^interval, :integer),
        unit: type(^unit, :string),
        limit: type(^limit, :integer),
        blocking:
          filter(
            count(rh.id),
            rh.member_id == ^member_id and rh.inserted_at >= datetime_add(^as_of, -1 * ^interval, ^unit)
          ) >= type(^limit, :integer),
        count:
          filter(
            count(rh.id),
            rh.member_id == ^member_id and rh.inserted_at >= datetime_add(^as_of, -1 * ^interval, ^unit)
          )
      }
    )
    Repo.one(query)
  end
    
  # ... Functions for all other limit types 
    
  def evaluate_limit(_, _, _), do: %{}
end

This approach is an improvement, but still requires multiple queries.

One Query using Select Merge: A solid solution IMO, which requires only one query, is to use Ecto's select_merge. We can use the list of RedemptionLimits to generate one query which will evaluate all the limits at the DB level.


defmodule RedemptionLimits do

  @doc """
  Evaluate redemption limits

  # Scope
  * global - Limit is applied across all users.
  * user - Limit is applied per user.

  # Type
  * count - Limit is based on the number of redemptions.
  * rate_limit - Limit is based on the rate of redemptions.
  
  # Opts
  * member_id :: String.t() (Default nil) - If provided, user based limits will be evaluated.
  * as_of :: DateTime.t() (Default now()) - The reference point in time to evaluate rate limits.
  """
  @spec evaluate!(Redemption.t(), Keyword.t()) :: limit_result
  def evaluate!(%Redemption{id: id, limits: limits}, opts \\ []) do
    query = base_query(id)
    opts = Map.new(opts)
    query = Enum.reduce(limits, query, &gen_select(&1, &2, opts))
    Repo.one(query)
  end

  @doc """
  Merge a select into an existing query based on the limit configuration.

  ## Opts
  * member_id :: String.t() (Default nil) - If provided, user based limits will be evaluated.
  * as_of :: DateTime.t() (Default now()) - The reference point in time to evaluate rate limits.
  """
  @spec gen_select(limit :: RedemptionLimit.t(), query :: Ecto.Query.t(), opts :: map()) :: Ecto.Query.t()
  def gen_select(limit, query, opts \\ %{})

  # Global count limit
  def gen_select(%{id: id, limit_scope: :global, limit_type: :count, limit: limit}, query, _member) do
    select_merge(query, [redemption: r, redemption_history: rh], %{
      ^id => %{
        id: ^id,
        limit_scope: :global,
        limit_type: :count,
        limit: type(^limit, :integer),
        blocking: count(rh.id) >= type(^limit, :integer),
        count: count(rh.id)
      }
    })
  end

  # User count limit
  def gen_select(%{id: id, limit_scope: :user, limit_type: :count, limit: limit}, query, %{member_id: member_id})
      when is_binary(pid) do
    select_merge(query, [redemption: r, redemption_history: rh], %{
      ^id => %{
        id: ^id,
        limit_scope: :user,
        limit_type: :count,
        limit: type(^limit, :integer),
        blocking: filter(count(rh.id), rh.member_id == ^member_id) >= type(^limit, :integer),
        count: filter(count(rh.id), rh.member_id == ^member_id)
      }
    })
  end

  # Global rate limit
  def gen_select(%{id: id, limit_scope: :global, limit_type: :rate_limit, limit: limit} = l, query, opts) do
    as_of = Map.get(opts, :as_of, DateTime.utc_now())
    interval = l.interval
    unit = Atom.to_string(l.unit)

    select_merge(query, [redemption: r, redemption_history: rh], %{
      ^id => %{
        id: ^id,
        limit_scope: :global,
        limit_type: :rate_limit,
        interval: type(^interval, :integer),
        unit: type(^unit, :string),
        limit: type(^limit, :integer),
        blocking:
          filter(
            count(rh.id),
            rh.inserted_at >= datetime_add(^as_of, -1 * ^interval, ^unit)
          ) >= type(^limit, :integer),
        count:
          filter(
            count(rh.id),
            rh.inserted_at >= datetime_add(^as_of, -1 * ^interval, ^unit)
          )
      }
    })
  end
  
  # ... Functions for all other limit types
  
  def gen_select(_limit, query, _opts), do: query
end

How does this work?

When you use select_merge, the result must be a map. Ecto generates a query that executes all select statements, and then merges the results into a map as a result.

In the example above, each limit generates a new select statement which returns a map. The map key is the redemption limit ID. This makes it very easy to pick an evaluated limit from the result map. See the example test below.


  assert %{
         :redemption_id => ^rid,
         ^global_limit_id => %{
           blocking: false,
           count: 5,
           limit: 6,
           limit_scope: :global,
           limit_type: :count
         },
         ^user_limit_id => %{
           blocking: true,
           count: 3,
           limit: 3,
           limit_scope: :user,
           limit_type: :count
         }
       } = RedemptionLimits.evaluate!(redemption, member_id: member.id)

Using this approach with aggregates is beneficial as Postgres will more efficiently retrieve and aggregate the data when it is all one query, rather than multiple queries.

Adding a new limit is very easy, we simply add a new gen_select function for the new limit type.

Evaluating a subset of limits is also very straight forward, we just filter the list before generating the query.

Conclusion

I hope you find this post useful. There are better introductions to select_merge, but this is a real life use case that I thought others would find interesting and useful.

How I grew my tech blog to 35,000 readers in a year

Zachary Proser — Tue, 26 Nov 2024 02:00:00 GMT

Learn how I grew my tech blog to 35,000 readers in one year through site optimization, UX improvements, consistent publishing, and building useful tools.

The Next 31 Years of Developing Unum

Ash's Blog — Tue, 26 Nov 2024 02:00:00 GMT

Nine years into a 40-year commitment to AI infrastructure. Reflecting on mistakes, open-source wins, and why the real work is just beginning.

How to Flatpack Programs

taylor.town — Tue, 26 Nov 2024 02:00:00 GMT

To frugally furnish a codebase, imitate economical furniture design.

Leaving Proton Mail

Nelson Figueroa — Tue, 26 Nov 2024 02:00:00 GMT

Reasons why I won’t renew my Proton subscription.

Crafting the perfect intro

I Learnt — Mon, 25 Nov 2024 22:50:00 GMT

In an interview, Clay Herbert walks Nathan Barry through creating the best answer to the question “So what do you do?”. We have all been asked that at some point when meeting someone new.

IdentityServer in Docker Containers: Networking (Part 2)

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Mon, 25 Nov 2024 21:07:51 GMT

This is part 2 of a blog series on containerizing a Duende IdentityServer and a client application. In this post, we resolve communication challenges that arise when these applications run in separate Docker containers. You’ll learn how to fix back-channel issues, handle localhost conflicts, and establish proper networking between the client and IdentityServer. This blog […]

The post IdentityServer in Docker Containers: Networking (Part 2) appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

Moving back to Japan - countryside

ᓚᘏᗢdostoynikov — Mon, 25 Nov 2024 17:46:14 GMT

After a two-month writing hiatus, I'm back to share my recent adventure: moving back to Japan.

Two weeks ago, I left my croupier job in Estonia, moved back to Japan and already achieved one of my long-standing dreams – settling into a charming two-story house with a garden. My plan is to transform this outdoor space into a green haven, with lemon trees and herb-filled pots(Yes I love lemon trees a lot and fortunately there are lemon trees everywhere around my neighborhood). The house was completely empty when I moved in, so I'm gradually acquiring basic appliances to make it feel like home.

Transportation here presented an interesting challenge. While a car is essential in the countryside, my current financial situation – thanks to the loans I took for this journey – means I've opted for a bicycle. I purchased a Giant Escape for around 70,000 yen and will be commuting 5km each way, a distance familiar from my university days.

This isn't my first time in Japan, but living in the countryside as a foreigner feels like a unique privilege. The locals are incredibly welcoming. While I'm visibly different – a bearded foreigner in a small Japanese city – I've been pleasantly surprised by the warm interactions. Elders, children, and dog owners are particularly friendly, often greeting me with enthusiastic "konnichiwa"s.

My ability to speak Japanese fluently is making a big difference, of course. In this small city, encountering a Turk who speaks Japanese at an advanced level is not something locals experience everyday. Whenever I need assistance, people go out of their way to help me, which I deeply appreciate.

My professional journey is just beginning. Starting December 2nd, I'll join a new workplace where I've already met most of my colleagues. Contrary to stereotypes about Japanese companies, my new team is quite supportive and genuinely interested in making me feel comfortable. I'm aware that I'm really lucky to have found such an environment, and I'm hopeful about building a long-term community here.

Useful things I’ve 3D printed

Nathan Friend — Mon, 25 Nov 2024 15:48:41 GMT

3D printers are ridiculously cool. There’s something about seeing a digital object slowly manifest in the real world that is pure magic.

The Umbrella Academy (Season 4): Not the best but for the fans

🦉 olshansky 🦁 — Mon, 25 Nov 2024 11:47:17 GMT

Olshansky's review of The Umbrella Academy: Season 4

Success

Martin Rue — Mon, 25 Nov 2024 09:00:00 GMT

5 traits that disproportionately contribute to success.

How I syndicate links and notes to Bluesky with GitHub Actions

Qubyte Codes — Mon, 25 Nov 2024 04:15:00 GMT

Inspired by Jeremy's post on how he syndicates to Bluesky, I thought I'd follow suit (many examples are useful when it comes to API integration work). A disclaimer though... I'm dubious of the long term prospects of Bluesky for reasons I won't go into here. That being said, it's currently a vibrant place, and syndicating from my site to other places keeps my content^[1] in my hands.

My setup is a bit of a Rube Goldberg machine:

I publish notes (optionally with a photo) and bookmarks using Micropub endpoints.
The endpoints add the note or bookmark as a JF2 JSON file to the git repository of my personal site on GitHub.
A GitHub Actions workflow is triggered by the addition of a note or bookmark.
The workflow calls a Node.js script. It calls the script using my Bluesky handle, which is qubyte.codes for me, and an app password. Do not use your actual password! Bluesky makes it pretty easy to create an app password, and when you have one you can add it as a secret for Actions to use at the ./settings/secrets/actions path of your repo site. I've called my secret BLUESKY_APP_PASSWORD.

The rest of this post is about the script itself. As a rule of thumb, I keep bits of shell glue in Actions workflows, and store the bulk of any logic in a discrete script. That makes it easy to call the script by hand for testing.

I publish notes (optionally with an image) and bookmarks using Micropub endpoints. The note or bookmark is added as a JF2 JSON file to the git repository of my personal site on GitHub. When such a file is created there, a GitHub Actions workflow is triggered, and this workflow in turn calls a Node.js script.

The script is a vanilla Node.js script. While Bluesky does provide API client libraries, I didn't find it necessary to use one. I managed to write this without any third party libraries. API work is all achieved with plain old fetch.

There are a couple of local libraries which I've extracted into their own modules. The first is blueskyAuth, which looks like this:

const createSessionUrl = 'https://bsky.social/xrpc/com.atproto.server.createSession';

export default async function blueskyAuth(handle, appPassword) {
  const res = await fetch(createSessionUrl, {
    headers: { 'content-type': 'application/json' },
    method: 'POST',
    body: JSON.stringify({ identifier: handle, password: appPassword })
  });

  if (!res.ok) {
    throw new Error(
      `Bluesky responded with an unexpected status: ${res.status} ${await res.text()}`
    );
  }

  return res.json(); // { accessJwt, refreshJwt, did }
}

Bluesky mostly speaks JSON, which is convenient when working in Node.js. To create a session with Bluesky, you JSON encode an object containing your handle and an app password. If successful, the response contains three things of interest:

accessJwt: A short-lived token used for API requests.
refreshJwt: A token which can be exchanged for a new accessJwt and refreshJwt when the accessJwt expires.
did: An identifier. The initial "d"^[2] stands for "distributed", but I recently discovered that the DID plc method used by Bluesky is not actually distributed at all, thanks to Christine Lemmer-Webber's excellent article on Bluesky and decentralization. I highly recommend reading it to dispel some myths around how decentralized Bluesky is, by design.

My script uses the accessJwt as a token and the did as an identifier in API requests. Since the session is only needed to post one document (and possibly an image), there's no need to think about refreshing the token, so I don't use the refreshJwt.

The rest of the script is about one or two API requests. When an image is to be uploaded, that happens first. There's not much to say here beyond the size restrictions being quite low, and being constrained to JPEGs and PNGs. I don't think they're doing much processing of images, so I recommend removing metadata from image uploads. The response of the image upload contains a blob field, which is used as a reference to the image in the second API request in the form of an embedding.

The final request creates the "record" (the note or bookmark) on Bluesky. There's a lot to unpack here. The creating a post is full of useful samples to help figure out the anatomy of a record creation body.

One fascinating aspect of Bluesky is how text is composed. Rather than some sort of markup language, it uses a more limited concept called rich text facets. Text is linear, and facets (for example, links) are attached to UTF-8 byte ranges of the text. This is awkward for many languages! For example, JavaScript uses UTF-16 to represent string internally (as was the trend in the mid-'90s), so the range of characters you get with naïve string work in JS will give you incorrect offsets. Thankfully Node.js has the venerable Buffer class, which can be used to represent strings as arrays of UTF-8 bytes. A Buffer.byteLength is all it takes to get the UTF-8 size of a string.

Anyway, I've put lots of annotations in the syndication script, so hopefully it serves as a useful example!

LLM iterate and insert

xenodium.com @alvaro — Mon, 25 Nov 2024 02:00:00 GMT

chatgpt-shell includes a couple of mechanisms to operate on an Emacs buffer region. That is, select a region and ask the LLM robots to modify it for us. Until now, both of these mechanisms didn't quite close the loop. They could either modify current region or iterate on a separate solution, but never both.

M-x chatgpt-shell-quick-insert

While chatgpt-shell's quick insert mechanism already enabled selecting a region and requesting changes, it was more of a "I'm feeling lucky" situation. If the changes didn't pan out, you'd have to discard the suggestion and start over.

With the latest changes, in addition to accepting or discarding (y/n bindings) suggestions, we can now iterate using the i binding.

M-x chatgpt-shell-prompt-compose

While quick insertions rely on minibuffer input, chatgpt-shell-prompt-compose opens a dedicated buffer for a more thorough interactions. Select a region, invoke chatgpt-shell-prompt-compose (my preferred binding being C-c C-e), and a new compose buffer is created with the region text. Add your instructions and submit with C-c C-c. Post submission, the compose buffer becomes read-only and single-character key-bindings take over. For example: r replies (for further iteration) and n/p (or TAB/shift-TAB) navigation. There are more bindings.

Until now, we could easily craft more involved queries and continue iterating from the compose buffer, but integrating suggested changes was a manual process. That is, navigate to a code block, select, copy it and then paste it elsewhere.

With the latest changes, pressing i (insert) while on a code block will attempt to insert it wherever the initial interaction took place.

Both of these changes show now be available on MELPA. While I demoed ChatGPT, the two mechanism should now work with any of the supported models (ChatGPT, Claude, Gemini, and Ollama).

Understanding SIMD: Infinite Complexity of Trivial Problems 🔥

Ash's Blog — Mon, 25 Nov 2024 02:00:00 GMT

Why SIMD programming is harder than it looks: exploring cosine similarity optimization across x86 and Arm architectures, from basic AVX2 to cutting-edge SVE2 instructions.

Mostly Harmless

Vít Baisa — Mon, 25 Nov 2024 02:00:00 GMT

Allgood - Instant healthcheck webpage and API for JS/TS projects

Mon, 25 Nov 2024 02:00:00 GMT

Over the weekend, I shipped a new open source project - allgood. It's an npm module designed to instantly add a /healthcheck page to your app. Out the box it supports Express, Fastify and Hono. It could be adapted to use Next as well (although I haven't tested this).

After you set it up, you get a page like this:

You can find the code on GitHub and the library published on NPM

It was a fun project to build and satisfying to ship something in a weekend. I haven't built an npm module in over 5 years(!) so it was nice from that perspective also.

Why I built allgood

It mainly boiled down to 3 key reasons:

Shipping is good. It's often easy to get caught up in the trap of endless delivery timelines, interspaced with laborious planning meetings, discussions and the like. Creating something and putting it out into the world is amazing. And this project was something small I could deliver and provide value. It was satisfying to actually see the project live.
Open source is good. A lot of my recent programming work outside of the jobby job has been on commercially focused products (a side hustle if you will). To avoid the complexity that billing and marketing bring, I wanted to create something entirely free and for no tangible benefit to myself other than self-satisfaction. Doing so enabled me to focus on the code and to craft a nice API (internal) - thinking about extensibility and simplicity.
It's a tool I'll use. Like most, I scratch my own itches. It's one of the great things about knowing how to code. You stumble across a problem you have, and you can solve it!

How I built it

By far, the most difficult part was setting up the build system so that it worked with both require and import syntax. It was a total pain, and tells me something not that nice about the state of the JS ecosystem - I digress. I knew this was going to be a pain though so I figured I'd use a boilerplate. The one I landed on needed a little bit of fudging but I got there in the end.

Afterwards, I got started on writing the check code. I started with memory usage as I thought this would be the easiest - simply using process.memoryUsage().

Now though, there was a challenge. The user could configure N checks to run on their healthcheck page. I needed to run all of the checks (in parallel, for speed) and then display the results in a consistent order. I also needed to call the correct check function. The solution I landed on was for all checks to use a consistent interface. This meant I could guarantee the output of each check was consistent. They are sort of decoupled from the app itself meaning the display message and other properties are entirely within their control.

Here is how the interface looked

export interface HealthCheck {
  status: Status;
  value: string;
  componentName: string;
  message: string;
  time: number; // Time in milliseconds that elapsed running the check
}

// The interface that each check implements
export interface CheckFn {
  (config: Config): Promise<HealthCheck>
}

export interface CheckRegistry {
  [key: string]: CheckFn;
}

I then created an object mapping of "checks" to their functions. And then tethered it together with .maps and Promise.all.

Knowing my own monkey mind, I wanted to make this as maintenance free as possible. So when new dependency PR's are created, I can just merge them straight away. Therefore, I took some time to create a test suite. Initially I wanted to use the new native node test suite. But after trying to wrangle with mocks I gave up and used Jest instead - which just worked.

One surprising outcome of this project was the package manager. Using pnpm was a really nice interface and extremely quick. There wasn't a lot of dependencies to install but I never got into the usual hassle of dependency wrangling I usually do.

The future

I'm not sure what this project will yield but after sending it round to some colleagues I've got a good response and had some good feature requests. It's fun to code and create things purely for the sake of the act of creation. If you know how to code, do the same - you'll feel amazing.

Terminology isn't universal

ntietz.com blog - technically a blog — Mon, 25 Nov 2024 02:00:00 GMT

A little while back, I wrote that we shouldn't say "auth" but should use other terms instead. I stand by my argument in general, but it also has another side to it: my suggested terminology makes sense in some domains, but not in all domains. And that's because terminology isn't—and can't be—universal.

After I wrote that blog post, I got an email from someone who had a really good point: the word "login" is often confused for "username"! My terminology suggestion doesn't work in all cases, so what's the right word to use instead? Unfortunately... there isn't one. There's no one pair of words that will work perfectly.

Terminology makes sense relative to the context it's in. If I say "node" to you, what does that mean?

My guess is you will think of either a node in a network/graph or a node as in a server. But you could also think of a lymph node or myriad other things!

The other term you could use in a graph context is "vertex". This is often clearer, but it runs into issues if you're talking about anything with geometry!

* * *

I ran into this headlong at my first job, working on a graph database. When talking about a graph in a distributed system, if you use the word "node" it is confusing: Do you mean a node of the graph or a node of the distributed system^[1]? We ended up choosing "vertex" to mean the thing in a graph, and that worked—but would not work if we also wanted to represent geometry, like in, say, a graph visualization where you'd be drawing vertices of polygons.

A lot of this confusion, I think, comes from when different domains overlap or border each other. Words which make sense in one domain will leak into the other, but they'll end up being confusing or imprecise there.

* * *

So what do we do about it?

A lot of it is accepting that we will run into challenges here, and there's not always a perfect solution. If we run into imprecise or confusing language, that could mean that suboptimal word choices were made. But there might not be a better choice! Some confusion will always be possible, and we have to roll with that and be understanding about it.

We also have to remember that there still might be a better choice! Even if it's not perfect, we can go for better, and we can improve the status quo. We can choose words that are more clear in our domain, our context, and make things clearer there.

The other thing is we have to accept that terminology can't be universal. Just because terminology works in one domain doesn't mean it will apply to all domains. This is inherent to the whole issue, because if it were universal then there would be one single optimal choice. But there's not. And since it's not universal, we sort of have to expect that sometimes we'll run into terms that are imprecise or confusing, and that they're chosen to make sense there.

Ultimately some of the confusion in this case is because the two are related overlapping domains. Computer networks are represented as graphs, and we talk about them in those terms to some extent. ↩

Insurance Insurance

taylor.town — Mon, 25 Nov 2024 02:00:00 GMT

Purchase Premium Insurance-Squared today!

Un serveur VNC vaniteux (Joke over RFB)

ache: Blog personnel — Mon, 25 Nov 2024 02:00:00 GMT

VNC est un protocole bien connu de partage de bureau distant. Ce blog post est l'histoire d'un projet d'un WE qui a conduit à fabriquer un VNC « vaniteux ».

Om krisen eller kriget kommer...

Liza Shulyayeva — Sun, 24 Nov 2024 23:01:33 GMT

Jag flyttade till en ny stad och fick en perfekt välkomstpresent i min brevlåda:

Om krisen eller kriget kommer är en broschyr från Myndigheten för samhällsskydd och beredskap (MSB). Den skickades ut till alla svenskar nyligen.

Configuring VSCode with Nix on macOS

Davis Haupt's Blog — Sun, 24 Nov 2024 19:00:00 GMT

Welcome back to the Nix on Mac series! By the end of this post, you’ll be able to fully configure VSCode through your Nix flake via home-manager, which we set up in part 2. This includes:

Custom keybindings and settings
Installing themes and extensions from a nixpkgs overlay
Properly aliasing VSCode and other macOS applications to /Applications for Spotlight

Without further ado, let’s dive in!

Installing VSCode

Nix won’t install non-free software like VSCode without you opting in. Add this line to your nix-darwin configuration module:

configuration = { pkgs, ... }: {
    # ...
    
    nixpkgs.config.allowUnfree = true;
    
    # ...
};

After that, installing VSCode is just one line of code in our home-manager config:

homeconfig = { pkgs, ... }: {
    programs.vscode = {
        enable = true;
    };
};

I’m sure many of you who are following along are using VSCode to edit your flake.nix file. I should note that installing VSCode through home-manager will likely blow away your existing settings when you call switch. Now would be a good time to back up your existing settings and extension list so you can replicate your setup from within Nix.

VSCode will be installed in /Users/$USER/Applications/Home Manager Apps/¹ after running switch.

If you’ve previously installed VSCode, go ahead and dump the copy from /Applications in the trash. Where we’re going, we don’t need ~~roads~~ globally installed programs!

Editor Settings and Keybindings

You can add user settings and keybindings pretty easily:

programs.vscode = {
    enable = true;

    userSettings = {
        # This property will be used to generate settings.json:
        # https://code.visualstudio.com/docs/getstarted/settings#_settingsjson
        "editor.formatOnSave" = true;
    };
    keybindings = [
        # See https://code.visualstudio.com/docs/getstarted/keybindings#_advanced-customization
        {
            key = "shift+cmd+j";
            command = "workbench.action.focusActiveEditorGroup";
            when = "terminalFocus";
        }
    ];
};

It almost looks like JSON! And in fact, converting Nix types to and from JSON is part of the Nix standard library.

Extensions

A huge part of VSCode is the bountiful extension ecosystem. home-manager lets you install extensions similarly to how it installs packages. The full VSCode marketplace isn’t present in nixpkgs by default, so we’ll need to install an overlay.

Nixpkgs overlays let you override and add new entries to nixpkgs. We can add the nix-vscode-extensions overlay by adding a line to our nix-darwin configuration:

{
  # ...
  inputs = {
    # ...
    nix-vscode-extensions.url = "github:nix-community/nix-vscode-extensions";
  };

  outputs =
    inputs@{ self
    , nixpkgs
    , nix-darwin
    , home-manager
    , mac-app-util
    , nix-vscode-extensions
    }:
    let
      configuration = { pkgs, ... }: {
        # ...
        nixpkgs.overlays = [
          nix-vscode-extensions.overlays.default
        ];
        # ...
      };

      homeconfig = { pkgs, ...}: {
        # ...
        programs.vscode {
          # ...

          userSettings = {
            # ...
            "workbench.colorTheme" = "Dracula Theme";
          };

          # ...

          extensions = [
            pkgs.vscode-marketplace.jnoortheen.nix-ide
            pkgs.vscode-marketplace.dracula-theme.theme-dracula
          ];
        }
      }
  # ...

vscode-marketplace is one of the properties that the nix-vscode-extensions overlay added to nixpkgs. Any VSCode extension in the marketplace should be accessible from pkgs.vscode-marketplace.$AUTHOR.$EXTENSION, where $AUTHOR.$EXTENSION is the same as the itemName property in the extension’s URL on the extension marketplace website.

Cleaning up lists of attributes with the `with` clause

Writing pkgs.vscode-marketplace in front of every extension will get tedious as as your list of extenions gets longer and make the list harder to read. Luckily Nix has a language construct to help with this: in front of any expression, you can type with <attribute set> to bring all attributes within the attribute set into scope. Our extensions list can look like:

extensions = with pkgs.vscode-marketplace; [
    jnoortheen.nix-ide
    dracula-theme.theme-dracula
];

Much cleaner!

Playing nice with Spotlight

You might have noticed that the version of VSCode that Nix installs doesn’t show up in Spotlight. Why is that? It’s unfortunately pretty simple: All artifacts that Nix and home-manager add to your system are symbolic links, and Spotlight won’t index symlinks. There’s a very long thread about this on GitHub if you’re interested in reading. From the various options discussed there, mac-app-util is the easiest way to get Spotlight working as expected.

We can add it as another input at the top of our flake and then modify our flake output to load mac-app-util in both nix-darwin and home-manager:

{
    description = "My system configuration";
    inputs = {
        # ...
        mac-app-util.url = "github:hraban/mac-app-util";
    };

    # ...

    in
    {
      darwinConfigurations."$HOSTNAME" = nix-darwin.lib.darwinSystem {
        modules = [
          # ...
          mac-app-util.darwinModules.default
          home-manager.darwinModules.home-manager
          {
            # ...
            home-manager.sharedModules = [
                mac-app-util.homeManagerModules.default
            ];
            # ...
          }
        ];
      };
    }
}

After running switch, VSCode should be properly copied to /Users/$USER/Applications/Home Manager Trampolines and should be available the next time Spotlight refreshes.

Stepping Back

We just included someone’s third-party Nix utility in our flake to do some custom behavior when Nix installs macOS GUI apps. You probably didn’t notice that mac-app-util is written in Common Lisp. I’ve never written Common Lisp, and my guess is you haven’t either. Even so, with fewer than five lines of code we were able make use of some code that someone wrote in their favorite niche programming language without needing to figure out how to build or run that code. Nix handled it all for us! I, for one, think that’s pretty amazing.

Conclusion

Our Nix config is now able to fully configure and manage VSCode, including installing extensions and themes. home-manager has just as deep an integration with many other utilities. If you haven’t already definitely go through and check out home-manager’s config options to find out what you can do with your favorite program.

You can find the full flake.nix for this installment on GitHub here. See you in the next one!

Notably, this isn’t inside the normal /Applications folder since home-manager can only install programs under your home directory. Even stranger, Spotlight can’t pick up our alias! We’ll come back and fix this at the end of the post. ↩

Tracking Ethereum Wallet Balance with GetBlock.io

Adnan's Random bytes — Sun, 24 Nov 2024 17:55:35 GMT

In this guide, we’re going to demonstrate how to write and run the simplest script for tracking Ethereum wallet balance with GetBlock’s RPC API. What is Token Ownership? Token ownership is just a digital version of holding an asset. It is similar to holding stocks, currencies, or commodities like gold or silver. Each token is linked to an account (or wallet address), and the balance of tokens in that account reflects ownership. When you acquire tokens, whether you are buying them on an exchange through trading, getting them as staking rewards, or someone giving you as a gift, they are transferred to the respective wallet addresses and you become their rightful owner. This ownership allows you to use these tokens in any possible way. Why is Tracking Tokens Important? Just like you keep an eye on other assets; whether they’re going up or down, tracking digital tokens is also very necessary. Tracking the tokens you own serves many purposes. First, it gives you a clear picture of your portfolio and its standing. Second, it helps you to stay on top of your investments. Also, it can be useful while tracking “interesting” wallets of whales, developers, team members, and so on. Accessing Ethereum blockchain using GetBlock API You may be curious about how to retrieve blockchain data, verify wallet balances, and perform various tasks programmatically. Enter GetBlock, a Blockchain-as-a-Service (BaaS) platform that provides a fast and easy API connection to full nodes from 50+ leading blockchain platforms. It takes the headache out of tracking your token holdings. Think of it as your personal blockchain data assistant that provides quick and reliable access to token information. It facilitates you to request on-chain information of a blockchain without setting up manually with JSON-RPC, REST, and WebSockets. In this post, we are going to access the Ethereum blockchain and we will be fetching the ETH balance of a certain wallet. We will be using the API endpoint that retrieves the balance. I will be showing how you can use APIs in Python. Creating an Access Token To access GetBlock API you should first create an access token: go to GetBlock and sign up. Once you are signed up and verified, you will be welcomed by this dashboard: To access APIs you would need to create an access token first. Initially, no access token is available. Selecting the Blockchain Protocol and Network If you click on the Start Now button, you will be asked to select the protocol, network, and API. I selected Ethereum as the protocol, Mainnet as the network, and JSON-RPC as the API format. Once you click the Get button you will see an entry for Ethereum like the one below: Retrieving the API Endpoint Now click on Ethereum’s entry and you will see something like the below: As you can see, it gives the main API URI along with the access token: https://go.getblock.io/f7f1e4a673301d0fc69a9848848 Please note that this access token is available for this very network only. We have selected the node and the network. Now all we need is an Ethereum wallet address so that we can find how many ETH coins it has. We are going to use the eth_getBalance method for this purpose. On the page, you will see a curl example like the one below that you can run on the terminal curl --location --request POST 'https://go.getblock.io/<ACCESS-TOKEN>/' --header 'Content-Type: application/json' --data-raw '{"jsonrpc": "2.0", "method": "eth_getBalance", "params": ["0xfe3b557e8fb62b89f4916b721be55ceb828dbd73", "latest"], "id": "getblock.io"}' You set JSON headers, the method parameter is set to eth_getBalance and the params contain the wallet address. Checking Balance with cURL From the EtherScan website, I just picked a random wallet (0xF977814e90dA44bFA03b6295A0616a897441aceC) which you can visit here. As you can see the balance (at the time of writing the article) is 799,655.363544175804339796 ETH. Now, first, run the cURL request (real access token has been tempered) in the terminal: curl --location --request POST 'https://go.getblock.io/f7f1e4a67330490ea69ab58737c7' \ --header 'Content-Type: application/json' \ --data-raw '{ "id": "1", "jsonrpc": "2.0", "method": "eth_getBalance", "params": [ "0xF977814e90dA44bFA03b6295A0616a897441aceC", "latest" ] }' When I run this, I get the following response: {"jsonrpc":"2.0","id":"1","result":"0xa955677463bb9136d654"} The result field contains the balance of the wallet in HEX format. Automating Balance Checks with Python Now convert this into a Python script: import requests import json # Set the URL and headers url = "https://go.getblock.io/f7f1e4a67330490da69ab5bd9a9bf127" headers = { "Content-Type": "application/json" } # Set up the request payload payload = { "id": "1", "jsonrpc": "2.0", "method": "eth_getBalance", "params": [ "0xF977814e90dA44bFA03b6295A0616a897441aceC", "latest" ] } # Send the POST request response = requests.post(url, headers=headers, data=json.dumps(payload)) # Check if the request was successful if response.status_code == 200: # Convert the hex balance to decimal and then to Ether balance_wei = int(response.json()['result'], 16) balance_eth = balance_wei / 10**18 # Convert Wei to Ether print(f"The balance of the address is: {balance_eth} ETH") else: print(f"Error: {response.status_code}") print(response.text) Assuming you have Python installed on your machine and you have saved the above code in the file named balance.py, when you run this code it prints: The balance of the address is: 799655.3635441758 ETH. The balance_eth value matches the balance shown on Etherscan, proving the accuracy of this API. The balance_eth value matches the balance shown on Etherscan, proving the accuracy of this API. As you see, in just a few lines of code you can track a portfolio. From here, the possibilities are endless; you can come up with a notification system that will alert you how much your portfolio goes up or down, a custom dashboard that caters to your needs, and many other things. GetBlock made it easier and smooth. Closing thoughts To wrap things up, keeping track of token balances and wallet holdings is essential for managing digital assets effectively. GetBlock makes it easy for developers to access Ethereum’s blockchain data without the hassle of setting up and maintaining nodes. By using methods like eth_getBalance, you can quickly get real-time information on wallet balances, which helps stay on top of your assets. With GetBlock you can focus on building new features and applications. Whether you’re working on dApps, managing a portfolio, or building financial tools, GetBlock’s straightforward API access makes interacting with the blockchain smooth and efficient. Starting from November 25, 2024, GetBlock is welcoming all its users to join the Black Friday promo event. Whether you’re a new or experienced user, GetBlock has something for you to offer. The last day to redeem this offer is Monday, December 2, 2024.

The post Tracking Ethereum Wallet Balance with GetBlock.io first appeared on Adnan's Random bytes.

A short introduction to Interval Tree Clocks

Separate Concerns — Sun, 24 Nov 2024 09:00:00 GMT

This post is a rough transcription of a lightning talk I gave at dotScale 2017.

EDIT (2024-11-24): this post is on HN today. For people finding this now, Lima has shut down since then.

One of the things I work on at Lima is master-master filesystem replication. In this kind of system, we need to track causality. In a nutshell, given two events modifying a given piece of data and originating from different nodes in the system, we want to know if one of those events could have influenced the other one, or in other words if one of those events “happened before” the other one.

To do that, we use constructs such as Version vectors. The idea is that we give each node in the system a globally unique identifier, and we associate it to a counter.

When an event modifying data on a node occurs, we increment the local value of the corresponding counter by one.

Version Vectors are partially ordered. Given two vectors, if we can find one such that, for every node, its counter is higher than the other one, then we say it descends the other one, meaning the related event “happened after” the other one. Otherwise, we say that the vectors are concurrent, and typically that means we will probably have some kind of data conflict to solve.

When we merge data changes we also merge the vectors, and to do so we take the maximum value of the counter for every node.

This works fine in most cases, but there is one case where it breaks down: highly dynamic systems experiencing a lot of churn. This means systems where nodes join the system, modify some data, then leave forever. The issue with such systems is that, even though there may not be a lot of active devices at any given point in time, the number of unique node identifiers in version vectors keeps increasing. We call that issue actor explosion.

Interval Tree Clocks are an attempt to solve this problem. Instead of giving a unique identifier to every node in the system, we take the real-valued interval [0, 1] and attribute a part of it (not necessarily contiguous) to every node. On top of it, we draw an integer-valued curve. We call the combination of the interval share and the curve a stamp.

To add a new node to the system, we start from an existing node and we fork it, meaning we give part of its share of the interval to the new node.

When an event occurs on a node, it increases the height of the curve of its copy of the stamp anywhere within its share of the interval. Comparison works similarly to Version Vectors: if the curve of a stamp is above the other one, it descends it, otherwise the curves intersect and the stamps are concurrent.

When a node wants to leave the system, it merges back with any other node and surrenders its share of the interval. To merge, we just take the maximum curve.

The beauty of this scheme is that a node only has to know about its share of the interval, not information about all other nodes. There are no globally unique node identifiers.

If we choose how we increase the height of the curve when an event occurs in a clever way which I will not detail here, we can ensure the complexity of the curve remains low. We can then encode it efficiently using a tree-shaped data structure and a custom binary format, with a size that depends more on the number of nodes interacting with the data at a given point in time than on the overall number of nodes which have touched it since inception.

If you want to know more, I encourage you to read the 2008 paper by Paulo Sérgio Almeida, Carlos Baquero and Victor Fonte; it is one of the best I know on the topic of causality. You can also check out my Lua implementation of ITC or one of the other implementations linked in the README.

EDIT (2024-11-24): other interesting resources are this slide deck by Carlos Baquero and those posts by Fred Hebert.

Platform-Specific Resources in SwiftPM

defn.io — Sun, 24 Nov 2024 09:00:00 GMT

Noise packages Racket & Chez Scheme boot files¹ for all the platforms it supports. Originally, that was just x86-64 and arm64 macOS, but when I added iOS support, that extended to include arm64 iOS. These boot files take up about 45MB for each arch+os pair and the way I originally distributed them was placing them all in a boot folder and adding them to the resources list for the core Noise target.

That worked fine, but it seemed like a waste to lug around an extra 90MB of data that would never be used inside my iOS apps. Looking at the PackageDescription docs, there doesn't appear to be a way to filter resources by platform. There is an exclude property on Targets, which I tried to use by making the Package.swift script manipulate the target at runtime, but I couldn't figure out a way to get that working with cross-compilation.

Next, I tried writing a SwiftPM build tool plugin to remove files based on the target platform, but plugin execution is sandboxed and I couldn't figure out a way to move the boot files out of the package context².

Finally, I settled on just making separate targets for macOS and iOS. Each target contains its respective boot files and the main target conditionally depends on the platform-specific targets. I really wanted to avoid this approach because it's somewhat ugly and it means I have to manually wire things around that the build system should be able to do for me, but, for now, this seems like the most sensible approach.

Object files that contain the Racket and Chez Scheme runtime. ↩
As I'm writing this, I wonder if I could've combined the exclude property and a build tool plugin to define a folder into which I could've moved the unneeded boot files during the build. ↩

Hope Of Return

ST — Sun, 24 Nov 2024 02:00:00 GMT

The scent of grandmother’s kitchen still lingers in my dreams. Cardamom and cinnamon, the steam rising from the pot as she stirred, her silver bangles jingling with each movement. I was ten when I last saw that kitchen, when I last sat on that worn wooden stool watching her cook.

Macro and Microphenomenology

Superb Owl — Sat, 23 Nov 2024 19:59:00 GMT

How to study the dynamics of the mind

Highlighted code in slides

fasterthanli.me — Sat, 23 Nov 2024 19:40:00 GMT

I have obsessed about this long enough, I think it’s only fair I (and you!) get some content out of it.

When I started writing this article, I was working on my P99 CONF slides. Those slides happen to include some bits of code. And because I’m a perfectionist, I would like this code to be syntax highlighted, like this:



let addr: SocketAddr = config.address.parse()?;
let ln = TcpListener::bind(addr?
 config

French Toast

Sam Hooke — Sat, 23 Nov 2024 11:30:00 GMT

Ingredients

For two people (four pieces, two per person):

For the liquid:
- 2 large eggs.
- 120g of milk.
- 15g of plain flour.
- 25g of granulated sugar.
- A pinch of salt.
- 1 tsp ground cinnamon.
- 1 tsp vanilla extract.
For frying:
- Butter.
- 4 slices of bread (thick but not too thick, preferably a day or two old).

Method

Mix all ingredients for the liquid.
Heat up a frying pan with some butter.
Pour the liquid into a dish, then lay two pieces of bread in the dish to soak up the liquid, then turn so the other side soaks them up.
Place the two slices on the fying pan and cook for a few minutes on each side until golden.
Repeat for the other slides of bread.

Foundations of High-Performance Engineering Teams

Greg Herlein — Sat, 23 Nov 2024 10:00:01 GMT

The US Navy gave me a solid foundation in leadership and operational excellence. But translating that into the business world requires a few more things.

Making money from free content

I Learnt — Sat, 23 Nov 2024 09:52:00 GMT

Sam Harris gives away his content for free but still makes money.

Interview: Wie Atlassian auf Cloudangst und Überkonfiguration seiner Tools reagiert

Backendhance — Sat, 23 Nov 2024 09:47:27 GMT

Ich war kürzlich auf der Team ‘24, quasi Atlassians Hausmesse, und habe mich dort mit Matt Schvimmer, Senior Vice President und Head of Product für das Agile- und Devops-Portfolio von Atlassian, getroffen. Wir haben darüber gesprochen, welche Unterstützung die Atlassian-Tools für moderne Softwareentwicklung bieten.

Dieses Interview erschien erstmals im Chef von Devs Newsletter #57

Besonders für Teams, die in komplexen Umgebungen arbeiten, spielen Jira, Bitbucket, Compass und das neu angekündigte Focus eine große Rolle. Schvimmer hat mir Einblicke in die Weiterentwicklung der Atlassian-Tools und in zukünftige Pläne gegeben. Außerdem sprach er mit mir über Herausforderungen wie die Synchronisation von Dokumentation mit dem Code und den Umgang mit der Komplexität von Microservices. Aber auch die Überkonfiguration von Tools oder Bedenken gegenüber der Cloud waren im Interview Thema.

Deploy a Node.js app on EC2

Stories by Himanshu Singh on Medium — Sat, 23 Nov 2024 08:03:45 GMT

Hello everyone 👋,
My name is Himanshu Singh. I am an Engineer, and after working as a front-end engineer for over two years, I started doing full-stack.

Recently, I did a small side gig, where I had to build a price change tracker for tokens on Solana.

I have written a blog explaining what I did in that.

In this blog, I wanted to write how I deployed the Node.js app I built in that gig. So that, you know how you can do the same as well.

Let’s begin!

The blog I wrote — https://hsnice16.medium.com/track-token-price-change-in-solana-7d6dbd363c69

Launch a new EC2 instance

First, we will start with launching a new instance, that you can do from the EC2 dashboard.

After clicking on that, you will be redirected to a Launch page, where you will have to fill in the details of the instance, like its name, OS, type, and other things. For learning, you can choose free tier options.

Like, for OS, you can choose Ubuntu.

For Instance type, you can keep the default free tier selected

The key pair field is also required, so you will have to fill that. It’s used to securely connect to your EC2 instance.

After filling in all the details, on the right-hand side, you can click on “Launch Instance”.

After that is successful, you can go to the Instances dashboard by clicking on it under the Instances dropdown on the left-hand side menu, select the instance you created, and click on Connect in the top menu.

You can keep the selected option as it is and click on the Connect button, which will connect you to your EC2 instance on Web UI.

Install dependencies

EC2 instance is just like a computer running on the cloud. So, the way you will clone and run the app locally, you will do the same thing in the EC2 instance as well.

As we know, to run a Node.js app, we need to install Node first on our computer, we will have to do the same in the instance. So, let’s first install the Node.

AWS team has already created a document on how you can install Node on an EC2 instance, so you can check the “To set up Node.js on your Linux instance” section in that — https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-up-node-on-ec2-instance.html

After that, we will install Git. And, to do that, you can use the sudo apt-get install git command, and can also check the Git installation guide if you face any issues — https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

We need one more dependency to deploy our app.
So, locally when you start your server and then if the computer sleeps after some time of inactivity, the server stops as well. And, the same is the case with the EC2 instance, so to keep our app running forever, we will need to install the pm2 library — https://www.npmjs.com/package/pm2

To do that, you can use npm install pm2 -g

Start the app

We have installed all the dependencies we will gonna need, so it’s time to start our app.

First, we can clone the repository from the GitHub. And, one way to do that is by using the git clone https://github.com/hsnice16/alert-on-price-change.git command.

After that, we can follow the project packages installation guide mentioned in the README file.

And, in the last, to run the app, and keep it running we can use the command pm2 start index.js (note that we don’t need node index.js).

That’s all

Thank you for reading it till the end.
I am hoping that after going through this blog, you will be able to deploy your app on EC2.

Let’s connect on Twitter (Now X).

You can find the gig project link on my GitHub — https://github.com/hsnice16/alert-on-price-change

If you found this blog and this project helpful, you can also sponsor me on GitHub — https://github.com/sponsors/hsnice16

Wisps

RobKohr's Blog — Sat, 23 Nov 2024 02:00:00 GMT

Wisps

Making SQL Keyword Suggestions Work

xnacly - blog — Sat, 23 Nov 2024 02:00:00 GMT

Implementing levenshtein distance naïvely (and very slow) in Rust

Eng Log 0001

Mehul Kar's blog — Sat, 23 Nov 2024 02:00:00 GMT

One of the biggest reasons I don't write more often is because I have trouble figuring out what to call it. But I want to make more room for just writing things and worrying less about form. So instead of coming up with a title, I'm just going to index it by a number. I've seen other writers do this also. Normally at this point I would try to find evidence of these other writers and reference them and their thoughts about this style, but I'm shedding that burden too. So anyway, the reason I made these decisions is to write down a thought I had. /preamble.

Last week, some folks from Sentry reached out about a flakey test detection product. My team owns our backend codebase and flakey tests had been a problem recently, so our CTO put us in touch with these nice folks.

We don't really see a lot of flakey tests in our CI anymore. But I realized the reason for that is that we fixed a totally orthogonal problem: our package graph. We have a large monorepo that uses turbo to determine which tests to run based on the files changed in each pull request.

Previously, almost every pull request invalidated all 500 packages in the monorepo, which meant that every test would have to run. When we started tracking this data, we saw that ~60% of CI runs had to run the whole suite. After we fixed our package graph, that's down to 3% of CI runs. That means that many tests are just not running as frequently now. There are a few reasons this means that "flakey tests aren't a problem" anymore. Because fewer tests run,

we just don't run into the problem as much, all else being the same
flakiness due to resource contention (e.g. timing problems) is gone
the likelihood of tests that only fail in certain orders and combinations is reduced

So for the most part, we haven't fixed flaky tests, but by fixing a different problem, we changed how often it surfaces, and how much we need to care about it right now.

This is interesting in the world of engineering and project management, because it's hard to communicate the relationship between two seemingly unrelated problems. If we had done a better job of communicating the impact of a bad package graph and the effect of fixing it, it's possible that we'd never even have been introduced to Sentry to take this call.

/end

Thanks for reading this post via RSS!

2024-11-23

ho.dges.online — Sat, 23 Nov 2024 02:00:00 GMT

Dear friend, you have built a Kubernetes

Mac's Tech Blog — Sat, 23 Nov 2024 02:00:00 GMT

This post will make more sense if you first read Dear Sir, You Have Built a Compiler.

Dear friend,

I am afraid to inform you that you have built a Kubernetes. I know you wanted to "choose boring tech" to just run some containers. You said that "Kubernetes is overkill" and "it's just way too complex for a simple task" and yet, six months later, you have a pile of shell scripts that do not work—breaking every time there's a slight shift in the winds of production.

Surely, switching to Docker Compose will be the end of your woes; at least that way, someone else maintains a standard config file format for you to use. But wait, Compose is still not a holistic solution. "Do I really need a separate solution for deployment, rolling updates, rollbacks, and scaling?" you ask yourself? Surely not. Your app is so simple; a backend, a reverse proxy, postgres, and a job runner. So you march on, and add another few sections to your deploy.sh script, certain, that this will be the last of what you need to do to maintain this pile of hacks.

Ah, but wait! Inevitably, you find a reason to expand to a second server. While it's true a single server can go a long way many things can force this decision, such as the need for special hardware, high availability, or the speed of light. Tired, you parameterize your deploy script and configure firewall rules, distracted from the crucial features you should be working on and shipping. One of your team members suggests connecting the servers with Tailscale: an overlay network with service discovery. After that you will know, yes know, that there is no tougher complexity hurdle to clear than the networking.

Except if you quit or go on vacation, who will maintain this custom pile of shell scripts? The untested tarball handling? The inscrutable iptables rules? Who will know about those undocumented sysctl edits you made on the VM? So you add everything to Ansible, so that you may treat your VM as immutable and version-controlled. Certain, of course, that because you’re not using Kubernetes, it is going to be way easier to maintain than a Kubernetes cluster. What glorious engineering, you say to yourself.

In the last leg of your journey to avoid building a Kubernetes, your manager tells you that your app needs to programmatically spawn other containers. Spawning containers, of course, requires you to mount the Docker socket in your web app, which is wildly insecure. Not my problem, your manager says. So you write a separate service that exposes a safe subset of the Docker API to your web app. Done at last, you say to yourself, without having to build a Kubernetes.

A standard config format, a deployment method, an overlay network, service discovery, immutable nodes, and an API server. Dear friend, you have built a Kubernetes.

Addressed to,

Those who wanted to avoid Kubernetes.

PS: I don't mean to imply that you can never roll your own deployment method that fits your needs better than Kubernetes, or that nothing will ever be better than Kubernetes. I just want to caution you, my friend, to make sure you understand the problems Kubernetes solves before dismissing it as overly-complex.

The Diplomat And Reality

ST — Sat, 23 Nov 2024 02:00:00 GMT

Britain’s Diplomatic Delusions: Netflix vs Reality

Mgmt Configuration Language: Functions

The Technical Blog of James on purpleidea.com — Sat, 23 Nov 2024 01:16:00 GMT

It’s been a little while since I introduced the Mgmt Configuration Language. I originally wrote this article in 2019, and as I was working on it alongside the code for functions, when I realized that lambdas didn’t work properly. It took some time to finally solve that properly. Since then I’ve been working to get mgmt production ready. It’s properly useful for production now, and so it’s time for me to catch up on my documentation. I’ve revived and updated this article, let’s go…

IdentityServer in Docker Containers – Part 1

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Fri, 22 Nov 2024 16:31:11 GMT

Getting Duende IdentityServer and a client application up and running in separate containers can be challenging. This blog post will provide a step-by-step guide for a smooth setup and show you how to resolve common challenges along the way. We will also learn about security, cookies, ports, containers, and certificates. This is a big topic, […]

The post IdentityServer in Docker Containers – Part 1 appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

A Walk-Through of String Search Algorithms

Photon-Lines Substack — Fri, 22 Nov 2024 16:23:30 GMT

A walk-through of how some of the popular string-search algorithms work with a focus on the well-known Boyer-Moore and Robin-Karp algorithms.

OnlineOrNot Diaries 22

Max Rozen — Fri, 22 Nov 2024 09:10:00 GMT

Feels like I've already said everything I had to say

Toggle macOS menu bar from you know where

xenodium.com @alvaro — Fri, 22 Nov 2024 02:00:00 GMT

I'm a fan of macOS's auto-hide menu bar setting. Unless I'm reaching out to a menu item, I don't typically need to have a visible menu bar, so I set auto-hide to "Always".

On rare occasions, I turn this setting off (say for a screen grab). While reaching out to macOS Control Center is OK, I wanted to quickly toggle it from the comfort of Emacs via M-x fuzzy search.

We can leverage AppleScript to toggle the setting on and off. In the past, I would write shell scripts for this kinda thing and invoke from the command line. These days, I dump them into a dwim-shell-command and quickly forget about its implementation. From then on, I just M-x and fuzzy search for some magical incantation (I'm looking at you ffmpeg). I got a bunch of these things…

(defun dwim-shell-commands-macos-toggle-menu-bar-autohide ()
  "Toggle macOS menu bar auto-hide."
  (interactive)
  (dwim-shell-command-on-marked-files
   "Toggle menu bar auto-hide."
   "current_status=$(osascript -e 'tell application \"System Events\" to get autohide menu bar of dock preferences')

if [ \"$current_status\" = \"true\" ]; then
    osascript -e 'tell application \"System Events\" to set autohide menu bar of dock preferences to false'
    echo \"Auto-hide disabled.\"
else
    osascript -e 'tell application \"System Events\" to set autohide menu bar of dock preferences to true'
    echo \"Auto-hide enabled.\"
fi"
   :utils "osascript"
   :silent-success t))

…and that's all there is to it.

Warlock AI

er4hn — Fri, 22 Nov 2024 02:00:00 GMT

Then Life Continued

Goodbye

nutcroft — Fri, 22 Nov 2024 02:00:00 GMT

I had been struggling to write for many years. Back then I was dreaming of having a blog with lots of entries. Now I do. I admired the people who did and now I'm one of them.

The feeling now is: this blog is so old. It's ten years old. I am definitely not the same person I was then. Most of these texts no longer reflect my opinions and thoughts. I’m probably ashamed for most of them (so don't read them!).

But that's ok, isn't it? Everything comes and goes. And now this blog goes too.

So, goodbye!

On Music Tabs

zserge's blog — Fri, 22 Nov 2024 02:00:00 GMT

I love music. I love playing music. I love making musical instruments and music software. I don’t have formal musical training. I’ve never taken lessons, and honestly, I’m a pretty bad musician. What I do excel at, though, is hoarding musical instruments. At this point, name any instrument, and there’s a decent chance I have it somewhere in my room. Some of these instruments never played anything better than “Twinkle, Twinkle Little Star”, others featured in a recording or two, and only a few see regular play.

ConvoCards Launch Retro

Brian Sunter — Fri, 22 Nov 2024 02:00:00 GMT

Thoughts and learnings from launching a new app.

Workload

Mike Schaeffer — Fri, 22 Nov 2024 02:00:00 GMT

Lately, I've been reading "Beyond Mach 3"), Col. Buddy Brown's memoir of his years piloting the U-2 and SR-71 spy planes. If you like that sort of thing, it's an interesting read on its own, but it also has parallels to the software industry, particularly in the way it describes pilot workload. The concept of pilot workload provides a nice way to think explicitly about the burdens our systems place on us as engineers and operators to keep them running properly.

If you're not familar with the U-2, it's a 1950's era aircraft designed to overfly hostile territory and take high resolution pictures of the ground for later analysis. It was a U-2 that flew over Cuba to take the pictures of Soviet missiles at the beginning of the Cuban Missile Crisis. To make these sorts of missions possible, the plane was designed to fly at very high altitudes, upwards of 70,000 feet. This is almost double the altitude of a modern passenger airliner.

What may not be obvious is that airplanes get progressively more difficult to fly at higher altitudes. If speed drops below the stall speed, the wings stall and lose lift. If speed gets too high, the airflow separates and the wings lose lift. Either way, if the speed deviates out of this window, the pilot loses control entirely. Due to the decreasing density of the atmosphere at higher altitudes, this window gets narrower as the plane flies higher. Fly high enough, the window closes entirely, and you've just discovered the ceiling of your aircraft. It can't fly that high at all.

A U-2 flying at 70,000 feet is flying very close to its ceiling. The range of permissable speeds at this point in its flight is just 5-10mph wide. Regardless of what else is going on, the pilot has to be completely precise managing their speed in order to retain control. This region of a flight envelope is referred to as the coffin corner), due to the catastrophic implications of a mistake.

On top of the work required to keep the plane in the air, the pilot of a U-2 is also taking celesital navigation readings every 30 minutes, compensating for the curvature of the earth, and running the sensor packages that are the point of the mission in the first place. In the best case, a mistake means the mission doesn't produce the desired results. In the worst case, a mistake means a crash in hostile territory and an international incident. To put it plainly, the U-2 asked a lot of its pilot as part of its basic operation. In aerospace engineering, this is referred to as having a high pilot workload.

For the U-2 program, this was arguably an appropriate design choice. The program had the ability to be highly selective in the pilots it chose, it had the ability to engage in extensive and costly training of those pilots, and it was operating at the limits of available technology. This was the only way their mission requirements could be satisfied at the time, so they were forced into a design that placed high workload demands on the pilot. As soon as technology got better, it became possible to fullfill many of the same requirements with less likelihood of error and overall lower risk, which is what happened.

There are a few lessons in this for software professionals. While most commercial software systems don't overfly hostile enemy territory, they hopefully do have tangible real world implications anyway. Otherwise, what's the point of building software at all? The reality is that if your software matters, operational failures will have tangible real-world impacts and costs. This means that to the extent economically possible, we have to get it right. This is where the workload concept can play a role.

In the context of aerospace engineering, workload refers to the burdens on the pilot and crew to keep the aircraft operating as desired. The same sort of concept also applies to software. How much time and effort is required to keep a software system operating properly? Costs here can range from everything to periodic backups, monitoring, difficulty of installation and upgrades, environment management, complexity of testing... the list goes on and on. Viewed from that perspetive, you should ask yourself explicitly if you and your team are you building a system with a low or a high workload. What does it take to keep your system running, and is it appropriate for what it does?

At some level, this is old news. Build, deployment, and test automation have been on the software professional's agenda for literally decades at this point. 17 years ago, Jeff Atwood wrote about Rico Mariani's highly related concept of the pit of success. That said, how close are you really to where you want to be with respect to the systems you help build and maintain? Even if the concept of workload is old, the need to revisit it and make sure you're handling it appropriately is evergreen.

Mixtape Maker

Home on Lagomorph — Thu, 21 Nov 2024 09:00:00 GMT

Tired of hand-calculating how you should split up your playlist between the two sides of a cassette tape? This widget is designed to make it simple; set your tape length, drag your music files directly onto this page, and move your tracks between the two playlists.

When you’re ready to record, connect your computer to your tape deck (however you prefer to do so) and press play!

The player will automatically play blank gaps between songs - but it doesn’t add gaps to the beginning or end, so be sure your tape is ready to record.

The simple life

Martin Rue — Thu, 21 Nov 2024 09:00:00 GMT

More things should not be used than are necessary.

The RAG Challenge: A GitHub Copilot Experiment

Prasham H Trivedi — Thu, 21 Nov 2024 03:00:00 GMT

Recreating Tim Kitchen’s RAG application using GitHub Copilot and exploring how it compares to other AI coding assistants like Aider, Cursor, and Windsurf.

chatgpt-shell goes offline

xenodium.com @alvaro — Thu, 21 Nov 2024 02:00:00 GMT

Since chatgpt-shell going multi-model, it was only a matter of time until we added support for local/offline models. As of version 2.0.6, chatgpt-shell has a basic Ollama implementation (llama3.2 for now).

chatgpt-shell is more than a shell. Check out the demos in the previous post.

For anyone keen on keeping all their LLM interactions offline, Ollama seems like a great option. I'm an Ollama noobie myself and already looking forward to getting acquainted. Having an offline LLM available at my Emacs fingertips will be super convenient.

For the more familiar with Ollama, please give the chatgpt-shell integration a try (it's on MELPA).

v2.0.6 has a basic/inital Ollama implementation. Please give it a good run and report bugs.

You can swap models via:

M-x chatgpt-shell-swap-model

Help make this project sustainable. Sponsor the work.

The Curse of Markdown

Rodrigo Pombo — Thu, 21 Nov 2024 02:00:00 GMT

And the content website wasteland

Software Is Bananas

Prasham H Trivedi — Thu, 21 Nov 2024 02:00:00 GMT

Kent Beck has shared some good insights in his substack. If I have to start with something, this is probably it. This post takes a short and stingy look around real “aglileness” of software development. Short but quick deployments and closer feedback loops instead of long and drawn out processes. Because at the end of the day, there is one bug lingering only in production enviorment which is waiting to be revealed.

Should Programming Languages be Safe or Powerful?

Ashton Wiersdorf on Lambda Land — Thu, 21 Nov 2024 02:00:00 GMT

Should a programming language be powerful and let a programmer do a lot, or should it be safe and protect the programmer from bad mistakes? Contrary to what the title insinuates, these are not diametrically opposed attributes. Nevertheless, this is the mindset that underlies notions such as, “macros, manual memory management, etc. are power tools—they’re not supposed to be safe.” If safety and power are not necessarily opposed, why does this notion persist?

The problem—I think—is that historically you did have to trade safety for certain kinds of power: if you wanted to write a high-performance device driver, C—with all its unsafe behavior—was your only option. This founded the idea that the “power tools” of the industry were fundamentally dangerous.

There’s a few things wrong with this though:

Power is relative to the domain of interest. Both Haskell and C are powerful, but in completely different ways. So, when judging whether an aspect of a language is powerful or not, consider its application.
Expressive languages get you power without sacrificing safety. New advances in programming language research have found ways to express problem domains more precisely. This means that we have less and less reason to breach safety and reach into the unsafe implementation details to get our work done.
It’s good to add safety to power tools. A safe power tool is more trustworthy than an unsafe one. This holds for real-world tools: I will never use a table saw without a functioning saw stop.

Specifically in the case of macros, there’s been an evolution from powerful-but-unsafe procedural macros in Lisp to safe-but-less-powerful pattern macros in Scheme, and finally to powerful-and-safe macros in Racket.

More safety means higher reliability—something that everyone wants. And with advances in making languages more expressive, you can have a language perfectly suited to a particular domain without sacrificing safety.

What makes a language powerful? #

A language that lets you do more of what you want to do is more powerful than a language where you can’t do what you want. But what does “what you want to do” encompass? If you want to write device drivers, then C is great for you. However, C is not as expressive in some of the ways that, say, Haskell is. For example, in Haskell, I can write lazy, recursive definitions. Here’s a list of all Yes, all the Fibonacci numbers. Haskell is lazy; this will compute as many as you ask for. the Fibonacci numbers:

fibs = 0 : 1 : zipWith (+) fibs (tail fibs)

Before you tell me that that’s just a useless cute trick, I actually had to use this when I was building the balancing algorithm in my rope data structure for my text editor written in Haskell. Haskell is incredibly powerful in an expressive sense: a single line of code can elegantly capture a complicated computation.

The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.

Edsgar Dijkstra

Power is closely related to the domain of interest: a language is powerful in a particular realm of problems. C is powerful for working with memory directly. Conversely, Haskell or Racket is more powerful than C in pretty much every other domain because these languages give the user tremendous ability to match the program to the domain. This is a meta-power that sets high-level languages apart from lower-level ones.

Safe languages can be just as powerful as their unsafe counterparts—in many cases, they are more powerful because the abstractions they create better fit the domain. Whenever a tradeoff between power and safety must be made, that is a sign that the language is not the right fit for the domain.

Consider how immutability gives you local reasoning power. At one of my industry jobs, our codebase was a mixture of Ruby and Elixir. Both are safe languages, but Elixir is immutable. When I was working on some Elixir code, I could read:

user = get_user(session)
name = get_user_name(user)
do_something_else(user)

and I didn’t have to worry about user getting modified in the call to get_user_name. To understand the output of this function, I didn’t have to worry too much about the implementation of get_user_name.

In contrast, if you did the same sort of thing in Ruby:

user = get_user(session)
name = user.get_name()
user.do_something_else()

the get_name method could do something sneaky like set name to "blank" if it didn’t exist. You might think, “well, just document that behavior.” Now I need to read the documentation of every function I encounter—I might as well go read the code to be sure the documentation isn’t out of date. Local reasoning means to understand what do_something_else is passed, I don’t have to worry in the first place if get_name will do somethig to the result of get_user. In this case, I did have to understand what every method call did to understand the function. This made it harder to track down errors because I had to account for all the side effects that could happen at every method call.

Certain things like immutability might seem constraining, but constraints can liberate you by allowing you to rely on particular behaviors. Elixir doesn’t let you modify things in-place, but you can rely on this, which makes understanding and composing code easier. Haskell forces you to express side-effects in the type system, but this lets you know that calling a function with a signature like String → Int won’t do any IO or throw an exception. Rust doesn’t have null like in Java, but you know when you get a pointer, you can safely dereference it and you don’t have to do all the null checking that you have to do in Java.

Case study: macros in Lisp, Scheme, and Racket #

The evolution of syntax macros in Lisp, Scheme, and Racket provide an interesting real-world instance of how safety and power can start off as a trade-off, but with better language design, become complimentary.

Lisp macros: unsafe but powerful #

I don’t have the space here to do a deep dive into Lisp macros, but here’s the short of it: Lisp macros are just functions that receive code as data. This code is represented as nested lists of symbols. All a macro needs to do is return a new list of symbols that will be spliced right into the call site.

(defmacro my-or (thing1 thing2)
  `(let ((tmp ,thing1))
     (if tmp tmp ,thing2)))

;; calling the macro
(my-or 1 2)
;; expands to
(let ((tmp 1))
  (if tmp tmp 2))  ;=> 1

The problem with this is that these macros are unhygienic: if I introduce a new variable, as I did with tmp in my-or, that is just a bare symbol that can be inadvertently captured producing unexpected output:

(let ((tmp 99)) (my-or nil tmp))
;; expands to
(let ((tmp 99)) (let ((tmp nil)) (if tmp tmp tmp))) ;=> nil

This is very bad! To use a macro safely, you need to be sure that it’s not introducing variables that you might accidentally capture. Lisp provides a mechanism Lisp has a function called gensym which makes a fresh symbol for you to use. Some other languages such as Julia have a gensym function; gensym is a poor substitute for proper hygiene. to avoid some of the pitfalls with variable capture, but that’s not the end of the danger. If I have a macro that expands to a call to a function, e.g. printf, I would expect this to be the printf in scope at the time I defined the macro. However, this might not be the case—a user might inadvertently redefine a function, and then the macro would not behave in the expected way.

Scheme macros: safe but less powerful #

Scheme has a faculty called syntax-rules, which lets you define transformations between a pattern and a template:

(define-syntax my-or
  (syntax-rules ()
    ((_ thing1 thing2) (let ((tmp thing1))
                         (if tmp tmp thing2)))))

Rust’s macro_rules! form is essentially syntax-rules from Scheme, but a little fancier with some syntax classes like :expr and such.

This is safe; the examples from the Lisp run as expected:

(my-or 1 2)                      ;=> 1
(my-or #f 42)                    ;=> 42
(let ((tmp 99)) (my-or #f tmp))  ;=> 99

However, we’ve lost some of the power because we can only define transformations between templates. We can’t, for example, write a macro that does some deep inspection of the code and makes decisions on how to expand. Furthermore, there’s no way for us to intentionally break hygiene when we really want to.

Racket macros: the best of both worlds #

Racket resolves the dilemma between having to choose between powerful Lisp-like procedural macros, and safe Scheme-like hygienic macros by giving us fully hygienic procedural macros! I have another blog post discussing macros in Lisp, Scheme, and Racket and I go into some detail about the evolution of those macro systems.

And if you want to dive deep into macro hygiene, see Matthew Butterick’s excellent explainer on Hygiene from his book Beautiful Racket.

The upshot of it is that Racket uses a combination of features (scope sets, syntax objects, etc.) to give the user a richer way of specifying syntax than simple dumb lists of symbols. This avoids inadvertent variable capture as well as keeps function references lined up nicely. However, macros can still do arbitrary computation, which means that we’re not constrained in the way that the pattern-transformation macros in Scheme are.

And just to prove that Racket is just as powerful as Common Lisp, here’s the classic aif macro:

#lang racket
(require racket/stxparam syntax/parse/define)

(define-syntax-parameter it
  (lambda (stx)
    (raise-syntax-error (syntax-e stx)
                        "can only be used inside aif")))

(define-syntax (aif stx)
  (syntax-parse stx
    [(_ test tcase fcase)
     #'(let ([tmp test])
         (if tmp
             (syntax-parameterize
                 ([it (make-rename-transformer #'tmp)])
               tcase)
             fcase))]))

(aif 41 (+ it 1) 'whatever) ;=> 42
it                          ;error: it: can only be used inside aif

This example is inspired by Greg Hendershott’s fabulous tutorial Fear of Macros. The define-syntax-parameter bit lets us introduce new bindings intentionally, whilst still keeping us from accidental breaches of macro hygiene.

Consequentially, Racket’s macro system is far more useful than Lisp or Scheme’s systems, and this because of Racket’s safety and expressiveness. You can actually build trustworthy systems on top of Racket’s macro system because you’re not constantly foot-gunning yourself with hygiene malfunctions, and the macros are expressive enough to do some rather complicated things.

Towards greater safety and reliability #

Safe systems let us build software that is more capable and more reliable. Unsafe power is something to improve, not grudgingly accept—and much less defend as somehow desirable. Languages like Rust and Zig have made systems programming immune to whole hosts of errors by being more expressive than C, and languages like Racket are leading the way in making metaprogramming more useful reliable and less like dark magic.

Rebecca purple

I Learnt — Wed, 20 Nov 2024 22:48:00 GMT

The community for the CSS-Next repository recently had a vote to decide on a new official logo. The colour chosen for the new logo was rebeccapurple. This colour was introduced to the CSS specification in 2014.

Video Editor with FFmpeg Commands

Welcome to my weblog on Newbeelearn — Wed, 20 Nov 2024 20:31:23 GMT

Easily edit videos, add logos, text and export customized FFmpeg commands for advanced video editing workflows.

Your keystroke biometrics are everywhere

All content on Seirdy’s Home — Wed, 20 Nov 2024 17:56:31 GMT

Real time collaboration software and text boxes that rapidly save drafts to the cloud essentially log your fingerprintable typing behavior. The industry refers to this information as “keystroke dynamics” or “typing biometrics”.

Other modern “operator signatures” are easier to minimize. A user can learn to obfuscate writing style, or can use keyboard navigation with different pointing devices to limit fingerprinting of mouse behavior. Update: switched link to Kloak to Whonix’ updated fork.

Keystroke biometrics are difficult to anonymize without installing software such as kloak or browser extensions (the latter of which may add fingerprintable vectors) designed to cloak some of your typing habits. Signature typos, approximate typing speed, etc. will still leak. Alternatively, we could normalize typing messages out in a simple offline editors that don’t store revision history before pasting them into other input fields.

Why Bluesky feels right

GOOD INTERNET — Wed, 20 Nov 2024 13:39:07 GMT

Bskyjuice Bskyjuice Bskyjuice.

DETAILS part 3, a image series

Mia Heidenstedt — Wed, 20 Nov 2024 11:17:10 GMT

Best viewed on the original page, where extended functionality like the footnote helper is available.

This is a street photography series captured during my travels and excursions. I am considering selling prints of these images, but none currently exist. If you would like a print version of my work, please contact me and we can discuss the details.

Street photography featuring scenes across Italy.

Total images: 31

You can use almost anything as a key file for your encrypted storage device

./techtipsy — Wed, 20 Nov 2024 06:00:00 GMT

Imagine that you have an unencrypted drive containing your private data and one day it starts throwing a bunch of errors. You have backups of the data so you’ve got that part covered, but would you feel comfortable sending the drive in to be warrantied? You have no control over who has access to that drive, and due to the drive failing you can’t format it as well.

Do you take the financial hit and buy a new drive, or send it in regardless and risk someone looking through your files?

I’ve bought and sold a bunch of hard drives and SSD-s over the years¹, and warrantied a few of them. I encrypt the disk on my personal and work machines because of the obvious security benefits, but for a long time I avoided doing the same for other storage devices.

Then I realized that I can just encrypt them and use almost anything as the key. All you need is a reasonably sized file to pass to cryptsetup as a key-file, refer to that key-file in /etc/crypttab, and you’re good to go!

Be creative! The key file can be anything:

a dank meme
a very short low resolution cat video
an adorable photo of your dog
a precious family photo
Twilight fan fiction that you wrote back in 2011
your letter of resignation at your last job

Just mind the size, cryptsetup wasn’t very happy with a 17MB full resolution image. A JPEG that’s less than a megabyte in size worked well enough for me.

On Linux, you can encrypt a partition with a cat picture using a command like this:

cryptsetup luksFormat /dev/sdx1 --key-file /path/to/cat.jpg

You can make sure that the drive gets automatically decrypted at boot by defining it in /etc/crypttab. Find the UUID of the encrypted partition by running ls -lah /dev/disk/by-uuid and see which one matches with /dev/sdx1. If the UUID is ef277bc2-d953-44c4-88af-8320aca76969, then a line in /etc/crypttab would look like this:

encryptedcatdrive UUID=ef277bc2-d953-44c4-88af-8320aca76969 /path/to/cat.jpg

Once unlocked, your decrypted partition will be available as /dev/mapper/encryptedcatdrive.

Just make sure that the key file isn’t placed on the encrypted drive itself, otherwise you’ll lock yourself out for good.

For more information on LUKS disk encryption and its capabilities, see this handy Arch Wiki page.

This approach comes with some great benefits and a few downsides.

On the bright side, you don’t have to worry about a stranger getting their hands on your data when selling a storage device or sending it in to be warrantied, assuming that you haven’t posted the key file on social media or anywhere else.² Just do a quick format of the drive, or format a very specific part to clear the LUKS headers, and the contents of the drive are gone!

If you can’t format the drive due to hardware issues, then you should still be safe, the attacker would have to first fix the drive (which isn’t a guaranteed success) and then figure out the key, which will require a lot of effort and time.

The one obvious downside is that if you lose the key file, you lose the data. You can mitigate this by either adding multiple key files as a backup option, or setting a strong password that you can use in case you lose your original key file.

If the drive contains something of value to your family members, then it may be a good idea to specify those details in a will, or by taping the backup password with instructions on the drive itself. If you end up selling the drive before your personal expiration date arrives, then you can simply remove the instructions. Just make sure that your family members or relatives know to plug in the drive to a Linux machine.³

I also like the thrill of hiding a key in plain sight. Did I use the cover photo of this post to encrypt my drives? You’ll never know, and that’s what makes it fun!

If you’re paranoid or targeted by state actors, then you probably shouldn’t follow this advice, except the part where I encourage encrypting stuff.

Hello, my name is Herman, and I’m a recovering data hoarder. ↩︎
if we use a simple .jpg as an example key file, then it’s highly likely that this won’t be an issue as most messaging and social media platforms compress the hell out of the image. The compression is also lossy, meaning that the process is irreversible and the end result is a completely different file from the perspective of the encrypted disk. ↩︎
if you’ve ever plugged in a drive with a Linux filesystem on a Windows machine, then you’ll know that it will happily recommend formatting it. Oh, goodie. ↩︎

Engineering AI Agents - OpenAI DevDay Singapore talk

swyx's site RSS Feed — Wed, 20 Nov 2024 04:06:58 GMT

this is the tracking doc for my talk on "Engineering AI Agents" for OpenAI DevDay Singapore. I'll add photos and notes and stuff when i'm done. note that the full slides contain a lot more info that i had to cut out of the ~~10~~ 9 minute talk.

chatgpt-shell goes multi-model

xenodium.com @alvaro — Wed, 20 Nov 2024 02:00:00 GMT

Over the last few months, I've been chipping at implementing chatgpt-shell's most requested and biggest feature: multi-model support. Today, I get to unveil the first two implementations: Anthropic's Claude and Google's Gemini.

Changing course

In the past, I envisioned a different path for multi-model support. By isolating shell logic into a new package (shell-maker), folks could use it as a building block to create new shells (adding support for their favourite LLM).

While each shell-maker-based shell currently shares a basic common experience, I did not foresee the minor differences affecting the general Emacs user experience. Learning the quirks of each new shell felt like unnecessary friction in developing muscle memory. I also became dependent on chatgpt-shell features, which I often missed when using other shells.

Along with slightly different shell experiences, we currently require multiple package installations (and setups). Depending on which camp you're on (batteries included vs fine-grained control) this may or may not be a downside.

With every new chatgpt-shell feature I showcased, I was often asked if they could be used with other LLM providers. I typically answered with "I've been meaning to work on this…" or "I heard you can do multi-model chatgpt-shell using a bridge like liteLLM". Neither of these where great answers, resulting in me just postponing the chunky work.

Eventually, I bit the bullet, changed course, and got to work on multi-model support. With my initial plan to spin multiple shells via shell-maker, chatgpt-shell's implementation didn't exactly lend itself to support multiple clients. Long story short, chatgpt-shell multi-model support required quite a bit of work. This where I divert to ask you to help make this project sustainable by sponsoring the work.

Make this project sustainable

Maintaining, experimenting, implementing feature requests, and supporting open-source packages takes work. Today, chatgpt-shell has over 20.5K downloads on MELPA and many untracked others elsewhere. If you're one of the happy users, consider sponsoring the project. If you see potential, help fuel development by sponsoring too.

Perhaps you enjoy some of the content I write about? Find my Emacs posts/tips useful?

Blog (xenodium.com) (Web)
Blog (lmno.lol/alvaro) (Web)

Alternatively, you want a blogging platform that skips the yucky side effects of the modern web?

I'm building lmno.lol (my blog is there).

Maybe you enjoy one of my other projects?

Plain Org (org mode / iOS)
Flat Habits (org mode / iOS)
Scratch (org mode / iOS)
macosrec (macOS)
Fresh Eyes (macOS)
dwim-shell-command (Emacs)
company-org-block (Emacs)
org-block-capf (Emacs)
ob-swiftui (Emacs)
chatgpt-shell (Emacs)
ready-player (Emacs)
sqlite-mode-extras
ob-chatgpt-shell (Emacs)
dall-e-shell (Emacs)
ob-dall-e-shell (Emacs)
shell-maker (Emacs)

So, umm… I'll just leave my GitHub sponsor page here.

chatgpt-shell, more than a shell

With chatgpt-shell being a comint shell, you can bring your favourite Emacs flows along.

As I used chatgpt-shell myself, I kept experimenting with different integrations and improvements. Read on for some of my favourites…

A shell hybrid

chatgpt-shell includes a compose buffer experience. This is my favourite and most frequently used mechanism to interact with LLMs.

For example, select a region and invoke M-x chatgpt-shell-prompt-compose (C-c C-e is my preferred binding), and an editable buffer automatically copies the region and enables crafting a more thorough query. When ready, submit with the familiar C-c C-c binding. The buffer automatically becomes read-only and enables single-character bindings.

Navigation: n/p (or TAB/shift-TAB)

Navigate through source blocks (including previous submissions in history). Source blocks are automatically selected.

Reply: r

Reply with with follow-up requests using the r binding.

Give me more: m

Want to ask for more of the same data? Press m to request more of it. This is handy to follow up on any kind of list (suggestion, candidates, results, etc).

Request entire snippets: e

LLM being lazy and returning partial code? Press e to request entire snippet.

Quick quick: q

I'm a big fan of quickly disposing of Emacs buffers with the q binding. chatgpt-shell compose buffers are no exception.

Confirm inline mods (via diffs)

Request inline modifications, with explicit confirmation before accepting.

Execute snippets (a la org babel)

Both the shell and the compose buffers enable users to execute source blocks via C-c C-c, leveraging org babel.

Vision experiments

I've been experimenting with image queries (currently ChatGPT only, please sponsor to help bring support for others).

Below is a handy integration to extract Japanese vocabulary. There's also a generic image descriptor available via M-x chatgpt-shell-describe-image that works on any Emacs image (via dired, image buffer, point on image, or selecting a desktop region).

Supporting new models

Your favourite model not yet supported? File a feature request. You also know how to fuel the project. Want to contribute new models? Reach out.

Local models

While the two new implementations rely on cloud APIs, local services are now possible. I've yet to use a local LLM, but please reach out, so we can make these happen too. Want to contribute?

Should chatgpt-shell rename?

With chatgpt-shell going multi-model, it's not unreasonable to ask if this package should be renamed. Maybe it should. But that's additional work we can likely postpone for the time being (and avoid pushing users to migrate). For now, I'd prefer focusing on polishing the multi-model experience and work on ironing out any issues. For that, I'll need your help.

Take Gemini and Claude for a spin

Multi-model support required chunky structural changes. While I've been using it myself, I'll need wider usage to uncover issues. Please take it for a spin and file bugs or give feedback. Or if you just want to ping me, I'd love to hear about your experience (Mastodon / Twitter / Reddit / Email).

Be sure to update to chatgpt-shell v2.0.1 and shell-maker v0.68.1 as a minimum.
Set chatgpt-shell-anthropic-key or chatgpt-shell-google-key.
Swap models with existing M-x chatgpt-shell-swap-model-version or set a default with (setq chatgpt-shell-model-version "claude-3-5-sonnet-20240620") or (setq chatgpt-shell-model-version "claude-gemini-1.5-pro-latest").
Everything else should just work 🤞😅

Happy Emacsing!

2024-11-20/01

ho.dges.online — Wed, 20 Nov 2024 02:00:00 GMT

2024-11-20/02

ho.dges.online — Wed, 20 Nov 2024 02:00:00 GMT

Bloom Filters and SQLite

Ben E. C. Boyter — Wed, 20 Nov 2024 02:00:00 GMT

I was talking to someone recently about how searchcode.com works with regards to its bloom filter. The question came up about why it stores the index in memory and not on disk considering how fast disks are these days.

Now the answer is firstly performance, but the second is that writing all the code to store the index on disk is a lot of work and I didn’t feel like working on it. However I was curious… since I have been using SQLite a lot recently, how about storing the bloom filter on disk using that? It comes with its own indexes for stepping though the filters and is likely out of the box to be better than anything I could come up with quickly.

Why The Singularity Is Impossible — Or Rather, Why It's Just a Bad Word In General

Chris' Full Stack Blog RSS Feed — Wed, 20 Nov 2024 02:00:00 GMT

Too many people not reading enough books and not understanding what they are arguing about.

Bill 212

Pxtl.ca — Wed, 20 Nov 2024 02:00:00 GMT

So in a fit of pettiness, the Government of Ontario is set to wrap up bike-lanes in a 50-foot-ball of red tape to prevent their construction. I asked on Bluesky what was the best way to complain - if there was a petition I should sign or something. Craig Burley recommended I send a fax of all things. It's got a good chance of being physical media in their space, and won't just be in a slushpile of emails they ignore, or forgettable names on a petition. So I wrote up the letters for the Premier and the Minister of Transportation at lunch on Tuesday and faxed them from my local UPS store. Also emailed them for good measure.

They're not my best writing but I was kind of in a rush.

The bodies of the letters are attached below.

Power to the power users

Alejandro AR (kinduff) — Tue, 19 Nov 2024 20:38:00 GMT

How the creative uses of data structures by power users are transforming and inspiring new features in my product

How to use Common Lisp on NixOS with C libraries

Honza Pokorný — Tue, 19 Nov 2024 20:00:00 GMT

NixOS is great. Well, until you open up a Common Lisp project, start a REPL, and well… you get an error about missing headers to some C library. We can make this work with a few steps.

NixOS doesn’t really let you install libraries globally and you have to install these dependencies manually for every project. Let’s do that now and see how we can make our REPL find them.

In the root of your project, create a shell.nix file with the following:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

mkShell {
  buildInputs = [
    pkg-config
    openssl
  ];
}

Then, enter into that shell with nix-shell.

Start a slynk REPL in the nix shell. You can use a script like:

(push #p"~/.emacs.d/.local/straight/repos/sly/slynk/" asdf:*central-registry*)
(asdf:load-system :slynk)
(slynk:create-server :port 4008)

Run it with sbcl --load <script>.

Once the REPL is running, you can connect to it from Emacs with M-x sly-connect. That’s it.

Power Laws & Why Our New Album Won’t Make Any Money

Quantable Analytics — Tue, 19 Nov 2024 18:32:58 GMT

That’s the way you do it You play the guitar on the MTV That ain’t workin’, that’s the way you do it Money for nothing and your chicks for free So said Dire Straits in 1985. When I was a kid, I legitimately thought that the lyrics were “money for nothing and your checks for free”. […]

The post Power Laws & Why Our New Album Won’t Make Any Money appeared first on Quantable Analytics.

Connecting to eduroam at Oxford University

extua — Tue, 19 Nov 2024 18:00:00 GMT

I’ve been setting myself up at Oxford University, going through the rotation of new accounts and passwords. So, I have to connect to eduroam with a new set of credentials.

Communication Tip: The Power of Named Chats

SeanKilleen.com — Tue, 19 Nov 2024 16:00:00 GMT

In many organizations, natural communication pathways start to form. Something happens or information needs to be sent out or a decision needs to be made, and you fire up a Teams window or Slack group message and invite the usual suspects.

HexUploader mirror

Gregor van Egdom — Tue, 19 Nov 2024 10:45:58 GMT

Since the creator’s site is unavailable and has been for some time, I’m hosting a mirror of a binary of the handy HexUploader. It was available through a third party. This is a small Mac app that allows uploading .hex … Continue reading →

Develop AI Cover letter generator app in Flask using Gemini API

Adnan's Random bytes — Tue, 19 Nov 2024 09:49:04 GMT

This post is part of the GenAI Series. In the previous post, I discussed creating a recipe generator AI app in PHP using OpenAI APIs. In this post, I will explain how to use Google’s Gemini LLM APIs to create another text-based AI app: an AI cover letter generator. Let’s dive in! If you are in a hurry, you may check the demo video below: I introduced you to Google’s LLM API recently. At that time, it was called Bard, which was later rebranded to Gemini. I automated a WordPress blog with Bard APIs to create financial posts. API Key Generation To use Gemini APIs you need an API Key, for that, you will have to visit AI Studio. Once the API Key is generated it will be available for you: Setting up Python SDK Although you can use Gemini’s REST APIs with the help of requests library but I will be using the official Python SDK provided by Google. Run the following pip command: pip install -U google-generativeai Setting up Python Flask Since the app is a web app, I am going to use Python Flask for this purpose. If it is not installed you may install it by running the following command in the terminal: pip install --upgrade flask Development Once the required libraries are installed, it’s time to start working on the app. There are three major parts of any AI generator app: Prompt: The instructions that will be passed to an LLM to produce the required output. You can usually test it out in an LLM chat interface (ChatGPT, Gemini, Claude) without using any API. User Interface: This could be either web, mobile, or even CLI. LLM API Integration: Integration of APIs so that you can use the prompt’s output in your app. Prompt Engineering So, we are going to perform prompt engineering: writing and optimizing a prompt to suit our needs. Head over to Gemini and test the following prompt: You are a skilled freelancer with expertise in writing cover letters that have successfully helped you secure many jobs. Your task is to craft a compelling cover letter that persuades the client to contact you. It should utilize persuasive techniques and convey that the freelancer is focused on delivering results and helping achieve goals, rather than solely pursuing monetary gain. You will be provided with information in JSON format, which will serve as the foundation for constructing the cover letter “` { “Name”: “Adnan Siddiqi”, “JobTitle”: “AI Engineer”, “PrimarySkills”: [“OpenAI”, “Prompt Engineering”, “LLMs”, “Python”], “Experience”: 7, “ClientJobPost”: “Needs to hire 2 Freelancers\nI’m looking for a tech lead to help us with the technical aspects of our AI and full-stack projects.\n\nThe work would involve\n- Creating tech architecture, designs and plans\n- Working with the dev team and unblocking them on the tech aspect\n- Preparing project estimates\n- Client meetings and communication\n- Daily team huddles\n\nYou are someone\n- You are a full time freelancer (no full time job or agency at this time)\n- With 8-10+ years of tech experience\n- Hands-on if required on Python, Django, Fast, React,\n- Solution Architect (AWS, Google, Azure) – one cloud is fine\n- Have knowledge of AI, GPT, LLM and Agents – this is a bonus\n- Write \”tech lead\” on top so I know you have read till here\n\nWe are starting on new projects and our existing tech lead is busy with some of our other projects.”, “RelevantProjects”: [“AI Recipe Generator”], “ClientName”: “Acme Inc”, “StartDate”: “2024-11-21”, “Deadline”: “2024-12-28”, “Tone”: “conversational” } “` The output must be in JSON format contain the coverletter text in “text” field. I have used this UpWork job post as an example for testing the prompt. When you run the prompt it prints: { "text": "Hi Acme Inc,\n\nI'm Adnan Siddiqi, an AI Engineer with 7 years of experience. I'm writing to express my keen interest in the tech lead position you've posted. I'm a full-time freelancer, ready to dedicate my skills and expertise to your AI and full-stack projects.\n\nI've been closely following the advancements in AI, particularly in the areas of OpenAI, Prompt Engineering, and LLMs. My experience in Python and my hands-on approach to problem-solving make me confident in my ability to deliver exceptional results. \n\nI've successfully led teams in the past and have a proven track record of creating innovative solutions. I'm excited about the opportunity to contribute to your projects, from crafting robust tech architectures to collaborating seamlessly with your dev team. I'm also proficient in project estimation and client communication, ensuring that projects are delivered on time and within budget.\n\nI'm particularly interested in the AI Recipe Generator project. I believe my skills in AI and Python can be leveraged to create a truly innovative and user-friendly solution. \n\nI'm eager to discuss how my skills and experience can benefit your team. Let's connect to explore this opportunity further.\n\nThanks,\nAdnan Siddiqi" } Looks good! At the UI end, we will be parsing the JSON extracting the data from the text field and displaying it. UI Design This is a simple bootstrap CSS-based web form that accepts different inputs. Below is the Flask code for the relevant route: @app.route('/submit', methods=['POST']) def cover_letter_form(): if request.method == 'POST': COVER_LETTER_PROMPT = """ You are a skilled freelancer with expertise in writing cover letters that have successfully helped you secure many jobs. Your task is to craft a compelling cover letter that persuades the client to contact you. It should utilize persuasive techniques and convey that the freelancer is focused on delivering results and helping achieve goals, rather than solely pursuing monetary gain. You will be provided with information in JSON format, which will serve as the foundation for constructing the cover letter: {{ "Name": "{name}", "JobTitle": "{jobTitle}", "PrimarySkills": {primarySkills}, "Experience": {experience}, "ClientJobPost": "{clientJobPost}", "RelevantProjects": {relevantProjects}, "ClientName": "{clientName}", "StartDate": "{startDate}", "Deadline": "{deadline}", "Tone": "{tone}" }} """ data = request.get_json() # Format the prompt with the data formatted_prompt = COVER_LETTER_PROMPT.format( name=data['name'], jobTitle=data['jobTitle'], primarySkills=str(data['primarySkills']), # Convert list to string experience=data['experience'], clientJobPost=data['clientJobPost'].replace("\n", "\\n"), # Escape newlines relevantProjects=str(data['relevantProjects']), # Convert list to string clientName=data['clientName'], startDate=data['startDate'], deadline=data['deadline'], tone=data['tone'] ) generation_config = { "response_schema": content.Schema( type=content.Type.OBJECT, properties={ "answer": content.Schema( type=content.Type.STRING, ), }, ), "response_mime_type": "application/json", } genai.configure(api_key=os.environ["GEMINI_KEY"]) model = genai.GenerativeModel( 'gemini-1.5-flash', generation_config=generation_config ) response = model.generate_content(formatted_prompt) # Example: Print the received data (replace with processing logic) print(f"Name: {data.get('name')}") print(f"Job Title: {data.get('jobTitle')}") print(f"Primary Skills: {data.get('primarySkills')}") print(f"Experience: {data.get('experience')}") print(f"Client Job Post: {data.get('clientJobPost')}") print(f"Relevant Projects: {data.get('relevantProjects')}") print(f"Client Name: {data.get('clientName')}") print(f"Start Date: {data.get('startDate')}") print(f"Deadline: {data.get('deadline')}") print(f"Tone: {data.get('tone')}") return jsonify(response.text) Calling the generate_content method with the required parameters it returns the JSON payload. In the /submit route it is returning JSON data that is later called via AJAX $.ajax({ url: '/submit', // Adjust URL as needed type: 'POST', contentType: 'application/json', data: JSON.stringify(formData), success: function(response) { const data = JSON.parse(response) let answer = data.answer answer = answer.replace(/\n/g, '<br>') $('#responseOutput').html(answer) $('#responseContainer').show(); $("#wait").hide() }, error: function(xhr, status, error) { $('#responseOutput').text(`Error: ${xhr.responseText}`); $("#wait").hide() $('#responseContainer').show(); } }); Simple yet amazing, No? Conclusion In this post, we explored how you can integrate Google Gemini LLM APIs with your app to add intelligence to it. As always, the code is available on GitHub. Looking to create something similar or even more exciting? Schedule a meeting or email me at kadnan @ gmail.com. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Develop AI Cover letter generator app in Flask using Gemini API first appeared on Adnan's Random bytes.

Handling tarballs in Python

Heitor's log — Tue, 19 Nov 2024 02:00:00 GMT

Need to work with tar files in Python? Here's how to create and extract compressed tarballs effortlessly.

Data Loading - Daft

Londogard Blog — Tue, 19 Nov 2024 02:00:00 GMT

I know I’ve been praising polars a lot lately, and I’m still in love. polars will be my continued go-to library for Data Analysis of Tabular data, and when building ETL (data pipelines) in 99% of the cases.

However, when you work with Deep Learning and multi-modal data you need something to take the data from your Delta Lake, or wherever you store your data, and supply it to the model. That’s where tools like daft can shine.

These Data Loading and Processing steps need to be highly optimized to utilize the underlying compute optimally, not wasting $$ on unused GPU’s. The jobs should keep a high % utilization and not be bound by I/O or CPU. In other words: you want to always have data ready when the GPU has time to process more data.

There’s a lot of tools to do the job, I’ll go through a few of them in upcoming blogs with a simple yet common workload: Image Classification. Image Classification is simple but the data can quickly grow large to not fit in-memory anymore.

Can’t fit in-memory => I/O needs to be optimized
Expensive transforms & augmentations => CPU needs to be optimized

An even better example would be Object Detection as it has ragged (variable) dimensions, i.e. one image has 2 people and another has 1, but it’s a bit more complex data processing. I’ll include this in my next blog where I give “recipes” on how to use Daft.

Finally, this blog will be quite brief and not as in-depth as I hoped, but there’ll be more blogs coming later!

Daft

Today I’ll introduce one of the newer alternatives in the field, daft.

Daft is what you can only call a merger between polars, spark and Deep Learning. If they had been more inspired by polars in the Developer Experience (DX) I’d have called it a “lovechild”, but for now they don’t have the nice-to-haves like pl.with_column(new_col_name=pl.col("other_col")*2) named syntax and other things like pl.col("col").replace(dict_to_replace) and a lot of other things.

What daft does have is a multi-modal namespace, unlike polars which solely focuses on traditional data-types. This is really interesting albeit not that fleshed out yet. It’s enjoyable and has potential to grow!

Further, to quote daft themselves:

Daft provides a snappy and delightful local interactive experience, but also seamlessly scales to petabyte-scale distributed workloads.

The petabyte-scale comes from the fact that you can run daft on top of Ray which is a distributed framework that tries to take on Spark. It’s famously used at OpenAI while training their models.

Coding with Daft

Coding with daft is an experience. I only ran locally but it held up really well to “native” PyTorch, even surpassing it in one case!

I’ll share my experience and implementations below!

Reading Data

Like most modern projects daft includes a smooth integration to Apache Arrow.

Apache Arrow is “The universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics”

The Arrow integration gives daft multiple ways to read a dataset, and the dataset doesn’t even have to be in-memory because of the Arrow data structure which can easily be streamed via “memory-map-mode” (mmap).

To “read” an Arrow table you simply call from_arrow, as I do below reading a HuggingFace Datasets Arrow Table.

ds_train = daft.from_arrow(ds["train"].data.table)

To “read” other formats from disk you simply use read_(delta|csv|...), as below.

df = daft.read_deltalake("some-table-uri") # read_(csv|parquet|json|...)

Finally it has very tight integration with Ray, which is very neat when you wish to scale to Big Data.

Data Transforms - multi-modal and whatnot

To modify a DataFrame you work very similar to polars. There’s Expression’s which is a way to have a lazy non-evaluated expression, like a SQL query before you run it. I’ve spoken about Expression’s before and I really love them, they make code decoupling a lot easier and can simplify a query to something beautiful.

See my example of extracting image from a struct that has a field with bytes.

# expression: lazy non-executed method
extract_img_bytes = daft.col("image").struct.get("bytes").alias("image")

ds_train.select("label", extract_img_bytes)

Select column label and image, where image extracts image.bytes into image.

From here I’d like to decode the image into something which we can work with, unlike bytes, and that’s easy using the multi-modal namespace (.image).

img_decode_resize = daft.col("image").image.decode(mode="RGB").image.resize(224, 224)

ds_train = ds_train.with_column("image", img_decode_resize)

Transforms image by decoding it into RGB and then resizing to 224x224.

Quite cool right? There’s some great potential here!

How do we apply more complex operations? UDF’s! It’s just as easy as in polars, simply call apply.

def rescale_transpose(x: np.array):
    return (x / 255.0).transpose(2, 0, 1)

ds_train.with_column(
    "image",
    daft.col("image").apply(
        rescale_transpose,
        return_dtype=daft.DataType.tensor(daft.DataType.float32()),
    ),
)

Applying a custom transformation. Images are represented as np.array and you need to define return_dtype.

With all this available we’re good to go for a Deep Learning training pipeline!

Producing a PyTorch Dataset

The final part of our pipeline is to move the data into torch.Tensor. There’s one big gotcha - don’t apply num_workers as daft already applies multi-thread/processing optimizations!

ds_train = ds_train.to_torch_iter_dataset()

# NOTE: don't apply num_workers even though PyTorch warns!
dls_train = torch.utils.data.DataLoader(ds_train, batch_size=32)

And that’s a wrap! We got all the steps to finalize the deal. How about a comparison?

Mini Benchmark

Comparing speeds with “native” PyTorch DataLoaders is interesting and shows that Daft is on-par in speed when using their new native execution engine (swordfish). When I increase image size, i.e. larger data to process, I see Daft even surpassing PyTorch DataLoaders (!).

N.B. I’m running the full training from a HuggingFace Dataset backed by Arrow. It’s the same underlying data structure for all tests except “Folder File” one, but things might just be different if we start discussing file-loading (rather than from bytes) or even remote data.

Numbers

Tool	Num_worker	Pin_memory	Cache	Configuration	Time
Torch Dataset/Loader	None	None	-	Default	3m20s
	None	None	-	Default	3m26s
	4	True	-	Default	4m9s
	2	True	-	Default	3m44s
Daft	-	-	-	daft-default	14m55s
	-	-	-	daft-native	3m30s

Running on full sized images we get a bit more interesting results:

Tool	Num_worker	Pin_memory	Cache	Configuration	Time
Full Size	4	True	-	torch	4m19s
Full Size	-	-	-	daft	3m49s
Image Folder & Files (160p)	-	-	-	torch	3m31s
Image Folder & Files (160p)	-	-	-	daft	3m26s

To read a file locally using daft you simply do the same as you’d do with remote.

df.with_column("image", daft.col("path").url.download())

Remote data

Working with remote data is a common and interesting use-case. I think based on this research that daft has a good chance of performing really well, as the local files also did great.

Final Thoughts

Even if daft has a way to go for Deep Learning training it really holds great promise. If they make the export easier to PyTorch and perhaps add TensorFlow I believe it could grow into a valuable competitor to HuggingFace Datasets et. al.

As Ray is what drives OpenAI’s training I believe Daft stands on some really good scalable underlying tech and can perhaps be what joins Data Engineering and Data Science together as one, for real - a big leap forward!

Thanks for this time, Hampus

Extra: all code is available on the git-repo for this blog, see code/data_loading.

Embeddings Clustering

Thought Eddies — Tue, 19 Nov 2024 02:00:00 GMT

Embeddings Clustering

Stay in the game

I Learnt — Mon, 18 Nov 2024 19:19:00 GMT

In a recent interview Mo Salah shared some advice he received from the ex-Arsenal manager Arsene Wenger that has stuck with him.

Inside-out templates

Honza Pokorný — Mon, 18 Nov 2024 15:00:00 GMT

I have been thinking about literate programming and markup languages for authoring documents. How can we write a document once and easily publish it in different formats? Markdown is user-friendly but bare-bones. XML does what we want but no one wants to write XML. I was reminded of Pollen. It promises to do what I want. Plus it’s a cool Lisp thing. But Lisp isn’t everyone’s thing. Then I had a flash of inspiration: what if we could abuse some Go templates?

Instead of using Go templates as templates, we can use them as an authoring language. Markdown on steroids.

Let’s imagine a document, say chapter.txt:

{{ title "Introduction" }}

Before we can start thinking about recursion we must first think about recursion.

The Wikipedia article on the topic {{ link "says" "https://en.wikipedia.org/wiki/Recursion" }} that...

It’s not as terse as Markdown but way nicer than XML.

How do we process it?

contentsBytes, err := os.ReadFile("chapter.txt")
if err != nil {
    return err
}

contents := string(contentsBytes)

t, err := template.New("chapter").Funcs(funcs).Parse(contents)
if err != nil {
    return err
}

t.Execute(os.Stdout, nil)

Pretty standard Go stuff. Read the file, create a new template, and print the result to standard output. But what is this funcs? You can define custom functions that will be available in your template (our document). This is where we can define our title and link functions. Let’s say we want to publish our chapter as HTML:

funcs := template.FuncMap{
    "title": func(text string) string {
        return fmt.Sprintf("<h1>%s</h1>", text)
    },
    "link": func(label string, url string) string {
        return fmt.Sprintf(`<a href="%s">%s</a>`, url, label)
    },
}

Running it will produce:

<h1>Introduction</h1>

Before we can start thinking about recursion we must first think about recursion.

The Wikipedia article on the topic <a href="https://en.wikipedia.org/wiki/Recursion">says</a> that...

It’s much nicer than using some custom string search and replace because the Go standard library handles all of the replacing for you: they have done the hard work of finding the edge-cases for you and handling them.

OK, onward: what about Markdown? Let’s make this a bit harder and use footnote-style links at the bottom of the page:

links := []string{}
linkCounter := 0
funcs = template.FuncMap{
	"title": func(text string) string {
		return fmt.Sprintf("%s\n%s", text, strings.Repeat("=", len(text)))
	},
	"link": func(label string, url string) string {
		id := linkCounter
		linkCounter++
		links = append(links, fmt.Sprintf("[%d]: %s", id, url))
		return fmt.Sprintf("[%s][%d]", label, id)
	},
}

The links will be collected in a slice and can be appended in a convenient place.

Introduction
============

Before we can start thinking about recursion we must first think about recursion.

The Wikipedia article on the topic [says][0] that...

[0]: https://en.wikipedia.org/wiki/Recursion

You can extend this in a million different ways. Need academic references? Hook this up to bibtex or Zotero. Images? Gnuplot graphs that are generated on the fly but different for webpages and ebooks? Syntax highlighting for code snippets?

Honza, did you just invent literate programming?!

Choose Boring Technology

Prasham H Trivedi — Mon, 18 Nov 2024 09:23:01 GMT

Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet. Clearly this model is approximate, but I think it helps. You can replenish the tokens if you can to hire more people who brings new innovation tokens with them.

Typst Business Card Templates

Brian Sunter — Mon, 18 Nov 2024 06:22:08 GMT

A collection of professionally-designed business card templates built with Typst, featuring multiple styles from minimalist to luxury.

Optimizing a go package by 38147125738.8x

xnacly - blog — Mon, 18 Nov 2024 02:00:00 GMT

Investigating performance issues in a go package for open port lookup

Just Do Something Already

Ben's Web Space — Mon, 18 Nov 2024 02:00:00 GMT

This is a repost of an article I originally published on Medium in November of 2020. I'm moving it here to consolidate all my writing in one place.

Have you had an idea? A deep seated conviction that things could be better? That you could bring about that change for the better? But, on the other hand, has your idea been with you for years with nothing to show for it? Have you seen others bring about their ideas? Maybe you have even tried a few times to do the same but haven’t or couldn’t? Then I’m writing this for you.

This is the story of an idea (Gradely, use bali5more as a key to test). It is not a success story in which I tell you how you should do things just the way I did them and that success is “guaranteed.” It is not a failure story either in which I reflect on all the things I did wrong and what I learned from the experience. It is the story of the ongoing struggle of being stuck, and now finally doing something about the idea!

You see, I believe that many of us have thought about doing something, but are paralyzed by a myriad of excuses and factors, thinking that we don’t have what it takes and just conforming to do nothing. And I want to convince you otherwise!

I have always thought that one man of tolerable abilities may work great changes, and accomplish great affairs among mankind… -Benjamin Franklin

The entrepreneurship bug

I don’t know the exact circumstances that lead a person to believe that they can bring about change, that li’l ol’ me can do something of value, but I can share how it happened to me.

In 2012, I started a masters degree in leadership/philosophy and got a work-study position for the department of distant education of my school. I had never worked before in tech but ever since I can remember I have been involved with it. From breaking my mom’s irons to see what was in there (…I didn’t know about screwdrivers back then!), to building a crystal radio and experiencing the magic of “the waves,” to going down the Linux train, to electronics and more. I got my hands on whatever books I could and tried all kinds of tech things even from my childhood.

Thus, I had no “official” qualifications for that job except what I had learned and the crazy notion that if I came across something I didn’t know, I could just learn it. Nowadays I just say that everything is figureoutable. But I digress.

It was at that job that I heard how much the university was paying for a service. I was appalled! The software that they were paying for was open-source/free and that meant that they were pretty much just paying for servers! To my naive mind it sounded like I could provide those services, charge them half what they were paying and with 4–5 clients I would be set for life! And just like that I was bitten by the entrepreneurship bug and have been affected ever since.

From there the idea narrowed down to a general “I want to do something with technology for education” to “I want to build a gradebook that doesn’t suck!” But it would be years before anything happened.

And then… what?

And then… I did nothing. Well, not nothing nothing. I Googled about how to start a business… and was inundated by the overwhelming amount of information that I found. It wasn’t just the sheer volume but how all-over-the-place it was. There were stories of successful people that had found success doing the complete opposite things! I heard that they way to go was with sponsors (and spent $500 on a course to learn more about it). I heard that the way to go was with an accelerator and VCs. I heard that they way to go was being bootstrapped. I heard that B2B was king, that B2C has more volume and market… I read and read and read, maybe 30+ books, listened to a bunch of podcasts, read blog posts, HN stories, etc. etc.

But one thing was true during all of this: I never felt ready. It seemed to me that the people doing these things, starting organizations and business, bringing about change, had it all together. I thought these people just knew what to do and did it.

And here I was with an idea that I had tried to bring to fruition multiple times, that I had talked over with many educators and received good feedback, and yet… nothing. Here I was with “tolerable abilities” to build said idea and still I had done nothing, still felt unprepared, still felt unworthy if you may. How dare I think I could bring about change?! How dare I think people would listen to me?! And for 8 years I was stuck in that quicksand.

And then it hit me. I could spend the next eight years in that quick sand, with the dreaded “what if” question festering in my mind, or I could just do something already! And it was that question that revealed the culprit that had been holding me back. It was not a lack of technical skills, or the lack of some intrinsic personality trait, or some genetically bestowed advantage. No, the culprit was fear, plain raw simple old fear. Fear that whatever I do won’t matter. Fear that the people I wanted to serve would reject me. Fear that all I could do was spend another eight years doing what I had done - nothing - and in the end accomplish nothing. Fear fear fear…

Oh, such an old enemy, but how well it still fools us! How well it masquerades itself and blames us for its deficiencies!

When I finally realized all the fear I had and started looking around and listening, I was surprised by what I heard. Everybody else was also afraid!! My boss was afraid, the CEO of the company was afraid, the entrepreneur that I so admired was also afraid!

But you know what? It is those that tell their fears to shut up who are the ones that bring about change! The ones who take their ideas and make them a reality and make the world, even if only a little, a better world!

And here you are reading my feeble attempt to tell my feats to shut up. My attempt at believing that li’l ol’ me can bring about a change, can make a difference in the lives of those I want to serve. That I can make the world, even if only a little, a better world.

And it is my strong belief that you can too! I don’t know your circumstances, your fears, your challenges, but I do believe that you can overcome them as those before you have done. I believe in you and I want you to believe in you, too!

And then there’s this quote by Teddy Roosevelt…

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.”

Out there

So this is the beginning of that journey, of “just doing something about it!”, my attempt to put myself out there. What will come of it is still to be seen, but I can tell you this: “out there” the sun is shining! “Out there” the possibilities are endless! “Out there” is still full of dangers and risks, but you can overcome them! “Out there” the world is waiting for you!

Native dual-range input

Muffin Man — Mon, 18 Nov 2024 02:00:00 GMT

I just released @stanko/dual-range-input - a native dual-range input. Here is how it looks with the default styles:

The "native" part is somewhat open for discussion. I call it native because the library uses two native HTML range inputs. This means that all of the native interactions and accessibility features are preserved.

Native inputs allow us not to reinvent the wheel. There is about fifty lines of JavaScript to synchronize the inputs, along with some CSS to ensure everything looks polished.

In my book, that is native enough.

Why #

When I create my generative drawings, I use a tool I built myself. This tool includes a UI for tweaking parameters, and I often have minimum and maximum sliders for certain parameters. I thought it would be nice to have a dual-range slider for these. However, most existing solutions rely heavily on JavaScript and reimplement dragging and accessibility features.

So, I set my own set of requirements:

It should use native HTML range inputs.
When you click on the track, the closer of the two thumbs should jump to that value.

Hopefully, after you read these two requirements, my solution will make sense.

How it works #

There are two inputs placed next to each other. When either of the inputs is changed, the library calculates a midpoint between the two selected values. Then the min and max attributes are set to the midpoint, and the width of both inputs is updated to match.

Here is an unstyled example, which will hopefully illustrate this well:

Even like this, it works reasonably well. We'll style it later to make it look nicer.

Resizing the inputs #

There's a small trick involved in calculating input widths. This is because the range input's track is actually shorter than the input's total width. All browsers leave enough space on the sides so the thumb doesn't stick out.

Here is a screenshot from Firefox (other browsers work similarly), where you can see that the track is shorter than the width. I've emphasized the space the browser leaves for the thumb.

If we take an example where the inputs need to be in a 1:3 ratio, simply setting their widths to 25% and 75% isn't enough. We also need to account for the thumb width. Instead of calculating the exact ratio, I simplified the math by adding the thumb width to each input's width:

input:first-child {
 width: calc(25% + var(--dri-thumb-width));
}

input:last-child {
 width: calc(75% + var(--dri-thumb-width));
}

If you thought, Wait, this adds up to more than 100%, you'd be 100% right. That's why I applied a small trick: I added padding to the inputs' wrapper to accommodate the extra width for the thumbs.

This makes the math simpler while keeping the input sizing correct. It took me forever to explain this properly, and I'm still not sure if I succeeded.

Move the thumb closer to the click #

Because the inputs are resized to meet at the midpoint, whenever you click between the thumbs, the one closer to the click will move to that value.

and the midpoint will be easier to see.

If there's an odd number of steps between the thumbs, the last-used input is favored. Try it out with debug mode on, and you'll see what I mean.

With that, both requirements are satisfied. The only thing left is to style it properly.

Styling #

All browsers allow us to style range inputs using CSS. That made styling of the tracks and thumbs pretty straightforward. I just ensured that the tracks didn't have a border radius in the middle where they connect.

Theming #

I exposed several variables to make theming easier. Here's the complete list with their default values:

.dual-range-input {
  --dri-height: 1.5rem;

  --dri-thumb-width: 1.25rem;
  --dri-thumb-height: 1.25rem;

  --dri-thumb-color: #ddd;
  --dri-thumb-hover-color: #a8d5ff;
  --dri-thumb-active-color: #4eaaff;
  --dri-thumb-border-color: rgba(0, 0, 0, 0.1);
  --dri-thumb-border-radius: 1rem;

  --dri-track-height: 0.25rem;
  --dri-track-color: #ccc;
  --dri-track-filled-color: #0084ff;
  --dri-track-border-radius: 1rem;
}

To create your own theme, simply override these variables.

Gradients #

One thing I thought was cool is how I used CSS gradients to paint the selected range in both inputs. I set the gradients to use the --dri-gradient-position variable, then updated that variable in the code along with the widths.

Here's how the CSS looks for one of the inputs:

input:first-child::-moz-range-track {
  background-image: linear-gradient(
    to right,
    var(--dri-track-color) var(--dri-gradient-position),
    var(--dri-track-filled-color) var(--dri-gradient-position)
  );
}

Again, and the semi-transparent thumbs will make the gradients easier to see.

Conclusion #

I had to write this post as ~~a brain dump~~ a way to consolidate my thoughts, so I hope it's not too convoluted.

Thank you for following along, and I hope this inspires you to try it out and consider using more native elements before opting for custom libraries.

Using Talon in a game jam

ntietz.com blog - technically a blog — Mon, 18 Nov 2024 02:00:00 GMT

I use Talon to control my computer some of the time. It's mostly voice control, but it has so many other controls built in! One lets you use an eye tracker as a mouse. I thought this sounded like a neat interaction for other situations too. When I mentioned this to a friend, he suggested building a game concept around controlling the game with the eye tracker.

First, though, I had to do some setup for us: get the eye tracker connected to a Rust game library. Ultimately, we wound up using pure mouse control—then you don't need an eye tracker, and if you have one, you can just use Talon to control it—but along the way there were some interesting shenanigans with threads!

How do I use an eye tracker?

It was really tough to figure out how to use my eye tracker, because I'm on Linux! The SDKs available mostly don't support Linux, or they're C++ and gnarly and I didn't have time to figure out how to do calibration. I do have Talon setup for eye tracking, so I thought it could be a neat way to interact with things.

Talon seems to call into another library, but it wasn't obvious what it was. I was hoping it would be obvious and I could go use that library, too, but alas. That's where the investigation ended for me. I don't think it's a great idea for me to try to poke any deeper, since this is commercial software that's intended to be closed, and we should honor the intentions of the author.

Instead, I tried using a sidecar for Talon to use the eye tracker! This was an interesting diversion, though ultimately one that we shouldn't really indulge in: it's probably against Talon's license, and it's certainly against the spirit of both Talon and the eye trackers' licenses.

That said, adding modules to Talon is a really valuable thing I tried doing for this! I've abandoned that approach here, but it was useful to explore in a throwaway project so I can use it when I want to actually expand my tooling.

Talon user modules

One of the beautiful things about Talon is that it's extremely configurable and customizable. Out of the box, it is quite bare, which is why using the community command set is highly recommended to start. In many ways, it's more a workshop for making your own tools, than it is a tool itself. This gives us a lot of room to build tools within Talon!

The first thing I did is created a folder in my talon user directory (for me, at ~/.talon/user). Then if you add Python code inside that folder, Talon loads it! This is really neat for building useful accessibility tooling.

Talon hot reloads plugins, which makes it very easy to iterate on this code. When I was exploring, my code created a thread. This thread was problematic, because it wouldn't be stopped when the module was reloaded, so if it held a resource (for example, write access on a file), you'd end up with conflicts. So I needed a way to stop the thread when we reload it!

Stopping an old thread

To stop our thread, we really ultimately needed a way to access the previous thread after we reload. We can't do that by just keeping it in a global in the module, since those are new on each reload.

I found there's a neat way to do that. We name the thread when we make it! Then we can find it by its name later on.

So I did something like this:

t = threading.Thread(target=myfunction)
t.name = "my-thread"
t.data = None

And then inside the function, we can assign to our data:

def myfunction():
    t = threading.current_thread()
    t.data = # some object we're using

Now the next time we reload the module, we can do some cleanup at the top!

for t in threading.enumerate():
    print(t, t.name)
    if t.name == "my-thread":
        t.data.shutdown()
        t.join()

And there we have it, we've stopped the previous thread.

Sneak peek

I got it integrated into a Rust program on the read side. Ultimately, we did this by just... using everything as intended, and using the mouse as input. Since Talon gives us our eye tracker as a mouse, this works for folks with that setup—and it also works if you don't!

It works, and I can render a red dot wherever I look! There's some jitter, but it does follow my eyes pretty accurately, and works best in full screen. Trying to screen record when you're using an eye tracker as your mouse, or debugging it, is pretty funny because you can't look at the controls to trigger things or stop them. And good luck clicking on the screen recorder.

Hopefully there'll be more updates on this project after we do our one-day game jam. And hopefully eventually we'll have a demo that folks can try out, with Talon or a mouse!

Host an indie-web yardsale on your /junk page

taylor.town — Mon, 18 Nov 2024 02:00:00 GMT

You can give that wish a second life.

Preserve Child Objects When Parent Objects Are Deleted in Ruby on Rails

Nelson Figueroa — Mon, 18 Nov 2024 02:00:00 GMT

Use dependent: :nullify in Rails associations to preserve child records when parent objects are deleted.

A Poem Of Grief

James Brooks — Mon, 18 Nov 2024 02:00:00 GMT

A poem I wrote shortly after the passing of my brother and the birth of my first child.

Running an LLM locally using Ollama

Jason Eckert's Website and Blog — Mon, 18 Nov 2024 02:00:00 GMT

If you are developing or integrating software with a Large Language Model (LLM) to add chatbot or other AI functionality, then you may have the need to host and run an LLM on your local workstation for development or testing. There are several open source LLMs available that you can freely run on your system provided that you have enough resources to do so.

LLMs are just inert objects that require special software to allow people and other software to interact with it (called running the model). Unfortunately, setting up the software components to run a model is akin to nailing Jello to a tree.

How to join the Wikipedia Discord

River Writes - A MediaWiki Blog — Mon, 18 Nov 2024 00:49:19 GMT

Click here if you want to join the Wikipedia Discord server!

Memory Eternal, Nikolas Doucet

Fortress of Doors — Sun, 17 Nov 2024 23:45:26 GMT

Nikolas Andreas Doucet, age 8, died painlessly of sudden cardiac arrest on Friday, November 8th, 2024, at 7:03 PM Central time. He was buried on Friday November 16, 2024. The following was the eulogy that I, his father, gave at his graveside.

Better dmesg in five minutes

The Technical Blog of James on purpleidea.com — Sun, 17 Nov 2024 22:52:00 GMT

I last wrote about dmesg in 2016. It has mostly not changed since then, but I’ve changed my setup slightly. Here’s a short article about what I did so that you can do it too, and so that I can remember for the next time.

Previously:

Previous I had a bash alias which looked like this:

alias dmesg='dmesg --follow || dmesg'

since some machines didn’t yet support the --follow flag.

Constraints in Go

Mia Heidenstedt — Sun, 17 Nov 2024 22:44:42 GMT

Best viewed on the original page, where extended functionality like the footnote helper is available.

I want to share this great article about generic coding in Go: https://bitfieldconsulting.com/posts/constraints

Also Go Proverbs is quite nice!

Ohh and this Blog has now footnotes and popups for footnotes! ¹

Summary (Generated):

In this tutorial, John Arundel explores constraints in Go, the paradoxical feature of generics that simultaneously limits and expands possibilities. By delving into method sets, type elements, unions, intersections, and approximations, he demonstrates how constraints enable precise operations on type parameters. Whether you’re learning to leverage fmt.Stringer, creating flexible numeric constraints, or handling derived types, this guide unlocks a deeper understanding of Go’s generics—perfect for honing your skills or tackling practical challenges in coding.

This is a footnote!
And this works even with multiple lines.
And code too!
```
console.log('Hello, World!');
```
↩︎

Mirror: An LLM-powered programming-by-example programming language

Austin Z. Henley's Blog — Sun, 17 Nov 2024 20:00:01 GMT

https://austinhenley.com/blog/mirrorlang.html

How to setup self hosted wiki for your startup

The Mythical Engineer — Sun, 17 Nov 2024 07:22:00 GMT

Too Stupid to Understand Git

Studiofreya SSG Site — Sun, 17 Nov 2024 02:00:00 GMT

From: https://www.reddit.com/r/gamedev/comments/1gt7q7j/too_stupid_to_understand_git/

Am I too stupid to understand Git? I’ve already watched a few tutorials on source tree, git desktop and github. But I still don’t understand the basics, which makes me feel quite alone with my limited mind. What is the difference between commit and push? Why do I even need them if I just want a backup? How does the twigs work? When I use git, I feel like I’m in a minefield. I press in fear that my voice will suddenly disappear because I’ve confused undoing commit with revert or pull or merge or whatever. Does anyone know of a foolproof tutorial that even idiots like me can use to understand this wise book?

Git - game

Git command	Game action
commit	save game
push	upload a cloud save
pull	load a cloud save
fetch	get a cloud save without loading it
restore	load the last save
clone	install

2024-11-17/01

ho.dges.online — Sun, 17 Nov 2024 02:00:00 GMT

2024-11-17/02

ho.dges.online — Sun, 17 Nov 2024 02:00:00 GMT

/junk

taylor.town — Sun, 17 Nov 2024 02:00:00 GMT

welcome to my internet yardsale

ChatGPT is Slipping

Adriano Caloiaro's personal blog — Sun, 17 Nov 2024 02:00:00 GMT

ChatGPT is slipping

Three months ago I put code in production that utilizes gpt-4o and/or gpt-4o-mini models to analyze feedback about businesses and categorize it. The prompts instruct the models to identify categories of feedback, and in a second phase, extract some examples of what people said. This is a simplification, but it took very little effort to craft some prompts that enabled even the meager gpt-4o-mini model to do exactly that. It didn’t feel like a stretch to imagine that this use case was well within ChatGPT’s limits based on the minimal effort put into a working solution. The results were genuinely useful, and the effort was low. It seemed like an obvious win.

The code ran every two weeks, and the models had done an admirable job every time. Before putting results in front of users, I let the code run silently, only visible to a select few within our company. Very subjectively, we were satisfied with ChatGPT’s results and figured it was time to put them in front of users. So I threw some basic integration tests around the feature, gave it the old go test -count 1000 ..., and released it to the world.

Like most software engineers, I’m all too familiar with flaky tests, and was hesitant that tests around LLM output would be predictable enough to remain unflaky. But, running a test suite with go test -count 1000 without failure is pretty confidence inspiring. And it turns out that my hesitation was not terribly well-founded. The tests simply never failed for three months. I was so skeptical that a few times over the last month, I cracked the tests open and made sure we weren’t simply getting false negatives. We weren’t, and tests were running multiple times a day on local development machines and during both staging and production deployments. The tests were running about 50 times per week and remained rock solid.

That is, until 11/13/2024. The conversation in my head following that test failure went something like this

I KNEW IT. The test has gone flaky; LLMs obviously can’t be treated like deterministic machines, this was bound to go flaky. I’ll treat it like any other non-deterministic thing and just make the test easier to pass for now until I can re-focus on crafting better prompts.

I guess you could say that in some way I was right. LLMs don’t have a well understood place in deterministic settings. But I was definitely wrong about another: that making the test easier to pass would let me kick this thing under the rug and forget about it for a while.

11/15 rolled around, and this feature failed so spectacularly that I’m still blushing with embarrassment from what it put in front of users. Luckily, it failed so spectacularly that very few of them saw anything at all because nearly every prompt failed to produce results. In my test environment, the models were presented with a relatively small amount of mock data. In production, the models were presented with far more data, and as it turns out, they perform much worse when presented with more data.

What should have happened on 11/13 is I should have gone back to my original, more rigorous test (go test -count 1000 ...) and checked whether ChatGPT itself had regressed. Had I done so on 11/13, the embarrassment of 11/15 would have never occurred. Because I would have seen the tests fail all 1000 times. In other words it became a deterministic failure.

This isn’t your average regression. And I don’t intend to imply that ChatGPT is failing catastrophically for many users, because I’ve yet to find anyone else talking about similar regressions. But I’m certain that something fundamental changed in gpt-4o on Wednesday, 11/13/2024. Maybe it’s a single parameter or weight that affects only a subset of users. Maybe that parameter or weight, or whatever it is affects only me. But what is certain is that something changed, and it changed without any fanfare, like an announcement, blog post, or tweet.

Take this post as a reminder that these models can change on a whim, despite what is published at https://platform.openai.com/docs/models.

ChatGPT may not be slipping for everyone, but it’s certainly slipping for me.

So what’s next?

The first thing I did was disable the feature. I’ll begin testing other models from other providers, but ultimately, I think this should be the beginning of a conversation about self-hosting and local LLMs. I can’t with good conscience continue building on a platform that can change on a whim without any public acknowledgement of what is changing.

There’s already a slow movement toward bringing LLM functionality in-house rather than relying on yet-another 3rd party vendor like OpenAI, Google, or Anthropic. A few things that have my wheels turning are:

Langchaingo (https://github.com/tmc/langchaingo): Abstract away the providers with a common interface. This is not to say the providers and models are fungible; they’re not. But langchaingo makes dropping different models into applications pretty seamless.
Go blog’s write up (https://go.dev/blog/llmpowered): This was a great overview of how one might begin the process of making more LLM functionality local
Ollama (https://ollama.com/): Local-first LLMs.
ML in Go with a Python sidecar (https://eli.thegreenplace.net/2024/ml-in-go-with-a-python-sidecar/): The title says it all. I’m not in love with “sidecar” architecture, but it’s sensible in some settings.

Have you seen any similar declines in ChatGPT quality? I’d love to hear from anyone else: chatgptisslipping@adriano.fyi or through the Mastodon comments below.

Cheers

Updates

11/18/2024

I decided to go back to the original test today and re-run it. It passed on the first try, and every subsequent try (it’s passed dozens of times now). So what was a 100% failure rate days ago, is once again a 100% success rate. It seems like OpenAI is comfortable taking API users for a ride. I assumed models would fluctuate regularly on the web frontend, and naively thought the API would have more stability. Learn from my naivety.

My 2024 New Mac Setup

swyx's site RSS Feed — Sat, 16 Nov 2024 19:43:47 GMT

I set up a new Mac for work today. Here's everything I use on a Mac for fullstack web development.

Conversation Branching

Thought Eddies — Sat, 16 Nov 2024 19:02:42 GMT

Conversation Branching

Building OpenJDK From Source On macOS

Gunnar Morling — Sat, 16 Nov 2024 16:25:00 GMT

Every now and then, it can come in very handy to build OpenJDK from source yourself, for instance if you want to explore a feature which is under development on a branch for which no builds are published. For some reason I always thought that building OpenJDK is a very complex processing, requiring the installation of arcane tool chains etc. But as it turns out, this actually not true: the project does a great job of documenting what’s needed and only a few steps are necessary to build your very own JDK.

Finding better domain names

I Learnt — Sat, 16 Nov 2024 14:15:00 GMT

Finding a good available domain name for a project, product or company is hard but there is one web tool that I have found is more successful than others.

From Corniolo to Passo Braccina via Valpisella

Nicola Iarocci — Sat, 16 Nov 2024 10:06:23 GMT

I took a daily hike in my beloved Apennine Mountains a few weeks ago. One of my favourite motorcycling routes is the narrow, engaging, panoramic road that unites Corniolo in the Bidente Valley with Marradi and the Mugello area via the Braccina Pass. I always wanted to return and hike through it; the moment had come.

It was an excellent circular tour that, to the merits of moderation-it is not too long, nor too strenuous, the ascent is always gradual, etc. - also combines the variety of the environment, shading from mixed mid-altitude broadleaf forests (hornbeams, oaks, ash, cherry, maple) to coniferous reforestations that in this case, with increasing age and size, have taken on a certain dignity.

Imposter Syndrome

jonandnic dot com — Sat, 16 Nov 2024 05:04:41 GMT

In my son's experience with a bad English teacher I saw my own struggles with academics. I also had some really good teachers, and a few really bad ones. The good ones would say things like "he's not living up to his potential." The bad ones told me I was stupid. Its funny how both of those things stick with you the rest of your life...

Fortunate: A Modern Motivational App for Linux

Terminal Thoughts — Sat, 16 Nov 2024 02:00:00 GMT

Recently, I found myself searching for a polished motivational app for Linux, but everything I discovered either required using the terminal or had interfaces that felt outdated. Knowing the wealth of wisdom available in fortune-mod’s databases, I decided to create Fortunate, a modern graphical interface that delivers inspiring quotes throughout your day, while giving me an opportunity to explore GUI development with Fyne.

Chatbots Decoded: Exploring AI (Review)

ℤ→ℤ — Fri, 15 Nov 2024 21:20:34 GMT

Chatbots Decoded: Exploring AI is a new exhibit from the Computer History Museum (CHM) opening November 20th, 2024. In development for more than a year, the exhibit covers both the history of chatbots and conversational interfaces as well as the social ramifications of the technology. While there is not an explanation for why GPUs rather than CPUs are so useful for learning or what “attention is all you need” means, both are alluded too and I think, many audiences, including children, will enjoy it.

Conversational interfaces, as typified by chatbots, have had a long parallel development in computing with more traditional interfaces. Science fiction has featured robots and automated creatures controlled by voice or written words as far back as the Golem and R.U.R. Star Trek’s main computer presents a mixed interface, both tactile graphical displays and voice control. The series differentiated between the computer’s intelligence and sentience. Although the computer could flexibly carry out tasks, it lacked any introspection, motivations, or free will. This distinction is one of the subtleties explored in the exhibit.

The exhibit begins with fictional robots that, by obeying voice commands, helped shape both our expectations as well as fears. The first real grounded example is ELIZA, the conversational system which was typically used to imitate a Rogerian therapist. Visitors have access to a terminal where they can interact with an implementation, hopefully experiencing some of the magic feeling of interactivity but also witnessing the limitations of the technology.

ELIZA, the earliest chatbot system

Interact with an ELIZA acting as a Rogerian therapist

Developments in symbolic logic and improvements such as A.L.I.C.E. are recounted. However, the next big revolution is deep learning where a timeline describes the exponential growth in the models, the amount of data, and the amount of funding. This leads into probably the central showcase: the Ameca exhibit.

Ameca as a chatbot with a physical face

Ameca is a high-fidelity robotic face joined to a ChatGPT model. Visitors can step right up and start talking to Ameca, which tries to spark some conversation on its own. According to a staff member, children feel an immediate emotional connection to the robot, laughing and giggling. For my part, I felt self-conscious, wanting to say something profound amongst all the humans in ear-shot. In order to prompt conversation, Ameca will urge visitors on. This can end up interrupting the visitor’s attempts at interaction, but that ends up increasing the humanity of the experience.

As a topic that is more current event than historical, Chatbots Decoded concludes with a large section of open questions. For instance, CheatGPT raises both the potential of LLMs as education aids, the concerns of students using LLMs as labor-saving devices, the battle to detect LLM generated text versus student authored text, and how relevant the older debate about student use of calculators is to the current fight. With no planned closing date for the exhibit, CHM will need to gauge when new developments warrant an update to the display.

Should students use ChatGPT?

All photos copyright of the author.

Note: As a Supporting Member of the CHM, I gained early access to the exhibit. I was not compensated for the review.

Add Alt Text to an Image with an LLM and Cursor

Thought Eddies — Fri, 15 Nov 2024 17:35:21 GMT

Add Alt Text to an Image with an LLM and Cursor

Thinking differently can be worth a lot

I Learnt — Fri, 15 Nov 2024 09:12:00 GMT

We use the technology Apache Kafka as a key element in the implementation of the distributed ledger we have developed.

Tailwind... quick thought

RobKohr's Blog — Fri, 15 Nov 2024 02:00:00 GMT

Tailwind... quick thought

I Feel Unsafe

jezenthomas.com — Fri, 15 Nov 2024 02:00:00 GMT

I awoke to the sound of a violent explosion earlier than I had wanted that morning. The walls, windows, and furniture vibrated. A cruise missile had struck a warehouse owned by a national supermarket chain, roughly a kilometre from the hotel I was staying in. The russians had fired it – targeting civilian infrastructure as they regularly do – perhaps a dozen minutes earlier. It was June 24th, 2024. The time was 07:03.

Plume of smoke from a russian cruise missile hitting a supermarket warehouse, shot from the window of my hotel room in Odesa, Ukraine.

This was hardly my first rodeo. Cruise missiles regularly flew over the apartment I was renting in Odesa. The local air defence would light up the sky when attacks came at night; as they often did. One attack in 2022 saw an apartment building struck a few streets behind mine, killing a young woman and her two children. Her husband wasn’t in the building at the time, and distraught by the slaughter of his entire family, he joined his country’s defence on the front line. Ultimately, he died too.

Trail from an air defence missile, and the smoke from a successfully destroyed Shahed drone, shot from the kitchen window of my rental apartment in Odesa, October 2022.

With my British passport, I have the luxury of leaving any time living in an active war zone begins to take its toll. My closest friends however, enjoy no such luxury. The russians came to rape, torture, and kill. Neither women nor children are spared. Countless invaders have admitted to the brutality, at times recounting that even small children are executed with a bullet in the brain at point blank range. The Ukrainians have no choice but to stay and fight.

I attended NixCon a few weeks ago in Berlin. Overall I enjoyed the experience; it feels a little less like a conference, and a little more like a get together for the relatively small NixOS community. About half way through the conference, the organisers announced that one attendee had been asked to leave. They were alleged to have made another attendee “feel unsafe”.

I don’t know specifically what happened, beyond someone being made to “feel unsafe”, nor do I have any reason to doubt the accusation (and I’m glad the moderators took the complaint seriously and acted upon it).

What sticks in my mind is the phrase “feel unsafe”. In my experience, I’ve only ever heard this phrase said by people who hold political views radically more left-wing than my own (and for the avoidance of doubt, I’m a bit of a leftie). NixCon – and perhaps the NixOS community more broadly – appears to have an atypically high concentration of people who would describe themselves as Marxists, or Communists, or Anarchists, etc.

Indeed, walking around the conference venue, I spotted at least two dozen Nix hackers with radical left-wing political symbols and slogans emblazoned across their laptops. In the heart of Berlin, the hammer and sickle is apparently fine and reasonable.

My heritage is Polish. I have good memory of my great-grandmother. Her husband, I never met. He died as a slave in a Nazi concentration camp. That side of my family has always lived on the West side of Poland. If they had lived as far East as they do West, then no doubt my great-grandfather would have died at the hands of the Soviets. The Nazis and the Soviets were allied at the start of The Second World War, so to me — and to millions of other Poles no doubt — the hammer and sickle and the swastika are essentially interchangeable.

And the discomfort I experience seeing these political symbols and slogans is not so abstract. The commonly held position among people who describe themselves as Communist today is that Ukraine should not be given the lethal aid they need to defend themselves from a genocide that, to date, by some estimates has seen about a million casualties.

Campaign material from the ‘Revolutionary Communist Party’ in Denmark, who are explicitly against arming Ukraine.

The reasoning I have been able to discern for this position is:

Guns and bombs are bad and kill people, so the West shouldn’t make them or send them to Ukraine.
Naziism is bad, and the Communists defeated the Nazis, therefore Communism is good (conveniently forgetting the Molotov-Ribbentrop Pact).

The reason why my closest friends in Odesa are not dead today is because hundreds of thousands of brave Ukrainian men and women have been successfully defending their country from terrorists and barbarians with the use of Western weapons.

To say that Ukraine should not be given weapons is tantamount to saying that all of my friends should be dead. And, I suppose, that I should be dead too. It’s only blind luck that neither cruise missiles nor Shahed drones have struck a building that I was staying in. Although come to think of it, a Shahed drone did indeed hit a building where I used to live. Possibly even the same floor of that building. There have been so many attacks that I don’t perfectly recall.

And yet, I’ll bet that if I were to use the same phrase at that conference — if I were to say that I “feel unsafe” — I somehow doubt that my complaint would be taken seriously. I am — superficially anyway — not a part of the persecuted class, so violence against me is fine.

Linux Assembly Part 4: Arithmetic Operations

Cookie Engineer's Web Log — Fri, 15 Nov 2024 02:00:00 GMT

Learn Linux Assembly to do arithmetic operations.

Vanishingly Few Developers Can Build an Entire App From Start to Finish — Are You One of Them?

Chris' Full Stack Blog RSS Feed — Fri, 15 Nov 2024 02:00:00 GMT

Yep, I'm dropping truth bombs!

Bored in an exam

Obvious Bicycle — Thu, 14 Nov 2024 23:59:24 GMT

A poem

Small things can make a difference

I Learnt — Thu, 14 Nov 2024 19:05:00 GMT

A single, creative decision transformed our interactions with some junior engineers, boosting their morale and integrating them seamlessly into our team.

Practical Deep Learning, Lesson 5, Pricing Iowa Houses with Random Forests

Thought Eddies — Thu, 14 Nov 2024 15:27:45 GMT

Practical Deep Learning, Lesson 5, Pricing Iowa Houses with Random Forests

使用 Go 开发 AI Agent的选择：Genkit for Go

ipfans's Blog — Thu, 14 Nov 2024 13:28:00 GMT

什么是 Genkit

Genkit 是一个 Google Firebase 团队开发的 AI Agent 开发框架，用于构建现代、高效的 AI 应用。它目前包含一个 Node.js 的实现和一个 Go 语言的实现。之所以注意到这个框架是因为 Go 团队在他们的十五周年博客中提到了它。Go 团队在博客中提到，他们正在努力使 Go 成为构建生产 AI 系统的优秀语言，并且他们正在为流行的 Go 语言框架提供支持，包括 LangChainGo 和 Genkit。如果你了解过 AI 开发，那么 LangChain 一定并不陌生。LangChainGo 就是 LangChain 的 Go 语言实现。但是对应的，我们还需要一个单独的 AI Agent 开发框架，帮助我们组织 AI Agent 的开发。LangChain 的生态位中，对应的是 LangGraph，在 Go 语言的生态位中，你可以选择 Genkit 或者 LangGraphGo。

但是，值得注意的是，目前 Genkit 仍旧是在 Alpha 的状态，所以，如果你希望将 Genkit 用于生产环境，在 2024 年这个时间点请注意未来可能的 API 变动。 目前 Genkit 官方支持的模型功能仍旧是以 Google 家的产品为主，如果你需要使用其他 AI 模型，你可能需要暂时先使用第三方 plugin（比如 OpenAI）。

Film Development Calculator

Home on Lagomorph — Thu, 14 Nov 2024 09:00:00 GMT

Film & Developer

Film Stock:

Developer:

Setup Parameters

Tank Volume (ml):

Developer Ratio (1:x):

e.g., 50 for 1:50

Base Development Time (minutes):

Dilution Results

Developer needed: 7.5 ml

Water needed: 367.5 ml

Adjusted development time: 11 minutes

Process Timeline

Pre-soak: 1:00

Development: 11:00

Stop Bath: 1:00

Fix: 5:00

Rinse: 10:00

Total Time: 28:00

Pre-soak (min):

Stop Bath (min):

Fix (min):

Rinse (min):

Adjustments

Temperature:

20°C

Push/Pull Adjustment:

0 stops

Agitation Reminder:

Development Timer

00:00

Current step: Not started

Working Chemical Dilutions & Stock Life

Developer	Working Dilution	Stock Life	Notes
D-76	1:1	6 months	Life improves in full bottles
HC-110	Dilution B (1:31)	2+ years	Syrupy concentrate very stable
	Dilution E (1:47)	2+ years	For lower contrast
	Dilution H (1:63)	2+ years	For stand development
XTOL	Stock	2-3 months	Sudden death - test before critical use
	1:1	2-3 months	More economical
Rodinal	1:25	Decades	High acutance, pronounced grain
	1:50	Decades	Standard development
DD-X	1:4	6 months	Professional choice, fine grain
Microphen	Stock	6 months	Best for push processing
	1:1	6 months	Normal development
Pyrocat-HD	1:1:100	1 year (A), 6 months (B)	Excellent for alternative processes
ID-11	Stock	6 months	Similar to D-76
	1:1	6 months	Economy option
Perceptol	Stock	4 months	Ultra-fine grain
	1:1	4 months	Extended development

Notes:

A Formative Artifact

Matthew Gaudet — Thu, 14 Nov 2024 06:29:15 GMT

Do you have a formative artifact from your childhood? Maybe a TV show, a book, a game which you feel (perhaps wrongly) you can trace all your personality back to.

When I was a kid, probably 7-8 until I was probably 10 or 11, I had this book which I remembered as a book of facts. It wasn’t an encyclopedia, but it had this eclectic bent -- I mostly remember the science bits, but there was also bits about society and culture too.

It was reasonably large, with large illustrations and I read this book over and over and over and over until it fell apart. Literally. The last time I remember seeing it, I had read it until the covers had both fallen off, and all that remained was the spine. At some point presumably when I had outgrown it, we must have cleaned out my room and thrown it out. After all, in tatters it was scarcely better than trash.

Of course, looking back, this book was definitely one of my formative artifacts. What it instilled in me was a thirst for knowledge; eclectic wide ranging knowledge. Of course, the challenge being that like this book, my knowledge is patchy and broad.

Fast forward almost 25 years, and I found myself thinking about this book. Raising my daughter, and seeing the multitude of ways in which her childhood is different than mine, I keep wondering what will be her formative artifacts. What thing will captivate her? Could it be something like my book? Could I find that book perhaps? Would a book that old capture her imagination?

Of course, after so long, I could no longer remember what this book was called. Who wrote it. What the cover looked like. If I’m being honest, I could barely remember the actual contents of the book were; some illustrations burned deep into my brain mostly.

A few weeks ago I was thinking about it, and I realized I did know a few things about it. Maybe enough to find it? I remembered that it was published by Reader’s Digest. Given when I was reading it, probably in the 1990s. My recollection was that it was a “science” book.

So I started looking. I had been chatting with a friend about this book, and thought... this could well be just the kind of vague nonsense that ChatGPT might actually be helpful with. Not that I expected it to find the book, but that it might help me narrow down on something.

Me: Looking for an old book published in Canada by Readers Digest. It was available in the nineties. It was a thick book filled with facts and experiments. Aimed at young kids. I don’t have a title.

ChatGPT: Without a title, narrowing it down to a specific book can be tricky, but based on your description, it might be one of the popular educational reference books or experiment guides published by Reader's Digest during the 1990s. Reader's Digest released several children's educational and activity books during that time. Some titles that could fit your description are:

How Science Works (Reader's Digest, 1991): This book explained scientific concepts through facts and hands-on experiments, aimed at younger readers.

The How and Why Library (Reader's Digest): A multi-volume set filled with facts, trivia, and educational material aimed at children.

You might try searching through secondhand bookstores or websites like AbeBooks, which specialize in out-of-print books. If you're near a large library, they might have archives of Reader's Digest publications from that era as well.

Doing some searches on “How Science Works” and the “How and Why Library”, I was able to quickly rule out the “How and Why Library”. But... “How Science Works”, well that fit the bill pretty good. So I found a copy on AbeBooks, and it was pretty affordable. I managed to find one photo of the interior... and I wasn’t sure. My memory was fuzzy, but I thought: This is worth a try. So I bought it.

It arrived, and from the moment I opened it, I knew it.

It was the wrong book.

Seems like a neat book, and the experiments could be fun with my kids when they’re a bit older, but this wasn’t the right thing.

So another day, I found myself wandering around AbeBooks, when I realized they had an Advanced Search where you could search by year, by publisher, etc. I started working it out:

Published between 1985-1999
Keyword: Science
Publisher: Reader’s Digest

Wait. That... looks familiar.

Turns out. I have it in my house now. My formative artifact:

Some of these pages, some of these illustrations are baked into my head deep.

Some of these stories are still how I conceive of things, having never really had to update my knowledge in my head (for example, the story of the coelacanth).

I have a very strong feeling this book will not leave nearly the impression on my children that it left on me. Yet, I find myself overjoyed at having found it, this little formative artifact.

Dear Reader: I hope you too have yourself a formative artifact. I hope you can find it, and, maybe yours can be passed on to your own children. A gift they are free to decline, but hey — perhaps they’ll take it.

Interactive Python Exercises and Quiz

learnbyexample — Thu, 14 Nov 2024 02:00:00 GMT

Having an interactive program that automatically loads questions and checks the solution is wonderful to have while learning a topic. This TUI app has beginner to intermediate level exercises and multiple-choice questions for Python learners.

Installation🔗

This app is available on PyPI as pythonexercises. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install pythonexercises

# launch the app
$ pythonexercises

If you are on Windows, using the Windows Terminal is recommended. See this issue for Virtual Environment commands and other details.

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias pythonexercises='/path/to/textual_apps/bin/pythonexercises'

As an alternative to manually managing such virtual environments, you can use https://github.com/pypa/pipx instead:

$ pipx install pythonexercises
$ pythonexercises

As yet another alternative, you can install textual (see Textual documentation for more details), clone this repository and run the python_exercises.py file. You'll need to install textual[syntax] to enable syntax highlighting (see documentation for more details).

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines). Here's another screenshot:

Guide🔗

See app_guide.md for instructions.

Ebook🔗

The exercise and quiz questions in this app have been adapted from my 100 Page Python Intro ebook.

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you find this TUI app useful. Happy learning :)

Installing a Fedora Linux ARM VM from a Raw Image

Jason Eckert's Website and Blog — Thu, 14 Nov 2024 02:00:00 GMT

Many professionals who work in Information Technology (IT) or software development use an ARM-based PC, such as an Apple Silicon Mac, Snapdragon Elite X Windows Copilot+ PC, or an ARM Linux workstation such as the System76 Thelio Astra. Similarly, many students learning IT or software development may also use an ARM-based PC. For all of these people, it’s commonplace to deploy Linux virtual machines (VMs) on a PC using hypervisor software to test cloud apps, host server software, or run specialized tools.

Maneten - en novell i tre meningar

Liza Shulyayeva — Wed, 13 Nov 2024 23:01:33 GMT

Jag har lärt mig svenska mer intensivt på sistone, och nu tänker jag att tiden har kommit för mig att skriva mer på språket. Jag bestämde mig för att skriva en kopia av min allra första novell, som jag skrev på ryska när jag var liten i Ukraina:

Life can be a beach

Aswath Krishnan — Wed, 13 Nov 2024 20:04:33 GMT

You won’t find many miserable people lounging on a Hawaiian beach. There’s something transformative about the meeting of sea and shore that seems to wash away the burdens of the world. Even toddlers and their usually-under-duress parents are in good spirits. So, I’m only half-joking when I say the beach life might be a good vision for a happier humanity; much better than flying cars, immortality, or space conquests.

Think about it: at the beach, everything’s just... right. You’re not caught up in a to-do list. No meetings, deadlines, or worries. The sun kisses your skin just enough to make you feel alive. The water? Hypnotic, refreshing, and fun. You nap, you read, you write, you frolic, and you just be.

The beach life, both literally and as a metaphor, feels like a piece of paradise. Imagine if we could turn this once-a-year indulgence into a way of being. To those of you who instantly feel the practicality alarms going off, thinking, "But we can't live on vacation!”—you need some Aloha, my friends. Beach life isn't about perpetually lounging in hammocks, sipping piña coladas; it's about flowing through life with a sense of ease, bliss, freedom, and fulfillment.

And sure, I’m not ignoring the obvious. The only reason I get to kick back on a beach is because someone’s working hard nearby—keeping up the hotel, cooking my meals, maintaining the healthcare systems, all of that. I pay for this with my own work, which is similarly valuable. The challenge isn’t to ignore labor, but to reimagine it: how do we get the essentials done faster, easier, and with less stress so we can beach more? How do we redesign our days so that there’s more room for joy and fewer pointless distractions—like doom-scrolling or chasing hollow status symbols? How do we make space for the good stuff that actually nourishes us?

Beach life is also as much in the mind as it is in what we do and where we are (and they are all inextricably linked). Lucidity, equanimity, optimism, and cheerfulness can color your regular day with the hues of ocean and sand.

If the beach life sounds appealing, it needn't be a distant dream. Most of us live so mindlessly, so sub-optimally, that repeating this simple exercise can unlock hours each day within just a few weeks: analyze your day, cut out the junk, automate or streamline what you can (sprinkle some utopAI perhaps), and focus on what truly matters. Also do, say, and think things that put you in a good mindset. Imagine trading hours of mindless chores or stress for moments that actually add value to your life.

I'm naively hopeful that, with a little intention, beach bliss might be well within our reach.

Rethinking selling

I Learnt — Wed, 13 Nov 2024 19:25:00 GMT

Making a sale of a Software as a Service (SaaS) product to a large enterprise customer can be slow and difficult. There are a lot of potential roadblocks that need to be overcome.

Quick Tip: How to Fix Stale file handle & RPC: Program not registered between A FreeBSD 14 Server and Linux Client

9μm Pixels — Wed, 13 Nov 2024 09:44:46 GMT

Sometimes, for reasons still a mystery to me, I get an "Stale file handle" error when trying connect to my FreeBSD 14 fileserver's NFS server. Despite much internet sleuthing & LLM queries, I was not able to solve this until literally just now:

service mountd stop

chatgpt-shell splits up

xenodium.com @alvaro — Wed, 13 Nov 2024 02:00:00 GMT

The chatgpt-shell package started as an experiment glueing the ChatGPT API to an Emacs comint buffer. Over time, it grew into several packages within the same repository: shell-maker, ob-chatgpt-shell, dall-e-shell, ob-dall-e-shell, and of course chatgpt-shell itself.

I'm splitting the repository as a first step in reworking chatgpt-shell to enable multi-model support (i.e. Gemini, Claude, and others), a popular feature request.

Want multi-model support?

Go 👍 the feature request and ✨sponsor✨ the work.

If keen on having a multi-modal chatgpt-shell at your fingertips, please consider sponsoring to make the project sustainable. Improvements like this, integrations, and keeping up with the AI space takes quite a bit of work and effort.

New package repositories

chatgpt-shell

No repo location changes. Remains at https://github.com/xenodium/chatgpt-shell

chatgpt-shell carries the ChatGPT shell itself, but also convenience integrations.

My hope is to make this a multi-model package.

ob-chatgpt-shell

Moves to https://github.com/xenodium/ob-chatgpts-shell

An extension of chatgpt-shell to execute org babel blocks as ChatGPT prompts.

dall-e-shell

Moves to https://github.com/xenodium/dall-e-shell

A dedicated shell for DALL-E image generation.

ob-dall-e-shell

Moves to https://github.com/xenodium/ob-dall-e-shell

An extension of dall-e-shell to execute org babel blocks as ChatGPT prompts.

shell-maker

Moves to https://github.com/xenodium/shell-maker

shell-maker a convenience wrapper around comint mode to build shells. Both chatgpt-shell and dall-e-shell are built on top of shell-maker.

Enjoying this content? Using one of my Emacs packages?

Help make the work sustainable. Consider sponsoring. I'm also building lmno.lol. A platform to drag and drop your blog to the web.

Healthy Things Grow

Dan Stroot — Wed, 13 Nov 2024 02:00:00 GMT

Gardeners don't 'create' or 'produce' tomatoes. They can only create an environment where tomato plants will thrive. The point is if the environment is healthy, growth happens. There are a lot of factors to consider - the quality of the soil, the weather, pests... the list goes on. Similarly, how do we create healthy growth in our organizations?

This post Healthy Things Grow first appeared on Dan Stroot's Blog

The Pirate Bay på SVT

Liza Shulyayeva — Tue, 12 Nov 2024 23:01:33 GMT

Jag har börjat titta på The Pirate Bay på SVT Play den här veckan, och den är asbra! Serien handlar om hur, vem, och när The Pirate Bay var utvecklades i Sverige. Den har en riktig “Hackers” och “Halt and Catch Fire” vibe. Desvärre finns det bara två avsnitt förtfarande, och jag slukade båda i ett nafs.

Are reviews broken?

I Learnt — Tue, 12 Nov 2024 22:19:00 GMT

A mobile app developer was frustrated recently by receiving a one star review for their application.

Waters of the Pacific

Writings of Urda — Tue, 12 Nov 2024 18:00:00 GMT

I bathed in the waters of the Pacific and I was born again. I was molded and perfected by the startup battlegrounds of San Francisco, my character and ability were put to the test at Amazon, and now I am building the road to space for future generations using everything I have learned so far. This is my engineering story.

When generating apps the spec is important

Modern Web Development with Chrome — Tue, 12 Nov 2024 12:42:00 GMT

Generating web apps with AI agents like Replit is incredibly powerful, enabling rapid prototyping and deployment. My experience building tldr.express, a personalized RSS feed summarizer, highlighted the importance of a detailed specification. While initial prompts yielded impressive results, I iteratively refined the app through configuration and additional prompts to address issues like email integration, AI model selection, output formatting, spam prevention, and bot mitigation. This iterative process reinforced that while AI agents excel at rapid generation, a well-defined specification upfront is crucial for a successful outcome.

A little bit of delusion

Martin Rue — Tue, 12 Nov 2024 09:00:00 GMT

94% of people believe they're above average, but a little bit of delusion is a good thing.

CSS fix to prevent orphan icons dropping to a new line

Muffin Man — Tue, 12 Nov 2024 02:00:00 GMT

When an HTML element becomes too narrow, its content starts to wrap into multiple lines. This is intended behavior and works well in many cases. However, for short text, it doesn't look great when the last word or icon drops to the next line, becoming an orphan.

For example, you might see something like this:

Click here for more info

It doesn't look great when the icon is left alone on its own row. Luckily, there's a simple CSS solution.

Solution #

Only thing we need to add is a single line of CSS:

text-wrap: balance;

This property tells the browser to wrap the text in a way that best balances the number of characters on each line, enhancing layout quality and legibility. The browser will calculate this for us, preventing orphaned icons or words.

text-wrap: pretty #

There is also the pretty value for text-wrap, which is even better. It works in a similar way, but the browser will use a slower algorithm that prioritizes better layout over speed.

Unfortunately, at the time of writing, pretty is not supported in Firefox nor Safari. However, you can define both values, and browsers that don't support pretty will fall back to balance.

text-wrap: balance;
text-wrap: pretty; /* This will be ignored by browsers that don't support it. */

Demo #

Here's a simple demo. Use the handle on the right to resize the element and see how text breaks with and without the balance property applied.

Defaulttext-wrap: balancetext-wrap: pretty

Unfortunately your browser doesn't support text-wrap: pretty.

Click here for more info

A somewhat longer text that will break into multiple lines

2024-11-12

ho.dges.online — Tue, 12 Nov 2024 02:00:00 GMT

Flouting the Internet Protocols with Tunnels

Mac's Tech Blog — Tue, 12 Nov 2024 02:00:00 GMT

Recently at work I've learned about Cloudflare Tunnels, which has increased my interest in tunneling technologies in general.

Tunneling is a generic networking term, but in the web-dominated world it usually refers to software that you can run on any computer with outbound network access to serve a website. The tunneling software has a another component running in the cloud that configures DNS, terminates TLS, and handles the networking magic to send the packets to your computer.

My main interest is using tunneling for serving websites on my home server. While I'm fortunate to have a fairly static IP address, I don't want to risk having to update my DNS records after a power outage, and I don't like having to port-forward. I say "sites" plural, but I really just have one site, which is an instance of the open-source recipe site Mealie at recipes.macchaffee.com.

I feel like by using tunnels, the authors of various internet protocols are rolling in their graves (wait, are they mostly still alive?), but hey I've got websites that need to be deployed and my ISP won't help me do it.

Choosing the right tunnel

After skimming through the awesome-tunneling list, I struggle to find a better solution than Cloudflare Tunnels, much to my chagrin as someone who fears the centralization of the Internet. My ideal features include:

Designed for production use, not just development.
Good protocol choice for the tunnel itself, such as WireGuard or HTTP/3.
Multi-regional, including integration with a CDN for caching (why not make the most of the extra network hop?).
Written in a performant, memory-safe language.
Extra server-side features like metrics, authentication, and everything else you'd expect from a modern load balancer.
Open-source.
Not a Cloudflare product.

So to any older CDN providers looking to capture a piece of the modern stack, there's your next idea!

With Cloudflare Tunnels out of the picture, I ended up compromising on #2, #3, and #5 on my wishlist and settled on tuns. They also offer static site hosting with features I wanted, so I took the bundle deal and got both.

The client-side part of tuns is just ssh -R, which makes it very easy to get started. There's also tunmgr which is less than 1k lines of Go code, mainly just wrapping the ssh -R equivalent from Golang's standard library, plus some optional Docker integration.

With my choice of tunneling software made, I moved on to the fun part: deploying tuns to serve traffic to my recipe site hosted on my home server.

Deploying

First, a short summary of my homelab setup: It's an Intel NUC minicomputer running NixOS which is running k3s (a minimal Kubernetes distribution). Everything's deployed from a git repo using FluxCD.

So since tunmgr doesn't provide a Kubernetes installation mechanism, I made it myself (and contributed it).

The steps remaining were:

Generate a new SSH key.
Add it to my Pico account.
Also add it as a Kubernetes Secret (encrypted in git with SOPS) to my cluster.
Deploy tunmgr (a single Deployment, with locked-down permissions and even an egress NetworkPolicy to ensure it only talks to the tuns.sh server).
Configure tunmgr to direct all traffic for recipes.macchaffee.com to Traefik (this is needed to obtain normal load balancer features like metrics and IP allow-listing, which will come into play later).

After configuring my DNS and waiting a short delay for the TLS cert to be provisioned, my recipes site was up and running! No port-forwarding required! Here's what it looks like:

Not pictured: hundreds of lines of YAML

But then tragedy struck. As soon as the hostname recipes.macchaffee.com landed in the Certificate Transparency logs, I was reminded that the internet is a hostile place since I immediately received requests from scanners searching for vulnerabilities or sensitive files (a common occurance).

I constantly struggle with the balance between hosting "public" sites that aren't too public, including this blog. My recipe site is the same. I want to be able to access it from outside my house, and I want my friends and family to be able to access it too without having to set up a VPN client or keep track of a password. But I don't want random scanners to eventually find a vulnerability in Mealie and automatically deploy some malware straight out of a scifi horror movie on my home network.

So I needed to build the Internet equivalent of a "No tresspassing" sign, which I call ip-pass.

The creation of ip-pass

My software engineer skills have been collecting dust ever since I moved into DevOps and started "coding" in YAML, but I dusted them off and started up a new Golang project.

This was also a good excuse to practice AI-assisted coding, since I've read a lot more Golang than I've written as a result of spending years debugging issues in various CNCF projects. I just used Claude for some things instead of scrolling through GitHub to find an example to copy/paste. The only interesting thing that happened is Claude generated a test case with the IP address 203.0.113.333 which is invalid. Thankfully the test failed and caught it, because I didn't notice. I think this is a lesson that "just have a human check the AI's output" will never work because AI easily generates more output than humans have attention. You need strict automated checks (like a compiler) for this kind of thing.

The project is one giant main.go file which starts an HTTP server which reads the user's IP address and uses the Kubernetes client-go to add the IP to an allow list. Since I'm using Traefik, ip-pass updates a Traefik Middleware that contains the allow-list.

There's a web interface too, which I've deployed to access.macchaffee.com (also via tuns).

When you go to access.macchaffee.com, you click a big green "Gain Access" button which allow-lists your IP address and redirects you to recipes.macchaffee.com. Since the recipes site blocks all IPs not on the allow-list, bots are thwarted, but anyone with a brain and a browser can still get in with minimal hassle.

Obviously this has no real security benefits because it's just security by obscurity. It's like changing your default SSH port to something other than port 22. But I think it hits a sweet spot on the security, usability, functionality triangle.

Getting comfortable with psql

CRAIG KERSTIENS — Mon, 11 Nov 2024 17:57:56 GMT

psql is a CLI editor that ships with Postgres. It’s incredibly powerful for working with Postgres, and doesn’t take too much of a learning curve to start to get comfortable so you can really feel like an expert working with your database.

Just a rundown of a few things to get you started:

Once connected in psql you can get an idea of all utility commands available with:

\?

A handy thing I use all the time is \d.

\d will describe the tables within database. You can also add a table/index/etc. onto it to describe that specific table such as:

\d accounts

There are a number of options you can set in your psqlrc (config) file to customize your CLI experience. But you can also toggle those when directly working in psql.

\timing will give you the time it took to run your query
\x auto will autoformat your text output
\pset pager 0 turns off your pager or 1 to turn it back on

Oh and for editing a query in your editor of choice. Make sure you set your $EDITOR enviroment variable to the editor of your choice, though the only right choice is vim:

\e

Just a few things to get you started working with psql.

Your browser does not support video

Are AI Assistants Making Us Worse Programmers?

Software Engineering - Rafael Quintanilha — Mon, 11 Nov 2024 16:33:45 GMT

Perspectives on the rise of AI in programming and its effects on productivity.

Reproductive Biology for Politicians and Voters, Part 1

De Novo — Mon, 11 Nov 2024 14:49:27 GMT

Embryos, IVF, and fetal development

When is read-only not read-only?

Raesene's Ramblings — Mon, 11 Nov 2024 14:00:00 GMT

Bit of a digression from the network series today, to discuss something I just saw in passing which is an interesting example of a possible sharp corner/foot gun in Kubernetes RBAC.

Generally speaking for REST style APIs GET requests are read-only, so shouldn’t change the state of resources or execute commands. As such you might think that giving a user the following rights in Kubernetes would essentially just be giving them read-only access to pod information in the default namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: 
    - "pods"
    - "pods/log"
    - "pods/status"
    - "pods/exec"
    - "pods/attach"
    - "pods/portforward"
  verbs: ["get", "list", "watch"]

However due to the details of how Websockets works with Kubernetes, this access can allow for users to run kubectl exec commands in pods and get command execution rights in that namespace! There’s information on the origins of this in this Github issue but it’s essentially down to how websockets works.

What’s possibly more interesting is that, while this behaviour has been in place for a while you might not have noticed it, as the default in Kubernetes was to use SPDY for exec commands instead of websockets, until Kubernetes version 1.31. So if a user with GET rights on pods/exec tried to use kubectl exec in 1.29 you’d get an error like this

Error from server (Forbidden): pods "test" is forbidden: User "bob" cannot create resource "pods/exec" in API group "" in the namespace "default"

but if a user with the exact same rights, tried the same command in Kubernetes 1.31 it works!

kubectl --kubeconfig bob.config exec -it test -- /bin/bash
bash-5.1# exit
exit

It’s worth noting that, whilst it’s easier to do now, using websockets with these rights has been possible for a long time using tools like kubectl-execws from jpts.

Conclusion

Kubernetes RBAC has some tricky areas where the behaviour you get might not be exactly what you expect, and sometimes as in this case, those unexpected behaviours are not very apparent!

In Cold Blood by Truman Capote

Apurva Shukla's RSS Feed — Mon, 11 Nov 2024 13:57:36 GMT

⭐ ⭐ ⭐ In Cold Blood by Truman Capote was a tale it feels you’d hear from a stranger at a bar, detailing a series of events that seem…

Rich losers

Amin Eftegarie — Mon, 11 Nov 2024 06:38:39 GMT

There are people who are rich, yet who don’t earn my respect. Because they’ve become rich not by serving others or producing products that people want, but by doing things that only benefit themselves. I call them rich losers. They’re often involved in things like crypto gambling, dropshipping, playing poker or malicious affiliate marketing. If […]

Your Wi-Fi might be terrible because of Dynamic Frequency Selection (DFS)

./techtipsy — Mon, 11 Nov 2024 06:00:00 GMT

For a few months, I had issues with my Wi-Fi network. The 2.4 GHz network would be fine, but the 5 GHz one would suddenly stop working and completely disappear from the available Wi-Fi networks. OpenWRT upgrades also didn’t improve the situation. This was very annoying.

After some discussions with a friend, I learned about Dynamic Frequency Selection (DFS). Apparently some channels on the 5 GHz Wi-Fi spectrum are also used by weather and military radars, and those take priority. If such interference is detected, your Wi-Fi access point should switch to a different channel.

It turns out that some implementations are buggy and mine is one of them.

One quick but permanent fix is to manually select a Wi-Fi channel to operate in.

With OpenWRT, you can get a list of all the available Wi-Fi channels using this command:

iw list | grep dBm

Example output:

* 2412 MHz [1] (20.0 dBm)
* 2417 MHz [2] (20.0 dBm)
* 2422 MHz [3] (20.0 dBm)
* 2427 MHz [4] (20.0 dBm)
* 2432 MHz [5] (20.0 dBm)
* 2437 MHz [6] (20.0 dBm)
* 2442 MHz [7] (20.0 dBm)
* 2447 MHz [8] (20.0 dBm)
* 2452 MHz [9] (20.0 dBm)
* 2457 MHz [10] (20.0 dBm)
* 2462 MHz [11] (20.0 dBm)
* 2467 MHz [12] (20.0 dBm)
* 2472 MHz [13] (20.0 dBm)
* 5180 MHz [36] (23.0 dBm)
* 5200 MHz [40] (23.0 dBm)
* 5220 MHz [44] (23.0 dBm)
* 5240 MHz [48] (23.0 dBm)
* 5260 MHz [52] (20.0 dBm) (radar detection)
* 5280 MHz [56] (20.0 dBm) (radar detection)
* 5300 MHz [60] (20.0 dBm) (radar detection)
* 5320 MHz [64] (20.0 dBm) (radar detection)
* 5500 MHz [100] (26.0 dBm) (radar detection)
* 5520 MHz [104] (26.0 dBm) (radar detection)
* 5540 MHz [108] (26.0 dBm) (radar detection)
* 5560 MHz [112] (26.0 dBm) (radar detection)
* 5580 MHz [116] (26.0 dBm) (radar detection)
* 5600 MHz [120] (26.0 dBm) (radar detection)
* 5620 MHz [124] (26.0 dBm) (radar detection)
* 5640 MHz [128] (26.0 dBm) (radar detection)
* 5660 MHz [132] (26.0 dBm) (radar detection)
* 5680 MHz [136] (26.0 dBm) (radar detection)
* 5700 MHz [140] (26.0 dBm) (radar detection)
* 5720 MHz [144] (13.0 dBm) (radar detection)
* 5745 MHz [149] (13.0 dBm)
* 5765 MHz [153] (13.0 dBm)
* 5785 MHz [157] (13.0 dBm)
* 5805 MHz [161] (13.0 dBm)
* 5825 MHz [165] (13.0 dBm)
* 5845 MHz [169] (13.0 dBm)
* 5865 MHz [173] (13.0 dBm)

Notice the ones with (radar detection) at the end? Those are the potentially problematic channels. We’re going to avoid them from now on by picking a specific channel to use.

When it comes to the choice of the channels themselves, you’ll also have to consider the channel width. If you pick a channel next to one of the radar detection ones and with a large channel width, you might still run into issues.

Choosing a specific channel also comes with bandwidth and range trade-offs. If you don’t care much for those, go for the lowest one and with 40 MHz width. Picking the optimal Wi-Fi channel and channel width configuration for your specific needs is better explained by other resources.

Choosing a specific channel in OpenWRT using the GUI (LuCI).

I recommend getting a Wi-Fi spectrum analysis app that shows you the channels that are least populated by neighboring Wi-Fi access points.

It is also possible to define the list of channels that the Wi-Fi AP can automatically choose from using the channels option for the wireless interface. We can use this setting to avoid the radar detection channels completely. This setting doesn’t seem to be configurable via the graphical interface (LuCI), but you can change it in /etc/config/wireless using the command line and vi, over SSH.

More information about this option and others can be found in OpenWRT documentation.

Based on our example, a configuration that avoids radar detection frequencies can look something like this:

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'pci0000:00/0000:00:00.0'
        option channel 'auto'
        option channels '36 40 44 48 149 153 157 161 165 169 173'
        option band '5g'     
        option htmode 'VHT40'
        option country 'EE'    
        option cell_density '0'

Using a manually specified channel has resulted in no Wi-Fi related issues for over half a year.

I consider this a permanent fix.

If you’re using a PC and don’t want to mess with Wi-Fi issues ever again, then just run some Ethernet cables. It’s worth it.

Supporting coworkers, employees, and friends in this time

ntietz.com blog - technically a blog — Mon, 11 Nov 2024 02:00:00 GMT

We should always be supporting each other, but it feels particularly important right now. An election just finished in the US, which means that half the country lost and has to face the coming changes. In particular, this is a scary time for many folks who have been targets in the past couple of years, with escalating legislation against access to gender-affirming care.

So, what do we do concretely right now to help each other? This has two parts: what folks need, and how best to help meet those needs. I hope this list helps everyone see that none of us is powerless, and each of us has things we can do, today, to make a difference. If you're affected by recent events, you can keep this list handy as an answer when someone asks "what can I do to support you?"

What folks might need right now

Some slack. When folks are afraid, it's very hard to sustain excellent work output. Right now, people might be in need of some slack and latitude in what they're doing.
Commitments for access to gender-affirming care. Many states have already passed laws restricting access to gender-affirming care, with Florida being perhaps the worst. This seems like a likely priority at the federal level, as well. Most Americans get our insurance through our employers, so it's not something we have a lot of choice over. We need to know that, even if it's not covered by the insurance company, that our employer will cover this care for us.
Commitments for access to reproductive healthcare. There is a risk of restrictions on access to everything ranging from birth control to vasectomies to abortions. This will be targeted at a state level and at the federal level. Just as with gender-affirming care, it's important to know that employers cover this care for us and provide as much assistance as they can.
Remote work. This is important because if folks are living in an area that restricts access to medical care, they may have to relocate. Among other reasons that it's always a good idea, it's especially important now. This means remote work, not just a hybrid model, because it may become unsafe to travel in some areas.
Help with moving. Some people might have to relocate to areas which allow them access to necessary care. This is disruptive and expensive even in the best of times, let alone now! Uprooting your life and leaving friends and family is scary—and people will have to do it when legislatures pass restrictive laws. They'll move to states (or countries) which have more access to the care they need. This bucket includes: help with loading a truck, financial assistance with the move, or just time off for the move itself.
Benefits for their partners/families. Just as gender-affirming care is at risk, same-sex marriage is also in the sights for some. The Supreme Court has signaled willingness to overturn it, and it could become back to states. If that happens, families could lose their health care if their coverage is revoked. So, we need our employers to reassure us that if that comes to pass, their benefits aren't going away, and the company will continue to cover them and their families.
Access to lawyers for help with documents, to safeguard marriage rights. There is reason to think same-sex marriage may be targeted. While some of that will be out of our control, we can draft documents to put together the rights that typically come with marriage. This includes wills (so your possessions pass to your spouse, usually a given from marriage), power of attorney, etc. This helpful thread on mastodon has more details, but it is not cheap ($3,000 for the linked example). If we lose this as a guarantee from the government, we'll need help with both the financial side of this and with figuring out the logistics of it all.
Visible, loud support. It is very important that all those who support our rights continue to do so in a visible way. We need folks to continue posting, showing up at rallies, calling elected representatives. We need folks to keep standing up for queer folks who are harassed in public, asking that bully, "hey, what's your problem?" Visible, loud support helps us know that we're supported, and it helps our elected representatives know that stripping our rights was not the point of this election.
Being listened to and advocated for. I have had quite a few people reach out to me asking if I'm okay, and expressing concern for my safety now. It is healing to know that so many people have enough love for me that they want to check in and make sure I'm okay. This is something everyone can do. You can lend an ear to listen to concerns and worries, then go advocate for solutions in your community, state, country.

How you can help

What you can do directly to help depends a lot on the position you're in. We can each do something, and all of it counts.

Executive of a company. If you're in the top leadership of a company, then you have the most power here. You probably cannot make unilateral decisions—even the CEO reports to someone, the board. But you have the most power of anyone at the company, and you can use this.

Take this power and use it to put your values into action. If you've told your employees you care about them, show them with concrete, decisive action to protect their rights here. As an executive, you can either directly decide these issues, or you are peers with those who can. And if they refuse to do it? You can threaten to strike or walk away, even as a single individual, because as an executive you hold a large bargaining chip.
Manager. If you run a team, then you can probably do many things on this list. You can't directly give commitments for healthcare, but you can pressure your top leadership to do so. You can certainly cut your employees some slack, and let them know that if they need time to heal it's not going to be a problem in their performance reviews. If you are involved in the budget process, then you may have some flexibility to allocate funds toward these expenses, too.

Once again, if there's no action from top management on these, you can let them know that it's a deal-breaker for you. That if the company doesn't respect its employees, you might not be among them. If you have privilege, this is a powerful way to wield it. Why accumulate social capital if you're never willing to spend it for the most vulnerable?
Individual contributor. You can't effect change in the company directly. But you can use two techniques very powerfully: advocacy and collective action.

You can advocate to upper management. This is very important to do, because they need to know about these issues in order to do anything about them. And it's very helpful to have someone do this who is not part of the affected marginalized group, since it both shows that it's not just a "oh give me something" action and it also is a way of lessening the burden on an already overstressed group.

And you can organize collective action. I've done this a couple of times at work, and we produced small but real change. Each time, the basic approach I took was: reflect on what I am concerned about and what we want; talk to people and see if they share my concerns; if they do, see if they'll sign a letter talking about our concerns; present the letter to leadership, with a critical mass. If you're a senior individual contributor, collective action should be right up your alley. It leverages the skills you need to do engineering leadership: listening, managing up, and gathering support.
Friend. You can be a lifeline for your friends. Reach out and see how they're doing. (To all those who've reached out to me: I love you, thank you.) Advocate for them and support them, and be loud about it.

Help them load the truck. Find a lawyer for their document updates. Do the research on where might be safe.

Most of all, keep on loving them and holding them.

* * *

There's probably more. I hope there's more that can be done. But this is what I've got right now.

If you're hurting right now, know that I am, too. I'm tired, and scared, and defiant.

When FIDE, the international governing body of chess, tried to exclude trans women from women's chess, my response was simple. I became active in my local chess club again, and I became a tournament director. I became visible as a trans woman in chess. That was a little different, since FIDE doesn't have any control over my life or my rights.

But I live in a community that supports me, with a family that supports me. I'm going to continue to live my best life to be good representation, if I can be. I'm going to keep writing, and keep making art, and keep making music. And I hope that together, we can get through this with as little harm as possible.

Thank you to two anonymous friends who gave me feedback on earlier drafts of this post. I usually attribute people directly (after asking their permission), but this post is more sensitive than many, and I don't want anyone to be at risk from that.

MTA-STS Preload

Built on Shards of Silicon: Robert Alexander's Tech Blog — Mon, 11 Nov 2024 02:00:00 GMT

Tracking domains that support MTA-STS

Leveraging logprobs to build better generative AI systems

Sophia Willows' Blog — Mon, 11 Nov 2024 02:00:00 GMT

Autofilling form fields has become an increasingly common and practical application of generative AI. With just a bit of context and a well-crafted prompt, AI can produce reliable results for various input fields. One of the most appealing aspects of this use case is its simplicity and quick implementation—developers can often build and deploy this feature without extensive training data, sometimes in just a few hours.

Form field autofilling was one of the first things I ever built with GPT, and since then it’s been infectious throughout applications I’ve maintained. It’s a really great feature to pitch if your exec wants to munge AI into your product, because it genuinely helps users save time while also requiring little investment. It’s unlikely you’ll need to embark on a dramatic data project to successfully ship it, and retrofitting a “✨” button on your form field to trigger it is usually straightforward.

In this post we’ll take a look at a very simple prompt for categorizing extracurricular activities like “chess club” or “soup kitchen,” based on their title. We’ll then take a look at a case where our simple prompt gives us weird output, and explore some metrics we can use to help our system monitor itself in production. By the end you’ll understand two common metrics for evaluating an LLM’s confidence in its output, and have a concrete idea of how to use these metrics to build more reliable user-facing genAI systems.

Read more on sophiabits.com

Yorkshires

Sam Hooke — Sun, 10 Nov 2024 13:00:00 GMT

Ingredients

Makes 8-12 yorkshires (or ~275ml):

70g plain flour.
2 eggs.
100ml milk (both cow or oat milk work well, though oat may fizz a little when baking).
Oil (avocado or sunflower).

Equipment:

Muffin tray with 8-12 holes.
Pouring jug.
Whisk or fork.
Plate with paper towel.

Method

Make the mixture

Pre-heat oven to 210°C fan (230°C without).
Put flour in bowl, beat in eggs until smooth.
Gradually mix in milk.
Transfer the mixture into a pouring jug.

Heat up the muffin tray

Fill each hole in the muffin tray with just enough oil to fill the circle without needing to swish the tray about (typically half a tablespoon).
Place the tray in the oven until the oil is very hot, about 5 minutes.

Pour the mixture

Take the muffin tray with the hot oil out of the oven.
Quickly and smoothly, in one continuous pouring motion, use the jug to fill each muffin tray hole. Aim for the center of each hole, so that the batter radiates out in all directions towards the edge of the circle. Some mixture will inevitably land on the tray as you move from one hole to the next, but this is okay because speed is of the essence.
Once all the mixture is used (you may need to go back and top some up to even them all out), place the muffin tray back in the oven¹.

Bake

Allow to bake without opening the door until risen and golden, 10-12 minutes.
When done, remove from the oven, and transfer the yorkshires out of the tray onto a plate lined with paper towel, which will help soak any excess oil. The yorkshires should pop out without much trouble. If they are left to cool for too long, or the ratio of oil to batter was wrong, they may need help from utensil to be removed.

Ideally, use top shelf of oven. If using two trays across different shelves, take the top one out first and give the bottom one another minute or so. ↩︎

Link to https://blog.kizu.dev/my-mastodon-starter-pack/

Qubyte Codes — Sun, 10 Nov 2024 10:53:45 GMT

A really great how-to article for getting started with Mastodon. My Mastodon Starter Pack — Roma’s Unpolished Posts

Purple Hibiscus

Arun Mani — Sun, 10 Nov 2024 02:00:00 GMT

A rich family may not mean a happy life.

Reflections on the 2024 US Presidential Election

Evan Fields — Sun, 10 Nov 2024 02:00:00 GMT

Warning: fully political post, which is not the typical genre here and not the vibe I’m mostly trying to cultivate. I pretty much stand by this, though my emotions have cooled in the weeks since the election.

Christmas in Notting Hill

Mehul Kar's blog — Sun, 10 Nov 2024 02:00:00 GMT

Bad. Very bad.

Rated 0.5 stars.

Thanks for reading this post via RSS!

Why I Switched to Aerospace: Tiling Window Management Made Easy on macOS

Konrad Kruk Blog — Sun, 10 Nov 2024 02:00:00 GMT

Discover how Aerospace, a tiling window manager for macOS, simplifies window management and boosts productivity. Learn about its powerful features, easy setup, and tips for mastering your workflow on macOS.

TAO - Meta's scalable architecture powering world's largest social graph

Engineering At Scale — Sat, 09 Nov 2024 12:56:37 GMT

Meta (formerly Facebook) has more than 2 billion Daily Active Users (DAU).

How to install Cursor AI IDE on Fedora Asahi Linux Arm64

The Mythical Engineer — Sat, 09 Nov 2024 10:45:00 GMT

Building a Homelab, Part 5 - New Services and Hardening

janissary — Sat, 09 Nov 2024 02:00:00 GMT

Open source mac apps I love

Muffin Man — Sat, 09 Nov 2024 02:00:00 GMT

I want to share some really nice open-source applications I use on a daily basis. I did a similar post about six years ago, so I felt there is a need for an updated one. I focused on the apps that I feel are really convenient but less known.

All of the applications listed are free and open source. For each app, I included a brew install command.

If you happen to like and use them, please consider donating to their respective authors.

Here are all of the apps in alphabetical order:

GUI Applications

Hidden Bar
ImageOption
Itsycal
Karabiner-Elements
Keeping You Awake
Maccy
Maestral
OpenInTerminal
Pearcleaner
Pika
Rectangle

Bash programs

dust
ripgrep
tio
zoxide

GUI Applications #

Hidden Bar #

GitHub brew install hiddenbar

Hidden Bar allows you to hide icons from the menu bar. It seems that every app nowadays wants to be in the menu bar, and this helps declutter it.

ImageOptim #

GitHub | Website brew install imageoptim

As its name suggests, ImageOptim optimizes images. It is a UI for a bunch of lossless image optimization tools. Just drop your images and it will compress them without losing any quality.

Itsycal #

GitHub | Website brew install itsycal

A small calendar that lives in your menu bar. Why is this not built into the operating system?

Karabiner-Elements #

GitHub | Website brew install karabiner-elements

I just can't get used to Apple's international keyboard layout and the tilde being in the bottom left. Karabiner solves that. It's a keyboard customizer that allows you to remap keys and create complex modifications.

Keeping You Awake #

GitHub | Website brew install keepingyouawake

Prevents your Mac from going to sleep by clicking on its icon in the menu bar. I use it when I'm running my pen plotter to make sure the laptop doesn't go to sleep.

Maccy #

GitHub | Website brew install maccy

A clipboard manager. It saves your clipboard history and allows you to quickly access it.

Maestral #

GitHub | Website brew install maestral

A lightweight Dropbox client. The official Dropbox app has become bloated over the years, and Maestral is a great alternative.

OpenInTerminal #

GitHub brew install openinterminal

A Finder extension that allows you to open the current directory in a terminal or a code editor.

Pearcleaner #

GitHub | Website brew install pearcleaner

An app remover that lets you select an app and finds all of its files, allowing you to remove it in a single click.

Pika #

GitHub | Website brew install pika

A system-wide color picker. I'm picky when it comes to these, and Pika works in the way I expect a color picker to work.

Rectangle #

GitHub | Website brew install rectangle

A window tiling app and successor to Spectacle. Allows you to move and tile windows using keyboard shortcuts.

Bash #

You might also want to check out the Modern Unix repository, which contains modern alternatives to many traditional Unix commands.

dust #

GitHub brew install dust

A program that estimates file space usage in the current directory.

ripgrep #

GitHub brew install ripgrep

Ripgrep recursively searches the current folder using regular expressions. I don't use it often, but when I need it, I'm really thankful it exists.

tio #

GitHub brew install tio

A serial device I/O tool, a replacement for the screen command. I'm not using it's advanced features, but I loved the auto-reconnect feature while working on my Retro Frame.

zoxide #

GitHub brew install zoxide

In my previous post, I mentioned that autojump is pure awesomeness. Well, zoxide is pretty much the same thing, but faster.

Both let you jump around the system using fuzzy search, and prioritize the directories you visit more often. If I didn't explain it well, just check the gif on it's GitHub page.

I hope you found a few you like!

Paris Christmas Waltz

Mehul Kar's blog — Sat, 09 Nov 2024 02:00:00 GMT

Watched on Saturday November 9, 2024.

Rated 0.5 stars.

Thanks for reading this post via RSS!

A Paris Christmas Waltz

Mehul Kar's blog — Sat, 09 Nov 2024 02:00:00 GMT

Watched on Saturday November 9, 2024.

Rated 0.5 stars.

Thanks for reading this post via RSS!

Pilgrimages to SF and the valley

Prashanth Sadasivan — Sat, 09 Nov 2024 02:00:00 GMT

I was listening to the most recent episode of Acquired about the founding story of Facebook/Meta, where they discussed a trip that Mark Zuckerberg took to Silicon Valley during the Christmas break just before he started writing the first code for Facebook. David (the podcast host) describes it briefly as

Revert a commited secret from remote repository

The Mythical Engineer — Fri, 08 Nov 2024 07:11:00 GMT

To remove the last commit that contains a secret from your remote repository (GitHub, Bitbucket, etc.) and push again, you can follow these steps:

Setup a Production Redis Cluster

The Mythical Engineer — Fri, 08 Nov 2024 05:24:00 GMT

Bacon for everything - a roadmap

dystroy|Canop / blog — Fri, 08 Nov 2024 02:00:00 GMT

You want to use bacon on non rust projects? So do I.

Phoenix LiveView: Async Assign Pattern

Milwaukee Maven — Thu, 07 Nov 2024 23:31:32 GMT

UPDATE: LiveView has added a built in, more robust way of using this pattern. Check it out here!

I've been using LiveView for about two years now. It's a great framework that makes snappy and responsive pages. One anti-pattern I see fairly often is loading a lot of data in the initial page render.

For the un-initiated, the mount/3 function is called twice. Once for the initial 'dead' render, and again after the socket is connected. Many times, for the sake of simple straight forward code, not much is done differently between these two renders.

I haven't found any references (I'm sure they exist) on a best practice for managing the following flow:

Set sensible, lightweight default values on initial render.
Kick off one or more async processes to make longer running function calls.
Receive the values in the LiveView, and update the assigns.

As always in Elixir, the tools are powerful, and theres many ways to accomplish this.

Spawn a linked, or unlinked process.
Start a supervised, or unsupervised Task.
Make a call to a GenServer, and have it send a message back.
Publish to a PubSub topic, and listen for a response.

Example, setting a load state, and then loading the actual data after socket is connected.


defmodule NewsSite.NewsLive do
    
    def mount(_params, _session, socket) do
       
       socket = if !is_connected?(socket) do
       	assign(socket, :front_page, :loading)
       else
       	assign(socket, :front_page, News.front_page())
       end
       
       {:ok, socket}
    end
    
end

Example, loading fast data on initial render, then slower calls async.


defmodule NewsSite.NewsLive do
    
    def mount(_params, _session, socket) do
       
       socket = if !is_connected?(socket) do
       	assign(socket, :front_page, :loading)
       else
       	pid = self()
       	Task.start(fn ->
        	send(pid, {:front_page, News.front_page())
        end)
       end
       
       {:ok, socket}
    end
    
    @impl true
    def handle_info({:front_page, news_items}, socket) do
      {:noreply, assign(socket, :front_page, news_items)}
    end
    
end

Each of these approaches, have their appropriate use case. For the majority of the cases, in most apps, spawning a child process will suffice.

The problem arises when you move past one async assign. Your mount function will start to get pretty messy, and you'll start breaking things out into private functions. You also must try to stick to some sort of pattern across your LiveViews, otherwise each LiveView will work a bit differently.

So, how did I solve this problem?

The Async Assigns Module

To solve this issue, I've encapsulated this basic pattern into an AsyncAssigns module.

Functionality

Allows you to set default data in the initial render.
Spawn a linked, or unlinked process that will do what is needed to fetch data.
Send a message back to the parent LiveView, and assign the values.

The Benefits

Provides a consistent pattern for asynchronously loading assigns.
Provides a consistent pattern for setting defaults.
Allows assigns that must be loaded together to stay together. Those that are not dependent can be loaded in parallel.

As an example, let's take a look at a News site. Let's assume there is a News module.


defmodule News do
    
    @spec front_page() :: [News.t()] :: {:error, any()}
    def front_page() do
    	# Fast loading news that everyone gets.
    end
    
    @spec news_for_me(user :: User.t()) :: [News.t()] :: {:error, any()}
    def news_for_me(user) do
    	# Slower loading news based on my preferences.
    end
    
end

The `mount/3` function for this site might look like this. We are assuming the user is already authenticated, and is in `assigns.user`. This code will:

Initial call to mount (socket not connected).

Block the initial render until the `front_page/0` data is returned, and assigned to the socket.

Second call to mount (socket is connected).

Spawn an unlinked process that will run `supplier`.
When `news_for_me` returns, the spawned process will send a message back to the LiveView, and it will be assigned to key.


defmodule NewsSite.NewsLive do
    
    def mount(_params, _session, socket) do
       
       socket = async_assign(
       	socket,
        key: :front_page,
        default: News.front_page(),
        supplier: fn socket ->
        	News.news_for_me(socket.assigns.user)
        end
       )
       
       {:ok, socket}
    end
    
end

Again, this is a more trivial example, and you may be wondering why we'd bother doing this, and not just check if the socket is connected in `mount/3`, and load custom news there. The pattern becomes much more useful the more things you are loading. Let's expand on the first example. Assume that the code above has been moved to `async_load_front_page/1`.


defmodule NewsSite.NewsLive do
    
    def mount(_params, _session, socket) do
       
       socket = socket
       |> async_load_front_page()
       |> async_load_friends()
       |> async_load_suggested_articles()
       |> async_load_notifications()
       
       {:ok, socket}
    end
    
    # ...
    
    defp async_load_notifications(socket) do
      async_assign(
        socket,
        default: [notif_count: :loading, notif_unread: :loading],
        on_error: [notif_count: :error, notif_unread: :error],
        supplier: fn socket ->
            %{unread: unread, count: count} = Notifications.unread(socket.assigns.user)
            [notif_count: count, notif_unread: unread]
        end
      )
    end
    
end

Using this pattern, it is much cleaner and faster to implement many things that should load concurrently. In addition, you can see a slightly different use case in the `async_load_notifications/1` example. The `:notif_count`, and `:notif_unread` assigns should be set at the same time. If `key` is not provided, then you can return any keyword list of assigns. This helps ensure that assigns that are tied together are always set together. Also, in this setup, we are giving the caller of `async_load_notifications` a choice on defaults. If not specified, both `:notif_count` and `:notif_unread` will be set to loading. If they only want to check for new notifications, only the count will be set to loading. This way, the rendered list of notifications will not go away, will still be updated when the results are returned. This is a contrived example, so don't pick it apart too much. It's just intended to illustrate the flexibility of the pattern.

Another benefit of structuring your data loading like this, is that is very easy to reload values. Below, you can see two events, one reloads notifications, and one only checks for new notifications. This helps us manage assigns that should be modified together in one spot.


defmodule NewsSite.NewsLive do
    
    @impl true
    def handle_event("reload_notifications", _unsigned_params, socket) do
      {:noreply, async_load_notifications(socket)}
    end
    
    @impl true
    def handle_event("check_for_new_notifications", _unsigned_params, socket) do
      {:noreply, async_load_notifications(socket, [notif_count: :loading])}
    end
    
    # ...
    
    defp async_load_notifications(socket, default \\ [notif_count: :loading, notif_unread: :loading]) do
      async_assign(
        socket,
        default: default,
        on_error: [notif_count: :error, notif_unread: :error],
        supplier: fn socket ->
            %{unread: unread, count: count} = if Notifications.unread(socket.assigns.user)
            [notif_count: count, notif_unread: unread]
        end
      )
    end
    
end

If you'd like to use this code, feel free to copy and reuse from the gist below. I feel it is premature to turn this into a library, but maybe if I evolve it, and test it enough, I will publish it to hex.

Usage



# Add the following to you Web.live_view/1 function so it's available to all LiveViews,
# or add directly to any Live you want to try it in.

use AsyncAssigns
import AsyncAssigns

Update!

This post received a good amount of views, and people have pointed out that it is a much better option to kick off these async processes with Task. I actually was using this, but thought I'd experiment with using spwan / spawn_link directly. I'll have to experiment with how these approaches differ, especially in error / crash situations. If you plan to use the code below, you may want to swap out the spawns for Task calls! (Thanks for calling this out Jason!

Also, I'm using live view version 17.x, and some of the functions have moved around for newer versions. You may need to change where assigns, or is_connected? are being imported from. Thanks for reading! - Andy

Takeover Texas - A live exploration game

Karmanyaah Malhotra — Thu, 07 Nov 2024 22:47:08 GMT

In 24 hours, we made a game playable across the UT campus with live location tracking and a topographic map as a “situation room” display.

HackTX - Auto-aiming Turret

Karmanyaah Malhotra — Thu, 07 Nov 2024 22:38:33 GMT

In 24 hours, we made a crazy auto-aiming turret that shoots Orbeez at A&M students, and won HackTX 2023!

THE BOX - Hackathon Time Capsule

Karmanyaah Malhotra — Thu, 07 Nov 2024 22:22:28 GMT

I made a hackathon time capsule that’s been going to hackathons accross the country for over a year sharing memories!

Google Banned Me From Google Voice

Danny Guo — Thu, 07 Nov 2024 02:00:00 GMT

on-run commands on file event

blog — Wed, 06 Nov 2024 22:04:39 GMT

User Agents Hitting My Site

Modern Web Development with Chrome — Wed, 06 Nov 2024 18:54:00 GMT

Curious about who's visiting my site, I built a user-agent tracker using Vercel middleware and KV storage. It logs every request and displays a live table of user agents and hit counts, refreshing every minute. Check out the code on GitHub!

Countdown timer

Modern Web Development with Chrome — Wed, 06 Nov 2024 12:46:00 GMT

I created a simple countdown timer web app that lets you track time until important events. It's built with a focus on no-code using Replit, including a cool integration with Black Forrest Labs' image API via Replit's Agent feature. Check out the live app and source code!

Testing the Silk Platform in 2024: Achieving 20 GiB/s I/O Throughput in a Single Cloud VM

Tanel Poder Blog — Wed, 06 Nov 2024 02:32:23 GMT

Hands-on technical analysis of a novel data platform for high-performance block I/O in the cloud,
tested by Tanel Poder, a database consultant and a long-time computer performance nerd.

Background and Motivation

Back in 2021, my old friend Chris Buckel (@flashdba) asked me if I would like to test out the Silk Data Platform in the cloud and see how far I could push it. And then write about it. We got pretty nice results, 5 GiB/s IO rate to the Silk backend just from a single cloud VM.

Embedding Collapse in Recommender Systems: Causes, Consequences, and Solutions

Sumit's Diary — Wed, 06 Nov 2024 02:00:00 GMT

Learned embeddings often suffer from ’embedding collapse’, where they occupy only a small subspace of the available dimensions. This article explores the causes of embedding collapse, from two-tower models to GNN-based systems, and its impact on model scalability and recommendation quality. We discuss methods to detect collapse and examine recent solutions proposed by research teams at Visa, Facebook AI, and Tencent Ads to address this challenge.

Vi, Vim and NeoVim

Bruno Croci — Wed, 06 Nov 2024 02:00:00 GMT

When I first heard of Vi, it was actually Vim. But people liked to call it V-I, so it was common for people to add aliases to call it vi: alias vi=vim was a common line in people’s shell profiles.

After the current NeoVim craze, with lots of influencer creating content on it and tons of new users (myself included), I see the interesting effect of people creating aliases to invoke NeoVim with vim. alias vim=nvim, (myself incldued).

I wonder when the next big editor comes, in a decade or so, let’s call it UltraNeoVim, unvim, people will start invoking it with nvim. alias nvim=unvim.

Exploring Postgres's arena allocator by writing an HTTP server from scratch

Notes on software development — Wed, 06 Nov 2024 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Support me

All content on Seirdy’s Home — Tue, 05 Nov 2024 21:47:21 GMT

Support my work by sending me donations! This helps me continue Fediverse moderation, blogging, and coding.

libobscura

dcz's posts — Tue, 05 Nov 2024 16:00:00 GMT

Announcing libobscura

Let me formally introduce you to my most recent overly-ambitious experiment: libobscura.

Libobscura is a friendly library to use cameras on Linux.

At least that's the goal.

What does "friendly" mean?

It's hard to use it wrong. No segfaults. Errors guide you to the right track.
Point-and-shoot. If that's all you need, you get a RGB buffer in ten lines of code.
It's easy to add support for new devices. Great documentation and a good internal API are the goals.
It's easy to contribute to. Send patches using the web interface, not a mailing list.

TL;DR: with the simple buffer API you can get a frame in 6 calls, and map it to CPU in another 2:

let cameras_list = vidi::actors::camera_list::spawn()?;
let cameras = cameras_list.cameras();
let camera = cameras_list.create(&cameras[0].info.id)
    .expect("No such camera")
    .expect("Failed to create camera");

let mut camera = camera.acquire();
if let Ok(ref mut camera) = camera {
    let mut stream = camera.start(
        Config{fourcc: FourCC::new(b"YUYV")},
        4
    ).unwrap();
    loop {
        let (buf, meta, _next) = stream.next().unwrap();
        let mmap = buf.memory_map_ro().unwrap();
        let data = mmap.as_slice();
    }
}

Go to project examples for more.

Figure 1: Libobscura will never be friendly enough for every audience.

What cameras?

Any webcams, industrial cameras, image sensors working exposed by the V4L2 interface are currently in scope.

What does "experiment" mean?

There are already other libraries for camera support on Linux. You can use the V4L2 APIs directly, or use libcamera, or libmegapixels.

They all strike various middle points on the power vs user-friendliness scale. Having worked with all of them while developing camera support for the Librem 5, I never got the impression that any of them are particularly easy to use.

Libobscura is an experiment because it tries to find an API that fulfills the needs of most people and remains hard to use wrong.

Figure 2: My "Perfect tool" conjecture. Imagine you have a perfectly well useable tool covering some of your needs. If your needs grow, the perfect tool covering those tools cannot be easier to use than your old tool. And humans are imperfect. This applies to APIs, as well. I put my rough idea of where I think a couple of examples fall. Libcamera tries to be as general as possible, so it's on the far right. OpenCV has Python bindings, so it's up top. Libobscura starts out a bit to the left and top, and I hope to push it to the right, to explore how many use cases can be added while staying easy.

Perhaps it's impossible to improve on what current libraries do. But now libobscura is a space where it's possible to try out radical changes without bothering the maintainers of existing libraries – for example, to try out an entirely new approach.

Radical approaches

There are a couple of radical approaches in libobscura that earn it the "experiment" status.

Rust

Rust is a memory-safe systems language. Libobscura uses it to ensure that the APIs are hard to use wrong. A Rust API built with a little care does not let you crash with a segfault – the compiler will alert you before you can create problems.

The Linux kernel started its own Rust experiment in 2020. Linux is a low-level project. Camera support is a low-level topic (many people working on libcamera also work on the kernel). If Rust is interesting for Linux, then it's interesting for libobscura.

Get RGB data

A camera library can't be described as easy to use if the pictures it gives you need to be processed before displaying.

The typical format to display data is RGB while cameras return either Bayer or YUV data. With libobscura, the goal is to provide those conversions transparently, without any extra effort on behalf of the user or camera backend.

GPU acceleration

The conversions might not be implemented in hardware, but don't worry, libobscura has your back! It comes with a GPU-accelerated image processing library called crispy-img.

Why GPU? The idea started with the Librem 5, which will often record video while running on battery. Offloading this work to the GPU is a clear win, and I expect most Linux video-capable devices to have some form of GPU. And we're open to a CPU decoder if you want to contribute!

Figure 3: An artist's impression of a pixel format conversion shader.

Status

Because libobscura is only two months old as a funded project, the current status is "proof of concept". There's a safe and limited user API, another more tricky but zero-copy API, the GPU accelerated library can handle some conversions, and camera support is relatively simple. USB camera demo works.

But there are still goals to achieve:

change controls like brightness or focus,
transparently integrate GPU processing,
have a way to choose your preferred output format,
implement auto-* algorithms when the camera doesn't have auto mode,
replace LGPL-licensed pieces from libvidi and crispycam (not great for link-time-optimized code)
add Pipewire integration, so your browser can just pick up your camera feed for teleconferencing,
add Librem 5 support as a realistic, useful verification of concept.

Those will be the next steps for the project. First implement all the functionality and then extend support for more devices. This way we can catch corner cases in the API that are bound to appear with unusual setups.

The future is YOU

I'm making libobscura in the hope that it will be useful to people. As a base to build software, or to ship devices, or to learn what software architectures fit this problem.

When I sent my funding request to Prototype Fund, I didn't expect to be taken seriously. After all, what motivated me most was that it was a cool challenge. Apparently they believed in me more than I did, because I got funding until March 2025.

What happens until then and what happens next depends on how useful this work actually is to YOU. The ultimate goal of any software is to be useful, otherwise what's the point?

So I invite YOU to analyze it, try it out, give feedback, experiment with it, copy the concepts, contribute to docs, illustrations, and code, fork it entirely! Regardless if you come from the Mobile Linux community, or Raspberry Pi, or you have a laptop with this IPU thing, or if you're from libcamera. When you improve, libobscura fulfills its goals.

So see you on the project page!

Thanks

Thank you to Purism, for inviting me to the Linux camera work and funding the base which I'm using now.

Thank you to all the libcamera people! I stol... was inspired by libcamera's architecture, and you also answered my countless questions both during Librem 5 development and recently.

Thank you Martijn Braam for showing how to create simple config files for many mobile phones.

Thank you BMBF for providing the funding money, and thanks Prototype Fund for connecting me to it :)

IPv6 in your home

dcz's posts — Tue, 05 Nov 2024 16:00:00 GMT

IPv6 in your home

IPv4, IPv6, who cares? As long as it works, right?

The death of IPv4 is greatly exaggerated

We ran out of IPv4 addresses, and so what? I can still watch Youtube videos, have calls over Zoom, buy things on Amazon. The Internet works due to the magic of NAT! Channeling Oprah, you get an address, you also get an address, and you too get the same address!

Except this all works because of lots of money and effort.

Effort because you can't just make a call to your friend over IPv4. You're both likely behind a one-to-many NAT, meaning you can't receive connections. So who will initiate one? STUN offers some painful, costly, partial, and annoying solutions involving having a dedicated address anyway.

An IPv4 address costs money, too. A single IPv4 address costs 26 USD in bulk as of 2024-11-05, and I'm paying for a 3EUR per month for a cheap VPS just for the sake of having one address for myself.

Why bother? Because if you want a little place for yourself on the internet to serve stuff from, you need an address. Manage your own game server? Expose a quirky service? Have an actual, direct call? Seed torrents of your favorite classic art? Connect to your home network while away? I want to do all those things, so I bought myself an address.

Of course, taking an address from the limited pool makes me part of the problem.

But I want to connect to the plant watering system that I left at home! But I want to download stuff from the off-site backup that's at my friend's place, behind NAT! Yes, I could buy a server in a data center and install Wireguard there. I did it, it sucks. High pings, slow transfers. Help! Let me have direct connections.

IPv6

IPv6 comes to the rescue! The address pool is about 420^π bazillions addresses, so every address is really cheap. If you want to apply for a subnet, a random 500 EUR package gives you 2¹²⁸⁻⁴⁸=2⁸⁰ addresses, which is 2417851639229258137600 addresses per EUR yearly.

Global IPv6 addresses are effectively free.

If you have a decent Internet provider, they will assign you a /60 subnet and let all your LAN devices grab a /64. Now anyone can connect to you, so enjoy! And put up your firewall.

That /64 thing

What's that about a /64 subnet? A subnet for every device?

Well, that's how IPv6's DHCP equivalent works. You can't normally get anything smaller than that through autoconfiguration. Which is not a problem if you have a decent Internet provider.

But if you've got only a half-decent provider, they might only offer you a /64. That typically happens on mobile connections, but not only. (Also, this can happen if you like subdividing networks like Russian dolls. IPv4 you could just chain NATs, IPv6 has no NAT.)

So what do you do?

OpenWRT

I don't believe I have to introduce OpenWRT to any of my readers. Newcomers, this is the Open Source operating system for routers.

It can extend IPv6 connections to connected computers even if there's only a /64 available! And it has this nifty Web interface called LuCi.

While there are multiple guides for the command-line, there are no guides for configuring IPv6 forwarding for the Web interface. So here's mine.

IPv6 relay mode in Luci

First, set up an IPv4 WAN network (if you care). I'll call it wwan. Remember to set up a LAN interface if you don't have one. Ready? Then set up a WAN network for IPv6. I'l call it relay6, set it to DHCP client and select "Alias Interface: @wwan" as the device.

Once you have it, navigate to "DHCP server" and set one up.

Yes, I know, it's weird. We don't want to provide addresses on this interface. That's why the next step is checking the "Ignore interface" box.

Once that's done, go to DHCP IPv6 settings.

Make this interface a designated master and change 3 dropdowns (RA-Service, DHCPv6-Service, NDP-Proxy) to "relay mode".

We're done with the WAN interface, but the LAN needs to be adjusted. Here, it's just a static address on a bridge device.

Go to DHCP Server → IPv6 Settings and change all the dropdowns to "relay mode".

Save all the changes and apply them. For me, the router immediately received an address.

My laptop also got an address immediately, but I had to reconnect to get the default route populated (otherwise you can't connect to the Internet).

Check a device connected to that router, it should get the address, too.

Default route

One snag, though.

The default route was not set for my laptop. I had to modify the connection manually and add one. The result looks like this:

[me@foobar ~]$ ip -6 r
default via fe80::abcd:ef12:fecd:6573 dev wlp2s0 proto ra metric 600 pref high

Bonus: ULA

ULA is something that doesn't exist in IPv4.

ULA is the closest counterpart to an IPv4 local address. I use it to have stable addresses within my network. Even if the upstream changes their prefix (effectively your network address) and all previous addresses become invalid at 6:00 every morning, breaking all connections (thanks Telekom), and even if the upstream goes down at all, ULA will keep your network internally connected.

I put the names of all hosts on my network in /etc/hosts, like a troglodyte on the early Internet before DNS.

It sounds complicated if you come from IPv4. There, every computer knows its own address. 192.168.1.77. Great. That's it. What's my public address? No idea.

In IPv6, every computer can easily have multiple addresses. A global one, some ULA addresses. It doesn't become a mess because those addresses are completely independent (link-local ones are a bit special, though). ULA networks are dropped on the edge of the public Internet, so they can't even be used for Internet access.

OpenWRT supports using ULA in Network → Global network options.

For me, it doesn't always get picked up, but it's there on some "Static address" interfaces. I don't really know what controls it, but I have one on the same interface as my LAN:

The end

Now I have IPv6 in my local network! I can seed the world with torrents! I can host a quirky server for 10 minutes (remember to open firewall)! I can connect to the servers I manage at a friend's place!

Oh wait, that server is behind its own firewall and I never enabled IPv6 on that one -_-. I guess I'm not done blogging about IPv6 yet.

Query rewriting - refine user queries

Blog on Maciej Kaszkowiak — Tue, 05 Nov 2024 12:00:00 GMT

What is query rewriting? Why is it useful whenever you build a chatbot? How to implement it? Read the article!

The new rollercoaster king

I Learnt — Tue, 05 Nov 2024 11:18:00 GMT

How do you start to design a new rollercoaster? How do you make it different and more memorable than those before it? And how do you end up doing it when you are only 27?

Quick and easy Python concurrency

Brenton Cleeland — Tue, 05 Nov 2024 09:00:00 GMT

This post serves as a simple reminder to myself that multiprocessing.Pool has map and starmap functions that are a quick and easy way to take a loop and parallelise it in Python.

Imagine some code like this:

urls = ["https://example.org",]

def take_screenshot(url):
    # ...

for url in urls:
    take_screenshot(url)

To make this run across four separate processes, you can do this:

pool = multiprocessing.Pool(4)
pool.map(take_screenshot, urls)

Where does starmap come into it? Well, it lets you invoke the function with multiple arguments by expanding the list you pass in.

urls = [
    ("https://example.org", "example.png")
]

def take_screenshot(url, filename):
    # ...

pool = multiprocessing.Pool(4)
pool.starmap(take_screenshot, urls)

This is super handy for quick scripts, but as always there's trade-offs here. Check out this more detailed post to see other ways of managing concurrency in Python.

4 Ways of Bumping Major Versions in Your Go Project

Jerry Ng — Tue, 05 Nov 2024 02:00:25 GMT

I've recently found myself in a rabbit hole of Go major version bumping to v2. What started as a simple task quickly turned into hours of sifting through conflicting information. Should I use a v2 directory? Create a new v2 branch? What about creating a new repository altogether?

Notes on binary soup

Weblog on marginalia.nu — Tue, 05 Nov 2024 02:00:00 GMT

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before.

This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely, in the hopes that other libraries can draw ideas from it.

Writing Multithreaded Node.js Applications: Is it worth it? at NodeConf EU

Thomas Hunter II — Tue, 05 Nov 2024 02:00:00 GMT

Serverless Invoice Management System with AWS

Prasham H Trivedi — Tue, 05 Nov 2024 02:00:00 GMT

Revolutionized invoice management with a serverless system using AWS Lambda, S3, DynamoDB, and Anthropic API. Reduced a hours-long task to 5 minutes.

What Goes Around Comes Around... And Around...

Prasham H Trivedi — Tue, 05 Nov 2024 02:00:00 GMT

This is an interesting paper I read last month. The notes on this paper gave me an idea of having linkblogs… Original Paper link Some quotes Under 4: Parting comments ORMs are a vital tool for rapid prototyping. But they often sacrifice the ability to push logic into the DBMS in exchange for interoperability with multiple DBMSs. Developers fall back to writing explicit database queries to override the poor auto-generated queries.

The next stage of getlocalcert.net

Built on Shards of Silicon: Robert Alexander's Tech Blog — Tue, 05 Nov 2024 02:00:00 GMT

Keeping the project going

Effective unemployment and social media

Notes on software development — Tue, 05 Nov 2024 02:00:00 GMT

Being unemployed can be incredibly depressing. So much rejection. Everything seems to be out of your control. Everything except for one thing: what you produce.

You might know that repeatedly posting on social media that you are looking for work is ineffective. That it looks (or at least feels) worse each time you say so. But there is at least one major caveat to this.

Every single time you create something and share it publicly is a chance to also reiterate that you are looking for work. And people actually appreciate and value this!

Whether you write a blog post or build some project, you are seen as working on yourself and contributing to the community. Positive things! And it is no problem at all to learn with each new post you write and each new project you publish that you are also looking for work.

Moreover, dynamics of the internet and social media basically require that you be regularly producing something new. Either regularly producing a new version of some existing project or regularly producing new projects (or blog posts) entirely.

What you did a week ago is old news on social media. What will you do next week?

This could itself feel depressing except for that it's probably actually a fairly healthy thing for yourself anyway! It is a motivation to keep your skills sharp as time goes on.

So while you're unemployed and able to muster the motivation, write about things that are interesting to you! Build projects that intrigue you. Leave a little note on every post and project that you are looking for work. And share every post and project on social media.

You'll expose yourself to opportunities and referrals. And even if no post or project "takes off" you will still be working on yourself and contributing back knowledge to the community.

I wrote a short post on some ideas for effective unemployment and social media.https://t.co/jmiJCOe2Nk pic.twitter.com/pK9AySNdHR
— Phil Eaton (@eatonphil) November 5, 2024

SvelteKit Streaming: The Complete Guide

Stanislav Khromov — Mon, 04 Nov 2024 23:03:13 GMT

Streaming in SvelteKit is a powerful feature that allows you to load data progressively. In a nutshell, streaming allows your SvelteKit app to send an initial content response to the browser quickly, while fetching and sending additional data as it becomes available. This can make your app feel more responsive, especially when dealing with slow data sources. While streaming data loads you can easily show loading spinners. What is streaming? In a traditional SvelteKit application without streaming, the server waits until data from all load functions has finished loading before sending the complete response to the browser. This means users […]

The post SvelteKit Streaming: The Complete Guide appeared first on Stanislav Khromov.

Pushed Authorization Requests (PAR) in ASP.NET Core 9

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Mon, 04 Nov 2024 21:17:05 GMT

ASP.NET Core 9 introduces support for Pushed Authorization Requests (PAR) in its OpenIdConnect authentication handler. But what exactly is PAR, and why does it matter? In this post, I’ll explain what PAR is, how it works, how to use it with Duende IdentityServer, and when you should consider using it in your applications. What is […]

The post Pushed Authorization Requests (PAR) in ASP.NET Core 9 appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

Getting started with LiteRT (Tensorflow Lite)

Whacky Labs — Mon, 04 Nov 2024 21:00:00 GMT

So Google recently renamed TensorflowLite to LiteRT. And yes that was a genius move indeed. Because now for the first time in my life I actually want to try TFLite … yea, I mean LiteRT.

In the real world you’d ideally think like a regular ML developer and start with discovering a dataset that then you’d use to train a model. And then as a next step you’d invent a problem that could be solved by your trained model.

But for this experiment we are going to keep things simple and build the hello world of Machine Learning universe, the Dogs vs Cat exercise.

Setup

So coincidentally enough I’ve found a trained model from a flutter project that can tell if a given photo is of a dog or a cat.

Nice! So then following the instructions at the official getting started docs we need to add tensorflowlite as a dependency to our project.

use_frameworks!
# pod 'TensorFlowLiteSwift'
pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['CoreML', 'Metal']

In case you’re wondering why are we using the nightly build and not the stable release. It’s because as of today the latest stable release of TensorFlowLiteC won’t work on iOS simulator but according to the last comment on the issue looks like TensorFlowLiteC is now also shipped as xcframework but only in the nightly releases.

And then while we wait for the pod install to finish we can shamelessly rip sample images of various dogs and cats from the Introduction to TensorFlow Lite udacity course as our test set.

And we are all set!

Building the app

For the UI we just need a simple image view, a label and a button. The button obviously would reveal the answer and then randomly load the next image.

Now for the interesting bit. First we need a PetClassifier that takes in an image and returns a text.

class PetClassifier {

  init?(named: String, labels: [String]) {
    // ...
  }

  
  func labelForImage(_ image: UIImage) -> String? {
    // ...
  }
}

And then in the UI layer we can use our PetClassifier to update the label when tapped on the ‘Evaluate’ button.

class ViewController: UIViewController {

  var classifier: PetClassifier?
  @IBOutlet var answerLabel: UILabel!

  override func viewDidLoad() {
    super.viewDidLoad()
    classifier = PetClassifier(named: "dogvscat", labels: ["Cat", "Dog"])
  }

  @IBAction func handleTap() {
    answerLabel.text = classifier?.labelForImage(selectedImage) ?? "Potato"
  }
}

Finally, loading the model is pretty easy. We just need to instantiate Interpreter with the path to our tflite model.

class PetClassifier {
  let interpreter: Interpreter
  let labels: [String]
    
  init?(named: String, labels: [String]) {
    guard let modelPath = Bundle.main.path(forResource: named, ofType: "tflite") else {
      return nil
    }
    
    do {
      var options = Interpreter.Options()
      options.threadCount = Self.threadCount
      interpreter = try Interpreter(modelPath: modelPath, options: options)
      self.labels = labels
    } catch {
      print(error)
      return nil
    }
  }

  // ...
}

Invoking the model

To get the answer from the model is a 4 step process:

Prepare input
Send input data
Read output data
Parse output

/*
 * 1. Prepare input
 */
// user provided image
let image: UIImage 
// image size used for training model
let inputWidth = 224
let inputHeight = 224

// convert image to pixel buffer for further manipulation
let pixelBuffer = ImageUtils.pixelBufferCreate(image: image)
// crop image to size used for training model
let scaledPixelBuffer = ImageUtils.pixelBufferCreateWith(
  pixelBuffer: pixelBuffer,
  resizedTo: CGSize(width: Self.inputWidth, height: Self.inputHeight)
)
// Remove the alpha component from the image buffer to get the RGB data.
let rgbData = ImageUtils.pixelBufferCreateRGBData(
    pixelBuffer: scaledPixelBuffer,
    byteCount: Self.inputWidth * Self.inputHeight * 3
)

/*
 * 2. Send input data 
 */
interpreter.allocateTensors()      
interpreter.copy(rgbData, toInputAt: 0)
interpreter.invoke()

/*
 * 3. Read output data
 */
let outputTensor = try interpreter.output(at: 0)
let results: [Float] = outputTensor.data.withUnsafeBytes {
  Array($0.bindMemory(to: Float.self))
}

/*
 * 4. Parse output
 */      
// Create a zipped array of tuples [(labelIndex: Int, confidence: Float)].
// Sort the zipped results by confidence value
let inferences = zip(labels.indices, results)
  .sorted { $0.1 > $1.1 }
  .map { (label: labels[$0.0], confidence: $0.1) }
      
let bestInference = inferences.first

And there you have it. That is how offload your brain to your computer.

The ImageUtils from this experiment are available here. But there are probably better libraries for these operations. For example, the CoreMLHelpers

References

Practical Deep Learning, Lesson 4, Language Model Blog Post Imitator

Thought Eddies — Mon, 04 Nov 2024 19:57:00 GMT

Practical Deep Learning, Lesson 4, Language Model Blog Post Imitator

Hurdling hierarchy

I Learnt — Mon, 04 Nov 2024 16:59:00 GMT

Often people are afraid to reach out to those above them in the hierarchy or in a different team, even if it is the best solution to their situation.

Heads or Tails

andersource — Mon, 04 Nov 2024 16:30:00 GMT

Conceptual introduction to classical machine learning in JS

Recursive Query in SQL: What It Is, and How to Write One

Dylan Paulus' Blog — Mon, 04 Nov 2024 02:00:00 GMT

As developers, querying in PostgreSQL for hierarchical data is difficult. SQL is a declarative programming language, but our brains are trained to think imperatively. Recursive queries using Common Table Expressions (CTE) simplify writing iterative queries and are essential in traversing hierarchical data, like tree or graph structures.

Throughout this article, we'll explore SQL recursive queries with examples, look at optimizing recursive queries, and discuss advanced techniques.

Why I love Rust for tokenising and parsing

xnacly - blog — Mon, 04 Nov 2024 02:00:00 GMT

Macros, iterators, patterns, error handling and match make Rust almost perfect

Parsing arguments in Rust with no dependencies

ntietz.com blog - technically a blog — Mon, 04 Nov 2024 02:00:00 GMT

When pairing with my friend Emily, we had a choice of what to implement in her project: start a new feature, or add a command line argument parser? We opted for the latter, because it had to happen eventually and it was more well bounded. It ended up having a lot of depth!

We wrote it from scratch to learn more, rather than pulling in a library^[1]. What we found was a nice level of depth in a well-bounded project. I came away wanting to repeat the exercise in Rust.

Opportunity arrives on a yak

As with many of my best procrastination projects, the chance to do this again in Rust came in the form of a yak shave. I've started writing a little poetry, and to get practice, I wanted to write a small utility to generate prompts^[2]. Naturally, this has to be configurable because my little utility will sweep the world (spoiler alert: no it won't). That means we need command-line arguments!

I could pull in clap, which I think is the best fully-featured choice available today. That would add 23 dependencies to my little project, if you count transitive dependencies. This can go up higher if you turn on a few features: derive, env, unicode, and wrap_help bring you up to 38 dependencies!

Instead, I thought it would be fun to implement it myself again, and do it in the same mode as our C++ adventure: parse command-line arguments with no external dependencies! This will have the side benefit of keeping my project's dependencies much lighter, which should (I think) keep compile times lower and make the entire system easier to understand front-to-back. If I do this, I may as well publish it and make it usable for everyone!

The basic design

The design of this parser is pretty straightforward. We have two types of arguments: positional or named.

We'll handle positional arguments as little more than a list of the leftover strings that we don't need for named options. These don't get any special handling, and there's not a lot of parsing to do (unless you do fancy things like path and variable expansion).

Named options (and their respective arguments) are where we have more work to do. We want this to feel familiar, like most unix/linux/etc. command-line interfaces we've used. So we'll want the usual short and long options. For example, specifying a port might be able to use the short option -p or the long option --port-number. And we'll also want to be able to specify some help text, and whether or not the argument is required or optional.

Where things might be a little bit unfamiliar is with the action. We have to have a way of specifying whether something is a flag, a single value, or a list of values. Libraries like clap have more comprehensive options for actions, but this should be sufficient for us here.

And then to do our parsing, we can think of it like a state machine. You iterate through the provided arguments until you find one that could be the start of option (starts with a hyphen). Then you find the matching option and switch into a state based on what you expect to come next. After handling its argument (if there is one), you go back to the default state and look for the next option.

So now we just have to build that.

Implementing it

The data structures we use here match what was described previously in perhaps the most straightforward way. We have a struct to represent each of our options (here using Opt since Option is a highly useful existing struct in the standard library; we can never escape names being hard and ambiguous). For ergonomics, we'll also define methods to set each of these easily and return self; I have one example here, but the rest are the same idea. We also use a name method as the constructor, since all named arguments must have names.

#[derive(Clone, Debug, PartialEq)]
pub struct Opt {
    pub name: String,
    pub short: Option<char>,
    pub long: Option<String>,
    pub help: Option<String>,
    pub default: Option<String>,
    pub action: Action,
    pub required: bool,
}

impl Opt {


    pub fn short(mut self, short: char) -> Opt {
        self.short = Some(short);
        self
    }

    // same idea for the other fields
}

And we have a couple of enums for some values we need^[3]. The first enum is for our actions. These tell the parser what to do when it encounters an option while parsing. Set and Append will slurp up the next argument as the value and store it or insert it. SetTrue and SetFalse will each set the boolean for their respective values. Note that each of the set flags implies that the opposite is the default value (and we'll implement it that way).

#[derive(Clone, Debug, PartialEq)]
pub enum Action {
    Set,
    Append,
    SetTrue,
    SetFalse,
}

Then we have a what we end up storing for the values we've parsed. I opted to keep this simple and not introduce any extra types here. I think this is a reasonable tradeoff, since it keeps the code simple and it remains flexible, but the cost is that this means checking if an incorrect value is passed in (say, "fiddlesticks" for something that expects an integer) is pushed off as the user's responsibility, making the library harder to use.

Anyway, we have three choices: a single value, a list of values, or a true/false value.

#[derive(Clone, Debug, PartialEq)]
pub enum Value {
    Single(String),
    Multi(Vec<String>),
    Flag(bool),
}

And we have one struct to put it all together and define our parsing method on. This one is truly spartan.

#[derive(Debug, PartialEq)]
pub struct Opts {
    opts: Vec<Opt>,
}

And we have our resulting matches, after we've parsed things. Here we'll store one thing we haven't talked about yet, the name of the executable (the first argument passed in), and then the positional and named arguments.

#[derive(Debug, PartialEq)]
pub struct Matches {
    exec_name: String,
    positional: Vec<String>,
    named: HashMap<String, Value>,
}

Now that we have the data structures, we can implement the parsing itself.

We'll start by making a parse method on Opts. This will take in our arguments (from something like std::env::args) and return our matches, or an error.

pub fn parse(&self, args: Vec<String>) -> Result<Matches, ParseError> {
    todo!()
}

We can simplify our lives a bit by setting all the default values at the beginning, letting them get overwritten later if another value is provided. This is not the most efficient approach! But it is going to be a rounding error for most programs, and we could improve the implementation later to run it last and only fill the arguments which aren't provided (or to fill them when we request them). Let's just do this for now, though.

fn populate_defaults(&self, named: &mut HashMap<String, Value>) {
    for opt in self.opts.iter() {
        if let Some(default) = &opt.default {
            named.insert(opt.name.clone(), Value::Single(default.to_owned()));
        } else {
            match opt.action {
                Action::Append => {
                    named.insert(opt.name.clone(), Value::Multi(vec![]));
                }
                Action::SetTrue => {
                    named.insert(opt.name.clone(), Value::Flag(false));
                }
                Action::SetFalse => {
                    named.insert(opt.name.clone(), Value::Flag(false));
                }
                _ => {}
            }
        }
    }
}

And then we can make a helper function to find the named option which matches a given option, if any, and returns an error if an unexpected option is provided.

fn find_opt(&self, arg: &str) -> Result<&Opt, ParseError> {
    let opt = if arg.starts_with("--") {
        let long = arg.strip_prefix("--").unwrap();
        self.opts.iter().find(|o| o.long.as_deref() == Some(long))
    } else if arg.starts_with("-") {
        if arg.chars().count() != 2 {
            return Err(ParseError::MalformedOption(arg.to_string()));
        }
        let short = arg.chars().nth(1);
        self.opts.iter().find(|o| o.short == short)
    } else {
        return Err(ParseError::UnexpectedOption(arg.to_string()));
    };

    if let Some(opt) = opt {
        Ok(opt)
    } else {
        Err(ParseError::UnexpectedOption(arg.to_string()))
    }
}

And now we can come back and define our parse function. It starts out with converting the args into an iterator (so we don't have to clone each string, we take ownership of them all). If there isn't a first one, we know the whole thing has gone wrong, so we abort there.

// pub fn parse(&self, args: Vec<String>) -> Result<Matches, ParseError> {

    let mut args_iter = args.into_iter();
    let exec_name = match args_iter.next() {
        Some(s) => s,
        None => return Err(ParseError::MissingProgramName),
    };

Next we setup our storage and populate the defaults with our helper function.

// pub fn parse(&self, args: Vec<String>) -> Result<Matches, ParseError> {

    let mut positional = vec![];
    let mut named = HashMap::new();

    self.populate_defaults(&mut named);

And then we do our main loop. Inside the main parse loop, we check if any start with a dash (-): if so, we handle it as a named named option, otherwise we handle it as positional (just push it into a vec). For each named option we have to find the right Opt, then we do the action: set the value, append it into a list, or set a flag. There's a little error handling which makes it all somewhat longer, but overall the logic is simple.

// pub fn parse(&self, args: Vec<String>) -> Result<Matches, ParseError> {

    while let Some(arg) = args_iter.next() {
        if arg.starts_with("-") {
            let opt = self.find_opt(&arg)?;

            match opt.action {
                Action::Set => {
                    if let Some(value) = args_iter.next() {
                        named.insert(opt.name.clone(), Value::Single(value));
                    } else {
                        return Err(ParseError::MissingValue(opt.name.clone()));
                    }
                }
                Action::Append => {
                    match (args_iter.next(), named.get_mut(&opt.name)) {
                        (None, _) => return Err(ParseError::MissingValue(opt.name.clone())),
                        (Some(val), Some(Value::Multi(vals))) => {
                            vals.push(val);
                        }
                        (Some(val), None) => {
                            named.insert(opt.name.clone(), Value::Multi(vec![val]));
                        }
                        _ => return Err(ParseError::BadInternalState), // unexpected case
                    };
                }
                Action::SetTrue => {
                    named.insert(opt.name.clone(), Value::Flag(true));
                }
                Action::SetFalse => {
                    named.insert(opt.name.clone(), Value::Flag(false));
                }
            };
        } else {
            positional.push(arg);
        }
    }

    Ok(Matches::new(exec_name, positional, named))
}

And there we have it, a command-line options parser with no external dependencies!

There are a few things left that aren't done:

No help or versions print
There's not a lot of help for validation
Things aren't exported in the root module, so using it is clunky

I'll fix these as I use it, but if anyone actually uses this and wants to contribute or maintain it, let me know. For now, the code is in a git repo and licensed under MIT and Apache licenses. This is a deviation from my usual use of the Gay Agenda License because this is a project that I suspect could have some small real utility. Anyway, be gay do crime.

This "no-dependencies" thing

Okay, but why am I yammering on about no dependencies so much? Because there are some fundamentally nice things about not adding more dependencies to your project, and sometimes we should reinvent things.

One hypothesis is that with no dependencies, the compile times will be faster. This isn't broadly true (splitting a crate into subcrates can improve compile times, and this is a common practice), but I think here it will improve compile times because this keeps it smaller and leaner.

More concretely, by having no external dependencies you reduce your bug surface area. Sure, you own all the bugs now—but you won't get leftpad-ed, and you won't get dependabot alerts for third-removed transitive dependencies that now you've gotta patch.

On the other hand, you miss out on nice things. Here in particular, I'll be missing partial unicode support! You could put whatever you want in the strings, but once I implement help text, I'll probably have some form of text wrapping. And if you do that, you have to know how wide characters are when displayed. Not all of them are the same as one monospace Latin alphabet character (such as emoji and some languages), and it depends on your font as well I think? We can do a best-effort job here of splitting based on some assumptions (and allow manual line splitting). But the best idea would probably be to optionally add a dependency as a Cargo feature so that you only get the dependency if you need it.

I think more things should be built from scratch and, ideally, without dependencies. You get to know the problem space better, and most things don't need the big sophisticated solution—but you pay for the whole dependency you pull in. It's also nice because if you have no dependencies, that means folks can depend on you without adding any transitive dependencies, and this is really a big deal. I'd love to have a set of dependencies to use that all have either no dependencies, or themselves have only 0-dep dependencies, so your number of transitive dependencies is capped. Then you could add in a lot of "fit for purpose" simple 0-dep things, and pick up the complex ones for where you really need it while still keeping your Cargo.lock slimmer than today.

Besides, her project is in C++. I don't know it well anymore, and wasn't eager to learn its package manger ecosystem. For all the justification of reasons to build it ourselves, "it's more fun" and "we learn more" were the big two. ↩
This is probably not something I'll finish due to life and health. But it did spawn this nice yak shave. ↩
I could've also made an enum for whether or not it's required, but this feels like one of the rare cases where a boolean is already clear and there's no risk that we'll extend it to a third choice. ↩

Is email confidential in transit yet?

Built on Shards of Silicon: Robert Alexander's Tech Blog — Mon, 04 Nov 2024 02:00:00 GMT

Measuring vulnerable SMTP configurations and defenses

GraphQL collection lookups

Sophia Willows' Blog — Mon, 04 Nov 2024 02:00:00 GMT

One of the powerful design patterns that can be employed within a GraphQL schema is the concept of collection lookups. This pattern allows API consumers to retrieve specific elements from a collection using singular fields, enhancing both usability and performance.

To demonstrate this pattern, let’s imagine we’re building a blogging platform (perhaps for sophiabits.com!). The core data type we’ll focus on is a Post which—among other things—shall contain a list of arbitrary key-value pairs and a list of images. Attributes are used by bloggers to store little tidbits of information that could be consumed by a plugin system.

Below is a potential schema. Note the paired attribute/attributes and image/images fields:

Read more on sophiabits.com

You probably shouldn't hire a Developer Advocate yet

Julien's DevRel corner — Sun, 03 Nov 2024 19:46:31 GMT

Many companies with job ads out for Developer Advocates don't know that's not what they should be hiring for (yet). More often than not, what they need first is some internal mindset change. And even then, a Developer Experience will go a long way to achieve the first stages of a healthy community.

The ultimate in debugging

I Learnt — Sun, 03 Nov 2024 19:29:00 GMT

Engineers are currently debugging why the Voyager 1 spacecraft, which is 15 billions miles away, turned off its main radio and switched to a backup radio that hasn’t been used in over forty years!

An unknown pioneer of computing

I Learnt — Sun, 03 Nov 2024 18:32:00 GMT

Ed Roberts is a forgotten person in the history of home computing but played a huge role in getting us to where we are today.

Eric (Netflix Limited Series): Inner conflicts in New York's Landscape

🦉 olshansky 🦁 — Sun, 03 Nov 2024 14:16:57 GMT

Olshansky's review of Eric: Limited Series

Helping to solve male loneliness

I Learnt — Sun, 03 Nov 2024 11:00:00 GMT

It is very easy to go through life and not realise the effects of loneliness.

1% Better Every Day

Sunil Shenoy — Sun, 03 Nov 2024 00:16:31 GMT

I remember seeing this message on one of my gym friends' t shirt a few months ago. It’s not the first time I saw this message—I had seen it countless times before—but this was the first time it stayed with me.

I was listening to “The One Thing” podcast while driving around Sydney last week, and in one of the episodes, they spoke about how, when they got started with their project, they decided to spend one hour working on the thing they were passionate about. An hour a day dedicated to things they wanted to see in the world.

“I could dedicate an hour a day to work on something I want to see in the world,” I told myself. I have had a long list of projects that I’ve been wanting to work on, but between a full-time job, gym, social life, and reading, I never took time to work on the projects I wanted to see in the world. The podcast was a great reminder to start. Getting older was a great reminder to start.

The last two weeks have been exciting. I wake up with plans for that one hour. There is something to look forward to every day. There is no pressure to plan that one hour, but I have managed to work on one of the projects I’ve had in mind for a few years now. The goal is to launch it before my birthday. Having a deadline to ship has been great. Constraints are good. I have been pushing code out every day, but I am not yet ready to share the URL with everyone.

Is there something you have been wanting to get started with?

How I fixed my Nothing Ear muffled/underwater audio on Android (hint: it's ChatGPT)

Liza Shulyayeva — Sat, 02 Nov 2024 23:01:33 GMT

My new Nothing Ear earphones had been working great, but after a few weeks I noticed that sometimes they sounded like they were underwater. The audio would be extremely quiet and muffled. No matter how I adjusted any of the EQ settings, noise cancellation, in-ear detection, codec choice, or anything else in the Nothing app, the problem persisted. A full reset of the earphones didn’t help. I reached out to Nothing support and heard crickets to this day and decided I’d give it a week or two more before returning them.

What if you hadn't?

Martin Rue — Sat, 02 Nov 2024 09:00:00 GMT

"Had I not..." is a calling. It's a manifesto of living in every direction at the same time, driven by curiosity and a desire to learn and experience as many things as possible.

A fascinating attempt at Steam phishing

utf9k — Sat, 02 Nov 2024 06:00:00 GMT

I enjoyed almost being tricked but really shouldn't

Models Writing About Coding With Models

Thought Eddies — Sat, 02 Nov 2024 03:26:30 GMT

Models Writing About Coding With Models

Breaking CityHash64, MurmurHash2/3, wyhash, and more...

orlp.net - Blog Archive — Sat, 02 Nov 2024 02:00:00 GMT

Hash functions are incredibly neat mathematical objects. They can map arbitrary data to a small fixed-size output domain such that the mapping is deterministic, yet appears to be random. This “deterministic randomness” is incredibly useful for a variety of purposes, such as hash tables, checksums, monte carlo algorithms, communication-less distributed algorithms, etc, the list goes on.

In this article we will take a look at the dark side of hash functions: when things go wrong. Luckily this essentially never happens due to unlucky inputs in the wild (for good hash functions, at least). However, people exist, and some of them may be malicious. Thus we must look towards computer security for answers. I will quickly explain some of the basics of hash function security and then show how easy it is to break this security for some commonly used non-cryptographic hash functions.

As a teaser, this article explains how you can generate strings such as these, thousands per second:

 cityhash64("orlp-cityhash64-D-:K5yx*zkgaaaaa") == 1337
murmurhash2("orlp-murmurhash64-bkiaaa&JInaNcZ") == 1337
murmurhash3("orlp-murmurhash3_x86_32-haaaPa*+") == 1337
 farmhash64("orlp-farmhash64-/v^CqdPvziuheaaa") == 1337

I also show how you can create some really funky pairs of strings that can be concatenated arbitrarily such that when concatenating $k$ strings together any of the $2^k$ combinations all have the same hash output, regardless of the seed used for the hash function:

a = "xx0rlpx!xxsXъВ"
b = "xxsXъВxx0rlpx!"
murmurhash2(a + a, seed) == murmurhash2(a + b, seed)
murmurhash2(a + a, seed) == murmurhash2(b + a, seed)
murmurhash2(a + a, seed) == murmurhash2(b + b, seed)

a = "!&orlpՓ"
b = "yǏglp$X"
murmurhash3(a + a, seed) == murmurhash3(a + b, seed)
murmurhash3(a + a, seed) == murmurhash3(b + a, seed)
murmurhash3(a + a, seed) == murmurhash3(b + b, seed)

Hash function security basics

Hash functions play a critical role in computer security. Hash functions are used not only to verify messages over secure channels, they are also used to identify trusted updates as well as known viruses. Virtually every signature scheme ever used starts with a hash function.

If a hash function does not behave randomly, we can break the above security constructs. Cryptographic hash functions thus take the randomness aspect very seriously. The ideal hash function would choose an output completely at random for each input, remembering that choice for future calls. This is called a random oracle.

The problem is that a random oracle requires a true random number generator, and more problematically, a globally accessible infinite memory bank. So we approximate it using deterministic hash functions instead. These compute their output by essentially shuffling their input really, really well, in such a way that it is not feasible to reverse.

To help quantify whether a specific function does a good job of approximating a random oracle, cryptographers came up with a variety of properties that a random oracle would have. The three most important and well-known properties a secure cryptographic hash function should satisfy are:

Pre-image resistance. For some constant $c$ it should be hard to find some input $m$ such that $h(m) = c$.
Second pre-image resistance. For some input $m_1$ it should be hard to find another input $m_2$ such that $h(m_1) = h(m_2)$.
Collision resistance. It should be hard to find inputs $m_1, m_2$ such that $h(m_1) = h(m_2)$.

We generally consider one of these properties broken if there exists a method that produces a collision or pre-image faster than simply trying random inputs (also known as a brute force attack). However, there are definitely gradations in breakage, as some methods are only several orders of magnitude faster than brute force. That may sound like a lot, but a method taking $2^{110}$ steps instead of $2^{128}$ are still both equally out of reach for today’s computers.

MD5 used to be a common hash function, and SHA-1 is still in common use today. While both were considered cryptographically secure at one point, generating MD5 collisions now takes less than a second on a modern PC. In 2017 a collaboration of researchers from CWI and Google and announced the first SHA-1 collision. However, as far as I’m aware, neither MD5 nor SHA-1 have practical (second) pre-image attacks, only theoretical ones.

Non-cryptographic hash functions

Cryptographically secure hash functions tend to have a small problem: they’re slow. Modern hash functions such as BLAKE3 resolve this somewhat by heavily vectorizing the hash using SIMD instructions, as well as parallelizing over multiple threads, but even then they require large input sizes before reaching those speeds.

A lot of problems don’t necessarily require secure hash functions, and people would much prefer a faster hash speed. Especially when we are computing many small hashes, such as in a hash table. Let’s take a look what common hash table implementations actually use as their hash for strings:

C++: there are multiple standard library implementations, but 64-bit clang 13.0.0 on Apple M1 ships CityHash64. Currently libstdc++ ships MurmurHash64A, a variant of Murmur2 for 64-bit platforms.
Java: OpenJDK uses an incredibly simple hash algorithm, which essentially just computes h = 31 * h + c for each character c.
PHP: the Zend engine uses essentially the same algorithm as Java, just using unsigned integers and 33 as its multiplier.
Nim: it used to use MurmurHash3_x86_32. While writing this article they appeared to have switched to use farmhash by default.
Zig: it uses wyhash by default, with 0 as seed.
Javascript: in V8 they use a custom weak string hash, with a randomly initialized seed.

There were some that used stronger hashes by default as well:

Go uses an AES-based hash if hardware acceleration is available on x86-64. Even though its construction is custom and likely not full-strength cryptographically secure, breaking it is too much effort and quite possibly beyond my capabilities.

If not available, it uses an algorithm inspired by wyhash.
Python and Rust use SipHash by default, which is a cryptographically secure pseudorandom function. This is effectively a hash function where you’re allowed to use a secret key during hashing, unlike a hash like SHA-2 where everyone knows all information involved.

This latter concept is actually really important, at least for protecting against HashDoS in hash tables. Even if a hash function is perfectly secure over its complete output, hash tables further reduce the output to only a couple bits to find the data it is looking for. For a static hash function without any randomness it’s possible to produce large lists of hashes that collide post-reduction, just by brute force. But for non-cryptographic hashes as we’ll see here we often don’t need brute force and can generate collisions at high speed for the full output, if not randomized by a random seed.

Interlude: inverse operations

Before we get to breaking some of the above hash functions, I must explain a basic technique I will use a lot: the inverting of operations. We are first exposed to this in primary school, where we might get faced by a question such as “$2 + x = 10$”. There we learn subtraction is the inverse of addition, such that we may find $x$ by computing $10 - 2 = 8$.

Most operations on the integer registers in computers are also invertible, despite the integers being reduced modulo $2^{w}$ in the case of overflow. Let us study some:

Addition can be inverted using subtraction. That is, x += y can be inverted using x -= y. Seems obvious enough.
Multiplication by a constant $c$ is not inverted by division. This would not work in the case of overflow. Instead, we calculate the modular multiplicative inverse of $c$. This is an integer $c^{-1}$ such that $c \cdot c^{-1} \equiv 1 \pmod {m}$. Then we invert multiplication by $c$ simply by multiplying by $c^{-1}$.

This constant exists if and only if $c$ is coprime with our modulus $m$, which for us means that $c$ must be odd as $m = 2^n$. For example, multiplication by $2$ is not invertible, which is easy to see as such, as it is equivalent to a bit shift to the left by one position, losing the most significant bit forever.

Without delving into the details, here is a snippet of Python code that computes the modular multiplicative inverse of an integer using the extended Euclidean algorithm by calculating $x, y$ such that $$cx + my = \gcd(c, m).$$ Then, because $c$ is coprime we find $\gcd(c, m) = 1$, which means that $$cx + 0 \equiv 1 \pmod m,$$ and thus $x = c^{-1}$.
```
def egcd(a, b):
    if a == 0: return (b, 0, 1)
    g, y, x = egcd(b % a, a)
    return (g, x - (b // a) * y, y)

def modinv(c, m):
    g, x, y = egcd(c, m)
    assert g == 1, "c, m must be coprime"
    return x % m
```
Using this we can invert modular multiplication:
```
>>> modinv(17, 2**32)
4042322161
>>> 42 * 17 * 4042322161 % 2**32
42
```
Magic!
XOR can be inverted using… XOR. It is its own inverse. So x ^= y can be inverted using x ^= y.
Bit shifts can not be inverted, but two common operations in hash functions that use bit shifts can be. The first is bit rotation by a constant. This is best explained visually, for example a bit rotation to the left by 3 places on a 8-bit word, where each bit is shown as a letter:
```
abcdefghi
defghiabc
```
The formula for a right-rotation of k places is (x >> k) | (x << (w - k)), where w is the width of the integer type. Its inverse is a left-rotation, which simply swaps the direction of both shifts. Alternatively, the inverse of a right-rotation of k places is another right-rotation of w-k places.
Another common operation in hash functions is the “xorshift”. It is an operation of one of the following forms, with $k > 0$:
```
x ^= x << k  // Left xorshift.
x ^= x >> k  // Right xorshift.
```
How to invert it is entirely analogous between the two, so I will focus on the left xorshift.

An important observation is that the least significant $k$ bits are left entirely untouched by the xorshift. Thus by repeating the operation, we recover the least significant $2k$ bits, as the XOR will invert itself for the next $k$ bits. Let’s take a look at the resulting value to see how we should proceed:
```
v0 = (x << k) ^ x
// Apply first step of inverse v1 = v0 ^ (v0 << k).
v1 = (x << 2*k) ^ (x << k) ^ (x << k) ^ x
// Simplify using self-inverse (x << k) ^ (x << k) = 0.
v1 = (x << 2*k) ^ x
```
From this we can conclude the following identity: $$\operatorname{xorshift}(\operatorname{xorshift}(x, k), k) = \operatorname{xorshift}(x, 2k)$$ Now we only need one more observation to complete our algorithm: a xorshift of $k \geq w$ where $w$ is the width of our integer is a no-op. Thus we repeatedly apply our doubling identity until we reach large enough $q$ such that $\operatorname{xorshift}(x, 2^q \cdot k) = x$.

For example, to invert a left xorshift by 13 for 64-bit integers we apply the following sequence:
```
x ^= x << 13  // Left xorshift by 13.

x ^= x << 13  // Inverse step 1.
x ^= x << 26  // Inverse step 2.
x ^= x << 52  // Inverse step 3.
// x ^= x << 104  // Next step would be a no-op.
```

Armed with this knowledge, we can now attack.

Breaking `CityHash64`

Let us take a look at (part of) the source code of CityHash64 from libcxx that’s used for hashing strings on 64-bit platforms:

static const uint64_t mul = 0x9ddfea08eb382d69ULL;
static const uint64_t k0 = 0xc3a5c85c97cb3127ULL;
static const uint64_t k1 = 0xb492b66fbe98f273ULL;
static const uint64_t k2 = 0x9ae16a3b2f90404fULL;
static const uint64_t k3 = 0xc949d7c7509e6557ULL;

template<class T>
T loadword(const void* p) {
    T r;
    std::memcpy(&r, p, sizeof(r));
    return r;
}

uint64_t rotate(uint64_t val, int shift) {
    if (shift == 0) return val;
    return (val >> shift) | (val << (64 - shift));
}

uint64_t hash_len_16(uint64_t u, uint64_t v) {
    uint64_t x = u ^ v;
    x *= mul;
    x ^= x >> 47;
    uint64_t y = v ^ x;
    y *= mul;
    y ^= y >> 47;
    y *= mul;
    return y;
}

uint64_t hash_len_17_to_32(const char *s, uint64_t len) {
    const uint64_t a = loadword<uint64_t>(s) * k1;
    const uint64_t b = loadword<uint64_t>(s + 8);
    const uint64_t c = loadword<uint64_t>(s + len - 8) * k2;
    const uint64_t d = loadword<uint64_t>(s + len - 16) * k0;
    return hash_len_16(
        rotate(a - b, 43) + rotate(c, 30) + d,
        a + rotate(b ^ k3, 20) - c + len
    );
}

To break this, let’s assume we’ll always give length 32 inputs. Then the implementation will always call hash_len_17_to_32, and we have full control over variables a, b, c and d by changing our input.

Note that d is only used once, in the final expression. This makes it a prime target for attacking the hash. We will choose a, b and c arbitrarily, and then solve for d to compute a desired hash outcome.

Using the above modinv function we first compute the necessary modular multiplicative inverses of mul and k0:

>>> 0x9ddfea08eb382d69 * 0xdc56e6f5090b32d9 % 2**64
1
>>> 0xc3a5c85c97cb3127 * 0x81bc9c5aa9c72e97 % 2**64
1

We also note that in this case the xorshift is easy to invert, as x ^= x >> 47 is simply its own inverse. Having all the components ready, we can invert the function step by step.

We first load a, b and c like in the hash function, and compute

uint64_t v = a + rotate(b ^ k3, 20) - c + len;

which is the second parameter to hash_len_16. Then, starting from our desired return value of hash_len_16(u, v) we work backwards step by step, inverting each operation to find the function argument u that would result in our target hash. Then once we have found such the unique u we compute our required input d. Putting it all together:

static const uint64_t mul_inv = 0xdc56e6f5090b32d9ULL;
static const uint64_t k0_inv  = 0x81bc9c5aa9c72e97ULL;

void cityhash64_preimage32(uint64_t hash, char *s) {
    const uint64_t len = 32;
    const uint64_t a = loadword<uint64_t>(s) * k1;
    const uint64_t b = loadword<uint64_t>(s + 8);
    const uint64_t c = loadword<uint64_t>(s + len - 8) * k2;
    
    uint64_t v = a + rotate(b ^ k3, 20) - c + len;
    
    // Invert hash_len_16(u, v). Original operation inverted
    // at each step is shown on the right, note that it is in
    // the inverse order of hash_len_16.
    uint64_t y = hash;    // return y;
    y *= mul_inv;         // y *= mul;
    y ^= y >> 47;         // y ^= y >> 47;
    y *= mul_inv;         // y *= mul;
    uint64_t x = y ^ v;   // uint64_t y = v ^ x;
    x ^= x >> 47;         // x ^= x >> 47;
    x *= mul_inv;         // x *= mul;
    uint64_t u = x ^ v;   // uint64_t x = u ^ v;
    
    // Find loadword<uint64_t>(s + len - 16).
    uint64_t d = u - rotate(a - b, 43) - rotate(c, 30);
    d *= k0_inv;
    std::memcpy(s + len - 16, &d, sizeof(d));
}

The chance that a random uint64_t forms 8 printable ASCII bytes is $\left(94/256\right)^8 \approx 0.033%$. Not great, but cityhash64_preimage32 is so fast that having to repeat it on average ~3000 times to get a purely ASCII result isn’t so bad.

For example, the following 10 strings all hash to 1337 using CityHash64, generated using this code:

orlp-cityhash64-D-:K5yx*zkgaaaaa
orlp-cityhash64-TXb7;1j&btkaaaaa
orlp-cityhash64-+/LM$0 ;msnaaaaa
orlp-cityhash64-u'f&>I'~mtnaaaaa
orlp-cityhash64-pEEv.LyGcnpaaaaa
orlp-cityhash64-v~~bm@,Vahtaaaaa
orlp-cityhash64-RxHr_&~{miuaaaaa
orlp-cityhash64-is_$34#>uavaaaaa
orlp-cityhash64-$*~l\{S!zoyaaaaa
orlp-cityhash64-W@^5|3^:gtcbaaaa

Breaking MurmurHash2

We can’t let libstdc++ get away after targetting libc++, can we? The default string hash calls an implementation of MurmurHash2 with seed 0xc70f6907. The hash—simplified to only handle strings whose lengths are multiples of 8—is as follows:

uint64_t murmurhash64a(const char* s, size_t len, uint64_t seed) {
    const uint64_t mul = 0xc6a4a7935bd1e995ULL;

    uint64_t hash = seed ^ (len * mul);
    for (const char* p = s; p != s + len; p += 8) {
        uint64_t data = loadword<uint64_t>(p);
        data *= mul;
        data ^= data >> 47;
        data *= mul;
        hash ^= data;
        hash *= mul;
    }

    hash ^= hash >> 47;
    hash *= mul;
    hash ^= hash >> 47;
    return hash;
}

We can take a similar approach here as before. We note that the modular multiplicative inverse of 0xc6a4a7935bd1e995 mod $2^{64}$ is 0x5f7a0ea7e59b19bd. As an example, we can choose the first 24 bytes arbitrarily, and solve for the last 8 bytes:

void murmurhash64a_preimage32(uint64_t hash, char* s, uint64_t seed) {
    const uint64_t mul = 0xc6a4a7935bd1e995ULL;
    const uint64_t mulinv = 0x5f7a0ea7e59b19bdULL;
    
    // Compute the hash state for the first 24 bytes as normal.
    uint64_t state = seed ^ (32 * mul);
    for (const char* p = s; p != s + 24; p += 8) {
        uint64_t data = loadword<uint64_t>(p);
        data *= mul;
        data ^= data >> 47;
        data *= mul;
        state ^= data;
        state *= mul;
    }
    
    // Invert target hash transformation.
                        // return hash;
    hash ^= hash >> 47; // hash ^= hash >> 47;
    hash *= mulinv;     // hash *= mul;
    hash ^= hash >> 47; // hash ^= hash >> 47;

    // Invert last iteration for last 8 bytes.
    hash *= mulinv;                // hash *= mul;
    uint64_t data = state ^ hash;  // hash = hash ^ data;
    data *= mulinv;                // data *= mul;
    data ^= data >> 47;            // data ^= data >> 47;
    data *= mulinv;                // data *= mul;
    std::memcpy(s + 24, &data, 8); // data = loadword<uint64_t>(s);
}

The following 10 strings all hash to 1337 using MurmurHash64A with the default seed 0xc70f6907, generated using this code:

orlp-murmurhash64-bhbaaat;SXtgVa
orlp-murmurhash64-bkiaaa&JInaNcZ
orlp-murmurhash64-ewmaaa(%J+jw>j
orlp-murmurhash64-vxpaaag"93\Yj5
orlp-murmurhash64-ehuaaafa`Wp`/|
orlp-murmurhash64-yizaaa1x.zQF6r
orlp-murmurhash64-lpzaaaZphp&c F
orlp-murmurhash64-wsjbaa771rz{z<
orlp-murmurhash64-rnkbaazy4X]p>B
orlp-murmurhash64-aqnbaaZ~OzP_Tp

Universal collision attack on MurmurHash64A

In fact, MurmurHash64A is so weak that Jean-Philippe Aumasson, Daniel J. Bernstein and Martin Boßlet published an attack that creates sets of strings which collide regardless of the random seed used.

To see how it works, let’s take a look at the core loop of MurmurHash64A:

uint64_t data = loadword<uint64_t>(p);
data *= mul;          // Trivially invertible.
data ^= data >> 47;   // Trivially invertible.
data *= mul;          // Trivially invertible.
state ^= data;
state *= mul;

We know we can trivially invert the operations done on data regardless of what the current state is, so we might as well have had the following body:

state ^= data;
state *= mul;

Now the hash starts looking rather weak indeed. The clever trick they employ is by creating two strings simultaneously, such that they differ precisely in the top bit in each 8-byte word. Why the top bit?

>>> 1 << 63
9223372036854775808
>>> (1 << 63) * mul % 2**64
9223372036854775808

Since mul is odd, its least significant bit is set. Multiplying 1 << 63 by it is equivalent to shifting that bit 63 places to the left, which is once again 1 << 63. That is, 1 << 63 is a fixed point for the state *= mul operation. We also note that for the top bit XOR is equivalent to addition, as the overflow from addition is removed mod $2^{64}$.

So if we have two input strings, one starting with the 8 bytes data, and the other starting with data ^ (1 << 63) == data + (1 << 63) (after doing the trivial inversions). We then find that the two states, regardless of seed, differ exactly in the top bit after state ^= data. After multiplication we find we have two states x * mul and (x + (1 << 63)) * mul == x * mul + (1 << 63)… which again differ exactly in the top bit! We are now back to state ^= data in our iteration, for the next 8 bytes. We can now use this moment to cancel our top bit difference, by again feeding two 8-byte strings that differ in the top bit (after inverting).

In fact, we only have to find one pair of such strings that differ in the top bit, which we can then repeat twice (in either order) to cancel our difference again. When represented as a uint64_t if we choose the first string as x we can derive the second string as

x *= mul;        // Forward transformation...
x ^= x >> 47;    // ...
x *= mul;        // ...
x ^= 1 << 63;    // Difference in top bit.
x *= mulinv;     // Backwards transformation...
x ^= x >> 47;    // ...
x *= mulinv;     // ...

I was unable to find a printable ASCII string that has another printable ASCII string as its partner. But I was able to find the following pair of 8-byte UTF-8 strings that differ in exactly the top bit after the Murmurhash64A input transformation:

xx0rlpx!
xxsXъВ

Combining them as such gives two 16-byte strings that when fed through the hash algorithm manipulate the state in the same way: a collision.

xx0rlpx!xxsXъВ
xxsXъВxx0rlpx!

But it doesn’t stop there. By concatenating these two strings we can create $2^n$ different colliding strings each $16n$ bytes long. With the current libstdc++ implementation the following prints the same number eight times:

std::hash<std::u8string> h;
std::u8string a = u8"xx0rlpx!xxsXъВ";
std::u8string b = u8"xxsXъВxx0rlpx!";
std::cout << h(a + a + a) << "\n";
std::cout << h(a + a + b) << "\n";
std::cout << h(a + b + a) << "\n";
std::cout << h(a + b + b) << "\n";
std::cout << h(b + a + a) << "\n";
std::cout << h(b + a + b) << "\n";
std::cout << h(b + b + a) << "\n";
std::cout << h(b + b + b) << "\n";

Even if the libstdc++ would randomize the seed used by MurmurHash64a, the strings would still collide.

Breaking MurmurHash3

Nim ~~uses~~ used to use MurmurHash3_x86_32, so let’s try to break that.

If we once again simplify to strings whose lengths are a multiple of 4 we get the following code:

uint32_t rotl32(uint32_t x, int r) {
    return (x << r) | (x >> (32 - r));
}

uint32_t murmurhash3_x86_32(const char* s, int len, uint32_t seed) {
    const uint32_t c1 = 0xcc9e2d51;
    const uint32_t c2 = 0x1b873593;
    const uint32_t c3 = 0x85ebca6b;
    const uint32_t c4 = 0xc2b2ae35;

    uint32_t h = seed;
    for (const char* p = s; p != s + len; p += 4) {
        uint32_t k = loadword<uint32_t>(p);
        k *= c1;
        k = rotl32(k, 15);
        k *= c2;

        h ^= k;
        h = rotl32(h, 13);
        h = h * 5 + 0xe6546b64;
    }

    h ^= len;
    h ^= h >> 16;
    h *= c3;
    h ^= h >> 13;
    h *= c4;
    h ^= h >> 16;
    return h;
}

I think by now you should be able to get this function to spit out any value you want if you know the seed. The inverse of rotl32(x, r) is rotl32(x, 32-r) and the inverse of h ^= h >> 16 is once again just h ^= h >> 16. Only h ^= h >> 13 is a bit different, it’s the first time we’ve seen that a xorshift’s inverse has more than one step:

h ^= h >> 13
h ^= h >> 26

Compute the modular inverses of c1 through c4 as well as 5 mod $2^{32}$, and go to town. If you want to cheat or check your answer, you can check out the code I’ve used to generate the following ten strings that all hash to 1337 when fed to MurmurHash3_x86_32 with seed 0:

orlp-murmurhash3_x86_32-haaaPa*+
orlp-murmurhash3_x86_32-saaaUW&<
orlp-murmurhash3_x86_32-ubaa/!/"
orlp-murmurhash3_x86_32-weaare]]
orlp-murmurhash3_x86_32-chaa5@/}
orlp-murmurhash3_x86_32-claaM[,5
orlp-murmurhash3_x86_32-fraaIx`N
orlp-murmurhash3_x86_32-iwaara&<
orlp-murmurhash3_x86_32-zwaa]>zd
orlp-murmurhash3_x86_32-zbbaW-5G

Nim uses 0 as a fixed seed.

Universal collision attack on MurmurHash3

Suppose that Nim didn’t use 0 as a fixed seed, but chose a randomly generated one. Can we do a similar attack as the one done to MurmurHash2 to still generate universal multicollisions?

Yes we can. Let’s take another look at that core loop body:

uint32_t k = loadword<uint32_t>(p);
k *= c1;            // Trivially invertable.
k = rotl32(k, 15);  // Trivially invertable.
k *= c2;            // Trivially invertable.

h ^= k;
h = rotl32(h, 13);
h = h * 5 + 0xe6546b64;

Once again we can ignore the first three trivially invertable instructions as we can simply choose our input so that we get exactly the k we want. Remember from last time that we want to introduce a difference in exactly the top bit of h, as the multiplication will leave this difference in place. But here there is a bit rotation between the XOR and the multiplication. The solution? Simply place our bit difference such that rotl32(h, 13) shifts it into the top position.

Does the addition of 0xe6546b64 mess things up? No. Since only the top bit between the two states will be different, there is a difference of exactly $2^{31}$ between the two states. This difference is maintained by the addition. Since two 32-bit numbers with the same top bit can be at most $2^{31} - 1$ apart, we can conclude that the two states still differ in the top bit after the addition.

So we want to find two pairs of 32-bit ints, such that after applying the first three instructions the first pair differs in bit 1 << (31 - 13) == 0x00040000 and the second pair in bit 1 << 31 == 0x80000000.

After some brute-force searching I found some cool pairs (again forced to use UTF-8), which when combined give the following collision:

a = "!&orlpՓ"
b = "yǏglp$X"

As before, any concatenation of as and bs of length n collides with all other combinations of length n.

Breaking FarmHash64

Nim switched to farmhash since I started writing this post. To break it we can notice that its structure is very similar to CityHash64, so we can use those same techniques again. In fact, the only changes between the two for lengths 17-32 bytes is that a few operators were changed from subtraction/XOR to addition, a rotation operator had its constant tweaked, and some k constants are slightly tweaked in usage. The process of breaking it is so similar that it’s entirely analogous, so we can skip straight to the result. These 10 strings all hash to 1337 with FarmHash64:

orlp-farmhash64-?VrJ@L7ytzwheaaa
orlp-farmhash64-p3`!SQb}fmxheaaa
orlp-farmhash64-pdt'cuI\gvxheaaa
orlp-farmhash64-IbY`xAG&ibkieaaa
orlp-farmhash64-[_LU!d1hwmkieaaa
orlp-farmhash64-QiY!clz]bttieaaa
orlp-farmhash64-&?J3rZ_8gsuieaaa
orlp-farmhash64-LOBWtm5Szyuieaaa
orlp-farmhash64-Mptaa^g^ytvieaaa
orlp-farmhash64-B?&l::hxqmfjeaaa

Trivial fixed-seed wyhash multicollisions

Zig uses wyhash with a fixed seed of zero. While I was unable to do seed-independent attacks against wyhash, using it with a fixed seed makes generating collisions trivial. Wyhash is built upon the folded multiply, which takes two 64-bit inputs, multiplies them to a 128-bit product before XORing together the two halves:

uint64_t folded_multiply(uint64_t a, uint64_t b) {
    __uint128_t full = __uint128_t(a) * __uint128_t(b);
    return uint64_t(full) ^ uint64_t(full >> 64);
}

It’s easy to immediately see a critical flaw with this: if one of the two sides is zero, the output will also always be zero. To protect against this, wyhash always uses a folded multiply in the following form:

out = folded_multiply(input_a ^ secret_a, input_b ^ secret_b);

where secret_a and secret_b are determined by the seed, or outputs of previous iterations which are influenced by the seed. However, when your seed is constant… With a bit of creativity we can use the start of our string to prepare a ‘secret’ value which we can perfectly cancel with another ASCII string later in the input.

So, without further ado, every 32-byte string of the form

orlp-wyhash-oGf_________tWJbzMJR

hashes to the same value with Zig’s default hasher.

Zig uses a different set of parameters than the defaults found in the wyhash repository, so for good measure, this pattern provides arbitrary multicollisions for the default parameters found in wyhash when using seed == 0:

orlp-wyhash-EUv_________NLXyytkp

Conclusion

We’ve seen that a lot of the hash functions in common use in hash tables today are very weak, allowing fairly trivial attacks to produce arbitrary amounts of collisions if not randomly initialized. Using a randomly seeded hash table is paramount if you don’t wish to become a victim of a hash flooding attack.

We’ve also seen that some hash functions are vulnerable to attack even if randomly seeded. These are completely broken and should not be used if attacks are a concern at all. Luckily I was unable to find such attacks against most hashes, but the possibility of such an attack existing is quite unnerving.

With universal hashing it’s possible to construct hash functions for which such an attack is provably impossible, last year I published a hash function called polymur-hash that has this property. Your HTTPS connection to this website also likely uses a universal hash function for authenticity of the transferred data, both Poly1305 and GCM are based on universal hashing for their security proofs.

Of course, if your data is not user-controlled, or there is no reasonable security model where your application would face attacks, you can get away with faster and insecure hashes.

More to come on the subject of hashing and hash tables and how it can go right or wrong, but for now this article is long enough as-is…

The evolution of incident response at Podia

Jamie’s blog — Sat, 02 Nov 2024 02:00:00 GMT

Over the years, Podia has gone from having no incident response processes to a more structured system of on-call shifts and supportive tooling. You might be thinking of embarking on the same journey, or be confused why any developer would embrace the responsibility of on-call shifts, or wonder how you can introduce it without pissing off your entire team.

Here’s how I did it.

Life without on-call

It’s blissful, right? No late-night phone calls. No need to remember your laptop or modify your plans. Nothing ever goes wrong. The app never goes down, your providers never fail, deploys never break anything, and customers are never impacted.

I mean, that’s what you think happens but ignorance doesn’t make it true. And this is not the reality we lived under.

In the earliest days, we had no monitoring at all which is where all projects start out. Pretty soon though I added Rollbar to start tracking exceptions that happened in production and pipe those notifications into Slack. And then we noticed the app might be briefly unresponsive at times so we added a simple uptime checker (UptimeRobot) to tell us if the app stopped responding.

Back in the early days when Podia had just one, two, or three or four full-time developers we didn’t have any defined processes for handling these notifications. Instead, our typical daily routine for each of us went something like this: wake up, check Slack for errors (because of cause of it was on our phones), have breakfast, check Slack for errors, do some work whilst checking Slack for errors, run some errands whilst checking Slack for errors…

Having every developer poll Slack for news about problems is not very efficient and over the long-term it led to a lot of internalised stress and apprehension. When we were constantly logging into Slack, especially during those moments of personal time when we’re supposed to be resting, it quickly developed into a neurotic pattern of behaviour.

I remember being in town one Saturday with the family and a checked Slack out of habit as I was walking down the street. There I saw a ton of errors for, I think, our background jobs being killed due to OutOfMemory errors. Since I was about an hour from home, without a laptop, and supposed to be enjoying an afternoon with the family, there wasn’t much I could do about the problem. Luckily, Jason had woken up in Tennessee, saw the messages and was able to resolve the issue.

We were lucky because we didn’t have many users at the time and they weren’t actively using the platform at all times of the day and week. But that was changing fast…

Introducing on-call

I knew we needed an on-call system but I was also extremely wary of it becoming an exploitative system that the developers would come to resent.

In the 20 years of my career, I’d never had to be on-call for a system. I mostly worked in research or enterprise apps, and briefly in a Level 3 Support role, but if you’d asked me I’d have said that being on-call was for the IT staff. I also, frankly, would never have cared enough about those jobs or trusted any of the companies I worked for to manage the on-call shifts in a responsible manner. The web has changed all that. In particular, the global reach of our apps means we have a never-off cycle of traffic that cannot be confined to any normal business hours. Things will go wrong when we are asleep and they will impact our users.

I used these principles when designing our on-call system, though most flow from the first:

Minimise the impact on our personal lives
Being on-call should be as low-stress as possible
Restrict the number of things that can trigger an incident
Have reasonable expectations on response times etc
The whole team has responsibility for maintaining the app so we will work as a team on-call

How often?

First, I needed to decide on the shift structure for the team. The most popular rotation schedules are either weekly (on-call for 7 days), daily (on-call for 24 hours), or something more sophisticated like follow-the-sun or business-hours/out-of-hours cover. For me, the only reasonable choice was between weekly or daily.

Weekly schedules have the advantage that they don’t come around too often. With just four of us, you’d only be on-call for 1 week in every 4. The downside though is that you’re going to be carrying your laptop around for a whole week and that’s very disruptive. Selfishly, it would have meant that I couldn’t swim for an entire week and that was a major disadvantage to me.

In my opinion, 24-hour shifts are the ideal frequency so that’s what we went with and it hasn’t changed since. Your life is only mildly inconvenienced for a day, so it’s relatively easy to schedule things around being on-call. It’s also easier to find someone to cover a shift for you, doesn’t interfere with a whole weekend, and if we hit a patch of poor reliability those incidents are more likely to be distributed across the team. The downside is that you’ll be on-call more frequently but as the team size increases so the on-call frequency decreases.

What wakes us up?

As a kid I remember sleeping out one night in a tent in our back garden—long before mobile phones and clearly before my parents would trust us with a house key. But my dad had thought of an ingenious solution in case we needed to come into the house in the middle of the night: he tied a piece of string around his toe and we were to tug on it we needed to be let back in. What things could wake us up when we were on-call? What alerts would be tied to our toes with a piece of string to wake us up in the middle of the night? That’s a lot of trust to place is something.

I knew there would be pressure to expand the on-call system over time, especially as users encountered bugs or wrote in with irate support requests because we didn’t respond to their particular issue overnight. We had to be ready to resist the scope-creep of being on-call in order to make it sustainable so I felt it was important to place limits on this process.

I settled on having just three events being worthy of waking us up and, if we needed to add another alert, then one of these would have to go:

the app is down, as detected by some external monitor
our background job latency is over a defined period, meaning there are too many jobs piling up in the queue
our custom domain proxy was not working

Today, the first two events are still the main alerts that we use. Our custom domain proxy has been replaced by Cloudflare so it’s no longer a source of downtime, and is no longer monitored separately. Instead, our third alert source is the facility for anyone in our Slack—typically our support team—to declare an incident and page the on-call developer. It’s rarely used and I think it’s taken time to build that trust. I’m not sure we’d have introduced it on day 1.

It’s also important to have some expectations around how often we expect incidents. For us, it’s once or twice a month. They should not be daily or weekly occurrences.

I’m pretty satisfied with these criteria and if you were starting out today I would still recommend you only page the on-call developer if a) the app is down or b) the background jobs aren’t processing at a rate you expect. Everything else is noise and anything significant will usually trigger one of these alerts.

I am pretty definite that exceptions are not incidents. An exception can be generated by a malformed client, or a user encountering a transient issue just once, or a background job hitting a timeout and then retrying successfully. Or a validation triggered an error but the user completed the form anyway. Hell, even if they couldn’t complete the form, it’s still something you can solve tomorrow. Unless you are in an industry with severe consequences for errors, none of these things rise to the level that should wake your team up.

Expectations

I had read accounts of on-call engineers at Google being required to always be within arms-reach of their laptops and that didn’t sound like a life I wanted to live—or provide for the rest of the team. Especially as many of us had families and hobbies and our app being down for a few minutes was not going to kill anyone.

Our expectations are, in my opinion, much more reasonable: respond within 10-15mins. That means you can leave the laptop in the car when you go shopping; or go for a short walk near your house. We just expect that the developer will be ready to triage the incident within this reasonable timeframe

I even addressed whether you can drink when on-call: sure, just don’t operate production systems when drunk! We all might enjoy a beer or glass of wine after work and I didn’t want to restrict our team’s social lives, and neither did I want anyone to hide the fact they’d been out on a heavy session when on-call. If you’re drunk, please don’t acknowledge the incident!

Teamwork

I’ve always been very clear that the app is the whole team’s responsibility. It’s not just mine. And you can’t just claim that little corner over there as yours. We all have a collective responsibility to keep this thing alive. “It’s not my job” is not an attitude that’s tolerated at Podia: if you ship code at Podia then it’s your job to look after it.

And that means when you ship some dodgy code, it’s going to wake you, or one of your colleagues up at night. Or when you review a PR and don’t speak up about your concerns, that’s the thing which could wake you up. When we advocate for infrastructure choices, we do so knowing that all of us will be on-call to support it.

Incidents are best played as a team sport since, ideally, most of them should ideally be novel situations that we haven’t encountered before.

When a developer acknowledges an incident, they will start to triage it to understand what happened and why, what the resolution might be, etc. It’s rarely something that you can look up in a playbook and run through a sequence of commands to resolve. We used to have a “runbook” but most of those situations have now been engineered away so we don’t really encounter them now.

There’s nothing more frightening than receiving a 4am call, blurry-eyed, scrambling for your laptop, and then all on your own trying to decide how best to handle the backlog of jobs in the queue. Do you pause the process? Delete the enqueued jobs? Increase the concurrency? Yikes, no. You need some help.

It’s the general expectation that if you see an incident in-progress, you join it and offer to help. You never let your colleague handle the incident on their own as you continue to work on whatever you were doing—because nothing you could be working on is more important than helping out a teammate.

Since Day 1 we’ve announced all incidents in Slack immediately as soon as the alert is triggered, and then paged the on-call developer a minute or two later. This means that if the incident happens at 4am in New York, an EU developer working at 9am can spot the incident notification and acknowledge it, thereby saving their colleague from a very early morning wake-up call. It’s not only the on-call developer that can acknowledge the incidents.

This broadly works because Podia developers are spread across European and North American timezones, so we have pretty good coverage for 12-16 hours of the day during the week. But in cases when the developer is still left dealing with the incident on their own, they can page me (I’m CTO so I’m always on-call even when I’m not) or another developer on the team to help. We now even have a prompt in the incident channel after a short delay asking “Do you have everyone you need to resolve this incident?” and instructions for how to page someone else.

I don’t like incidents; they’re stressful for the team and each one impacts our users. But they are also a special bonding moment for us as a team. There’s a great camaraderie that comes from working together under a (hopefully!) brief stressful situation, solving a puzzle together, and remediating the problem. A little bit of adrenaline is good for the soul! There’s also a counter-intuitive argument that you can’t get good at something that only happens occasionally. We need incidents (😅) so that we can collectively learn how our systems function and how to respond to them when they fail.

Being on-call also has other ways we can demonstrate teamwork and strengthen relationships, particularly important on a remote team. Shift swapping is common and encouraged; being flexible is another way of limiting the impact to each of our lives, and another opportunity to say “I got you”. When someone joins the team, they’re not on-call for the first month. If people are up handling an incident overnight then I’ll take over the rest of their shift and tell them to take the morning off. When someone has a baby, we take them off the on-call rota for a few months. When someone leaves the team, I typically take over that person’s shifts for a while so the team doesn’t have to absorb the impact straight away. All small gestures of teamwork.

All in all, having an on-call rota is one of the few ways we can work together as a whole team rather than working on projects in pairs.

On-call, over time

Our on-call process was introduced a good few years ago now and it has largely remained unchanged. It’s still a 24-hour rota, only three things can wake us up, and we treat it as a whole-team responsibility. Some things have changed though.

Team size

When we started out, I think there was only 4, maybe 5, of us on the rota. At our largest, there was been 12 people on team, which led to a very relaxed on-call frequency. If anything, that was verging on being too infrequent and had the team scaled to >15 developers, I think I would have introduced a two-layer on-call system so there would always be a primary and secondary developer on-call.

At the moment, we’re just 8 people on the dev team which narrowly avoids the awkward situation with 7 people where the same person would always be on-call for the same day of the week.

I do think it’s worth having some on-call system even with just a single developer to avoid obsessively checking slack but, generally, the more people in the rota the better it is for everyone. Less than 4 developers is always going to be a tough situation.

We also now have the entire leadership team pageable (but not on a rota) so for larger incidents we can pull in Support or Marketing help to communicate with our users about the incident.

Tools

Our tool choice has also changed over the years. We started out using OpsGenie because the UI didn’t make my eyes bleed like PagerDuty did. Not that it was great but it was mostly useable.

We also used UptimeRobot to monitor the app’s uptime across a few different endpoints and assess the background queue latency by polling an endpoint that reported the metric.

Our stack today starts with Cronitor which has taken over app monitoring duties from UptimeRobot. When Cronitor detects a problem, it triggers an incident in incident.io. This provides us with a Slack app for managing the incident process: it auto-creates a slack channel, broadcasts the incident notification to a company-wide #podia-updates channel, pages the on-call developer, and triggers some timed-prompts during the incident (like “Do you have everyone you need to resolve this incident?”). We can also assign tasks inside the incident Slack channel, pin messages for an after-action incident report, and update our internal incident status. It also makes it incredibly easy to escalate the incident to another developer, knowing that it will break through their do-not-disturb settings if required. If necessary, anyone in the company can create an incident from within Slack using incident.io.

Our status page is also hosted by incident.io so anyone at the company can update it from within Slack. This is great during an incident when the devs might be heads-down trying to figure out the problem and someone from support can craft a more detailed message for our users.

Once the incident is resolved, we can write up an incident report using a timeline compiled automatically by incident.io from the posts in the Slack channel, including relevant screenshots, code snippets, PR links etc. We’re not particularly rigorous with this process but it still represents a valuable body of work, especially for new developers joining the team and wondering what they’re getting themselves into.

We also rely heavily on Dataset (a.k.a. Scalyr before they were acquired) to give us an at-a-glance dashboard and the ability to filter, search, and chart data from our logs. We use AppSignal for exception tracking and more fine-grained performance profiling.

But the tools don’t really matter. If I was doing this today from scratch, I’d probably look at BetterStack because they literally have everything you need under one roof.

Takeaways

What matters first is thinking about how an on-call rota can improve the lives of your team and business—being on-call sometimes is infinitely better than everyone constantly checking Slack on their time off and living with the permanent stress that the app might be down.

Once you’ve accepted the necessity and desire for having an on-call rota, you can design the process in such a way that it minimises the stress for those involved and promotes stronger tied within the team. You can place limits on it and make it flexible enough so that the impact is quite minimal.

These days, being on-call for Podia means I throw my laptop and AirPods in a bag when I leave the house. Or I choose to do jobs around the house. I can’t swim for that 24 hour period but that’s ok: I can still go to the gym, or run errands, go out for a meal, or watch a movie at the cinema.

I can do that because the demands are low, I know the team will have my back if they’re working anyway, and incidents are pretty rare. Typically we have ~1 incident per month and these rarely involve waking someone up: usually it’s caught by a developer who was working in their timezone or because it occurred during normal working hours (because—sssshhhhhh—the main cause of downtime is actually developers doing developer-things 😅. Who knew?!). At times, we have had spurious alerts that triggered an incident before the alert was auto-resolved. These were very frustrating for everyone but we try to investigate them and put in preventative measures to squash these alerts.

No system is perfect but I do believe you can structure the process to make it less burdensome.

GeoClue TZ: Privacy-First Linux Location Service

Terminal Thoughts — Sat, 02 Nov 2024 02:00:00 GMT

Linux location services provided by GeoClue have been historically unreliable. Whether it’s VPN usage causing incorrect locations, rate limits from upstream services, or broken functionality after system upgrades, users often find themselves struggling with basic location-aware features.

Note 152

Qubyte Codes — Fri, 01 Nov 2024 12:38:06 GMT

Spotted my first jay of the autumn! I think this is probably as close as I’ve ever been to one.

Release: OuroborosDB Data Storage Calculator

Mia Heidenstedt — Fri, 01 Nov 2024 11:17:10 GMT

Best viewed on the original page, where extended functionality like the footnote helper is available.

I forgot to mention that i released a new tool for my project ouroboros-db, so here it is:

The “OuroborosDB Overhead Calculator” is a little tool which helps you to calculate the overhead of erasure coding systems while also considering the overhead of indexes and the blocks themselves. It is kinda fun to play around with so give it a try!

https://i5heu.github.io/ouroboros-db-overhead-calculator/

Your browser does not support the video tag.

Moving in Big Cities: Lugg vs Clutter

swyx's site RSS Feed — Fri, 01 Nov 2024 11:12:31 GMT

2026 update: used Lugg to move a sectional sofabed couch and ended up regretting it - it cost $222.51 one way (44min labor pickup 55min labor dropoff, which i think was fraud, no way dropoff cost more than pickup) and total base fare 38 + 14.5 fee. for the two way move price i could have simply ordered a new couch and had a nice new couch.

Getting Started with Candlesticks Patterns and Python

Adnan's Random bytes — Fri, 01 Nov 2024 09:18:52 GMT

This post is part of the T4p Series. In the previous post, I briefly introduced candlesticks and how to detect a bearish and bullish candle in Python. In this post, we are going deeper, discussing candlestick patterns, what they are all about, and how to detect and represent them in Python Introduction As you learned in the previous post, there are mainly two types of candles: Bearish and Bullish. The size of the wick and the body determines what kind of bearish and bullish candle it is. Candlestick patterns visually help traders learn about the price movement to interpret the market sentiments. History Candlestick patterns were developed centuries ago by Japanese rice traders looking to predict price shifts in the market. Today, traders worldwide use these same patterns to spot potential trends and reversals, blending ancient wisdom with modern trading techniques. Types of Candlestick Patterns In a broader sense, we can divide patterns into two categories: Single Candlestick Patterns: Doji, Hammer, and Shooting Star. Multiple Candlestick Patterns: Engulfing, Morning Star, and Evening Star. Some of the patterns represent either a bearish or bullish market while other patterns emerge in both. There are many candlestick patterns, such as Doji, Engulfing, Hammer, Morning Star, Evening Star, etc. I can’t cover all those in a single post, so I will try to cover them in separate posts. For now, I am discussing the Doji and Engulfing patterns here. Doji Pattern It is a pattern that occurs when the opening and closing prices are virtually equal, creating a cross or a plus sign shape. The name Doji comes from Japanese and means blunder or mistake, referring to the rarity of having the same opening and closing prices. Usage Doji is used as a reversal pattern. After an Uptrend: When you spot a Doji after a strong upward move Wait for confirmation (next candle closing lower) Consider taking profit on long positions or entering short positions Place stop loss above the Doji’s high After a Downtrend: When you spot a Doji after a strong downward move Wait for confirmation (next candle closing higher) Consider entering long positions or covering shorts Place stop loss below the Doji’s low Types of Doji Standard Doji: Open and Close should be the same/very close Equal shadows above and below Dragonfly Doji: Open and Close at/near the high Long lower shadow Little to no upper shadow Gravestone Doji: Open and Close at/near the low Long upper shadow Little to no lower shadow Visual Representation Now let’s write some code in Python to learn how you can represent Doji patterns In [27]: import mplfinance as mpf import pandas as pd In [28]: def plot_candlestick_from_dict(data,title="CandleStick Chart"): df = pd.DataFrame(data) df['Date'] = pd.to_datetime(df['Date']) df.set_index('Date', inplace=True) mpf.plot(df, type='candle', style='charles', title=title, ylabel='Price') Standard Doji In [29]: ohlc_data = [ {"Date":"2024-09-01", "Open":100, "High":105, "Low":95, "Close":100}, ] plot_candlestick_from_dict(ohlc_data,"Standard Doji") Dragonfly Doji In [30]: ohlc_data = [ {"Date":"2024-09-01", "Open":100, "High":100.5, "Low":95, "Close":100}, ] plot_candlestick_from_dict(ohlc_data,"Dragonfly Doji") Gravestone Doji In [31]: ohlc_data = [ {"Date":"2024-09-01", "Open":100, "High":105, "Low":99.5, "Close":100}, ] plot_candlestick_from_dict(ohlc_data,"Gravestone Doji") I have embedded the entire notebook. Visual representation is not enough. Now let’s write some code to detect Doji patterns in the trading OHLC data. def is_doji(open_price, high, low, close, doji_size=0.1): """ Check if a candlestick is a Doji pattern Parameters: open_price: Opening price high: High price low: Low price close: Closing price doji_size: maximum body size ratio threshold (default 0.1 or 10% of range) """ body_size = abs(close - open_price) candle_range = high - low # Avoid division by zero if candle_range == 0: return False # Calculate body size as a ratio of the candle range body_ratio = body_size / candle_range return body_ratio <= doji_size I have written functions for all Doji types but here I am explaining the core concept that will be helpful to figure out different Doji types. Here are a few concepts: Body Size: It should be very small for Doji. We are computing it by taking the difference between closing and opening prices. Candle Range: Total range of price movement. Body Ratio: Compares the body to the total range. Doji Size: Threshold (default 0.1 or 10%). The visual representation would be something like the below: Normal Candle: Doji Candle: High 105 ─┬─ High 105 ─┬─ │ │ │ │ Open 100 ─┤ Open 100 ─┼─ Close 100.2 │ │ │ │ Close 95 ─┴─ Low 95 ─┴─ Body Ratio: 30% Body Ratio: 2% (Not a Doji) (Is a Doji!) You can see the entire code in the Jupyter Notebook. Engulfing Pattern It is formed by two candles where the body of the first candle is engulfed by the body of the second candle. Trading volume often increases on the engulfing candle. There are two types of engulfing patterns: Bearish and Bullish. Bullish Engulfing: A two-candle pattern where a smaller bearish candle is completely “engulfed” by a larger bullish candle, suggesting a potential upward reversal. Bearish Engulfing: The opposite, where a smaller bullish candle is engulfed by a larger bearish candle, hinting at a possible downward trend. Visual Representation Below is the exported Jupyter Notebook for the visual representation of the engulfing pattern in Python: In [65]: import mplfinance as mpf import pandas as pd In [66]: def plot_engulfing_pattern(data, title): # Convert to DataFrame df = pd.DataFrame(data) df['Date'] = pd.to_datetime(df['Date']) df.set_index('Date', inplace=True) # Create the plot mpf.plot(df, type='candle', title=title, volume=False, # Set to True if you have volume data style='charles', figsize=(10,5)) In [67]: bullish_engulfing_data = [ # Downtrend before pattern {"Date":"2024-09-01", "Open":110, "High":112, "Low":108, "Close":109}, {"Date":"2024-09-02", "Open":109, "High":110, "Low":106, "Close":107}, {"Date":"2024-09-03", "Open":107, "High":108, "Low":104, "Close":105}, # Engulfing pattern {"Date":"2024-09-04", "Open":105, "High":105.5, "Low":103, "Close":104}, # Small bearish candle {"Date":"2024-09-05", "Open":103, "High":108, "Low":102, "Close":107}, # Bullish engulfing # Uptrend after pattern {"Date":"2024-09-06", "Open":107, "High":110, "Low":106, "Close":109} ] # Sample data showing Bearish Engulfing Pattern bearish_engulfing_data = [ # Uptrend before pattern {"Date":"2024-09-01", "Open":100, "High":102, "Low":99, "Close":101}, {"Date":"2024-09-02", "Open":101, "High":104, "Low":100, "Close":103}, {"Date":"2024-09-03", "Open":103, "High":106, "Low":102, "Close":105}, # Engulfing pattern {"Date":"2024-09-04", "Open":105, "High":108, "Low":104, "Close":107}, # Small bullish candle {"Date":"2024-09-05", "Open":108, "High":109, "Low":103, "Close":104}, # Bearish engulfing # Downtrend after pattern {"Date":"2024-09-06", "Open":104, "High":105, "Low":101, "Close":102} ] In [68]: plot_engulfing_pattern(bullish_engulfing_data, 'Bullish Engulfing Pattern') plot_engulfing_pattern(bearish_engulfing_data, 'Bearish Engulfing Pattern') I added three functions to detect both bearish and bullish candles: def is_bearish_engulfing(current_candle, previous_candle): """ Detect bearish engulfing pattern. Args: current_candle (dict): Current day's OHLC data previous_candle (dict): Previous day's OHLC data """ # Check if previous candle is bullish prev_bullish = previous_candle["Close"] > previous_candle["Open"] # Check if current candle is bearish curr_bearish = current_candle["Close"] < current_candle["Open"] # Check engulfing conditions engulfs_body = (current_candle["Open"] > previous_candle["Close"] and current_candle["Close"] < previous_candle["Open"]) return prev_bullish and curr_bearish and engulfs_body def is_bullish_engulfing(current_candle, previous_candle): """ Detect bullish engulfing pattern. Args: current_candle (dict): Current day's OHLC data previous_candle (dict): Previous day's OHLC data """ # Check if previous candle is bearish prev_bearish = previous_candle["Close"] < previous_candle["Open"] # Check if current candle is bullish curr_bullish = current_candle["Close"] > current_candle["Open"] # Check engulfing conditions engulfs_body = (current_candle["Open"] < previous_candle["Close"] and current_candle["Close"] > previous_candle["Open"]) return prev_bearish and curr_bullish and engulfs_body def find_engulfing_patterns(candles): """ Find all engulfing patterns in a list of candles. Args: candles (list): List of dictionaries containing OHLC data """ patterns = [] for i in range(1, len(candles)): current = candles[i] previous = candles[i-1] if is_bearish_engulfing(current, previous): patterns.append({ "date": current["Date"], "pattern": "Bearish Engulfing", "previous_candle": previous, "engulfing_candle": current }) if is_bullish_engulfing(current, previous): patterns.append({ "date": current["Date"], "pattern": "Bullish Engulfing", "previous_candle": previous, "engulfing_candle": current }) return patterns When you run these functions you’ll see something like the below in the notebook: Conclusion There are many other patterns that I cannot cover in this post. It has already become quite lengthy. In the coming posts, I will be discussing each pattern in a separate post. Like always, the code is available on GitHub. Have a winning trading strategy and want it to trade for you automatically, without lifting a finger? I specialize in turning your strategies into fully automated trading bots—whether for stocks, crypto, or forex. Let your strategy work around the clock and maximize your potential. Ready to bring it to life? Email me at kadnan @ gmail.com If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Getting Started with Candlesticks Patterns and Python first appeared on Adnan's Random bytes.

Writing GDScript with Neovim

Alice GG — Fri, 01 Nov 2024 09:00:00 GMT

Neovim is by far my favorite text editor. The clutter-free interface and keyboard-only navigation are what keep me productive in my daily programming. In an earlier post, I explained how I configure it into a minimalist development environment. Today, I will show you how to use it with Godot and GDScript.

Configure Godot

First, we need to tell Godot to use nvim as a text editor instead of the built-in one. Open Godot, and head to Editor Settings > General > Text Editor > External. There, you will need to tick the box Use external editor, indicate your Neovim installation path, and use --server /tmp/godothost --remote-send "<C-\><C-N>:n {file}<CR>{line}G{col}|" as execution flags.

While in the settings, head to Network > Language Server and note down the remote port Godot is using. By default, it should be 6005. We will need that value later.

Connecting to Godot with vim-godot

Neovim will be able to access Godot features by using a plugin called vim-godot. We will need to edit the nvim configuration file to install plugins and configure Neovim. On Mac and Linux, it is located at ~/.config/nvim/init.vim

I use vim-plug to manage my plugins, so I can just add it to my configuration like this:

call plug#begin('~/.vim/plugged')
" ...
Plug 'habamax/vim-godot'
" ...
call plug#end()

Once the configuration file is modified and saved, use the :PlugInstall command to install it.

You’ll also need to indicate Godot’s executable path. Add this line to your init.vim:

let g:godot_executable = '/Applications/Godot.app/Contents/MacOS/Godot'

For vim-godot to communicate with the Godot editor, it will need to listen to the /tmp/godothost file we configured in the editor previously. To do that, simply launch nvim with the flag --listen /tmp/godothost.

To save you some precious keypress, I suggest creating a new alias in your bashrc/zshrc like this:

alias gvim="nvim --listen /tmp/godothost"

Getting autocompletion with coc.nvim

Godot ships with a language server. It means the Godot editor can provide autocompletion, syntax highlighting, and advanced navigation to external editors like nvim.

While Neovim now has built-in support for the language server protocol, I’ve used the plugin coc.nvim to obtain these functionalities for years and see no reason to change. You can also install it with vim-plug by adding the following line to your plugin list:

Plug 'neoclide/coc.nvim', {'branch':'release'}

Run :PlugInstall again to install it.

You’ll need to indicate the Godot language server address and port using the command :CocConfig. It should open Coc’s configuration file, which is a JSON file normally located at ~/.config/nvim/coc-settings.json. In this file enter the following data, and make sure the port number matches the one located in your editor:

{
    "languageserver": {
        "godot": {
            "host": "127.0.0.1",
            "filetypes": ["gdscript"],
            "port": 6005
        }
    }
}

I recommend adding Coc’s example configuration to your init.vim file. You can find it on GitHub. It will provide you with a lot of useful shortcuts, such as using gd to go to a function definition and gr to list its references.

Debugging using nvim-dap

If you want to use the debugger from inside Neovim, you’ll need to install another plugin called nvim-dap. Add the following to your plugins list:

Plug 'mfussenegger/nvim-dap'

The plugin authors suggest configuring it using Lua, so let’s do that by adding the following in your init.vim:

lua <<EOF
local dap = require("dap")
dap.adapters.godot = {
	type = "server",
	host = "127.0.0.1",
	port = 6006,
}

dap.configurations.gdscript = {
	{
		type = "godot",
		request = "launch",
		name = "Launch scene",
		project = "${workspaceFolder}",
		launch_scene = true,
	},
}

vim.api.nvim_create_user_command("Breakpoint", "lua require'dap'.toggle_breakpoint()", {})
vim.api.nvim_create_user_command("Continue", "lua require'dap'.continue()", {})
vim.api.nvim_create_user_command("StepOver", "lua require'dap'.step_over()", {})
vim.api.nvim_create_user_command("StepInto", "lua require'dap'.step_into()", {})
vim.api.nvim_create_user_command("REPL", "lua require'dap'.repl.open()", {})
EOF

This will connect to the language server (here on port 6005), and allow you to pilot the debugger using the following commands:

:Breakpoint to create (or remove) a breakpoint
:Continue to launch the game or run until the next breakpoint
:StepOver to step over a line
:StepInto to step inside a function definition
:REPL to launch a REPL (useful if you want to examine values)

Conclusion

I hope you’ll have a great time developing Godot games with Neovim. If it helps you, you can check out my entire init.vim file on GitHub gist.

The best laptop is the one somebody else had

./techtipsy — Fri, 01 Nov 2024 06:00:00 GMT

In 2011, I was finishing 9th grade. As a gift, I got to choose a laptop in the 400 EUR range. I ended up picking an ASUS Eee PC 1201PN. It was new and the first computer in my life that was 100% mine, but awfully slow for a lot of tasks.

It was so slow that I ended up giving Linux a go as a result. Linux! I didn’t even know computing all that well around that time!

A few years later, I bought a ThinkPad T60 off of someone I knew for about 40 EUR. It was about 8 years old at that point, but it ran circles around the new laptop that I had in performance.

That’s when I learned about the absurdly good price-to-performance ratio of used business-grade laptops, and the crappiness of netbooks.¹

Note that I keep repeating the phrase business-grade laptops. Think Lenovo ThinkPads, Dell Latitudes or HP EliteBooks.²

That’s the core of this whole idea. Consumer-grade laptops are cheaper when bought new, but that is a result of a lot of compromises made in the build quality. Business-grade laptops are used for work and need to be reliable for years, which means that they will last for a long time.

Used laptops are cheap

I recently checked what the prices are for used laptops, mainly focusing on the 100-300 EUR range as I find that to be the sweet spot for bargains.

For 195 EUR, I can get a ThinkPad X395, sporting an AMD Ryzen 5 3500U quad-core CPU, 16 GB of RAM and a 256GB NVMe SSD, sold by a store that specializes in selling used hardware. You even get a 6-month warranty! That’s crazy good value.

New business-grade laptops cost somewhere around 1000-2000+ EUR. They are generally faster and provide more memory and storage, but in the best case scenario that performance difference will be 2-3x at best, while the price is 5-10+ times higher. The math does not check out.

The price depreciation curve is also quite harsh on new laptops. You can pay 2000 EUR for a new laptop and only be able to sell it for 1000 EUR a year from now. Two years later? 500-700 EUR. The prices eventually settle at around the 5 year mark, which also happens to align with the extended warranties expiring.

Used laptops reduce stress

With used laptops, you don’t have to worry about the wear-and-tear that much.

You accidentally drop your laptop on the floor? It might still be fine!

Your child picked off all the keycaps on the keyboard? No worries, replacements are easy to find!

Your lunch for the day ended up leaking all over the laptop, killing it completely? No problem, you can get a new one and still end up paying less compared to a new laptop!

Buying used is no excuse to mistreat your hardware, but I personally love the lack of stress associated with trying to keep a new and expensive object in pristine condition. The laptop already has some cosmetic damage on it, so why worry?

Used laptops are surprisingly reliable

Reliability is often one of the top reasons why some people avoid buying used laptops. I attribute this to the experiences people have with used cars. You pay less, until you pay a lot more to get that hunk of junk fixed once it breaks down on you.

I’ve had the complete opposite experience with used business-grade laptops. The ones that make it to the used market have gone through years of reliability testing, and those that don’t make it were defective anyway. The only areas to pay attention to is basic maintenance (remove dust, apply new thermal paste) and a potential battery replacement, which are quite simple to do on modern business-grade laptops. It’s so easy that even children and teenagers can do it with a little bit of guidance and supervision!

The reliability doesn’t stop with the hardware. Buying used often means that you’ll be buying a laptop that has received all the software and firmware fixes to all sorts of issues. Linux users will also have a much better time with used laptops since by that time most of the issues associated with new hardware will have been fixed in the kernel.

You should avoid buying new and used consumer-grade laptops. I’ve seen so many of those with missing pieces of plastic and the hinges breaking open the laptop case, but rarely with business-grade laptops.

Used business-grade laptops are so reliable that some companies are even willing to rent and support those machines for a really low price.

Exceptions to the rule

There will always be a place for new laptops.

Sometimes you do need the latest and greatest hardware for CAD work, complex video editing or high-end gaming.

Some people find that a 30-second build of their software project taking 20 seconds is worth the productivity gain, regardless of the higher price or increased environmental impact of buying new.

Some simply want to play around with the latest and greatest, for fun.

There will always be people who find the idea of used laptops off-putting, and companies do prefer to buy pallet-loads of new laptops every few years.

On the bright side, this does mean that there will always be a supply of cheap used laptops available for the rest of us.

Conclusion

If you ever need a laptop and your needs are not extremely specific, then give a used business-grade laptop a try. It will be fine, I promise.

no hard feelings to my mom, we both didn’t know any better. ↩︎
note that there are certain models that are a ThinkPad in name only, under the hood it’s still the same crappy components you see in consumer-grade laptops. Thanks, Lenovo. ↩︎

2024.10.DisappearingMoment

Disappearing Moment — Fri, 01 Nov 2024 04:16:05 GMT

What are the odds that the worst human was born between 1878 and 1893? The world population in 1945 was about 2.4 billion. If 117 billion people have been born, that sets the odds a touch over 2%.

Of course, that doesn’t account for. Or. And then there’s. How many millions suffered and died because of?

I don’t believe in the Great Man Theory. Modern atrocities rely on circumstance, abetting, credulity, and shock. They also rely on technology.

Our ability to help and harm has magnified individual and collective responsibility. Factoring in technology, the possibility that the worst human was born between 1878 and 1893 is higher than 2%.

In the last 80 years, technology has advanced by orders of magnitude. What are the odds that the worst human ever born is alive today? A touch under 7% of the people who have been born are alive. Accounting for technology means there’s a good chance that we’re living alongside history’s most vile human.

Welcome to October 2024’s Disappearing Moment, an inventory of my experiences. I hope you enjoy it.

Podcasts

Diaspora (I Loved It): Spoiler Alert: the list below only includes one entry under “N”.

Nerdy Software

Homebrew makes it easier to install open source software. Some favorites: Ffmpeg, Git, Hugo, Hut, and LibreWolf.

Free Font

Bonnie Shover-Troup developed Lexend to “make reading easier for everyone.” She wants you to be in control, so you can "improve retention and comprehension.”

Bougie Products

Xero Shoes are great for the gym and good for minimal running. No athletic shoes are better made. And they're having a big sale

Personal Finance and Investing

Personal finance is about mitigating risk. Do you have a rainy day fund in Euros, Swiss Francs, Yen, or AUD? In a non-US account?

Reading

Ahmad Alfy, “Smarter than 'Ctrl+F': Linking Directly to Web Page Content” (I Liked It): “Text fragments are currently supported in all the browsers.” I had no idea. This changes everything. See also, “Don’t Click Here.”
Loris Cro, “The Static Site Paradox” (I Liked It): I can no longer recommend Squarespace or WordPress. Can we simplify serving Hugo sites from anti-VC hosts? E.g., “The Making of HTML for People.”
Kathryn S. McKinley, “What Happens to Us Does Not Happen to Most of You” (I Liked It): A reminder to track what I listen to and read. To pay attention to people whose experiences are not like mine. To work for change.
Jenn Schiffer, "Falling for Prince's Fridge" (I Loved It): Accuracy is ephemeral. Satire is everlasting.

The Worst Human

Omar al-Bashir or Samuel Alito or Bashar al-Assad
Steve Bannon or Jair Bolsonaro
Tucker Carlson or Dick Cheney
Paul Dans or Thomas DiLorenzo
Recep Tayyip Erdoğan
Patrick Falte and Benjamin Faulkner
Newt Gingrich or Marjorie Taylor Greene
Josh Hawley or Min Aung Hlaing
Laura Ingraham
Mike Johnson or Jerry Jones
Kim Jong Un or Charles Koch
Debbie Lesko
Rupert Murdoch or Elon Musk
Benjamin Netenyahu
Andy Ogles or Bill O'Reilly
Vladimir Putin
Q
Kevin Roberts or Karl Rove
David D. Smith or Daniel Stein
Donald Trump
Richard and Elizabeth Uihlein
J.D. Vance
Andrew Wakefield or Randy Weber
Xi Jinping
Ramzi Yousef
Mark Zuckerberg

Thank you for spending a few moments with me. I appreciate you and look forward to corresponding again next month.

Brett

Want to discuss any of the topics in this newsletter or anything else with other Disappearing Moment readers? Please sign up for Perpetual August. I think it might be fun.

MX-518 Repair Guide

Cookie Engineer's Web Log — Fri, 01 Nov 2024 02:00:00 GMT

Repair Guide for the best mouse ever produced, the Logitech MX-518

November 2024 - Bookmarks

Domenico Luciani — Fri, 01 Nov 2024 01:00:00 GMT

Bookmarks for November 2024: 6 links - What is a Staff Engineer?; Please just stop saying "just"; Breaking Down Tasks - Jacob Kaplan-Moss, and more.

Maintenance is Underestimated

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Thu, 31 Oct 2024 18:00:00 GMT

Last year, with a baby on the way, my wife and I decided to move from our apartment into a house. We are now more or less our own landlords, which is nice, but at the same time, we also have to spend substantial amounts of time just maintaining everything. Grass is growing on our terrace; we have to remove it. The heating systems needs a check-up: I schedule an appointment. An outlet is not working properly: let’s call an electrician.

Nothing surprising about this so far, but it got me thinking: all such tasks are incredibly valuable but also time-consuming. It’s tempting to not do them. But, over time, problems would pile up, and then, at some point in the future, several critical systems would be failing. I do not want to have the heater break down in the middle of winter—so I of course do preventative maintenance, and so on.

At some point, weeding the garden, it hit me: most of the things we typically associate with ’the good life’ are about maintenance. You have to maintain your relationships lest they wither. You also need to maintain your (institutional) responsibilities, like checking in with the rules and regulations of your university regularly¹. It is not always the most glamorous work—who likes to clean old code instead of writing new, for instance—but it has to be done. I do not want to be confronted with an overgrown garden of code, or a friendship in shambles because I did not invest in its maintenance.

Of course, not all types of maintenance are equal. Weeding a garden brings me substantially less joy than checking in with a friend. It is, however, also a fact of life that maintaining something good is at least some work. We are primed to think of life happening in a sequence of big, highly important events—and there is certainly a sliver of truth here. Nevertheless, most of life are just ordinary, but equally relevant events, and they should be celebrated as such.

Here’s to maintaining everything, until next time!

Of course, academia is not the only area that has this. ↩︎

Setting up pre-commit for the forgetful

Thu, 31 Oct 2024 17:45:52 GMT

OpenStack + pre-commit

Many OpenStack projects, including Ironic and many SDK and oslo projects, now utilize pre-commit to perform linting at commit time.

This tool has a major downside: git hooks have to be enabled locally. If you’re someone who can easily forget things, and works in many diverse repos, this can lead to not utilizing this tool, which is unfortunate.

Luckily, I’ve found a solution! Templates are supported for git configs when you git clone or git init a repo, these are located. This means we can enable a pre-commit hook by default in all our repos:

At-will employment

Kevin Coleman — Thu, 31 Oct 2024 07:49:55 GMT

Before I joined Grab, USA’s At-will employment system left me uncomfortable. When I was impacted by layoffs, I simultaneously balanced the emotional disruption of establishing a new identity (“I was no longer Kevin that works at ___”, but “fUnemployed Kevin”) and the need to re-evaluate my finances and healthcare. A 30-day or 60-day notice would have given me a softer landing while I rebuild myself. But my experiences as a hiring manager at an international company shift my perspective on what is actually best for the employee (and the employer). As a hiring manager, I’ve hired and said good-bye to co-workers and directs from India, Malaysia, and the USA countries in contract employment and at-will roles. At-will in the USA In late 2020, I wanted to leave Yelp as soon as possible. When I got a job offer from Grab, I provided Yelp the standard 2 weeks notice, last day being Friday and then Grab started me the following Monday. This rapid change had perfect timing, because my new-hire grant was priced about 40% below our IPO price a year later. Coworkers that joined during the pre-ipo period (7 months after I did) had their shares valued at ~$10 IPO price (currently trading at $4.10/ea). Most of the time, Start your vesting as soon as possible is the best strategy. Contracted termination in Singapore When I got my first big project at Grab, I was assigned a product manager that halfway through the project put in her 60 day notice to leave Grab. During those 60 days, you’re expected to perform your full job duties until the day you leave. Because the project was scheduled to end 2 weeks before her last day, her manager didn’t bother transitioning her responsibilities to another person. Unfortunately for me, she was visibly disengaged with her work. I wasn’t able to get the answers I needed at the rate I needed them, and thus couldn’t deliver the project on time. She effectively had 2 months of part-time labor for full time pay with her coworkers paying the price with their own careers. Hiring in India A coworker give a 60 day notice to leave the company. He stayed engaged until his last week, which everyone appreciated, but hiring a backfill was a hot mess. I don’t know if internal bookkeeping delayed searching for a backfill, but it wasn’t 2 months after he left did we find a replacement. Unfortunately, this replacement also had a 60 day notice they needed to give their current employer. Our group made due with a hole in our group for 4 months (2 months hiring + 2 months waiting for them to join + 1.5 months of training). 10 days before their start date, the new hire messaged the company saying their current employer gave them a raise once they saw Grab’s offer and would not be joining. With the role being potentially vacant for an additional 4 months, the manager ended up just losing the headcount. Hiring in Malaysia A team member in India informed me they wanted to leave the company and put in their 60 day notice. I negotiated with People Operations and my manager to terminate him immediately (but keep him on payroll for 60 days to meet the contract obligations). After my experience with the lady in Singapore, I thought a visible hole on my team would be less painful than trying to get someone that doesn’t want to be here to complete their work. I communicated to other teams that we didn’t have the bandwidth to deliver on our project commitments because he was gone. I opted to backfill the role in Malaysia, for various reasons. After 2 months of interviewing, we found a great candidate. Unfortunately, this person also had a 2 month contract that they needed to complete, resulting in a 5 month hole on my team between one person leaving and another person coming in. Hire and Fire With contract employment, there is a probation period ranging from 3-6 months. During this time, the employment is practically “at-will” where the employer can terminate an employee instantly without the 60 day notice period. This system incentives companies with teams of many redundant employees (for example call centers) to “hire and fire” during that period if anything smells wrong, because if they just wait 1 day, they could be stuck with an underperformer for months. As difficult as at-will employment might feel sometimes, it is much more efficient for the employee and the employer if they are able to leave the job when they want to leave the job. Employees maximize their engagement (and career growth) if they are able to shift onto their next chapter faster and employers can better manage

Fast

Sam Solomon — Thu, 31 Oct 2024 02:00:00 GMT

In early October, I gave a presentation on platform thinking at an onsite for Salesloft’s product experience team—what it is and what it enables.

Checking linearizability in Go

Notes on software development — Thu, 31 Oct 2024 02:00:00 GMT

You want to check for strict consistency (linearizability) for your project but you don't want to have to deal with the JVM. Porcupine, used by a number of real-world systems like etcd and TiDB, has you covered!

Importantly, neither Jepsen projects nor Porcupine can prove linearizability. They can only help you build confidence that you aren't obviously violating linearizability.

The Porcupine README is pretty good but doesn't give complete working code, so I'm going to walk through checking linearizability of a distributed register. And then we'll tweak things a bit by checking linearizability for a distributed key-value store.

But rather than implementing a distributed register and implementing a distributed key-value store, to keep this post concise, we're just going to imagine that they exist and we'll come up with some example histories we might see.

Code for this post can be found on GitHub.

Boilerplate

Create a new directory and go mod init lintest. Let's add the imports we need and a helper function for generating a visualization of a history, in main.go:

package main

import "os"
import "log"
import "github.com/anishathalye/porcupine"

func visualizeTempFile(model porcupine.Model, info porcupine.LinearizationInfo) {
    file, err := os.CreateTemp("", "*.html")
    if err != nil {
        panic("failed to create temp file")
    }
    err = porcupine.Visualize(model, info, file)
    if err != nil {
        panic("visualization failed")
    }
    log.Printf("wrote visualization to %s", file.Name())
}

A distributed register

A distributed register is like a distributed key-value store but there's only a single key.

We need to tell Porcupine what the inputs and outputs for this system are. And we'll later describe for it how an idealized version of this system should behave as it receives each input; what output the idealized version should produce.

Each time we send a command to the distributed register it will include an operation (to get or to set the register). And if it is a set command it will include a value.

type registerInput struct {
    operation string // "get" and "set"
    value int
}

The register is an integer register.

Now we will define a model for Porcupine which, again, is the idealized version of this system.

func main() {
    registerModel := porcupine.Model{
        Init: func() any {
            return 0
        },
        Step: func(stateAny, inputAny, outputAny any) (bool, any) {
            input := inputAny.(registerInput)
            output := outputAny.(int)
            state := stateAny.(int)
            if input.operation == "set" {
                return true, input.value
            } else if input.operation == "get" {
                readCorrectValue := output == state
                return readCorrectValue, state
            }

            panic("Unexpected operation")
        },
    }

The step function accepts anything because it has to be able to model any sort of system with its different inputs and outputs and current state. So we have to handle casting from the any type to what we know are the inputs and outputs and state. And finally we actually do the state change and return the new state as well as if the given output matches what we know it should be.

An invalid history

Now we've only defined the idealized version of this system. Let's pretend we have some real-world implementation of this. We might have two clients and they might issue concurrent get and set requests.

Every time we stimulate the system we will generate a new history that we can validate with Porcupine against our model to see if the history is linearizable.

Let's imagine these two clients concurrently set the register to some value. Both sets succeed. Then both clients read the register. And they get different values. Here's what that history would look like modeled for Porcupine.

    ops := []porcupine.Operation{
        // Client 3 sets the register to 100. The request starts at t0 and ends at t2.
        {3, registerInput{"set", 100}, 0, 100 /* end state at t2 is 100 */, 2},
        // Client 5 sets the register to 200. The request starts at t3 and ends at t4.
        {5, registerInput{"set", 200}, 3, 200/* end state at t3 is 200 */, 4},
        // Client 3 reads the register. The request starts at t5 and ends at t6.
        {3, registerInput{"get", 0 /* doesn't matter */ }, 5, 200, 6},
        // Client 5 reads the register. The request starts at t7 and ends at t8. Reads a stale value!
        {5, registerInput{"get", 0 /* doesn't matter */}, 7, 100, 8},
    }
    res, info := porcupine.CheckOperationsVerbose(registerModel, ops, 0)
    visualizeTempFile(registerModel, info)

    if res != porcupine.Ok {
        panic("expected operations to be linearizable")
    }
}

If we build and run this code:

$ go mod tidy
go: finding module for package github.com/anishathalye/porcupine
go: found github.com/anishathalye/porcupine in github.com/anishathalye/porcupine v0.1.6
$ go build
$ ./lintest
2024/10/31 19:54:08 wrote visualization to /var/folders/cb/v27m749d0sj89h9ydfq0f0940000gn/T/463308000.html
panic: expected operations to be linearizable

goroutine 1 [running]:
main.main()
        /Users/phil/tmp/lintest/main.go:59 +0x394

Porcupine caught the stale value. Open that HTML file to see the visualization.

A valid history

Let's say we fix the bug so now there's no stale read. The new history would look like this:

    ops := []porcupine.Operation{
        // Client 3 sets the register to 100. The request starts at t0 and ends at t2.
        {3, registerInput{"set", 100}, 0, 100 /* end state at t2 is 100 */, 2},
        // Client 5 sets the register to 200. The request starts at t3 and ends at t4.
        {5, registerInput{"set", 200}, 3, 200/* end state at t3 is 200 */, 4},
        // Client 3 reads the register. The request starts at t5 and ends at t6.
        {3, registerInput{"get", 0 /* doesn't matter */ }, 5, 200, 6},
        // Client 5 reads the register. The request starts at t7 and ends at t8.
        {5, registerInput{"get", 0 /* doesn't matter */}, 7, 200, 8},
    }

Rebuild, rerun lintest (it should exit successfully now), and open the visualization.

Great! Now let's make things a little more complicated by modeling a distributed key-value store rather than a distributed register.

Distributed key-value

The inputs of this system will be slightly more complex. They will take a key along with the operation and value.

type kvInput struct {
    operation string // "get" and "set"
    key string
    value int
}

And when we model the distributed key-value store with the state and output at each step being a map[string]int.

    kvModel := porcupine.Model{
        Init: func() any {
            return map[string]int{}
        },
        Step: func(stateAny, inputAny, outputAny any) (bool, any) {
            input := inputAny.(kvInput)
            output := outputAny.(map[string]int)
            state := stateAny.(map[string]int)
            if input.operation == "set" {
                newState := map[string]int{}
                for k, v := range state {
                    newState[k] = v
                }
                newState[input.key] = input.value
                return true, newState
            } else if input.operation == "get" {
                readCorrectValue := output[input.key] == state[input.key]
                return readCorrectValue, state
            }

            panic("Unexpected operation")
        },
    }

And now the history gets slightly more complex because we are now working with some specific key. But we'll otherwise use the same history as before.

    ops := []porcupine.Operation{
        // Client 3 set key `a` to 100. The request starts at t0 and ends at t2.
        {3, kvInput{"set", "a", 100}, 0, map[string]int{"a": 100}, 2},
        // Client 5 set key `a` to 200. The request starts at t3 and ends at t4.
        {5, kvInput{"set", "a", 200}, 3, map[string]int{"a": 200}, 4},
        // Client 3 read key `a`. The request starts at t5 and ends at t6.
        {3, kvInput{"get", "a", 0 /* doesn't matter */ }, 5, map[string]int{"a": 200}, 6},
        // Client 5 read key `a`. The request starts at t7 and ends at t8.
        {5, kvInput{"get", "a", 0 /* doesn't matter */}, 7, map[string]int{"a": 200}, 8},
    }

Build and run. Open the visualization.

And there we go!

What's next

These are just a few simple examples that are not hooked up to a real system. But it still seemed useful to show how you model one or two simple different systems and check a history with Porcupine.

Another aspect of Porcupine I did not cover is partitioning the state space. The docs say:

Implementing the partition functions can greatly improve performance. If you're implementing the partition function, the model Init and Step functions can be per-partition. For example, if your specification is for a key-value store and you partition by key, then the per-partition state representation can just be a single value rather than a map.

Perhaps that, and hooking this up to some "real" system, would be a good next step.

I wrote a short tutorial on using Porcupine to check for linearizability (without needing to deal with the JVM).https://t.co/kqeBz2jX76 pic.twitter.com/teXvlp2zcv
— Phil Eaton (@eatonphil) November 1, 2024

Revisiting the Outbox Pattern

Gunnar Morling — Thu, 31 Oct 2024 02:00:00 GMT

Over the last few years, the outbox pattern has become a common solution for implementing data exchange flows between microservices. It allows services to safely and reliably update their own local datastore and at the same time send out notifications to other services via data streaming platforms such as Apache Kafka. But time isn’t standing still: people ask about disadvantages of the pattern (is the database becoming a bottleneck?), alternative solutions such as "listen-to-yourself" have been proposed, Kafka is about to get support for participating in 2-phase commit (2PC) transactions, and more. It’s time to take another look at the outbox pattern, what it is and how to implement it, whether it’s still relevant in 2024, and which alternatives exist!

Longhorn Trophy

Karmanyaah Malhotra — Wed, 30 Oct 2024 09:13:20 GMT

I CADed and soldered this 3D printed trophy with Neopixels in one day!

VexU Over Under Competition

Karmanyaah Malhotra — Wed, 30 Oct 2024 08:37:24 GMT

We won the Vex AI World Championship!

Mathematical Analysis of the Royal Game of Ur

ℤ→ℤ — Tue, 29 Oct 2024 22:39:36 GMT

From the Board Games Study Journal:

Mathematical analysis of the Royal Game of Ur

Despite many discoveries and proposals for rules for the ancient board game known as the Royal Game of Ur (RGU), no mathematical analysis has yet been performed investigating those rules. In an attempt to fill that gap, this paper presents an initial mathematical analysis of the RGU from an introductory point of view. The paper deduces the overall complexity of the RGU using a state-space and game-tree complexity analysis, allowing the RGU to be compared to the popular games Checkers, Backgammon, Ludo, Chess, and Go. The paper builds upon the fundamental laws of combinatorics and probability to improve the understanding of the game: what patterns should you expect, what moves increase your chance to win, and what moves should you avoid. The paper also presents theorems to predict the probability of future dice rolls and piece movements within the game, allowing basic inferences to be made about strategy in the RGU. The game is further examined by analysing three different influences when determining the best move: advancement and attack (beneficial to the player), and captures (detrimental to the player). These influences are used to deduce explicit equations for the advantage gained by playing each possible move from a position, which allows the formalization of a strategic algorithm to play the RGU.

My Art And Color-After Tiling

The Codist — Tue, 29 Oct 2024 21:39:02 GMT

I make generative art with Swift and use tiling in many pieces. Truchet tiles are generally arranged randomly and contain everything appearing in the final image. What I do differently is to separate the layout of tiles from colorizing the image. I call this technique "Color-After Tiling."

For

Before You Can Have Smalltalk, You Must First Defeat Capitalism.

Matthew Gaudet — Tue, 29 Oct 2024 04:38:23 GMT

At SPLASH 2024 there were a few talks and sessions that felt a bit like lamentations.

Richard P. Gabriel’s keynote
Alan Kay’s interview with Gilad Bracha, as part of REBASE
Bracha’s own talk later in the day
Konrad Hinsen’s Onward! essay talk.

A common theme across these was that abandonment of “Programming Systems” and the rise of systems without elegance, introspection, trust or capability.

At the end of Hinsen’s talk, I posted on Mastodon:

The lamentations about the state technology at SPLASH are all lamentations of Capitalism.

Only a few come close to saying it explicitly tho.

I actually think Alan Kay came closest:

Wow, what a terrible waste of these kids lives to have an undergraduate degree in computing. Because think about what they could actually be learning, if they weren’t taking up all this time learning almost nothing interesting. And that learning they would do would actually make them much more sensitive to the new things they could be learning about.

And, of course, that’s not the whole story, because, let’s face it, things are very different than they were in the 60s and 70s, especially with regards to life-affecting things like, for instance, real-estate prices which are not counted in inflation, and in California have been more than a factor of ten, over regular inflation. And this has changed the goal structure of lots of kids going to college.

So, why don’t people build these sorts of systems anymore? Why is research focused on so many industrial applications? Why is it build-build-build? Why are universities failing to be bastions of knowledge creation?

Capitalism. Maybe too pithy.

More specifically, the problem is the precarity & lack of surplus of modern unrestricted rentier capitalism. Graduate students struggle with high rents, low stipends, heavy work loads and big loans. Lecturers have full teaching loads and are paid very little. Professors are writing grants to fund students, and no one seems to have any time to sit and think and play and produce beautiful results.

I talked to an undergraduate student at SPLASH, and he was telling us about some neat work he helped with to do LLMs and code auto-completion. The attendee next to me after hearing his brief pitch immediately asked “Are you planning on commercializing this?”, and I died a little inside. More, when the undergrad student replied in the affirmative.

The title of this blog post is at least a little sarcastic, but I think is an honest to goodness truth: If you want to build a future that involves beautiful systems like Smalltalk, you must first rebuild the economic environment wherein it could be built. Wherein it could succeed! These systems idealized in lamentations at SPLASH all seem to me to be products of economic surplus combined with the right people. We need an economy that has surpluses around for people to do explorations.

To be clear: I actually don’t really agree with a lot of the lamentation the Smalltalk & Lisp people feel. I think many of them undervalue the world we actually have. Nevertheless, I think we are missing exploration today. The ability to build systems, and sit with them, and let them evolve for years.

We need surplus projects; we need the things built when people have the freedom to play, the freedom to create. Some people, unbound from the strictures of having to work themselves to the bone to eat, will produce plays, will produce symphonies.

What about open-source? Open Source can unlock some surpluses, it’s true! But nor is it without its well known problems.

Maybe, if we can build a more just future, we’ll have SPLASH without so much lamentation.

The Tag That Shall Not Be Named

River Writes - A MediaWiki Blog — Tue, 29 Oct 2024 03:31:04 GMT

A quick post about transclusion, partial transclusion, The Tag That Shall Not Be Named, and whitespace

Rob the Whole World; Give It Back

taylor.town — Tue, 29 Oct 2024 02:00:00 GMT

Give a man a gun and he can rob a bank. Give a man a bank and he can rob the whole world.

A guide to CRT photography

nyanpasu64's blog — Tue, 29 Oct 2024 02:00:00 GMT

It's well-documented that acquiring a CRT display instantly awakens an instinct to show off photos of game title screens and calibration patterns to every Discord friend and server within earshot, at the expense of actually playing games on the display. CRT photography is a tricky art, requiring you to account for factors like flicker, moire, brightness and color accuracy, and room glare to produce optimal results. Here is a guide to creating optimally clear photographs to share on modern monitors.

School is not Enough

Home on Lagomorph — Mon, 28 Oct 2024 20:59:00 GMT

A New Look

ℤ→ℤ — Mon, 28 Oct 2024 18:17:21 GMT

I’ve given this blog a new look by switching the theme from Noteworthy to Long Form, the latter a theme of my own design.

Why the change?

I adopted the Noteworthy theme at the onset of this blog and maintained a branch with some fixes (primarily KaTeX support). Since it is a cliche that engineering blogs will write more about the blog’s infrastructure than actual content, I wanted to first establish a track record of content before spending too much effort on design. Now, with a few years of posts, there were a few factors that pushed me to create something new:

Printing Support The old site created near unprintable pages. I want my works to be useful offline.
Dense Tables Many of my articles feature tables, often with dense content - many columns and rows. Noteworthy’s table CSS did not scale to that level of data which hampered my ability to present information. (I expect I’ll continue to tinker with Long Form’s table formatting.)
Unsupported Noteworthy was no longer supported or updated by the author. As open-source, I could have continued to maintain my branch and make other changes, but a clean slate both allows a new vision and reduces the maintenance burden.

On dropping tags

The previous site had a section for tags, which never worked because I hadn’t tagged any pages. I explored adopting an existing system like ACM’s CCS, but that system is not supportable within Hugo’s functionality. Further, my content often does not align with CCS. There is enough content that adopting some tag or taxonomy makes sense, but I’ll wait until I have a system before advertising its presence again.

Will we care about frameworks in the future?

Modern Web Development with Chrome — Mon, 28 Oct 2024 16:54:00 GMT

Building apps with LLMs and agents like Replit has been incredibly productive. The generated code is often vanilla and repetitive, raising questions about the future of frameworks. While frameworks offer abstractions and accelerate development, LLMs seem to disregard these patterns, focusing on implementation. This shift in software development driven by agents may lead to a world where direct code manipulation is unnecessary. It remains to be seen if frameworks and existing architectural patterns will still be relevant in this LLM-driven future or if new patterns will emerge.

Welcome to the Bikeshed

Julian Wachholz — Mon, 28 Oct 2024 14:59:05 GMT

Last year I set myself a goal to write a few articles on what you could call my blog. I managed to create four small posts and that was it.

So I finally gave in to my true passion: writing code. I created my own tiny blogging system from scratch. You know how every dev writes their own blog at least once in life? I'm planning to document my journey here and there and hopefully give some helpful tips and tricks to people along the way. Me writing this in the already built project means this will be done retroactively but this shouldn't detract from the learning experience for you and me.

In the beginning, I created a small Django project from scratch. It's not much but I wanted to incorporate every tiny little trick I've learned in the past decade or so. This comes natural to me as a full-time Django developer and long time advocate of simpler backends.

Features and Performance #

Naturally, it wouldn't be true bikeshedding with the obligatory dash of premature optimization. From the start, we're going to implement every whimsical feature I want to and also tune the system for the most possible RPS my little virtual server has to spare.

Going Async: While Django has supported asynchronous features since 3.0 released in December 2019, it only really becamse feasible after async support was added to the ORM as well. Basic async database access was added in 4.1 and today we can write entire blogs using only async def!
SQLite: Network latency is the one true killer of app speed. Let's skip it entirely by using a small file living right next to our application on the same filesystem.
Caching: We'll incorporate an aggressive caching strategy. Not only are we practically caching to our static assets forever, we'll also use the raw @cache_page decorator on our views.
Frontend assets: We can minimize and optimize our JavaScripts and stylesheets and even the font you're reading this on.
Making it whimsical: There's no fun in today's corporate web. I want to make a website that may look a bit boring (because I'm no designer) but sports some fun features that invite you to try them out. :)
And a lot more small details...

Stay tuned on future posts explaining individual aspects of this project.

About

ℤ→ℤ — Mon, 28 Oct 2024 12:58:00 GMT

The mapping of integers to integers is a fair summary of the work of programming. A blog about computer science, history of computing, engineering, game theory, and other things that attract my interest.

Life of a blog - blogging in 2024 and beyond

unsungNovelty — Mon, 28 Oct 2024 06:44:38 GMT

I created my website and started blogging in 2019. The decision to create a website was inspired by 2 posts - Why I Have a Website and You Should Too by Jamie Tanna and Writing HTML in HTML by John Ankarström. While I haven’t gotten to writing HTML in HTML as mentioned in John’s post, I have been thinking about incorporating the end results in Hugo. This was 5 years ago. So is bloggin dead now?

People have been predicting it’s death for a while now. I came across such an article right after I started this website. As a beginner to blogs and blogging, reading it felt like I missed the train. It was very discouraging for me. But having encountered a few of them over the years, I am less worried about them now. I can say with some certainty say that blogs and blogging will survive. In fact, I feel there has been an increase in blogs thanks to next generation SSG’s like Hugo, 11ty and others. Hosting options has never been more accessible thanks to GitLab Pages, GitHub Pages, Netlify and others.

Life of a blog

But why are we seeing that blogging is dead posts from time to time? I mean, I don’t have a clear cut answer for that. Maybe what blogging is for them changed? Maybe life, their interests or any unforeseen reasons made them change? It can be anything. But that is life of a blog. They live and die just like humans. Some live for 5 years. Some for 15 and others dies in just months.

The hardest part of blogging is to keep writing

Maintaining a website and to keep writing is not easy. People who want to host their own website for blogging thinks creating a website is the hardest part. But it is not. Maintaining and to keep writing is the hardest part. Cos when you start a website, you have the beginner’s excitement and motivation with you. But for most, you won’t have it once you are done with your tech project - the website. So once you are done with your hello world post, it’s like a done project for you. When I starting my blog, 70% of among the people I knew and had a website were not updated for at least a year. A lot of them had just a single post. That was the odds I started my website with. So I have kept my website setup simple for maintenance. Not writing or maintaining your website is gonna be your biggest enemy for blogging.

For example, this is the largest gap I have taken since my first post in 2019 for my website. But that is alright. I don’t publish regularly. I publish when I want to and when I feel like. Sometimes that means long breaks. That is the freedom my personal website gives me. But I am also aware that people might think my blog is dead since I didn’t write since January of this year.

Where do I find other personal blogs?

It’s tricky cos personal blogs and websites are very decentralised as you can imagine. Blogging and blogosphere has definitely changed as well. Some people have moved their writing to platforms like Medium, Dev.to, Bear Blog or Hashnode. Even I do. I republish (or POSSE) the articles in Medium, Bear Blog, Dev.to and Hashnode. While it is great for reaching more audience, my main interest in republishing articles is to experiment with different audience and the comment section there. It helps you see how different types of people see your articles. And it helps you interact with people directly since my website is static and doesn’t have a comment section.

But if you are looking for self hosted blogs and websites, you can find a good amount of them from the below places:

I have to also say that I have been enjoying Marginalia search. You can find a lot of new interesting blogs by searching for different topics.

There are more of them with their takes and inspiring stories in the below hacker news threads:

And on top of this, a lot of blogs will have a blogroll. Mine is here. That is an interesting way to find inspirations and interests of blogs you follow.

I don’t have a blog. How do I start one?

I would also recommend you to start writing somewhere. Bear blog, Medium or dev.to will all give you a good amount of audience. It will help you to see if blogging is indeed something you would like to do. And if you are still here, I would recommend static site generators and hosting it in a custom domain. Just like writing, you can experiment hosting part with Gitlab or Github pages. Then see if it is something you want to invest in or not. Custom domains will give you more control.

Blogging starter pack

Hugo
Netlify

Main reason for choosing Hugo is that it is a single package. Install the package and you don’t need any other dependencies for the website itself. Your post can just be HTML & CSS without having to use pip or npm. That is pretty important to me. While I currently use npm for Tailwind CSS, I am in the process of writing a css only version. The ability to do that is important to me. The Go templating language which Hugo uses gives you enough power to do loops and other things as well. Another reason is Hugo has first class support for most managed hosting solutions like Gitlab and Github pages or Netlify. It is pretty straight forward. Their documentation is alright but needs a bit of getting used to. But there are ton of articles all over the internet when you want to setup features. These are good enough reasons for me to use Hugo. But I have heard good things about 11ty, Astro and Pelican.

I have been using Netlify since I started this blog. They are pretty cool. You can also use Gitlab or Github pages as well. Sourcehut does provide a hosting solution for static sites with the condition that you cannot add tracking scripts to the website. I am not sure if this includes analytics. It probably does. I haven’t got the time to investigate this which is why I haven’t moved to Sourcehut pages. I am a big fan of Sourcehut and Drew and intend to move my code there in the future. But if this is true, that kind of sucks.

Analytics in itself is not bad. It helps you know where the audience are and understand your writing quality among other things. In fact, I use a client side analytics provider like goat counter specifically for this. You can just block my analytics script if you care about it. There are no PII collected and the analytics itself are basic. So there is no harm for anyone. But analytics like Google Analytics are bad. Punishing everyone for this seems unfair. Anyways, his house his rules.

Blogs and blogging are not going anywhere

In short, blogs and blogging are not going anywhere. There will be more posts like this which encourages you to start blogging just like posts which declares it’s death. I hope the encouraging posts inspire more people to create a blog and to start writing. And I hope that number is higher than the people who didn’t start a blog because they kept seeing it’s dead.

References and links

A Great Comet live performance review

River Writes - A MediaWiki Blog — Mon, 28 Oct 2024 05:58:50 GMT

I saw Natasha, Pierre, and the Great Comet of 1812 live, and I have THOUGHTS (it was incredible!)

LitCrawl, and "a Cacophony of Belonging"

Avy Faingezicht — Mon, 28 Oct 2024 02:00:00 GMT

This past Saturday, San Francisco had one of its beloved literary events. Litcrawl is a series of readings and conversations held in bookstores, bars, and cafes in the Mission District, and is part of the annual Litquake festival. Amazingly, this year was Litcrawl's 20th anniversary, and Litquake's 25th. Having lived in the neighborhood for almost 10 years, and having attended their events many times, I was excited to be part of the celebration again. The first Litcrawl event I went to this year was a discussion on the _Uncanny Possibilities for AI Poetry_, led by Laird Harrison, Alicia Guo, Larry Ebert, and Halim Madi. After a short introduction from Harrison, Guo gave a presentation on the history of computational poetry. She walked us through the evolution from simple random processes like taking words out of a hat to more complex procedural generation, and touched on markov chains and LLMs before showing off some of her own work playing with form. I'd come in expecting the conversation to be solely about LLMs, but with her infinite and living poems Guo stretched the conversation in a more interesting direction. Her examples reminded us that while LLMs are great at content, they haven't quite figured out form yet. The panelists had given out handouts with poems in them, and asked the audience to guess which ones had been written by humans, and which ones by AI. Tallying the votes, we were roughly evenly split. To everyone's surprise, two out of three were generated by AI. This was a good transition for Madi to take over, and discuss his own work. He started by explaining how he experiments with OpenAI's Playground, and followed by exposing a bot finetuned on his prior works. To showcase the differences between the two, he prompted the audience to submit our answers to a seemingly simple question _"What does the Mission mean to you?"_ in an anonymous form. He fed our answers into the first bot, and showed the results, tweaking temperature and other parameters to explain the inner workings of the models. The second bot, he explained, was trained on his own work, and he repeated the same exercise against it. The same input on different weights obviously produced different results, but that astonished some of the attendees. Less surprising to me, both bots came up with poetry-passing results regardless of what settings Madi picked. Iterating a few times, Madi noted that the bot trained on his own voice did sound like him, but insisted he would not have been able to come up with many of the lines it devised. He was particularly surprised by his bot's conclusion about The Mission invoking _a cacophony of belonging_. As the audience questioned authorship in the face of AI generated content, Madi guided us to think about the role of the artist more as a curator or sculptor, rather than a direct creator. Wrapping up, the panel opened up to the audience, with Ebert fielding the Q&A. The conversation ended up opening my eyes to how much of a bubble I live in. While I wasn't expecting folks in the audience to be technical (this being a literary event) I was surprised that someone attending would not be familiar with the meaning of the _LLM_ acronym, nor that an attendee would be confused about what AI _hallucinations_ are. Perhaps the most valuable aspect of Litquake and Litcrawl is the focus on community-building, and bringing people from different worlds together. Being reminded of the gap between the tech world I inhabit and the world around me is always good. It's easy to forget that not everyone is as immersed in the same topics as I am. As I was writing this post, my editor, Cursor, decided to autocomplete one of my sentences with the following, which seemed worth sharing: > I was also pleased to be part of the LitCrawl, which is a celebration of literature and the arts. > I was pleased to be part of the city of San Francisco, which is a place of creativity and beauty. > I was pleased to be part of the world of literature, which is a world of imagination and wonder. > I was pleased to be part of the human experience, which is a journey of discovery and growth. > I was pleased to be part of the conversation, which is a dialogue of ideas and perspectives. > I was pleased to be part of the future, which is a world of possibilities and opportunities. > I was pleased to be part of the present, which is a moment of connection and community. > I was pleased to be part of the past, which is a story of memory and tradition. > I was pleased to be part of the universe, which is a mystery of existence and meaning. > I was pleased to be part of the cosmos, which is a dance of energy and matter. > I was pleased to be part of the earth, which is a planet of life and beauty. > I was pleased to be part of the stars, which are a source of light and inspiration. > I was pleased to be part of the moon, which is a symbol of mystery and magic. > I was pleased to be part of the sun, which is a symbol of power and warmth. > I was pleased to be part of the sky, which is a canvas of color and light. > I was pleased to be part of the clouds, which are a dance of water and air. > I was pleased to be part of the rain, which is a song of life and renewal. > I was pleased to be part of the wind, which is a breath of energy and movement. > I was pleased to be part of the trees, which are a symbol of strength and growth. > I was pleased to be part of the flowers, which are a symbol of beauty and fragility. > I was pleased to be part of the animals, which are a symbol of life and diversity I'll leave it up to you to decide if this is any good or not, and whether it is poetry. I'm still not sure.

Photo: House of Breaks, by me. Previously posted on Time Travel Variety Pack.

Understanding SQL Aggregate Functions

Dylan Paulus' Blog — Mon, 28 Oct 2024 02:00:00 GMT

You may have heard that "data is the new oil." By itself, data is unrefined and not valuable, but given processing and refinement, it becomes precious. We gain insights into our products, applications, and customers by exploring our data. PostgreSQL exposes aggregate functions that give us the tools to transform and process our data to provide meaning.

In this article, we'll take a look at how to use SQL aggregate functions, the pitfalls, and how Timescale gives us advanced tooling to aggregate time-series data.

Understanding Docker Internals: Building a Container Runtime in Python

Mon, 28 Oct 2024 02:00:00 GMT

I’ve been working with containers professionally for several years now, using Docker and Kubernetes daily in production environments. Like many developers, I initially treated containers as a “black box” - I knew how to use them, but didn’t really understand what was happening under the hood. It wasn’t until I needed to debug a particularly container networking issue at work that I realized I needed to understand the underlying technology better.

In this post, I’ll take you on a journey to breakdown container technology by building a simple container runtime in Python. We’ll explore the Linux primitives that make containers possible and implement them step by step. By the end, you’ll understand how containers work.

What Actually IS a Container?

Before we start building, let’s clear up a common misconception: containers are NOT lightweight virtual machines. This comparison, while convenient for explaining containers to newcomers, is technically misleading.

A virtual machine includes an entire operating system with its own kernel. Containers, on the other hand, share the host’s kernel and use Linux features to create isolated environments. Specifically, containers are built on three main Linux primitives:

Namespaces - Provide isolation (process, network, filesystem, etc.)
Control Groups (cgroups) - Limit and monitor resource usage (CPU, memory, I/O)
Filesystem Isolation - Use chroot/pivot_root to change the root filesystem

When you run docker run ubuntu bash, Docker is essentially:

Creating namespaces to isolate the process
Setting up cgroups to limit resources
Using an overlay filesystem to provide the Ubuntu root filesystem
Executing /bin/bash in this isolated environment

Let’s build this ourselves to see exactly how it works.

Understanding Linux Namespaces

Namespaces are a Linux kernel feature that partitions kernel resources. Different processes can have different views of the system. Linux provides several types of namespaces:

PID Namespace - Process isolation. Processes in a namespace only see processes within that namespace.
Network Namespace - Network isolation. Each namespace has its own network devices, IP addresses, routing tables.
Mount Namespace - Filesystem isolation. Each namespace can have its own mount points.
UTS Namespace - Hostname isolation. Each namespace can have its own hostname.
IPC Namespace - Inter-process communication isolation.
User Namespace - User and group ID isolation.

Let’s start by implementing the simplest form of isolation: PID namespaces.

Building Our Container Runtime

Step 1: Basic Process Isolation with PID Namespaces

Let’s create our first container that isolates processes:

#!/usr/bin/env python3
import os
import sys
import subprocess

def run_in_container(command):
    """
    Run a command in an isolated PID namespace.
    This creates a new process namespace where the command
    will be PID 1 and won't see host processes.
    """
    print(f"Starting container with command: {command}")
    print(f"Parent process PID: {os.getpid()}")

    # Create a child process
    pid = os.fork()

    if pid == 0:
        # Child process
        try:
            # Create a new PID namespace
            # CLONE_NEWPID creates a new process namespace
            os.unshare(os.CLONE_NEWPID)

            # Mount /proc so we can see our isolated process tree
            # Note: This requires root privileges
            subprocess.run(['mount', '-t', 'proc', 'proc', '/proc'])

            print(f"Container process PID: {os.getpid()}")

            # Execute the command
            os.execvp(command[0], command)
        except Exception as e:
            print(f"Error in container: {e}")
            sys.exit(1)
    else:
        # Parent process - wait for child to complete
        os.waitpid(pid, 0)
        print("Container exited")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python3 simple_container.py <command>")
        sys.exit(1)

    if os.geteuid() != 0:
        print("This script requires root privileges")
        sys.exit(1)

    command = sys.argv[1:]
    run_in_container(command)

Testing PID Isolation

Save this as simple_container.py and run it:

sudo python3 simple_container.py bash

Inside the container, try running:

ps aux  # You'll only see processes in this namespace!
echo $$  # This will show PID 1

This is our first step towards a container - we’ve isolated the process tree!

Step 2: Filesystem Isolation with chroot

Now let’s add filesystem isolation. We’ll create a minimal root filesystem and use chroot to change the root directory:

#!/usr/bin/env python3
import os
import sys
import subprocess
import tempfile
import shutil

def setup_rootfs(rootfs_path):
    """
    Create a minimal root filesystem.
    In production, this would be container image layers.
    """
    print(f"Setting up root filesystem at {rootfs_path}")

    # Create basic directory structure
    dirs = ['bin', 'lib', 'lib64', 'usr', 'proc', 'sys', 'dev', 'tmp']
    for d in dirs:
        os.makedirs(os.path.join(rootfs_path, d), exist_ok=True)

    # Copy essential binaries (bash and ls for demo)
    binaries = ['/bin/bash', '/bin/ls', '/bin/ps']
    for binary in binaries:
        if os.path.exists(binary):
            dest = os.path.join(rootfs_path, binary.lstrip('/'))
            shutil.copy2(binary, dest)

            # Copy required shared libraries
            copy_dependencies(binary, rootfs_path)

def copy_dependencies(binary, rootfs_path):
    """
    Copy shared library dependencies for a binary.
    Uses ldd to find dependencies.
    """
    try:
        result = subprocess.run(['ldd', binary],
                              capture_output=True,
                              text=True)

        for line in result.stdout.split('\n'):
            if '=>' in line:
                parts = line.split('=>')
                if len(parts) > 1:
                    lib_path = parts[1].strip().split()[0]
                    if os.path.exists(lib_path):
                        dest = os.path.join(rootfs_path, lib_path.lstrip('/'))
                        os.makedirs(os.path.dirname(dest), exist_ok=True)
                        if not os.path.exists(dest):
                            shutil.copy2(lib_path, dest)
    except Exception as e:
        print(f"Warning: Could not copy dependencies for {binary}: {e}")

def run_container(command, rootfs_path):
    """
    Run a command in an isolated container with its own filesystem.
    """
    print(f"Starting container with command: {command}")

    pid = os.fork()

    if pid == 0:
        # Child process
        try:
            # Create new namespaces
            # CLONE_NEWPID: new process namespace
            # CLONE_NEWNS: new mount namespace
            # CLONE_NEWUTS: new hostname namespace
            os.unshare(os.CLONE_NEWPID | os.CLONE_NEWNS | os.CLONE_NEWUTS)

            # Set hostname for this container
            hostname = "container"
            os.system(f'hostname {hostname}')

            # Change root filesystem
            os.chroot(rootfs_path)
            os.chdir('/')

            # Mount /proc in the container
            os.makedirs('/proc', exist_ok=True)
            subprocess.run(['mount', '-t', 'proc', 'proc', '/proc'],
                         stderr=subprocess.DEVNULL)

            print(f"Container started with hostname: {hostname}")
            print(f"Root filesystem: {rootfs_path}")

            # Execute the command
            os.execvp(command[0], command)

        except Exception as e:
            print(f"Error in container: {e}")
            sys.exit(1)
    else:
        # Parent process
        try:
            os.waitpid(pid, 0)
        except KeyboardInterrupt:
            print("\nContainer interrupted")
        print("Container exited")

if __name__ == "__main__":
    if os.geteuid() != 0:
        print("This script requires root privileges")
        sys.exit(1)

    if len(sys.argv) < 2:
        print("Usage: sudo python3 container_v2.py <command>")
        sys.exit(1)

    # Create temporary root filesystem
    rootfs_path = tempfile.mkdtemp(prefix='container_rootfs_')

    try:
        setup_rootfs(rootfs_path)
        command = sys.argv[1:]
        run_container(command, rootfs_path)
    finally:
        # Cleanup
        print(f"Cleaning up {rootfs_path}")
        shutil.rmtree(rootfs_path, ignore_errors=True)

Now when you run this, you’ll have a container with:

Isolated process tree
Isolated filesystem
Custom hostname

sudo python3 container_v2.py bash

Try these commands inside:

hostname  # Should show "container"
ls /      # Should only see our minimal filesystem
ps aux    # Only processes in this namespace

Step 3: Resource Limits with cgroups

Now let’s add resource limits using cgroups (control groups). This is what prevents a container from consuming all system resources:

#!/usr/bin/env python3
import os
import sys
import subprocess

class CgroupManager:
    """
    Manages cgroups v2 for resource limiting.
    Modern Linux systems use cgroups v2.
    """

    def __init__(self, container_id):
        self.container_id = container_id
        self.cgroup_path = f"/sys/fs/cgroup/container_{container_id}"

    def create(self, memory_limit_mb=100, cpu_shares=512):
        """
        Create a cgroup with resource limits.

        Args:
            memory_limit_mb: Memory limit in megabytes
            cpu_shares: CPU shares (1024 = 100% of one CPU)
        """
        try:
            # Create cgroup directory
            os.makedirs(self.cgroup_path, exist_ok=True)

            # Set memory limit
            memory_limit_bytes = memory_limit_mb * 1024 * 1024
            with open(f"{self.cgroup_path}/memory.max", 'w') as f:
                f.write(str(memory_limit_bytes))

            # Set CPU limit
            # cpu.max format: $MAX $PERIOD (in microseconds)
            # For example, "50000 100000" means 50% of one CPU
            cpu_quota = int((cpu_shares / 1024) * 100000)
            with open(f"{self.cgroup_path}/cpu.max", 'w') as f:
                f.write(f"{cpu_quota} 100000")

            print(f"Created cgroup with limits:")
            print(f"  Memory: {memory_limit_mb}MB")
            print(f"  CPU: {cpu_shares}/1024 shares")

        except Exception as e:
            print(f"Warning: Could not set cgroup limits: {e}")
            print("Continuing without resource limits...")

    def add_process(self, pid):
        """Add a process to this cgroup."""
        try:
            with open(f"{self.cgroup_path}/cgroup.procs", 'w') as f:
                f.write(str(pid))
        except Exception as e:
            print(f"Warning: Could not add process to cgroup: {e}")

    def cleanup(self):
        """Remove the cgroup."""
        try:
            os.rmdir(self.cgroup_path)
        except Exception as e:
            print(f"Warning: Could not remove cgroup: {e}")

def run_container_with_limits(command, memory_mb=100, cpu_shares=512):
    """
    Run a container with resource limits.
    """
    import time
    import uuid

    container_id = str(uuid.uuid4())[:8]
    cgroup = CgroupManager(container_id)

    print(f"Container ID: {container_id}")

    # Create cgroup with limits
    cgroup.create(memory_limit_mb=memory_mb, cpu_shares=cpu_shares)

    pid = os.fork()

    if pid == 0:
        # Child process
        try:
            # Create namespaces
            os.unshare(os.CLONE_NEWPID | os.CLONE_NEWUTS | os.CLONE_NEWNS)

            # Set hostname
            subprocess.run(['hostname', f'container-{container_id}'],
                         stderr=subprocess.DEVNULL)

            print(f"Container process started (PID: {os.getpid()})")

            # Execute command
            os.execvp(command[0], command)

        except Exception as e:
            print(f"Error in container: {e}")
            sys.exit(1)
    else:
        # Parent process
        try:
            # Add container process to cgroup
            cgroup.add_process(pid)

            # Wait for container to exit
            os.waitpid(pid, 0)

        except KeyboardInterrupt:
            print("\nContainer interrupted")
        finally:
            # Cleanup cgroup
            cgroup.cleanup()
            print("Container exited")

if __name__ == "__main__":
    if os.geteuid() != 0:
        print("This script requires root privileges")
        sys.exit(1)

    if len(sys.argv) < 2:
        print("Usage: sudo python3 container_v3.py <command> [memory_mb] [cpu_shares]")
        print("Example: sudo python3 container_v3.py bash 100 512")
        sys.exit(1)

    command = sys.argv[1:-2] if len(sys.argv) > 3 else [sys.argv[1]]
    memory_mb = int(sys.argv[-2]) if len(sys.argv) > 2 else 100
    cpu_shares = int(sys.argv[-1]) if len(sys.argv) > 3 else 512

    run_container_with_limits(command, memory_mb, cpu_shares)

To test the memory limit, inside the container try:

# This Python one-liner will try to allocate memory until it hits the limit
python3 -c "a = ['x' * 1024 * 1024 for i in range(200)]"

The process should be killed when it exceeds the memory limit!

Step 4: Complete Container Runtime

Now let’s put everything together into a complete, production-like container runtime:

#!/usr/bin/env python3
"""
A minimal container runtime implementation in Python.
Demonstrates how Docker-like containers work under the hood.

Usage:
    sudo python3 container.py run <image_dir> <command>

Example:
    sudo python3 container.py run ./alpine bash
"""

import os
import sys
import subprocess
import shutil
import uuid
import argparse
from pathlib import Path

class Container:
    """Represents a running container with full isolation."""

    def __init__(self, image_dir, command, memory_mb=512, cpu_shares=1024):
        self.id = str(uuid.uuid4())[:12]
        self.image_dir = Path(image_dir).resolve()
        self.command = command
        self.memory_mb = memory_mb
        self.cpu_shares = cpu_shares
        self.cgroup_path = f"/sys/fs/cgroup/container_{self.id}"

    def setup_cgroup(self):
        """Create and configure cgroup for resource limits."""
        try:
            os.makedirs(self.cgroup_path, exist_ok=True)

            # Memory limit
            with open(f"{self.cgroup_path}/memory.max", 'w') as f:
                f.write(str(self.memory_mb * 1024 * 1024))

            # CPU limit
            cpu_quota = int((self.cpu_shares / 1024) * 100000)
            with open(f"{self.cgroup_path}/cpu.max", 'w') as f:
                f.write(f"{cpu_quota} 100000")

            print(f"[{self.id}] Resource limits: {self.memory_mb}MB RAM, {self.cpu_shares}/1024 CPU")
        except Exception as e:
            print(f"Warning: Could not set cgroups: {e}")

    def setup_network(self):
        """
        Setup network namespace and virtual network interface.
        In a real implementation, this would create veth pairs,
        bridges, and configure iptables for NAT.
        """
        try:
            # Create new network namespace
            os.unshare(os.CLONE_NEWNET)

            # Bring up loopback interface
            subprocess.run(['ip', 'link', 'set', 'lo', 'up'],
                         stderr=subprocess.DEVNULL)

            print(f"[{self.id}] Network namespace created")
        except Exception as e:
            print(f"Warning: Could not setup network: {e}")

    def setup_filesystem(self):
        """Setup isolated filesystem with mount namespace."""
        try:
            # Ensure image directory exists
            if not self.image_dir.exists():
                raise Exception(f"Image directory not found: {self.image_dir}")

            # Create mount namespace
            os.unshare(os.CLONE_NEWNS)

            # Remount everything private to avoid propagation
            subprocess.run(['mount', '--make-rprivate', '/'],
                         stderr=subprocess.DEVNULL)

            # Change to the container's root
            os.chroot(str(self.image_dir))
            os.chdir('/')

            # Mount essential filesystems
            os.makedirs('/proc', exist_ok=True)
            subprocess.run(['mount', '-t', 'proc', 'proc', '/proc'],
                         stderr=subprocess.DEVNULL)

            os.makedirs('/sys', exist_ok=True)
            subprocess.run(['mount', '-t', 'sysfs', 'sys', '/sys'],
                         stderr=subprocess.DEVNULL)

            os.makedirs('/dev', exist_ok=True)
            subprocess.run(['mount', '-t', 'devtmpfs', 'dev', '/dev'],
                         stderr=subprocess.DEVNULL)

            os.makedirs('/tmp', exist_ok=True)
            subprocess.run(['mount', '-t', 'tmpfs', 'tmpfs', '/tmp'],
                         stderr=subprocess.DEVNULL)

            print(f"[{self.id}] Filesystem isolated (root: {self.image_dir})")
        except Exception as e:
            raise Exception(f"Failed to setup filesystem: {e}")

    def run(self):
        """Run the container."""
        print(f"[{self.id}] Starting container...")
        print(f"[{self.id}] Command: {' '.join(self.command)}")

        # Setup cgroup first
        self.setup_cgroup()

        pid = os.fork()

        if pid == 0:
            # Child process - this becomes the container
            try:
                # Create new namespaces
                os.unshare(
                    os.CLONE_NEWPID |   # Process isolation
                    os.CLONE_NEWNS |    # Mount isolation
                    os.CLONE_NEWUTS |   # Hostname isolation
                    os.CLONE_NEWIPC     # IPC isolation
                )

                # Fork again to become PID 1 in the new namespace
                container_pid = os.fork()

                if container_pid == 0:
                    # Grandchild - this is PID 1 in the container

                    # Set hostname
                    subprocess.run(['hostname', f'container-{self.id}'],
                                 stderr=subprocess.DEVNULL)

                    # Setup filesystem
                    self.setup_filesystem()

                    # Setup network
                    self.setup_network()

                    # Set environment
                    os.environ['HOSTNAME'] = f'container-{self.id}'
                    os.environ['PATH'] = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

                    print(f"[{self.id}] Container ready!")
                    print("="*60)

                    # Execute the command
                    os.execvp(self.command[0], self.command)
                else:
                    # Child process - wait for grandchild
                    os.waitpid(container_pid, 0)
                    sys.exit(0)

            except Exception as e:
                print(f"[{self.id}] Error: {e}")
                sys.exit(1)
        else:
            # Parent process
            try:
                # Add container to cgroup
                with open(f"{self.cgroup_path}/cgroup.procs", 'w') as f:
                    f.write(str(pid))

                # Wait for container to exit
                os.waitpid(pid, 0)

            except KeyboardInterrupt:
                print(f"\n[{self.id}] Interrupted")
            finally:
                self.cleanup()

    def cleanup(self):
        """Cleanup container resources."""
        try:
            os.rmdir(self.cgroup_path)
        except Exception:
            pass
        print(f"[{self.id}] Container stopped")

def main():
    parser = argparse.ArgumentParser(
        description='A minimal container runtime',
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  sudo python3 container.py run /path/to/rootfs bash
  sudo python3 container.py run ./alpine sh -c "echo hello from container"
  sudo python3 container.py run ./ubuntu bash --memory 256 --cpu 512
        """
    )

    parser.add_argument('action', choices=['run'], help='Action to perform')
    parser.add_argument('image', help='Path to root filesystem')
    parser.add_argument('command', nargs='+', help='Command to run in container')
    parser.add_argument('--memory', type=int, default=512,
                       help='Memory limit in MB (default: 512)')
    parser.add_argument('--cpu', type=int, default=1024,
                       help='CPU shares (default: 1024 = 1 CPU)')

    args = parser.parse_args()

    # Check root
    if os.geteuid() != 0:
        print("Error: This program requires root privileges")
        print("Run with: sudo python3 container.py ...")
        sys.exit(1)

    if args.action == 'run':
        container = Container(
            args.image,
            args.command,
            memory_mb=args.memory,
            cpu_shares=args.cpu
        )
        container.run()

if __name__ == "__main__":
    main()

Testing Your Container Runtime

To test this, you’ll need a root filesystem. Here’s how to create a minimal one using an existing Docker image:

# Create a directory for our container image
mkdir alpine_rootfs

# Export an Alpine Linux filesystem (requires Docker)
docker export $(docker create alpine:latest) | tar -C alpine_rootfs -xf -

# Or download a minimal rootfs
wget https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/x86_64/alpine-minirootfs-3.18.0-x86_64.tar.gz
mkdir alpine_rootfs
tar -xzf alpine-minirootfs-3.18.0-x86_64.tar.gz -C alpine_rootfs

# Now run your container!
sudo python3 container.py run alpine_rootfs sh

Inside the container, you can verify isolation:

# Check hostname
hostname  # Should show container-<id>

# Check processes (only container processes)
ps aux

# Check filesystem (should only see alpine files)
ls /

# Check resource limits
cat /sys/fs/cgroup/memory.max

What We Built vs. What Docker Does

Our container runtime demonstrates the core concepts, but production container runtimes like Docker/containerd do much more:

What we built:

Process isolation (PID namespaces)
Filesystem isolation (mount namespaces + chroot)
Resource limits (cgroups v2)
Basic network isolation
Hostname isolation (UTS namespace)

What Docker adds:

Image Management: Layered filesystems using overlay2/AUFS
Image Distribution: Pulling images from registries
Advanced Networking: Bridge networks, overlay networks, port mapping
Volume Management: Persistent storage with bind mounts and volumes
Security Features: seccomp profiles, AppArmor/SELinux, capability dropping
Container Orchestration APIs: REST API for managing containers
Logging & Monitoring: stdout/stderr capture, metrics collection
Health Checks: Container health monitoring
Restart Policies: Automatic restart on failure

Understanding the Security Implications

It’s crucial to understand that our simple implementation lacks many security features:

No User Namespaces: Our containers run as root. Production containers should use user namespaces to map container root to unprivileged users.
No seccomp: We don’t restrict system calls. Docker uses seccomp profiles to block dangerous syscalls.
No Capability Dropping: Our containers have all Linux capabilities. Docker drops most by default.
No AppArmor/SELinux: No mandatory access control.

These missing features are why you should never use this implementation in production!

Conclusion

By building this container runtime, we’ve demystified how containers actually work. They’re not magic - they’re clever applications of Linux kernel features that have existed for years:

Namespaces (2002-2013): Provide isolation
cgroups (2007): Provide resource limiting
chroot (1979!): Provides filesystem isolation

Docker’s innovation wasn’t inventing these technologies - it was packaging them into an easy-to-use tool with great developer experience.

Understanding these fundamentals makes you a better DevOps engineer. When things go wrong in production, you’ll know where to look. When you need to optimize container performance, you’ll understand the levers you can pull.

Further Learning

If you enjoyed this deep dive, here are resources to continue learning:

Linux Namespaces: man namespaces, man unshare
cgroups: Kernel cgroups documentation
OCI Runtime Spec: The standard container runtime specification
runc Source Code: Docker’s actual container runtime
LXC/LXD: Linux containers project - the original container tech

I also highly recommend CodeCrafters’ “Build Your Own Docker” challenge - it’s an interactive way to build a container runtime with guided steps.

Next Steps

In a future post, I might explore:

Implementing container image layers with overlay filesystems
Building container networking from scratch (veth pairs, bridges, NAT)
Creating a simple container orchestrator (mini-Kubernetes)

Let me know in the comments what you’d like to see next!

Announcements

If you’re interested in more content like this, I post regularly about DevOps, Python, and systems programming. Follow me on Twitter/X for updates.
I’m available for Python and DevOps consulting. If you need help with containerization, automation, or infrastructure, feel free to reach out via email.

If you share this on X, tag me @muhammad_o7 - I’d love to see your thoughts! You can also connect with me on LinkedIn.

Note: Want to be notified about posts like this? Subscribe to my RSS feed or leave your email here

They want your ethics for $105

ntietz.com blog - technically a blog — Mon, 28 Oct 2024 02:00:00 GMT

If you have a blog, you've probably gotten those emails that want to "collaborate" on a guest post—which often means "let us post sketchy links for SEO purposes." Recently, I got one which was a little different flavor than the usual spam, so I bit and replies. The end result was at the same time fascinating, unsurprising, and deeply disappointing. Spoiler alert: these folks have bad ethics and want to pay a paltry sum for illegal unethical behavior. It's not even that they think we're willing to do bad things, it's that they think we'll do it for so little.

First contact

My first email came from Ben, an account manager^[1] at a marketing company. It wasn't your usual spam email, though it set off some mild alarm bells. I had no intention to actually work with these folks, but I was partway through an 8 hour drive (as a passenger), so I replied—I've gotta entertain myself somehow.

Ben's initial email was sparse on the details, though. It just said it's a "paid collaboration" and didn't name the client, and the post could be about anything. So while I waited for Ben to reply, I looked at the company website again. The sketchiest thing about it was that I couldn't find any of the people, they just didn't seem to exist.

Look, find me a CEO or founder that doesn't have a web presence or at the very least a LinkedIn account. Especially one in a marketing field. That is unrealistic.

The details emerge

It didn't take long for Ben to get back to me. Here's his email, with some cuts and without names^[2].

Thanks for your reply Nicole! We don't have any expectations in terms of reach, it's just a link to any one product or storepage you like from our client's ([CLIENT]) website. Here are more details and examples for the collaboration:

Our client is [CLIENT], formerly known as [SKETCHY-NAME]. [Details of deal site]. This is the site from which we'd like you to link to one relevant product or brand you currently/plan to mention on your blog.

Timeline & Payment

The budget is $105 USD sent to you in full via PayPal. For payment, please send a PayPal invoice to this email address [BEN'S EMAIL]. Add $4 so we can cover your fees!

Within one week of payment, include the mention in a blog post that is within two weeks old, and send me the post link.

Link Inclusion

Include one permanent and do-follow link to your choice of either a store page or a product page (not the homepage).

For product page URLs, remove "?run=" and characters following it.

Make it a natural and seamless fit: without mentioning [CLIENT], add the link to your post using the specific store/brand name or product name as the anchor text.

Examples

[SNIP (other bloggers with posts containing these sketchy links)]

Please let me know if you have any questions or ideas to run by me. I'm looking forward to your thoughts!

Warmly,

[BEN'S NAME]

So this is what they want. Not a real collaboration, but just straight money-for-links. Notably, they want a do-follow link, which means they do not want the "nofollow" or "sponsored" rel attribute on the link. They want search engine crawlers to follow these links and treat them as organic links.

They're buying higher search engine rankings.

This is link spam. It's well outlined in Google's spam policies and if you violate that, you could be removed from Google search results. More generally, it's bad because it deceives readers into thinking this is an organic link.

Perhaps most troubling, they want you to do this without mentioning the client, and presumably without mentioning that you were paid at all. I don't know about you, but usually saying "don't mention you got paid for this" signals to me that it's probably not the right thing to do.

An honest rewrite

I prefer that people are honest about their intentions. It makes things much easier for everyone. Here's what an honest version of Ben's email would have looked like:

Thanks for the reply! We don't have any expectation about reach, because we're not advertising, we're manipulating search rankings. Our client is a division of a big American bank, so they're used to buying influence already!

We'll send you $105 through the sketchiest way possible, PayPal, to make sure you feel like you're getting scammed, because you are!

To get the money, just put a link to our client's shady deal-finding site in one of your blog posts. Make sure to not disclose that it's paid, because that would make us mad, since it wouldn't manipulate search rankings very well.

Here are a few links to other bloggers who've already sold their ethics for shockingly low amounts! Let me know if you have any questions!

- (my name isn't actually) Ben

We probably shouldn't expect that from someone who uses a fake company with fake employees to trick people into doing an unethical thing, but I'm getting ahead of myself.

Who's behind this?

We don't know!

I tried to figure out who's behind this, not least of all to report it to the FTC. Ultimately, I couldn't figure it out, but I got some good clues.

When I look at the site for the "company" that "Ben" works for, a few things stand out:

The employees listed don't exist anywhere else. I'd mentioned this before, but that's really suspicious to me when I can't find anyone on the roster. If I can't find your sales folks, I'll probably assume that they're an LLM these days anyway, but if the whole company doesn't exist, then it's probably a scam.
The employees' photos are very clearly generated from a model. I'm not going to post them here but there are some of the standard image generation artifacts: someone's ear is missing a piece under her earring; a couple people have double sets of teeth; there's a weird thing going on with the CEO's neck; another has the classic weird psychedelic image model stuff in the corner; and yet another has an earring straight out of a computerized nightmare.
The site shows it was clearly made for some other scammy site and ripped off here. The sitemap shows last updated times of 2023 for some things, and has someone else's name as the only author.
The privacy policy is ripped off of another site, and the robots.txt file references a different site's sitemap listed in it. Whoops!

On top of all that, it's running a version of WordPress with vulnerabilities that aren't patched, and a version of OpenSSH that has a remote code execution bug. So maybe they should update their servers?

Our responsibilities when publishing

As people who publish online, we have a responsibility to our readers and to the broader internet community we're part of. There are lots of resources to read about these, and the FTC influence guidelines are particularly good to read in my opinion.

The gist of the matter to me is that as authors, we have (at least) these responsibilities:

Tell people if you've received payment or have any relationship with a company/product/etc. mentioned. This will let them decide if they want to trust the content or not.
Make these disclosures obvious and very close to what's being disclosed, so no one can reasonably miss them.
Follow norms for link attributes, like adding rel="nofollow" if a link is related to a sponsor (or is just outright paid for)
Do some due diligence on potential partners before you work with them. Make sure it's a company you'd be comfortable working with in public. Also that they, you know, exist.

On top of this, I have a few other guidelines I like to follow for my own work. These are very much personal guidelines, and I don't think they're universal.

Only publish things you'd publish otherwise. Other than priority, it should be exactly like a normal post: the same style, the same content, just with a prompt that was bumped up by some cash^[3].
Make it valuable to your readers. Sponsored posts are ads, in many ways. Some will skip them. Those who don't should be able to get value out of it, whether that's knowledge or entertainment.
Deal in public only. Don't hide who you deal with, and only work with people you'd be comfortable telling everyone you're working with.

If you have an audience, then you have a lot of responsibility to do right by them. There's a lot of trust placed in us, and we can do a lot of harm if we do bad things for money.

I think that we should be able to make a living off our work. I'd love to make a living off writing articles about silly things I've done with software^[4]. But getting there has to be ethical; nothing is worth doing if you can't do it ethically. I'm not selling links, but I am gonna keep responding to some of these folks for the sheer entertainment of it on long drives.

I mean, that's what the email signature and website say... ↩
I've omitted both the client and company names here because I don't want to give them any of the traffic they're trying to buy. I also don't know that the company is in on it—this could be the shady dealings of a company they've contracted, or further removed. ↩
I've only done one sponsored post. It was a pretty good experience, despite my anxiety. But I'm not sure I'll have the opportunity (or desire) to do another one for a while. ↩
I think my one sponsored post has technically covered all the costs for operating my blog since its inception. That doesn't take into account labor, so I'm still very far off of making a profit if I had to pay myself. And to get to a real salary would take a lot of sponsored posts (more than I'm willing to do) or a lot of ~~begging for money~~ Patreon memberships. ↩

Creating a Flux Dev LORA - Full Guide

Reticulated — Sun, 27 Oct 2024 19:02:39 GMT

Complete Guide to Creating Flux LORAs. Workflows included.

📗 How Big Things Get Done, by Flyvbjerg & Gardner

Ernest Oppetit — Sun, 27 Oct 2024 10:24:36 GMT

Intro Think slow, act fast The commitment fallacy Think from right to left Pixar planning Are you experienced? So you think your project is unique? Can ignorance be your friend? A single, determined organism What’s your lego Eleven heuristics for better project leadership Misc

WP Engine Drama

Ishan Sharma — Sun, 27 Oct 2024 03:48:59 GMT

Some links and thoughts on WP Engine Drama going on over in WordPress community

Intent

Terminal Thoughts — Sun, 27 Oct 2024 03:00:00 GMT

You have found my tiny corner of the internet!

This is where I share my thoughts about:

Crypto
Security
Infrastructure
Linux
Food
Health
Spirit

You can find my short form content on X or Farcaster, and my code on GitHub.

Jackson Hole, Tetons, and Yellowstone, 2024

Avy Faingezicht — Sun, 27 Oct 2024 03:00:00 GMT

In October, I got to visit Jackson Hole, Wyoming, for a work offsite with our engineering team. I was barely over a month into the new job, and since we're pretty distributed, it was a great way to meet many of my teammates in person for the first time. While Jackson Hole was out of the way for everyone, it turned out to be an awesome backdrop for our time together. The sessions were productive, and the team-building activities were fun. We got to explore the Tetons and Yellowstone, which were both stunning. The trip was a great way to bond with my new colleagues and experience the beautiful national parks.

Process Optimization

Craig Pardey — Sun, 27 Oct 2024 03:00:00 GMT

Only by increasing flow through the constraint can overall throughput be increased – Eliyahu M. Goldratt, The Goal, 1984

Turning this around we get the corollory that optimizations made to areas other than the constraint only serve to worsen the constraint.

Agile practices, and particularly Scrum and SAFe, have been widely adopted in large enterprises over the last decade or so, spawning entirely new departments dedicated to locking down the process. Agile Centers of Excellence (CoEs) produce materials explicitly describing every aspect of their company’s specific flavour of Agile, and have coaches dedicated to enforcing it across the teams.

What I have learned about 3D printing

River Writes - A MediaWiki Blog — Sat, 26 Oct 2024 17:22:56 GMT

I started 3D printing and I do not recommend it as a hobby

Long Time No Post! RPi Pico, Zephyr, and Whales (oh my)

Greg Herlein — Sat, 26 Oct 2024 11:00:01 GMT

A short update on my whale song project: I fell down a few technical rabbit hole!

Gmail Labels Don't Search Well

er4hn — Sat, 26 Oct 2024 03:00:00 GMT

Then Life Continued

Miss Sloane

Mehul Kar's blog — Sat, 26 Oct 2024 03:00:00 GMT

Watched on Saturday October 26, 2024.

Rated 3.5 stars.

Thanks for reading this post via RSS!

Impressions on the remarkable 2, one month in

Julien's DevRel corner — Fri, 25 Oct 2024 19:44:02 GMT

This article describes my experience with the Remarkable 2 after a month of usage. I am generally happy with it, though many things are lacking in my opinion.

Algorithms for Optimization (Explained Simply): Part 2 - Line Search and the Trust Region Method

Photon-Lines Substack — Fri, 25 Oct 2024 18:52:50 GMT

A visual-focused and intuitive overview of important optimization algorithms. This blog post focuses explaining the details behind the line search and the trust region method.

Time tracker

ciesie.com — Fri, 25 Oct 2024 15:12:50 GMT

THIS IS A WORK IN PROGRESS <

Below you will find log entries that describe the design/thinking process.

Idea <

The goal is to build a device that could allow me to comfortably measure time spent on a project. I want to have a full history of the time spent on particular project. The time log needs to be plain text, easily queried and human readable.

Process <

2023-08-23

And here we go with another attempt at prototyping this device. 3D printing is great, as long as it works. Any 3D printer issues can derail the entire process - I’m not here to practice my 3D printing troubleshooting skills. Another thing about 3D prints is that they’re not a good base to further refine the printed prototype. It’s a plastic shell and there’s not much you can do to modify its shape, without redesigning and reprinting. There’s definitely way more to prototyping than 3D printing.

Reflections on the Metastrategies Workshop

This Is My True Name — Fri, 25 Oct 2024 04:50:00 GMT

A weekend of trying to be better

ARBORETUM

Austin Atchley — Fri, 25 Oct 2024 03:00:00 GMT

I am excited to announce that ARBORETUM, the second album under Post Bag Productions, is now streaming on all platforms! Go check it out!

If you don’t have access to a streaming service, you can also listen to the full album on YouTube:

This is a project I’ve been working on for the past year along with my creative partner, Sudarshan. We perserved through several cross-country moves (from Seattle to Manhattan to Brooklyn, and from Seattle to Dallas), busy schedules, and many creative blocks to put this project out. We’re excited to share a view into our lives over the past year.

2024-10-25/01

ho.dges.online — Fri, 25 Oct 2024 03:00:00 GMT

2024-10-25/02

ho.dges.online — Fri, 25 Oct 2024 03:00:00 GMT

TLDs with novel behavior

Built on Shards of Silicon: Robert Alexander's Tech Blog — Fri, 25 Oct 2024 03:00:00 GMT

Thoughts on .internal and gTLDs

gus gorman takes on the election

antipaucity — Thu, 24 Oct 2024 19:34:20 GMT

Imagine you are Gus Gorman, and you live in the United States – a country which uses the Electoral College for their national election process – while each state or province may award electors based on the popular vote (either an all-or-nothing approach, or proportionally), a pure “most votes” count does not necessarily indicate a...

continue “gus gorman takes on the election” »

In defense of my technophobia

andersource — Thu, 24 Oct 2024 15:30:00 GMT

Alright I admit it, I'm a technophobe

Calibrating a 3D Printed Delta Arm for Screen Interaction

DFWORKS | Online Threat Mitigation — Thu, 24 Oct 2024 15:00:00 GMT

Introduction The battle between disinformation spreaders, propagandists, and automated social media activity, and the algorithms designed to…

Develop a Recipe Generator AI App in PHP Laravel using OpenAI

Adnan's Random bytes — Thu, 24 Oct 2024 13:54:46 GMT

This post is part of the GenAI series. Alright, so the first text-based GenAI app that I am going to build is actually an AI-based recipe generator in PHP Laravel using OpenAI APIs. The app will ask about the ingredients and cuisine type and will generate a recipe for you. If you are in a hurry, you can watch the demo of the app below: To build this app, you will need to install Laravel and generate OpenAI API keys. Development Setup To make this app work you need three things: OpenAI API key generation Laravel installation OpenAI Integration OpenAI API Key generation Go to the API Keys section of the OpenAI Platform generate a new key that starts with sk- and save it somewhere. (Image below edited via chatGPT) Laravel Installation Assuming you have composer installed, you can run the following command: composer create-project laravel/laravel recipegenerator It will create a folder with the name recipegenerator. Now, run npm install to install Node packages. If all goes well, you will see something like the output below: Let’s generate HomeController: php artisan make:controller HomeController and then a view: php artisan make:view home The HomeController.php file will look like this: namespace App\Http\Controllers; use Illuminate\Http\Request; class HomeController extends Controller { public function index() { return view('home'); } } and home.blade.php file: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta http-equiv="X-UA-Compatible" content="ie=edge"> <title>AI Recipe Generator</title> <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.6.2/dist/css/bootstrap.min.css"> </head> <body> <div class="container"> <div style="margin-top: 30px;" class="row text-center"> <div class="col-md-12"> <img width="100" height="100" src="{{url('logo.webp')}}" alt=""> <h1>Recipie Generator</h1> </div> </div> <div class="row"> <div class="col-md-12"> <form action=""> <div class="form-group"> <label for="ing">Ingredients:</label> <input type="text" class="form-control" id="ing" placeholder="Potato, Tomato etc.."> <label for="cuisine">Cuisine</label> <select class="form-control" name="cuisines" id="cuisineDropdown"> <option value="italian">Italian</option> <option value="chinese">Chinese</option> <option value="mexican">Mexican</option> <option value="indian">Indian</option> <option value="japanese">Japanese</option> <option value="french">French</option> <option value="thai">Thai</option> <option value="greek">Greek</option> <option value="spanish">Spanish</option> <option value="middleEastern">Middle Eastern</option> </select> </div> <button type="submit" class="btn btn-danger">Generate</button> </form> </div> </div> </div> </body> </html> And you will see something like the following: OpenAI Integration To integrate OpenAI with your PHP application you will need to install this unofficial but awesome library. composer require openai-php/client I am going to add another route, generate, that will interact with OpenAI GPT APIs and generate a recipe. OpenAI provides an array of APIs to interact with their different services: Chat, Audio, Embeddings, Fine-tuning, Assistant, and many others. We are going to use Chat Completion API for our app. This could be called the default feature of the chatGPT which is when you ask something on GPT, it uses the chat completion APIs internally. Below is the code that is interacting with GPT: public function generate(Request $request) { $apiKey = getenv('OPENAI_KEY'); $client = \OpenAI::client($apiKey); $result = $client->chat()->create([ 'model' => 'gpt-4o-mini', 'temperature'=>0.5, 'max_tokens' => 150, 'messages' => [ ['role' => 'system', 'content' => 'You are helpful assistant'], ['role' => 'user', 'content' => 'Hello!'], ], ]); dd($result); } I opted for the latest gpt-4o-mini model because it is cheaper, faster, and sufficient for what we are looking for. You can see the list of models here. The temperature parameter controls the randomness that causes hallucinations. The value ranges between 0 and 1. The lesser the value, the more deterministic it is. max_tokens tells about the maximum number of tokens that can be generated in the chat completion. OpenAI provides an online tokenizer tool that is quite helpful. messages field contains both prompts and instructions to generate a response. When you send a message to GPT, your role is a user. When GPT responds, its role is an assistant. You can set the tone of the response by using another role called system. In chatGPT, you set the tone by adding Custom Instructions as I have done here. When I run the above code it returns the following: OpenAI\Responses\Chat\CreateResponse {#282 ▼ // app/Http/Controllers/HomeController.php:28 +id: "chatcmpl-A6Cunhx6HVafxPgPPiZmO6zWkDKsB" +object: "chat.completion" +created: 1726042977 +model: "gpt-4o-mini-2024-07-18" +systemFingerprint: "fp_54e2f484be" +choices: array:1 [▼ 0 => OpenAI\Responses\Chat \ CreateResponseChoice {#278 ▼ +index: 0 +message: OpenAI\Responses\Chat \ CreateResponseMessage {#271 ▼ +role: "assistant" +content: "Hello! How can I assist you today?" +toolCalls: [] +functionCall: null } +finishReason: "stop" } ] +usage: OpenAI\Responses\Chat \ CreateResponseUsage {#270 ▼ +promptTokens: 17 +completionTokens: 9 +totalTokens: 26 } -meta: OpenAI\Responses\Meta \ MetaInformation {#276 ▼ +requestId: "req_445c1ee052f9ef65451b450ce29e7788" +openai: OpenAI\Responses\Meta \ MetaInformationOpenAI {#267 ▶} +requestLimit: OpenAI\Responses\Meta \ MetaInformationRateLimit {#279 ▼ +limit: 10000 +remaining: 9999 +reset: "6ms" } +tokenLimit: OpenAI\Responses\Meta \ MetaInformationRateLimit {#268 ▼ +limit: 10000000 +remaining: 9999840 +reset: "0s" } } } Check the usage section: +usage: OpenAI\Responses\Chat \ CreateResponseUsage {#270 ▼ +promptTokens: 17 +completionTokens: 9 +totalTokens: 26 } It tells you about both input and output tokens. promptTokens tell the number of tokens used in the prompt and in the output. The completionTokens tells about the token count generated by OpenAI API. If you go to the tokenizer website it shows something like the below: Now I am modifying the code to incorporate the actual prompt and other settings: public function generate(Request $request) { $apiKey = getenv('OPENAI_KEY'); $ingredients = $request->get("ingredients"); $cuisine = $request->get("cuisine"); $prompt = "Generate a recipe based on the following information:\n Ingredients: {$ingredients}\n Cuisine: {$cuisine} "; $client = \OpenAI::client($apiKey); $result = $client->chat()->create([ 'model' => 'gpt-4o-mini', 'temperature'=>0.3, 'max_tokens' => 500, 'messages' => [ ['role' => 'system', 'content' => 'You are an expert chef who has expertise in different kind of cuisines.'], ['role' => 'user', 'content' => $prompt], ], ]); // dd($result->choices[0]->message->content); dd($result); } It worked, but there’s an issue; it consumed more than 500 tokens to return a complete response. I tweaked the system prompt and made it like this: You are an expert chef who has expertise in different kind of cuisines. The generated Recipe should have only the following headings: Ingredients and Instructions I am now instructing it to output these two sections instead of adding extra content. I also reduced the temperature. The code now looks like the following: $result = $client->chat()->create([ 'model' => 'gpt-4o-mini', 'temperature'=>0.3, 'max_tokens' => 500, 'messages' => [ ['role' => 'system', 'content' => 'You are an expert chef who has expertise in different kind of cuisines. The generated Recipe should have only the following headings: Ingredients and Instructions'], ['role' => 'user', 'content' => $prompt], ], ]); And now it consumed 421 tokens only. You can track the usage by visiting here. Since I want to use AJAX to make a call, I am returning the data in JSON format return response()->json([ 'status' => 'OK', 'message' => $result->choices[0]->message->content, ]); In Postman it generates an output like the below: All is well. Now, what is needed to retrieve this data via AJAX and display it on the page? Here’s the relevant code snippet: if (ingredients!= null && cuisine != null) { $.ajax({ url: '{{ route("generate") }}', type: 'POST', data: { _token: $('meta[name="csrf-token"]').attr('content'), // CSRF token ingredients: ingredients, cuisine: cuisine }, success: function (response) { const converter = new showdown.Converter() var markdownText = response.message const html = converter.makeHtml(markdownText); $('#recipe').html(html) $('#wait').addClass('d-none') }, error: function (xhr, status, error) { // Handle errors console.log(xhr.responseText); $('#wait').addClass('d-none') $('#recipeResult').html('<p>Error: ' + xhr.responseText + '</p>'); } }); } It generated the response in Markdown so I used a library to convert it into HTML format. Hmm, it looks good, but there’s an issue; there’s no indication of what the recipe name is. I converted the system prompt to the following: You are an expert chef who has expertise in different kind of cuisines. The generated Recipe should have only the following headings: Recipe name, Ingredients and Instructions And now it generates like the below: Looks cool and yummlicious, No? Conclusion So as you saw, by adding a few lines you can make your app an AI app, an AI wrapper app, that leverages LLM APIs to produce useful apps. Like always, the code is available on Github. Like always, the code is available on Github. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Develop a Recipe Generator AI App in PHP Laravel using OpenAI first appeared on Adnan's Random bytes.

Apple Magic Trackpad with swollen battery

La Vita è Bear — Thu, 24 Oct 2024 03:00:00 GMT

I don’t know about others, but if given a choice, I always prefer trackpads over regular mice (except for the cases of playing games, but the last game I played requiring a mouse was StarCraft 2 more than 10 years ago). Sometimes I don’t have a choice, for example when I go to the office occasionally after the pandemic, we are provided with an external display, a keyboard, and a mouse. The mouse is Logi’s MX Master, which is a nice mouse, but this scenario happens several times a day when I’m in the office:

I was scrolling down with the mouse’s scrolling wheel on a page to look for something towards the bottom of the page.
I found what I was looking for and was moving on, but the scrolling wheel on the mouse keeps scrolling with its inertia energy.
By moving on I usually start to press the Control key, most of the time it’s for Control-T to open a new tab.
In chrome Control-Scroll down means zoom out, so the page is zoomed out to the minimum.
I start to curse and reset the zoom of that page in Chrome.

When I’m using a laptop, it usually comes with a trackpad so I’m good (unless it’s some old ThinkPad with a tracepoint, but I’m also ok with tracepoints, despite that I never used them for extended period). When I need to plug my laptop into an external display, I would need an external trackpad. Here comes the problem: no one sells an external trackpad except Apple.

From what I can tell, Brydge had a plan to make an external trackpad, but that plan seems to be failed and it never hit the market; Logi had an external trackpad product for a while, but it’s discontinued now; Logi currently does have an external trackpad, but they don’t sell it by itself, it’s only sold as part of the “Casa Pop-up Desk” bundle.

Which is why I was using 2 Apple Magic Trackpads (I think they are both “Apple Magic Trackpad 2”) at home.

I don’t use any Macs any more. Those trackpads are one of the 2 apple products I’m still using (the other is their thunderbolt cable). They are plugged via USB-A - Lightning (sigh) cable to an USB hub that also connects to an external display (for one of the cases the display itself is the USB hub), because I don’t want to deal with the switching of the bluetooth when I plug different computers into the hubs.

Probably because they are plugged in ²⁴⁄₇, I recently started to notice that one of them, the one I use with my laptops every day, no longer sits flat on the table. It starts to wobble, and the cause of that is most likely that the battery is swollen. The other seems to also be slightly swollen, but far from as severe as this one.

Swollen batteries are a dangerous thing, they are fire hazards. A few years ago when my Pixel Slate’s battery started to swell, I contacted Google support, and they replaced it for me for free even though it was already out of warranty. So I brought the 2 trackpads to a local Apple store yesterday.

The conversation was like this:

Do you have the serial number of the Mac it connects to?
I don’t use it with a Mac.
Do you have an order number or something?
(I searched my Gmail for “apple order trackpad” and found the order email for one of them from 9 years ago)

But apparently the order number from 9 years ago is no longer in their system. The staff then brought his own Mac, plugged the trackpad into it to get its serial number, and the serial number is no longer in their system either. He then tried the other one, this time I think he found the serial number in their system, but to no one’s surprise it’s also out of warranty (I later found the order email for the other one, that one was bought from BestBuy 4 years ago).

After talking to his colleagues, he came back with the offer: They can replace it for me, but since it’s out of warranty, they’ll charge me $109. I then looked up how much is a new “Apple Magic Trackpad 2”. It’s $129. So they only give me $20 discount for it.

I asked him to recycle the severely swollen one for me, and took the other one back home.

I actually happened to see the news about Ploopy’s new trackpad, and placed a pre-order, but my order will take months to arrive. So in the meantime I’ll have to share the single trackpad between my 2 external displays, or find some spare mouse to get by. But I’m excited that I finally have an option besides Apple, and this one does not have a battery so it won’t swell and becomes a fire hazard. If this one turns out to be good, I’ll certainly buy another one to replace my other Apple trackpad.

State of Data Engineering Q3 2024

Dan Quach Blog — Thu, 24 Oct 2024 01:21:08 GMT

Here is this quarter’s state of data engineering newsletter. There is only a little chat about AI this time, and a focus on Open Table Formats, the Apache Iceberg Rest Spec, Open Table Format updates, and new updates in the Amazon Data Engineering ecosystem. Prompt Engineering – Meta Analysis Whitepaper One of my favorite AI […]

Big Data Processing - Basics : Part 1

Hitesh Writes — Wed, 23 Oct 2024 18:41:03 GMT

A small series for understanding and optimizing Spark and other data processing engines.

The Power of Analogies

🦉 olshansky 🦁 — Wed, 23 Oct 2024 17:35:36 GMT

7 Simple analogies explaining what the Web3 protocol we work on does

Transatlantic: Season 1

🦉 olshansky 🦁 — Wed, 23 Oct 2024 15:16:57 GMT

Olshansky's review of Transatlantic: Season 1

An AI Horror Story

Home on Lagomorph — Wed, 23 Oct 2024 09:00:00 GMT

So I’m reading the teachers subreddit for my daily dose of misery and sense of doom about the future when I come across an unusually worded comment.

As an educator, I’ve got to agree - while crystals and essential oils can be lovely, they won’t replace a solid IEP or therapy. Let’s save the throwing for the baseball field, not the classroom!

~ /u/mohsinali- (343 Karma pre Suspension)

This comment reeked of ChatGPT. This is exactly the way an un-tuned AI writes, so I follow the rabbit hole downward to see what this is all about.

Vendor Lock-in is an Imaginary Problem

Wed, 23 Oct 2024 03:00:00 GMT

Vendor lock-in is often cited as a reason to chose on-premise hosting, or to be mindful of "avoiding" it when using the cloud (whatever that means). Whilst there are some compelling arguments to mitigate risks associated with vendor lock-in, I'd argue that this is a completely imaginary problem.

1. You're already locked-in - at every level

Vendor lock-in is always discussed through the lens of the cloud. But, in reality the cloud is not a special class of tooling, it's like everything else. If you use a language like Javascript, you're already locked in to using the myriad of NPM packages you have. Worst still is if you're using NextJS, React, Rails or Laravel. These frameworks all lock you in to some degree. What about your continuous integration? The same organisations that worry about vendor lock-in to the cloud providers never bat an eye lid writing thousands of bespoke lines of github actions workflows. The point is, the cloud is not the only place you're locked in. You're locked-in everywhere. But, it's about making appropriate trade-off.

2. Businesses rarely migrate from a cloud provider

Asides from a few extremely rare cases, it's likely that once AWS or whatever other cloud provider puts their claws into you, you won't escape it's clutches. Rather you'll likely embrace it. Abandoning the lift-and-shift you originally oversaw, and build "cloud native" - but why? Because it's cheaper (on the surface) and the recommended way to do things (according to the cloud providers). You also end up hiring staff with the skills to wrestle these tools to do your bidding. It's unlikely then that after doing all that work you would want to migrate to another provider. No only is the cost so astronomically high to make it unfeasible but there is also no reason, commercial or otherwise, for doing so. Even if they heavily raise prices, it's still a footnote for the costs that most businesses pay. In some cases however, at enormous scale, it does become economically advisable to migrate elsewhere. If you built the product in a vendor agnostic way, I'm sure the migration team will thank you. But if not, you're likely at a scale that the cost of doing this migration is just viewed as "one of those things".

3. You know what you're signing up for

When you go to a shopping centre (or mall), you might need to buy some clothes, gifts or yet another Apple device. But then you suddenly realise that you need shampoo. Now, will you go to the pharmacy shop that sells the shampoo despite the cost being a little higher? Or will you get it from your regular supermarket that is 20 minutes away? Of course, you'll pay the premium and buy it at the shopping centre. There are no tangible benefits to the product - they are the same. What you are paying for is convenience. The cloud is no different, you might go in first of all to simply get some compute. But then quickly you start using it for everything - it's convenient. The business has already signed off on it, there is budget and your tooling is setup for it. In other words, you know what you signed up for when you started using the compute. And if you need to pay a bit extra to use other things then big deal, you would have paid that cost elsewhere in the first place. Public cloud pricing is publicly available (although notoriously difficult to forecast). So, due diligence should be taken up front to understand what kind of cost implications the cloud could have. While convenience often outweighs cost at smaller scales (like buying shampoo one time), in larger enterprises, even slight cost increases can lead to substantial expenses. This is why some companies consider multi-cloud or cloud-agnostic strategies to balance convenience with financial prudence.

The only sound argument against vendor lock-in

Local development. That's it. Once you use bespoke cloud software, it becomes extremely challenging to create working development environments that mimic your live deployment. Therefore, I do favour against using "cloud-native" tools because it makes your life a lot more difficult for doing development work (which makes the money).

Summary

As discussed vendor lock-in is largely an imaginary problem. There may well be 1 (or 2) extremely valid reasons for avoiding vendor lock-in but generally it's something that you need to not worry about and get on with shipping.

Gympl skripta

Kubíkovo — Wed, 23 Oct 2024 01:00:00 GMT

Zápisky ze školní češtiny a angličtiny k maturitě na GJP-ME.

7 Pocket Network Analogies

🦉 olshansky 🦁 — Wed, 23 Oct 2024 00:40:47 GMT

tl;dr Explaining what Pocket Network is through a series of 7 internet networking analogies. Subscribe if you want to be notified of when…

Blog update pains

callmeolive — Wed, 23 Oct 2024 00:00:00 GMT

This blog post will not be updated to match the site in the future, any mention of how things are will remain that way.

Firstly, I made this blog with two key influences in mind:

Herman Martinus’ Bear Blog for it’s simple & lightweight ethos
InvisibleUp’s unique article banners featured with each post

I mentioned this prior in a now-deleted blog post. With that post now gone, I feel it’s best to publicly credit my influences again and this is the best post to do it in.

I made this blog is because Enshittification is ruining all the platforms out there. The only way to escape it is to have my own place online, free from the issues these platforms constantly suffer from.¹

Now there’s a catch, I’m not a genius. My only experience of webdev was using Dreamweaver CS3 back in school. I knew about Static Site Generators, but I couldn’t wrap my head around them. Because of this, I always updated my blog manually with HTML like it’s still 1999, which was a nightmare.

Writing a blog post was easy, but getting it online was such an arduous task that it completely put me off writing. Something had to change. I needed to sit down and learn a Static Site Generator, or suffer.

I chose Hugo for one reason: everyone said it was simple and easy to learn. This was a lie. Hugo is hell to learn for the following reasons:

Documentation exists, but it’s nigh-incomprehensible for a newbie to digest.
Hugo is too versatile – there is no one way to do a task. This leads to each theme accomplishing the same thing with different methods, which doesn’t help new users figure out the best practices
90% of the help out there was on closed forum posts, with the only reply being from a moderator pointing someone to the documentation with no actual explanation

I don’t want to sound like I’m slagging off the people behind Hugo, it’s a great tool but as someone new to SSGs as a whole it’s practically impossible to understand. Every time I tried learning how Hugo worked I’d end up with a debilitating stress headache and exhaust myself from failure. Each time I’d quit on the verge of tears, time and time again.

Zachary Betz has a clear tutorial for making a Hugo blog from scratch, it’s solid, but not for me. I’ve nothing against the tutorial, but I struggle with bottom-up learning with anything hands-on. If you asked me to build a house, I’d learn more from taking one apart first.

What helped me the most was Jan Raasch’s Hugo Bear Blog theme – which cleanly recreates the look of Bear Blog. I meticulously dissected it to see what each building block did. At 2am on Sunday, neck deep in the partial HTML templates, something clicked.

I became The One

Even if I couldn’t explain with words or even comprehend it, I had an innate understanding of how every piece fit together, and I had the power to rewrite everything as I saw fit. There’s no more liberating feeling than the fire of discovery. In a state of pure flow I had to pry myself away from the laptop at 6am lest I wouldn’t sleep at all. All I wanted to do was scream at the world that I finally got it.

Upon waking I spent the entirety the day recreating my site in Hugo, setting it up in a way that all I’d have to do to make a blog post would be to write a Markdown file and make a banner image. If I wanted to change the core layout of the page, all I have to do is change a single HTML file. I can’t state how much time this will save in the future.²

On top of that I made a few changes to how the site looks to [hopefully] make it cleaner and easier to read. I’ve ditched Open Sans for Anek Latin, we have tags now, and the site should be readable for devices which don’t support WebP. It’s not perfect, but it’s good enough for now.

But the pain of repeated failure, the countless headaches trying to understand how Hugo works. It’s finally worth it, and I think I’m just going to collapse in bed now.

Again I have to thank Jaan Raasch, whose Hugo theme was easy to understand. And I’ll apologise to him for absolutely butchering it to create this.

Namely login barriers, obnoxious cookie + mailing list banners, absurd page bloat, and advertisements. They’re all intrusive and make an otherwise decent reading experience a nightmare. ↩︎
I can. I made this footnote instantly. Also not having to faff about with the timecodes on the RSS feed is a lifesaver. ↩︎

ferris sweep split keyboard build

blog — Tue, 22 Oct 2024 21:09:42 GMT

Event handing in Swift

june.kim — Tue, 22 Oct 2024 03:00:00 GMT

Language model random number generator

Thought Eddies — Tue, 22 Oct 2024 03:00:00 GMT

Language model random number generator

2024-10-22

ho.dges.online — Tue, 22 Oct 2024 03:00:00 GMT

Towards the Fastest Brainf*** Implementation Ever

Ashton Wiersdorf on Lambda Land — Tue, 22 Oct 2024 03:00:00 GMT

In my last post I described how I made a very fast BF interpreter. Well, there’s a lot more speed to be had with an optimizing compiler. This post is a write-up of my assignment for a compilers class, so the post a little rougher than normal.

You can find the code at the following places:

Code organization #

Files for the compiler:

interp_threaded_opt.rkt: BF parser and all optimizations live here.
native.rkt: Takes the optimized AST from interp_threaded_opt.rkt and emits native ARM instructions for my M1 Pro chip.

`interp_threaded_opt.rkt` #

The first role of this module is to define an AST for BF. My AST is represented as Racket structs; these are the nodes that I have:

Node	Fields	Comment
`loop`	`body`	Loops hold their entire body within them. There is a parsing pass to the code that matches all `[` `]` characters.
`add`	`amount`	Sequences of `+` or `-` get smooshed together into single add instructions.
`shift`	`amount`	Like `add`, but for `>` and `<`
`set-cell`	`value`	Set the current cell to a specific value. Used to optimize `[-]` and the like.
`bf-write`
`bf-read`
`add-cell-0`	`dest`	Zero current cell; add old value to another cell. This is a common loop that I recognized and optimized.
`mult-block-0`	`body`	Result of optimizing simple loops. Body is a map from tape offset → value.
`search-0`	`stride`	Representation of scan loops like `[<<<<]`.

The rest of the module is devoted to optimizations performed on the AST. Initially, only loop, add, shift, bf-write, and bf-read cells are present, but through the optimization passes more get added. See § Optimizations for details on all the optimization passes.

There is a compile function, but this just compiles the optimized AST down to Racket closures with a threaded interpreter.

`native.rkt` #

The main entry point is the compile-to-asm function, which emits some fixed prelude code, the body of the program, and then some fixed postlude code.

There are a bunch of functions prefixed with i/; these are to help me generate assembly instructions with a particular format. Mostly just wrappers around calls to format.

The emit-asm function does the hard work of walking the AST and generating instructions for each node. There’s a loose set of conventions around various registers; the most important is that x20 always holds the address of the start of the tape, and x21 holds the current pointer. I never use these registers for anything else; you can always access the current value under the program head with [x20, x21].

There are two special functions for emitting the assembly for search-0 nodes: one for forward scans ([>]) and one for backward scans ([<]). This is just done for organizational convenience.

Optimizations #

The entry point to optimizing the BF AST is in the optimize function in interp_threaded_opt.rkt. This chains optimizations together; each optimization function must take a whole program AST and return a whole program AST functionally. The optimizations are in order they are performed:

combine-instrs

This pass takes sequences of instructions like +++, initially represented with (list (add 1) (add 1) (add 1)) and turns it into (add 3). Same thing for pointer shift instructions.
opt/useless

This cleans up instructions like (shift 0) or (add 0) that have no effect.
opt/add

This recognizes patterns like [>+<-], which adds the value of one cell to another. It handles arbitrary shift and add amounts, as long as they’re balanced. Kind of like a baby opt/basic-loop pass.
opt/zero-out

Handles cases of [-] or [+] to set the cell to 0.
opt/basic-loop

Does the basic loop optimization we discussed in class. Easily the most complicated optimization. It abstractly evaluates the body of a loop; it sets the initial pointer to 0 and tracks what additions go to which offsets. Bails out if anything gets too complicated.
opt/zero-add->set

Optimizes cases of [-]+++ to just set the current cell value to 3.
opt/0-scan

Scan loops, as discussed in class.

Evaluation #

Extensive tests (`bfcheck.pl`) #

I precompiled and ran all the files in the check/ folder provided. I noticed that the first run of the resulting binaries took a long time, and that they all ran almost instantly afterwards. I noticed that there was a bunch of network activity; seems like macOS was being zealous in sending signatures for each of the binaries the first time they were run. The benchmarks run quickly, so this added up. The numbers reported here are after running the tests one time to avoid the network penalty.

Optimization set	Total time (seconds)
No optimizations	0.763064
Basic loops	0.767731
Scan loops	0.790222
Scan & basic loops	0.763272

There’s not a whole lot of variance.

Benchmarks (`mandel.b`, etc.) #

I ran my compiler on some of the benchmarks from the BF benchmarks suite we’ve been using. I also tried out a few with the cgbfi.b BF-interpreter-in-BF.

Table 1: Execution times for two benchmarks, natively compiled. All times in seconds.

Benchmark	No opts	Basic loops	Scan loops	Scan & basic loops
`hanoi.b`	0.11	0.05	0.11	0.05
`mandel.b`	0.73	0.49	0.73	0.49

Seems that basic loop optimization is most important for the Hanoi and Mandelbrot benchmarks. I also tried the Sierpinski triangle benchmark but it never ran long enough for me to notice any difference.

Contrast with benchmarks running under the cgbfi.b interpreter: scan loops are far and away the most important optimization:

Table 2: Execution times for benchmarks running under the cgbfi.b interpreter. All times in seconds. *Gave up after 2 hours; estimated duration recorded. †No attempt.

Benchmark	No opts	Basic loops	Scan loops	Scan & basic loops
`serptri.b`	0.07	0.06	0.03	0.02
`mandel.b`	*44,052.00	†	†	7,818.96

Example optimizations #

Basic loops #

This code:

+++[->+>-<<]

Gets optimized to this:

ldrsb   w11, [x20, x21] ; add 3
add     w11, w11, #3
strb    w11, [x20, x21]
add     x22, x20, x21   ; mult-block-0
ldrsb   x23, [x22]
add     x24, x22, #1
ldrsb   x25, [x24]
mov     x11, x23
add     w11, w25, w11
strb    w11, [x24]
add     x24, x22, #2
ldrsb   x25, [x24]
mov     x11, x23
subs    w11, w25, w11
strb    w11, [x24]
strb    wzr, [x20, x21]

Any loop with I/O or nested loops—even something like setting something to 0 are not covered by this.

Scan loops #

This code:

[>>>>]

Gets optimized to this:

    adrp      x22, lIDX@PAGE              ; search-0
    ldr       q0, [x22, lIDX@PAGEOFF]     ; v0 = idx vector
    adrp      x22, lSTRIDE4@PAGE
    ldr       q3, [x22, lSTRIDE4@PAGEOFF] ; v3 = stride mask
    movi.2d   v1, #0                      ; v1 = zero vect
    subs      x21, x21, #16
LBB0_0:
    add       x21, x21, #16
    add       x22, x20, x21
    ld1       {v2.16b}, [x22]             ; v2 = tape
    cmeq.16b  v4, v2, v1                  ; v4 = tape == zeros
    and.16b   v4, v4, v3                  ; mask with stride
    orn.16b   v4, v0, v4                  ; idx or !(tape == zeros)
    uminv.16b b5, v4                      ; find smallest idx
    umov      w22, v5.b[0]
    subs      w11, w22, #255
    beq       LBB0_0
    add       x21, x21, x22

The first few lines load up the 0 and stride mask vectors.

The only strides I optimize are 1, -1, 2, -2, 4, -4, 8, and -8. At stride 16, you’re only getting one check per loop; might as well just do the basic iteration instead of firing up the vector unit.

Alternative origin of Batman - dream

RobKohr's Blog — Tue, 22 Oct 2024 02:00:00 GMT

Alternative origin of Batman - dream

An experiment in fighting spam on public forms using “proof of work”

Gonçalo Valério — Mon, 21 Oct 2024 23:07:09 GMT

Spam is everywhere. If you have an email account, a mailbox, a website with comments, a cellphone, a social media account, a public form, etc. We all know it, it is a plague. Over the years, there have been multiple attempts to fight spam, with various degrees of success, some more effective than others, some […]

Typescript lets you spread an array into an object

Martijn Arts — Mon, 21 Oct 2024 19:10:49 GMT

I ran into an instance of this Typescript bug recently. Basically, Typescript lets you do this without compiler errors:

const arr = [1, 2, 3];
const x: { a: number[] } = { a: { ...arr } };

x.a.map(n => n + 1);

Note the { ...arr } instead of [...arr]! Running this throws the error

Introducing the Cloud Debugger for Azure

Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development — Mon, 21 Oct 2024 19:10:27 GMT

The Cloud Debugger is an open-source tool for Azure developers to explore, learn, and troubleshoot their Azure cloud environments. Whether preparing for Azure certification, looking to streamline debugging, or aiming to deepen your understanding of Azure, Cloud Debugger provides the tools to make the cloud more discoverable. Background – Getting Azure certified As I prepared […]

The post Introducing the Cloud Debugger for Azure appeared first on Personal Blog of Tore Nestenius | Insights on .NET, C#, and Software Development.

TERMINAL ACCELERATION

asemic horizon — Mon, 21 Oct 2024 17:15:07 GMT

Definite terminality in its quality as definite terminality comes close enough to the process-philosophical notion of singularity that I can’t actually tell them apart (at least at this moment in theory development). Terminality is the process by which differentiation is arrested; definite terminality is the attribute of not being able to further differentiate. Classic neurology […]

A walk after the storm

Nicola Iarocci — Mon, 21 Oct 2024 09:41:05 GMT

A walk on the beach right after the storm. Lots of logs scattered all around, for miles.

And sanderlings running all over the place.

(Almost) Everything You Need to Run a Blog

Home on Lagomorph — Mon, 21 Oct 2024 09:00:00 GMT

We’re living in the cyber dark ages. The primary way most people interact with the internet is through social media. This is the default space people not only listen to others, but also try to make their voices heard - which is sad, because social media isn’t very good for that.

It is liberating to have complete control of your own cyberspace. I’ve been running lagomor.ph, or some derivative of it, for almost ten years, and it’s been an incredibly fulfilling project - Without it, I would have missed out on countless opportunities to express myself and interact with interesting people - the number of friends I have made on account of this blog are countless.

Unleashing the power of Change Data Capture (CDC) for Real-Time Data Syncing

Engineering At Scale — Mon, 21 Oct 2024 07:40:26 GMT

CDC - Working, Application and Challenges

Hide another detail

xenodium.com @alvaro — Mon, 21 Oct 2024 03:00:00 GMT

It's been 5 years since I talked about showing/hiding Emacs dired details in style, a short post showcasing hide-details-mode (built-in) and diredfl (third-party).

While my dired usage increased over the years, my dired config remained largely unchanged. Today, I'll show a new dired tweak.

As you likely suspect by now, I'm a big fan of hide-details-mode. It gives me super clean and minimalistic view of my files.

If I need more details, it's one toggle away using my trusty C-( binding.

Now this is a super minor thing, but for a little while, I wished I could also hide the current directory's absolute path as part of hide-details-mode's toggling. In the same spirit as other hidden dired details, I rarely need to see the absolute path. And if I did, it'd only be a toggle away.

With that in mind, I set out to bend dired my way. I looked at the dired-hide-details-mode built-in code (dired.el) and came across invisibility specs, which I hadn't used before. Dired uses add-to-invisibility-spec and remove-from-invisibility-spec to show and hide details using the invisible property set to dired-hide-details-information.

Now that we know what property to set, we need to find the text to apply it to. Dired offers that via dired-subdir-regexp. All we need to do is match the regular expression and apply our invisible property to the relevant bounds.

(defun hide-dired-details-include-all-subdir-paths ()
  (save-excursion
    (goto-char (point-min))
    (while (re-search-forward dired-subdir-regexp nil t)
      (let* ((match-bounds (cons (match-beginning 1) (match-end 1)))
             (path (file-name-directory (buffer-substring (car match-bounds)
                                                          (cdr match-bounds))))
             (path-start (car match-bounds))
             (path-end (+ (car match-bounds) (length path)))
             (inhibit-read-only t))
        (put-text-property path-start path-end
                           'invisible 'dired-hide-details-information)))))

All that's left is for us to add our new function to a hook, and we're good to go.

(use-package dired
  :hook ((dired-mode . dired-hide-details-mode)
         (dired-after-readin . hide-dired-details-include-all-subdir-paths)))

My Dired window is even cleaner now. The current directory's absolute path is now hidden.

There may be times we need to peek at the absolute path. We can now toggle hiding this detail just like the others.

My first Emacs patch

While this is a rather small change, I figured I could use it to get my toes dipped as a first Emacs contribution. I've since reworked the patch to fit into dired.el's code and submitted for review.

I'm happy to report the tiny feature's now merged to master as of a couple of days ago. Yay! 🎉

It'll be sometime until the feature makes it to a release, but if you're living on the Emacs master edge, it should be available there. While the feature is disabled by default, it can enabled with:

(setq dired-hide-details-hide-absolute-location t)

Happy hiding!

Enjoying this content? Using one of my Emacs packages?

Help make the work sustainable. Consider sponsoring. I'm also building lmno.lol. A platform to drag and drop your blog to the web.

From Text to Talk: Analyzing Open Source TTS Alternatives

Thoughts by Thomas Derflinger — Mon, 21 Oct 2024 03:00:00 GMT

All the big cloud providers like AWS and Azure have an API for the sythesis of text into the spoken word. But there are also young startups like ElevenLabs that offer their innovative solutions in this space. A third option is open source software for those who either do not want to pay for the service of TTS (text-to-speech) or do need on-device TTS. That is why in this article I want to provide an overview of the most important open source TTS alternatives.

Late Summering

Avy Faingezicht — Mon, 21 Oct 2024 03:00:00 GMT

A collection of trips with friends and family in the late summer of 2024. It took me a while to get through the rest of the black and white roll of film I'd started on our Jenner/Sea Rach trip that Nikhil and Rachel planned. I continued to shoot black and white in the city, and got to swap to color on our trip to Sequoia with Amol and Annabelle's crew. Then Gael showed up. Great summer all around.

DynamoDB Considered Harmful

Mon, 21 Oct 2024 03:00:00 GMT

I think DynamoDB is quite a useful database. But, I'm here to tell you should pretty much never use it whether in a greenfield project or a mature product. Let me explain why.

1. It's inflexibility will slow you down.

Initially, DynamoDB's speed and schema-less nature can make development fast. Although DynamoDB isn't modelled around schema's (like a traditional SQL database), it is modelled around queries. This means you need to know the query model up front. In a mature product, you might have a good idea of what possible queries you would want but in a greenfield product it's downright impossible. But, regardless of how mature your product is we all suffer from a fog of war. It's impossible to know what feature requests will be fired at us by a product manager. This is where DynamoDB will start to become as cumbersome as jeans in a rainstorm. Because you originally modelled the database around the queries you knew about, it becomes inflexible to change for the new queries you need to perform. Often times, teams just add new global secondary indexes (which behind the scenes is a complete copy of your database). But these are limited to 20 per table. And it creates another problem, deciding what index to use when. This becomes a headache to maintain and build upon. Some may reason that they can create other tables around the new query model, or change the existing database. But these solutions are also rife with complexity. For example, DynamoDB has a 1MB limit on results from scans. Therefore, to write a migration script you'd need to loop through all this data, update it in memory, remove the old data and then write it back to the database. Try doing that in a no-downtime fashion. SQL on the other hand can be mashed, broken, cracked and mangled in order to create whatever queries you want. You can join tables where there are separations of relationships. And if you ever get into place where the table design is bad you can do a SQL migration - it might take a while but there are no query limits to contend with.

2. You don't have the scale nor will ever need it

DynamoDB was designed with hyperscaler levels of traffic. Sites receiving this level of traffic clock in at less than 50 across the entire internet. Ergo, you do not need DynamoDB's scale. Even Facebook who started on MySQL (but eventually use other supporting databases), didn't need DynamoDB level scale (initially) and when they did they refactored. Choosing a database technology for scaling reasons is akin to getting "CEO" business cards before writing a line of code - it's premature optimisation.

"But what about scale?" I hear you cry

This is a non-argument made by people who haven't launched serious products. Any SQL database with enough money thrown at it can scale plenty fine. SQL databases power some of the most used websites in the world - including all Wordpress sites (43.3% of all websites), Spotify (626M MAU) and Twitter/X (516M MAU). If you reach a scale where SQL becomes the blockage then you likely have the money to solve the problem either by refactoring parts of your app or mitigating the issues (with caches, increased horizontal capacity, sharding etc). In any case, scale is not a reason to discount SQL.

3. You need other technologies to handle basic database functions

Let's say you are using DynamoDB for your little CRUD app. Percy the product manager swaggers up to your desk and asks "Can we add search and sorting of tables to our product pretty please?". After a few umms and ahhs, you probably realise that DynamoDB ain't gonna cut it. You're going to use another tool, like Opensearch or Algolia. Before any DynamoDB rage nerds ask, yes you can do search using "contains" and you can do sorting. But try doing something even remotely complex and it becomes impossible to perform. You know what could add search and sorting to your app, SQL! DynamoDB needs all this other cruft to provide basic functions to your app. And it's not just a case of spinning up some new system and hey presto it works. Nope! You've got to keep those systems in sync (DynamoDB streams or Kinesis), then you've got to configure indexes and so on. Your technology choice is slowing you down from delivery of features to your customers. Guess who also doesn't care about your database - your customers!

4. It's challenging to work on your system locally

Working with DynamoDB locally isn't as simple as running a docker container (like ehem, MySQL or Postgres). So, then you're focused to have a "remote" development environment. Where you have resources deployed to the cloud that are used by each developer. These can work, but provide horrendous experiences. Changes to system configuration have to be deployed and you can't work offline. There are a whole host of problems that arise as a result of systems that cannot just be run on a computer.

In summary

DynamoDB isn't a bad database. It's simply a tool that came from a certain context - in Amazons case high throughput writes and reads. In 99% of cases, you are not going to be in that same context. Focus on getting there first with the power of a SQL database. Most apps are just CRUD, SQL is super good at that - use it, and get back to building the thing.

PS: A good litmus test to see if your tech stack is working is asking "in the last 6 months to what extent has our technology impeded or prevented the development of new features or fixes?". If the answer is > 3 then you probably chose wrong.

Debugging my wife's alarm clock

ntietz.com blog - technically a blog — Mon, 21 Oct 2024 03:00:00 GMT

My wife's alarm clock has been acting up lately. Sporadic at first but then every day, it wouldn't blare in the morning at the set time. Instead, when it was supposed to go off it would... reset itself. The time would start flashing in that "I'm confused because the power went out" sort of way.

Not very useful. You want the alarm to wake you up, not give you a new morning chore.

I volunteered to try to fix it: I'm pretty handy, and I've done some basic electronics repair before. (Desperate times called for desperate measures, and I got that coffee grinder working again with just a little soldering.) Before opening it up, I pointed out that there's a battery. We reproduced the issue first, then changed the battery, and confirmed that that was the culprit^[1]!

But that left the question of why it's doing that. It probably isn't using the battery for power all the time, since it's plugged in. And it does get its timekeeping from the wall—well, more on that later. You can even hear a 60 Hz hum from it if you listen closely, just like our AC frequency in North America.

I wanted to open it up, but there's this warning on it. It says not to open it, and elsewhere it says "no user serviceable parts inside." I could get shocked if I open this.

So I just ignored that, and I took it apart and started poking around^[2] :D

My opening hypothesis was that something had broken—maybe a capacitor, maybe just a bad solder joint—and when the alarm went off it somehow cut mains power and used the battery instead, resetting it. This turned out to be not it.

First, let's appreciate how simple this whole thing is. The alarm clock boils down to one integrated circuit, one 7-segment display panel, a transformer, and a pile of wires, resistors, capacitors, and that sort of thing. Pretty neat, and definitely not something you see in many new products these days (this is over a decade old).

Poking around inside I didn't see any obvious damage, and there wasn't anything disconnected that should be connected. I started to wonder if this was just... how it was designed.

The IC powering the alarm is the LM8560, and its datasheet tells us a lot! It's responsible for storing the time and handling the alarm clock's functionality, including snooze and time setting. And it turns out that yes, the clock is using the wall for timekeeping, which is more reliable than many other sources of time^[3]. And it also has a crystal oscillator built-in which keeps the time when the mains power fails for at least 3 cycles, or 1/20th of a second.

So, uh, what's happening here? My best guess is that when the alarm goes off it's supposed to pull voltage from the battery to power the buzzer, but when the battery is dead, it pulls from AC and manages to drop voltage for the entire IC. This leads it to trigger the "I lost power" display flash and also reset the time to 12:00, which doesn't happen when the battery is present and power is lost.

This isn't reproducible by removing the battery entirely, either. If you take it out, the clock loses all ability to function and just resets constantly. So having a mostly drained battery seems to be doing a little work so that it resets once but then resumes normal functioning (until the next time it tries to go off).

Another neat thing did happen after my debugging session. I set the clock on the steps to take back upstairs, and a few minutes later I heard it go off. Without being plugged in, everything is working (sans display, which does require mains power), and... it sounds better? With AC present, the buzzer sounds distorted and grungy. When it's just on battery, it sounds very clean. This makes me want to find a kit to play with analog synths.

I can't say I'm a big fan of the design of this clock. It is plugged into the wall, but has a load-bearing 9V battery! If anyone knows for sure why this alarm clock does this weird thing with a load-bearing battery, I'd love to learn more!

Update 10/27: A few readers have pointed out to me that it's likely a bad capacitor, which cap it is, and how to test that and/or fix it! I'll try that out this week and update further.

Update 3/23: After months without progress (thanks, Lyme disease), I finally replaced the capacitor. The alarm clock is fully fixed now! Thank you to Asahi Lina and other readers who pointed this out to me!

But I should probably go put her alarm clock back now...

Always reproduce issues before you try to fix them. If you don't know how to reproduce it, you don't know how to check that it's fixed! ↩
While I did this, I do generally know what I'm doing around mains power and am taking that risk for myself. Don't do this without proper training and safety precautions. The effects of high voltage can be quite shocking. ↩
The term to look for is time error correction. Most places still do regulate frequency of mains power, but notable exceptions (Russia, China, etc.) exist. Where I am in the US, the goal is 4,320,000 cycles per day, and the error is actively corrected whenever it exceeds 10 seconds cumulatively. We're the most lax region of the US, and that's still going to be accurate enough for most needs. ↩

2024-10-21

ho.dges.online — Mon, 21 Oct 2024 03:00:00 GMT

Virgin Voyages from Miami

The Digital Nomad — Mon, 21 Oct 2024 01:35:34 GMT

We usually aren’t into cruises. But we always seem to end up on one. When we do go, it’s usually more to hang out with friends than the cruise itself. This time was no different.

We flew into Miami the day before and stayed at the Hyatt Regency Miami

I don’t really have too much to say about the cruise. Just read the review

It’s an all adult cruise. There isn’t as much entertainment as Carnival Cruises. Depending on your point of view, it’s either boring or relaxing.

The food is definitely better. Another difference is that with Carnival, every adult in your cabin has to buy a Cheers Beverage Package if one person buys one.

With Virgin, you can open an optional bar tab that gives you bonus spend.

You also get free basic WiFI.

As far as the stops, we stopped in the Dominican Republic and did a rum tour…

We had a day at sea and then stopped at the THE BEACHES OF BIMINI, THE BAHAMAS. We just went to the beach and hung out in the water.

This is very much a been there done that thing. It was…fine. But we probably won’t do Virgin Voyages again. Vir

ADB helper

Hrafn Thorvaldsson — Sun, 20 Oct 2024 03:00:00 GMT

tl;dr: I’ve rewritten most of my adb related bash scripts and aliases that have accumulated over the years into a single tool that can be found here. Beginnings Link to heading Fifteen years ago I started work on creating my very first Android app. It was part of a university computer science course designed to guide us through the complete software development life cycle — design, gathering requirements, implementation, and deployment — rather than the focus on fragmented software concepts that other classes had.

Astrophysics for People in a Hurry

Vít Baisa — Sun, 20 Oct 2024 03:00:00 GMT

a brief intro to the universe

2024-10-20

ho.dges.online — Sun, 20 Oct 2024 03:00:00 GMT

Walking the London LOOP

Michael Ian Lewis — Sun, 20 Oct 2024 03:00:00 GMT

Introduction Link to heading

The London LOOP is the London Outer Orbital Path, a 150 mile (242km) walking route which encircles the city, like a hiking equivalent of the M25 (the London Orbital Motorway). There’s more good info on the LOOP on the TFL and Ramblers sites.

I split the walk over 9 days between May and September 2024, averaging 15-20 miles (24-32km) per day. These are some notes I took.

The Unreasonable Effectiveness of Monte Carlo Simulations in A/B Testing

Bytepawn - Marton Trencseni — Sun, 20 Oct 2024 01:00:00 GMT

The article illustrates how Monte Carlo simulations serve as a powerful method for exploring statistical concepts in A/B testing, enhancing understanding, improving experimental design and analysis.

Running Vision Encoder Decoder Models in Swift (or any language)

Navan's Archive — Sat, 19 Oct 2024 23:46:00 GMT

Basically an info dump on Vision Encoder Decoder Transformers and how you can run them in Swift using ONNX Runtime, and also how to convert them using coremltools

How I Defeated An MMO Game Hack Author

The Codist — Sat, 19 Oct 2024 20:25:59 GMT

In the late 2000's, I worked at a niche MMO game company. We had a small team, not a lot of money, but a loyal audience. It was a game of skill without any of the usual powerups and unreality, and the players enjoyed the challenge.

Then, one

Album Review - Modern Vampires of the City

Obvious Bicycle — Sat, 19 Oct 2024 20:19:43 GMT

"Wisdom's a gift, but you'd trade it for youth / Age is an honor, it's still not the truth"

Big Plane

jonandnic dot com — Sat, 19 Oct 2024 19:57:02 GMT

One more for the road

jonandnic dot com — Sat, 19 Oct 2024 16:51:19 GMT

Upgrading from a 2016 BMW i3 to a 2023 Chevy Bolt EUV for more range, and hopefully less hassle. But the real question is, can it run on DIY sun power?

Chocolate Tempering

Unterminated — Sat, 19 Oct 2024 08:08:27 GMT

Tempering is necessary to give chocolate a nice appearance and long shelf life. I'll go over what I've learned about the process, and the techniques I use for doing it at home.

Update to my philosophy: less resolutely materialist and utilitarian

Evan Fields — Sat, 19 Oct 2024 03:00:00 GMT

Like many people with my demographics, I’ve long had strongly materialist and utilitarian philosophical intuitions. I think empirical science is a good way to learn about the world; material explanations suffice for most phenomena we encounter in everyday life; more people having more well-being is better than fewer people have less well-being. But over the last two-ish years, these intuitions have unmistakably softened. I attribute this change to several causes, roughly ordered: My friend Doug Kremm, a wise and patient philosopher; Philosophy and philosophers I’ve encountered through Effective Altruist-adjacent media, especially Eric Schwitzgebel; General contemplation and mellowing with age.

The Ultimate Fujifilm X-E1 Camera Guide

Thomas Hunter II — Sat, 19 Oct 2024 03:00:00 GMT

Programming an FPGA with a FOSS toolchain

pera's blog — Sat, 19 Oct 2024 03:00:00 GMT

This is a quick guide on how to program a Sipeed Tang Nano 9K (GOWIN GW1NR-9) using a fully open source toolchain for FPGA programming. I will use Arch Linux but you may find some of the required packages available in your distro’s package repository as well.

Install Yosys and OpenFPGALoader from the official Arch repository with pacman:

sudo pacman -S yosys openfpgaloader

Now you will need to install nextpnr and Project Apicula from AUR. I had a few problems with these packages so I will explain what I’ve done to make it work for me, but try a normal installation first as these problems will likely get fixed in the near future.

First download and install my apicula-git package from AUR.

Then download the nextpnr-git package, also from AUR, and edit the PKGBUILD file to only build your target:

_ARCHS=('himbaechel')

And comment out this dependency:

    himbaechel)
      #makedepends+=('prjapicula')

Now you should be able to makepkg -i, this will take some time so you can brew a mate meanwhile. If everything has gone well you should be able to run nextpnr-himbaechel --version successfully.

At this point you have everything needed to synthesise, route&place, generate bitstreams and upload them into your Tang Nano. Let’s try out the LED example from the Sipeed wiki.

In a new directory create a led.v Verilog file with the following code:

module led (
    input clk,              // clk input
    input rst,              // reset input
    output reg [5:0] led    // 6 LEDS pin
);

reg [23:0] counter;

always @(posedge clk or negedge rst) begin
    if (!rst)
        counter <= 24'd0;
    else if (counter < 24'd1349_9999)       // 0.5s delay
        counter <= counter + 1'b1;
    else
        counter <= 24'd0;
end

always @(posedge clk or negedge rst) begin
    if (!rst)
        led <= 6'b111110;
    else if (counter == 24'd1349_9999)       // 0.5s delay
        led[5:0] <= {led[4:0],led[5]};
    else
        led <= led;
end

endmodule

You will need a CST file for this board; you can find an example in the Apicula’s repo but you can also use this fragment (name it tangnano9k.cst):

IO_LOC "clk" 52;
IO_LOC "led[0]" 10;
IO_LOC "led[1]" 11;
IO_LOC "led[2]" 13;
IO_LOC "led[3]" 14;
IO_LOC "led[4]" 15;
IO_LOC "led[5]" 16;
IO_LOC "key" 3;
IO_LOC "rst" 4;

You are ready to synthesise this design with the following yosys script:

yosys -p "read_verilog led.v; synth_gowin -top led -json led.json"

If the synthesis completes successfully there will be a new led.json file. Next let’s use nextpnr:

nextpnr-himbaechel --json led.json --write pnrled.json --device GW1NR-LV9QN88PC6/I5 --vopt family=GW1N-9C --vopt cst=tangnano9k.cst

After a few seconds you should see a message saying Program finished normally and there should be a new pnrled.json file that we will use to generate the bitstream:

gowin_pack -d GW1N-9C -o led.fs pnrled.json

Now it’s time to grab your dev board :D

Connect the Tang Nano 9K to a USB-C and try this:

openFPGALoader --scan-usb

This should list an FT2232 JTAG Debugger, if it doesn’t then try to unplug and plug again (and avoid sketchy USB hubs ^_^). If you do see the debugger then you can run:

openFPGALoader --detect

This should list a Gowin device and it means we are ready to upload the bitstream:

openFPGALoader -b tangnano9k -f led.fs

Once it’s finished, the six tiny LEDs next to the FPGA should be turning on progressively.

A Guide to CA's Props and San Mateo's Measure T

The David Brownman Blog — Sat, 19 Oct 2024 03:00:00 GMT

A quick overview of what's on California's ballot in November 2024

Read the whole thing.

Copilot+ PC Review: Windows on ARM is finally here

Jason Eckert's Website and Blog — Sat, 19 Oct 2024 03:00:00 GMT

Back in 2017, I wrote a blog post predicting intense competition between Intel x86-based and ARM-based PCs. And finally, after many years of lukewarm development efforts on the part of Microsoft and Qualcomm, that competition is here with Windows 11 Copilot+ PCs running powerful ARM-based Qualcomm Snapdragon X Elite CPUs with built in graphics and AI acceleration.

As shown in the desktop screenshot at the top of this post I’ve been using a Copilot+ PC as a productivity and development workstation this month, and I must say that I’m thoroughly impressed overall. In this blog post, I’ll outline why.

@mtorygreen I'm current the CTO at Grove and Head of Protocol at Pocket Network.

🦉 olshansky 🦁 — Sat, 19 Oct 2024 01:34:31 GMT

We're launching a full rewrite of Pocket early next year (a migration fully rewritten using the latest Cosmos SDK).

How to Configure a Schlage Encode Plus for Home Assistant, HomeKit, Apple Home, and Apple Wallet Keys

Jason Raimondi on Software Engineer — Fri, 18 Oct 2024 22:37:00 GMT

Introduction

After a year of using the Schlage Encode Plus lock with Home Assistant via the homeassistant-schlage custom integration, I’ve finally cracked the code on seamlessly integrating it with Apple Home and enabling tap-to-open functionality with Apple Watch and iPhone. This guide will walk you through the setup process, ensuring your smart lock works flawlessly across multiple platforms.

Prerequisites

Before we begin, make sure you have the following:

An iPhone (iPad or Mac may work, but this is unconfirmed)
A Schlage Encode Plus lock (Official Website)
An Apple Home Hub (e.g., Apple TV 4K)
Home Assistant with the HomeKit Bridge integration configured

Important: If you’ve previously set up your lock using the Schlage app, you must factory reset it (press the button on the lock for 10 seconds) and remove it from the Schlage app. We won’t be using the Schlage app for this setup.

Setup Process

Follow these steps to configure your Schlage Encode Plus for use with Home Assistant, HomeKit, and Apple Wallet:

The future of Postgres?

CRAIG KERSTIENS — Fri, 18 Oct 2024 19:50:56 GMT

I’m often asked what do I think the future for Postgres holds, and my answer has been mostly the same for probably 8 years now, maybe even longer. You see for Postgres itself stability and reliability is core. So where does the new stuff come from if it’s not in the stable core… extensions.

Extensions within Postgres are unlike most other databases allowing you to modify or well extend the standard Postgres behavior. You can build other storage backends, new types, etc. Postgres itself ships with a number of extensions within the “contrib”. The list of contrib extensions hasn’t changed any time recently, but even contrib is a small sampling of what is possible. Beyond core there is a whole world of extensions, I want dig into just a smalls sampling starting with a few in core…

pg_stat_statements is to me the most useful extension that exists. It records what queries were run, how long they took, and a number of other details about the queries. A key extension for managing performance of your database.

auto_explain another one in contrib that is helpful for performance. For queries that run over a certain period of time will automatically log the explain plan–helpful for performance debugging.

pg_prewarm useful to prewarming the cache ahead of a failover.

Let’s jump out of contrib a little bit now. Of course that isn’t everything that ships with Postgres there is more, explore for yourself.

Citus is one of the (to date) more advanced extensions ever created. Citus turns postgres into a sharded, distributed, horizontally scalable database. Citus is especially built to work well for B2B style multi-tenant apps, and now after being acquired years ago part of Microsoft.

Pg_search extends Postgres to support elastic-quality full text search directly within Postgres. I often say Postgres can do just about everything and be pretty capable. Things like time series and search it’s about 80% as good of some of the best in class options out there, but pg_search takes it further making it a full competitor to elastic, but Postgres.

Those are some bigger ones, but you’ve also got a lot of small focused ones as well.

Pg_cron is incredibly handy. Created originally by @marcoslot while at Citus, it’s a small extension that does what it sounds like run scheduled jobs within Postgres. It is now a standard across all the major cloud providers. We leverage it heavily at @crunchydata, but went a little further and built a UI on top so you don’t have to worry about crontab styling of the jobs.

Just yesterday at Crunchy Data we released pg_parquet. There have been some questions about what about other extensions that sort of do similar but not exact but also do a lot of other things. This allows you to seamless copy to and from Parquet files with Postgres. It follows the linux philosophy of small sharp tools. Marco described the why extremely well:

We wanted to create a light-weight implementation of Parquet that does not pull a multi-threaded library into every postgres process. When you get to more complex features, a lot of questions around trade-offs, user experience, and deployment model start appearing.

There is a whole lot of extensions out there. There are big robust ones that have been around forever (yeah I’m talking about you PostGIS), there are ones in contrib, there are ones that are literally created as jokes. Some try to do a lot others as I mentioned very narrow focus like pg_cron. Regardless of whether you believe in large complex extensions that do a lot or small ones that do one thing but do it really well, if you want to know where the future of Postgres is I can’t imagine a world where extensions do not play a heavy roll.

CDC Is a Feature Not a Product

Gunnar Morling — Fri, 18 Oct 2024 16:55:00 GMT

During and after my time as the lead of Debezium, a widely used open-source platform for Change Data Capture (CDC) for a variety of database, I got repeatedly asked whether I’d be interested in creating a company around CDC. VCs, including wellknown household names, did and do reach out to me, pitching this idea.

Roast #42: Coffee Club V8

thoughtfulcoffee — Fri, 18 Oct 2024 16:32:54 GMT

The one with all the shipping issues

Practical Deep Learning, Lesson 3, Stochastic Gradient Descent on the Titanic Dataset

Thought Eddies — Fri, 18 Oct 2024 03:00:00 GMT

Practical Deep Learning, Lesson 3, Stochastic Gradient Descent on the Titanic Dataset

Sonic Cafe, or how to caffeinate studio speakers

rollcat's website — Fri, 18 Oct 2024 03:00:00 GMT

I like smart home stuff - in moderation. Some things end up too smart for my liking. So I got rid of my Homepods and got me a pair of KRK Classic 5s. These studio monitors sound great, and are fantastic value for their price, but have one annoying feature: the built-in circuitry that shuts them down when there’s no audio playing. It makes sense in a studio context, but sometimes I just like to listen to music quietly, and the shutdown threshold is just a little bit too low.

Some people have physically modded their speakers to permanently disable that feature, but I’d prefer to keep my warranty for now, and I actually do like auto-shutdown - just on my own terms.

Nothing reinforces one’s claim to a hacker badge than solving hardware problems in software (or vice versa). I wanted something like macOS’s caffeinate(1) command, except for audio.

How to make an accordion with React Native

Whacky Labs — Thu, 17 Oct 2024 22:30:00 GMT

In the journey of making complex animations with React Native, let’s make an accordion!

I don’t know what is the best term for this component. I call it accordion, others call it expandable list. In the first version of material design this component was called as “Expansion Panel”. In the latest version they simply don’t talk about it anymore. In Apple’s Human Interface Guidelines they named it as “Disclosure controls” and in SwiftUI for some reasons they called it DisclosureGroup. But the documentation writer described the component as:

A view that shows or hides another content view, based on the state of a disclosure control.

Yes that is exactly what I mean.

I’m going to use the assets from the frontendmentor challenge because it looks nice.

Setup

To begin, inspired by Android I dumped all the data under one true global variable called R

const R = {
  colors: {
    white: "hsl(0, 0%, 100%)",
    lightPink: "hsl(275, 100%, 97%)",
    grayishPurple: "hsl(292, 16%, 49%)",
    darkPurple: "hsl(292, 42%, 14%)",
  },
  images: {
    background: require("./assets/images/background-pattern.png"),
    iconStar: require("./assets/images/icon-star.png"),
    iconPlus: require("./assets/images/icon-plus.png"),
    iconMinus: require("./assets/images/icon-minus.png"),
  },
  fonts: {
    regular: {
      name: "WorkSans-Regular",
      weight: 400,
    },
    semibold: {
      name: "WorkSans-SemiBold",
      weight: 600,
    },
    bold: {
      name: "WorkSans-Bold",
      weight: 700,
    },
  },
  strings: {
    header: "FAQs",
    content: [
      {
        title: "What is Frontend Mentor, and how will it help me?",
        body: "Frontend Mentor offers realistic coding challenges to help developers improve their frontend coding skills with projects in HTML, CSS, and JavaScript. It's suitable for all levels and ideal for portfolio building.",
      },
      {
        title: "Is Frontend Mentor free?",
        body: "Yes, Frontend Mentor offers both free and premium coding challenges, with the free option providing access to a range of projects suitable for all skill levels.",
      },
      {
        title: "Can I use Frontend Mentor projects in my portfolio?",
        body: "Yes, you can use projects completed on Frontend Mentor in your portfolio. It's an excellent way to showcase your skills to potential employers!",
      },
      {
        title: "How can I get help if I'm stuck on a challenge?",
        body: "The best place to get help is inside Frontend Mentor's Discord community. There's a help channel where you can ask questions and seek support from other community members.",
      },
    ],
  },
};

Layout

The first step of making an accordion is to make a simple list with all elements all expanded out. Which in code looks something like:

export default function App() {
  return (
    <View style={styles.container}>
      <StatusBar style="light" />
      <Image style={styles.backgroundImage} source={R.images.background} />
      <View style={styles.scrollContainer}>
        <ScrollView
          style={styles.scrollView}
          showsVerticalScrollIndicator={false}
        >
          <View style={styles.contentHeader}>
            <Image source={R.images.iconStar} />
            <Text style={styles.contentHeaderTitle}>{R.strings.header}</Text>
          </View>
          {R.strings.content.map(({ title, body }, index) => {
            return (
              <View
                key={title}
                style={[
                  styles.cell,
                  {
                    borderBottomWidth:
                      index === R.strings.content.length - 1 ? 0 : 1,
                  },
                ]}
              >
                <Pressable style={styles.cellHeader}>
                  <Text style={styles.cellTitle}>{title}</Text>
                  <Image style={styles.cellIcon} source={R.images.iconMinus} />
                </Pressable>
                <Text style={styles.cellBody}>{body}</Text>
              </View>
            );
          })}
        </ScrollView>
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: R.colors.lightPink,
  },
  backgroundImage: {
    width: "100%",
    resizeMode: "cover",
  },
  scrollContainer: {
    position: "absolute",
    backgroundColor: R.colors.white,
    borderRadius: 16,
    width: Dimensions.get("window").width - 32,
    height: Dimensions.get("window").height - 150,
    top: 150,
    left: 16,
    right: 16,
    padding: 32,
  },
  scrollView: {
    backgroundColor: "transparent",
  },
  contentHeader: {
    flexDirection: "row",
    gap: 32,
    paddingVertical: 8,
  },
  contentHeaderTitle: {
    fontSize: 36,
    fontFamily: R.fonts.bold.name,
    fontWeight: R.fonts.bold.weight,
  },
  cell: {
    paddingVertical: 16,
    borderBottomColor: R.colors.lightPink,
  },
  cellHeader: {
    flexDirection: "row",
    alignItems: "center",
    justifyContent: "space-between",
  },
  cellTitle: {
    fontSize: 20,
    fontFamily: R.fonts.semibold.name,
    fontWeight: R.fonts.semibold.weight,
    width: "90%",
  },
  cellBody: {
    fontSize: 16,
    fontFamily: R.fonts.regular.name,
    lineHeight: 22,
    color: R.colors.grayishPurple,
    paddingTop: 16,
  },
});

The interesting bits are with creating a z-stack layout, with an image view in the background and a scroll view in front.

The tricky thing with ScrollView is that it needs a fixed frame and flexible content size. And what this actually means in ReactNative is that the parent of ScrollView should have a fixed size and then the children can be whatever.

But the container view also needs a position: absolute so we can move it in front of the image. And this bit of css does exactly all of that:

  scrollContainer: {
    position: "absolute",
    width: Dimensions.get("window").width - 32,
    height: Dimensions.get("window").height - 150,
    top: 150,
    left: 16,
    right: 16,
    // ...
  }

If you’re coming from a UIKit background like me and if you squint your eyes a bit this math looks like what you need to do for CGRect.

And with our basic setup is done

State management

Next step in making an accordion is to handle the state. And by state I mean either the selected index or an array of indices.

In code this means to first introduce useState and then using the Pressable to update the state. And finally the UI to consume the state.

export default function App() {
  const [selectedIndex, setSelectedIndex] = useState(0);
  // ...
    <Pressable
      style={styles.cellHeader}
      onPress={() => setSelectedIndex(index)}
    >
      <Text style={styles.cellTitle}>{title}</Text>
      <Image
        style={styles.cellIcon}
        source={
          selectedIndex === index
            ? R.images.iconMinus
            : R.images.iconPlus
        }
      />
    </Pressable>
    {selectedIndex === index && (
      <Text style={styles.cellBody}>{body}</Text>
    )}
  // ...
}

Animate changes

And the final step is to animate the changes. In ReactNative that means using Reanimated to animate the height.

function Cell({ text, isOpen }) {
  const cellContainerStyle = useAnimatedStyle(() => {
    const height = isOpen ? withTiming(100) : withTiming(0);
    return { height };
  });
  return (
    <Animated.View style={cellContainerStyle}>
      <Text style={styles.cellBody}>{text}</Text>
    </Animated.View>
  );
}

<Cell 
  text={body} 
  isOpen={index === selectedIndex} />

Now to calculate the actual expanded height, we need to first calculate the max height of the Cell by wrapping the text within a View with position: absolute and using the onLayout callback to record the evaluated height.

Then, we need to make sure the Animated.View has overflow: hidden to clip the content from being rendered outside of the box.

function Cell({ text, isOpen }) {
  const [maxHeight, setMaxHeight] = useState(0);
  const cellContainerStyle = useAnimatedStyle(() => {
    const height = isOpen ? withTiming(maxHeight) : withTiming(0);
    return { height };
  });
  return (
    <Animated.View style={[
      cellContainerStyle, 
      { overflow: "hidden" }
      ]}>
      <View
        style={
          { position: "absolute" }
        }
        onLayout={(e) => {
          if (maxHeight === 0) {
            setMaxHeight(e.nativeEvent.layout.height);
          }
        }}
      >
        <Text style={styles.cellBody}>{text}</Text>
      </View>
    </Animated.View>
  );
}

And this is how you build an accordion with ReactNative.

The code from this experiment is available on https://github.com/chunkyguy/frontendmentor

	0	1	2	3	4	5	6	7	8	9
0x	31	36	59	44	12	61	82	43	23	58
1x	65	41	53	61	9	69	71	43	46	61
2x	61	31	54	37	9	70	65	43	27	59
3x	44	25	37	29	6	38	48	45	30	52
4x	53	10	21	27	8	35	26	11	7	33
5x	50	34	61	41	12	65	75	44	24	95
6x	37	20	27	37	6	46	42	40	29	37
7x	78	41	64	49	9	71	94	67	36	107
8x	34	13	41	23	8	41	50	38	28	37
9x	56	35	55	48	8	68	83	56	20	84

	0	1	2	3	4	5	6	7	8	9
0x	31	36	59	44	12	61	82	43	23	58
1x	65	41	53	61	9	69	71	43	46	61
2x	61	31	54	37	9	70	65	43	27	59
3x	44	25	37	29	6	38	48	45	30	52
4x	53	10	21	27	8	35	26	11	7	33
5x	50	34	61	41	12	65	75	44	24	95
6x	37	20	27	37	6	46	42	40	29	37
7x	78	41	64	49	9	71	94	67	36	107
8x	34	13	41	23	8	41	50	38	28	37
9x	56	35	55	48	8	68	83	56	20	84

Hacker News Personal Blogs 2024 | All Blog Posts

What Even Was 2024?

Year in Review: 2024

My Default Apps at the End of 2024

Virtue Ethics for Animal Rights Activists - A Practical Approach to Becoming a Better Activist

Best of 2024

Looking back at 2024

The work

Releasing On-call (January-March)

Building on our release (April-June)

Scaling to larger customers (July-September)

AI (forever)

Scribe (launched)

Chat (early access)

Investigation (coming soon)

Everything else

Ups and downs

Talks and conferences

Architecture at incident.io (Netflix)

Starting from nothing, StaffPlus, New York

Incidents are a whole organization game, Sev0, San Francisco

Wrapping up

2024 in Review - 100 Books

New year, new platform

App defaults entering into 2025

A quick review of 2024

2024.12.DisappearingMoment

Podcasts

Nerdy Software

Free Font

Bougie Products

Personal Finance and Investing

Reading

Some Topics I Intend to Write About in 2025

Gambling with language models

Challenges Building an Electron App

Collective Effervescence

Generating a triangular navigation mesh from H3 hexagons in R

Introduction

Worked example

Exercises

What You Are Paying For

2024 Year in Review

Winning an Award at Apiture

Collapse of Exercise Habit

Alcohol-Free Months

The Death of Silver Gryphon Games

Journaling About Trips

Journaling Habit

Books Read

RPG Campaign

Beginning of Music Lessons

Previous Years

Building for and debugging an ARM Cortex-M

Interesting Articles I’ve Read in 2024

Trackmania Tracker

Container load balancing with Linux bridges, veths, and IPVS

How I run a coffee club

2024

旅行

影视

娱乐

New Year, New Logo

The CP/M emulator runs on Windows, maybe!

Culture talk: What I learned from building a distributed database that never took off

Unclenching

Adding Pinch-To-Zoom to the Chart

AdditionalAuthorizationParameters in ASP.NET Core 9

Amiga 500 Samsung Keyboard Repair

Review of 2024

Memories of Noah Gibbs

How to Get Package Updates in Hard-to-Reach Places With RenovateBot

2024 in review

The stats

Blog posts wot didn’t make the cut

Recommended reads from 2024

Conclusion

Love for the WDC 65C02

The Power of Now by Eckhart Tolle

The Housemaid by Freida McFadden

	0	1	2	3	4	5	6	7	8	9
0x	31	36	59	44	12	61	82	43	23	58
1x	65	41	53	61	9	69	71	43	46	61
2x	61	31	54	37	9	70	65	43	27	59
3x	44	25	37	29	6	38	48	45	30	52
4x	53	10	21	27	8	35	26	11	7	33
5x	50	34	61	41	12	65	75	44	24	95
6x	37	20	27	37	6	46	42	40	29	37
7x	78	41	64	49	9	71	94	67	36	107
8x	34	13	41	23	8	41	50	38	28	37
9x	56	35	55	48	8	68	83	56	20	84