rand[om]

Two-step structured outputs for LLMs

2026-01-27T00:00:00+00:00

I’m experimenting with a two-step pipeline for structured outputs. Instead of asking for JSON in a single prompt, I capture the raw free-text response first, then format it in a second call.

Why bother?

Schema validation errors don’t invalidate the LLM response. If the JSON is malformed, I only retry the formatting step. The original reasoning is still there.

Most errors I see in data extraction pipelines are the model messing up JSON tokens, not the response being wrong. A missing bracket shouldn’t waste a complex reasoning chain.

Software Willy Wonka

2026-01-19T00:00:00+00:00

I call my LLM agents “Oompa-Loompas”. It started as a fun way to refer to them, especially when talking to non-technical people. They normally just know “ChatGPT”, but I use Claude Code, Codex, and a bunch of others. At some point it felt weird to keep saying “ChatGPT” when that’s not even what I’m using. So I needed a term that covered all of them. Oompa-Loompas.

Now I realize the metaphor goes deeper than I thought.

Fixing Claude Code's broken file search

2026-01-14T00:00:00+00:00

Claude Code has a nice feature where you can type @ followed by a file name to reference files in your conversation. The problem is that it’s broken. At least for me, it doesn’t work well when the .git folder is not in the repo root; like when using git worktrees. There’s an open issue about it.

After some frustration, I decided to write my own file suggestion hook.

The Hook System

Claude Code lets you override the file suggestion behavior with a custom command. You configure it in your settings file (.claude/settings.json):

Getting better at coding with LLMs

2026-01-11T00:00:00+00:00

In LLM coding, people will spend weeks learning frameworks and tooling, and spend basically zero time learning how to talk to the thing that’s actually writing the code.

I’m pretty sure I’m the #1 power user of coding AI tools at my company (Claude Code, Codex, etc). People asked me to showcase how I use them. As I’m planning it, I notice the main idea I want to share is: using these tools is a skill. Like sports. You don’t get good by copying someone else’s “perfect routine”.

Preact without a build step, including routing and signals

2025-12-21T00:00:00+00:00

Build tools add a lot of complexity. They solve real problems. But for small projects, they add layers of abstraction I don’t need. I want to write code and run it. No waiting for builds, no watching for changes, no debugging why the bundler failed.

I wanted to test the React API without the build step. Preact is a good candidate: lightweight, same API, and it ships ES modules you can load directly in the browser. Combined with import maps, you get clean imports without a bundler.

Taming your shell for LLMs

2025-09-26T00:00:00+00:00

I recently got frustrated with Codex’s command permissions. They don’t let you configure which commands should always be allowed or denied. There’s an issue about it, but it’s been open for almost 4 months as of writing this.

Their sandboxing is less convenient than tools like Claude Code, since it’s more of an all or nothing approach. So I decided to build my own solution.

The Problem

To make LLM CLI agents useful, you need to let them run commands more freely, including ones with network access. But you want control over what they can execute. I think “Claude Code” has a very good user experience for this.

Stop reaching for file appends. Create new files instead.

2025-09-15T00:00:00+00:00

Creating new files is safer and simpler.

The Problem with Appending

Appending to files causes problems:

Multiple threads writing to the same file requires locks
Other processes might append to the same file without you knowing about it (very rare, but could happen)
Buffered writers don’t flush immediately
Race conditions corrupt data
Hard to debug when writes fail
It’s hard to remove just part of the data

Better Approach: Create New Files

Instead of appending, create one file per event/message/log entry.

Simple strategy to run Ansible tasks in parallel

2025-03-05T00:00:00+00:00

This post shows how to run some Ansible playbook tasks in parallel. The approach requires tags for all tasks you want to run in parallel.

How It Works

The script runs multiple Ansible tasks simultaneously by:

Defining a list of tags to run in parallel
Using a ThreadPoolExecutor to create a thread for each tag
Running the Ansible playbook (as a subprocess) with a specific tag in each thread

The example code (below) assumes the playbook files are part of the Python package, and uses importlib.resources to access them. But you can use any method to access the playbook files.

Why I use executable scripts instead of shell aliases

2025-03-04T00:00:00+00:00

I stopped using shell aliases and moved everything into executable scripts.

Main Benefits

No need to source your .bashrc after adding new scripts
No need to edit .bashrc, just drop executable files in your ~/bins folder (already in my $PATH)
Use immediately without reloading your shell
Faster iteration (edit script, re-run command, repeat).
Available in more places:

Executable scripts in your PATH can be used inside other scripts, sub-shells, find -exec, xargs -I{}, and other environments where the aliases defined in your .bashrch are not read by default ¹

Exploiting Cursor to write better code

2025-03-03T00:00:00+00:00

I have been using the Cursor IDE for some time already and I really like its documentation indexing feature. Recently, I discovered a better way to leverage Cursor’s tools to make better references to official documentation and write better code.

The local docs advantage

Here’s what I’ve been doing: Instead of relying on URL-indexed documentation, I download official docs directly to my machine. Take Caddy, for example. I grabbed their docs from GitHub (see src/docs/markdown), keeping just the markdown files in a local subfolder called docs. Now when I’m tweaking my Caddy config, the documentation sits right next to my work.

ZIP as a bundle format

2025-01-11T00:00:00+00:00

At some point in the past, I created a tool for a project (not relevant for this post). This tool generated multiple artefacts from an input (embeddings file, JSON metadata, and other files). The project was successful, but one thing I didn’t like about the “tool” I wrote was that it was not easy to manage those artefacts. The artefacts had to be uploaded and downloaded to some remote(s). Apart from that, the code required all the files to be available, in the same directory, and in sync. For example, if you re-generated the embeddings, you had to re-generate the JSON metadata, but if you forgot to do so, or the process failed and didn’t notice, the tool wouldn’t work as expected.

From HTML template strings to elements

2024-12-23T00:00:00+00:00

Some time ago, I found (and used for a while) the htl library. Before that, I was using raw template literals to build HTML, and then setting some element.innerHTML = newString. htl uses template literals, but instead returns a regular DOM Node object. This makes it a lot easier to add even listeners, setting other properties, and overall manipulating the object before appending it to the current document.

I was curious to see how it was done, and it turns out it’s easier than I thought. The full library is a single index.js file. Most of the code is parsing the HTML string to support different features from their “templating-format” (or that’s what I understood), which allows doing things like:

Handling keyboard shortcuts in JavaScript

2024-12-21T00:00:00+00:00

I was recently reading the source code of mizu.js¹ and I liked the utility “function”² they have to handle keyboard shortcuts/combinations. I decided to slightly modify it to have it as part of my JavaScript utility functions.

The function

This function returns another function itself. You pass in a string that represents a combination of key presses like Shift+Alt+C, and it returns a function that will return true if a KeyboardEvent matches that combination. I had to make some modifications to the original code. First, I converted it to JavaScript, because I don’t like having a build step, and I want to be able to copy-paste this code. I also changed the matcher on the event.key to match on event.code because I found event.key was returning a string generated by the keyboard combination, not the actual key pressed.

Query a list of dictionaries with DuckDB

2024-12-16T00:00:00+00:00

How to query a list of Python dictionaries using DuckDB, without previously converting to another format.

We can pass the list of dictionaries to the query directly and use unnest to convert to a list of rows. We can also expand the struct to multiple columns.

Note: all the dictionaries must have the same keys.

# /// script
# requires-python = ">=3.12"
# dependencies = ["duckdb"]
# ///

import duckdb


DATA = [
    {"foo": 1, "bar": "some string", "baz": 1.23},
    {"foo": 2, "bar": "some other string", "baz": 2.34},
    {"foo": 3, "bar": "yet another string", "baz": 3.45},
]

# Select as struct
rel = duckdb.query("SELECT unnest($data) as x", params={"data": DATA})

# Expand struct to rows
rel = duckdb.query("SELECT x.* FROM (SELECT unnest($data) as x)", params={"data": DATA})

result = duckdb.query("SELECT * FROM rel WHERE foo > 1")

print(result.fetchall())
# [(2, 'some other string', 2.34), (3, 'yet another string', 3.45)]

Here’s a longer script, including timings, and different approaches. I’m using .fetchdf().to_markdown() to pretty-print the results:

Cron jobs inside your web app

2024-12-13T00:00:00+00:00

When you have a web application, almost always, you need to run some functions/scripts regularly. This what cron is for. I wanted to find an easy way to run “cron-like” jobs inside a FastAPI app. The app runs inside docker, and I just need those jobs to run as long as the app is also running.

Since the app is “async”, I found I can just have recurring background tasks as part of the app. We need to use the app lifespan and some functions that run a task, the asyncio.sleep for a pre-defined time. I decided to apply some random jitter to avoid multiple tasks starting at once. (Note: this can also be done in “sync” apps using threads).

My take on Web Components

2024-12-12T00:00:00+00:00

Lately, I’ve been trying to learn more JavaScript.

I wanted to explore web components. After checking out some frameworks and tools, mainly lit, tonic and facet, I wanted to build my own thing.

It ended up being heavily inspired by tonic, but way simpler and with less features. I also really liked lit-html, which you can use as a separate package without having to use lit. It solves the problem of re-rendering only the UI parts that have changed, and it’s awesome at doing it.

DuckDB as a vector database

2024-02-13T00:00:00+00:00

DuckDB version 0.10.0 has been released, bringing some new array functions. You can use them to turn DuckDB into a vector database. (Make sure you have at least version 0.10.0 of DuckDB version to follow python3 -c "import duckdb; print(duckdb.__version__)")

Insert data from numpy arrays

import duckdb
import numpy as np

conn = duckdb.connect(database=":memory:", read_only=False)

conn.execute("CREATE TABLE data (id INTEGER, vector FLOAT4[768]);")


def normalize(vec: np.ndarray) -> np.ndarray:
    return vec / np.linalg.norm(vec)


# insert 3000 random vectors, each vector has 768 dimensions
for i in range(3000):
    vector = np.random.rand(768).astype("float32")
    # normalize vector before inserting
    norm_vector = normalize(vector)
    conn.execute("INSERT INTO data VALUES (?, ?)", (i, vector))

Some notes about datatypes:

Building HTML components from Python functions

2024-01-11T00:00:00+00:00

This post could also be called “Or how to do React in Python” or “HTML as a function of state”.

Most people use templating libraries like jinja2 to render HTML. I think that’s probably the best way to do it in production. However, for very simple / internal / proof-of-concept apps, I wanted to generate the HTML directly from Python functions to avoid needing extra files. I tried using f-strings to do that, but it gets messy pretty quickly. I recently found a nice way to render HTML using lxml. As a nice side effect, the overall architecture is similar to React, where functions become UI components. At the same time, it allows rendering only individual components easily. This can be especially useful when used together with HTMX.

Python profiling and timing utils

2023-09-08T00:00:00+00:00

These are just a few Python snippets that I have been using a lot for timing and profiling functions using only the standard library. You can also find the code here as a GitHub Gist

import contextlib
from functools import wraps
import cProfile
from typing import Optional
import io
import pstats
import time
from pathlib import Path
from functools import wraps


@contextlib.contextmanager
def profile(filename: Path, *args, **kwargs):
    profile = cProfile.Profile(*args, **kwargs)

    profile.enable()
    yield
    profile.disable()

    s = io.StringIO()
    sortby = pstats.SortKey.CUMULATIVE
    ps = pstats.Stats(profile, stream=s).strip_dirs().sort_stats(sortby)
    ps.print_stats()
    with open(filename.with_suffix(".txt"), "w") as f:
        f.write(s.getvalue())

    profile.dump_stats(filename.with_suffix(".prof"))


def profiled(name: Path):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            with profile(name):
                return fn(*args, **kwargs)

        return wrapper

    return decorator


@contextlib.contextmanager
def timeit(output_file: Optional[Path] = None):
    start = time.perf_counter()
    yield
    end = time.perf_counter()
    if output_file is not None:
        with open(output_file, "a") as f:
            f.write(f"{end - start}\n")
    else:
        print(f"{end - start}")


def timed(output_file: Optional[Path] = None):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            with timeit(output_file):
                return fn(*args, **kwargs)

        return wrapper

    return decorator

Usage

You can use a context manager or the decorator both for profiling and timing

Replacing FFI with a CLI

2023-07-23T18:00:00+00:00

I recently read a post from vercel about porting turborepo to rust. In that post, they mention calling a go binary from the rust code instead of having to deal with FFI and C type compatibility.

I wrote a post called “Use a subprocess instead of a dependency”, which follows a similar philosophy, although applied to something different. Here are some extra notes related to the vercel post. As an example, I’ll use a Python app that wants to call some Rust code, but the idea could apply to any combination.

macOS port 5000 not working

2023-07-23T12:00:00+00:00

macOS AirPlay Receiver now listens on port 5000 by default, which conflicts with the default port used by some frameworks (like Flask). This may cause some confusion if you’re trying to forward port 5000 from a different VM machine and you don’t see anything in your browser.

To fix it, you need to uncheck the option in System Preferences -> Sharing -> AirPlay Receiver. Then it works again, otherwise, if you try to make something listen on port 5000 it may not work

Named parameters in shell scripts and Makefiles

2023-05-21T00:00:00+00:00

I recently learned ¹ a nice pattern to improve parameter handling in shell scripts. The pattern involves parameter expansion:

#!/usr/bin/env bash

set -euo pipefail

function main {
    local arg1="${A1:?'FAIL. Provide A1 var'}"
    echo $arg1
}


main

When calling the script (called t.sh throughout this post), it will fail:

bash t.sh
# t.sh: line 7: A1: FAIL. Provide A1 var

But this will succeed:

A1=foo bash t.sh
# foo

It can also be used to provide default/required positional arguments.

Makefile tricks for Python projects

2023-05-19T00:00:00+00:00

I like using Makefiles. They work great both as simple task runners as well as build systems for medium-size projects. This is my starter template for Python projects.

Note: This blog post assumes some basic knowledge of how make and Makefiles work.

Basic configuration

I like using bash as the default shell. Then set some flags to exit on error (-eu -o pipefail), warn about undefined variables and disable built-in rules.

Using IPython for timing and profiling

2023-04-18T00:00:00+00:00

Before you read

I found out the code here doesn’t work outside IPython, I didn’t realize that get_ipython() only works when running inside IPython. But the examples here work in isolated scripts if you run the script using ipython: ipython benchmarh.py.

Introduction

I like using IPython to measure the execution time of Python functions. You can also time functions using the timeit module, but the IPython utilities are just more convenient, and I install IPython in almost all my virtual environments anyway.

Validate Python dataclass types

2023-03-21T00:00:00+00:00

In Python you can create dataclasses with the wrong type. The type checker should show an error, but nothing prevents creating the object. This small function validates that each attribute is of the correct type by using the __annotations__ attribute of the dataclass. You can also create a base class to inherit from, but it won’t work if you override the __post_init__ method in the child classes.

import dataclasses


def val(t):
    """
    Validate the object `t` types based on the
    __annotations__ dictionary.
    """
    for k,v in t.__annotations__.items():
        assert isinstance(getattr(t, k), v)


#Base class
@dataclasses.dataclass(frozen=True, slots=True)
class Base:
    def __post_init__(self):
        val(self)

# Inherit from base class
@dataclasses.dataclass(frozen=True, slots=True)
class User(Base):
    name: str
    age: int

Now when initializing the object, it will throw a runtime exception if any of the attributes is not of the correct type.

Using mmap to share data between processes

2023-03-20T00:00:00+00:00

As part of my recent experiments with mmap¹ I have learned how to share data between processes using a memory-mapped file. Here I’ll show how to do it between two independent Python processes, but the same principles apply to any programming language.

Create a file to store the data

I will use the tempfile module so that the file gets deleted after the script finishes. I prefer using this when learning or testing new things, otherwise my folders end up filled with random files. You can also use another location instead of the temporary file. We need to know where this file is stored. We can use the .name attribute of the temporary file to get its path. Note that this will change every time you re-run the script because the files are temporary.

Running regexes on memory-mapped files

2023-03-17T00:00:00+00:00

Here are some benchmarks I ran to compare the speed of running a list of regexes on all the Markdown files in my Obsidian folder (210 when I wrote this). The benchmark compares running the regexes on memory-mapped files versus loading the file contents as a string and running the regexes on that string. I’m using hyperfine to run the benchmarks.

Considerations

When we memory-map a file, we work with bytes. Python can run regexes over those bytes too, but the pattern has to be .encode()‘ed for it to work.

Moving from Apple Notes to Obsidian

2023-03-14T00:00:00+00:00

After trying a few note-taking apps, I ended up using Apple Notes. It was super fast on macOS and iOS. This was more or less my thinking:

The syncing was good and it worked offline. But it had a couple of drawbacks. The format is not open. There are some scripts to reverse-engineer the Apple Notes SQLite database¹, but that still needs a computer with iCloud sync, otherwise you can’t get the Notes.app SQLite database on your disk.

This is now a stream of my thoughts

2023-03-14T00:00:00+00:00

This blog started as a place to share new things I learned about programming. I initially wrote Jupyter notebooks that I converted to Markdown. Then I started writing Markdown directly.

After some time, I felt like having to open my text editor in my laptop to write a blog post was making me write less. It also felt like I had to write something longer than a tweet. That’s how I started using Apple Notes a lot more¹. Some notes were long, others were shorter than a Tweet. I’ve been enjoying this, so I decided to take a similar approach with my website. Instead of just publishing posts that I’ve thought about and re-read multiple times, I will also make my TILs (Today I Learned), small code snippets, quick benchmarks, random thoughts about work and life, etc. public.

Using Apple Notes as a CMS

2023-03-14T00:00:00+00:00

Intro

I was exploring different ways I could use Apple Notes as a CMS (Content Management System). I found out that you can add a Google account to sync some notes, but those notes are synced using the IMAP protocol to a folder in the email account. This post explains how to use the Gmail API to retrieve those notes. The idea for the CMS is:

Write notes in Apple Notes, avoid using tables and drawings (there may be other elements that are not compatible with this method)
Add a new Google account and synchronize the notes
Duplicate the note you want to sync and move it to the Google account folder in the Notes app.
Use the Gmail API to retrieve the note programatically from another server.

My main objective was being able to read my notes from a linux computer that doesn’t have access to iCloud. This is how I did it.

Using SQLite for logging and ad-hoc profiling

2023-03-12T00:00:00+00:00

I was recently reading the documentation of nnethercote/counts, in there, Nicholas explains a few pain points when using a table as a profiling output. But using SQLite as the “table” can relieve some of those pains. I frequently use SQLite for debugging and I wanted to share some techniques that have been helpful (and relate them to some arguments mentioned in nnethercore/counts)

Defining the table and exposing it to multiple modules

These 2 points are not a problem with SQLite, the “table” is a file outside your program, any module can write to it or read from it.

Python automation utils

2023-03-11T00:00:00+00:00

Here are some utility functions that have been useful when writing automation scripts.

requests

I like using requests or httpx, but if you add a dependency to your script, now you’ll need to create an environment, install the depdencies, etc. just to run it. I like keeping automation scripts in a single file I can copy around. Instead of using the requests, module, you can use this. This function has been adapted from the urlrequest function in fastai/fastcore.

Stream a CSV from S3 to a parquet file

2023-03-11T00:00:00+00:00

I wanted to query a large TSV file stored in S3. To achieve this, I decided to convert it to Parquet and query it using DuckDB. However, I didn’t want to download the full file and then convert it. Instead, I wanted to stream the CSV file directly from S3 and write the output to a Parquet file. Here are a couple of approaches that worked quite nicely.

Setup

I’m running these experiments on an EC2 instance with 8 cores and 32GB of RAM. The data is stored in a 700GB gp2 volume with 2100 IOPS.

Use a subprocess instead of a dependency

2023-03-11T00:00:00+00:00

Sometimes calling a subprocess is better than using a dependency/package. At least in Python, once you add a third-party dependency, distribution becomes slightly harder. I like writing automation scripts in a single .py file. If that script doesn’t use any third-party dependencies, distributing it is as easy as copying the file to the machine. Otherwise you need to package your project, deal with virtual environments, PyPi, pipx, etc. I don’t think all of those tasks are hard, but rsync‘ing a file is easier.

Passing arrays as parameters to SQLite

2022-08-30T00:00:00+00:00

tl;dr

conn.execute(
    "select * from ... where ... in (select value from json_each(?))",
    (json.dumps(list_of_values),),
)

Lists of values in SQLite

SQLite doesn’t have an array type as Postgres does, but we can exploit the JSON extension to do operations with arrays.

An example of why this may be needed. Let’s say you have a table with names, and you want to select all the id’s of people from a list of names. Something like:

Musings about automation

2022-08-14T00:00:00+00:00

Engineers like automating tasks. You find yourself running a set of commands, copying values between files and prompts, and you think I can automate this!

Aside from the joke: “why spend 30 minutes doing something when you can spend 2 days writing a program to automate it”, I believe we don’t think enough about automation. Here I want to share some ideas.

Broken automation is more painful than manual work

Automation is wonderful until it breaks. When that happens, the fix may be easy, but maybe, having the automation in place made you forget how that task is actually done. Now you have to spend time reading code and understanding all the parts of the process. This may take a few minutes or a few days.

The unreasonable developer experience of SQLite

2022-08-10T00:00:00+00:00

In this post, I want to share some reasons why using SQLite turns into a pretty convenient developer experience.

This is a “live” post that may be updated in the future with more contents.

Sharing SQLite data is straightforward, just rsync the file to a new VM or upload it to any file storage service. When you want to share data from other databases, it usually requires generating an SQL dump of the data. Then, as a consumer of the data, you’re responsible for starting a DB server and loading the dump. You also need to know if the DB versions are compatible. The SQLite format has been stable for years, so older versions of SQLite can open databases generated with newer versions.

Building a remote SQLite explorer

2022-06-23T00:00:00+00:00

One of the main pain points of using SQLite in production deployments or VMs is managing the database. There are lots of database GUIs, but only work with local SQLite databases. Managing an SQLite database remotely requires:

Adding a new service to the deployment (like Adminer, sqlite-web or postlite)
Giving the new service permissions to access the volume with the database
Exposing a port to access the service The alternative is usually SSH’ing to the remote VM and use the sqlite3 CLI to manage or explore the database.

With this in mind, I decided to build a remote SQLite management GUI that does not require running any service in the remote VM and only needs an SSH connection between you and the remote machine.

Peeking and backtracking Python generators

2022-05-28T00:00:00+00:00

Python generators are mighty, but they lack a couple of useful features. One of them is peeking the next item without consuming the generator. Even better, what if we could peek any number of items? Another feature lacking from generators is rewinding/backtracking. We will implement both of those features in a couple of different ways. Note: if you need a better tested implementation that lets you peek 1 item, check out more_itertools.peekable.

Extending SQLite with Rust

2022-05-15T00:00:00+00:00

SQLite has a powerful extension mechanism: loadable extensions.

Being an in-process database, SQLite has other extensions mechanisms like application-defined functions (UDF for short). But UDFs have some shortcomings:

They’re local to an SQLite connection, not shared for every process connected to the DB
They have to be defined in your program. That means that you need to have the function available in the same scope as your application. This is where loadable extensions come in. Loadable extensions can be written in any programming language that can be compiled to a shared library/DLL. Then you can just share the compiled object and load them from any application or programming language. In this post, we’ll see how we can use Rust to write an SQLite loadable extension.

Intro

This post is a simplified version of some techniques I learned from phiresky/sqlite-zstd. That is an SQLite extension that enables zstd compression on SQLite, I highly recommend checking it out if you want to look at more advanced examples than this post.

Python, SQLite, and thread safety

2022-04-04T00:00:00+00:00

If you have ever used the built-in sqlite3 module in a multithreaded Python application, you may have seen this message.

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 4316169600 and this is thread id 6190804992.

This code can be used to reproduce the error:

from threading import Thread
import sqlite3

conn = sqlite3.connect(":memory:")


def f():
    conn.execute("select 1").fetchall()


Thread(target=f).start()

The Python docs

After seeing those errors, I checked the Python docs. The sqlite3.connect method accepts a check_same_thread parameter, which can be set to False and the error will disappear.

The best code is easy code

2021-07-23T00:00:00+00:00

The best code is the one that is easy to understand.

The problem

Source

People don’t focus on code readability. Readability is like writing documentation. It takes extra time, more key presses, and you need to stick to it to make it work in the long term. However, sometimes people write code like this:

settings = {"foo": 1, "bar": 2, "baz": 3}

results = [parse_document(settings, r)["somekey"].get("foobar", "") for r in get_documents()]

This code is understandable, but I need to guess:

I suppose parse_document() is returning some kind of dictionary
There’s another indexing operation. The dictionary is probably nested.
What is r?
Is parse_document() mutating settings? Can settings have other keys?

When you find a situation like this, the solution is usually going inside the functions and reading the code, which may be tedious. Yes, maybe you are avoiding creating a couple of variables, but at what cost?

Heroku-style deployments with Docker and git tags

2021-04-25T00:00:00+00:00

In this post I want to explain a new deployment method I came up with while working on drwn.io.

I wanted it to meet a few requirements:

Simple
Based on git tags
Zero-downtime
Easy rollbacks

Creating an empty remote in the server

Imagine you already have your project with some code that is being synchronized with a git service like GitHub. To have a git push based deployment, we need to have our own remote. We will push our code to that remote the same way we do to GitHub.

Bloom filters explained in an image

2021-04-11T00:00:00+00:00

A Bloom filter is a probabilistic data structure present in many common applications. Its purpose is answering the question: “is this item in the set?” very fast and not using a lot of space. The answers can be NO, or MAYBE YES. They work using hash functions, we learned about them some time ago.

For example, one use case of Bloom filters is the following: you have a huge list of malicious URLs. In your browser, before a user navigates to a new URL, you want to check if it’s inside the list of dangerous URLs. You can use a bloom filter to do that! It will take less space than saving the full list of URLs, and if the answer from the Bloom filter is “no” (the URL is not a malicious one), you can safely let the user visit it. If you want to learn how to implement a Bloom filter from scratch, you can do it here. And if you liked this post, feel free to share it and tag me!

Understanding Bloom Filters by building one

2021-04-06T00:00:00+00:00

What is a Bloom Filter?

A Bloom filter is a probabilistic data structure. It tells you if an element is in a set or not in a very fast and memory-efficient way. A Bloom filter can tell if an element is not in the set (“being 100% sure”) or that it may be in the set, but not “being 100% sure”. It only has 2 operations: add, to add an element, and query, to check if an element exists in the set or not.

Hash functions explained in an image

2021-03-11T00:00:00+00:00

Hash functions are a fundamental part of modern programming. They map a variable-sized input to a fixed-size output.

We can use them to:

Verify the integrity of files and make sure they have not been modified.
Save a password in a database without the risk that if somebody steals the data, they will be able to know the passwords.
Create a hash table to store items. With that, we can find them faster and using less memory.

Multiple, shared, in-memory SQLite databases in Python

2021-02-16T00:00:00+00:00

This works only in the same Python process, you can’t share an in-memory SQLite database between processes in this way.

import sqlite3

# NOTE: you need to use uri=True

# 3 connections to the same in-memory database (DB1 / memdb1)
DB1_1 = sqlite3.connect("file:memdb1?mode=memory&cache=shared", uri=True)
DB1_2 = sqlite3.connect("file:memdb1?mode=memory&cache=shared", uri=True)
DB1_3 = sqlite3.connect("file:memdb1?mode=memory&cache=shared", uri=True)

# 2 connections to a *new* in-memory database (DB2 / memdb2)
DB2_1 = sqlite3.connect("file:memdb2?mode=memory&cache=shared", uri=True)
DB2_2 = sqlite3.connect("file:memdb2?mode=memory&cache=shared", uri=True)

# create a table in both DBs
# DB1
DB1_1.execute("create table if not exists dict(key text, value text)")

# DB2
DB2_1.execute("create table if not exists dict(key text, value text)")

# insert some values in both DBs with *one* of the connections
with DB1_1:
    DB1_1.execute("insert into dict values ('asdas', 'foobar')")

with DB2_1:
    DB2_1.execute("insert into dict values ('barasd', 'barfoo')")

# check that all the connections for a database have the same data
# DB1
assert DB1_1.execute("select * from dict").fetchall() == [("asdas", "foobar")]
assert DB1_2.execute("select * from dict").fetchall() == [("asdas", "foobar")]
assert DB1_3.execute("select * from dict").fetchall() == [("asdas", "foobar")]

# DB2
assert DB2_1.execute("select * from dict").fetchall() == [("barasd", "barfoo")]
assert DB2_2.execute("select * from dict").fetchall() == [("barasd", "barfoo")]

Vanishing feedback button with FastAPI and hyperscript

2021-02-11T00:00:00+00:00

Feedback is one of the most importante things when you are creating a product (probably the most important one!), so it should be easy for users to give feedback. In this post we’ll see how to implement a feedback button easily. This exact code is what I’m using at drwn.io, a little project I’m working on with a friend.

The button does the following: after you use it, it will change its text to “Thank you!”, then it will fade out we will have our backend handle the message. Let’s see it in action:

Managing Python environments with mamba and pip-tools

2021-01-25T00:00:00+00:00

Some time ago I published a post about managing Conda environments. My workflow has changed quite a bit since then, and I’ve moved to venv + pip-tools, but I’ll leave that for another post. Today I want to talk about new experiments I’m making with mamba + pip-tools. If you don’t know about mamba, it’s kind of a miniconda replacement. It just works a lot better, and it’s VERY fast.

Triggered by this tweet, I feel like writing about this new experimental workflow. We will see how it works, some problems that may come up and workarounds to fix them.

Single file applications

2020-11-03T00:00:00+00:00

Storytelling is part of the human essence. Stories have let humans survive until today, they became the medium to move information between individuals.

Programming can be considered another form of moving information. With different levels of abstraction, programming becomes a medium to transfer information between humans and computers.

A programming paradigm mixing those two concepts is literate programming. According to its definition:

Literate programming is a methodology that combines a programming language with a documentation language, thereby making programs more robust, more portable, more easily maintained, and arguably more fun to write than programs that are written only in a high-level language. The main idea is to treat a program as a piece of literature, addressed to human beings rather than to a computer. ¹

Notes on Simple Made Easy

2020-09-22T00:00:00+00:00

These are my notes on the talks “Simple Made Easy” by Rich Hickey.

Simplicity is a prerequisite for reliability - Dijkstra

Simple:

One role.
One task.
One concept.
One dimension.
Lack of interleaving. (Does not mean there’s only one thing)

Easy:

“Near”. To our understanding, skill/set or capabilities.
Easy is relative.

We can only make reliable those things we can understand.

We can only consider a few things at a time. If a piece is intertwined with another we already have to keep both in mind. It’s combinatorial.

Short posts

2020-07-31T00:00:00+00:00

Our attention spans are short. Even more when looking for new information. We get flooded by different sources. Most of them contain little useful information. Content should be concise, packed with information and clean of empty prose. Focused around examples. Remove everything not related to the title.

When was the last time you spent more than 15 minutes reading a single piece of information?

That is why I started writing recipes.

Connect to remote docker host via SSH.

2020-07-29T00:00:00+00:00

I could not get reliable connections with docker-machine or docker context. This is how I got 100% reliable connections through ssh forwarding.

It’s also more secure. You don’t need to expose any additional ports.

# create bash function
function dockcon() {

    # if used with the argument `unset`, close the connection
    # see the part after this conditional to understand it better
    if [[ "$1" == "unset" ]]; then
        # get the PID of the ssh process and kill it
        kill "$(cat docker_connection.pid)"

        # unset the environment variable
        unset DOCKER_HOST

        # remove the file with the PID
        rm docker_connection.pid

        # end
        return
    fi


    # generate a random integer between 27000 and 37000 to choose as a port
    PORT=$(( ( RANDOM % 10000 )  + 27000 ))

    # connect via ssh
    # forward the remote docker socket to a local port on your machine
    # the argument "$1" is the remote host
    # echo the PID of the ssh connection to a text file
    ssh -NL localhost:$PORT:/var/run/docker.sock "$1" & echo $! > docker_connection.pid

    # set the environment variable to tell docker which host to use
    export DOCKER_HOST="tcp://localhost:$PORT"
}

Usage:

Run your containers as non-root

2020-07-22T00:00:00+00:00

Docker runs as root. The programs executing inside containers too. Make it a bit better.

# do everything you need as root (maybe installing some dependencies)

# create a non-root user
RUN addgroup --gid 1001 appgroup
RUN useradd --create-home --gid 1001 --uid 1001 appuser

# set new workdir
WORKDIR /home/appuser

# activate non-root user
USER appuser

# copy new files with correct permissions
COPY --chown=appuser:appgroup app.py .

If you need to mount or use files from your file system, give permissions to the UID/GID defined above:

Put your bash code in functions

2020-06-16T00:00:00+00:00

I wrote my degree dissertation in jupyter notebooks. Then I converted them to markdown and finally to pdf with pandoc. I also had to output an office file so that my tutor could work on it too. The initial code was:

#!/usr/bin/env bash


ProcessName="PDF Expert"
number=$(ps aux | grep -v grep | grep -ci "$ProcessName")
# echo $number
if [ "$number" -le 0 ]
then
    open -a 'PDF Expert'
fi
pandoc final.md \
    format.yaml \
    --filter pandoc-citeproc \
    --table-of-contents \
    --number-sections \
    --pdf-engine=xelatex \
    --indented-code-classes=python \
    --highlight-style=pygments \
    --template=./eisvogel2.tex \
    --listings \
    -o final.pdf
pandoc final.md \
    format.yaml \
    --filter pandoc-citeproc \
    --table-of-contents \
    --number-sections \
    --indented-code-classes=python \
    --highlight-style=pygments \
    --template=./eisvogel.tex \
    -V lang=es \
    -o final.docx

open final.pdf

This code does the following:

Conda management

2020-02-21T00:00:00+00:00

Manage conda environments and Jupyterlab easily.

TL;DR Use this script to create a base conda environment with Jupyterlab and some plugins, and this to create new environments and make them available when launching Jupyter.

Managing environments in Python can be tricky. I used to have a default script to create a conda environment, in that script I would install all the packages I consider basic, plus Jupyterlab and some plugins. Yes a new fresh Jupyterlab+plugins for every environment.

Education before innovation

2020-02-07T00:00:00+00:00

We live in a fast changing world, everyone seems to be working on the next big thing while you still haven’t got used to the previous big thing. We look at technology, science and innovation as our new saving god, while overlooking something more fundamental: education.

It seems we just want to solve everything with technology and new discoveries: travelling, diseases, waiting times to visit a doctor, etc. I believe many of those things could be alleviated with a better education instead of throwing money and headlines at the next company trying to solve it.

Fixing a bug in the Haskell installer

2020-01-16T00:00:00+00:00

The story of a bug fix. I found a bug in the default Haskell installer ghcup, fixed it in my computer and had the fix merged to the public installer.

I started doing the MIT course Programming with Categories, so I had to install Haskell on my computer. But following the recommended method kept throwing me an error: mktemp: too few X's in template ‘ghcup’. After some time searching I could only find 1 discussion where someone had had a similar problem.

Divide, Search, Abstract

2019-11-28T00:00:00+00:00

Some problems are harder than others, but having a system to approach them makes a big difference.

Yesterday I finished teaching my first Python course. Apart from teaching about generators, decorators, coroutines, multiprocessing, pandas, flask, pytest, docker… and a long list of development tools and Python modules, I tried to transmit an important message I believe is a lot more fundamental than knowing all the technicalities, and it is problem solving.

Language identification with fastText

2019-08-20T00:00:00+00:00

When dealing with a multilingual dataset doing language identification is a very important part of the analysis process, here I’ll show a way to do a fast ⚡️ and reliable ✨ language identification with fasttext.

<TL;DR>

wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin

import fasttext

lid_model = fasttext.load_model('lid.176.bin')

def detector(text):
    # return empty string if there is no tweet
    if text.isspace():
        return ""
    else:
        # get first item of the prediction tuple, then split by "__label__" and return only language code
        return lid_model.predict(text)[0][0].split("__label__")[1]

df['language'] = df['Tweet'].apply(detector)

</TL;DR>

First fo all we need to download a dataset to play with. We’ll use the kaggle CLI tool. Run the following command after installing it or download the dataset directly from kaggle.

Linking notebooks

2019-07-25T00:00:00+00:00

There is a little trick I just found about, which I think can be very useful when you split tasks across different notebooks.

Jupyter notebooks keep gaining popularity and use cases, the rapid growth of the tooling ecosystem around it is just one proof (for example voilà or panel).

Most people choose Jupyter when publishing a course or tutorial on Github and that’s great, but when there are a lot of notebooks it can become messy.

Formatting jupyter notebooks with black

2019-07-22T00:00:00+00:00

After nearly a year coding in Python (although not consistently), I started trying code formatters and discovered they were more useful than I thought. Here are some reason why.

First, I must admit these tips may be biased by the fact I use Jupyter Notebooks quite a lot. I think they are an incredible tool for learning although they may also instill some bad coding habits.

Before talking about black lets go through the step to install it and use it with Jupyter Notebooks.

About me

2019-06-01T00:00:00+00:00

My name is Ricardo Ander-Egg Aguilar. I am a medical doctor working as a sofware engineer and machine learning developer. I currently live in Barcelona.

I really love programming, maths, statistics and thinking about how current problems can be solved.

Regarding my other interests, I have been a swimmer for 20 years, sport is just a part of me. I love hiking and travelling, I am also a salsa dancer.

rand[om]

Two-step structured outputs for LLMs

Why bother?

Software Willy Wonka

Fixing Claude Code's broken file search

The Hook System

Getting better at coding with LLMs

Preact without a build step, including routing and signals

Taming your shell for LLMs

The Problem

Stop reaching for file appends. Create new files instead.

The Problem with Appending

Better Approach: Create New Files

Simple strategy to run Ansible tasks in parallel

How It Works

Why I use executable scripts instead of shell aliases

Main Benefits

Exploiting Cursor to write better code

The local docs advantage

ZIP as a bundle format

From HTML template strings to elements

Handling keyboard shortcuts in JavaScript

The function

Query a list of dictionaries with DuckDB

Cron jobs inside your web app

My take on Web Components

DuckDB as a vector database

Insert data from numpy arrays

Building HTML components from Python functions

Python profiling and timing utils

Usage

Replacing FFI with a CLI

macOS port 5000 not working

Named parameters in shell scripts and Makefiles

Makefile tricks for Python projects

Basic configuration

Using IPython for timing and profiling

Before you read

Introduction

Validate Python dataclass types

Using mmap to share data between processes

Create a file to store the data

Running regexes on memory-mapped files

Considerations

Moving from Apple Notes to Obsidian

This is now a stream of my thoughts

Using Apple Notes as a CMS

Intro

Using SQLite for logging and ad-hoc profiling

Defining the table and exposing it to multiple modules

Python automation utils

requests

Stream a CSV from S3 to a parquet file

Setup

Use a subprocess instead of a dependency

Passing arrays as parameters to SQLite

tl;dr

Lists of values in SQLite

Musings about automation

Broken automation is more painful than manual work

The unreasonable developer experience of SQLite

Moving and sharing data, backups

Building a remote SQLite explorer

Peeking and backtracking Python generators

Extending SQLite with Rust

Intro

Python, SQLite, and thread safety

The Python docs

The best code is easy code

The problem

Heroku-style deployments with Docker and git tags

Creating an empty remote in the server

Bloom filters explained in an image

Understanding Bloom Filters by building one

What is a Bloom Filter?

Hash functions explained in an image

Multiple, shared, in-memory SQLite databases in Python

Vanishing feedback button with FastAPI and hyperscript

Managing Python environments with mamba and pip-tools

Single file applications