Creating new files is safer and simpler.

The Problem with Appending

Appending to files causes problems:

• Multiple threads writing to the same file requires locks
• Other processes might append to the same file without you knowing about it (very rare, but could happen)
• Buffered writers don’t flush immediately
• Race conditions corrupt data
• Hard to debug when writes fail
• It’s hard to remove just part of the data

Better Approach: Create New Files

Instead of appending, create one file per event/message/log entry.

Benefits:

• No locks needed
• Multiple threads write safely
• Each file is atomic
• File system handles concurrency
• Easy to read with Unix tools
• Easy to do partial deletes of the data

Implementation

File Naming

• Include timestamp: 20250828_143022_event.log
• Add metadata: user_123_login_success.log
• Use unique IDs: req_uuid_abc123.json

I normally use JSONL, with each file being a single JSON line, that makes merging + querying (with duckdb) easy.

Reading Data

Use standard Unix tools or duckdb:

cat *.log | sort
ls -t *.log | head -10
rg "ERROR" *.log

duckdb -c "select * from read_json_auto(events/*.jsonl')"