sqlite-worker
Sqlite-worker is a concurrency-safe wrapper implementing better-sqlite3, which safely isolates database operations to a background thread in Node.js applications.This package grew out of a data-ingestion/processing pipeline where SQLite was serving as both a durable queue and data store. Instead of allowing each read/write module to interact with the database directly, I wanted a single persistence boundary that was predictable under concurrent workloads and easy to reuse across projects.
Overview
SQLite is often a good fit for local-first tools, data pipelines, and small system services, but usage can become complicated as projects grow, particularly around threading and asynchronous processing. sqlite-worker helps to simplify this by creating a single boundary for database operations. This allows application code to treat persistence as a service, rather than as a shared mutable state.
System Outline
- User provides path to schema and database
- Worker owns SQLite connection
- Operations are managed in one location
- Main app sends requests to the worker — queuing is handled internally by the worker
- Response comes back through defined interfaces
Key Technical Decisions
Dedicated worker ownership of the SQLite connection
- Isolated database ownership allows operations to be independent of the main thread.
Message-based operation
- Message passing creates a cleaner communication surface rather than a direct, scattered database access.
- Using worker-messages takes advantage of the internal message queue (event loop) that exists inside of the worker. Therefore, we have no need to build an explicit message queue for database events.
Reusability across multiple projects
- Originally wanting a no-nonsense durable queue, I found myself using SQLite in ever-widening contexts. As usage expanded, the need for a thread-safe local database option emerged — and the package grew from there.