Exploring NanoTDB: A Lightweight, Append-Only Time Series Database in Go

Time series data—sequences of data points indexed in time order—is the backbone of modern monitoring, IoT, and telemetry. However, many existing solutions are either too heavyweight for edge devices or too simplistic to provide meaningful aggregation. NanoTDB emerges as a specialized Golang implementation designed to bridge this gap, focusing on an append-only architecture optimized for resource-constrained environments.

The Architecture of NanoTDB

At its core, NanoTDB is an append-only time series database. Unlike traditional relational databases that may perform in-place updates, an append-only system ensures that new data is always written to the end of the storage medium. This approach is particularly beneficial for time series data, which is naturally chronological and rarely requires modification of historical points.

Optimization for SD Cards and Edge Hardware

One of the primary design goals of NanoTDB is efficiency on SD cards, which are common in IoT devices (such as Raspberry Pi clusters). SD cards have limited write cycles and can suffer from significant performance degradation with random writes. By utilizing an append-only strategy, NanoTDB minimizes random write overhead, extending the lifespan of the hardware and improving write throughput.

To ensure data integrity, NanoTDB employs a Write-Ahead Log (WAL), though this feature is optional depending on the user's requirements for durability versus performance. Furthermore, the database is designed to produce highly compressible data. According to the author, tests show an average of 2-3 bytes per data point, with compression efficiency increasing as the volume of data grows.

Beyond the Log File: The Role of Rollups

A common critique of append-only systems is that they are essentially just glorified log files. While a raw log is efficient for writing, it becomes prohibitively expensive to query over long time horizons because the system must scan vast amounts of data to calculate a simple average or maximum.

NanoTDB addresses this through the implementation of rollups. Rollups allow the database to pre-aggregate data into coarser time buckets. Instead of scanning every single 10-second data point from a month ago, a query can target a rolled-up version of that data, drastically reducing read amplification and increasing query speed.

Community Perspectives and Trade-offs

The introduction of NanoTDB has sparked discussions regarding the current state of time series storage in the ecosystem.

The "Append-Only" Trap

Some developers warn that the simplicity of append-only designs is often temporary. As noted by user @PunchyHamster:

"the history of every append only database: * we will make it append only... whoops, devs fucked something up and added a bunch of nonsense that have to be removed, let's figure out how to make at least occasional deletes work"

This highlights the inevitable tension between the purity of an append-only design and the real-world necessity of data correction or deletion (GDPR compliance, for example).

Application in Home Automation

There is a perceived gap in the home automation market for efficient time series storage. For instance, users of Home Assistant have noted that current handling of time series data can be inefficient. While heavy-duty options like Clickhouse exist, a lightweight, embedded solution like NanoTDB could potentially offer a more balanced approach for local edge processing without the overhead of a full OLAP database.

Summary of Key Features

Feature	Implementation/Benefit
Language	Golang
Storage Model	Append-only, optimized for sequential writes
Hardware Target	SD Cards / Low-power IoT devices
Data Integrity	Optional Write-Ahead Logging (WAL)
Efficiency	High compression (avg 2-3 bytes/point)
Query Optimization	Rollups for aggregated historical data