Introducing Jetro: A High-Performance Rust JSON Query Engine

Processing JSON data at scale requires a balance between expressiveness and performance. While tools like jq are the industry standard for JSON manipulation, embedding a query language into a Rust application requires an engine that is both lightweight and efficient. Enter Jetro, a JSON query engine for Rust that combines a functional-style DSL (Domain Specific Language) and a a high-performance VM.

Jetro is designed to be embeddable, providing a developer-friendly API for Rust developers to query JSON data using a a concise, functional paradigm. By leveraging simd-json as its primary parser, Jetro focuses on minimizing overhead and avoiding unnecessary computations during the query execution process.

The Architecture: Planner-Driven Execution

Unlike traditional query engines that process data in stages—where each stage must fully materialize its output before the next stage begins—Jetro employs a planner-driven approach. This architecture allows the engine to analyze the query chain from the end backward to determine exactly what data is required to satisfy the request.

Demand Propagation

This concept, known as demand propagation, propagation allows Jetro to stream elements through the pipeline. Instead of processing an entire dataset, the engine pulls only the necessary items until the query's requirements are met.

Consider a query chain such as:

let out = j.collect("$.orders .map({ id: @.id, total: @.items.map(@.price * @.qty).sum() }) .filter(@.total > 100) .map(@.id) .first()");

In a typical pipeline, the engine would parse all orders, compute totals for every single order, filter them all, and then finally take the first one. In contrast, Jetro's demand propagation works as follows:

1. Need one ID: The engine identifies that the final operation is .first(), meaning it only needs one result.
2. Need one matching order: It moves backward through the filter and map operations to determine it needs one order that matches the the filter criteria.
3. Pull orders one at a time: It pulls a single order from the source, shapes it, and tests it against the filter.
4. Emit and Stop: As soon as a single matching order is found and the first ID is returned, the engine stops processing further data.

This approach significantly reduces CPU and memory overhead, especially when dealing with large JSON documents.

Expressive Querying and Object Shaping

Jetro Jetro provides a powerful set of tools for object shaping and transformation. It supports complex transformations using match statements and functional operators like .map(), .filter(), .filter(), .filter(), .drop_while(), and .take().

For example, a complex query can shape a variety of different result sets into a single object:

let out = j.collect("{ errors: $.events .drop_while(@.level != 'error') .filter(@.service == 'checkout') .map(match @ with { { level: 'error', message: msg, timestamp: ts } -> { kind: 'error', ts: ts, msg: msg }, { level: 'warn', message: msg } -> { kind: 'warn', warning: msg }, _ -> { kind: 'other' } }) .take(20), slow_orders: $.orders .filter(@.latency_ms > 500) .map({ id: @.id, latency: @.latency_ms }) .take(10), first_vip: $.customers .filter(@.tier == 'vip') .map({ id: @.id, region: @.region }) .take(10) }");

Ecosystem and Tooling

To make the engine more accessible, the author has provided a complementary ecosystem of tools:

• Jetro CLI: A terminal-based tool for testing and querying JSON data, similar to jq.

• Jetro Book: A comprehensive guide for learning the language and the language's syntax and capabilities.

The Challenge of JSON Query Standardization

While Jetro aims to provide a jq-like experience, the community discussion highlights a broader challenge in the JSON query landscape: the lack of a formal specification for the jq language.

As noted by one user on Hacker News, there are multiple implementations of jq (including Python's yq and Go's yq), but they are often only "mostly compatible." This gap in a formal grammar or spec makes the creation of new, high-performance implementations like Jetro even more important, as they provide a clear, and a consistent implementation of a functional query language for Rust developers.

Conclusion

Jetro is a more than just a jq alternative; it is a high-performance, embeddable engine that optimizes query execution through demand propagation. By focusing on a functional paradigm and a more approachable query language, it provides Rust developers with a powerful tool for processing JSON data efficiently.