Schema and types¶

rypipe can infer column names and types, but inference passes cost time and memory. Providing schema_order and field_types up front avoids those passes, stabilizes column order, and enables numeric compare filters.

Avoiding inference passes¶

Some formats need a discovery pass to infer column names. For example, an XML adapter may scan the file to find all field names before parsing. This doubles I/O work and delays the first row.

Provide schema_order when the columns are known:

source = MyAdapter(
    "data.log",
    schema_order=["id", "ts", "amount", "status"],
)

With schema_order, the engine does not need to discover column names. It also sorts columns to this order at finish time, making output deterministic.

Stable column order across chunks¶

In parallel mode, each chunk may encounter columns in a different order. Without a shared schema_order, the engine must reconcile column order at merge time. This adds a small per-chunk cost and can produce unexpected ordering when chunks disagree.

schema_order fixes the output order regardless of the order in which fields arrive.

Casting during parse¶

field_types tells the engine which storage type to build for each column:

source = MyAdapter(
    "data.log",
    field_types={
        "id": "int64",
        "ts": "string",
        "amount": "float64",
        "is_active": "bool",
    },
)

The engine builds the correct Arrow array from the first row. It does not store intermediate strings and recast later. This saves memory and CPU.

Supported types include:

Type	Rust `FieldType`	Notes
`string` / `str`	`FieldType::String`	Default for text data.
`int64` / `int`	`FieldType::Int64`	Parses integer strings during parse.
`float64` / `float`	`FieldType::Float64`	Parses float strings during parse.
`bool` / `boolean`	`FieldType::Boolean`	Parses common bool representations.

In Rust:

use rypipe_core::{ExecutionPlan, FieldType};

let plan = ExecutionPlan::new()
    .type_as("amount", FieldType::Float64)
    .type_as("quantity", FieldType::Int64);

Numeric compare filters¶

Casting during parse is especially important for filters. When a column is stored as Int64 or Float64, the engine can use Arrow compute kernels for Compare filters (<, <=, >, >=). These kernels are vectorized and avoid Python-level comparisons.

If the column is left as a string, the engine must either skip the filter or fall back to Python. Declare the type explicitly to keep numeric comparisons in Rust.

Combining schema hints with fusion¶

schema_order and field_types are part of the ExecutionPlan. They merge cleanly with RenameFields, DropFields, and FilterRows:

result = (
    MyAdapter("data.log", schema_order=["id", "amount"], field_types={"amount": "float64"})
    | RenameFields({"old_name": "amount"})
    | FilterRows(field="amount", op=">", value="100.0")
).to_arrow()

The filter runs on the renamed, typed column. Without field_types, the filter would fall back to Python or be skipped.

Summary¶

Provide schema_order to skip inference and stabilize output columns.
Provide field_types to cast during parse and enable numeric Arrow filters.
Combine both with fused stages for the fastest path through the engine.