Lambda Compiler¶

When you pass a lambda to FilterRows, rypipe analyzes its bytecode at construction time and tries to compile it into a fusable filter spec. Detected patterns are pushed into the Rust parse loop; unknown patterns fall back to Python execution.

How it works¶

FilterRows(lambda r: r["amount"] > 100)
  │
  ▼
_analyze_lambda(fn) → dict | None
  │  Inspects bytecode via dis.get_instructions()
  │
  ├─ pattern detected → convert to filter spec dict
  │  {"field": "amount", "op": ">", "value": "100"}
  │  → fusable via existing Rust CompareLiteral path
  │
  └─ pattern unknown → fall back to Python (current behavior)

The compiler runs once, at FilterRows construction time. If the lambda matches a known pattern, the original lambda is replaced with a fusable predicate. If not, the lambda runs in Python as before.

Supported patterns¶

How pattern detection works¶

The compiler uses dis.get_instructions() to inspect the lambda's bytecode. Each pattern has a specific instruction sequence:

field op literal:

LOAD_FAST(r)  →  LOAD_CONST(field)  →  BINARY_OP([])
LOAD_CONST(value)  →  COMPARE_OP(op)  →  RETURN_VALUE

field.startswith("x"):

LOAD_FAST(r)  →  LOAD_CONST(field)  →  BINARY_OP([])
LOAD_ATTR(startswith)  →  LOAD_CONST(arg)  →  CALL  →  RETURN_VALUE

compound AND:

... first comparison ...
COPY  →  TO_BOOL  →  POP_JUMP_IF_FALSE(target)
... second comparison ...
target: RETURN_VALUE

The compiler filters out RESUME, CACHE, COPY, TO_BOOL, and jump instructions before pattern matching. This makes it resilient to Python version differences in bytecode encoding.

Limitations¶

The compiler detects common patterns but cannot handle everything:

Closures¶

threshold = 100
FilterRows(lambda r: r["amount"] > threshold)  # falls back to Python

The value threshold is loaded via LOAD_GLOBAL, not LOAD_CONST. The compiler cannot resolve it at construction time.

Nested function calls (beyond cast)¶

FilterRows(lambda r: r["name"].strip().lower() == "alice")  # falls back

Method chains with multiple calls are not detected. Simple casts like int(r["age"]) > 30 are fully supported.

Complex compound logic¶

Single-level AND, OR, and NOT are supported. Mixed or nested compound expressions fall back to Python:

# Supported (single-level):
FilterRows(lambda r: r["a"] > 1 and r["b"] < 2)
FilterRows(lambda r: r["status"] == "a" or r["status"] == "b")
FilterRows(lambda r: not r["active"])

# Falls back (nested/mixed):
FilterRows(lambda r: (r["a"] > 1 or r["b"] < 2) and r["c"] == "x")

Use keyword combinators for complex logic:

from my_adapter import FilterRowsAny, FilterRowsNot

FilterRowsAny(
    FilterRows(field="a", op=">", value="1"),
    FilterRows(field="b", op="<", value="2"),
)

Other string methods¶

Only startswith and endswith are supported. Other string methods (strip, lower, contains, etc.) fall back to Python.

What happens when compilation fails¶

When the compiler cannot match a pattern, it returns None and the original lambda runs in Python:

# This lambda falls back to Python execution
f = FilterRows(lambda r: r["name"].strip().lower() == "alice")

# f._filter_spec is None (not fusable)
# f._predicate is the original lambda (runs in Python)

The filter still works correctly, but it runs in Python over the full table instead of in the Rust parse loop. For small files this is fine. For large files, consider rewriting the lambda as a keyword form.

Performance impact¶

For the common case (r["field"] > value), the compiler eliminates:

• Python function call overhead per row (~50ns)
• Python comparison overhead per row (~20ns)
• The need for a CastTypes stage when comparing numbers

On a 10M row file, this saves ~700ms of Python overhead.

Diagnostics¶

To check whether a lambda was compiled, inspect the filter spec:

f = FilterRows(lambda r: r["amount"] > 100)
print(f._filter_spec)
# {'field': 'amount', 'op': '>', 'value': '100'}  ← compiled

f2 = FilterRows(lambda r: r["name"].strip() == "alice")
print(f2._filter_spec)
# None  ← fell back to Python

Recap¶

• The lambda compiler analyzes bytecode at FilterRows construction time.
• Common patterns are compiled to fusable filter specs:
• Field comparisons: r["field"] > 100, r["a"] > r["b"]
• String methods: r["name"].startswith("A"), r["name"].endswith("z")
• Membership: r["status"] in ("active", "pending")
• Truthiness: not r["active"]
• Compound logic: a and b, a or b
• Cast + compare: int(r["age"]) > 30
• Arithmetic: r["amount"] * 2 > 100
• Constants: lambda r: True, lambda r: False
• Unknown patterns (closures, nested method chains) fall back to Python.
• The compiler is a best-effort optimization: if it cannot detect a pattern, the lambda still works correctly, just slower.