Skip Regions¶

When splitting input into chunks, some byte ranges must not be split on. Comments, CDATA sections, quoted fields, and string literals all contain bytes that look like record boundaries but aren't.

The SkipRegionFinder trait¶

pub trait SkipRegionFinder: Send + Sync {
    /// Openers that start a skip region (e.g., `b"<!--"`, `b"<![CDATA["`).
    fn openers(&self) -> &[&'static [u8]];

    /// The closer for a given opener (e.g., `"-->"` for `"<!--"`).
    fn closer_for(&self, opener: &[u8]) -> &'static [u8];

    /// Maximum backward scan window in bytes. Default 64 KiB.
    fn window(&self) -> usize { 64 * 1024 }
}

How it works¶

The engine calls in_skip_region(bytes, at, finder) for each candidate split point. The function does a bounded backward scan from at (capped at window() bytes), looking for an unclosed opener.

If openers() is empty, the function returns false immediately (O(1)).

Implementation example: XML comments + CDATA¶

struct XmlSkipRegions;

impl SkipRegionFinder for XmlSkipRegions {
    fn openers(&self) -> &[&'static [u8]] {
        &[b"<!--", b"<![CDATA["]
    }

    fn closer_for(&self, opener: &[u8]) -> &'static [u8] {
        if opener == b"<!--" { b"-->" } else { b"]]>" }
    }
}

// Wire into your splitter:
impl Splitter for MyXmlSplitter {
    fn skip_regions(&self) -> Option<&dyn SkipRegionFinder> {
        Some(&XmlSkipRegions)
    }
    // ...
}

Implementation example: CSV quoted fields¶

struct CsvSkipRegions;

impl SkipRegionFinder for CsvSkipRegions {
    fn openers(&self) -> &[&'static [u8]] {
        &[b"\""]  // double-quote opens a quoted field
    }

    fn closer_for(&self, _opener: &[u8]) -> &'static [u8] {
        b"\""  // double-quote closes it
    }
}

Performance¶

The backward scan is bounded by window() (default 64 KiB). For formats with no skip regions, openers() returns an empty slice and in_skip_region returns false in O(1).

When regions exist, the cost is O(window × num_openers) per candidate. For typical XML/CSV with 1-2 openers and 64 KiB windows, this is negligible compared to the chunk-planning cost.

Build and test¶

The correctness property to test: no split point ever lands inside a skip region. This test uses a CSV-style splitter whose skip_regions treats double quotes as regions, on input containing a quoted field with an embedded newline:

$ cargo test skip_regions
running 1 test
test skip_region_tests::skip_regions_reject_splits_inside_quotes ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.00s

A plain newline splitter accepts a bogus boundary at the y inside "x\ny"; with skip_regions the candidate is rejected and the engine falls back to one safe chunk ([0, len]). Note that with same-byte openers and closers (quotes), the rejection is conservative: the candidate right after the closing quote is dropped too, because the backward scan reads the closing quote as an opener. Prefer distinct opener/closer pairs (like <!-- and -->) when your format allows it.