Chunking is a retrieval decision, not a preprocessing step

A team reports that the assistant gives confident, wrong answers about their contracts. They have tried three models and two rerankers. The corpus was split every thousand characters, so a clause and the exception that qualifies it live in different chunks, and retrieval returns one of them. No model fixes that, because the passage the model needed was never retrieved.

Split on the document's structure

Legal documents have clauses. Manuals have procedures. Support articles have a problem and a resolution. Splitting on characters ignores every one of those and cuts through the middle of the unit a reader would have quoted.

# Chunking is a retrieval decision. Split on the document's own
# structure, keep the heading path, and let neighbours overlap.
def chunk(doc: Document) -> list[Chunk]:
    out = []
    for section in doc.sections():                    # headings, not bytes
        for window in section.windows(target=700, overlap=120):
            out.append(Chunk(
                text=window.text,
                heading_path=" > ".join(section.ancestors()),   # prepended at index time
                doc_id=doc.id,
                effective_from=doc.effective_from,             # answers "which version"
                acl_group=doc.owning_group,
            ))
    return out

Three things travel with the chunk and each answers a question the system will be asked. The heading path tells the model where the passage came from, which is most of the improvement people attribute to a bigger model. The effective date lets the answer be correct about which version applied. The owning group is what makes the chunk filterable, so retrieval can respect permissions.

The tuning that actually moves the number

The order matters more than any individual row. Teams tend to start at the bottom, because changing a model is a config change and changing a pipeline is work. The bottom of the table is where the least improvement is.

Measure retrieval on its own

Score recall at k against a labeled set before you look at answer quality. If the right passage is not in the top twenty, nothing downstream can recover, and you will spend weeks tuning a generator that was never the constraint.

More field notes