Governed RAG on a medallion architecture

Most retrieval prototypes read from an export. The export carries the text but not the permissions, so the index returns passages the person asking is not entitled to see. Teams find this at the security review, after the demonstration has set expectations.

What each layer holds

Bronze stores the source data unchanged, so you can reproduce what the source returned on a given date. Silver resolves types, removes duplicates, applies late-arriving updates, and joins each document to an owner and a region. Gold serves one consumer. For retrieval, that means chunked text with the access metadata on the same row.

You can chunk and embed at bronze, and it is faster to build that way. Chunking at gold instead means each chunk carries what silver resolved, including the group allowed to read it.

The gold job selects the executed contracts, splits the body text into chunks, and copies owning_group onto every chunk as acl_group:

from pyspark.sql import functions as F

# Gold is shaped for one consumer. For retrieval that means chunked text
# with the access metadata attached to the row, not looked up later.
gold = (
    spark.table("silver.contracts")
        .where("status = 'executed'")
        .select(
            F.col("contract_id").alias("doc_id"),
            F.explode(chunk_text("body_text")).alias("chunk"),
            F.col("owning_group").alias("acl_group"),   # carried, never recomputed
            F.col("region"),
            F.col("updated_at"),
        )
)

gold.write.mode("overwrite").saveAsTable("gold.contract_chunks")

Carry the grant with the row

A permission that lives only in the source system does not reach the vector index. Store it as a column on the chunk, then apply it at query time against the groups of the person asking.

Define the filter once in Unity Catalog. A notebook, a dashboard, the index build, and the application then read the same governed table:

-- One rule. The same filter applies to a notebook, a dashboard,
-- the index build and the application.
CREATE OR REPLACE FUNCTION gold.acl_filter(acl_group STRING)
RETURN is_account_group_member(acl_group);

ALTER TABLE gold.contract_chunks
SET ROW FILTER gold.acl_filter ON (acl_group);

Nothing downstream reapplies the rule, which removes the most common place this breaks.

Retrieval filters the search result on the caller's groups, so a chunk the caller cannot read does not become context:

SELECT chunk, doc_id, score
FROM vector_search(
    index => 'gold.contract_index',
    query_text => :question,
    num_results => 40
)
WHERE array_contains(:caller_groups, acl_group)
ORDER BY score DESC
LIMIT 8

Use the caller's identity

In many stalled programs the application authenticates as a service principal with broad rights. Retrieval runs as that principal, and the answer contains whatever the corpus held. Your security team will stop this, and they are right to.

Take the groups from the request rather than from application configuration:

# The identity that matters belongs to the person asking, not to the
# service principal the application happens to run as.
groups = caller_groups_from_request(request)

context = spark.sql(
    RETRIEVE_SQL, args={"question": question, "caller_groups": groups}
).collect()

What the pattern gives you

You get lineage from an answer back to the source row, which auditors ask for first. Revocation works: remove someone from a group, and the next question stops returning those chunks without a re-index or a backfill. Quality also has somewhere to live, because gold is a table you can test on a schedule.

None of this is model work. It is what allows the model to run.

More field notes