DataHub

Knowing what is in your corpus

Context sources makes the case that every source has a freshness and a trust level, and that a payload assembled from sources of different ages looks consistent to the model. It leaves a question open: how do you know those things about a source at all?

DataHub is one answer. It describes itself as "a 3rd generation data catalog that enables Data Discovery, Collaboration, Governance, and end-to-end Observability", and it is an open-source metadata platform rather than anything AI-specific.

The reason it belongs here is narrower than data governance in general. Once a retrieval corpus is real, three questions arrive that nothing in your AI stack can answer: what is in it, where each piece came from, and whether you are allowed to use it.

The model, in three terms

URNs identify entities uniquely. Entities are the objects: datasets, pipelines, dashboards. Aspects are typed metadata attached to an entity.

The consequence of that shape is that metadata is structured rather than documentation. "This dataset contains personal data" is an aspect you can query across everything, not a sentence in a README nobody reads.

Metadata arrives through REST, GraphQL, and AVRO-based Kafka streams, and the platform is stream-oriented, so changes appear within seconds. A central search index and graph make it discoverable across the organisation.

Lineage is the part that pays for an AI system

Lineage records where data came from and what it feeds.

For retrieval, that answers the question you cannot otherwise answer: a chunk retrieved into a prompt came from a document, which came from a table, which came from a pipeline, which read a source system. When an answer is wrong because the data was wrong, lineage is how you find where it went wrong rather than guessing.

It also answers the inverse, which is the one that arrives under time pressure. A source system had a problem last Tuesday. Which parts of your corpus are affected, and do you need to re-index? Without lineage that question has no answer and somebody re-indexes everything.

Context sources argues for knowing each source's staleness well enough to state it. A catalogue is where that lives when you have more than a handful of sources.

Governance, which is the compliance half

Security and privacy concerns sets out a pre-launch list including what your own systems store and what you tell users. A retrieval corpus is one of those systems, and it is usually the one nobody added to the data map.

Classification aspects are how you keep that honest. Tag the datasets holding personal data, and the question "does our RAG index contain personal data" becomes a query rather than an investigation. That matters because indexing is where the decision gets made silently: a document that was fine to store may not be fine to send to a third-party model.

When it is worth it

Not for a small corpus. Ten documents from one source need a note, not a platform. This is infrastructure, and it has an operational cost.

Worth it when sources multiply. Several systems feeding an index, owned by different teams, at different refresh rates, is the point where nobody can answer the three questions above from memory.

Worth it when you already run one. Many organisations have a catalogue for analytics reasons. If yours does, registering your retrieval corpus in it is cheap, and it is the difference between your AI feature being inside the data governance story and being an exception to it.

That last case is the common one, and the practical version of this page: the question is usually not whether to adopt DataHub for your AI system, but whether your AI system is visible in the catalogue the rest of the company already uses.

Further reading

• DataHub, Architecture: the metadata model, ingestion channels and the search and graph surface.
• Context sources: freshness and trust as properties of a source.
• Security and privacy concerns: the data map your corpus belongs in.
• What is RAG: the pipeline whose inputs this describes.