plugin_crm compatibility path is separate.
What a snapshot holds
Names arrive qualified when the table sits outside the default namespace. PostgreSQL returns
accounting.invoices for a table in another schema and plain users for one in public. SQL Server applies the same rule around dbo. Oracle returns OWNER.TABLE when the owner differs from the connected user.
A snapshot carries names and nothing else. It holds no rows, no credentials, and no connection string. System tables never reach it: the database adapter drops pg_*, information_schema, and the Oracle dictionary views before the snapshot leaves the adapter.
Live discovery and cached discovery
The Manager exposes two schema surfaces, and they differ on purpose.
The split follows the two use cases. Somebody who asks for a schema wants the current truth, often right after a migration added a column. Validation runs on the way into every job, so a round trip to the database on each call would cost far more than it returns.
Discovery follows a fixed order, and each gate runs before the next one acquires anything:
1
Check the tenant
Fetcher validates the tenant scope before it touches any resource.
2
Resolve the connection
Fetcher resolves the connection inside that scope. An unknown connection — or one that belongs to another tenant — stops here as
404 Not Found.3
Consult the cache
On the cache-first path, a hit returns at once. Fetcher builds no connector and opens no database session.
4
Open the datasource
On a miss, Fetcher resolves the driver for the datasource type, opens a connector, and reads the catalog. It closes the connector on every path, success or failure.
5
Write through
Fetcher stores the snapshot under the tenant and the config name, then returns it.
What the schema cache buys
The cache turns a database round trip into a lookup. A hit skips the connector build and the catalog read together, so a job that validates twenty tables across three datasources pays for none of them a second time inside the window.
- Key. Every read and every write is scoped to the tenant and the config name. One tenant never sees another tenant’s snapshot and never poisons it.
- Lifetime. Five minutes by default.
SCHEMA_CACHE_TTL_SECONDSsets it on the Manager. - Backing store. The Manager keeps the cache in Valkey or Redis, and falls back to process memory when that store is unreachable.
The cache is an optimization, and Fetcher treats it as one. A failed cache read degrades to a live discovery. A failed cache write still returns the discovered snapshot to the caller. Neither failure reaches your response.
Running without a cache
The Engine schema-cache port is optional. Without one, discovery runs live from the datasource. Validation remains correct; it pays the database round trip each time. Add a cache when validation traffic repeats against stable schemas. Leave it out when the host runs occasional extractions, or when a live read on every call is the behavior you want.Discovery per database
Catalog reads carry a 30-second timeout.
MongoDB inference
MongoDB has no declared schema, so Fetcher builds one in two passes. An aggregation over the collection produces field names. A sample of up to 50 documents then gives each field its type. The field-name pass is bounded by collection size. For collections of up to 10,000 documents, the aggregation processes up to 1,000 documents through$limit; it does not guarantee which documents are selected. Above 10,000 documents, it takes a random sample instead:
The snapshot names fields carried by that limited set or sample. Generic extraction proceeds only when planning validates the selected field in its schema snapshot; do not rely on a field omitted from that snapshot extracting successfully. When aggregation fails on one collection, Fetcher falls back to sampling for that collection and continues discovery of the remaining collections.
Validation before extraction
POST /v1/management/connections/validate-schema takes the same mappedFields map an extraction job carries. Send it before you submit the job.
A clean validation returns 200 with status: success. Schema inconsistencies return 422 application/problem+json with code FET-1060; each issue is an errors detail with a location and message, and can include a structured value. If none of the requested datasources resolves, Fetcher instead returns top-level 400 FET-1062.
The Engine checks selection shape and configured limits before it reads a schema. It then resolves each datasource inside the tenant scope and validates table and field membership against the cache-first snapshot.
Next steps
Extraction jobs
Submit a job, follow it, and read the result.
Connections
Register, test, update, and delete a datasource connection.
Datasources
What each of the five database engines does differently.
Core concepts
The Fetcher model in one place.