> ## Documentation Index > Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt > Use this file to discover all available pages before exploring further. # Fetcher core concepts > The Fetcher model in one place: connections, datasource types, schemas, extraction jobs, filters, results, hosts, and the tenant scope that separates them. Fetcher has a small model. You register a **connection** to an external database. Fetcher **discovers the schema** behind it. You submit an **extraction job** that names the fields you want. Fetcher gives you back a **result** with an integrity digest over the exact bytes it produced. This page defines each noun. Each section then points to the page that covers it in full. ## The model at a glance *** | Concept | Definition | | --------------- | ----------------------------------------------------------------------------- | | Connection | A stored, named reference to one external database. | | Datasource type | The database engine behind a connection. Fetcher accepts five. | | Schema | The tables and fields that Fetcher finds on a datasource. | | Extraction job | An asynchronous request for named fields from named tables. | | Filter | A per-field condition that limits the rows a job returns. | | Result | The extracted rows, plus the digest that identifies them. | | Engine | The extraction core. It decides what an extraction means. | | Host | The application that runs the Engine. It owns queues, storage, and transport. | | Tenant | The only isolation boundary. Every operation carries one tenant ID. | ## Connections *** A connection holds the datasource type, host, port, database name, credentials, and optional TLS settings for one external database. A short `configName` identifies it inside the tenant, and every job and schema call addresses the datasource by that name. Fetcher encrypts the password with AES-256-GCM before it reaches storage. The stored record also keeps the key version that protected it. A connection never travels back to a caller with its password. Read [Connections](/en/fetcher/fetcher-connections) for the full lifecycle, the connection test, and the rule that blocks an update or a delete while jobs still run. ## Datasource types *** A connection declares one of five types. Send the value upper-case: request validation matches these five strings exactly and rejects any other casing with `400`. * `POSTGRESQL` * `MYSQL` * `ORACLE` * `SQL_SERVER` * `MONGODB` The five behave differently under the hood. PostgreSQL and SQL Server qualify tables outside the default schema. Oracle works in owner namespaces. MongoDB has no declared schema at all, so Fetcher infers one from a document sample. ## Schemas *** A schema snapshot lists the tables of a datasource and the fields of each table. Fetcher builds it directly from the datasource, so you do not maintain a separate catalog. Two things build on a snapshot. Schema **validation** checks a job mapping against it before extraction starts. Schema **caching** keeps a recent snapshot under the tenant and the config name, so repeated work skips the round trip to the database. Read [Schema discovery](/en/fetcher/fetcher-schema-discovery) for live-versus-cached behavior, the per-database differences, and the validation report. ## Extraction jobs *** A job names the fields to extract, per table, per datasource: ```json theme={null} { "dataRequest": { "mappedFields": { "my_postgres": { "accounts": ["id", "email", "created_at"] }, "my_mongo": { "transactions": ["*"] } } } } ``` One job can span several datasources, several tables per datasource, and several schemas. Use `["*"]` to take every field of a table. The Manager accepts a job and answers `202 Accepted`. A repeated request inside a five-minute window answers `200 OK` and returns the job that already exists. A job that failed does not suppress a retry. Engine limits bound generic datasource work. The defaults allow 10 datasources per extraction, 20 tables per datasource, 50 fields per table, and a five-minute deadline. Embedded Engine callers can lower, but never raise, those limits with `ExtractionRequest.Overrides`. The standalone Manager job payload has no limit-override field, and the Worker accepts only a positive `ENGINE_MAX_RESULT_BYTES` override. The `plugin_crm` portion uses the Worker's explicit compatibility path. ## Filters *** A filter narrows the rows of one table. The job payload nests filters four levels deep: datasource, then table, then field, then operator. ```json theme={null} { "dataRequest": { "filters": { "my_postgres": { "transactions": { "status": { "in": ["completed", "pending"] } } } } } } ``` Ten operators exist: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `in`, `nin`, and `like`. Every operator takes a JSON array. Several operators on the same field combine with `AND`. ## Results *** An extraction produces exactly one result shape. In **direct mode** the Engine returns the rows inline as indented JSON and stamps a SHA-256 digest over those bytes. The payload leaves the Engine unencrypted, and the host decides what to do next. In **store mode** the Engine streams the rows to a sink that the host provides, one JSON object per line. It returns a reference instead of the bytes, with a SHA-256 digest over exactly the bytes written. The Engine holds no complete result in memory on this path. Each mode hashes what it emits: direct mode the indented document, store mode the streamed lines. The Engine canonicalizes its planned field and step order before serialization. A digest identifies the exact bytes emitted by that execution; do not treat it as a cross-run equivalence guarantee unless the datasource query order and all host-side processing are controlled. The two modes write different shapes, so compare a digest only against another digest from the same mode. The generic Engine runner stops on its first failing step and does not return a successful direct result. Hosts define their own behavior for compatibility or multi-stage orchestration paths. The standalone Worker drives direct mode. It then signs the plaintext with HMAC-SHA256, encrypts it with AES-GCM, and writes it to S3-compatible object storage. See [Architecture](/en/fetcher/fetcher-architecture). ## Engine and hosts *** The Engine is the part of Fetcher that owns connection lifecycle, schema discovery, query planning, extraction, limits, and tenant safety. It ships as its own Go module with no third-party dependencies, and it talks to the outside world only through ports that a host provides. A **host** supplies those ports and owns everything the Engine refuses to know: HTTP, queues, object storage, authentication, and the job lifecycle. The Manager and the Worker are two such hosts. Your own application can be a third. Read [Architecture](/en/fetcher/fetcher-architecture) for the two services, the ports, and what each side owns. ## Tenants *** Tenant ID is the only isolation boundary in the Engine. There is no organization concept and no product concept below it. Every operation validates the tenant before it touches a connection, a cache entry, or a datasource. Single-tenant mode is the default. Multi-tenant mode gives each tenant its own metadata database, which Fetcher resolves from JWT claims. Without a tenant database in context, the call fails. It never falls back to the shared database. See [Multi-tenancy](/en/multi-tenancy) for the platform-wide model. ## Next steps *** The Manager, the Worker, and the Engine they both run over. Register, test, update, and delete a datasource connection. How Fetcher reads a schema, caches it, and validates a job against it. Run Fetcher locally and execute your first extraction job.