Fetcher core concepts

The Fetcher model in one place: connections, datasource types, schemas, extraction jobs, filters, results, hosts, and the tenant scope that separates them.

Fetcher has a small model. You register a connection to an external database. Fetcher discovers the schema behind it. You submit an extraction job that names the fields you want. Fetcher gives you back a result with an integrity digest over the exact bytes it produced.

This page defines each noun. Each section then points to the page that covers it in full.

The model at a glance

---

Concept	Definition
Connection	A stored, named reference to one external database.
Datasource type	The database engine behind a connection. Fetcher accepts five.
Schema	The tables and fields that Fetcher finds on a datasource.
Extraction job	An asynchronous request for named fields from named tables.
Filter	A per-field condition that limits the rows a job returns.
Result	The extracted rows, plus the digest that identifies them.
Engine	The extraction core. It decides what an extraction means.
Host	The application that runs the Engine. It owns queues, storage, and transport.
Tenant	The only isolation boundary. Every operation carries one tenant ID.

Connections

---

A connection holds the datasource type, host, port, database name, credentials, and optional TLS settings for one external database. A short configName identifies it inside the tenant, and every job and schema call addresses the datasource by that name.

Fetcher encrypts the password with AES-256-GCM before it reaches storage. The stored record also keeps the key version that protected it. A connection never travels back to a caller with its password.

Read Connections for the full lifecycle, the connection test, and the rule that blocks an update or a delete while jobs still run.

Datasource types

---

A connection declares one of five types. Send the value upper-case: request validation matches these five strings exactly and rejects any other casing with 400.

• POSTGRESQL
• MYSQL
• ORACLE
• SQL_SERVER
• MONGODB

The five behave differently under the hood. PostgreSQL and SQL Server qualify tables outside the default schema. Oracle works in owner namespaces. MongoDB has no declared schema at all, so Fetcher infers one from a document sample.

Schemas

---

A schema snapshot lists the tables of a datasource and the fields of each table. Fetcher builds it directly from the datasource, so you do not maintain a separate catalog.

Two things build on a snapshot. Schema validation checks a job mapping against it before extraction starts. Schema caching keeps a recent snapshot under the tenant and the config name, so repeated work skips the round trip to the database.

Read Schema discovery for live-versus-cached behavior, the per-database differences, and the validation report.

Extraction jobs

---

A job names the fields to extract, per table, per datasource:

{
  "dataRequest": {
    "mappedFields": {
      "my_postgres": {
        "accounts": ["id", "email", "created_at"]
      },
      "my_mongo": {
        "transactions": ["*"]
      }
    }
  }
}

One job can span several datasources, several tables per datasource, and several schemas. Use ["*"] to take every field of a table.

The Manager accepts a job and answers 202 Accepted. A repeated request inside a five-minute window answers 200 OK and returns the job that already exists. A job that failed does not suppress a retry.

Engine limits bound generic datasource work. The defaults allow 10 datasources per extraction, 20 tables per datasource, 50 fields per table, and a five-minute deadline. Embedded Engine callers can lower, but never raise, those limits with ExtractionRequest.Overrides. The standalone Manager job payload has no limit-override field, and the Worker accepts only a positive ENGINE_MAX_RESULT_BYTES override. The plugin_crm portion uses the Worker’s explicit compatibility path.

Filters

---

A filter narrows the rows of one table. The job payload nests filters four levels deep: datasource, then table, then field, then operator.

{
  "dataRequest": {
    "filters": {
      "my_postgres": {
        "transactions": {
          "status": { "in": ["completed", "pending"] }
        }
      }
    }
  }
}

Ten operators exist: eq, ne, gt, gte, lt, lte, between, in, nin, and like. Every operator takes a JSON array. Several operators on the same field combine with AND.

Results

---

An extraction produces exactly one result shape. In direct mode the Engine returns the rows inline as indented JSON and stamps a SHA-256 digest over those bytes. The payload leaves the Engine unencrypted, and the host decides what to do next.

In store mode the Engine streams the rows to a sink that the host provides, one JSON object per line. It returns a reference instead of the bytes, with a SHA-256 digest over exactly the bytes written. The Engine holds no complete result in memory on this path.

Each mode hashes what it emits: direct mode the indented document, store mode the streamed lines. The Engine canonicalizes its planned field and step order before serialization. A digest identifies the exact bytes emitted by that execution; do not treat it as a cross-run equivalence guarantee unless the datasource query order and all host-side processing are controlled. The two modes write different shapes, so compare a digest only against another digest from the same mode. The generic Engine runner stops on its first failing step and does not return a successful direct result. Hosts define their own behavior for compatibility or multi-stage orchestration paths.

The standalone Worker drives direct mode. It then signs the plaintext with HMAC-SHA256, encrypts it with AES-GCM, and writes it to S3-compatible object storage. See Architecture.

Engine and hosts

---

The Engine is the part of Fetcher that owns connection lifecycle, schema discovery, query planning, extraction, limits, and tenant safety. It ships as its own Go module with no third-party dependencies, and it talks to the outside world only through ports that a host provides.

A host supplies those ports and owns everything the Engine refuses to know: HTTP, queues, object storage, authentication, and the job lifecycle. The Manager and the Worker are two such hosts. Your own application can be a third.

Read Architecture for the two services, the ports, and what each side owns.

Tenants

---

Tenant ID is the only isolation boundary in the Engine. There is no organization concept and no product concept below it. Every operation validates the tenant before it touches a connection, a cache entry, or a datasource.

Single-tenant mode is the default. Multi-tenant mode gives each tenant its own metadata database, which Fetcher resolves from JWT claims. Without a tenant database in context, the call fails. It never falls back to the shared database. See Multi-tenancy for the platform-wide model.

Next steps

---

Architecture

The Manager, the Worker, and the Engine they both run over.

Connections

Register, test, update, and delete a datasource connection.

Schema discovery

How Fetcher reads a schema, caches it, and validates a job against it.

Getting started

Run Fetcher locally and execute your first extraction job.