Extraction jobs

How a Fetcher extraction job describes datasources, tables, schemas, and fields — and what happens to the job from creation to a stored result.

An extraction job is an asynchronous request to read data out of one or more registered databases. The Manager accepts the job and queues it. A Worker runs the extraction and stores the result. This page explains the request you send and the path the job takes.

The request

---

A job carries two parts: dataRequest and metadata.

{
  "dataRequest": {
    "mappedFields": {
      "my_postgres": {
        "public.accounts": ["id", "email", "created_at"],
        "accounting.invoices": ["*"]
      },
      "my_mongo": {
        "transactions": ["*"]
      }
    }
  },
  "metadata": {
    "source": "payments",
    "correlationId": "settlement-2026-07-13"
  }
}

mappedFields is the heart of the request. It maps each datasource config name to the tables you want, and each table to the fields you want.

metadata.source is required. It names the product that owns the job. The Manager rejects a request without it. The Manager also rejects the request when a referenced connection belongs to a different product, or belongs to no product at all. Internal datasources carry no product, so this check skips them.

Field projection

---

Name the fields you want, or ask for all of them:

• A list of field names extracts exactly those fields.
• The single entry ["*"] extracts every column of the table. The wildcard must be the only entry in the list.

The generic Engine validates selected fields against its cache-first resolved schema before it queries. An invalid selection fails the job with the safe extraction request failed schema validation error. Use the schema-validation workflow for per-field diagnostics.

Multiple datasources, tables, and schemas

---

One job can read from several datasources at the same time. Each datasource can contribute several tables. Each table can sit in a different schema.

Write a table name in one of two forms:

• Unqualified — accounts. Fetcher reads it from the engine default schema: public on PostgreSQL, dbo on SQL Server.
• Schema-qualified — accounting.invoices. The prefix before the dot is the schema. On Oracle it is the owner.

Fetcher collects the distinct schema prefixes in mappedFields and discovers only those namespaces. If any table name is unqualified, discovery adds the engine default schema as well: public on PostgreSQL, dbo on SQL Server.

Two engines do not take a schema list. MySQL treats the connected database as the namespace. MongoDB has collections instead of schemas, so a collection name is always unqualified.

Limits

---

The extraction engine bounds every job. These are the default values:

Bound	Default
Datasources per job	10
Tables per datasource	20
Fields per table	50
Datasources extracted in parallel	4
Extraction timeout	5 minutes
Serialized result size	256 MiB

The Manager rejects a job with more than 10 datasources before it reaches the queue. A result over the size limit fails the job. Fetcher never returns or stores a truncated result. For embedded Engine callers, ExtractionRequest.Overrides can lower, but never raise, an Engine limit. The standalone Manager job payload has no limit-override field and maps overrides as nil; the Worker only accepts a positive ENGINE_MAX_RESULT_BYTES override. These bounds cover generic datasource work; plugin_crm uses the Worker’s explicit compatibility path.

Duplicate jobs

---

The Manager computes a SHA-256 hash over the whole request — dataRequest and metadata together. It then looks for a job with the same hash created in the last 5 minutes.

• A match returns the existing job with HTTP 200. No second extraction runs.
• No match creates a new job and returns HTTP 202.
• A match that already failed does not block a retry. The Manager creates a new job.

Metadata is part of the hash. Two requests that read the same data under different correlation IDs are different jobs.

Job lifecycle

---

Status	Meaning
pending	The Manager accepted the job and published it to the queue.
processing	A Worker claimed the job and is reading the datasources.
completed	The result is stored, and job.completed is published.
failed	The extraction stopped, and job.failed is published.

Before it creates the job, the Manager resolves every datasource name to a connection and opens a real connection to each one. A datasource that fails this test rejects the whole request. Internal datasources configured through environment variables skip the test.

A Worker performs a best-effort check that a job is pending, then moves it to processing and starts the extraction. This is not an atomic pending-to-processing compare-and-set, so concurrent deliveries can both observe pending. Generic datasource work runs in parallel up to the concurrency bound and is fail-fast; plugin_crm uses the Worker’s explicit compatibility path.

On success the Worker writes the result to object storage and records two values on the job: the result path and the HMAC signature of the result. It then publishes the terminal event.

Next steps

---

Filters

The ten filter operators and the value shape each one takes.

Datasources

What behaves differently on each of the five database engines.