Data & Databases

The database connectors share a common shape: a query action for reads, a command action for writes, and (where the engine supports it) bulk operations. Connection strings and pool settings live on the connection, never in the flow.

SQL databases

Connector	`invoke:`	Typical actions
PostgreSQL	`postgresql.*`	`query`, `command`, `query-cursor`
MS-SQL	`mssql.*`	`query`, `command`, `query-cursor`
MySQL	`mysql.*`	`query`, `command`, `query-cursor`
Oracle	`oracle.*`	`query`, `command`, `query-cursor`
Redshift	`redshift.*`	`query`, `query-cursor`, `unload`, `copy`, `create-table`, `typed-insert`

using:
  - environment/prod
  - zenvara/postgresql

output:
  rows: !obj-list

steps:
  - $orders:
      invoke: postgresql.query
      on: warehouse
      with:
        Query: "SELECT id, customer, total FROM orders WHERE order_date = '${date}'"

  - return:
      rows: "${orders.rows}"

Output: rows (the result set, a typed list) and metadata. Reference as ${orders.rows}. Parameterise queries through the flow’s typed input: rather than string-concatenating untrusted values.

NoSQL

Connector	`invoke:`	Notes
MongoDB	`mongodb.*`	Document find / insert / update / aggregate. Filter inputs are validated to prevent injection.

The runtime-native transformers

Three of the most common data operations are not connectors — they are built into the runtime and need no using: import. They are documented in full under Transformers:

join: — relational join of two sources (inner / left / right / full).
delta: — change detection across runs (compare, hash, per-row by: hash).
filter: — SQL-like filtering on payload arrays (where:, order-by:, limit:, fields:).

A common pattern is to query a database, delta: against a persisted snapshot to find only what changed, and write back just the delta — incremental sync without an external CDC system.

Spill-to-disk for large result sets

The engine streams and spills large datasets to disk rather than holding everything in memory, so a query that returns millions of rows does not exhaust the process. This is automatic; you author the flow the same way regardless of size.