Storage Backends
Storage holds everything the platform runs: flows, connections, environments, secrets, run logs, persisted state. Pick a backend at install; switching later is a backup-restore exercise, not a config flip.
The three backends
Section titled “The three backends”| Backend | When | Configured by |
|---|---|---|
| File | Single-host, on-prem, evaluations | Storage.StorageType: File + Storage.Directory |
| S3 | Cloud-resident installs, AWS-native | Storage.StorageType: S3 + bucket / IAM role / access key |
| Minio | Object-storage semantics on-prem | Storage.StorageType: Minio + endpoint + credentials |
S3 and Minio share the same connector-kind contract and are mostly interchangeable from the platform’s perspective.
On-disk layout (File backend)
Section titled “On-disk layout (File backend)”{Storage.Directory}/├── flows/{name}.flow.yaml # pipeline definitions├── scripts/{name}.script.yaml # custom-connector manifests├── scripts/{name}/ # …and the bundle beside it (main.py, …)├── auth/ # identity & credentials│ ├── users.yaml # local-provider users│ ├── apikeys.yaml # API keys│ ├── entity-policy.yaml # unified access-control policy (secrets, flows, connections, environments)│ └── secrets.enc # encrypted; never git-tracked├── policies/ # access-control rules├── teams/{name}.team.yaml # organisational entities├── config/ # operational deployment config│ ├── connections/{name}.connection.yaml│ ├── environments/{name}.environment.yaml│ ├── cache-policies/{name}.cache-policy.yaml│ ├── stores/{name}.store.yaml│ ├── shapes/{name}.shape.yaml # named typed schemas referenced by flow input:/output:│ ├── task-boards/{id}.task-board.yaml # inbound task-board definitions│ └── mappings/{name}.mapping.yaml # reusable lookup tables├── knowledge/ # standalone & entity-pinned knowledge│ ├── folios/{name}.folio.md # in-product knowledge folios (one OKF concept)│ ├── skills/{name}.skill.yaml # reusable AI skill bundles│ ├── skills/{name}/document.md # …and the skill's body beside it│ ├── templates/{name}.template.yaml # report/EDI/HL7/email templates│ ├── templates/{name}/template.liquid # …the Liquid body beside it│ ├── templates/{name}/sample.json # …and a sample payload beside it│ ├── annotations/{type}/{id}.yaml # entity annotations (tags, descriptions)│ ├── template-bodies/{name}.liquid # flat content bucket (body: reference indirection)│ ├── template-samples/{name}.json # flat content bucket (body: reference indirection)│ └── context/global.md # installation-wide notes├── activity/│ ├── runs/ # per-run logs and outputs│ ├── persistence/ # persist: state across runs│ ├── indexes/ # run indexes│ ├── shares/ # share links│ ├── tasks/ # background task records│ └── introspection/ # captured tool-call events└── archives/ # archived activity (zip, tar.gz, parquet) ├── runs/ # archived run logs and outputs ├── indexes/ # archived run indexes (parquet) └── introspection/ # archived tool-call eventsA folio is a single file. knowledge/folios/{name}.folio.md is one Open Knowledge Format (OKF) concept: the folio’s YAML frontmatter followed by its Markdown body, in the same document. There is no separate .folio.yaml manifest, no document.md sidecar, and no flat folio content bucket — a folio’s body is part of the concept rather than a body: reference into another path. Templates keep the split layout shown above, together with their flat template-bodies/ and template-samples/ content buckets.
Three categories worth keeping straight
Section titled “Three categories worth keeping straight”- Stateful, must back up:
flows/,scripts/,auth/,policies/,teams/,config/,knowledge/,activity/, the git-versioning path, andauth/secrets.enc. - Stateful, regeneratable: the backup archives themselves (copy off-host on a schedule).
- Ephemeral:
Cache.Directory— anywhere, separate disk OK, safe to wipe between runs.
Splitting buckets across backends
Section titled “Splitting buckets across backends”The platform has individually configurable storage buckets — one per entity type. StorageBuckets in appsettings.yaml is the per-bucket override, so you can keep flow definitions on local disk and run activity in S3 for retention scaling. Bucket settings (Path, GitTracked, Retention, Archive) inherit from ancestor groups in the nested tree.
Entity names are flat basenames
Section titled “Entity names are flat basenames”An entity’s name is its file’s basename, and a name never contains /. flows/billing/invoice-sync.flow.yaml names the flow invoice-sync — the billing/ folder is pure organisation and never enters the identity. This is consistent across every storage backend and is why the layering (provider → git → branch → overlay) composes cleanly: whatever folder you organise an entity under, the identity a layer resolves is always the same flat kebab-case basename. Group entities operationally with tags (a sibling annotation), not with folder paths.