Storage Backends

Storage holds everything the platform runs: flows, connections, environments, secrets, run logs, persisted state. Pick a backend at install; switching later is a backup-restore exercise, not a config flip.

The three backends

Backend	When	Configured by
File	Single-host, on-prem, evaluations	Storage.StorageType: File + Storage.Directory
S3	Cloud-resident installs, AWS-native	Storage.StorageType: S3 + bucket / IAM role / access key
Minio	Object-storage semantics on-prem	Storage.StorageType: Minio + endpoint + credentials

S3 and Minio share the same connector-kind contract and are mostly interchangeable from the platform’s perspective.

On-disk layout (File backend)

{Storage.Directory}/
├── flows/{name}.flow.yaml               # pipeline definitions
├── scripts/{name}.script.yaml           # custom-connector manifests
├── scripts/{name}/                      # …and the bundle beside it (main.py, …)
├── auth/                                # identity & credentials
│   ├── users.yaml                       # local-provider users
│   ├── apikeys.yaml                     # API keys
│   ├── entity-policy.yaml               # unified access-control policy (secrets, flows, connections, environments)
│   └── secrets.enc                      # encrypted; never git-tracked
├── policies/                            # access-control rules
├── teams/{name}.team.yaml               # organisational entities
├── config/                              # operational deployment config
│   ├── connections/{name}.connection.yaml
│   ├── environments/{name}.environment.yaml
│   ├── cache-policies/{name}.cache-policy.yaml
│   ├── stores/{name}.store.yaml
│   ├── shapes/{name}.shape.yaml         # named typed schemas referenced by flow input:/output:
│   ├── task-boards/{id}.task-board.yaml # inbound task-board definitions
│   └── mappings/{name}.mapping.yaml     # reusable lookup tables
├── knowledge/                           # standalone & entity-pinned knowledge
│   ├── folios/{name}.folio.yaml         # in-product knowledge folios
│   ├── folios/{name}/document.md        # …and the folio's body beside it
│   ├── skills/{name}.skill.yaml         # reusable AI skill bundles
│   ├── skills/{name}/document.md        # …and the skill's body beside it
│   ├── templates/{name}.template.yaml   # report/EDI/HL7/email templates
│   ├── templates/{name}/template.liquid # …the Liquid body beside it
│   ├── templates/{name}/sample.json     # …and a sample payload beside it
│   ├── annotations/{type}/{id}.yaml     # entity annotations (tags, descriptions)
│   ├── folio-documents/{name}.md        # flat content bucket (body: reference indirection)
│   ├── template-bodies/{name}.liquid    # flat content bucket (body: reference indirection)
│   ├── template-samples/{name}.json     # flat content bucket (body: reference indirection)
│   └── context/global.md                # installation-wide notes
├── activity/
│   ├── runs/                            # per-run logs and outputs
│   ├── persistence/                     # persist: state across runs
│   ├── indexes/                         # run indexes
│   ├── shares/                          # share links
│   ├── tasks/                           # background task records
│   └── introspection/                   # captured tool-call events
└── archives/                            # archived activity (zip, tar.gz, parquet)
    ├── runs/                            # archived run logs and outputs
    ├── indexes/                         # archived run indexes (parquet)
    └── introspection/                   # archived tool-call events

Three categories worth keeping straight

• Stateful, must back up: flows/, scripts/, auth/, policies/, teams/, config/, knowledge/, activity/, the git-versioning path, and auth/secrets.enc.
• Stateful, regeneratable: the backup archives themselves (copy off-host on a schedule).
• Ephemeral: Cache.Directory — anywhere, separate disk OK, safe to wipe between runs.

Splitting buckets across backends

The platform has individually configurable storage buckets — one per entity type. StorageBuckets in appsettings.yaml is the per-bucket override, so you can keep flow definitions on local disk and run activity in S3 for retention scaling. Bucket settings (Path, GitTracked, Retention, Archive) inherit from ancestor groups in the nested tree.

Entity names are flat basenames

An entity’s name is its file’s basename, and a name never contains /. flows/billing/invoice-sync.flow.yaml names the flow invoice-sync — the billing/ folder is pure organisation and never enters the identity. This is consistent across every storage backend and is why the layering (provider → git → branch → overlay) composes cleanly: whatever folder you organise an entity under, the identity a layer resolves is always the same flat kebab-case basename. Group entities operationally with tags (a sibling annotation), not with folder paths.