Skip to content

feat(storage): allow local filesystem paths as tablePath #30

@gordonmurray

Description

@gordonmurray

DuckDB + the Iceberg extension can query a table sitting on the local filesystem. Cloudfloe currently rejects anything that doesn't match `s3://...` (see `_S3_PATH_RE` in `backend/main.py`).

Scope

  • Add a `LOCAL_PATH_RE` validator in `ConnectionConfig` matching absolute paths (/...), with a tight character class that rejects shell escape / semicolons / nulls.
  • Accept `storageType: "local"` in the model and skip S3 config + secret setup in `_apply_s3_config` / `_duckdb_connection` when that's the case.
  • Probe works unchanged — iceberg_metadata('/path/to/table') is the same call.
  • Frontend: add "Local filesystem" to the storage type dropdown; when selected, hide credential fields.

Security gate

Local-path support is behind an env flag: `CLOUDFLOE_ALLOW_LOCAL_PATHS=1`. Off by default. Cloud deployments never accidentally expose the backend container's /etc or /var/secrets.

When the flag is off and a local path arrives, return 400 with "Local paths are disabled. Set CLOUDFLOE_ALLOW_LOCAL_PATHS=1 to enable (self-hosting only)."

Why

Unlocks "I ran a local Flink/Spark job, let me peek at the output" without standing up MinIO. Onboarding wedge for data engineers debugging pipelines.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions