adds dynamic buckets docs #1292
Conversation
4db6cdc to
14365b0
Compare
github now rendering ```mermaid !
| **Flow (validate):** | ||
| AuthN → Admin RBAC → resolve secret/assume role → optional HEAD to storage → return `{ok, reason}` (no secrets). | ||
|
|
||
| ## 3) Data Model (Shared DB) |
There was a problem hiding this comment.
We should not have a shared db. We need to abstract microservices from each other with APIs as the only coupling.
|
|
||
| **Separation of concerns** | ||
| - `presigned-urls`: *read-only* DB, read secrets or assume-role, generate presigned URLs. No DB writes. | ||
| - `secret-admin`: CRUD + validate; never returns raw secrets; enforces admin RBAC. |
There was a problem hiding this comment.
The raw secrets are required for signing URLs
| This package contains a focused **architecture + rollout plan** to split signing from admin into two services: | ||
|
|
||
| - **`presigned-urls`** — high-throughput signer (download/upload single + multipart), read-only DB + secrets access. | ||
| - **`secret-admin`** — low-QPS (queries per second) admin CRUD + validation for the bucket registry, write access to DB; validate-only access to secrets/STS. |
There was a problem hiding this comment.
We will need this service to be able to be configured as included or excluded, b/c we will likely not utilize it. We'd like to still have the option of the current state: configure creds explicitly for the presigned-urls service
|
|
||
| ### A) `presigned-urls` | ||
| **Endpoints (Fence-compatible):** | ||
| - `GET /data/download/{guid}` → signed GET |
There was a problem hiding this comment.
should also include /drs/{{guid}}/access endpoints
| * **`FENCE_DYNAMIC_BUCKETS=true`** | ||
|
|
||
| * Split service deployment. Fence continues to handle auth flows and GA4GH endpoints. | ||
| * Presign and bucket-admin endpoints are served by new services. |
There was a problem hiding this comment.
GA4GH should be included in presigned. presigned needs a config flag on whether or not to rely on bucket-admin or just look for creds in configuration
motivation |
|
|
||
| * **`FENCE_DYNAMIC_BUCKETS=false` (default)** | ||
|
|
||
| * Fence behaves as today: all endpoints are served by the Fence service itself. |
There was a problem hiding this comment.
we should consider a clean split, at least for AWS. Have PRs to remove AWS signed URL logic from Fence and add new service(s) that handle AWS, GCP. The new presigned url service should handle GCP without all the crazy group management we have currently in Fence. We don't need that anymore. We should model the AWS approach in GCP in the new service. e.g. one admin service account signing URLs.
Overview: https://github.com/calypr/fence/blob/feature/dynamic-buckets/docs/fence-buckets/docs/index.md
New Features
S3_BUCKETSYAML lookups at runtime).FENCE_ENABLE_DYNAMIC_BUCKETSwith YAML fallback for backward compatibility.Breaking Changes
Bug Fixes
Improvements
Dependency updates
Optional runtime deps when corresponding backend is used:
boto3(for AWS STS/S3 clients in examples)kubernetes(for Kubernetes Secret API mode)No mandatory dependency changes for existing (YAML‑only) deployments.
Deployment changes
New DB objects:
bucket,bucket_prefix(optional),bucket_policy_binding, and audit table (optional).App config/env:
FENCE_ENABLE_DYNAMIC_BUCKETS=trueto activateFENCE_SECRETS_BACKEND=aws_sm|k8s(etc.)FENCE_BUCKET_CACHE_TTL_SECONDS,FENCE_SECRET_CACHE_TTL_SECONDSAuthZ/IAM:
sts:AssumeRolefor configured ARNs.Helm/K8s:
Migration:
fence-create sync-buckets-from-yaml) before enabling the flag.