Skip to content

Architecture

System Overview

Floecat tracks a resource graph anchored at Accounts and spanning Catalogs, Namespaces, Tables, Views, Snapshots, statistics, and query-lifecycle artifacts.

Account
 ├── CatalogIntegration (external catalog identity)
 │    └── CatalogOverlay (upstream namespace selection)
 └── Catalog (locally managed catalog)
      └── Namespace (hierarchical path)
           ├── Table (Iceberg/Delta metadata, upstream reference)
           │    └── Snapshot (immutable upstream checkpoints)
           └── View (stored definition)

Two storage primitives underpin every service:

  • BlobStore – immutable protobuf payloads such as catalog.pb, table.pb, snapshots/{snapshot_id}/snapshot/{sha}.pb, and stats blobs. Blobs are content-addressed via SHA256 ETAGs.
  • PointerStore – versioned key→blob indirection to support atomic compare-and-set (CAS), hierarchical listing, and name→ID lookups. Keys use deterministic prefixes such as /accounts/{account_id}/catalogs/by-name/{name} and /accounts/{account_id}/tables/{table_id}/snapshots/by-id/{snapshot_id}.

Table roots, coherent snapshot selection, and snapshot visibility

Per table, the two primitives compose into an immutable TableRoot: a content-addressed root blob referencing a chain of immutable snapshot-manifest pages, with a single CAS'd pointer per table naming the current root. Every writer publishes through TableRootCommitter — the single owner of every root mutation — which reads the current root, applies the caller's mutator, writes the new root blob, and CASes the pointer; a lost CAS re-runs the mutator against the winner's root so concurrent commits merge instead of clobbering.

Queries resolve one coherent snapshot selection: TablePin copies the refs it needs (definition, snapshot, constraints, stats generation) out of one root at resolution time, so every later schema, scan, stats, and constraints read in that query is coherent by construction. This selection is process-local query state, not a GC root or lifetime lease. Retention controls which snapshots new queries may select; explicit deletion or later collection can make an already-selected snapshot unavailable. Such a query fails with the snapshot-unavailable error, and Core replans only when the original query semantics allow a different selection.

Snapshot visibility is gated at read time (StatsVisibilityGate): registration and resync advance the root's current_snapshot_id freely, but when the stats store tracks generations a manifest entry without its stats_generation_ref is not yet query-visible — a read or pin resolves to the newest finalized snapshot at or before it instead, so logical metadata can move current without exposing an unfinalized scan.

The gRPC service (Quarkus) enforces tenancy, authorization, idempotency, and transactional CRUD for catalog integrations and overlays. Catalog integrations include typed, non-secret authentication configuration plus write-only credential create/rotation primitives; secret payloads are stored separately and never returned. A typed credential-store resolver derives the internal secret key from integration identity and credential generation for follow-on clients. The resources remain operationally inert: this API does not perform connectivity validation, adapter selection, or reconciliation. Legacy connectors remain unchanged and are not part of either new contract.

An overlay references one existing Catalog. Multiple overlays may target the same Catalog, which remains directly writable and retains its independent name and lifecycle. Overlay creation is conditioned on the current Integration and Catalog pointer versions and atomically publishes both dependency indexes while advancing fixed dependency markers. Integration and Catalog deletion check the strongly consistent indexes and matching markers, closing concurrent create/delete races. Account deletion removes overlays before integrations and catalogs.

Components

The following modules compose the system (see linked docs for deep dives):

Component Responsibility
proto/ All protobuf/gRPC contracts (catalog, query lifecycle, execution scans, connectors, statistics, types).
core/catalog-access-spi/ Connector-independent upstream catalog access contracts.
catalog-access/iceberg-rest/ Iceberg REST discovery and metadata provider for the catalog-access SPI.
service/ Quarkus runtime, resource repositories, query lifecycle service, GC, security, metrics.
client-cli/ Interactive shell for humans; exercises every public RPC.
core/connectors/spi/ Connector interfaces, stats engines, NDV helpers, auth shims.
connectors/catalogs/iceberg/ Iceberg REST + AWS Glue connector implementation.
connectors/catalogs/delta/ Unity Catalog/Delta Lake connector using Delta Kernel + Databricks APIs.
core/connectors/common/ Shared connector utilities (Parquet stats, NDV sketches, planners).
reconciler/ Connector scheduler/worker, reconciliation orchestration, job store.
core/storage-spi/ Blob/pointer persistence contracts shared by service and GC.
storage/memory/ In-memory dev/test stores (CAS semantics maintained).
storage/aws/ Production DynamoDB pointer store + S3 blob store.
types/ Logical type system utilities, coercions, min/max encoding.
extensions/builtin/ Plugin architecture for engine-specific builtin catalogs (functions, operators, types, etc.).
log/ Runtime log directory layout (service log + audit channel).

Data & Control Flow

  1. Connectors (Delta/Iceberg) enumerate upstream namespaces, tables, snapshots, and file-level stats via the shared SPI.
  2. The Reconciler schedules connector runs, materializes local Tables/Snapshots/Stats through repository APIs, and records incremental NDV, histograms, and scan manifests. Reconcile execution is mode-split:
  3. METADATA_ONLY for table/snapshot state
  4. CAPTURE_ONLY for stats enrichment only (via stats control-plane / engine registry), with failure on fully non-captured batches and degraded success on partial capture
  5. METADATA_AND_CAPTURE for metadata ingest plus capture using table-scoped capture requests
  6. The Service exposes CRUD RPCs for catalogs/namespaces/tables/views, plus query-lifecycle and statistics APIs. Requests traverse interceptors that inject PrincipalContext, correlation IDs, and optional query leases before hitting service implementations.
  7. Repositories translate RPCs into pointer/blob mutations, enforce optimistic concurrency, and update idempotency records.
  8. Query lifecycle RPCs hand planners query descriptors (snapshot selections, obligations) plus any connector-provided scan metadata needed before execution.

Consistency Model (Current)

  • Core table state (single-table commit): synchronous request path updates table/snapshot state before returning success.
  • Post-core side effects (connector/snapshot sync actions): best-effort after commit apply and not part of atomic commit state. This includes gateway-triggered service-side CAPTURE_ONLY stats bootstrap for managed REST connectors tagged with floecat.connector.mode=capture-only.
  • Multi-table transaction endpoint: request-level atomic apply through backend transactions, with idempotent replay and optimistic preconditions.