Data Layer Implementation

The Release 3 API Server separates Connect service handlers from storage code through a datalayer pattern. This is not an ORM: there is no generic object–relational mapping layer. Each datalayer interface defines the operations a service domain needs, and store-specific packages implement those operations with database-native queries and logic.

Layers, not ORM

Concern	Where it lives
RPC contract, permissions, orchestration	services/*/ Connect handlers
Domain persistence operations	datalayer/ interfaces (e.g. MetadataV3Alpha1Layer)
Store-optimized implementation	datalayer/mongostore/, datalayer/influxstore/, …
Blob bytes (S3-compatible)	objlayer/ (e.g. miniostore/) — separate from file metadata

Service code depends only on the interface. MongoDB aggregation pipelines, Influx query construction, and index management live in the store package, where they can be tuned per backend without leaking storage details into handlers.

Connector Binding

At startup, each service module binds its datalayer to a physical store through environment configuration.

1. Connector registry — conns/ registers named connections from env lists such as MONGO_CONNECTORS=metadata,… and INFLUX_CONNECTORS=cdfw,….
2. Service binding — each module reads SVC_<MODULE>_CONNECTOR as store:key (for example SVC_METADATAV3ALPHA1_CONNECTOR=mongo:metadata).
3. Implementation selection — the module’s Setup function resolves the connector and constructs the matching *Layer implementation.

The timeseries module is unique and can bind multiple locations (SVC_TIMESERIESV3ALPHA1_LOCATIONS) each with its own SVC_TIMESERIESV3ALPHA1_<LOCATION>_CONNECTOR, while still exposing one query surface to callers. Which database backs a domain is configured at deploy-time. The locations defined here are referenced under shardspaces defined in api.toml.

# Illustrative env (see sample.env in api-services)
MONGO_CONNECTORS=metadata
MONGO_METADATA_URI=mongodb://…

SVC_METADATAV3ALPHA1_CONNECTOR=mongo:metadata
SVC_FILEV3ALPHA1_CONNECTOR=mongo:metadata
SVC_DATAINTEGRATIONV3ALPHA1_CONNECTOR=mongo:metadata

Service Module Wiring

---
config:
  flowchart:
    nodeSpacing: 40
    rankSpacing: 60
    padding: 16
---
flowchart TB
    subgraph Env["Environment"]
        SVC["SVC_*_CONNECTOR<br/>store:key"]
        REG["MONGO_CONNECTORS, INFLUX_CONNECTORS, …"]
    end

    subgraph Registry["Connector registry"]
        MC[(mongo:metadata)]
        IC[(influx:cdfw)]
    end

    subgraph Interfaces["Datalayer interfaces"]
        MD["MetadataV3Alpha1Layer"]
        FL["FileV3Alpha1Layer"]
        TS["TimeseriesV3Alpha1Layer"]
        DI["DataintegrationV3Alpha1Layer"]
    end

    subgraph Stores["Store implementations"]
        Mongo["mongostore.LayerImpl"]
        Influx["influxstore.LayerImpl"]
    end

    subgraph Services["Connect service modules"]
        MetaSvc["metadata/v3alpha1"]
        FileSvc["file/v3alpha1"]
        TSSvc["timeseries/v3alpha1"]
    end

    REG --> Registry
    SVC --> MetaSvc
    SVC --> FileSvc
    SVC --> TSSvc

    MetaSvc --> MD
    FileSvc --> FL
    TSSvc --> TS

    MD --> Mongo
    FL --> Mongo
    DI --> Mongo
    TS --> Influx

    Mongo --> MC
    Influx --> IC

Each service module’s Setup function creates handlers, binds one primary datalayer, and returns an Exports struct when other modules need the same bound layer.

Cross-cutting Concerns via Exports

Some RPC services span more than one domain. The primary module owns the datalayer binding; it exports the bound interface for others to reuse.

Example: FileService owns FileV3Alpha1Layer (file metadata in Mongo) and objlayer (object bytes in MinIO). When a user confirms a station photo upload, the file service:

1. Writes object bytes through the object layer.
2. Calls metadataV3Alpha1Exports.DataLayer.UpdateStationPhotos to attach the file reference on the station record.

Integration and timeseries modules follow the same pattern — they import metadataV3Alpha1Exports.DataLayer to resolve organizations, stations, or table info while performing their own persistence through a separate bound layer.

---
config:
  flowchart:
    nodeSpacing: 40
    rankSpacing: 60
    padding: 16
---
flowchart LR
    Client["API client"]
    FileSvc["FileService"]
    FileDL["FileV3Alpha1Layer"]
    Obj["objlayer · MinIO"]
    MetaExp["metadata Exports<br/>MetadataV3Alpha1Layer"]
    Mongo[(MongoDB)]

    Client --> FileSvc
    FileSvc --> FileDL
    FileSvc --> Obj
    FileSvc -->|"UpdateStationPhotos"| MetaExp
    FileDL --> Mongo
    MetaExp --> Mongo
    Obj --> MinIO[(MinIO)]

This keeps a single bound metadata layer across the process: file uploads, integration config, and metrics all see consistent station and organization data without duplicating connector wiring.

Init and Schema Setup

Each layer also implements InitActions — operational hooks for indexes, default seed data, and store setup. These run outside normal request handling (via inittools and SETUP_* / CREATEINDEXES_* env in api-services), keeping schema maintenance separate from Connect service code.

Extending the Datalayer

To add a new store or backend:

1. Implement the relevant *Layer interface under datalayer/<store>/.
2. Register a connector type in conns/ if needed.
3. Add a store branch in the service module’s Setup using SVC_*_CONNECTOR.
4. Export the layer from Exports if other modules must share it.