System Design

Dendra is a modern, scalable, cloud-based system for managing and querying environmental data. Dendra Release 3 operates on two core server executables: API Server and Job Scheduler. Workers are additional executables, and are designed as hosts for various job types (ingestion, orchestration, etc.). A variety of third-party services are used to support the system, including databases, object stores, and messaging systems.

Our production system is hosted on NSF’s ACCESS Jetstream2 where we manage our own Kuberernetes clusters at the The Indiana University Jetstream2 supercomputing center.

While Dendra is a SaaS platform (meaning we host the system for you and you don’t need to worry about the infrastructure), the core system and many third-party services can be scaled down to run on a home server for development and testing.

System Components

Core system components and their relationships in Dendra Release 3.

architecture-beta
    group platform[Platform]
    service apiserver(mdi:api)[API Server] in platform
    service metadata(logos:aws-documentdb)[Metadata] in platform
    service platformkv(logos:aws-dynamodb)[KV Cache] in platform
    service search(logos:aws-open-search)[Search] in platform

    apiserver:T -- B:metadata
    apiserver:L -- R:platformkv
    apiserver:L --> R:search

    group jobprocessing[Job Processing]
    service scheduler(mdi:timer-cog-outline)[Job Scheduler] in jobprocessing
    service workerbus(logos:aws-eventbridge)[Worker Messaging] in jobprocessing
    service workers(mdi:cogs)[Workers] in jobprocessing
    service jobkv(logos:aws-dynamodb)[KV Cache] in jobprocessing

    workerbus:T -- B:scheduler
    workers:L -- R:workerbus
    workers:R -- L:jobkv
    workers:T --> B:apiserver

    group storage[Storage]
    service messages(logos:aws-mq)["Events/Data Streaming"] in storage
    service objectstores(logos:aws-s3)[Object Stores] in storage
    service timeseriesdbs(logos:aws-timestream)[Timeseries DBs] in storage

    apiserver:B --> T:scheduler
    apiserver{group}:R -- L:objectstores{group}
    apiserver{group}:R -- L:messages{group}
    apiserver{group}:R -- L:timeseriesdbs{group}

    workers{group}:T -- B:objectstores{group}
    workers{group}:T -- B:messages{group}
    workers{group}:T -- B:timeseriesdbs{group}

Component	Description
API Server	Main API server that handles requests from applications and API clients.
Events/Data Streaming	Event bus for resource change events and persistence engine for data streaming.
Job Scheduler	Scheduler that coordinates job processing across workers.
KV Cache	Key/value store and cache.
Metadata	Database for storing metadata (stations, sites, datastreams, etc.). Dendra has the option to store metadata in different databases to separate concerns and to support scaling.
Object Stores	S3-compatible object stores for files, images and working datasets. Dendra can utilize multiple object stores for different security, scalability, and performance purposes.
Search	Self-hosted Search Engine for full-text, semantic, and vector search to support cross-org metadata and data retrieval.
Timeseries DBs	Databases for storing timestamped data — accessed through tables. Dendra can utilize multiple timeseries databases and technologies for different orgs, timeframes, and purposes.
Worker Messaging	Event bus and queue group that handles communication between workers and the scheduler.
Workers	Processors (executables) that handle various job types (ingestion, orchestration, etc.).

Apps & API Clients

How applications and API clients interact with the Platform API and external services.

architecture-beta
    group apps[Apps]
    service mainapp(mdi:application-brackets)[Main App] in apps

    group platform[Platform]
    service apiserver(mdi:api)[API Server] in platform

    service auth(logos:aws-iam)[Auth IAM]
    service maps(logos:google-maps)[Maps]

    mainapp{group}:L -- R:auth
    mainapp{group}:B -- T:maps
    mainapp{group}:T --> B:apiserver

    service analytics(logos:google-analytics)[Analytics]

    analytics:L -- R:apiserver
    analytics:B -- R:mainapp{group}

    service apiclient(mdi:account-hard-hat)[API Client]

    apiclient:R --> L:apiserver

Component	Description
Analytics	Third-party analytics for monitoring app and API usage. We use Plausible and PostHog.
API Client	Any user, developer, or tool interacting with the Platform API.
API Server	Main API server that handles requests from applications and API clients.
Auth IAM	External IAM (Identity and Access Management) service for user authentication and base permissions.
Main App	The main web app that allows users to query and manage the Dendra system.
Maps	Third-party maps service for rendering maps (Google Maps, Mapbox, etc.).

Workers & External Services

How the workers interact with external services.

architecture-beta
    group jobprocessing[Job Processing]

    service workers(mdi:cogs)[Workers] in jobprocessing

    service mail(logos:aws-ses)[Mail Delivery]
    service slack(logos:slack-icon)[Slack]
    service dataproviders(cloud)[Data Providers]

    workers:R -- L:dataproviders
    workers:R -- L:mail
    workers:R -- L:slack

Component	Description
Data Providers	Third-party time-series and metadata providers (Campbell Scientific, LI-COR, GOES, etc.).
Mail Delivery	Third-party mail delivery service for sending emails.
Slack	Third-party real-time messaging service for sending messages.
Workers	Processors that handle various job types (ingestion, orchestration, etc.).

Other Services

Other services that are not part of the Dendra system but are still part of running and maintaining Dendra as a whole.

architecture-beta
    service systemmonitoring(mdi:monitor-eye)[System Monitoring]
    service statuspage(mdi:lan-check)[Status Page]
    service mainsite(mdi:home)[Main Site]
    service docsite(mdi:help-circle-outline)[Docs Site]

Component	Description
Docs Site	Documentation website for Dendra (this site).
Main Site	Main public website for the Dendra Science.
Status Page	Status page for the production Dendra system.
System Monitoring	Third-party system monitoring services for tracking the health and performance of the Dendra system.