Datastream Search

The Data Explorer lets users find datastreams across organizations from a map. Behind that page, Release 3 keeps a compact, read-only index of sites, stations, and datastreams. The index answers three questions quickly: which datastreams match these filters, which sites are inside this map view, and which were deployed during this time window.

That index is built and queried with dendra-search, a Go library that also runs in the browser as WebAssembly. This page covers how the files are built, published, and queried.

Why a Separate Index

Release 2 explored data one organization at a time, so the metadata database could answer each request. The Data Explorer asks much more of a single page:

• Facet counts that update as you filter. A facet is a list of choices, such as measurements or sites, with a count next to each. Checking one choice changes the counts on the others.
• A map that answers for any view. Pan or zoom and the page needs the sites in view, grouped into clusters when there are too many to draw.
• Many organizations at once, including organizations the user is not a member of.

Running that against the metadata database on every pan would be slow and costly. Instead, Dendra builds the index ahead of time, stores it as files, and lets the browser run the queries itself.

The Pieces

Piece	Role
dendra-search	Go library. Builds an index, opens it, queries it, and merges results. It does not serve traffic or talk to storage.
dendra-job-workers	Reads metadata through the API, calls the library to build the index files, and stores them. See Workers.
Object storage	Holds the index files and small pointer files that say which file is current.
dendra-api-services	Returns download URLs: one public, and short-lived signed ones for a member’s organizations.
Browser	Downloads the files and runs the same Go library, compiled to WebAssembly, to answer queries.

The library has one rule: it returns bytes and answers queries, and the host decides who may download what. Storage, permissions, and compression stay outside it.

How We Index

Each organization is one shard, a single file holding that organization’s whole index. A shard is built in a fixed order: sites, then stations, then datastreams.

• Sites carry two locations: a shown location and an exact one. Shown may be intentionally blurred for sensitive sites.
• Stations bring their deployment history, the time ranges each station was at each site.
• Datastreams point at a station, or attach to a site directly, and carry their measurement, purpose, output field names, and access flags.

Deployments are stored as time intervals. An open interval means the deployment is still active. Closed intervals are kept, so a query with a date range can find a datastream at a site it has since left. An interval can also carry a group id.

At query time, the library also builds an R-tree for each organization. This is a spatial index that finds the sites inside a map rectangle without checking every site. Trees are built when a shard is opened and are not stored in the file.

Public and Private Shards

From the same data, the worker writes two outputs:

Output	Contents	Who can read it
Private shard	Every datastream, with both the shown and exact site locations.	Members of that organization
Public shard	Only datastreams a guest could already list, and only the sites they use. Exact locations are copies of the shown ones.	Anyone, including guests

A member-only datastream is simply absent from the public shard, rather than hidden by a flag. Anyone who can download a file can read every byte in it, so a flag inside the file would not protect anything. Because the public shard has no true coordinates, it is safe to hand to the browser as is.

How We Publish

Shards are rebuilt when metadata changes, never on the request path.

---
config:
  flowchart:
    nodeSpacing: 40
    rankSpacing: 60
    padding: 16
---
flowchart TB
    Change["Metadata change feed"]

    subgraph OrgJob["Per-organization job"]
        Pages["Read sites, stations, datastreams<br/>(through the API)"]
        Build["Build index"]
        Private["Private shard"]
        Chunk["Public shard (one per organization)"]
    end

    subgraph SysJob["System job"]
        Splice["Splice changed organizations<br/>into one container"]
        Gzip["Gzip the container"]
    end

    Pointer["current.json pointer"]
    Browser["Browser<br/>(WebAssembly)"]
    API["API<br/>(signed URLs for members)"]

    Change -->|"debounce per organization"| Pages
    Pages --> Build
    Build --> Private
    Build --> Chunk
    Chunk --> Splice
    Splice --> Gzip
    Gzip --> Pointer
    Pointer --> Browser
    Private --> API
    API --> Browser

1. Change. A change to a site, station, or datastream is seen on the change feed. After a short debounce, a build job is queued for that organization.
2. Build. The organization job pages through metadata and calls the library twice, once for the private shard and once for the public shard. Organizations that are private, disabled, or hidden publish no public shard.
3. Container. A single system job assembles every organization’s public shard into one container file. It reuses the previous container and replaces only the organizations whose shard changed, so a single edit does not re-read everyone’s metadata. The container is gzipped for download.
4. Pointer. Each set of files has a small current.json that names the current file. Readers follow the pointer, rather than listing files and guessing. A new file is always written before the pointer moves, and older files are kept for a grace period (24 hours by default), so a download that started before an update still finishes. A daily tidy job removes leftovers.

Deleting an organization retires its shard and rebuilds the container without it.

Note

The container is fetched whole, as one URL, instead of one request per organization. The trade is that a changed container downloads again in full. Browsers cache the file, since each version has its own URL.

How We Query and Combine

Each organization’s shard is queried on its own, and the results are then merged.

1. Query one shard. The caller names the filters (measurements, sites, groups, purpose, a time window, a map rectangle), the zoom level, and which location to use, shown or exact. Filters you leave empty are skipped.
2. Combine. The per-organization results are merged. Measurement counts are summed, and the sites are clustered once, across organizations. Two nearby sites from different organizations stay two sites, or two members of the same cluster. Clustering happens only in this step, never inside a per-shard query.

How Filters Combine

• Choices across facets intersect. A measurement and a site returns datastreams that are that measurement and at that site.
• Choices within a facet are a union. Checking two measurements returns datastreams that are either one.
• Each facet’s counts leave out that facet’s own filter. Selecting one measurement does not make the others disappear, because their counts are computed as if it were not selected.

Clusters or Sites

The result has one of two layers. When the map is zoomed out, or too many sites are in view, the result is clusters: grouped markers with a site count, plus measurement counts. When few enough sites are in view, the result is sites: one marker per site, per-site counts, and the matching datastreams.

The Data Explorer switches to sites when the zoom is 1 or higher and the number of matching sites in the current view is at or below the site limit (MapSiteLimit, which is 100 by default in dendra-search). That is why datastreams appear only after zooming in. See Datastreams appear when you zoom in.

Who Sees What

Browsers start with the public container, which is available to everyone, including guests. A signed-in user also asks the API for signed URLs to the private shards of the organizations they belong to. When those finish downloading, the page swaps once: a member’s organizations are now queried from their private shard, with exact locations and member-only datastreams, and everything else stays on the public shard.

The page does not redraw after each private shard arrives, because the map would jump once per organization. A person who can administer every organization still receives only the shards for organizations they belong to. For how access is decided elsewhere, see Permissions.

Groups and the Picker

A group is an optional label on a deployment. The shard stores the group’s id, not its name. Names come from metadata.

The Data Explorer shows organizations and groups side by side as two lists. Checking items in one list narrows the other, and dendra-search computes this reduction (which rows are relevant, and which are checked but no longer fit) so the page does not carry its own copy of the rules. Renaming or hiding a group does not rebuild any shard. Setting or clearing a group on a deployment does, because that is a station or datastream change.

In the Browser

The Data Explorer runs the same Go code as WebAssembly (under 1 MiB gzipped). It opens the downloaded shards, runs queries, and combines the results. Pans, zooms, and filter changes are answered locally, with no search request to the API.

The cost is paid once: a first visit downloads the module and the public container. In exchange, the API does not hold the catalog in memory or spend CPU on every pan.

Related

Data Explorer The page users open: map, filters, cart, and chart.

Job Scheduler and Workers The jobs that build and publish the search files.

System Design Where these components sit in the platform.