A disposable SDMx teaching lab: metadata registry, authoring workbench, the .Stat Suite, an open-data portal and notebooks — behind one TLS door, seeded end to end, torn down after class.
Nothing is precious. The box is rebuilt from this repository and a set of idempotent seed scripts, so the disposal-and-rebuild cycle is the normal way to operate it.
OpenTofu creates a Hetzner box with a reserved IP that outlives it, so DNS is a one-time setup. cloud-init installs Docker & git.
A single Compose project: Traefik fronts FMR, the FMR Workbench, CKAN, Jupyter and the landing page, with automatic Let's Encrypt certs.
The OECD platform runs side by side, vendored at a pinned tag and left pristine; a thin overlay wraps it into our domain.
Idempotent scripts create the lab users, the SDMx structural foundation, the sample data, and the CKAN catalogue — and trigger indexation.
Traefik is the only thing listening on 80/443. It terminates TLS for every subdomain and sits on both Docker networks, bridging the umbrella services and the side-by-side .Stat Suite. Traffic on the connectors animates toward the services.
.Stat is a dozen cooperating services plus the FMR Workbench that authors into it. Edges are coloured by what flows: structures, data, identity & permissions, and config, search, storage. The DLM is the hub — most producer actions are it handing off to the right service.
A verified nuance: get-data reads structures through the NSI API; transfer reaches structures and data directly in MSSQL (it holds the struct/data DB connections), so there is no transfer→NSI call. Not drawn (peripheral support services): share (saved charts → Mongo), data-viewer, and url-shortener.
The same components, put to work. Edges keep their colours: structures, data, identity, config & search. Kong's routing (by content negotiation): SDMx-REST 2.0 data requests → get-data; legacy ?format=csv/plain data and all structures → the NSI.
Data Explorer · browser
script / client · api.
The Explorer's JSON / AllDimensions queries take the other Kong route, to get-data.
FMR Workbench · opened from the DLM
DLM → transfer → sfs
sfs is the only writer to the .Stat Solr; everything else reads it through sfs's search API.
api. is a Kong gateway that routes by Accept header and format, so two requests for the same dataflow can diverge here (verified by the response's Server header). The split is content negotiation, not CSV-vs-JSON: SDMx-REST 2.0 requests (Accept: application/vnd.sdmx.data+json or +csv, ?format=jsondata) go to get-data; legacy ?format=csv, plain, and all structure requests go to the NSI. get-data reads the data from MSSQL and fetches the DSD from the NSI only on a cache miss (measured: +2 NSI calls cold, 0 when warm).
?format=csv / plain / structures → NSI
get-data's DSD fetch (only on cache miss)
The seeds walk a single dataflow (BAFFELAN:DF_EXAMPLE) through the producer's loop. Two facts it makes concrete: each dataspace has its own structure registry, and structures must exist before data.
Agencies, canonical codelists, a DSD & dataflow — in the design space.
18 observations imported via the transfer service.
Structures + data transferred to the dissemination space.
sfs indexes the dataflow into Solr for discovery.
Browsable, filterable and charted in the Data Explorer.
make deploy, end to endA single command runs the whole sequence across laptop and box. Because the IP is reserved, there is no DNS step on a rebuild — indexation and all.
reserved IP, cloud-init
repo → box
umbrella + TLS
.Stat + users
structures→data→
promote→index
brand + catalogue
Every public subdomain and the container behind it. Traefik learns the umbrella routes from Docker labels and the .Stat routes from a file-provider config.
| Subdomain | Service | Role |
|---|---|---|
domain (bare) | landing | Front door: directory + live status |
fmr. | FMR | Fusion Metadata Registry |
fwb. | FMR Workbench | Authoring structures into .Stat |
ckan. | CKAN | Open-data portal |
jupyter. | Notebooks | Python + pysdmx / sdmx1 |
dotstat. | Data Explorer | Dissemination front-end |
dlm. | Data Lifecycle Manager | Load, validate, promote |
keycloak. | Keycloak | Single sign-on (realm demo) |
api. | Kong | Data + structure API gateway |
sfs. · transfer. · auth. | sfs · transfer · auth | Search · import · permissions |
Grouped by layer. Every script is idempotent and documented at the top of the file.
Makefilethe whole lifecycle: provision / sync / deploy / destroy + box targetscompose.yamlumbrella Compose; includes each stack, wires the shared networkARCHITECTURE.md · deployment-{design,notes}.mdthis guide, the design, the ops lessons.env.exampleevery tunable, documentedtofu/main.tfreserved IPv4/IPv6, firewall, servertofu/cloud-init.yamlDocker + git + swap at first boottofu/{variables,outputs,versions}.tfinputs, the IPs, the provider pinfmr/ · notebooks/registry and Jupyter, behind Traefikfwb/Workbench + the template wiring it to the .Stat dataspacesckan/portal image + seed-ckan.sh (branding + catalogue)landing/the front-door app: server-side health checksoverlay/render-{config,traefik}.shrouters + re-point config to our domainoverlay/dotstat.override.ymlenv fixes: SSO, DLM URLs, the get-data fixoverlay/patch-{keycloak,vendor-network}.shsilent-SSO CSP, shared external networkoverlay/seed-users · seed-permissionsthe lab roster + authorization rulesoverlay/seed-structures/ · seed-data/the SDMx foundation, data, promote, index