Company logo

Anatomy of IoT cybersecurity

How we hardened high‑frequency power metering from flaky doors to the cloud ⚡🚪🔐

Posted by hrvoje on 11.08.2025

NixOS

2 years of iteration on the hardest project I’ve ever had: take messy physical realities (noisy cables, half‑broken doors affecting Wi‑Fi, people unplugging stuff 🙃) and still deliver trustworthy, frequent energy data without over‑engineering the security model. Here’s the story in layers, but told like a system we actually live with—not a museum piece.

TL;DR (for the impatient)

  • Edge concentrators (Raspberry Pi + NVMe + Modbus hat) run NixOS with a Rust daemon called pidgeon, local Timescale buffering, and basically nothing else exposed.
  • Secrets + identities are produced declaratively (Vault + custom “rumor” generator + SOPS/age) and baked at image-build time.
  • Data collection runs as fast as the Modbus line allows (tight loop; effective cadence ≈ 1–5s depending on meter count + baud).
  • Batches are sent every minute over HTTPS with an API key (PBKDF2 hashed server-side), while Nebula handles remote ops.
  • We accept what we can’t change (plaintext Modbus) and secure what we control (OS image, VPN, validation, buffering).
  • Main near-term focus: health telemetry + binary cache reliability → then anomaly detection + pruning.

1. Why This Even Exists

Traditional multi-tenant electricity billing is still depressingly “square meters times a fudge factor.” That’s not just inefficient; it’s ethically questionable when granular, device-level consumption could exist. We built a system that actually samples real energy usage at a much higher temporal resolution than the 15‑minute (or worse) norm. Not because the world is already demanding sub‑second settlement, but because it inevitably will as tariffs, dynamic pricing, and demand response spread. High frequency now = clean historical runway later (think future tariff windows, peak demand shaping, carbon-aware scheduling, etc.). Getting the architecture right early also means we preserve timing fidelity—a sneaky but crucial piece for any tariff logic: if your devices drift or smear timestamps, even perfect cryptography won’t save billing fairness.

2. Physical Reality vs Our Responsibility

The physical layer is delightfully messy. We have Schneider iEM3xxx meters (others prototyped), long Modbus runs, occasional noise, and in at least one site a partly busted door that literally changes observed RTT depending on how “closed” it feels like being that day 😂. Wires and meters are not under our full control and Modbus simply does not encrypt. So we drew a line: plaintext is tolerated only until it enters the Raspberry Pi concentrator. Everything after that point must be deterministic, reproducible, and securable. That mental boundary kept us from wasting cycles on unsolvable link paranoia and focused us on the software supply chain, secret hygiene, and reliable delivery.

3. Edge Stack & OS Story (Why NixOS Wasn’t Just “Because Nerd”)

Each concentrator: Raspberry Pi + custom case + UPS + NVMe (USB3) + Modbus interface hat. The Pi runs NixOS so that the entire state that matters (services, users, networking, daemon config, journald limits, firewall, Nebula setup) is expressed in a flake. If I lose a device tomorrow: nixos-generate the image, embed secrets (age-decrypted during boot to /run/secrets), flash to SSD, plug in, power up, and it boots into exactly the expected shape—no mysterious “golden snowflake” drift. The reproducibility means we tackle reliability and security in the same motion: the fewer mutable steps, the fewer unscripted exposures.

pidgeon (the Rust process) polls meters in a tight loop—no sleep-based scheduling; actual effective sampling interval is just bounded by total cycle time (meter count × per-read latency × baud rate negotiation). That means the raw feed is “as fast as physics + cable noise allow,” not an arbitrary configured 5-second timer. Local TimescaleDB absorbs that high-tempo stream so network hiccups become a backlog issue, not data loss.

4. Secret & Identity Lifecycle (Rumor + Vault + SOPS)

Provisioning was non-negotiable: it had to be boring. We wrote a Nushell-based tool (rumor) that, when invoked for a new device, does a deterministic cascade:

  • Ask Vault for or generate: DB creds, Nebula cert pair, Wi‑Fi credentials, API key, SSH key, age key, pidgeon env bits, CA material where relevant.
  • Create SQL init + systemd environment fragments.
  • Age-encrypt outputs via SOPS, commit them, making the Git history the audit ledger (contents encrypted, structure visible).
  • During image build we inject the age private key via guestfish that gets used during boot to decrypt secrets into /run/secrets

Vault itself: single instance (intentionally—complex HA wasn’t step one). That’s “pragmatic trust” rather than full distributed cluster, but the backup cadence (weekly VM snapshot to Azure Blob) gives acceptable recovery posture.

Rotation today is manual-ish: secrets marked “renew” get regenerated if rumor reruns. It’s not elegant automation yet, but it’s safe, traceable, and avoids pretending we have a rotation framework we don’t.

API keys: generated randomly, stored salt + PBKDF2-HMAC-SHA256 (100k iterations, 32-byte output) Base64’d; verification uses constant-time comparison. Clean, conventional, no premature fancy TLS client cert matrix. We’ll only evolve when we actually have cadence + revocation tooling ready.

5. Network & Transport Path (And Why REST Is Still Here)

Data leaves via a simple minute-batch POST (JSON array of readings) over HTTPS, auth header = API key. Operational access (SSH, Postgres when needed) rides Nebula. That separation means the ingest surface stays extremely narrow while we still preserve remote maintainability. Nebula’s noise-based crypto plus the OS firewall trimming inbound to basically “SSH + altered Postgres port” keeps scanning noise negligible.

Yes, a streaming bus (proto-binary framing, backpressure, fan-out consumers) is on the roadmap because minute batching + verbose JSON is wasteful. But early on the JSON choice saved debugging hours—human-readable payloads are gold when half your lab problems turn out to be swapped A/B wires or an off-by-one register map. We deliberately chose debuggability over compression first.

6. Data Flow & Integrity (Continuous Loop → Buffered Batches)

Inside pidgeon:

  1. Endless loop over all configured meters for a particular serial port. One worker <-> one serial port.
  2. Modbus read per register group; parse into structured values (kept as Rust decimals internally to dodge float weirdness).
  3. Insert result into a flume channel
  4. Every minute a collector green thread reads the channel and dumps it to TimescaleDB (all numeric stored as strings intentionally for schema simplicity + cross-language parity with C# decimal on server).
  5. Every minute another collector green thread slices off a configurable amount of unsent measurements and JSON-batches it to the cloud.

Effective volume: for 15 meters and a loop cadence that yields roughly 1–5 seconds per full sweep (depending on baud + line stability), we end up with hundreds of samples per minute. With 18 fields (including model identifier + ISO timestamp) the minute payload is in the range of a few hundred KB in pathological best-case speed. Multiply out → hundreds of MB / day at maximum theoretical pace; this is why future binary framing + pruning matter, but NVMe + 1 TB buys us comfortable delay. Local DB currently grows unbounded; we intentionally punted TTL until after health telemetry because that is a practical trade-off for now.

Validation lives server-side using a “validator profile” concept: per meter min/max for voltage, current, active + reactive power. Out-of-range events raise alerts but the raw row persists—omitting potentially suspicious data is worse than flagging it.

Time fidelity: clocks matter for tariff boundaries. The build ensures NTP sync, and we treat timestamp alignment as a correctness property (no “the clock drifted 90s over a week” tolerated). High-frequency without trustworthy time is just noise.

7. Runtime Hardening (The Unsexy Bits That Actually Matter)

We trimmed the process list so an attacker landing on a Pi basically finds: kernel, systemd, pidgeon, Postgres, Nebula, SSH, NetworkManager, journald. That’s it. The pidgeon systemd unit gets hardening flags (private temp dirs, limited capabilities; not full seccomp fanfare yet—incremental). SSH is key-only, password auth gone. Firewall restricts inbound to SSH and the Postgres port (shifted away from 5432). Postgres local connections for pidgeon are trusted via socket; remote (rare) must be over Nebula with explicit creds. No log shipping agents, no dynamic package managers. Reproducibility reduces the opportunistic “just install tool X” temptation that leads to configuration entropy.

8. Failure Modes & Field Incidents

Real life provided colorful tests:

  • Yanked power or data cables mid-loop → confirmed local buffering worked (backfill replay intact).
  • Latency oscillating with semi-closed door → pushed us to externalize timeout parameters rather than baking constants.
  • Long bus segment noise → termination and bias adjustments solved it without protocol hacks.
  • Meter mismatch episodes (one reading “half” current vs neighbor that is not ours) usually traced to configuration or wiring.

Current visibility gap: we still don’t publish structured health telemetry (CPU load average, NVMe wear level, memory %, backlog depth, failed vs successful batch counts). We feel the absence whenever we troubleshoot blind. That’s why it’s roadmap item #1.

9. Trade‑Off Boundaries (Chosen, Not Accidental)

We explicitly tolerate plaintext Modbus because the alternative would be fantasy crypto layers that don’t stop physical tampering anyway. We accept single-instance Vault plus robust backups rather than pretending to run a clustered HA service we’d barely exercise. We store numeric strings instead of wrangling float vs decimal edge cases right now; it costs disk, buys simplicity. We let local DB grow for a while because early pruning logic mistakes risk silently losing the only ground truth we have. We haven’t bolted on rate limiting or per-tenant segmentation yet because device population + uniform trust domain make that premature. Each of these was argued, not hand-waved.

10. Outcomes (What We Actually Got For All This)

We now have a pipeline that turns “plug in a new concentrator” into a deterministic act: build image, flash, connect, watch it appear. Continuous high-frequency sampling is stable enough that when networks go flaky, we talk about backlog size rather than lost intervals. We have never had a security incident; every operational wobble has tied back to physical environment or early assumptions (like naive timeouts). And crucially we’ve banked a growing corpus of dense historical data whose existence is an asset for future tariff logic—even if today’s billing only slices daily or monthly. That strategic “collect now, monetize precision later” posture only works when integrity is defensible, which is why the secret pipeline and reproducible OS matter as much as loop speed.

11. The One Big Lesson (And Its Ripple Effects)

If I had to distill a single “wish we’d framed it this clearly on day zero” lesson: terminate messy, insecure field protocols at the earliest stable, controlled compute boundary you own, and treat that boundary as the start of the trustworthy timeline. We never tried to shuttle raw Modbus frames half across the network, and that saved a whole category of reliability and security pain (encryption wrappers, replay speculation, partial frame debugging, etc.). That decision cascaded: it justified local buffering (because that’s where trust begins), insisted that clock sync was an integrity concern (data without time = useless for tariff windows), and made declarative provisioning obviously worth the time (since the “trust anchor” device had to be rebuildable without artisan rituals). Everything else—VPN, validator profiles, backlog replays—slots in after that foundation.

12. Roadmap (Near / Mid Horizon Only)

  1. Hourly (maybe configurable) health telemetry: battery / UPS info, NVMe SMART-ish metrics, CPU + memory, filesystem %, backlog depth, success/fail counters, clock drift indicators. This directly cuts troubleshooting mean-time.
  2. Binary cache reliability + automated update flow (fix the cache population bug → allow devices to pull signed derivations rather than us manually pushing through deploy-rs).
  3. Lightweight anomaly scoring: sudden step deltas, persistently borderline readings, meter-to-meter sanity correlators (enough signal to prioritize site visits).
  4. Pruning policy: watermark-based retention (e.g. keep X days or until Y% free space, whichever is safer) with safe batch deletion (no half-chopped time windows).
  5. Transport evolution: introduce a message bus / streaming path as an additional ingestion mode; keep the existing JSON as a fallback while we benchmark throughput + resilience.
  6. Encoding optimization: optional compressed or binary framing (protobuf / flatbuffers) once observability around health + anomalies is in place (because you only compress confidently when you’ve already made introspection cheap).

(Deliberately trimmed: no premature rate limiting fantasies, no half-specified cert rotation ceremony.)

13. Closing

Security here isn’t a brag list; it’s the scaffolding that lets us bank high-frequency, time-faithful measurements now so that when the ecosystem finally pivots to fine-grained tariffs or real-time settlement, we’re not scrambling to invent history. We made a bunch of “boring” decisions—NixOS, declarative secret generation, local buffering, VPN segmentation—that compound into something resilient without demanding a giant team. And yeah, we still owe ourselves health telemetry and a cleaner binary cache story. But the core spine is solid: measure fast, trust the boundary, store reliably, move upward with integrity.

Cybersecurity IoT