IoT edge-to-cloud: optimise costs without sacrificing performance

The challenge of IoT projects

IoT projects generate a lot of data. Sensors send measurements every second, devices report their state continuously. If you send everything to the cloud without filtering, infrastructure costs can explode quickly. We know because we've been there. That's why we're sharing our tips.

The goal? Process as much data as possible at the edge (on the device or a local gateway) and send only what's truly necessary to the cloud. That's what edge-to-cloud architecture is. It may sound complex, but it's just common sense. Here's how we do it.

Strategy 1: Filtering and aggregation at the edge

The principle: Don't send all raw data to the cloud. Filter, aggregate, and only send useful data.

Threshold filtering

If a sensor sends a temperature every second but the value rarely changes, there's no need to send every value. Send only when:

• The value changes significantly (e.g. ±2°C)
• An event occurs (e.g. threshold exceeded)
• At regular intervals to confirm the device is active (e.g. every 5 minutes)

Time aggregation

Instead of sending 60 values per minute, aggregate them:

• Average: For continuous values (temperature, pressure)
• Min/Max: To identify spikes
• Last value: For binary states

This reduces data volume from 60 to 1 per minute—a 98% reduction.

Strategy 2: Data compression

The principle: Compress data before sending it to the cloud.

Payload compression

Use compact data formats:

• MessagePack or CBOR: Binary formats more compact than JSON
• Protobuf: Schema-based binary format, very efficient
• Gzip: Classic compression, supported everywhere

Example: a 1 KB JSON payload can be reduced to 200–300 bytes with MessagePack + compression.

Strategy 3: Batching and buffering

The principle: Instead of sending each message individually, group them and send in batches.

Time-based batching

Accumulate messages for X seconds/minutes, then send them all at once. This reduces HTTP/MQTT requests and connection costs.

Size-based batching

Send a batch as soon as you've accumulated X messages or Y KB of data. This improves the useful data / network overhead ratio.

Local buffering

Store data locally (SD card, flash memory) and send it in batches when the connection is available. Useful for devices with intermittent connectivity.

Strategy 4: Protocol choice

MQTT vs HTTP: For IoT, MQTT is generally more efficient than HTTP:

• Lower overhead: Lighter headers than HTTP
• Persistent connection: No need to reconnect for each message
• Pub/Sub: Model well-suited to IoT
• QoS: Service quality levels to ensure delivery

MQTT-SN: A lightweight version of MQTT for low-bandwidth networks (LoRa, Sigfox).

Strategy 5: Cloud storage and processing

Time-series databases: For IoT data, use databases optimised for time series:

• InfluxDB: Free up to 1M points/month
• TimescaleDB: PostgreSQL extension for time series
• AWS Timestream: AWS managed service

These databases are optimised for heavy writes and time-based queries, making them more performant and less expensive than classic relational databases.

Retention and archiving

Don't keep all data indefinitely:

• Recent data: Fast storage (SSD) for the last 30 days
• Older data: Archive to cold storage (S3 Glacier, etc.) for data older than 30 days
• Aggregation: Compute hourly/daily averages and archive raw data

Strategy 6: Edge computing

The principle: Process data as close to the source as possible, on the device or a local gateway.

Examples of edge processing

• Anomaly detection: Identify outliers locally and send only alerts
• Local prediction: Lightweight ML models on the device (TensorFlow Lite)
• Automatic decisions: Actuators controlled locally without going to the cloud

Reduces latency, bandwidth and cloud costs.

Concrete example: cost reduction

Initial scenario:

• 100 sensors
• 1 message/second per sensor
• 100 bytes per message
• Cost: ~€500/month (bandwidth + storage + processing)

With optimisations:

• Threshold filtering: -80% messages
• Aggregation: average every 5 minutes: -98%
• Compression: -70% size
• Result: ~€50/month (90% reduction)

In summary

Optimising IoT costs is possible—and it's quite simple. Here's what we do:

1. Filter at the edge: Don't send all raw data. Keep only what's useful.
2. Aggregate: Reduce data volume. One average is better than 60 individual values.
3. Compress: Reduce message size. It seems obvious, but it's often forgotten.
4. Batch: Reduce the number of requests. Group, send, done.
5. Choose the right protocol: MQTT over HTTP. It's made for IoT, so use it!
6. Use time-series DBs: Optimised for IoT, more performant and cheaper.
7. Process at the edge: Reduce what goes to the cloud. Fewer data sent = lower costs.

These strategies significantly reduce costs without sacrificing performance. The key is to start simple (filtering, aggregation) and optimise gradually. We didn't become experts overnight—and neither will you. Take it step by step.