Using XMPP and Jingle to carry huge amounts of sensor data
XMPP works quite well to access sensor data in the Internet of Things. Our friends Michael Kirsche and Ronny Klauck from BTU Cottbus know a lot about this issue as they implemented the Chatty Things XMPP stack for Contiki.
Ourselves at TU Dresden did a proof of concept implementation with RemoteBot, where XMPP is used to control Lego robots.
When I presented this demo at the yearly in-house exhibition of our faculty, many industry guys came by and liked the idea to use a chat protocol for the Internet of Things. But every one of them raised the same question:
Will it scale?
Unfortunately, there is no answer to this question in the current literature. No one did exhaustive performance tests of XMPP given IoT traffic. So we thought about a use case to put some load on our XMPP server. We found a sensor dataset that perfectly matched our need: The DEBS 2013 Grand Challenge dataset.
Christopher Mutschler from University Erlangen-Nuremberg together with Fraunhofer IIS provided a dataset with live sensor data from a soccer game where each of the players was wearing at least two sensors. The data provides position, acceleration and speed measurements in three dimensions with 200 measurements per second. The ball is even logged at 2000 Hz. This creates an event stream of 15,000 events per second.
So if XMPP could handle this quite futuristic scenario, the industry guys will stop worrying about XMPP performance.
We set up a team of smart students working on the issue. They set up an OpenFire server, put each sensor event from the soccer log in a message and opened the tap. But OpenFire refused to forward more than 10,000 events per second over the line. No chance. While there might for sure have been some options for accelerating the server or even server clustering, this would only have prolonged the suffering.
But the good thing about XMPP is that if you have a problem, there is always already an XEP solving the issue. In this case the solution was to use the Jingle extension to set up a direct UDP connection between sender and receiver. This streaming-like approach was able to serve millions of messages per second on standard hardware between two nodes.
You can see the full demo here on YouTube.
tldnr: We used Jingle-based streaming, PubSub and BOSH to build a cool visualization of sensor-equipped soccer players.