Sport Event Analyzer

Using XMPP and Jingle to carry huge amounts of sensor data

XMPP works quite well to access sensor data in the Internet of Things. Our friends Michael Kirsche and Ronny Klauck from BTU Cottbus know a lot about this issue as they implemented the Chatty Things XMPP stack for Contiki.

Klauck, Ronny, Kirsche, Michael: Chatty Things - Making the Internet of Things Readily Usable for the Masses with XMPP. In: Proceedings of the 8th Conference on Collaborative Computing: Networking, Applications & Worksharing (CollaborateCom), IEEE, 2012.

Ourselves at TU Dresden did a proof of concept implementation with RemoteBot, where XMPP is used to control Lego robots.

When I presented this demo at the yearly in-house exhibition of our faculty, many industry guys came by and liked the idea to use a chat protocol for the Internet of Things. But every one of them raised the same question:

Will it scale? 

Unfortunately, there is no answer to this question in the current literature. No one did exhaustive performance tests of XMPP given IoT traffic. So we thought about a use case to put some load on our XMPP server. We found a sensor dataset that perfectly matched our need: The DEBS 2013 Grand Challenge dataset.

Christopher Mutschler from University Erlangen-Nuremberg together with Fraunhofer IIS provided a dataset with live sensor data from a soccer game where each of the players was wearing at least two sensors. The data provides position, acceleration and speed measurements in three dimensions with 200 measurements per second. The ball is even logged at 2000 Hz. This creates an event stream of 15,000 events per second.  

So if XMPP could handle this quite futuristic scenario, the industry guys will stop worrying about XMPP performance.

We set up a team of smart students working on the issue. They set up an OpenFire server, put each sensor event from the soccer log in a message and opened the tap. But OpenFire refused to forward more than 10,000 events per second over the line. No chance. While there might for sure have been some options for accelerating the server or even server clustering, this would only have prolonged the suffering.

But the good thing about XMPP is that if you have a problem, there is always already an XEP solving the issue. In this case the solution was to use the Jingle extension to set up a direct UDP connection between sender and receiver. This streaming-like approach was able to serve millions of messages per second on standard hardware between two nodes.

The server than ran a Bot doing some event processing. It built up an XMPP PubSub model with player and ball positions as well as some more data needed at more coarse grained level. Thus XMPP is used for both, sending the raw sensor data as well as publishing the processed data to be viewed by some application. Last but not least, they used JavaScript and BOSH for building some cool visualization of the sensor data.

You can see the full demo here on YouTube.


tldnr: We used Jingle-based streaming, PubSub and BOSH to build a cool visualization of sensor-equipped soccer players.