SWEET - Join The Crowd Emotion Monitor

Want to know where the action is? Interested to get real time feedback about conference hotspots? By participating in this experiment, you will receive real time updates about the appreciation of several conference events by you and your fellow participants. Everybody who has an Android smartphone can easily join. Alternatively, you can follow the experiment via Twitter on an iPhone, iPad or any other mobile device. 

By participating in this live demonstration (results will be presented on Thursday 28th August, 16:50-17:10, Pomonazaal), you will be the first to use this app which can enhance the experience during conferences, but can also increase safety at mass events by monitoring the emotion of a crowd. This will also demonstrate how multimodal measurements can create insight into group processes, and inspire you to think about other applications of this new technology. 

What is SWEET?

SWEET is short for Sense & Tweet. It’s a software tool combining state-of-the-art sensing, event processing, reasoning and data communication technologies. The result of this development is a Crowd Emotion Monitor, an application that is able to assess the behavior and emotion of groups of people and relay that information to stakeholders in a highly intuitive manner. 

Example screenshot of central visualization

The purpose of this experiment is to enhance the experience during the conference: Information is collected and fed back to participants, which tells you at which locations there is a lot of activity or where many smiles have been detected. In a broader sense, the Crowd Emotion Monitor can be used at events where people gather for professional reasons (such as Measuring Behavior 2014) or for entertainment (e.g. dance parties) and where they can benefit from an app that guides them to locations where they can gain the best experience. The Crowd Emotion Monitor can also contribute to increasing safety: information that is collected can be interpreted in order to assess the stress level in a crowd, to detect behavior or emotion that can lead to an incident, or to detect incidents at an early stage.  

SWEET is being developed in the context of COMMIT, a Dutch public-private research program in information and communication science. Partners of the consortium are CWI/MonetDB, TNO, University of Twente, VicarVision and Noldus.  

How does it work?

The measurement platform uses the built-in sensors of smartphones. Combining many smartphones makes it possible to measure the behavior of a crowd. The SWEET app will measure:

  • Location: Using the ‘fingerprint’ of WiFi access points to track the participants. During Measuring Behavior, this will be done in the ‘Terraszaal’ on the ground floor, which is the exhibition area where also lunch and refreshments are served.  
  • Emotion (of the smartphone owner or others): Using facial expressions detected with camera and analyzed using the FaceReader analysis server. This can be done using pictures taken by participants of themselves or other participants using their own smartphone, and by cameras installed in the ‘Ir Haakzaal’, which is the large lecture room on the first floor. 
  • Subjective feedback: With an event-driven questionnaire tool about how the participants are feeling.

The collected information is relayed in a highly intuitive manner and used to give real time feedback to its users, for instance to direct them to the location with the most positive experience. The app shows an ‘emotion map’ of the conference venue, with hotspots visualizing the responses of all participants.

People who have not installed the SWEET app can be informed by tweets containing feedback. Aggregated data will also be visualized on a large display at the Noldus InnovationWorks booth during the conference (booth #4 - see the exhibition floor plan here). 

Example screenshots of the SWEET App  

For more information about the SWEET project, you can also join the demonstration during the conference, which is scheduled for Thursday 28th August, 16:50-17:10 (in the Pomonazaal). 

How can I join?

There are two ways to participate:

  • SWEET participants: download the SWEET app from the Google Play Store, install it on your smartphone and agree to be monitored at the conference. You can carry your phone in a normal way, measures are made continuously. You can make photos of yourself and fellow participants for analysis of facial expressions. Only a few sensors (the microphone for sound and WiFi for location tracking) are used continuously, in order to minimize energy consumption.

    If you want to download the SWEET app, please contact us at noldusnbd@gmail.com and you will receive an e-mail with the download details at the start of the conference.

  • SWEET followers: if you haven’t installed the SWEET app on your smartphone, you can still sign up as a follower of the SWEET Twitter channel (#MB2014SWEET) to receive messages related to the event. Tweets are broadcasted to inform people of special activities or where something nice or interesting is going on. These tweets can be received with any Twitter application on any device.

Feedback

SWEET is in an experimental stage: it is work in progress, which means not all functionality has been completed and the software may contain bugs. No personal information is stored. All images being taken are temporary and privately stored on the device and are only accessible by the SWEET app. After feature extraction, the images are automatically deleted. All audio recordings are kept privately on the device and are only accessible by the SWEET app. After analysis the recordings are removed from the device. 

We love to hear your feedback! Please let us know what you think of this tool and its applications. Drop us a note at noldusnbd@gmail.com. 

Copyright © 2014 The SWEET Consortium. All rights reserved.