We could already demo, that sharky may become controlled by a Vaadin web UI properly. Now we are going to help sharky in its natural evolution.
The main idea
We would like to use a natural interface to remote control two sharkies at the same time. A 3D-sensor should observe the left and the right hand movements. Gestures by the left hand should remote control sharky-1 and gestures by the right hand should control sharky-2.
The technical solution should look like the image below.
So we bought Xbox Kinect. The problem was, that Kinect SDK only supports windows and OpenNI dropped linux support for license reasons. Again we explored the web and found Asus Xtion Pro Live. A 3D-sensor developed for developers and native OpenNI support. The sensor will capture 3D-images and sends them to OpenNI. NiTE – as an openNI plugin – provides java API for skeleton and hand tracking. Our first idea was to install OpenNI and NiTE on a beagle bone. But NiTE does not support ARM processors for now. So we adjusted our architecture again and have been installing OpenNI and NiTE on a X86 Ubuntu. Writing some java glue code allows us to track the positions of a hand in 3-dimensions. Since we are addicted to M2M-technologies we do not further process that information on the Ubuntu device, but send them using Eclipse Paho to a M2M-Server using the MQTT protocoll.
The M2M-Server (a Mosquitto server) is running on a Beagle Bone Black. And it aims as a publish/subscribe server. Clients can subscribe to topics and will get messages sent to the topic. The Ubuntu device sends all messages to the “handinfo” topic at the M2M-server.
A very nice project called Jnect provided by Jonas Helmig and Maximilian Kögel (Eclipsesource Munich) implements a body model based on EMF. It also supports gesture recognition. Your own gesture handler may be registered using extension points. So the idea is, that eclipse equinox is becoming installed on an additional beagle bone. Using Paho the beagle bone connects to the M2M-Server at the topic “handinfo”. So all changes of the human hands in any of the 3-dimensions is sent to this beagle bone. Implementing some glue code, the body model based on EMF is prepared.
At a next step, we have to add GestureHandlers. These are being notified about changes in the body model and have to calculate whether a gesture was detected. For instance “left hand up”, “right hand down”, “hands clapped”,… The gestures will be sent to the M2M-Server at the topic “gestures”.
These gestures are the base information for the Sharky controller. The sharky controller is also installed on a beagle bone black and is based on Mihini. Using Lua it connects to the M2M-Server and subscribes the topic “gestures”.
So if the human raises its right hand, the SharkyController gets the information “right hand up”. Sharky controller will use that information to calculate the required GPIO outputs. The GPIOs are connected to the remote controller and sharky follows the commands introduced by hand movements.
Planned commands – left hand controlls sharky-1 and right hand controlls sharky-2
- hand left -> “sharky turn left”
- hand right -> “shary turn right”
- hand up -> “sharky raise”
- hand down -> “sharky dive”
- hand towards the sensor -> “sharky get faster”
- hand away from sensor -> “sharky get slower”
So we have a lot of work to do, to implement things properly. Lets see what happens…