Augmented reality sand box

I have been involved in constructing augmented reality sand box (ARSandbox) lately. It is a beautiful project created by Dr. Oliver Kreylos at UC Davis. The system uses a Microsoft XBox 360 Kinect sensor to digitize the sand in a box and then uses a projector to project color-coded elevation and contours on the sand, thus augmenting the sand with colors and contours. Here is a photo I took on a prototype that we replicated from the ARSandbox created by Dr. Kreylos:

When someone manipulates the sand, thus changing the topography, the projected colors and contours change accordingly. You can also rain over the terrain by a hand gesture over the terrain.

For those that didn’t know, the Microsoft XBox 360 Kinect sensor is a sophiscated set of sensors that include IR projector and camera for depth sensing, body movement and gesture capture, and regular RGB camera and microphone array. The software Dr. Kreylos developed takes the depth image and calculates a topography map and projects it onto the very same sand using a calibrated projector. To show you how good the simulation is, here is a photo:

Did you notice the white cone and the colors/contours on it? The contours are depicting 1cm heights and the cone is about 4cm tall. Here is a close-up of the cone placed at a different location:

It shows roughly 4cm tall and the contours are very well centered around the tip of the cone.

Here is a video:

Like it? The whole setup is not cheap. It needs an expensive video card for the simulation, especially the water. It also needs a decent desktop computer and projector, sand box, frames etc. Here is what my setup looks like:

I didn’t buy a more expensive (thus shorter throw ratio) projector so my setup is very tall even without any legs. I am hoping to develop it into a portable system so I can take a few of them to teachers’ training workshops, museums, schools, fairs etc. for basic education and outreach for water resources.

What I’m thinking about doing using my Arduino/Raspberry pi skills is to add sensors to help preserve the projector’s bulb and have kids and operators interact with the sand box without having to use the keyboard and mouse or understanding linux. Big buttons will do certain predefined things such as rain, drought, etc.