Natural User Interfaces in a school environment

Microsoft’s Surface is a nifty, fancy and cool appliance. It still has an air of being an expensive gadget, however the application for it are endless. At least, that’s my opinion. Recently, with the arrival of Windows 7, many people are working in the field of ‘multi-touch application’. The idea, basically, is that one can operate a device our application using hand/fingers gestures in a, hopefully, intuitive way. It’s the intuitive manner which has triggered my attention.

Natural User Interfaces (NUI) are a larger family of any form of user interfaces that can be handled in an intuitive and less complex way. In other words, even granny who has never seen a computer-mouse, let alone a keyboard, has to be able to control a NUI-device. The first examples that appeared on Microsoft’s Surface are a first step in that direction. With simple drag and ta gestures, one is able to interact with fairly complex application. Perhaps I’m stretching the definition a bit, but all I’m saying is that using a Surface-like application, one can build powerful, yet easy to use, software application.

In one of our research project at Artesis we are focusing on discovering how NUI’s can be beneficial in a school environment. Imagine a class of 10 kids listening to their teacher in front of the class. Though he has to explain certain key elements of the given subject (let’s say geometry), the real ‘learning’ happens when the kids themselves start to toy around with parallel lines, congruent corners, etc. In the ‘old days’ this was done using pen and paper. It worked, though it was a ‘on-your-own’ activity. Recently tools such as Cabri [1] enable kids to test geometric properties on a computer…however, it remains a private activity. Even if the whole class is looking over the shoulder of the kid behind the person, only that kid can manipulate the objects and test whether the teacher was really talking sense or not.

Wouldn’t it be nice if those 10 kids could sit together around a large Surface-like er surface? They could ‘toy’ around with the circles, lines and squares on the surface and they could together discover new, exciting geometric properties. It could even have a game-aspect where the kids have to ‘compete’ against each other, they could unlock achievements (“You have discovered property x. +2 points”) and race against time to unlock as many new properties as possible. These applications don’t exist…yet, but our goal at Artesis is to show that it could be possible to have these types of applications which are a) educational (ugh, ugly word) and b) fun to use.

I should note that there are a gazillion ‘educational games’ around, but it my impression that all these game are better termed ‘educational with a hint of gaming’, because in the end they are oh so boring on the gaming side (“Huuray, you get 2 points for typing the word une fille correct”). Perhaps the other way around is better? Dunno, we’ll see.

Cool examples

Here are some neat tools that I am currently considering as being ‘good for inspiration’.

Mouse Mischief: A Microsoft application where each student has its own cursor on the powerpoint presentation (and thus each student has a mouse). Together they can manipulate objects on the slides. It was developed using the MultiPoint Mouse SDK (One PC, multiple Mice, endless possibilities) [2]. Check out the nice lesson examples on http://www.microsoft.com/multipoint/mouse-mischief/lessons.aspx
Live Geometry: Though nothing a complex as Cabri, this Silverlight demo demonstrates (including code!) how easy it is to make these kind of applications [3]
WiiBox: There thousands of applications that utilize a Wii-mote as input device. The way of handling a Wiimote is a lot more intuitive than a mouse. Who knows that our future application can be manipulated with certain kids touching the surface and other wielding a Wiimote? WiiBox is just one of many fun example which enables you to manipulate your powerpoint presentation and draw additional details on the slides while presenting (e.g. sticky laser pointer).
- Wiimote SDK: connect you Wiimote with Bluetooth and using this SDK you can code any type of .NET application that uses the wiimote as input (or output) device
- Low-cost MultiTouch whiteboard using the wiimote: video
Light Touch: Projector that enables you to ‘touch’ the presentation. Expensive probably, yet sexy.
Cheap MultiTouch Pad: making your own Surface is still expensive, however for testing purposes (.e.g. a small device) perhaps this video can be a good starter. . By the way, Seth Sandlers blog I a must-follow if you’re interested in these kinds of multi-touch toys [4]
Multitouch on Windows: I’m eagerly awaiting the arrival of this book. Perhaps this will become the reference work when it comes to coding multi-touch application for Windows? Check the link for a free pdf of the first chapter.
Make your own cheap multi-touch Surface.