computers should be less friendly

One of the common questions I was asked after demoing Majority Desk was how the heck is it built?  To explain, I will first break down the individual components and then talk about how the Wiihands move across the screen.  There is a lot more to the interactions in this system but, moving the Wiihands should give you a pretty good insight into how the whole thing is built.  Just to frame the following discussion Majority Desk is mostly an AIR application built using Flex Builder 3.

ODE2Paper

ODE2Paper is the main visualization and physics library I wrote that underpins the Majority Desk application.  It binds the Open Dynamics Engine (ODE for short), which is the underlying physics system to Papervision3d (PV3D for short), the 3d rendering system.  The foundation of this interaction is actually very simple: every time Flex fires an ENTER_FRAME event I issue a refresh command over a socket that is connected to the physics server.  The physics server is a python application that uses pyODE to wrap the ODE libraries.  The physics server “parses” the refresh command,

<refresh/>

iterates across all the objects in its world producing a new XML document with the position and transformation matrices of all the existing objects and sends that XML back to Flex. 

<world>
    <o1 x="0.0" y ="0.0" z="0.0" r0="0.0" r1="0.0" r2="0.0" r3="0.0" r4="0.0" r5="0.0" r6="0.0" r7="0.0" r8="0.0"/>
    <o2 x="10.0" y ="15.0" z="10.0" r0="0.0" r1="0.0" r2="0.0" r3="0.0" r4="0.0" r5="0.0" r6="0.0" r7="0.0" r8="0.0"/>
<collisions/>
</world>

When this data transfer is complete on the Flex side, I move all the objects in the PV3d scene and then call the camera render.  Simple right?

A Wii bit of background

The Wiimote is a blue tooth transmitter, it reports its position and state to the blue tooth receiver on the client computer.  It determines its position by “looking” through the IR port on the front of the Wiimote at the poorly named sensor bar.  The sensor bar in reality has no sensors in it — it has two IR emitters that allow the Wiimote to triangulate its X and Y position in space.  The Wiimote also has an accelerometer on board that allows it to measure movement in the X, Y, Z planes (pitch, yaw, and roll).  This page has a mountain of technical detail about the Wiimote.

WiiFlash

The information from the Wiimote is sent to the WiiFlash server.  WiiFlash removes all the complexity out of using Wiimotes to control any Flash application.  The API is pretty complete and we found it to be fairly stable.  This is the section Eddie dealt with most so, I can’t talk too intelligently about it except to say that we use the WiiFlash AS3 API to access all the position and button state information from the Wiimote.  The API handles all the calls through a local connection that streams the information from the projects WiiFlash Server, which for now only runs on windows, sorry Mac faithful. 

This pretty much rounds out the subsystems that lie underneath the application.  This doesn’t really explain how the thing works so, I will walk you through how the Wiihands move around the screen to hopefully give you a better idea of what is going on.

Moving Wiihands

The most common thing we do in Majority Desk is move our Wiihands to interact with the world.  To allow interactions with the state of the physical world maintained by the physics server we have a set of commands in ODE2Paper.  The ones that matter for the Wiihands are createSphere, setObjPos, createJoint and breakJoint.  These methods serve as proxy for the real commands flying over to the physics server.  The rationale for this is again the main underlying concept that the physics server knows where the objects are and PV3d simply renders them.  So all the interactions are really taking place on the physics server.  The Wiihands are represented in the physics system as sphere because they are in reality just a cool texture wrapped on a sphere in PV3d.

So as a Wiimote moves we get its position and call a setObjPos method in ODE2Paper library which moves the object in the physics server.  Again the physics server leads and PV3d simply renders a view of the world.

Grabbing stuff

If you got the chance during Hacker Night or our Meet the Jammers session to mess with Majority Desk you may remember that if you hold the trigger and bump into something with your Wiihands you can grab it.  Letting go of the trigger will release the object and let it float around again.  This is achieved using the collisions XML node that I didn’t talk about earlier.  If you look back at the XML doc that the physics server creates to represent the world you will see an empty node there for collisions.  ODE2Paper notifies our Flex application that two bodies are about to collide via this XML node.  This allows us to detect that the Wiihand is touching something and then build a joint, using the ODE2Paper method createJoint.  A joint is a type of attachment between two bodies in ODE, we use simple static joints but a whole slew of different joints types are supported by ODE.  This is why the widget seems to follow a Wiihand around as it moves.  The great thing is this affect is achieved by just calling setObjPos on the Wiihand, which is the normal behavior described previously.   To release the object we simply call breakJoint and the physics server breaks the joint allow the attached object to move fluidly again.

If you want a run down of links and a good video that shows Majority Desk in action, check out Eddie’s blog post about it.

This entry was posted on Tuesday, October 9th, 2007 at 10:32 am and is filed under Flex, SAP, SDN blogger, innovation. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.