computers should be less friendly

At SAP TechEd Las Vegas I wanted to bounce some ideas off the RedMonk Gang( Michael Cotè and James Governor ) about what I think the new Developer will look like instead I somehow got roped into a full blown interview on Synthesizers.  So, I’ve frantically spent the last few weeks solidifying my ideas in this area.

The new Synthesizer will have a few required tenets, listed in order of importance, underneath are a many “optional” components.  It’s okay if you don’t exhibit these characteristics — you might still be able to get a job flipping burgers:

Code

Sorry, namby-pamby non-coder types need not apply.  A real understanding of programming concepts is required.  However, I’ve found that Synthesizers are generally not language Nazis.  They are more opportunistic in their choice of language, they know a lot about probably one language but, also know when it’s not a good fit for what they are doing.  For instance, why would write a screen saver in ABAP?

Community

This is the area that allows Synthesizers to move faster then anyone else.  They know what is going on, they’re “plugged in.”  They know who is doing good work and they know how to stick those disparate parts together in order to get something better.  Doug Mccune ( my model synthesizer ) has put it best in a post about TileUI:

…how badass [the] open source community libraries for Flex are. I was able to grab these open source libraries and within a few days have something pretty sweet to show for it. A big thanks to everyone behind the PaperVision project, and to Alec Cove for the APE engine. You guys make this stuff easy. [ The making of TileUI ]

What else is in this quote?  Attribution, “A big thanks to Alec Cove for the APE engine.”  Promoter, you can tell from Doug’s tone how much the open source community helped him write this application.  Collaboration, without giving back to these communities you end up looking like a strip miner taking all the best out and not returning anything back.  A example from a totally different world is Thomas Jung, SAP geek and ABAP coder #1.  Thomas has built a ton of innovative stuff in ABAP from Matrix Screen Savers to a full replacement for the delivered web rendering system, changing it from HTML to Flex, named Flob.  At TechEd this year he released Flob, built using my library AJS, which he mentioned he had a few fixes for functionality.  This back and forth consumption and production allow open source projects to grow organically as need.

Communication

This is an obvious one, but again has a similar ebb and flow characteristic.  The most effective Synthesizers have a blog or are really active in forums.  They produce not only code but content, documentation, opinions — they care about what’s out there and what they are putting out there.  They probably read RSS feeds from dozens(at least) of different sources from many different areas.

Broad Knowledge

Underlying all these is less a tenet and more an obvious prerequisite.  Synthesizers want to understand stuff, they want to know about many areas not just Coding.  They might read about other sciences or be super into music.  Samuel Agesilas Pastel is great example of someone with groundings in music and design who also happens to be a sick programmer.  His UML modeling tool, called Saffron UML, parses actionscript libraries and generates great looking UML diagrams.  Just read his blog of a bit and you will see the diverse view he takes to software development.  That diversity allows Synthesizers to keep an anything goes mentality that enables them to see connections where other people see chasms they will never cross.

The funny thing with this set is maybe if you replace the first tenet with “industry analysis” or “accounting” you might get some other folks that are producing new Synthesizers — but I’m just a programmer what do I know.

Thanks to Synthesizer, maverick for the images.

One of the common questions I was asked after demoing Majority Desk was how the heck is it built?  To explain, I will first break down the individual components and then talk about how the Wiihands move across the screen.  There is a lot more to the interactions in this system but, moving the Wiihands should give you a pretty good insight into how the whole thing is built.  Just to frame the following discussion Majority Desk is mostly an AIR application built using Flex Builder 3.

ODE2Paper

ODE2Paper is the main visualization and physics library I wrote that underpins the Majority Desk application.  It binds the Open Dynamics Engine (ODE for short), which is the underlying physics system to Papervision3d (PV3D for short), the 3d rendering system.  The foundation of this interaction is actually very simple: every time Flex fires an ENTER_FRAME event I issue a refresh command over a socket that is connected to the physics server.  The physics server is a python application that uses pyODE to wrap the ODE libraries.  The physics server “parses” the refresh command,

<refresh/>

iterates across all the objects in its world producing a new XML document with the position and transformation matrices of all the existing objects and sends that XML back to Flex. 

<world>
    <o1 x="0.0" y ="0.0" z="0.0" r0="0.0" r1="0.0" r2="0.0" r3="0.0" r4="0.0" r5="0.0" r6="0.0" r7="0.0" r8="0.0"/>
    <o2 x="10.0" y ="15.0" z="10.0" r0="0.0" r1="0.0" r2="0.0" r3="0.0" r4="0.0" r5="0.0" r6="0.0" r7="0.0" r8="0.0"/>
<collisions/>
</world>

When this data transfer is complete on the Flex side, I move all the objects in the PV3d scene and then call the camera render.  Simple right?

A Wii bit of background

The Wiimote is a blue tooth transmitter, it reports its position and state to the blue tooth receiver on the client computer.  It determines its position by “looking” through the IR port on the front of the Wiimote at the poorly named sensor bar.  The sensor bar in reality has no sensors in it — it has two IR emitters that allow the Wiimote to triangulate its X and Y position in space.  The Wiimote also has an accelerometer on board that allows it to measure movement in the X, Y, Z planes (pitch, yaw, and roll).  This page has a mountain of technical detail about the Wiimote.

WiiFlash

The information from the Wiimote is sent to the WiiFlash server.  WiiFlash removes all the complexity out of using Wiimotes to control any Flash application.  The API is pretty complete and we found it to be fairly stable.  This is the section Eddie dealt with most so, I can’t talk too intelligently about it except to say that we use the WiiFlash AS3 API to access all the position and button state information from the Wiimote.  The API handles all the calls through a local connection that streams the information from the projects WiiFlash Server, which for now only runs on windows, sorry Mac faithful. 

This pretty much rounds out the subsystems that lie underneath the application.  This doesn’t really explain how the thing works so, I will walk you through how the Wiihands move around the screen to hopefully give you a better idea of what is going on.

Moving Wiihands

The most common thing we do in Majority Desk is move our Wiihands to interact with the world.  To allow interactions with the state of the physical world maintained by the physics server we have a set of commands in ODE2Paper.  The ones that matter for the Wiihands are createSphere, setObjPos, createJoint and breakJoint.  These methods serve as proxy for the real commands flying over to the physics server.  The rationale for this is again the main underlying concept that the physics server knows where the objects are and PV3d simply renders them.  So all the interactions are really taking place on the physics server.  The Wiihands are represented in the physics system as sphere because they are in reality just a cool texture wrapped on a sphere in PV3d.

So as a Wiimote moves we get its position and call a setObjPos method in ODE2Paper library which moves the object in the physics server.  Again the physics server leads and PV3d simply renders a view of the world.

Grabbing stuff

If you got the chance during Hacker Night or our Meet the Jammers session to mess with Majority Desk you may remember that if you hold the trigger and bump into something with your Wiihands you can grab it.  Letting go of the trigger will release the object and let it float around again.  This is achieved using the collisions XML node that I didn’t talk about earlier.  If you look back at the XML doc that the physics server creates to represent the world you will see an empty node there for collisions.  ODE2Paper notifies our Flex application that two bodies are about to collide via this XML node.  This allows us to detect that the Wiihand is touching something and then build a joint, using the ODE2Paper method createJoint.  A joint is a type of attachment between two bodies in ODE, we use simple static joints but a whole slew of different joints types are supported by ODE.  This is why the widget seems to follow a Wiihand around as it moves.  The great thing is this affect is achieved by just calling setObjPos on the Wiihand, which is the normal behavior described previously.   To release the object we simply call breakJoint and the physics server breaks the joint allow the attached object to move fluidly again.

If you want a run down of links and a good video that shows Majority Desk in action, check out Eddie’s blog post about it.

So far the highlight of TechEd for me has been people’s reaction to Majority Desk.  I said to a number of people before the demo that everyone’s reactions would be the same, “Oh my god…”  The problem is that that reaction would come because of two very different reasons, “Oh my god… what is wrong with you people that has no business value!” or “Oh my god… that is the coolest thing I’ve seen in a long time.” 

It seems that most people fall in the latter category.  I’ve already spoke to a number SDNers from all different industries who see how they can use this type of interface in a broad range of applications.  A airplane manufacturer navigating large engineering drawings to business people visualizing data sets in 3d to see patterns.  This was the reaction we were hoping to create in the community a true sense of innovation and possibility.  It’s also been a blast watching people mess around with the interface.

Here are some great photos from hacker night with Mark Finnern and Marilyn Pratt messing around with Majority Desk.