Skip to content

April 3, 2010


Multi-Touch Table – Cameras and LEDs

The next set of components I’d like to talk about work together as the “source” and “detector” for touch-based input collection.  But before I get into the components themselves I’d like to talk a bit about how touch detection works on a DSI table.  I shall do my best to keep this post a bit shorter than the last two!

DSI Explained

Diffused Surface Illumination (DSI) works, as many multi-touch technologies do, by utilizing an infrared (IR) light source and specially filtered cameras to detect “touches” on the table surface.  How the light source is generated is one of the key factors that differentiate the various technologies used to build a table.  Previously I had planned to use IR lasers to build an LLP table, but for most other builds (FTIR, Rear/Front DI, DSI) IR LEDs are typically used as the light source.  Configuration of the LEDs will vary depending on the build type but for DSI you typically have a strip of LEDs that inject IR light into a specially made acrylic that reflects the light out of the top and bottom of the sheet.  I have discussed this property before in an earlier post.

So, imagine if you will, that you can see IR light.  If you were to lay down directly beneath an IR lit piece of specialty acrylic you would see a nice even illumination of the surface – it would “glow” with light.  Since you are underneath the table you would only be seeing the light that was projected “downwards” from the surface of the table.  The light that was reflected “upwards” would largely be invisible – or basically overwhelmed by the light shining directly downward.  This is best illustrated with some pictures and since I don’t have my table up yet and I didn’t want to steal anyone else’s images I have made some of my own to simulate how this would all look.

So this is what the table would look like through our “infrared vision” eyes (Pretend you are an Elf, or Half-Elf) when viewed from beneath:

Table surface lit with infrared

If someone were to touch or place an object on the surface of the table some of the light that was shining upward and away from us would be reflected back down towards us.  We would see this as a “brightening” of the area where the touch or object was placed.  It might look something like this:

Table and a touch point before processing

Now, for us Elves, this point is easy to see, but it’s not so easy for a machine to see.  It would be a lot better if we could process the image such that the “bright” points really stood out.  Well we can.  I’m going to horribly oversimplify the process but one of the key bits is called background subtraction.  As you might guess from the name, this processing involves “removing” the static areas of the image.  In our case this means we can “remove” the light exiting from the bottom of the table surface since it doesn’t really change very much.  By removing it, the lighted area of the table pretty much becomes “black”.  If we then collapse the image to pure black and white – literally just two colors, not shades of grey – then the entire area of our view becomes black.  Now when we touch the surface the most intense areas of reflected light will become pure “white” and stand out clearly from the background.  These white points can then be tracked as touches – shown below:

Touch detection after image processing

And there you have it.  Once you can detect the touches you can send touch events, just like mouse events (click, double-click, move, etc.), to applications.

So to recap, DSI uses a combination of IR LEDs and a specially manufactured acrylic that redirects light injected from the edges out of the front and rear surfaces.  Touches on the top of the surface reflect some of the IR light downwards causing a brightening of the touched areas that can be detected by Elfin eyes (or specially prepared cameras which I’ll get to shortly).  Got it?  Good.

Lighting the Acrylic

Since the acrylic used in a DSI setup must be lit from the edge most people build or buy an LED strip.  In order to achieve the required brightness you might need hundreds of LEDs – so I wasn’t terribly motivated to build.  Thankfully the folks over at Environmental Lights are kind enough to have already built a few strips just for this sort of thing.  I ended up buying the High Brightness 5050 strip which provides

5 meters of invisible light!

5 meters of surface-mounted LED goodness.  If you decide to go this route remember to also buy a PSU for the reel.  Imagine how annoying it would be to get this lovely little number only to realize you can’t turn it on!  You can bring your own PSU but make sure to adhere to the proper specs for voltage and current.

This light strip will be strung around the edge of the acrylic to provide the required illumination for touch detection.  The LEDs in this strip produce light in the 850nm frequency range.  This is important to know because you’ll want to make sure your cameras can receive light in that range.  I can assure you that these LEDs are indeed bright.  While I personally do have the eyes of an Elf, I took the chance to fire up the cameras and turn the reel on as-is, without un-spooling it.  Let me assure you, the reel lit up like a supernova!  Guess what else?  It also heated up like a supernova!  In hindsight I cannot believe the level of stupidity it took to do this.  It was only a minute or so before I could smell a faint burning and when I picked up the reel it was… intensely hot.  I quickly pulled the plug and threw it in front of a fan to cool.  Thankfully no damage was done (except maybe a shortening of LED life?) or at least no damage I can see.  Learn from these mistakes people.  I do these kinds of things so that you don’t have to.

A closer view of the LEDs

Do you See the Light!?

As Reverend Cleophus James once said to Jake and Elwood Blues – “Do you see the light?!?”  Well, no, because it’s beyond the visible spectrum.  But, with the right kind of camera you won’t have any problems.  What kind of camera is the right kind?  Well most modern digital camera sensors can pick up infrared and if you’ve ever seen One Night in Paris (harumph, harumph) then you know this to be true.   Sony’s Night-Shot mode is a great example of this but in practice most cameras put in a filter to block IR light and most camcorders aren’t really well suited to building this kind of table.  What we need is something that easily connects to a PC and can deliver streaming framerates of at least 30fps (and ideally something more like 60fps) at a decent resolution.  It just so happens there’s a really economical choice that perfectly fits these requirements – the Sony PS3 Eye.

Capable of 640x480@60fps - Can you dig it?

Yeah that’s right.  In fact the PS3 Eye will drive up to 60fps at 640×480 resolution over it’s USB interface when the proper drivers are installed.  But in order to work for our purposes it requires some modifications.  Namely removing the IR filter that prevents it from seeing anything outside the visible spectrum of light.  There’s plenty of guides covering this on the net if you look around, but just like the LED strips there are people selling ready-to-go PS3 Eye cameras with the IR filter of your choice pre-installed.  Some will even attach an m12 lens mount allowing you to choose from a variety of focal lengths from wide-angle to telephoto.  I bought mine from Peau Productions.  The m12 mounts also allow you control focus much better than the stock lens and focus has been found to be very important especially for fiducial recognition.  Something  else also matters for fiducial recognition – resolution.

Since I’m building a table with a wide aspect ratio (16:10) if I were to try and cover it with a single PS3 Eye (4:3) I’d end up giving up a good bit of resolution in the “vertical” dimension to cover the width of the table surface or I’d need to use a very wide-angle lens which tends to smear the image at the edges.  Since I want as much resolution as possible this didn’t seem like a great option.

Two cameras are indeed better than one

Fortunately there is a very simple solution – buy more cameras.  Recent updates to both the PS3 drivers and the camera processing software (more on that in later posts) allow for the outputs of multiple cameras to be combined into a single image.  This is called “stitching”.  In order to maximize the resolution of my captured images and give myself the best chance for reading small fiducials and possibly even barcodes I’ll be using a pair of cameras.  At this point you begin to understand why having a fairly burly PC is required.  If I am doing the math right (and I’m probably not) then for a single camera streaming a 2bpp (pure black and white) image there will be: 640*480*2bpp*60fps=36.8Mbps of data coming in.  Add a second camera and you’re looking at almost 74Mbps.  This doesn’t include any of the additional processing required for background removal and other image processing, blob detection and tracking, and object recognition.  It should be noted that all of that processing needs to be completed in no more than 16.6ms per frame to maintain a 60fps framerate.  Modern CPUs are pretty damn impressive, no?

Well, that covers pretty much all the raw components outside of the acrylic itself but I’m holding off on ordering that until I can get a mock-up assembled and verify all my measurements are correct.  Next time up I’ll be assembling the PC, and the mock-up frame.  Until then be happy that this post is about 900 words less than the previous one.  I know I am.

4 Comments Post a comment
  1. Steel_Wind
    Apr 11 2010

    Caught the mention of the website on Order 66 live this eve. Thanks!

    I’ve bookmarked this and will be watching your final build quite closely. My own projection rig is top down projection which suite our needs well. After severak years, I an IR pen via Wiimote – but to be honest – we don’t use the pen much. My table is quite large (5 x 8.5) and it’s difficult enough for everybody to reach their minis on the map as it stands. So I mustly just use a mouse.

    Still – mutitouch is cool and has its own charm and rewards, so I’m very interested in how this works out for you. When your table is up and running – I hope you don’t mind if I point ENWorlders to your blog via the Coolest. Gaming Set-up. Evar. thread which has been a fixture on EnWorld going on five years now.

  2. RexRemus
    Apr 12 2010

    Glad you dropped by Steel Wind. If it’s easier for you I typically update Twitter, Facebook, and Google Buzz in addition to the D20 forums and my thread on Sam’s fansite. That way you’ll know when there’s something worth coming back for :) Of course there’s also an RSS feed if you use Google Reader or any other feed reader.

    RexRemus – Twitter
    Chuck Hirstius – Facebook, Buzz

    Would be interested in the details of your top down rig. Got any pics?

  3. Aug 31 2010

    Howdy there. I’m following your posts with great interest. Working on my own DSI, (currently struggling to incorporate a NEC WT600 someone gave me, so that I don’t have to buy a new one. but it seems impossible to reflect the image. Waiting wiht baited breath to hear if the Dell actually worked out!).

    The main difference in my approach is : I’m planning to use an OptiTrack v100 camera (which does onboard image processing at 640×480 at 100fps) for about 600 bucks.

    Very curious to hear if modding PS3 Eyes is worth the headache. Please keep posting!

  4. Sep 2 2011

    Aweosome project


Share your thoughts, post a comment.


Note: HTML is allowed. Your email address will never be published.

Subscribe to comments