One PC, Three Kinects

Recently I managed to get a hold of three Microsoft Kinects.  A few of my family members had one (including myself) and ended up not needing them since they never used them.  Being the "techy" I am, the first thing I did was attempt to integrate them into my recent Oculus Rift projects.  While the code seems to be correct, there was an issue from a hardware perspective that I was hoping to not run into.  Alas, I have, and thus am writing this blog post about what exactly is going on.  It's an interesting little hardware problem that I'm hopeful Kinect 2.0 will solve a bit more, and that'd I decided to do a quick write-up about.

Actually Kinecting with Multiple Kinects

Figure 1a - No Kinects Detected example.

To start off explaining this issue, I want to explain how exactly you can connect to more than one Microsoft Kinect in C#.  If you already are familiar with how this works (Or don't really care), skip down to the next section for the hardware side of things.

We'll use a form that has four PictureBox objects, each containing a default image that says there are no Kinects detected (See figure 1a).

What we'll want in our code side of things for this example is just one global variable, which will be a Dictionary object:

 Dictionary<string, int> KinectIDs;  

These will be used for storing the IDs of storing each Microsoft Kinect sensor that is connected to the system, that way we can output the image from each Kinect to the proper PictureBox from before.

Don't forget to initialize these somewhere.  I did so within the constructor like such:

 public mainForm()  
 {  
    KinectIDs = new Dictionary<string, int>();  
    InitializeComponent();   
 }  
   

Next, within the main form we'll need to setup each Kinect when the form is loading.  While we won't focus on error-handling in this blog post, the code does have a small amount if you would like an example of some things that can happen.  The code is as follows:

     private void mainForm_Load(object sender, EventArgs e)  
     {  
       // Count how many Kinects are attached.  
       int kinectCount = 0;  
       // Loop through each Kinect.  
       foreach (KinectSensor kinect in KinectSensor.KinectSensors)  
       {  
         // Checks that the Kinect is properly Kinected (had to put one pun in).  
         if (kinect.Status == KinectStatus.Connected)  
         {  
           // Add the Kinect's Unique ID as a key and the kinect sensor index as the count.  
           KinectIDs.Add(kinect.UniqueKinectId, kinectCount);  
           // Increment the Kinect count.  
           kinectCount += 1;  
   
           // Setup the color frame event handler;  
           kinect.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(mainForm_ColorFrameReady);  
           // Enable the color stream at 640x480 resolution and 30 frames per second.  
           kinect.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);  
           // Start the Kinect camera.  
           kinect.Start();  
         }  
       }  
     }  
   

The comments should be self-explanatory, especially if you've worked with the Microsoft Kinect before, but to sum it up, this method will count the number of Kinects, adding the ID of each one to our dictionary object from before, while also setting up the Kinects to stream the color image data.  Note that it only does this if the Kinect is properly connected though.

For the actual handling of the image data, here is the code:

     void mainForm_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)  
     {  
       // Create a color frame to hold the color frame that was recieved.  
       KinectSensor sensor = (KinectSensor)sender;  
       // Create an image index.  
       int imageIndex = 0;  
   
       // Try to get the index associated with the Kinect's unique ID.  
       if (KinectIDs.TryGetValue(sensor.UniqueKinectId, out imageIndex))  
       {  
         // Get the color frame from the device.  
         ColorImageFrame frame = e.OpenColorImageFrame();  
         // Set the picture box image at the specified index with the color image frame.  
         setImage(imageIndex, frame);  
       }  
     }  
   

All this method does is check that the ID for the Kinect Sensor was stored previously (IE: that it was connected), and then proceeds to pass the color image frame and the index of the image to draw to into a method called setImage.  Set image simply displays the image to one of our four picture boxes from before via the following code:

     void setImage(int imageIndex, ColorImageFrame frame)  
     {  
       // Check that the frame is not empty.  
       if (frame != null)  
       {  
         // Create an array of bytes to hold the data.  
         byte[] pixelData = new byte[frame.PixelDataLength];  
         // Copy the pixel data to the array.  
         frame.CopyPixelDataTo(pixelData);  
   
         // Create a bmp to hold the actual image.  
         Bitmap bmp = new Bitmap(frame.Width, frame.Height, PixelFormat.Format32bppRgb);  
         // Create the bitmap data to hold the data for the image.  
         BitmapData bmpData = bmp.LockBits(new Rectangle(0, 0, frame.Width, frame.Height), ImageLockMode.WriteOnly, bmp.PixelFormat);  
         // Create the pointer to the bmp data.  
         IntPtr ptr = bmpData.Scan0;  
         // Then copy the actual data to the bmp.  
         System.Runtime.InteropServices.Marshal.Copy(pixelData, 0, ptr, frame.PixelDataLength);  
         // Unlock the bmp data bits so they can be used.  
         bmp.UnlockBits(bmpData);  
         // Switch through the image index.  
         switch (imageIndex)  
         {  
           case 0:  
             // Set the color image for the 1st picture box to the bmp image.  
             this.kinectPB1.Image = bmp;  
             break;  
           case 1:  
             // Set the color image for the 2nd picture box to the bmp image.  
             this.kinectPB2.Image = bmp;  
             break;  
           case 2:  
             // Set the color image for the 3rd picture box to the bmp image.  
             this.kinectPB3.Image = bmp;  
             break;  
           case 3:  
             // Set the color image for the 4th picture box to the bmp image.  
             this.kinectPB4.Image = bmp;  
             break;  
         }  
   
         // Dispose of the frame.  
         frame.Dispose();  
       }  
     }  
   

And there's a single Kinect working!

Again, the comments should explain everything, but to summarize, it simply takes each frame, converts it to a Bitmap image to display, and then displays it to the appropriate picture box.

Overall, pretty simple.

The Problem

This is where things get tricky, as while the code above should work, on many computers it probably won't.  This is entirely due to a hardware problem within the Kinect.  Really, it's not a problem even, it's a simple limitation of computers these days, and can appear in a variety of ways.

First, the big thing to consider is that the Kinect is streaming a TON of data to your computer.  It has 16 microphones, infrared data, color image data, depth data, skeleton data, and more things being sent to you computer, all via one little USB port.  Most computers these days have USB 2.0 support, but that only allows for 35 MB/s, which while quite a bit of data, still doesn't really provide enough for that much data, especially with the quality of everything.  While yes, it is not full HD, it's still a lot of data to be sending over a USB 2.0 port.  USB 3.0 and 3.1 fix this, with 500 MB/s data rates, and will help to eliminate this problem, but USB 3.0 and 3.1 are still not widespread enough currently.

If only there was another way!

This means you'll run into the error that I had, where you don't have enough bandwidth to use more than 1 Kinect at a time.  Unfortunately, there really isn't much that can be done on Microsofts end about this.  It's an unfortunate limitation on computers currently that will be fixed as time goes on.  As more computers roll out with USB 3.0 and 3.1, the issue will be resolved, and then we'll be able to go on from there!  If only there was a way to mess around with this now....

Potential Solutions

One possible solution that I may try out is networking multiple PCs together with a Kinect on each one.  This would most likely introduce quite a bit of lag however, but would still make for an interesting challenge to see how well it can be done.  Another idea would be to look into switch which Kinect is used each time you want to grab a frame.  This would most likely severely hinder the speed of the program, but would allow for a single computer to be used instead.  Ultimately, the best solution would be to get a computer that can handle this kind of task, with USB 3.0 ports, or to see if the Kinect 2.0 has resolved this issue in another way already.

Updates on 3-1-2014
There is yet another solution that I found recently that does seem to solve this!  If you have a laptop with an Expresscard port, adding additional USB slots there may allow for at least 1 more Kinect to be used.  Having recently installed some new USB 3.0 ports on my laptop, I found that this did indeed work and allowed me to use a second Kinect at the same time!

Voila! Two Kinects running simultaneously!

To those who are interested in the code, you can find the entire project over on my Github account here: https://github.com/gemisis/Multi_Kinect_Demo
It's a pretty simple project, and has very minimal error handling, but should be a good base to get started with multiple Kinects if you're interested.  Enjoy!

Any comments about this first tutorial would be most appreciated as well!