We have come a long way until the final show day. We were trying everything technology-wise from using a webcam and OpenCV library, to Processing Flob library without any success in getting the clean data. At the end we were successful at using a Kinect to get the depth data. Since we didn’t have much time to work on it, we decided to change our setup scene so that it would detect the current position of the users, and draw triangle shapes for each step they move.
The idea is to create a game where Jessica can challenge herself to create an aesthetic view of the scene like a kaleidoscope. Check out our video recording!!
It will be a miracle if this works tonight.
We were still having issues with Flob getting random reading despite all efforts. Last Thursday we decided to move the project indoor but it couldn’t solve the issue with inconsistent reading.
Like what I predicted, Flob library cannot track users well enough even after we put in a “hack” to look for possibly identical blobs existed in 10 previous frame (this is to pick up blobs that somehow get lost.)
Now, with just one day left, I have no hope in making this work.
We talked to Professor Hwaryoung about this issue. She said we could simplify our goal. So, we are going to make it just track only one person, using the depth data from the Kinect.
Hopefully we can make something presentable after tonight.
In the past week, we’ve been working on improving the performance of Flob library for the user tracking piece. We also had the permission to get into the room 211 in Glasscock, tried out the webcam and the projector’s quality.
Then, I was working on the color tracking part. It is to differentiate Jessica from the rest of the audience (people who walk by), and here’s the code:
https://dl.dropboxusercontent.com/u/84358699/ColorTracking.pde
It works really well in detecting movement of objects with skin tone color. Hue threshold can be adjust to make it detect different color. However, since it doesn’t fit the group’s end-goal of tracking objects of ANY color, this program will be nothing more than a little project of my own.
The following video is a demonstration of how it works. It can detect my face when the face pops up in the scene, and follows the face.
However, on a brighter side note, although what I have gotten so far is subtle, but still an achievement. I have learned a lot about the histogram, color theory, threshold, background subtraction technique, meanshift algorithm, back projection, etc. Those are the things that I had not known before. I will definitely revisit this little program during summer for any possible improvement.
So, I talked to my group about the alternative option (track users based on some specific colors instead of covering all colors), but that idea was not supported.
Kaitlyn suggested to go back to Processing (Java-based), using a library called Flob. We tried it this evening in the real condition, it didn’t track people well enough.
1. Users are easily got lost in between frames if they walk too slow or remain at the same position. (The reason for that is because Flob does background subtraction from the previous frame, so if there is no, or little movement, it won’t be able to see the difference between frames.)
2. Another issue with it is in low light, we had to lower the threshold of the binary image so that the program can actually detect the difference. However, by doing that, we also allowed it to pick up even the lightest movement of other arbitrary objects like the bushes, the grass, and so on.
3. Also, when people walk too close to each other, their blobs are merged (hopefully this is just a matter of changing a settings value). However, when they separate again, they get reset to new IDs.
I think we might not have a working final project if we don’t tweak our end goal. I failed to persuade my group to do so. And I can’t use what I have got so far in C++ with OpenCV. Frustration.
I spent 5 days straight working on getting the user tracking data using C++ and OpenCV.
My goal: be able to track multiple people from the time they enter the scene to when they exit the scene.
Techniques:
1. Background subtraction utilizing BackgroundSubtractorMOG2 class in OpenCV to get the initial box to track.
2. MeanShift algorithm via CamShift function in OpenCV to find the highest possible position of a given initial box in the next frame.
However, with Meanshift/CamShift:
– It’s color-based detection, so it will try to match the given region of interest to the closest possible area that has a similar color. Therefore, if the scene already has some other objects in the same color as the object we’re tracking, the algorithm may give false result by returning the current position of the other object instead.
– It only work well when I give it a limited range of hue, for example red only, blue only, or so forth. If I extend this range of hue to cover all colors, it immediately gives false results, because the scene ALWAYS has some object in the same color as the object I’m tracking.
My knowledge of OpenCV is limited. In the past week, I’ve been researching everywhere to see besides color-based detection, what else MeanShift can do. However, all the examples I found are color-based.
I worked with Peizhao 5 hours on Sunday (Thank you so much, Peizhao!!) but could not find any other solution.
Alternative solution I can think of right now: Tweak our end goal so that the program will track users based on some specific color instead of covering all possible colors.
Stable Multi-Target Tracking in Real-Time Surveillance Video
Click to access benfold_reid_cvpr2011.pdf
Interactive Performance and Technology 