11/28/09
RotoCap and Microcontroller Applications
Good news. My thesis paper has been initially accepted by my committee. This has freed me to continue research in the fascinating world of human-computer interaction. The Arduino is an open-source microcontroller from Italy. It's cheap, well documented, and has numerous online communities each with their specific niches (diy-drones, music-production, art-installations). Above is my tinkering which will soon be connected to rotoCap to utilize its color-handling/vision functions.
9/15/09
How Color-Tracking Enhances Sony's PS3 Motion Controller
Sony's motion interface uses not only accelerometers like the Wii, but also uses the color emitted from the sphere to maintain tracking of different controllers. Since the sphere is a known dimension, Z-depth can also be extracted.
8/4/09
8/3/09
rotoCap Successfully Controls a Character
This is a test movie of my MFA thesis project, rotoCap. The person is repeating "Hello world" in different ways to drive the controllers of the 3D character ("Andy" by John Doublestein) only using three markers. The performance was shot on a scratch-built face-camera rig made from:
- discarded webcam
- used plastic 3D glasses
- steel bailing wire
- heat shrink wire wrap
5/14/09
Tracking fast movement and optimized centroid detection
I have updated the search method to handle motion blur and find centroids faster with greater accuracy. In this video I show how easy it is to setup targeting and simple key cleanup. The cleanup portion of the video will soon be automated and will be an option for the user to implement.
4/14/09
Arduino-based Motion Capture System
Another mocap project based on the Arduino microcontroller, from: http://blog.makezine.com/archive/2009/04/wireless_inertial_data_glove_using.html
Wireless Inertial Arduino Data Glove v0.3 from Noah Zerkin on Vimeo.
This is a quick vid of one of my later prototype glove units. The software is a proof-of-concept exercise. Regardless, you should be able to get the gist of what I'm trying to do. The thing that makes this special is how insanely inexpensive it was to build. The ultimate aim is to create a low-cost modular full-body interaction capture system for use in gaming, AR, and creative applications. The first commercial app I'm targeting for integration is Maya. After that, we'll look at integration with a game engine. I'm not sure what a glove/arm kit will cost, but it should be under $250 (Perhaps well under... we'll see... there are costs besides parts). Mass-production units would cost considerably less.
Wireless Inertial Arduino Data Glove v0.3 from Noah Zerkin on Vimeo.
This is a quick vid of one of my later prototype glove units. The software is a proof-of-concept exercise. Regardless, you should be able to get the gist of what I'm trying to do. The thing that makes this special is how insanely inexpensive it was to build. The ultimate aim is to create a low-cost modular full-body interaction capture system for use in gaming, AR, and creative applications. The first commercial app I'm targeting for integration is Maya. After that, we'll look at integration with a game engine. I'm not sure what a glove/arm kit will cost, but it should be under $250 (Perhaps well under... we'll see... there are costs besides parts). Mass-production units would cost considerably less.
3/25/09
Making rotoCap Data Useful
In order to be adaptable to each user's situation I have come up with a scheme to re-interpret data coming from rotoCap into a usable format. Motion data such as this is converted to keyable attributes which could be used to drive the user's own character rigs or anything possible via the connectionEditor. All this is done by essentially drawing the slider around the drone.
The image above shows the boundary box of the sphere's motion-extremes across the image surface. The dark triangles were some previous diagnostic graphics, ignore them. Scaling the green box redefines the parameter range of the drone (sphere) outputs by clamping the X and Y attributes. Additional number fields will be added to the drone's GUI section for North, South, East, West. Each field will define what the extreme value is for that direction or to ignore that parameter or clamp it to infinity. The user should be able to just plug in the extreme values from the control rig, blank fields are ignored:
Simulation
User Defined Drone Attributes
.85 would equal a slightly closed eye, or 85% open. The eye is parented to a null-dummy node which acts as a buffer between rotoCap and the user's rig just in case the user needs to remove portions or all of rotoCap from a scene which actually is the ultimate goal. rotoCap should work behind the curtains and leave as little impact to a scene beyond passing attributes in the form of a bare-bones node network.
The image above shows the boundary box of the sphere's motion-extremes across the image surface. The dark triangles were some previous diagnostic graphics, ignore them. Scaling the green box redefines the parameter range of the drone (sphere) outputs by clamping the X and Y attributes. Additional number fields will be added to the drone's GUI section for North, South, East, West. Each field will define what the extreme value is for that direction or to ignore that parameter or clamp it to infinity. The user should be able to just plug in the extreme values from the control rig, blank fields are ignored:
- yourFaceRig_eyeControlSlider.minY = 0.0 = drone.South
- yourFaceRig_eyeControlSlider.maxY = 1.0 = drone.North
Simulation
User Defined Drone Attributes
- Target Name = eyeControlSlider
- North = 1.0
- South = 0.0
- West = SKIP //ignore
- East = SKIP //ignore
.85 would equal a slightly closed eye, or 85% open. The eye is parented to a null-dummy node which acts as a buffer between rotoCap and the user's rig just in case the user needs to remove portions or all of rotoCap from a scene which actually is the ultimate goal. rotoCap should work behind the curtains and leave as little impact to a scene beyond passing attributes in the form of a bare-bones node network.
3/2/09
rotoCap Progress (Updated)
The past weekend has yielded many successes. After finally fixing a bug that wouldn't advance the frame properly I've fully re-engineered the core tracking engine and even ran some benchmark tests which will be posted shortly (updated, see below). The movie above shows the raw output, single pass with "Simplify Curves" command in from the Graph Editor, and a final hand-tweaked version. The raw output creates lots of key frames that aren't all necessary so by keeping the important key frames, this makes it much easier for the final manual cleanup due to the reduced key set.
Test Results & Observations
Environment
- 50 JPG image sequence
- 1 Drone sampled (Pink)
- timerX MEL function for timing
- Tolerance: (0.0 to 1.0) range of RGB value rotoCap accepts as close enough to desired color.
- Step: (0.0 to 1.0) distance to next sampled UV coordinate
- # of Keys: how many keyframes rotoCap created for a drone
- Time Elapsed: (seconds) starts when "Launch Drones" is pressed, ends after last drone is tested
- Tolerance .2
- Step .05
- # of Keys 7
- Time Elapsed = 101.47
Almost all of the keys occurred at the beginning while the target color was stationary. Tracking was not detecting motion over time. I suspected the step size was too large and rotoCap just couldn't sample enough UV's to find the drone.
Test 2: Smaller Step Size/Centering Off
- Tolerance .2
- Step .025
- # of Keys 16
- Time Elapsed = 1217.69
Reducing the step turned out to fix the sparse key issue. rotoCap still seemed to favor the edge of the color mass instead of centered which makes sense since this was run with self-centering turned off.
Test 3: Centering On
- Tolerance .2
- Step .025
- # of Keys 35 (simplified: 17)
- Time Elapsed = 71.86
The time difference is substantial due to the centering sub-process passing the centered UV coordinates to the next cycle of the colorSeek process. The key generation also went up significantly. Running a simplify curves on the XY curves in the Graph Editor essentially deletes the unnecessary stationary keyframes. I'm very happy how well the script is working now.
2/28/09
Alternate Search Modes
While working on rotoCap today I thought a nice feature would be allowing the user to manually launch rotoCap on a per-frame basis instead of a run of the whole image sequence. Additionally I am adding a flag to the keying sub-process whether or not to overwrite existing keyframes on a drone. If existing keyframes are found then the engine can use them as initial search coordinates, saving time by not having to start at the origin of a UV plane during the initial UV survey.
2/20/09
Homogenous RGB Mass Self-Centering
The colorSeek engine of rotoCap has been given some new methods for searching and evaluating masses of targeted RGB ranges. Instead of basing tracker position on a single pixel test, the new method treats a positive RGB test as the origin for a localized survey of the surrounding pixels. If it detects the pixels immediately surrounding it are within the RGB range it will expand the survey (patch) till it finds the edges of the color mass.
The engine then calculates the center of the mass by simply offsetting the target tracker half the size of the mass. This theoretically will eliminate any false positives due to the quality of the source image due to sensor noise and blurs due to fast movement. The centered coordinates are then passed on to the next frame's processing cycle to use as an origin for the new search.
The engine then calculates the center of the mass by simply offsetting the target tracker half the size of the mass. This theoretically will eliminate any false positives due to the quality of the source image due to sensor noise and blurs due to fast movement. The centered coordinates are then passed on to the next frame's processing cycle to use as an origin for the new search.
2/16/09
Break for Debugging
While not moving as quickly as I had hoped, the ideas I've been integrating have been encouraging. Though like any grand scheme, there comes a time to pause and make sure future development is on a stable foundation. For the past weekend I've been going back and fixing little bugs so I don't have to worry about them later. Though not as satisfying, it is essential to maintain a level of coding discipline.
1/19/09
Integrating Path Prediction
The core tracking engine has been successfully installed into the main program. It remains virtually unchanged since its creation but there are some adjustments to the search-style I'm working out currently. When the engine spots a target RBG value it uses that "hit" as the initial center for successive UV searches. Limiting the search area was the first attempt at speeding up processing and now predicting where the target is in the given area is the next step.
Noting Newton's First Law of Motion, a target will tend to continue along if not a straight line, a near approximation. The engine will make a list of coordinates to look at first in order of probability (the opposite direction being the last). This will require re-writing the engine but every engine needs an overhaul eventually. I believe the time I spend on this portion will be worth it in terms of process efficiency.
Noting Newton's First Law of Motion, a target will tend to continue along if not a straight line, a near approximation. The engine will make a list of coordinates to look at first in order of probability (the opposite direction being the last). This will require re-writing the engine but every engine needs an overhaul eventually. I believe the time I spend on this portion will be worth it in terms of process efficiency.
12/3/08
Drone Generation Subprocess "makeDrone"
I'm calling the visual representations of each color track drones. These, like worker bees and pollen, will track the RGB elements across the UV space. Currently the makeDrone subprocess creates:
- visible geometry
- RGB attributes from a colorEditor call
- unique object name
- colorSeek inclusion toggle (include in batch color tracking or not)
- RGB swatch via canvas command
- target-rig attribute to control (what does this drone control)
- "edit RGB" and delete options
- per-drone area constraints
- initial search pattern
- per-frame census (how many frames does drone find target RGB)
11/10/08
Over 100!
My method of file numbering is when I get a portion of the script running it gets saved as a separate file. This past weekend, the bugs squashed broke 100 (as of this writing at 103).
The GUI is being refined into a sleeker interface, but still with the consideration of an Adobe-like familiarity. Currently I'm working on a timeline-independent playback system for an animated texture. While scrubbing and stepping frames was relatively easy, the animation of the texture at a constant framerate is a little trickier. I've been using the timerX function to wait a certain amount of time then step the frame. It works but I'm trying to make it run consistently on different machines. The amount of times the function checks the elapsed time varies depending on the speed of the machine. I don't think this is a big issue, but I'm trying to make rotoCap as consistent and stable as I'd rather have all the processing power for the color-tracking engine than the GUI.
The GUI is being refined into a sleeker interface, but still with the consideration of an Adobe-like familiarity. Currently I'm working on a timeline-independent playback system for an animated texture. While scrubbing and stepping frames was relatively easy, the animation of the texture at a constant framerate is a little trickier. I've been using the timerX function to wait a certain amount of time then step the frame. It works but I'm trying to make it run consistently on different machines. The amount of times the function checks the elapsed time varies depending on the speed of the machine. I don't think this is a big issue, but I'm trying to make rotoCap as consistent and stable as I'd rather have all the processing power for the color-tracking engine than the GUI.
10/16/08
Simple Shape Tracking
Andy Wilson has developed an elegant control method based on the shape of the hand. By comparing the background shapes to newly created shapes (formed by the hand), he is able to track the movement allowing for mouse-less interactions. (Source: http://procrastineering.blogspot.com/2008/10/andy-wilson.html)
9/25/08
Video-Based vs Laser-Based Tracking of Dynamic Environments
In both fields of motion-capture and surface-capture (scanning) the need to discriminate unique points in space is paramount to the accuracy of the capture system. When I was first conceiving rotoCap I tried to figure out this problem by looking what I had to work with and not having ready access to the specialized cameras of a Vicon system or laser scanner I turned to consumer video cameras. I conceived I could lessen the amount of computer load by not having the program calculate trajectories but color. An infrared machine-vision camera is suited to higher resolutions and frame rates, how could the lowly camcorder compete? The answer is color.
Most machine vision cameras used in mocap and 3d scanning are grayscale in which their output is turned into a binary image. A binary image is simply a high-contrast image that is only black and white with no gradient. This makes the image easier to process for the software since it only deals with white dots on a black background, but the other calculations it uses to discriminate between two white dots which come near to each other can be complex and costly to the performance. By adding in the color value, the need to track each dot is made easier by giving each its own RGB value.
Why bring this up? The DARPA Grand Challenge is a great source for inspiration because of the real-world applications of scanning/mocap. The Stanford University team's Standley uses a hybrid system of laser scanning and a video overlay allowing it to use color to tell the difference between the road color and everything else. This is a great confirmation that we're both on the right track when it comes to using color over binary alone.
Most machine vision cameras used in mocap and 3d scanning are grayscale in which their output is turned into a binary image. A binary image is simply a high-contrast image that is only black and white with no gradient. This makes the image easier to process for the software since it only deals with white dots on a black background, but the other calculations it uses to discriminate between two white dots which come near to each other can be complex and costly to the performance. By adding in the color value, the need to track each dot is made easier by giving each its own RGB value.
Why bring this up? The DARPA Grand Challenge is a great source for inspiration because of the real-world applications of scanning/mocap. The Stanford University team's Standley uses a hybrid system of laser scanning and a video overlay allowing it to use color to tell the difference between the road color and everything else. This is a great confirmation that we're both on the right track when it comes to using color over binary alone.
9/14/08
Sequence Viewer and Shader Progress (UPDATE)
rotoCap is really coming along as more pieces of code unlock multiple areas of development. The past few weeks have seen a streamlining of file management, GUI revisions for better interactivity, and it's now ready to start processing images.
Since the core tracking engine relies on an image plane to calculate movement, rotoCap needs to:
Currently, the shader generator works great the first time the user loads up the image sequence into rotoCap but changing the image directory results in errors as the script tries to create new shaders connected to previous file nodes. The tricky part at this time is how to query if a shader exists since the shaderNode command does not have an -exists flag to test. To get around this I'm loading up all the shaders into an array and deleting the rotoCap-created shader network based on wildcard name-space search.
UPDATE
The shader handling is fixed now. The bug was due to old nodes conflicting with the newly generated nodes. Each time the user changes the image sequence directory a process is called to delete all nodes generated by rotoCap from previous executions. The nodes are filtered by a gmatch wildcard name-space execution, then deleted.
Since the core tracking engine relies on an image plane to calculate movement, rotoCap needs to:
- generate a shader with an animated texture
- disconnect the animated texture from timeline dependency
- connect animated texture to rotoCap control
- apply the shader to an image plane
- scale image plane the same aspect ratio as source image file
- create a camera to lock onto the image plane perpendicularly
- use the new camera's viewport as part of the rotoCap GUI
- create playback widget for different framerates
Currently, the shader generator works great the first time the user loads up the image sequence into rotoCap but changing the image directory results in errors as the script tries to create new shaders connected to previous file nodes. The tricky part at this time is how to query if a shader exists since the shaderNode command does not have an -exists flag to test. To get around this I'm loading up all the shaders into an array and deleting the rotoCap-created shader network based on wildcard name-space search.
UPDATE
The shader handling is fixed now. The bug was due to old nodes conflicting with the newly generated nodes. Each time the user changes the image sequence directory a process is called to delete all nodes generated by rotoCap from previous executions. The nodes are filtered by a gmatch wildcard name-space execution, then deleted.
8/20/08
3D Scanning for Animation
After attending SIGGRAPH this year and talking with Dr. Paul Debevec about the subject I am convinced 3D scanning will progress more and more into the realm of motion-capture where the fields will merge within 5 years. With Image Metrics' release of Emily, a 3D face tracked onto an actor's head, we see the current state of the synergy. So far, the use of 3D scan data in animation has been to capture key poses, this still leaves simulations to drive the actual motion. By capturing the full performance in 3D, the best of both worlds can be had by being able to use the animation uncut and or pulling key poses off choice portions.
Are we there yet? Not exactly. It has been pointed out that Emily was still animated by humans using the scan data to target a face rig. But, it is a good running start for the industry's jump over the valley.
Subscribe to:
Posts (Atom)