Interactive Computational Imaging for Deformable Object Analysis (aka Project Squishy)
We describe an interactive approach for visual object analysis which exploits the ability of a robot to manipulate its environment. Knowledge of objects' mechanical properties is important in a host of robotics tasks, but their measurement can be impractical due to perceptual or mechanical limitations. By applying a periodic stimulus and matched video filtering and analysis pipeline, we show that even stiff, fragile, or low-texture objects can be distinguished based on their mechanical behaviours. We construct a novel, linear filter exploiting periodicity of the stimulus to reduce noise, enhance contrast, and amplify motion by a selectable gain - the proposed filter is significantly simpler than previous approaches to motion amplification. We further propose a set of statistics based on dense optical flow derived from the filtered video, and demonstrate visual object analysis based on these statistics for objects offering low contrast and limited deflection. Finally, we analyze 7 object types over 59 trials under varying illumination and pose, demonstrating that objects are linearly distinguishable under this approach, and establish the viability of estimating fluid level in a cup from the same statistics.
Publications
• D. G. Dansereau, S. P. N. Singh, and J. Leitner, “Interactive computational imaging for deformable object analysis,” in Robotics and Automation (ICRA), 2016. Available here.
Collaborators
This work was a collaboration between Donald Dansereau and Juxi Leitner at the Australian Centre for Robotic Vision and Surya Singh from the Robotics Design Laboratory at the University of Queensland.
Acknowledgments
This research was partly supported by the Australian Research Council
(ARC) Centre of Excellence for Robotic Vision (CE140100016) and the
ARC Biomechanics Meets Robotics Project (DP160100714).
Themes
Gallery
(click to enlarge)
An overview of the method: The robot applies a time-varying stimulus d to a compliant object while observing the video sequence x. The video is filtered to enhance contrast and motion, allowing fragile, stiff and low-contrast objects to be analyzed. The optical flow of the filtered sequence is decomposed into orthogonal components which yield a set of flow statistics. These in turn drive object and material discriminations, and ultimately decisions on manipulation strategy. Feedback to d indicates a potential to modify the stimulus to yield more information, e.g. starting gently and increasing as appropriate.
With the novel, low-complexity linear filter, the motion in the cup is sufficiently amplified to allow optical flow analysis.
The filter is a single-stage linear filter that treats the video as a 3D block of data. It passes periodic motion (A) matching the stimulus while rejecting spatial motion that's too fast, e.g. noise (B) or too slow, e.g. robot arm drift (C). The combination of passband and stopband components creates a single-stage filter that rejects noise and arm drift and amplifies motion.
Dense optical flow over the stimulated object forms the basis for several visual material properties. Here the object is being squeezed horizontally with a gripper near the top of the frame, and motion has been amplified to allow optical flow analysis.
Optical flow is decomposed into perpendicular and parallel components, relative to a point and the axis of compression. In blue are the instantaneous values, and in red the running mean. Different materials show different ratios of these values, independent of texture and colour. The visual compressibility, for example, is akin to the Poisson's ratio, and allows stiff and soft objects to be distinguished.
Observing from the side while squeezing 7 different household objects over 59 videos, with pose and illumination variation.
PCA on the flow shape statistics allows liner discrimination of objects on the basis of their mechanical properties. No texture, brightness or colour information is used. This establishes the significance of the mechanical information extracted by the interactive technique.
The "Visual Compressibility Ratio", akin to the Poisson's Ratio, is particularly promising as a summary statistic of the mechanical object properties. The visual compressibility ratio for cup type C2 over five fill levels, over two trials, reveals a strong potential to use this statistic for estimating the fill level of an opaque cup while observing it from the side.
Optical flow for a compliant object. Here the gripper is squeezing horizontally near the vertical center of the frame.