BuFF Burst Feature Finder

A commercial drone (top-left) captures imagery that is too noisy for conventional 3D reconstruction in low light (top-middle). This is because of the high rate of spurious feature detection and low-quality feature matches offered by conventional features like SIFT (top-right and bottom, red). The proposed BuFF feature yields fewer higher-quality features resulting in many more correctly matched pairs (yellow). In this work we show the BuFF feature enables 3D reconstruction in previously prohibitive low light conditions.

We introduce BuFF, a 2D + time feature detector and descriptor that finds features with well defined scale and apparent motion within a burst of frames
We propose the approximation of apparent feature motion as either 1D or 2D linear segments under typical robotic platform dynamics, enabling critical refinements relative to prior work on hand-held imagery
We establish variations of BuFF matched to these apparent motion types and demonstrate it significantly outperforming 2D feature detectors applied to conventional and burst imagery in low-SNR scenes.

This work opens the way for a broad range of applications in which low light commonly complicates vision such as delivery drones, mining and behavioural studies on nocturnal animals.

Publications

• A. Ravendran, M. Bryson, and D. G. Dansereau, “BuFF: Burst feature finder for light-constrained 3D reconstruction,” Robotics and Automation Letters (RA-L) and Conference on Robotics and Automation (ICRA), 2024. Preprint available here.

This work was carried out within the Robotic Imaging Group at the Sydney Institute for Robotics and Intelligent Systems, Australian Centre for Field Robotics, University of Sydney.

Themes

Robust Vision, Algorithms and Architectures

Downloads

The code is on GitHub here.

The dataset used for validation is available here (83 GB download).

The dataset was captured with a UR5e robotic arm-mounted monocular camera, and contains 20 bursts of 10 images each. We provide light-constrained bursts and corresponding ground truth bursts with 1D and 2D apparent motion between frames.

Gallery

(click to enlarge)

In 3D scenes under general 6 degree-of-freedom platform motion, features in a burst exhibit apparent motion that is well approximated by line segments which we exploit to find BuFF features.

In the special case (right) of a platform moving orthogonal to the principal axis of the camera, the apparent motion follows parallel line segments.

An input burst is converted to a motion stack by shifting and summing frames across a range of putative apparent motions.

Each frame of the motion stack passes through a difference-of-Gaussian scale-space filter, and features are detected as extrema in the resulting joint scale-slope search space. Each feature is described using a SIFT-style histogram of gradients applied to its corresponding motion-stack image, rather than the input frames.

The resulting features have distinct location, scale and apparent motion (slope), and exhibit high precision, recall, and matching performance.

BuFF 1D admits fewer spurious features because of the smaller search space, and both variants of BuFF exhibit higher true positive rates.

We use the ROC curves to select comparable peak thresholds for each method, such that 10% of detected features are false positives.

Feature performance in noise: Validation using a test target scene of varying scale. SIFT finds features well under bright illumination, but suffers from many false positives in low light.

Employing conventional burst imaging prior to applying SIFT improves performance, but BuFF 2D and 1D show much greater performance due to the joint scale-motion search.

Average SfM reconstruction performance for light-constrained scenes. With COLMAP's default settings, our approach outperforms conventional images and burst-merged images in all measures.

Our method yielded more accurate and reconstruction with high match score, match ratio and precision overall.

Conventional imagery and burst-merged images yield poor camera pose estimates. The proposed BuFF variants successfully employed the most images and yielded the most accurate pose estimates,

Mean rotational and translation error in camera pose estimation. Our approach outperforms conventional image capture and burst-merged images.

Citing

If you find this work useful please cite

@article{ravendran2024buff,
  title={{BuFF}: Burst Feature Finder for Light-Constrained {3D} Reconstruction},
  author={Ravendran, Ahalya and Bryson, Mitch and Dansereau, Donald G.},
  journal={Robotics and Automation Letters ({RA-L}) and Conference on Robotics and Automation ({ICRA})},
  year={2024}
}