sfmflex-release

Public code for "Visualizing Spectral Bundle Adjustment Uncertainty", 3DV 2020.

This project is maintained by wilsonkl

Visualizing Spectral Bundle Adjustment Uncertainty

Kyle Wilson and Scott Wehrwein, 3DV 2020. (Poster, 9MB pdf)

You can find online interactive versions of our visualizations here.

Abstract

Bundle adjustment is the gold standard for refining solutions to geometric computer vision problems. This paper develops an uncertainty visualization technique for bundle adjustment solutions to Structure from Motion problems.

Propagating uncertainty through an optimization–from measurement uncertainties to uncertainties in the resulting parameter estimates–is well understood. However, the calculations involved fail numerically for real problems. Often we cope by considering only individual variances, but this ignores the important mutual dependencies between parameters. The dominant modes of uncertainty in most models are large motions involving nearly all parameters at once. These frequently look like flexions, stretchings, and bendings in the overall scene structure.

In this paper we present a numerically tractable method for computing dominant eigenvectors of the covariance of a Bundle Adjustment solution. We pay careful attention to the mismatched scales of rotational and translational parameters. Finally, we animate this spectral information. The resulting interactive visualizations (included in the supplemental) give insight into the quality and failure modes of a model. We hope that this work is a step towards broader uncertainty-aware computation for Structure from Motion.

Teaser Figure

Overview

This repository contains:

Visualizations

Live versions of the visualizations in our paper can be accessed by the links below:

These visualizations are included in the supplemental to our paper and in this repository. Please see the supplemental commentary for interpretive remarks.

Demo controls

Action Control
Toggle scene point visibility p
Rotate the model click + drag
Translate the model altclick + drag
Next/previous eigenvector j/k
Increase/decrease translation scale h/l
Increase/decrease rotation scale g/;
Help popup ?

How to run the demos locally

Please clone this repository. There are HTML files for each scene in the visualizations directory. Open these in any web browser (tested: Firefox, Chrome, Safari).

Research Code

The code pipeline works as follows:

The preprocessing stages are based on ceres-solver and its included bundle adjuster. This is in C++. The eigenvector computations are performed in Julia, and the final interactive visualization is based on ThreeJS in Javascript.

Setup and Building

The C++ targets depend on Eigen, GFlags, Glog, and hdf5. They also require ceres-solver, which should be built from source. Be sure to build ceres with suite-sparse.

For convenience on Ubuntu: apt-get install libeigen3-dev libgoogle-glog-dev libgflags-dev libhdf5-dev

Also, download the header-only library HighFive:

# from the sfmflex-release/code/ directory
mkdir lib && cd lib
git clone https://github.com/BlueBrain/HighFive.git

To build the C++ targets:

# from the sfmflex-release/code/ directory
mkdir build
cd build
cmake ..
make

The Julia code also has many dependencies. Install all of these by running julia deps.jl from the sfmflex-release/code/ directory.

Datasets

We have run our method on two datasets:

  1. Bundle Adjustment in the Large problems
  2. 1DSfM problems

A script is provided below to scrape and download all of the BAL problems. The 1DSfM problems are available as a single download from the 1DSfM project page.

Scripts and bins

Run all of these scripts from the sfmflex-release/code/ directory.

Pipeline scripts: These script don’t take CLI options—edit their options in the file.

Dataset scripts: These are useful for downloading problem instances and converting between file formats.

Preprocessing scripts: These scripts are used within the pipeline scripts.

Viewing the Visualizations

To view the interactive visualizations, first run one of the pipeline scripts above to get a json file containing the scene parameters and the computed covariance eigenvectors. Then edit sfmflex-release/code/vis/vis_supp.html (line 43) to include that json file. (The include path is relative to the vis_supp.html file.) Now open vis_supp.html in any web browser.

File Formats

BAL problems The Bundle Adjustment in the Large datasets use the same camera model as Bundler. See the BAL project page for details. Notice that unlike Bundler, the 2D coordinate system has its origin at the image center.

The BAL (and Bundler) cameras map 3D world coordinates X into 3D camera-centered coordinates Y via Y = R*X + t.

The camera parameters are ordered as:

r1, r2, r3, t1, t2, t3, f, k1, k2

Jacobian text files (Some older code stores Jacobians in text files. This has been mostly replaced by HDF5 files for performance.)

The first line of each file gives dimensions:

num_cameras num_points num_observations

The remainder of the file gives the Jacobian in (i, j, value) sparse triplets.

The Jacobian has dimensions (2*num_residuals) x (9*num_cameras + 3*num_points). The cameras are ordered first, and then the points, using the indexing of the source problem.

For now, no robust loss is applied when computing the Jacobian. This seems like an oversight that we’ll have to come back to.

Bundle Adjustment problem HDF5 files See jl/JacobianUtils.jl:readproblem() for details about how a Bundle Adjustment problem is stored in an HDF5 file. Both a solution and its Jacobian, along with some other metadata, are all stored in binary format.

Visualization JSON files We use json files to hold a scene + eigenvectors description in a way that is easy to read into our WebGL visualizer. See scripts/generate_json_for_vis.jl for a sample script that takes a BAL problem and outputs a vis json.

These json files are dictionaries at the base level containing the following fields:

To make it easy to read into JavaScript, the base level dictionary has a name:

sample_scene = {...}

(yes, this is hacky…)