Spherical Track

Usage  |  Controls

The Spherical Track node can be used to track features in an equirectangular spherical panorama and solve for the camera motion. The node has a single input, which must contain a 360 degree equirectangular image with a resolution twice as wide as it is tall. The spherical tracker node generates six output cameras, one for each planar face of a cube map generated from the input panorama.

An equirectangular panorama can be obtained by stitching together multiple images, although this must be done using external software, and the spherical panorama imported into PFTrack as a single clip.

Note that the quality of the overall camera track will depend significantly on the accuracy of the equirectangular panorama. Image deformations and mis-alignments caused by inaccurate stitching or parallax will affect the quality of the camera track, and in some severe cases prevent the camera being tracked at all.

Usage

Tracking the camera motion for a 360 degree equirectangular panorama is achieved in two stages: First of all features must be tracked throughout the clip. Then, the feature tracks are used to estimate the overall motion of the camera.

In order to achieve an accurate camera track, it is important to ensure that the equirectangular panorama is free from image deformations and mis-alignments caused by stitching. Parallax effects that are created when the cameras used to capture the panorama do not have a coincident centre of projection can also caused problems when tracking, so these should be minimised as much as possible during the stitching process.

Tracking features

It is important to mask out any moving objects during tracking, including the rig on which the camera is mounted, or the person carrying the camera. This can be achieved by drawing a mask around the offending area using any of the mask nodes. A region of interest (ROI) can also be used to quickly specify a reduced area for feature tracking if required.

Once tracking has finished, features are displayed in the feature list, along with the start and end frame in which they were tracked. Selected features are displayed in purple in the Cinema window.

Cleaning feature tracks

The Marquee button can be used to select features in the Cinema window. With this button enabled, clicking and dragging with the left mouse button in the Cinema window will draw a selection rectangle. Holding the Shift key will allow a lasso to be drawn instead of a rectangle.

Selected features can be deleted to remove features that have not been tracked accurately, or features that have been placed on moving objects. Often, it can also help to remove features that have been tracked on very distant points such as clouds in the sky if they are moving significantly over the course of the clip.

Instead of deleting features entirely, the Remove, R-, and R+ buttons can be used to remove feature tracks from specific frames. Generally, you should attempt to ensure that the majority of feature tracks look accurate and appear to be tracked on a static object in the scene.

Solving for camera motion

Once features have been tracked, the camera motion can be solved. The two parameters affecting this process are the Inlier Threshold and Camera Smoothing.

Selecting a High inlier threshold will tolerate larger tracking errors whilst attempting to improve the overall camera path. This can be useful in situations where the equirectangular panorama being tracked contains stitching or parallax errors that otherwise would prevent the camera motion from being solved. Selecting High camera smoothing means that the camera motion will be smoothed more during the solve, but this may also increase the overall tracking error.

Once the solve has finished, the projection solved feature points will be displayed in the Cinema and perspective/orthographic viewer windows. Features are colour coded according to their distance from the tracked positions. Feature points that are below the inlier threshold are coloured green, whilst those above the threshold are coloured red. Features that were not tracked in the current frame are displayed in white.

The current camera path and feature positions can be refined. to reduce the overall error. This can be used after deleting poorly solved features, or adjusting the Inlier Threshold or Camera Smoothing parameters.

Once the camera has been solved, the overall orientation of the scene can be quickly set by specifying the origin point and up direction.

Output

The Spherical Track node generates six output cameras, each corresponding to a different face of a cube map centred at the location of the spherical camera. These cameras can be passed down-stream for further processing, for example, by attaching a Test Object node to introduce test objects into the scene, and then to an Scene Export node where all six cameras and corresponding image data can be exported.

Using another camera solver

In some situations, it may be desirable to use another node to track one of the output cameras. For example, if the focus of a matchmoving task is one small area of the equirectangular image, an additional node such as the Camera Solver or Survey Solver can be used.

In this case, the Spherical Track node is used simply to cut the equirectangular image into six planar cameras, and not for the actual feature tracking and camera solving task. Once one of the six cameras is tracked using other means, the Spherical Orient node can be used to re-assemble the other five cameras, and the Spherical Render node can then be used to bring the results back to an equirectangular panorama.

Controls

Feature Tracking

From Frame: Set the From Frame to the current frame in the Cinema window. The From Frame is the first frame for feature tracking. If the edit box is empty, feature tracking will start at the first frame of the clip.

To Frame: Set the To Frame to the current frame in the Cinema window. The To Frame is the last frame for feature tracking. If the edit box is empty, feature tracking will end at the last frame of the clip.

Pick Threshold: The threshold for feature picking. Increasing this value will mean that only strong image features are selected for tracking, and areas of the image that do not contain significant feature points are ignored. The default value is 1.00

P: Display a preview of selected features points in the current frame using the Pick Threshold.

Fail Threshold %: The threshold at which feature tracks are considered to fail. This is generally caused by the image pixels around the feature point changing significantly, which signifies that the feature point has failed to track correctly. The default value is 30%.

Backtrack: When enabled, feature tracking will be performed in both the forwards and backwards directions. This can often extend the lifetime of feature tracks which will help provide a better overall camera track.

Consistency check: When enabled, feature tracks that do not match the overall motion of the camera will be rejected. This will can often help provide a better overall camera track. Note that the consistency check assumes an accurate equirectangular panorama. If the input clip contains significant stitching or parallax errors then this option can be switched off in order to increase the lifetime of feature tracks.

Edit ROI: Allow the region of interest (ROI) to be adjusted in the Cinema by dragging its edges with the left mouse button.

Start: Start the feature tracking process. Holding the Shift key whilst clicking this button will perform feature tracking as a background process.

Feature List

Marquee: Allow a selection marquee to be drawn using the left mouse button in the Cinema window (for tracking point selection) or the perspective/orthographic viewer window (for solved point selection). Holding the Ctrl key whilst drawing will ensure that previous selections are kept. Holding the Shift key will use a lasso selection instead of a rectangle.

R-, Remove and R+: Remove a feature from frames in the clip. The R- will remove selected features from all frames before the current frame, and R+ will remove from all frames after the current frame. Remove will remove features from only the current frame. These buttons can be used to remove a feature from frames where it has not been tracked accurately.

Delete: Delete all selected features.

Spherical Camera Solver

Inlier Threshold: The inlier threshold level. The options are Low, Normal and High. Selecting a High inlier threshold will tolerate larger tracking errors whilst attempting to improve the overall camera path. This can be useful in situations where the equirectangular panorama being tracked contains stitching or parallax errors that otherwise would prevent the camera motion from being solved.

Camera Smoothing: The amount of smoothing to apply to the camera translation during the solve. The options are None, Low, Medium and High. Selecting High means that the camera motion will be smoothed more during the solve. Note that smoothing the camera motion may also increase the overall tracking error.

Start: Start the camera solver process. Holding the Shift key whilst clicking will run the camera solver in the background.

Refine: Refine the current camera path and tracker locations. This can be used to improve the overall solution after editing or deleting trackers, or changing the inlier threshold or camera smoothing parameters, without needing to start the solver from scratch.

Orientation

Orient Ground: Adjust the position and orientation of the ground plane to match selected feature points in 3D space. Options are X-Y Plane, X-Z Plane and Y-Z Plane. The PFTrack ground plane lies on the X and Z axes, so to position the ground plane to selected features, the X-Z Plane option should be selected. At least three features must be selected in order to orient the ground plane.

Set Origin: Snap the origin of the ground plane to the average of all selected feature points.

Display

Show All Trackers: When enabled, all solved features will be displayed in the Cinema window, regardless of whether they have been tracked in the current frame or not. Disabling this option will only show features that have been tracked in the current frame.

Show Projections: When enabled, the projection of solved feature points will be displayed in the Cinema window, colour coded according to their distance from the tracked position. Feature points that are below the inlier threshold are coloured green, whilst those above the threshold are coloured red. Features that were not tracked in the current frame are displayed in white.

Show Ground: Toggle display of the ground plane in the Cinema window.

Show Horizon: Toggle display of the horizon line in the Cinema window.

Planar Camera Output

Output Size: The resolution of the planar cameras that are output by the Spherical Tracker node. By default this is set to the vertical height of the equirectangular panorama that was tracked.

Multi-Sample: The amount of multi-sampling used to generate each planar output camera image. Increasing this value will reduce aliasing, but also increase the time required to render each output frame.

Default Keyboard Shortcuts

Keyboard shortcuts can be customised in the Preferences.

From Frame	Shift+F
To Frame	Shift+G
Edit ROI	Shift+E
Remove Before	[
Remove	Ctrl+R
Remove After	]
Delete	Delete
Marquee	Shift+M
Start Tracking	Shift+T
Start Solving	Shift+S
Refine	Shift+R
Set Origin	Shift+O
Show All	Ctrl+A
Show Projections	Ctrl+P
Show Ground	Ctrl+G
Show Horizon	Ctrl+H