Auto Track

Usage  |  Example ASCII tracker text file used for import and export

The Auto Track node can be used to automatically generate trackers to use for camera or object solving, and has a single input and output.

In contrast to the Auto Match node, the Auto Track node attempts to pick likely points in a frame and track each one into the next frame, whereas the Auto Match node attempts to pick likely points in each frame and match them together.

Tracking groups

All trackers generated by a single Auto Track node are placed into a tracking group. By default, this group is called Camera and corresponds to tracking the overall camera motion. When tracking an object, the group should be changed to something representative of the object being tracked (for example, "Car" or "Head").

Usage

Clicking the Auto Track button start the tracking process, generating a number of candidate trackers for each frame, tracking these throughout the clip, and then selecting the best of these trackers to pass-down stream.

The number of candidates is specified using the Candidate number parameter. The Target number parameter controls how many of these candidates will be kept. The life-time and spatial distribution of each candidate tracker is used to decide which to keep and which to reject.

Candidate trackers can be tracked forwards and backwards through the clip. The backwards tracking pass is used to extend the life-time of each tracker by tracking backwards from the frame in which it was first introduced during the forward pass.

Masks can be generated to restrict automatic feature tracking to a specific part of the image. This is essential when the clip contains objects moving independently from the camera, and trackers are being generated for either camera or object tracking.

To mask feature tracking, connect either a Roto or Keyer node or an Image-based mask to the Auto Match node as described in the Workpage section of this documentation.

Tracking and search windows

Trackers have a rectangular window area that contains the image data to be tracked between frames, and a larger rectangular search area that shows the area that will be searched for the best matching window pixels. The tracking window size should be set so it neatly surrounds typical image features. Having a tracking window that is too large compared to the feature it is tracking may cause tracking to fail.

The size of both the tracking window and search area can directly affect both the accuracy and performance of the tracking algorithm. If the tracking window is too large compared to the feature being tracked (i.e. it contains a lot of pixels that are very similar to the surrounding image area, and not unique to the feature) then the feature may not be tracked accurately, or the time taken to track the feature may increase. However, if the tracking window is too small, it can become difficult to estimate motion from one frame to the next.

The screenshot below shows a search window (the outer square, displayed with short dots) and a tracking window (the inner square, displayed with longer dashes) positioned around a small feature. The tracking window on the left is too large, because the feature being tracked is only occupying a small area in the centre of the window. The window on the right is better, because it provides a tighter bound around the image feature.

Tracking Groups and Presets

Create a new tracking group.

Current group: Specify the group that new tracking points will be associated with. Groups can be renamed by typing into the menu box.

Delete the current tracking group.

Store current parameters as a new preset.

Current preset: Current preset for the matching parameters. The current preset name can be changed by typing into the menu box.

Delete the current preset.

Preset files are stored as XML data in the user's user's presets/auto_track directory.

Tracking Window

The Tracking Window displayed the image pattern for the first selected tracker, along with the current x and y tracker position and matching score. A matching score of 1.0 indicates a perfect match of the tracker's reference pattern and the pattern in the current frame. Tracker positions can be adjusted interactively in this window.

Trackers

The tracker list displays all trackers that have been generated by the node or fetched from other nodes upstream.

The tracker list contains all trackers that have been generated, along with their in and out frames. Trackers can be renamed in the list and selected trackers are also highlighted in the Cinema window.

At the top-right of the panel are tools to control the visibility of tracking points in individual frames:

R-: Remove selected trackers from all frames before the current frame.

Remove: Remove selected trackers from the current frame.

R-: Remove selected trackers from all frames after the current frame.

Hide / Show: Hide or show the selected trackers in the current frame. In contrast to the Remove button, hiding a tracker will prevent the tracker from contributing to any camera solve in the current frame, instead of just removing the keyframe position.

Tracking Parameters

The following parameters are available for adjusting tracking behaviour:

Channels: Image and mask channels to use to match reference patterns.

Candidate number: The number of candidate trackers to generate for each frame.

Target number: The number of trackers to keep, after removing short-lived trackers from the set of candidates.

Window size: The size of tracker windows in pixels. Larger windows can often be tracked more easily, but take longer to process and can introduce drift if the contents of the window is changing significantly from frame to frame.

Search range: The width and height of the search range in pixels to use for the initial pattern search. This option is only available when the search mode is set to either Better Speed or Better Accuracy. Note that the search area defines the possible locations of the centre-point of the tracker, rather than the entire tracking window itself.

Pick threshold %: The threshold used to identify new tracker positions in the range zero to one. New trackers are positioned at corners and significant image gradients. The higher the threshold, the stronger the corner or image gradients need to be before a tracker is created. Setting this to a very small value will position trackers on weak corners where they may not track well.

Min length: The minimum length of a tracker (measured in number of frames) that a tracker must exist for before it is added to the tracker list. Increasing this value will mean new trackers are rejected if they do not track over the specified number of frames.

Failure threshold: The minimum matching score allowable for new trackers. During tracking, a tracker will be rejected if its matching score falls below this value. The matching score for a tracker is displayed in the tracking window.

From: Set the first frame to use for automatic tracking to the current frame number. The corresponding frame number is displayed in the edit box. Reset with R button.

To: Set the last frame to use for automatic tracking to the current frame number. The corresponding frame number is displayed in the edit box. Reset with R button.

Search mode: The algorithm used to perform tracking. Better Speed and Better Accuracy will perform a search over the area of pixels defined by Search Range to find the most likely match for the reference pattern, with the latter giving more reliable results at the cost of more processing time. The final option is Optical Flow, which will use motion analysis to assist with the search. This can often increase the reliability of feature tracking. In this case, the Search Range options are not available. Motion analysis is GPU accelerated on supporting platforms.

Prediction: The method used to predict tracker motion from one frame to the next. The default value is None, indicating there is no prediction. When set to Image Motion, the average motion exhibited over the 2D image is used to predict how individual trackers will move from one frame to the next. When set to From Metadata, the metadata describing approximate camera motion is used to predict how trackers move from one frame to the next. Both these predictions are used to guess the starting location for each search during the tracking process.

Image proxy: Specify the image resolution to use for tracking. The default value (None) will track the image at full resolution. Other options are Half, Third and Quarter, and can be used to track the image at half, third or quarter resolution. This can increase tracking speed, but may also mean that small features are not tracked as accurately if there is significant motion in the image.

Feature scale: Specify the scale of features expected in the image. When set to Large, only corners and image gradients that are spread over a large area will be examined to see if a significant feature exists. Using a Small feature scale (the default value) can sometime mean noise or image grain is mistaken for a significant corner, especially when using high resolution images.

Deformation: Allowable deformation of the pattern windows over time during tracking. By enabling Rotate, Scale and/or Skew, tracking windows can warp and deform in order to better match the reference pattern. This will slightly increase the time it takes to track, but can often mean that trackers exist for longer before their matching score drops below the failure threshold.

Consistency: The expected overall tracker motion in the image. The default value (Local Motion) will reject trackers whose motion is inconsistent with the overall motion of the scene near the tracker. Free Camera will reject trackers whose motion is not consistent with a freely moving camera viewing a static scene. Rotation/Planar will reject trackers whose motion is not consistent with that of a rotating camera viewing a static scene, or a moving camera viewing a planar scene.

Back-track: Track in two passes: the first is forwards from the start of the clip to the end, and the second is backwards, from the end of the clip back to the start. During the backwards pass, no new trackers will be included, but those generated during the forwards pass will be tracked backwards from the frame in which they were first created. This has the effect of increasing the average length of a tracker.

Spread: Spread newly created trackers around the image, rather than being positioned only at the highest-rated corners and image gradients. This has the benefit of ensuring that trackers are evenly distributed around the image, which can help improve the reliability of the Camera Solver. This comes at the cost of sometimes choosing positions that cannot be tracked as reliably.

Guide: Trackers generated by a previous User Track node can be used as guides to assist with the consistency check. These trackers will be displayed in the Cinema window in purple and can help the consistency test identify the desired motion of the scene or object. Any number of guide trackers can be used, but two or more will produce better results than a single guide tracker.

Blur image: Blur image data slightly before tracking. This has the benefit of limiting the effect that image noise can have on reducing the overall matching score.

Illumination: Take into account global image brightness when tracking, so effects such as flicker or lighting changes do not reduce the overall matching score as much.

Reset: Reset the tracking parameters back to their default values.

Consistency Checks

The Consistency parameter specifies type of motion that the trackers are expected to follow relative to the image. When viewing a static scene through a free-moving camera, this can be set to Free Camera which uses a pin-hole camera model to reject trackers that are moving incorrectly. For a camera that is only rotating (or one that is rotating a very small amount relative to the distance from the camera to the tracking points, or viewing a planar scene), choose Rotation/Planar.

Note that in some cases where there is significant lens distortion, the Free Camera and Rotation/Planar models may not model tracker motion well enough.

Additional problems may arise when the Free Camera model is used for a scene that contains significant lines that are parallel to the direction of motion (such as horizontal lines viewed by a horizontally translating camera). In these cases, the Local Motion mode (which is the default option) should be used which will reject feature tracks that are inconsistent with the overall motion of the scene around the feature point.

When using a consistency check to help reject bad trackers, trackers generated by a previous User Track node and passed into the Auto Track node can be used as guides to help determine the correct motion. These trackers will be displayed in the Cinema window in purple and can help the consistency test identify the desired motion of the scene or object. Any number of guide trackers can be used, but two or more will produce better results than a single guide tracker.

Errors Graph

The error graph plots the tracking error (measured in pixels) for each tracker in each frame, along with the average tracking error for all trackers visible in a frame. The tracking error is 0.0 when the matching score for a tracker is 1.0, indicating a perfect match. Selected trackers are shown in yellow, and unselected trackers are shown in blue. The average tracking error graph is shown in white.

The graph can be panned or zoomed using the right or middle mouse buttons respectively. Clicking on a tracker line will select a tracker from the graph. Holding the Ctrl key will allow multiple trackers to be selected at the same time.

Show All: Error graphs will be shown for all trackers, otherwise a graph will only be shown for selected trackers.

Trim: Display a trim line, allowing all trackers whose errors are larger than a particular value to be removed. Trackers that have been trimmed in a particular frame are shown in red. The trim line can be moved up and down by dragging it with the mouse. Once activated trimming will remain active even if the Trim button is disabled - the Trim button only controls whether the trim line is displayed or not; not whether trimming itself is active.

D: Disable trimming (only available when the trim line is not displayed).

Edit Trim Curve: Toggles between moving the trim line as a whole and editing the shape of the trim line to allow more flexible trimming where a single value for the entire sequence is not sufficient.

R: Resets the shape of the trim line.

Fit View: Scale and translate the error graph so all graphs are visible.

Acceleration Graph

The acceleration graph plots the acceleration of each tracker in each frame, along with the average acceleration for all trackers visible in a frame. This can be used to spot frames where a tracker jumps incorrectly from one location to another by looking at the difference between the acceleration for a particular tracker and the average acceleration for all trackers in the frame. Selected trackers are shown in yellow, and unselected trackers are shown in blue. The average acceleration is shown in white.

The graph can be panned or zoomed using the right or middle mouse buttons respectively. Clicking on a tracker line will select a tracker from the graph. Holding the Ctrl key will allow multiple trackers to be selected at the same time.

Show All: Acceleration graphs will be shown for all trackers, otherwise a graph will only be shown for selected trackers.

Trim: Display a trim line, allowing all trackers whose accelerations are larger than a particular value to be removed. Trackers that have been trimmed in a particular frame are shown in red. The trim line can be moved up and down by dragging it with the mouse. Once activated trimming will remain active even if the Trim button is disabled - the Trim button only controls whether the trim line is displayed or not; not whether trimming itself is active.

D: Disable trimming (only available when the trim line is not displayed).

Edit Trim Curve: Toggles between moving the trim line as a whole and editing the shape of the trim line to allow more flexible trimming where a single value for the entire sequence is not sufficient.

R: Resets the shape of the trim line.

Fit View: Scale and translate the error graph so all graphs are visible.

The Coverage Panel

The Coverage Panel displays information about the frames in which each tracking point has been tracked:

This can be used to evaluate how well tracking points are distributed throughout the clip, which will help to provide an accurate camera solve without any jumps in the camera path.

The Coverage Panel displays each tracker as a horizontal chart, illustrating the frames in which each tracker is visible. The current frame is displayed using white vertical lines.

Each frame in which the tracker is present is indicated by a blue square. Lighter-blue squares indicate frames in which the tracker was first located. Clicking on an indicator will select the tracking point and display that frame in the Cinema window. Holding the Ctrl key will allow multiple tracking points to be selected.

The panel can be panned horizontally or vertically by clicking and dragging with the right mouse button. Clicking and dragging with the middle mouse button will zoom either horizontally or vertically to increase the number of tracking points and frames that are displayed in the panel.

The mouse wheel can also be used to zoom horizontally, or vertically if the Alt/Option key is held.

Additional controls are also available:

All: Switch between displaying all trackers, or only those trackers visible in the current frame.

Name: Sort the tracking points by name, in alphabetical order.

Start: Sort the tracking points according to the first frame in which they are tracked.

End: Sort the tracking points according to the last frame in which they are tracked.

Fit: Fit the tracking points display to the window. This will zoom in or out as necessary, displaying as many tracking points and frames as will fit in the viewport.

Other Buttons

Auto-Track: Start the automatic tracking process, picking new trackers in the From frame and tracking them towards the To frame. If trackers already exist, a popup window will be displayed asking whether to remove the existing trackers first, or extend them into frames where they have not been tracked. As tracking progresses, trackers that move out of bounds or cannot be matched accurately will be replaced by new trackers in an attempt to maintain the target number. Holding the Shift key whilst clicking this button will run the automatic tracking process in the background.

Import: Display a file browser allowing an ASCII text file containing tracker locations to be imported. An example ASCII text file is given at the bottom of this document.

Export: Display a file browser allowing an ASCII text file containing tracker locations to be exported. An example ASCII text file is given at the bottom of this document.

Fetch: Display a list of all trackers that are being piped into this node, allowing trackers to be taken from up-stream for further tracking. This is useful in situations where footage has been replaced and trackers must be extended into new frames.

Split: Split the selected tracker into two at the current frame. The selected tracker will be terminated at the previous frame, and a new tracker will be generated covering frames from the current frame to the out frame. This can be used in cases where a tracker jumps between two valid feature points to create two separate trackers, each covering a feature point before and after the jump.

Merge: Merge all selected trackers together into a single tracker. This can be useful in situations where a tracker that is covering a feature is terminated, and then another new tracker is introduced at the same feature point. By merging these trackers together, one tracker is created that covers a wider range of frames. For trackers that exist in the same frame, their positions will be averaged during the merging process.

Delete: Delete all selected trackers.

All/None: Select either all trackers, or unselect all trackers, depending on how many of the trackers are currently selected.

Rename: Display a popup window allowing all selected trackers to be renamed at the same time. Trackers will be numbered sequentially starting at 1 using the new name entered in the popup window.

Skip Frame: Skip the current frame during auto tracking. Once skipped, the button label will change to Un-Skip Frame to include the frame during tracking. When a frame is skipped, the tracking path will not be updated, and will instead be interpolated from nearby frames. Skipping frames can be useful when one frame is missing or corrupted due to some sort of image degradation.

Edit ROI: Adjust the region of interest (ROI) for tracking in the Cinema window. Click and drag with the left mouse button to adjust edges of the ROI. Trackers outside the ROI will be rejected during tracking.

Centre View: Translate the image displayed in the Cinema window so that the first selected tracker is positioned in the centre of the window. This can be useful when checking the motion of one particular tracker to see how well it manages to track an image feature over multiple frames.

Preview Track: Display new tracker locations in the Cinema window. This can be used to quickly assess how different parameters affect positioning before tracking. Hovering the mouse near a preview point will display numerical readouts of the tracking motion to the next frame.

Marquee: Draw a selection marquee in the Cinema window to select trackers. Holding the Ctrl key whilst drawing will ensure that previous selections are kept. Holding the Shift key will allow a lasso selection to be used instead of a rectangle.

Live Update: Update the Cinema window whilst tracking. When this option is disabled, a progress window will be displayed instead. Disabling live updates can provide increased tracking speed in certain situations, especially when multiple viewer windows are open and the graphics card is configured to sync to vertical blanking.

Show Windows: Display the current tracking window and search range around the mouse position. This can be used when setting the window size and search range parameters to ensure that they are suitable for the types of feature in the image, and the amount of displacement between image frames.

Show Paths: Display all tracker paths in the Cinema window. Otherwise, only the paths of selected trackers will be displayed.

Show Names: Display the names of selected trackers in the Cinema window.

Show Info: Display positional information of selected trackers in the Cinema window.

Show Selected: Display only selected trackers in the Cinema window.

Mouse controls

Mouse controls can be customised in the Preferences.

Click and drag left mouse button in Cinema or Tracking Window

Adjust tracker position. Holding the Shift or Ctrl keys will allow multiple selections to be made. The ↑ and ↓ cursor keys can also be used to change the selected tracker

Default Keyboard Shortcuts

Keyboard shortcuts can be customised in the Preferences.

Auto Track	Shift+A
Split	Shift+S
Merge	Shift+E
Marquee	Shift+M
All/None	Shift+L
Delete	Del
Show Selected	Shift+H
Show Paths	Shift+P
Show Names	Shift+N
Show Windows	Shift+W
Centre View	Shift+C
Remove Earlier	[
Remove	Shift+R
Remove Later	]
Hide/Show	H
Skip/Un-Skip Frame	Ctrl+S
Preview Track	Ctrl+P

Example ASCII tracker text file used for import and export

# "Tracker Name"
# clip number
# frame count
# frame, xpos, ypos, matching score

"Tracker0001"
1
4
15 506.41 320.16 1.0000
16 407.83 318.54 0.9956
17 406.44 315.12 0.8854
18 404.22 311.14 0.8774

"Tracker0002"
1
5
11 274.11 776.34 1.0000
12 277.73 781.45 0.9951
13 281.16 783.44 0.8873
14 285.11 791.62 0.9312
15 288.76 794.12 0.8901