Mocap Solver

Usage  |  Controls

The Mocap Solver node can be used to obtain the motion of individual tracking points viewed from two or more camera positions. This is often used to track the motion of an actor's body or face, where tracking points have been identified using physical markers.

In contrast to standard object tracking, the Mocap Solver node does not assume that the object is moving in a rigid fashion. The motion of each tracking point is completely independent and can therefore represent movement of non-rigid objects.

The Mocap Solver node can have multiple inputs, each corresponding to a different camera view of the subject. The first input is the primary camera, and the second (or third etc..) inputs correspond to witness cameras. The witness cameras can be at a different resolution to the primary camera, but must have the same frame-rate.

Usage

In order to solve for tracker motion viewed by multiple cameras, it is important to ensure that the frame numbers of witness cameras match those of the primary camera. If this is not the case, adjust the in/out points or frame offset of the witness camera in its Clip Input node so that the each frame of the witness camera is viewing the actor at the same point in time as the primary camera. To help with this, it is often useful to identify a reference event such as an eye blink or other action, and use that to synchronise the witness cameras to the primary.

Trackers should be tracked with a User Track node, making sure that each tracker is tracked in at least two of the input clips. Note that these trackers must be placed into a new tracking group, as trackers associated with the camera cannot be used for motion capture. The screenshot below shows how a User Track node should be connected to two input clips (the primary camera and one witness camera), and then to the Mocap Solver node:

The screenshot below shows four trackers: Tracker0001, Tracker0002, Tracker0003 and Tracker0004, that have each been tracked in both the primary and witness cameras:

Once trackers have been created, information about the primary and witness cameras must be set in the Mocap Solver node. If focal length information is available for either the primary or witness cameras, entering that data will help the solver produce a more accurate result.

Note that when entering a focal length in a physical unit such as millimeters, it is important to first ensure the sensor/film back size are correct for your camera sensor. This can be set in the Clip Input node.

In order for the mocap solver to estimate the position of the tracking points in 3D space, the witness cameras must be stationary (i.e. they must not translate or rotate, and must have a constant focal length). If only one witness camera has been used then the primary camera must also be stationary. If two or more witness cameras are used, then the primary camera is allowed to move.

Once solved, trackers are shown in the Cinema window coloured according to how well their projected position matches their 2D tracker locations. Trackers that match their 2D location well (with a projection error less than 1 pixel) are coloured green, trackers with projection errors less than 2 pixels are coloured orange, and trackers with projection errors larger than 2 pixels are coloured red. An error line is also drawn connecting the projected tracker position with its 2D tracker location.

Motion Capture Using Pre-Solved Cameras

When the primary camera is moving and viewing a scene containing enough tracking markers, better mocap results can often be obtained by solving for the camera positions before using the Mocap Solver node.

This can be done using the Camera Solver node to solve for the position of the primary and witness cameras, feeding the results into a User Track node to track the mocap markers, and finally attaching a Mocap Solver node. The mocap tracker positions can then be generated by either:

- Enabling the Use Input Cameras option and clicking the Solve All button.

- Clicking the Solve Trackers button to solve for the tracker positions only.

The following screenshot shows an example tree used for motion capture with a pre-solved moving primary camera and one witness camera (footage from http://www.hollywoodcamerawork.us/trackingplates.html). The Auto Match and User Track nodes were used to create features for tracking the motion of the primary camera and the position of the witness camera in the Camera Solver node. The User Track 1 node was then used to create trackers for mocap (making sure to place them into a new tracking group).

Controls

Current clip: The clip that is being displayed in the Cinema window.

Current group: The tracker group that will be used for motion capture. Note that trackers present in the camera group cannot be used for motion capture.

Solver

Start/end frames: The start and end frames to use for motion capture. The S button will store the current frame number as either the start or end frame.

Solve All: Solve for camera and tracker positions. The motion capture solve can be run in the background by holding the Shift key whilst clicking on the Solve All button.

Solve Trackers: Solve for motion capture tracker positions only. This can be used in situations where new trackers have been added up-stream and need to be solved using the existing camera positions..

Refine All: Adjust camera and motion capture tracker positions to better match the tracker paths. Refinement can be run multiple times, and a longer refinement can be ran by holding the Shift key whilst clicking on the button.

Exactly match mocap trackers in the primary camera: When enabled, the position of mocap trackers in 3D space will be set to exactly match the tracker path in the primary camera during solve and refinement operations. Note that it will often be the case that doing this will increase the error in the witness cameras. When this option is not enabled, the error will be spread as evenly as possible between the primary and witness cameras.

Use input cameras: When enabled, the cameras that are input to the Mocap Solver node will be used to solve for the tracker positions. Enabling this option will lock the input primary and witness cameras so they are not changed during the mocap solve.

Match background trackers: When trackers have been used up-stream to solve for the position of the primary and/or witness cameras, enabling this option will mean that these trackers are taken into consideration when trying to minimise the error during refinement, in addition to the trackers used for motion capture. Note that in order for an up-stream camera position to be modified during refinement, it must be un-locked in the Camera parameters tab.

Camera

The camera control tab contains information about the camera associated with the current clip. If more than one input clip is present, the current camera (and clip) can be changed using the Current clip menu option.

Focal length: The camera focal length at the current frame. Focal length can be set as Known, Unknown or Initialised. If the focal length of the camera is known beforehand, entering the value here can often improve both the speed and accuracy of the mocap solver. Setting focal length to Initialised will allow a minimum and maximum value to be specified in the Focal range edit boxes, and an initial value to be entered into the Focal length edit box. Note that in order to enter a focal length measured in any unit other than Pixels requires that the camera sensor/film back size is set correctly

Focal range: When focal length is set to Initialised, these edit boxes define the minimum and maximum allowable values of focal length.

Field of view: The horizontal and vertical field of view at the current frame, measured in Degrees.

Sensor size: The horizontal and vertical sensor/film back size. To change sensor size, edit the camera preset values in the Clip Input node.

Near/far planes: The near and far camera clipping planes.

Pixel aspect: The current pixel aspect ratio. To change the pixel aspect ratio, edit the camera preset value in the Clip Input node.

Translation: The smoothness of the camera translation path. This option is only available for a non-stationary primary camera. Smoothing options are None, Low, Medium and High.

Rotation: The smoothness of the camera rotation path. This option is only available for a non-stationary primary camera. Smoothing options are None, Low, Medium and High.

Sync offset: The temporal offset to apply to the witness camera during the mocap solve. This edit box is only available for witness cameras. This is used to correct situations where there is a small difference between the times at which the witness camera captured frames, compared to the primary camera. Setting this value to 0.5 means that the actor viewed by the primary camera at frame N is in exactly the same position when viewed by the witness camera at frame N+0.5.

Auto-sync: When enabled, the temporal sync offset will be calculated automatically during the mocap solve.

Lock: When enabled, the position and orientation of the camera will be locked to prevent adjustment when refining the solve.

Stationary: Assume the primary camera to be stationary. This option is only available for the primary camera. This information is used to help improve the accuracy of the mocap solve. All witness cameras are assumed to be stationary.

Trackers

The tracker list contains information about all trackers passed into the mocap solver node.

Columns

Name: The tracker name.

Active: Indicates whether the tracker is active in the mocap solver or not. To activate or de-activate multiple trackers at the same time, select them and click the Activate or Deactivate buttons on the right of the tracker list.

Rigid: Indicates that the tracker is moving in a rigid fashion with respect to all other trackers that have this flag set. This is used to help improve the accuracy of the mocap solver. For example, when placing trackers on an actor's face, it is often the case that points on their forehead are all moving in a rigid fashion with respect to each other.

Weight: The weight given to a particular tracker in the solution. The default value is 1.0, but increasing this will mean the mocap solver expends more effort to match a 3D tracker position to its path, possibly at the cost of decreased accuracy elsewhere. Changing the tracker weight can often help to lock a solution down onto a particular tracker.

Residual: The residual projection error (measured in pixels) for the tracker in the current frame. The projection error is the difference between the tracker path position and the projection of the 3D tracker point onto the camera plane. Ideally, the projection error should be close to zero for each tracker.

Distance: The distance from the current camera frame to the tracker's 3D position in space. This can be useful to detect when the mocap solver has failed to determine which trackers are in the foreground and which are in the background.

Controls

All / None: Select all or none of the trackers from the list.

Activate: Activate all selected trackers in the mocap solver. Active trackers will contribute to the solution. Trackers can also be activated individually by ticking the Active column in the tracker list.

Deactivate: Deactivate all selected trackers in the camera solver. Inactive trackers will not contribute to the solution. Trackers can be deactivated individually by un-ticking the Active column in the tracker list.

Smoothing: The amount of smoothing to apply to each mocap tracker in 3D space. The available options are None, Low, Medium and High. When smoothing is enabled, jitter on the 3D tracker position will be reduced.

Error Graph

The error graph plots the projection error (measured in pixels) for each tracker in each frame, along with the average projection error for all trackers visible in a frame. The projection error is the difference between the tracker path position and the projection of the 3D tracker point onto the camera plane. Ideally, the projection error
should be close to zero for each tracker. Selected trackers are shown in yellow, and unselected trackers are shown in blue. The average projection error graph is shown in white. The error graph can be translated and scaled by clicking and dragging with the right or middle mouse buttons.

Individual trackers can be selected from the graph by clicking with the left mouse button. When a tracker is selected, the current frame will change to match the frame number that was clicked in the graph. If the Centre View display option is also enabled, the main image window will be panned to display the tracker projection in the centre. Multiple trackers can be selected at once by either holding the Ctrl key whilst clicking, or by clicking and dragging with the left mouse button to draw a selection rectangle.

If a tracker has been tracked accurately but has a projection error that increases significantly in certain frames, it often indicates that the camera position or focal length in those frames is incorrect. Adding more trackers to the solution, or providing estimates of camera focal length can often help out here.

Click and drag right mouse button in error graph to pan the graph. Click and drag middle mouse button in error graph to zoom the graph (or use the mouse wheel, holding Alt/Option to zoom vertically instead of horizontally).

All: When enabled, error graphs will be shown for all trackers, otherwise graph will only be shown for selected trackers.

Fit: Scale and translate the error graph so all graphs are visible.

The Coverage Panel

The Coverage Panel displays information about the frames in which each tracking point has been tracked:

This can be used to evaluate how well tracking points are distributed throughout the clip, which will help to provide an accurate camera solve without any jumps in the camera path.

Coverage Keys display

By default, the Coverage Panel displayed keyframe information showing how tracking points have been positioned and tracked. Each frame in which the tracker is present is shown with a blue square. Yellow squares show frames in which manually generated tracking points were keyframed.

Frames in which the tracker is visible but has not been positioned are displayed in dark red. It is important to ensure that tracking points have been positioned in all frames in which they are visible, as this can significantly affect the accuracy of the camera solve.

Coverage Error display

Alternatively, the Coverage Panel can also display the projection error for each tracker by clicking the Errors button. This switches the colour-coding of each indicator to show the error of the solved tracking point in each frame. This is colour-coded to show green for less than 0.5 pixels error, yellow for 1.5 pixels and red is greater than 2.5 pixels.

Coverage Panel controls

The Coverage Panel can be panned horizontally or vertically by clicking and dragging with the right mouse button. Clicking and dragging with the middle mouse button will zoom either horizontally or vertically to increase the number of tracking points and frames that are displayed in the panel.

The mouse wheel can also be used to zoom horizontally, or vertically if the Alt/Option key is held.

Clicking on an indicator with the left mouse button will select the tracking point and display that frame in the Cinema window. Holding the Ctrl key will allow multiple tracking points to be selected.

Double-clicking on an indicator with the left mouse button will select the tracking point and immediately switch to display the node which generated that tracking point. This can be used to quickly jump to a User Track node to manually adjust a tracking point to correct a tracking error.

All: Switch between displaying all trackers, or only those trackers visible in the current frame.

Keys: Display keyframe information, showing where targets have been tracked and manually positioned.

Errors: Display projection error information.

Name: Sort the tracking points by name, in alphabetical order.

Start: Sort the tracking points according to the first frame in which they are tracked.

End: Sort the tracking points according to the last frame in which they are tracked.

Fit: Fit the tracking points display to the window. This will zoom in or out as necessary, displaying as many tracking points and frames as will fit in the viewport.

Solver Log

This window contains useful information generated by the solver as the mocap solution is estimated, including the estimated field of view and focal length, and the average pixel error of each tracker as it is solved. By default, the solver log is not stored in the project file, although this behaviour can be changed from within the General Preferences window.

Here is an example output when for two cameras with unknown focal length:

Initial solve using cameras [Camera01] and [Camera02]
[Camera01]: Found focal length 9.075 mm (FOV= 51.7046 x 39.9662)
[Camera02]: Found focal length 12.3 mm (FOV= 39.3439 x 30.0367)

In this case, a field of view of 51.7 x 39.9 degrees was found for the primary camera, and 39.4 x 30.0 degrees for the witness camera.

[Tracker0001]: initial error= 1.313 (min= 0.021193, max= 2.31726)
[Tracker0003]: initial error= 1.33468 (min= 0.0559052, max= 2.99335)
[Tracker0004]: initial error= 1.25211 (min= 0.0108167, max= 6.99517)
[Tracker0006]: initial error= 0.785642 (min= 0.0752704, max= 1.47206)
[Tracker0007]: initial error= 0.797631 (min= 0.022447, max= 1.46361)
[Tracker0008]: initial error= 1.483 (min= 0.155891, max= 3.52983)
[Tracker0009]: initial error= 0.857057 (min= 0.0644003, max= 3.43896)

This text describes the initial projection error for the trackers after the camera positions have been estimated. For each tracker the average error (measured in pixels) is displayed, along with the minimum and maximum over all frames.

[Tracker0001]: error= 0.685231 (min= 0.00299527, max= 3.03761)
[Tracker0003]: error= 1.18003 (min= 0.0414735, max= 2.69461)
[Tracker0004]: error= 0.872787 (min= 0.0132997, max= 4.90216)
[Tracker0006]: error= 0.727525 (min= 0.00213882, max= 1.78646)
[Tracker0007]: error= 0.612514 (min= 0.00653726, max= 1.55146)
[Tracker0008]: error= 0.385082 (min= 0.00313076, max= 1.49515)
[Tracker0009]: error= 0.52303 (min= 0.000858293, max= 2.61386)

The last section of the text describes the final projection error for the trackers after refinement.

Display

Show Ground: When enabled, the ground plane will be displayed.

Show Horizon: When enabled, the horizon line will be displayed.

Show Geometry: When enabled, the geometric objects from up-stream will be displayed.

Show Names: When enabled, selected tracker names will be displayed.

Show Info: When enabled, selected trackers will have position and residual error information displayed.

Show BG Trackers: When enabled, any background trackers that were used to solve for the camera position up-stream are displayed.

Show Frustum: When enabled, the camera frustum will be displayed in the Viewer windows.

Marquee: Allow a tracker selection marquee to be drawn in the Cinema window or in a Viewer window. Holding the Ctrl key whilst drawing will ensure that previous selections are kept. Holding the Shift key will allow a lasso selection to be used instead of a rectangle.

Centre View: When enabled, the Cinema window will be panned so the projection of the first selected tracker is fixed at the centre of the window.

Default Keyboard Shortcuts

Keyboard shortcuts can be customised in the Preferences.

Solve All	Shift+S
Refine All	Shift+R
Solve Trackers	Shift+T
All/None Trackers	Shift+L
Activate	Shift+A
Deactivate	Shift+D
Show Ground	Ctrl+G
Show Horizon	Ctrl+H
Show Geometry	Ctrl+E
Show Names	Ctrl+N
Show Info	Ctrl+I
Show BG Trackers	Ctrl+B
Show Frustum	Ctrl+F
Marquee	Shift+M
Centre View	Shift+C
All Errors	Shift+E
Fit	Shift+D
Next Clip	C