Geometry Track

Usage  |  Controls

The Geometry Track node can be used to track either the camera or a moving object using a triangular mesh instead of tracking points. This can be useful in situations where it is either not possible or not desirable to track using multiple tracking points.

For example, if a geometric model of an actor's head or arm is available, that body part can be tracked very quickly and easily using the Geometry Track node.

In its basic mode of operation, the object being tracked must be undergoing a rigid transformation consisting of changes to its translation, rotation and scale parameters. Geometry tracking can also be used to track the motion of a camera relative to the set if a model of (part of) the set is available. In many cases, a simple object such as a flat rectangle or a cube can be used to track part of a background set.

The Geometry Track node can also be used to track a deformable object like a talking face. This can be achieved by creating one or more deformable tracking groups, assigning some of the triangles in the mesh to those groups, and specifying how the groups can transform relative to the rest of the mesh. Deformable tracking groups can be created or deleted, and modified by painting with a brush in a window. Groups are created automatically when an imported Wavefront OBJ file contains group definitions.

Helpers (in the form of manually tracked points passed into the Geometry Track node from a User Track node, can be activated to assist with tracking quickly moving objects where the mesh might otherwise drift because of difficulty in following the image data. Helpers can also be of use in situations where the lighting on an actors face is such that automatically tracking the motion over a large number of frames becomes difficult.

Each vertex of the mesh can be assigned a weight, corresponding to how much influence that vertex has towards calculating the overall motion of the object or camera.

Meshes can be imported using either the Wavefront OBJ, Autodesk FBX, or Alembic file formats, and saved using an internal file format that maintains all group assignments and vertex weights.

Note that the geometric mesh model should be a good approximation of the object that is being tracked, although it does not have to be exact.

The resolution of the mesh only has an impact when tracking deformable geometry because individual triangles must be assigned to different tracking groups. When deformable tracking is not being used, there is no real advantage to using a very high resolution model compared to a lower resolution model that still approximates the geometry fairly well.

Note: Deformable object tracking will perform best when the subject is facing the camera, is illuminated evenly, and covers a large proportion of the image area.

Usage

Importing geometry

Geometry can be imported using either the Wavefront OBJ, Autodesk FBX, or Alembic file formats (as well an an internal format). Geometry can also be fetched from upstream.

Geometry is represented in PFTrack using a left-handed coordinate system using Y as the up direction. If the imported model differs from this it can be converted from its original coordinate system and up direction into left/y using the Coordinate system menus. Normal vectors can also be flipped using the Flip Normals button.

If no pivot position is available in the imported geometry file, the origin point (0, 0, 0) in the mesh's coordinate system will be used instead. This can be changed to the centre of the object's bounding box if required using the Center Pivot button.

The entire object can also be translated so its centre point lies at the origin (0, 0, 0) of the mesh's coordinate system. This can be used to ease the tracking process in situations where the object has not been modelled around the origin point. Note that this is different from moving the pivot to the centre of the object, as the latter relates only to the pivot point used for manual tracking adjustments.

The object can be scaled by a fixed amount after import.

Positioning geometry

After a mesh has been loaded, it must be positioned manually in one frame before it can be tracked. This can be achieved by first making sure the geometry is visible infront of the camera in the current frame, by clicking the View button. The transformation tools listed in the Transform mode menu can then be used to position to geometry to cover the object in the background image (see below for a description of the various transform mode options).

Take care to make sure the camera near/far planes are suitable for the scale of your geometry, otherwise it may not be visible in the Cinema window. Switching the 'Render style' to 'Transparent' can often help when lining the mesh up with the background image.

Tracking geometry

After the mesh has been positioned in one frame, the motion of the object can be tracked into other frames using the Tracking buttons <<, <, >, >>. As tracking commences, the object motion will be adjusted to match the background image. For complex motions or environmental lighting, this can often be difficult to achieve automatically, so it is recommended to use the following scheme to track geometry:

1. Position the geometry manually in the first keyframe
2. Track the geometry forwards using the '<<' or '>>' buttons. If the model drifts from the background image, stop tracking.
3. Manually adjust the position of the geometry in current frame to re-fit to the image. This will create a new keyframe.
4. Step backwards one frame (so you are viewing a frame in-between the two keyframes) and click the 'Refine' button.

Clicking the 'Refine' button in step 4 will attempt to re-track the geometry between the two keyframes and correct for any jumps that may have been introduced when the geometry was manually positioned in step 3.

Repeat this process throughout the clip until the geometry is tracked in every frame. Keyframes are shown as yellow markers in the scrub bar. The 'Refine' button can also be used when tracking deformable object groups, as described below.

Painting vertex weights

In the most basic mode of operation, the Geometry Track node can be used to track the motion of rigid objects. Often, part of the subject that is being tracked will not satisfy this rigid constraint. For example, if the rigid motion of an animal head is required, but the animal is moving its mouth, the the motion of the mouth will affect the accuracy of tracking for the rest of the head.

In these cases, weights can be painted onto the mesh to indicate how much each vertex should contribute to the overall solution. Vertices with a weight of zero will not contribute at all to the estimate of motion. In the example described above, the vertices associated with the mouth area would be painted with a weight of zero before the head is tracked.

Weights can also be painted over parts of the image that, whilst still undergoing a rigid transformation, may present problems to the tracking algorithm because of rapidly changing shading or illumination. For example, when tracking an actor's head that is brightly lit from one angle and rotating, bright specular highlights on the side of their face may adversely affect the tracking accuracy. By reducing the weights in these areas of the mesh, the influence of these areas on the overall motion estimate can be reduced.

Using helpers

Helpers can be used to assist with tracking either the main rigid mesh object or a deformable tracking group. Helpers should be generated using a User Track node, and each must follow the motion of a single point such as the corner of a mouth or a tracking marker positioned on an actor's face.

Tracking deformable groups

In many situations, the object that must be tracked will not be rigid and instead undergo some small amount of deformation. For example, when tracking an actor's face, their jaw may be moving and the motion of the jaw must also be estimated along with the overall motion of their head. This can be achieved by creating one or more deformable tracking groups and specifying how each triangle in that group can move relative to the rigid mesh.

An example of 4 groups assigned to track a talking face is given below: "Brow" (blue), "Top lip" (green), "Bottom lip" (yellow) and "Jaw" (red). White triangles are not present in any deformable group and therefore correspond to the rigid part of the object. The pivot location for the "Brow" group is shown as X, Y and Z axes.

Here is the corresponding group table, showing how the allowable motion of each group has been specified. Pivots for each group are at the same orientation as the one in the image above, with Y vertical and X pointing to the right as you look at the face, although their positions are different for each.

When tracking a mesh with deformable groups, both the rigid mesh motion and the deformation of each group will be calculated. If it is unclear how well a mesh will track, or how each pivot should be set up to best model the motion, the recommended way of working is as follows:

1. Create deformable tracking groups, but ensure each one is locked by ticking the Lock column,

2. Track the rigid position of the mesh as usual, manually adjusting keyframes and refining the motion where necessary,

3. Tick the Lock Base Motion option in the Tracking controls. This will lock the motion of the rigid portion of the object and prevent it from being changed,

4. For each group in turn, un-tick the Lock column, position the group's pivot, specify the allowable motion and then track through the clip using the << and >> buttons. Refinement can be used after a group is manually adjusted to improve tracking between manually placed keyframes.

Here are 10 (out of 100) frames from the geometry track, which was obtained entirely automatically once the groups were set up. The actor's head is moving away from the camera slightly whilst talking. In this case, all deformable groups were allowed to track at once, and the geometric mesh is a generic face model that does not exactly match the actor.

Controls

Camera

The Camera section contains information about the camera associated with the current clip.

Focal length: The camera focal length at the current frame. When tracking a camera instead of an object, focal length can be set as Known or Initialised, and a value entered in the edit box. Note that in order to enter a focal length measured in any unit other than Pixels requires that the camera sensor/film back width and height is set correctly

Note that when tracking a camera using geometry, the camera focal length can be automatically adjusted whilst pinning vertices provided that the camera's focal length has been set to Initialised. At least 4 pins must be placed before the focal length will adjust to better fit the pin locations.

Field of view: The horizontal and vertical field of view at the current frame, measured in Degrees.

Sensor size: The horizontal and vertical sensor/film back size. The sensor size is defined in the Clip Input node.

Pixel aspect: The current pixel aspect ratio. To change the pixel aspect ratio, edit the camera preset value in the Clip Input node.

Near/far planes: The near and far camera clipping planes that are used to display geometry in the Cinema. The near and far clipping planes can be changed in situations where geometry is very small or very large compared to the scene.

Geometry

Coordinate System: The original coordinate system in which the geometry was modelled. This can be useful to ensure the geometry is correct when importing from a modelling application that does not use a left-handed coordinate system with Y as the up direction.

Load: Display a file browser to locate either an OBJ, FBX, ABC or PFM mesh file to be loaded and used for tracking. Note that if a moving object is to be tracked, make sure the Moving Object tick-box is enabled before loading geometry. If geometry has already been tracked, this button will change to Replace and can also be used to replace the geometric object whilst keeping the current tracking transformation by answering Yes when queried.

Geometry can also be imported by dragging an object file out of the file browser and dropping it into the node.

Fetch: Fetch a geometric object from an up-stream node and used for tracking. Note that if a moving object is to be tracked, make sure the Moving Object tick-box is enabled before fetching geometry.

Save: Save the current geometric object (including vertex weights and deformable groups) to disk for later use. This can be helpful when preparing a geometric model for tracking in multiple shots.

Moving Object: When enabled, the geometry will be tracked as an object moving independently from the camera. Tracking a moving object will ensure the current camera motion is kept, and only the object changes position. If the Moving Object option is not set, the position of the object will remain fixed, and the camera will be adjusted accordingly.

View: Clicking this button will scale and translate either the camera, or if Moving Object is enabled, the object itself, so the geometry is visible in front of the camera.

Centre Object: Translate the object so its centre point lies at the origin (0, 0, 0) of the mesh's coordinate system.

Centre Pivot: Position the pivot (around which the object is transformed) so it sits at the centre of the geometry bounding box.

Flip Normals: Toggle the direction of surface normals. This can be useful when importing geometry with reversed front/back facing triangles.

Display

Render Style: The style used for rendering primitives in the Cinema and Viewer windows. Options are

- Wire-frame: Display the mesh using wire-frame outlines only.

- Hidden-line: Display the mesh using hidden line wire frame.

- Facet-shaded: Display the mesh using flat shaded facets for each triangle.

- Smooth-shaded: Display the mesh using smoothly interpolated vertex normals.

- Transparent: Display the mesh using a semi-transparent colour. The transparency can be adjusted by clicking the Mesh Colour button and changing the Alpha value.

- 2D Checkerboard: Display the mesh using a 2D checkerboard texture. Currently, this is only possible if the imported mesh already has a UV map associated with it. The scale of the checkerboard texture can be adjusted by clicking the Checker +/- buttons.

- 3D Checkerboard: Display the mesh using a 3D checkerboard texture. This is a procedurally generated texture that does not require a UV map. The scale of the checkerboard texture can be adjusted by clicking the Checker +/- buttons.

- Textured: Display the mesh in a Viewer window using projective texturing from the camera. In the Cinema window, hidden-line rendering will be used instead.

- Flat: Display the mesh using flat, un-shaded triangles only. This mode can be helpful when assigning triangles to deformable tracking groups.

- Weights: Display the mesh using red/green encoding for vertex weights in the range zero to one. Red indicates a low weight (0.0) and green indicates a high weight (1.0).

Note that all display modes apart from Textured and Weights will display the group assignment colours if the Show Groups option is enabled.

When Textured mode is used, the image from the current camera frame will be projected onto the primitive and displayed in the Viewer windows only. The Checkerboard option will only be available if the geometry has a set of UV texture map coordinates.

Checker +/Checker -: Increase or decrease the size of the checkerboard texture that is displayed when the using the Checkerboard render style.

Mesh Colour: Display a colour selection window that allows the colour of the current geometry mesh to be changed. The mesh transparency can also be adjusted by changing the Alpha value in the colour selection window.

Centre View: When enabled, the Cinema window will be translated so that the geometry pivot point is fixed in the centre of the window.

Show Ground: When enabled, the ground plane will be displayed in the Cinema and Viewer windows.

Show Horizon: When enabled, the horizon line will be displayed in the Cinema window.

Show Geometry: When enabled, the geometric object will be rendered in the Cinema and Viewer windows.

Show Groups: When enabled, the geometric mesh will be coloured to indicate the deformable group to which each triangle belongs. These colours can be changed within the Deformable Groups table. Triangles that do not belong to any deformable group will be coloured white.

Show Selected Group: When enabled, the geometric mesh will be coloured to show the selected group only.

Show Helpers: When enabled, trackers that are passed into the node and used as helpers will be displayed in the Cinema window.

Show Trackers: When enabled, solved tracker positions will be displayed in the Cinema and Viewer windows.

Show Frustum: When enabled, the camera frustum will be displayed in Viewer windows.

Uniform Scale

Scale %: The percentage scale factor.

Scale: Scale the object uniformly by the percentage scale factor. This is best done as soon as the geometry is imported.

Helpers

The Helpers list displays a list of all trackers that were generated with a User Track node that can be used as helpers when tracking geometry.

Name: This column shows the name of the tracker.

Active: Activate the tracker as a helper for geometry tracking.

Weight: The weight of each helper. Helpers with larger weights will influence the geometry tracking more than those with lower weights. The default weight for a helper is 1.

Activate: Activate all selected helpers.

De-activate: De-activate all selected helpers.

Set Weight: Set the weight for all selected helpers using a popup edit box.

All/None: Select all or none of the helpers in the list.

Vertex Weights

Vertex weights can be used to control how much influence different areas of the geometric mesh have when estimating motion. Vertex weights can be displayed by changing the render style to Weights, where green corresponds to a high weight and red corresponds to a low weight. Areas of the mesh with low weights will not contribute as much to the estimate of motion compared to areas with high weights.

Weight: The weight value that will be painted onto the mesh. A value of 0 corresponds to the lowest weight (red) and a value of 1 corresponds to the highest weight (green).

+/-: Increase or decrease the vertex weights by painting onto the mesh.

R: When enabled, painting on the mesh will reset vertex weights back to their default value of 1 (green).

Radius: The radius of the brush used to paint vertex weights.

Paint: When enabled, vertex weights can be painted by clicking and dragging with the left mouse button in either the Cinema or a Viewer window.

Depth Test: When enabled, painting will only occur on the front-most part of the surface, and hidden parts of the object will not be affected.

Back Faces: When enabled, painting will occur on front and back-facing surfaces. When disabled, painting will only occur on front-facing surfaces

Reset: Clicking this button will reset all vertex weights to their default value if 1 (green).

Note than whilst painting in the Viewer window, the camera rotation cannot be adjusted, because this is usually done by clicking and dragging with the left mouse button. In order to rotate the camera, either switch paint mode off, or hold the Alt/Option key to rotate around the origin (you can also hold the Ctrl key to rotate around the object's origin as well).

Deformable Groups

Groups table

Name: The name of each deformable group. Group names can be edited by double clicking with the left mouse button in this column.

Colour: The colour assigned to each deformable group. Group colours can be displayed by enabling the Show Groups display option, and can be edited by double clicking in this column with the left mouse button.

Lock: When enabled, the deformable group motion will be locked, meaning it will not be adjusted when tracking.

Tx, Ty, Tz: Control whether the group is allowed to translate along the X, Y and Z axes, as defined by the group's pivot point.

Rx, Ry, Rz: Control whether the group is allowed to rotate along the X, Y and Z axes, as defined by the group's pivot point.

Sx, Sy, Sz: Control whether the group is allowed to scale along the X, Y and Z axes, as defined by the group's pivot point.

Controls

Create: Create a new empty deformable group. Triangles can be added or removed from the group using the + and - buttons.

Delete: Delete the selected group. When a group is deleted, all triangles assigned to it will be re-assigned to the rigid base group that does not undergo any deformation.

Reset: Reset the deformable group transform to its initial state (i.e. no deformation).

Translate: When enabled, a translation widget will be displayed, allowing the group to be translated relative to its pivot point.

Rotate: When enabled, a translation widget will be displayed, allowing the group to be rotated relative to its pivot point.

Scale: When enabled, a translation widget will be displayed, allowing the group to be scaled relative to its pivot point.

Move Pivot: When enabled, a translation widget will be displayed, allowing the group's pivot point to be moved.

Rotate Pivot: When enabled, a translation widget will be displayed, allowing the group's pivot point to be rotated.

Radius: This slider controls the size of the paint brush used to assign triangles to deformable groups.

+: When enabled, triangles can be added to the current group by clicking and dragging with the left mouse button in either the Cinema or Viewer windows.

-: When enabled, triangles can be removed from the current group by clicking and dragging with the left mouse button in either the Cinema or Viewer windows. When a triangle is removed from a group, it is re-assigned to the rigid base group, which does not undergo any deformation.

Grow: Grow the current group assignment into neighbouring triangles that are not assigned to any deformable group.

Shrink: Shrink the current group assignment, so triangles that are next to triangle in the rigid base group will be re-assigned to the rigid base group, which does not undergo any deformation.

Depth Test: When enabled, painting will only occur on the front-most part of the surface, and hidden parts of the object will not be affected.

Back Faces: When enabled, painting will occur on front and back-facing surfaces. When disabled, painting will only occur on front-facing surfaces

Tracking

<<, <, >, >>: Track the geometric object backwards by multiple frames, backwards by one frame, forwards by one frame or forwards by multiple frames.

Refine/Refine Group: Attempt to refine motion of either the rigid object, or the selected group, to better fit the image data after geometry has been tracked between two manually placed keyframes. This button is only available when viewing a tracked frame that exists between two manually placed keyframes.

Auto Refine: When this button is enabled, the motion of the object will be automatically refined after adjusting a manual keyframe. This is equivalent to clicking the Refine/Refine Group button for frames on either side of the current manually placed keyframe.

Translation Axes: Control whether a moving object will be allowed to translate along the camera's X, Y or Z axes. The Z+ and Z- buttons can be used to specify whether an object is allowed to move along the camera's Z axes away (Z+) or towards (Z-) the camera.

Rotation Axes: These buttons control whether a moving object will be allowed to rotate around its X, Y or Z axes.

Z Smoothing %: The amount of smoothing that will be applied to object translation when moving towards/away from the camera. This can be increased to reduce z-jittering when tracking small objects. Values at or above 100% will disable translation towards/away from the camera entirely.

Transform Mode: The transformation mode that will be used to manually adjust motion and create a keyframe. Options are

- None: This mode disables all manual interaction with the object or camera.

- Fly: When this mode is enabled, the object or camera can be translated or rotated by holding the Alt/Option key and clicking and dragging with the right or left mouse buttons respectively in the Cinema window. Clicking and dragging with the middle mouse button will adjust either the distance of the object from the camera (in the direction towards the object's pivot point, if the Lock Object Size option is set) or the size of the object itself.

- Pin: This mode allows vertices of the mesh to be dragged into position in the Cinema window by clicking and dragging with the left mouse button. As each pin is positioned, the transformation of the object/camera will be updated to best match all pin positions in the current frame. To remove a pin, hover the mouse over the mesh vertex and press the Delete or Backspace key.

- Translate, Rotate, Scale: These interaction modes can be used in either the Cinema or a Viewer window, and will create a manipulator widget that can be used to translate, rotate or scale the object/camera. These manipulators are positioned at the object's origin (0.0, 0.0, 0.0) by default, or at the centre of the object's bounding box if the Centre Pivot button was pressed. Holding down the Ctrl key whilst dragging will allow finer adjustments to be made.

Lock Rigid Motion: When enabled, rigid motion will be locked and not updated during tracking. This can be helpful when tracking deformable geometry to ensure that the overall mesh is not adjusted, and only deformable groups are tracked.

Lock Object Size: When enabled, the object scale will not be adjusted when manually transforming the object using Fly mode, and any scale transformations will instead move the geometry towards or away from the camera.

Add Key: Add a keyframe to either the object or camera transformation at the current frame.

Reset Frame: Remove transformations at the current frame. Frames that do not have any transformation will interpolate motion from nearby keyframes.

Previous Key: Move to the previous keyframe in the scrub bar.

Next Key: Move to the next keyframe in the scrub bar.

Default Keyboard Shortcuts

Keyboard shortcuts can be customised in the Preferences.

Track Forwards	]
Track Backwards	[
Track Forwards Frame	Shift+}
Track Backwards Frame	Shift+{
Refine	Shift+P
Add Key	Shift+K
Reset Frame	Shift+E
Previous Key	N
Next Key	M
Centre View	Shift+C
"Fly" transform mode	Shift+I
"Pin" transform mode	Shift+N
"Translate" transform mode	Shift+T
"Rotate" transform mode	Shift+R
"Scale" transform mode	Shift+S
Translation X Axis	Ctrl+C
Translation Y Axis	Ctrl+V
Translation Z+ Axis	Ctrl+B
Translation Z- Axis	Ctrl+N
Rotation X Axis	Shift+X
Rotation Y Axis	Shift+Y
Rotation Z Axis	Shift+Z
Translate Group	Ctrl+T
Rotate Group	Ctrl+R
Scale Group	Ctrl+S
Move Pivot	Shift+M
Rotate Pivot	Shift+O
Add To Group	Shift+A
Remove From Group	Shift+V
Paint Weight	Shift+W
Show Ground	Ctrl+G
Show Horizon	Ctrl+H
Show Geometry	Ctrl+M
Show Groups	Ctrl+O
Show Selected Group	Ctrl+Q
Show Helpers	Ctrl+L
Show Trackers	Ctrl+E
Show Frustum	Ctrl+F

-----

Alias|Wavefront is a trademark of Alias|Wavefront, a division of Silicon Graphics Limited in the United States and/or other countries worldwide.

Maya, Softimage, 3DS Max and FBX are registered trademarks of Autodesk Inc. in the USA and other countries.

Alembic is trademark of and copyright 2009-2015 Lucasfilm Entertainment Company Ltd. or Lucasfilm Ltd. All rights reserved. Industrial Light & Magic, ILM and the Bulb and Gear design logo are all registered trademarks or service marks of Lucasfilm Ltd. Copyright 2009-2015 Sony Pictures Imageworks Inc. All rights reserved.