3.2 Tracking a Deformable Object with Geometry
This tutorial is part of Level 3. Advanced Matchmoving Strategies in PFTrack of The Pixel Farm Training Academy’s PFTrack course. Find out more and register for the next available live class.
During the Advanced Matchmoving Strategies live class we used a geometric model of an immovable structure on set to track a camera in PFTrack. PFTrack’s Geometry Tracking node, however, is not limited to tracking cameras, and can also be used to track moving objects.
This tutorial will guide you through the necessary steps of tracking the head of an actress. To go even further, we will allow PFTrack to deform the tracking geometry to capture the actress’ performance in the clip.
– Setting Up the Camera in the Geometry Track Node
– Importing the Geometry
– Positioning the Geometry
– Changing the Render Style
– Copying the Geometry Track Node
– Using Vertex Weights
– Tracking the Object
– Creating Deformable Groups
– Adding Triangles to a Group
– Creating the Remaining Group
– Tracking the Deformable Groups
To learn this tutorial you will need to download and use the footage below.
01. Setting Up the Scene
Import the clip into a PFTrack project and play through it. Before any object can be tracked, the camera for the clip must be solved. As the camera for the clip is static you can define it in an Edit Camera node, which you can find in the Utilities category, instead of using trackers and a Camera Solver node. However, for the purpose of tracking an object in a Geometry Track node for a clip with a static camera, even this step could be omitted, as a static camera can be created in the Geometry Track node itself.
Setting Up the Camera in the Geometry Track Node
In the Geometry Track node, check the Moving object checkbox. This tells PFTrack that we are going to track a moving object, and since no camera exists in the tree yet, a static camera will be created.
If a static or moving camera already exist in your tree, the Moving camera checkbox will be checked automatically.
02. Positioning the Tracking Geometry
With the camera and scene set up, the next step is to import and position the tracking geometry in one frame.
Importing the Geometry
Geometric models can be imported into the Geometry Track node from either OBJ, FBX or Open Alembic files, as well as an internal file format. The PFTDialog.zip archive contains a suitable model for this clip in PFTrack’s internal format, face.pfm.
Click the Load button, then select face.pfm, which you should find where you unzipped the PFTDialog archive, in the file browser dialog.
Initially, this particular model faces away from you, so you will only be able to just make out its edges.
Click the View button to reposition the geometry in front of the camera.
Positioning the Geometry
The model has to be positioned manually in one frame before it can be tracked. Select the appropriate mode, such as Translate, Rotate or Scale in the Transform mode menu to do this. Instead of doing those transformations individually, you can also use the Fly mode. When in Fly mode, hold down the Alt-key on Windows or Linux, or Option-key on the Mac, and use the left mouse button to rotate the model, the middle mouse button to scale and the right mouse button to translate the model.
Changing the Render Style
Changing the way the model is rendered on screen might help you position the geometry. Selecting Wire-frame or Hidden-line will only display the edges, rather than faces of the geometry, and lets you see what is ‘behind’ the model.
Use the eyes, nose, mouth and cheeks as guides to position the geometry in a way that fits the actress’ face.
03. Rigid Geometry Tracking
If you only want to track the overall motion of the actress’ head, you could now do so by clicking any of the tracking buttons shown below.
The aim of this tutorial is to track the actress’ performance by making the mesh deformable, as is described in the next chapter. In this chapter, we will perform a rigid track of the actress’ head movements. If you would like to follow the steps in this chapter, you can do so with a copy of the node.
Copying the Geometry Track Node
Using Vertex Weights
While, when tracking this way, the geometric model stays rigid, the actress starts talking, which means her mouth and chin will move. For cases like these, you can change how much the affected vertices (the ones covering the lower part of her face) should contribute to the overall solution. You do this by changing their weights.
Go to the Vertex Weights tab and click the Paint button.
The Render style for the mesh will automatically change to Weights, and the even green colour indicates that all vertices have the same weight and contribute fully to the resulting track.
Use the left mouse button to paint over the lower part of the model to decrease the weights in this area. You can paint in the clip view or perspective view.
Whilst painting the jaw, make sure to check in the perspective view for any areas that are hard to reach in the clip view, like the back of the chin.
When in Paint mode, you will have to hold the Ctrl-key (Windows, Linux) or Command-key (Mac) to rotate. Clicking this button will centre the rotation around the mesh, which will make it easier to navigate in the perspective view.
Tracking the Object
The model is now ready for tracking. If you prefer, you can change the Render style back to something different. Use the appropriate tracking buttons to track forwards and backwards.
As the track progresses, you can see how the mesh model sticks to the actress’ face and follows her every move. Make sure you have tracked all the frames.
04. Deformable Geometry Tracking
Instead of only tracking the overall motion of the actress’ head, we can go a step further and track the actress’ performance by allowing the mesh to deform as she speaks. To do so, we must create deformable tracking groups of mesh triangles, and specify how these groups can transform relative to the rest of the mesh.
If you have followed the steps of the previous chapter in a copy of the Geometry Track node, switch back to the original Geometry Track node by double clicking on it.
Creating Deformable Groups
You can create deformable tracking groups in the Deformable Groups tab. Click the Create button to create the first group.
You can double click in the Name column to rename your group. This group will be covering the actress’ jaw, so change the group’s name to Jaw.
Adding Triangles to a Group
Add all triangles of the lower jaw region, but do not include the lip. You can erase triangles from a group by clicking the button and painting over the triangles you would like to remove. The Radius slider controls the thickness of the brush.
Make sure to check in the perspective view for any areas that are hard to reach in the clip view, like the back of the chin.
Whilst adding triangles, you will have to hold the Ctrl-key (Windows, Linux) or Command-key (Mac) to rotate. Clicking this button will centre the rotation around the mesh, which will make it easier to navigate in the perspective view.
Check the Lock column to prevent the group to move at all during tracking, or any of the T, R and S columns to allow the group to translate, rotate or scale along the x, y or z axis respectively. You can adjust the pivot for these transformations by selecting the Move Pivot or Rotate Pivot buttons, but this will not be necessary for this tutorial.
Allow the Jaw group to translate in x and y, as well as rotate around x.
Creating the Remaining Group
Click the Help button to open the reference manual page for the Geometry Track node.
The page shows some example groups for a face track similar to the one in this tutorial (and uses the same mesh model). Add the remaining groups as outlined in the help page. Don’t forget to check the allowed transformations for each group.
You can change a group’s colour by double clicking the Colour column and selecting a new colour.
Tracking the Deformable Groups
With all the groups defined as described in the manual, you can now start tracking by clicking the appropriate tracking buttons to track forwards and backwards.
As the track progresses, you can see the whole mesh moving to accommodate the overall motion of the actress’ head, as well as all the deformable groups being tracked independently. Make sure you have tracked all the frames.
When exporting deformable meshes to FBX in an Export node, a bone skeleton and skin deformer will automatically be generated in the exported file.
In this tutorial you have learned how to use geometry to track an object. Tracking with geometry can be useful when it is not possible or not desireable to use multiple tracking points. Additionally, the mesh can deform to not only capture the overall movement of an object, but also deformations that may be happening in a clip.