PFTrack: Geometry Tracking
In this tutorial you are going use PFTrack’s Geometry Track node to track the performance of an actress. Instead of tracking points, the Geometry Track node uses a triangular mesh model of the object that is being tracked. This can be useful when it is not possible or not desirable to use multiple tracking points. In this tutorial, you are going to use the the Geometry Track node to track a moving object in a clip with a static camera. However, the Geometry Track can also be used to track a camera.
– Create the Node
– Camera Motion or Object Motion
– Changing The Render Style
– Creating Deformable Groups
– Adding Triangles to a Group
– Defining Allowed Transforms
– Add The Remaining Groups
To learn this tutorial you will need to download and use the footage below.
01. Importing and Reviewing Clips
Open the File Browser, and navigate to the tutorial clip on your hard disk. Then drag and drop the thumbnail into the Tree View. Play through the clip to review. It is a dialog shot with a static camera. The actress is the focus of attention. Stop playback and go back to the first frame.
02. The Geometry Track Node
Creating The Node
Make sure the clip node is selected, then right click the clip node and select Geometry Track from the menu.
Camera Motion or Object Motion
The Geometry Track node can be used to either solve for camera motion or for object motion. In this tutorial, you are going to solve for the movement of the actress’ head, which is object motion, rather than camera motion, so check the Moving object checkbox. If you already had a solved camera in your tree, Moving object would be checked automatically.
03. Importing The Tracking Geometry
The Geometry Track node uses a geometric model that should be a good approximation of the object that is being tracked, but doesn’t need to be exact. These models can be imported from either OBJ, FBX or Open Alembic files, as well as an internal file format. To import the model, click the Load button, then select face.pfm in the file browser.
The geometry faces away from you directly after import, so you will only be able to just make out its edges. Click the View button to position the geometry in front of the camera.
04. Positioning The Geometry
The geometry has to be positioned manually in one frame before it can be tracked. You can use the Transform mode menu to position the geometry. Select either Translate, Rotate or Scale as the Transform mode and use the manipulators to transform the mesh. Instead of doing those transformations individually, you can also select Fly. When in Fly mode, hold down the Alt key on Windows or Linux, or Option key on the Mac, and use the left mouse button to rotate the model, the middle mouse button to scale the model, and the right mouse button to translate.
Changing The Render Style
The Render style menu controls how the mesh is being displayed in the Cinema. Selecting Wire-frame or Hidden-line will let you see what is behind the model to help align it. Use the eyes, nose, mouth and cheeks as guides to position the geometry in a way that it fits the actresses face.
05. Vertex Weights
Scrub through the clip again as reminder of what you are dealing with today. While the geometric model is rigid, the actress starts talking which means her mouth and chin move. For cases like these, you can change how much the affected vertices should contribute to the solution by changing their weights. Go to the Vertex Weights tab and click the Paint button. The Render style for the mesh changes to Weights and the even green colour indicates that all vertices have the same weight. Use the left mouse button to paint over the lower part of the model to decrease the weights in this area.
06. Tracking The Object
Now the model is ready for tracking. To get a better look at the track, change the Render style back to Hidden-line and turn off Show Ground and Show Horizon in the Display box. Then click the >> button to start tracking. As PFTrack is tracking through the clip, you can see how the model sticks to the actresses face and follows her every move.
If necessary, it is possible to stop tracking at any time by clicking the Stop button, or pressing the Escape key, and adjust the position of the geometry in the same way as with the initial placement. This will create a keyframe for the geometry. Yellow markers in the scrub bar show the position of all keyframes.
Manually adjusting the geometry often leads to jumps in the motion, compared to the previous frame. Click the Refine button to refine the motion between two keyframes. You can find out more about adjusting your track in the Geometry Track reference page, which will open in your web browser when you click the Help button.
Click the >> button again to continue tracking.
Once tracking has reached the end of the clip, open the perspective view and play through the clip to review it.
07. Tracking The Performance
The scene is now ready for export. However, you can even go a step further and track the actresses performance by deforming the mesh as she speaks. Instead of creating a brand new Geometry Track node for that, you can re-use the initial placement of the Geometry by copying and pasting the existing node. Make sure the node is still selected, then click the copy button, and the paste button right next to it. Connect the new node to the clip.
Please note that even though you have done so in this tutorial, it is not necessary to do a static object track first before using deformable groups.
08. Clearing Vertex Weights
The copied Geometry Track node also includes the vertex weights you have assigned earlier. Select Weights as the Render style to confirm that. This time, you want our mesh to deform according to how the actress moves her mouth, rather than ignore it, so click and confirm Reset in the Vertex Weights tab to clear the weights.
09. Defining Deformable Groups
Creating Deformable Groups
You can track the actresses performance by defining which parts of the mesh are allowed to transform independently from each other. This is done by creating deformable tracking groups and assigning triangles of the mesh to those groups. Change the Render style to Facet-shaded to better see the triangles in the model. Then switch to the Deformable Groups tab. Then click the Create button to create a group. You can double click in the Name column to re-name your group. Change the group’s name to Jaw.
Adding Triangles To a Group
Click the Add To button. You can now add triangles to the current group by painting directly in the Cinema. Selecting Remove From removes painted triangles from the current group. You can adjust the Radius for more fine grained control. You can also paint in the perspective view. This is helpful to include triangles that may be hidden in the current frame. Whilst Add To is selected, hold the Alt or Option key to rotate the perspective view to reach the triangles at the back of the chin. Clicking this button will centre the rotation around the mesh, which will make it easier to navigate in the perspective view. Make sure all bottom triangles of the mesh are included in the current group, then rotate the perspective view back to view the front of mesh. For this group, we want to include all triangles of the jaw area, but not the bottom lip.
Defining Allowed Transforms
For each group you can control how it is updated during tracking by checking these columns. Check the Lock column to prevent the group to move at all during tracking or any of the T, R and S columns to allow the group to translate, rotate or scale along the x, y or z axis respectively. You can adjust the pivot for these transformations by selecting the Move Pivot or Rotate Pivot buttons. For this tutorial, allow the Jaw group to translate in x and in y, as well as to rotate around x.
Adding The Remaining Groups
Click the Help button to open the help page for the Geometry Track node. The page shows some example groups for a face track similar to the one in this tutorial. Add the remaining groups, as outlined in the help page. Don’t forget to check the allowed transformations for each group. You can change a group’s colour by double clicking the Colour tab and selecting a colour.
10. Tracking Deformable Groups
With all the groups defined as described in the manual, you can now start tracking by clicking the >> button. As with your first track, the whole mesh transforms to accommodate the actresses head’s moving. But on top of that, all the deformable groups are being tracked independently, and transferred onto the mesh. If you didn’t start tracking from frame 1, go back to the frame where you started tracking and click << to track backwards. Once you have tracked the whole clip play it back to review the result in real time.
Your scene is now ready to export. Create an export node and make sure it is connected to the Geometry Track 1 node. You will find our tracked geometry in the Objects tab in the Export node. When exporting deformable meshes to FBX, a bone skeleton and skin deformer will automatically be generated in the exported file. Click Export Scene to export the scene.