3.2 Tracking a Deformable Object with Geometry

11 Jul, 2017 | PFTRACK, TUTORIALS

This tutorial is part of Level 3. Advanced Matchmoving Strategies in PFTrack of The Pixel Farm Training Academy’s PFTrack course. Find out more and register for the next available live class.

During the Advanced Matchmoving Strategies live class we used a geometric model of an immovable structure on set to track a camera in PFTrack. PFTrack’s Geometry Tracking node, however, is not limited to tracking cameras, and can also be used to track moving objects.

This tutorial will guide you through the necessary steps of tracking the head of an actress. To go even further, we will allow PFTrack to deform the tracking geometry to capture the actress’ performance in the clip.

Contents

01. Setting Up the Scene

Setting Up the Camera in the Geometry Track Node

02. Positioning the Tracking Geometry

Importing the Geometry

Positioning the Geometry

Changing the Render Style

03. Rigid Geometry Tracking

Copying the Geometry Track Node

Using Vertex Weights

Tracking the Object

04. Deformable Geometry Tracking

Creating Deformable Groups

Adding Triangles to a Group

Creating the Remaining Group

Tracking the Deformable Groups

05. Conclusion

Tutorial Footage

To learn this tutorial you will need to download and use the footage below.

Footage: PFTDialog.zip

Download

01. Setting Up the Scene

Import the clip into a PFTrack project and play through it. Before any object can be tracked, the camera for the clip must be solved. As the camera for the clip is static you can define it in an Edit Camera node, which you can find in the Utilities category, instead of using trackers and a Camera Solver node. However, for the purpose of tracking an object in a Geometry Track node for a clip with a static camera, even this step could be omitted, as a static camera can be created in the Geometry Track node itself.

Setting Up the Camera in the Geometry Track Node

In the Geometry Track node, check the Moving object checkbox. This tells PFTrack that we are going to track a moving object, and since no camera exists in the tree yet, a static camera will be created.

If a static or moving camera already exist in your tree, the Moving camera checkbox will be checked automatically.

02. Positioning the Tracking Geometry

With the camera and scene set up, the next step is to import and position the tracking geometry in one frame.

Importing the Geometry

Geometric models can be imported into the Geometry Track node from either OBJ, FBX or Open Alembic files, as well as an internal file format. The PFTDialog.zip archive contains a suitable model for this clip in PFTrack’s internal format, face.pfm.

Click the Load button, then select face.pfm, which you should find where you unzipped the PFTDialog archive, in the file browser dialog.

Initially, this particular model faces away from you, so you will only be able to just make out its edges.

Click the View button to reposition the geometry in front of the camera.

Positioning the Geometry

The model has to be positioned manually in one frame before it can be tracked. Select the appropriate mode, such as Translate, Rotate or Scale in the Transform mode menu to do this. Instead of doing those transformations individually, you can also use the Fly mode. When in Fly mode, hold down the Alt-key on Windows or Linux, or Option-key on the Mac, and use the left mouse button to rotate the model, the middle mouse button to scale and the right mouse button to translate the model.

Changing the Render Style

Changing the way the model is rendered on screen might help you position the geometry. Selecting Wire-frame or Hidden-line will only display the edges, rather than faces of the geometry, and lets you see what is ‘behind’ the model.

Use the eyes, nose, mouth and cheeks as guides to position the geometry in a way that fits the actress’ face.

03. Rigid Geometry Tracking

If you only want to track the overall motion of the actress’ head, you could now do so by clicking any of the tracking buttons shown below.

The aim of this tutorial is to track the actress’ performance by making the mesh deformable, as is described in the next chapter. In this chapter, we will perform a rigid track of the actress’ head movements. If you would like to follow the steps in this chapter, you can do so with a copy of the node.

Copying the Geometry Track Node

Make sure the Geometry Track node is selected, then click the copy T3_2_copy_track and paste T3_2_paste_track buttons below the Tree View. Connect the newly pasted node to the clip and double click to make it the active node.

Using Vertex Weights

While, when tracking this way, the geometric model stays rigid, the actress starts talking, which means her mouth and chin will move. For cases like these, you can change how much the affected vertices (the ones covering the lower part of her face) should contribute to the overall solution. You do this by changing their weights.

Go to the Vertex Weights tab and click the Paint button.

The Render style for the mesh will automatically change to Weights, and the even green colour indicates that all vertices have the same weight and contribute fully to the resulting track.

Use the left mouse button to paint over the lower part of the model to decrease the weights in this area. You can paint in the clip view or perspective view.

Whilst painting the jaw, make sure to check in the perspective view for any areas that are hard to reach in the clip view, like the back of the chin.

When in Paint mode, you will have to hold the Ctrl-key (Windows, Linux) or Command-key (Mac) to rotate. Clicking this T3_2_rotate_track button will centre the rotation around the mesh, which will make it easier to navigate in the perspective view.

Tracking the Object

The model is now ready for tracking. If you prefer, you can change the Render style back to something different. Use the appropriate tracking buttons to track forwards and backwards.

As the track progresses, you can see how the mesh model sticks to the actress’ face and follows her every move. Make sure you have tracked all the frames.

04. Deformable Geometry Tracking

Instead of only tracking the overall motion of the actress’ head, we can go a step further and track the actress’ performance by allowing the mesh to deform as she speaks. To do so, we must create deformable tracking groups of mesh triangles, and specify how these groups can transform relative to the rest of the mesh.

If you have followed the steps of the previous chapter in a copy of the Geometry Track node, switch back to the original Geometry Track node by double clicking on it.

Creating Deformable Groups

You can create deformable tracking groups in the Deformable Groups tab. Click the Create button to create the first group.

You can double click in the Name column to rename your group. This group will be covering the actress’ jaw, so change the group’s name to Jaw.

Adding Triangles to a Group

New we need to add the mesh triangles which represent the actress’ jaw to the group. With the group selected, click the T3_2_plus_button button and paint over the triangles you would like to add to the group. You can paint in Cinema and the perspective view.

Add all triangles of the lower jaw region, but do not include the lip. You can erase triangles from a group by clicking the T3_2_minus_button button and painting over the triangles you would like to remove. The Radius slider controls the thickness of the brush.

Make sure to check in the perspective view for any areas that are hard to reach in the clip view, like the back of the chin.

Whilst adding triangles, you will have to hold the Ctrl-key (Windows, Linux) or Command-key (Mac) to rotate. Clicking this T3_2_rotate_track button will centre the rotation around the mesh, which will make it easier to navigate in the perspective view.

For each group you can control how it is updated during tracking by checking these columns. T3_2_controls_track

Check the Lock column to prevent the group to move at all during tracking, or any of the T, R and S columns to allow the group to translate, rotate or scale along the x, y or z axis respectively. You can adjust the pivot for these transformations by selecting the Move Pivot or Rotate Pivot buttons, but this will not be necessary for this tutorial.

Allow the Jaw group to translate in x and y, as well as rotate around x.

Creating the Remaining Group

Click the Help button to open the reference manual page for the Geometry Track node.

T3_2_help_button

The page shows some example groups for a face track similar to the one in this tutorial (and uses the same mesh model). Add the remaining groups as outlined in the help page. Don’t forget to check the allowed transformations for each group.

You can change a group’s colour by double clicking the Colour column and selecting a new colour.

Tracking the Deformable Groups

With all the groups defined as described in the manual, you can now start tracking by clicking the appropriate tracking buttons to track forwards and backwards.

As the track progresses, you can see the whole mesh moving to accommodate the overall motion of the actress’ head, as well as all the deformable groups being tracked independently. Make sure you have tracked all the frames.

When exporting deformable meshes to FBX in an Export node, a bone skeleton and skin deformer will automatically be generated in the exported file.

05. Conclusion

In this tutorial you have learned how to use geometry to track an object. Tracking with geometry can be useful when it is not possible or not desireable to use multiple tracking points. Additionally, the mesh can deform to not only capture the overall movement of an object, but also deformations that may be happening in a clip.

Related Posts

PFTrack: Photogrammetry

In this tutorial, you will be solving the tutorial clip using PFTrack’s photogrammetry tool set and then add a second camera to the scene. This will be done by creating a point cloud from a set of still images and then automatically solving cameras into the scene…

Read More

2.2 Creating a Triangular Mesh from Still Images

In the Extended Matchmoving in PFTrack class we have used PFTrack’s photogrammetry tools to help us solve multiple camera motions into a single scene. These tools can also be used to create textured triangular meshes. This tutorial will walk you through the necessary steps to create a mesh from the scene created in the live training.

Read More

Creating a Triangular Mesh in PFTrack

In this tutorial you will learn how to create a mesh model from still images with PFTrack’s Photo Mesh node. After creating the dense triangular mesh, you will simplify its geometry to pass the model downstream to create a texture in the Texture Extraction node.

Read More

Unity Project: Utilising PFTrack’s New Photogrammetry Toolset

The month of August was set aside for research and development within the walls of The Pixel Farm. New features have been added to PFTrack’s already powerful toolset, making the Photo Mesh node more versatile and more relevant to an unprecedented number of industries…

Read More
Share This

Share This Post On: