Tracking 360º Equirectangular Clips in PFTrack
In this tutorial, you are going to track and solve an equirectangular panorama using PFTrack’s new spherical tracking toolset.
– Mask out the Operator
– Track the Clip
– Solve for the Camera
– Initial Orient and Review
– Orient the Camera
– Other Uses for the Spherical Orient Node
To learn this tutorial you will need to download and use the footage below.
01. Import and Review Clip
First you need to create a new project. Then navigate to the tutorial clip in the File Browser and drag and drop the thumbnail into the Tree View. Play through the clip to review.
02. The Spherical Track Node
To use equirectangular 360º footage in PFTrack, you will need a Spherical Track node. Create the node and connect it to the clip. The Spherical Track has one input, which requires a 360º equirectangular image. The node’s six outputs represent one camera for each planar face of a cube map generated from the input panorama.
Mask out the Operator
One of the consequences of 360º panoramas is that there is no ‘behind the camera’ so the operator holding the camera is always visible in the clip. To prevent this from interfering with the tracking, you are going to create a mask to exclude the operator.
Click the Mask button to open the mask panel. A simple roto mask will be sufficient so click Bezier Roto and start creating vertices by left-clicking in the Cinema. Draw around the operator and his shadow, and close the mask by clicking on the first vertex again. Scrub through the clip to confirm that the mask is covering the operator throughout the clip. Then click the Parameters button to return to the Spherical Track.
Track the Clip
Leave the Feature Tracking parameters unchanged. If you want to know more about these parameters, you can click the Help button to open the node’s reference help page in your file browser.
Click Start to start tracking.
Once tracking is completed you can review the tracks by playing or scrubbing through the clip. All the trackers should have tracked well. If your mask didn’t fully cover the operator in all frames and you find rogue trackers, you can select and delete them. Refer to the node’s help page for more information.
Solve for the Camera
Now it’s time to convert the tracker motion into camera motion. Select Low in the Inlier Threshold menu and click Start. The Inlier Threshold controls how much tracking error will be accepted for the solve. Low is the strictest setting, using only the highest quality trackers, but in some situations, especially when your clip suffers from parallax or stitching errors, you might need a higher threshold to get a useable solution.
Initial Orient and Review
Click Marquee and select some features that have been tracked on the ground. Select X-Z Plane from the Orient Ground menu. This will orient the scene in such a way that the selected features lie on the ground plane. Then open the perspective window. As you scrub through the clip you can see the 360º camera moving along the solved path. Note, that it does not have a camera frustum.
03. Orient the Scene
Next you are going to refine the orientation of the scene. Close the perspective view again and create an Orient Scene node. Connect all outputs of the Spherical Track to the new node.
You can change which of the 6 cameras to view in the Current clip menu. The PFTChurch360_Left camera is facing the church. In that view, select the feature at the bottom of the gravestone and click Set Origin. Select Rotate in the Edit Mode menu and rotate the ground plane so the red x-axis faces right. Then select Scale as the Edit Mode and scale the scene until the gravestone is 2 squares wide. When you are done switch the Edit Mode back to None.
04. The Spherical Orient Node
The Spherical Orient node can be used to adjust the orientation of the virtual camera rig. Create a Spherical Orient node and make sure all of the Spherical Tracker’s outputs are connected to the corresponding inputs. This should happen automatically if the Spherical Tracker node was selected when you created the node.
Orient the Camera Rig
Select PFTChurch360_Front in the Current clip menu. The Cinema is now showing the front camera’s view, but the left camera is the one facing the church. To make the front camera face the church, the virtual camera rig needs to be rotated by 90º counter-clockwise. Type –90 in the Y column of the Rotation row and press enter. Then open the perspective view.
You can further adjust the camera’s orientation by clicking Orient and using the manipulators in the perspective viewport. When you are done, de-select Orient again.
Other Uses for the Spherical Orient Node
The Spherical Orient node also lets you use animated look-at targets for cameras, as well as stabilise camera rotation. Click the Help button and have a look at the node’s reference help page to find out more.
05. Test the Solution
It’s now time to test the solution. Close the perspective view and create a Test Object node. Connect all 6 outputs to the new node. Double click the Cow entry in the Available objects list to add a cow model to the scene. Select Rotate in the Interaction mode menu to rotate the cow so it is facing the camera. Then select Scale to increase its size. Finally select Place at selected feature and click on a feature in the Cinema to position the cow. Scrub through the clip. The cow should stay in position as if it was part of the original scene.
You can also position test objects in the other views. After you have added all the objects, de-select Show trackers, Show ground and Show horizon so you can focus on the test objects.
06. Export the Scene
The scene is now ready for export. Create an Export node and connect all outputs to the node. This will export the virtual rig of 6 cameras to disk.
07. Create a New Panorama
You can also create an equirectangular panorama from your 6 cameras with the Spherical Render node. Create the node and connect the 6 outputs from the Test Object node. You can see the panorama being built as you connect each output. Double-click the PFTChurch360 Clip Input node to compare the new panorama with the original. The differences are from your adjustments in the Spherical Orient node. You can also check Bake objects in the Test Object node to see them in the new panorama.
In this tutorial you have learned how to track and solve a 360º equirectangular panorama in PFTrack. Once solved, each of the 6 cameras can be used like any other camera in PFTrack. For example, you can create a triangular mesh from one or more views with the Photo Mesh node, or track an object using geometry tracking or the Object Solver. You have then oriented and exported the virtual camera rig, as well as created a new panorama from the result.