2.3 Using Multiple Views to Solve Nodal Pans
This tutorial is part of Level 2. Extended Matchmoving in PFTrack of The Pixel Farm’s Training Course for PFTrack. Find out more and register for the next available live class.
In the Extended Matchmoving in PFTrack (overview) class, we have solved two moving cameras into one scene for the purpose of creating a consistent coordinate system for multiple cameras. There is one special kind of camera motion, however, that requires additional views to extract any kind of 3D: Nodal pans. This tutorial explains what nodal pans are and how they are different from other camera movements. After that we will use the same technique shown in the live session to add additional camera views of the same scene in order to obtain a real 3D result.
– Solving Nodal Pans in PFTrack
– Create a Tracker
– Track the User Tracker
– Place the Tracker in the Reference Images
– Create Additional Trackers
01. What is a Nodal Pan?
The easiest way to imagine a nodal pan is a camera that is mounted on a tripod with no horizontal movement. Such a motion provides no parallax for tracked features, no matter how close or far away they are from the camera. Without parallax, a camera solver cannot determine the distance of trackers. As a result, all solved features will appear the same distance from camera.
Solving Nodal Pans in PFTrack
Luckily, most tripods shots are not true nodal pans. For a nodal pan to produce absolutely no parallax, the centre of rotation of the tripod would have to be the optical centre of the camera. This is normally only the case for specialist tripods and gimbals.
The Translation parameter in the Camera Solver node in PFTrack has a special setting just for tripods that are rotating around a point slightly offset from the camera’s true optical centre, Off-Centre. However, for a real nodal pan rotating around the optical centre, you would set Translation to None.
Using Off-Centre still provides useful 3D information for many tripod shots. However, for true nodal pans, using additional views, as explained in this tutorial, may be the only way to get 3D.
02. Using Reference Images to Track Nodal Pans
Import the clip PFTChurchyard_Pan, as well as the two reference images PFTChurchyard_ReferenceA and PFTChurchyard_ReferenceB into your PFTrack project and connect them to a User Track node.
Other than during the live training, we are going to use the reference images to help with solving the main camera. Please note that additional views, which are meant to assist solving the main camera, have to be single still images, or static, locked-off clips without any camera movement. However, the workflow of identifying and tracking common features in all views is the same that has been demonstrated during the live workshop.
Create a User Tracker
Click the Create button and click on the white patch on the rectangular gravestone to create the first tracker.
Then adjust the size of the inner tracking window and the outer search area.
Click the Help button to open the User Track reference help page for an example of how to place a good tracker and choose a good size for the tracking window.
Track the User Tracker
Press the Centre View button. This will keep the currently selected tracker in the centre of the Cinema, which makes it easier to judge the quality of the track whilst tracking. Then click the >> button to track the tracker forwards.
Depending on your placement of the tracker, tracking may stop before it reaches the end of the clip. This is because the matching score, a quality value that indicates how well the current frame matches the tracker’s reference pattern, is too low. The Failure threshold parameter defines when a matching score is considered too low. If necessary, adjust the positioning of the tracker. After manually repositioning a tracker, click Polish to smooth out any jumps that might occur after repositioning. In difficult situations, it sometimes can be helpful to click the > button to track a single frame forwards. Continue tracking until you have reached the end of the clip.
Place the Tracker in the Reference Images
You can use the Current clip menu to select between the different inputs of the User Track node. Select the first reference image, PFTChurchyard_ReferenceA, from the menu.
Locate the same white patch on the gravestone in the Cinema and click to place the tracker at the same spot. If necessary, adjust the tracker’s placement in the Tracking Window.
At this stage, your trackers table should only list a single tracker, Tracker0001, with PFTChurchyard_Pan and PFTChurchyard_ReferenceA listed in the Clips column. This confirms that the same tracker has been placed in both the clip and the reference image.
You have to make sure that a tracker is selected before trying to place it in another clip. If it isn’t, you will create a new tracker instead. If this happens, you can easily merge them into one tracker by selecting both (hold the Ctrl key (Windows/Linux) or Command (Mac) key to multiple select), and clicking the Merge button.
Create Additional Trackers
Click create and place the next tracker at the top of the triangular shaped gravestone.
Click the >> button to track forwards. The feature moves out of frame from frame 45 onwards, so tracking should stop automatically at frame 44. Despite tracking having stopped, the tracker is still active in frames 45 to 50, as is indicated by the dark red marks on top of the scrub bar (the blue marks before frame 45 indicate that the tracker is active and has been tracked).
This tracker doesn’t provide valid information for the final 6 frames, so it is important to make sure that it is not contributing to the camera solve for these frames. You can do this by hiding the tracker for these frames.
Whilst still at frame 44, click and confirm the H+ button to hide the tracker for all frames after the current one.
Other ways to hide a tracker are the Hide button, which hides the selected tracker in the current frame only, and the H- button, which hides the tracker for all frames before the current frame.
Place the tracker in the reference frames as well, then continue to create additional trackers as shown in the screenshot below.
If you have difficulties placing your trackers, or would like to compare your trackers to the ones used in this tutorial, the PFTChurchyard_Trackers.txt file in the PFTChurchyard_Trackers.zip archive contains the tracker locations used in this tutorial. Simply click Import in the User Track node and open the file to load these trackers into your node. Before loading the trackers from the file, make sure to have the clip and reference images connected to the node as outlined in this tutorial, with the movie clip connected to the first input, PFTChurchyard_ReferenceA to the second and PFTChurchyard_ReferenceB to the User Track’s third input.
03. Solve the Scene Using the Reference Frames
Create a Camera Solver and connect all three outputs from the User Track node.
Use the Current clip menu to switch to one of the reference frames. As these are single frames, the Helper checkbox has been checked automatically. This confirms that these inputs will be used to assist the solve of the main camera. Check out the Helper Frames section in Camera Solver’s reference help page to read more about using reference images as helpers.
Click Solve All to solve the scene.
Once the solve is completed, you can review the scene in the perspective and orthogonal views. Notice how the trackers are at the correct distance from the camera, and that the main camera as well as the location of the still images has been solved into the scene.
The scene is now ready to be used further in the tracking tree and then exported.
04. Getting 3D Without Reference Frames
Instead of using reference frames, an RGBD camera, which records depth and colour information in synchronisation, can be invaluable when solving nodal pans, as it provides the missing depth data caused by the lack of parallax in the scene.
PFTrack can harness the depth maps recorded by an RGBD camera through the Attach Z-Channel Node, assigning z-depth values to trackers created in Auto Track and User Track nodes. The advantages of this technology even beyond nodal pans are shown in this overview video.
In this tutorial you have learned why nodal pans are a difficult challenge in matchmoving. You have met this challenge by adding additional views to your camera solve in order to extract 3D, when the main clip is lacking parallax.