3.3 Stabilizing Footage Using Tracking
In our previous extended study material Level 3.2 Creating Custom Standards, we covered using an text editor to create custom standards for our imported footage. Find out more about The Pixel Farm Training Academy and register for the next available class here.
In this tutorial we will be looking at a couple of clips and using the tracking panel in the Workbench to assist us in our restoration.
If you haven’t completed the previous level 3 tutorial, or level 2 training, I would suggest going back and starting from there as some of things we will be covering build on the knowledge we acquired in the previous level.
NOTE: To follow along in this tutorial, you will need access to the clips provided for the class. If you don’t have them to hand, below you will find the link to download the footage again.
To learn this tutorial you will need to download and use the footage below.
Footage: Footage Name / Footage Info
01. Loading the Clips
Let’s play through both clips and see what the issues are starting with clip_003. Apart from the usual dirt, dust and scratches we can see both clips suffer from a stability problem.
In clip_003 we have a snatching issue causing the frame jump then fall back into alignment. In clip_004 we have a frame bump at the beginning of the clip. Normally these might occur where there is a shot transition and the bump is the first frame after the splice.
In this tutorial we are going to look at a number of ways to fix this, firstly demonstrating the existing automatic and manual tools, then we are going to look at using tracking to assist in stabilizing clip and finally the use of the F-Curve editor to refine the results.
02. Using Auto Stabilize
The Auto Stabilize effect will automatically start working the moment it is added to the stack and is used to remove high frequency fluctuations in frame position throughout the clip.
Let’s click on the playblast button and the and see what effect this has had on clip_003. What we can see is the Auto Stabilize has affected the clip in a positive way. All the unwanted high frequency movement has been removed included the bump at the front of the clip, however the low frequency gate movement remains.
03. Using the Stabilize Effect
The Mode menu specifies the type of motion that will be removed. For most situations, Translation, Translate and Rotate, Horizontal or Vertical mode can be used to remove horizontal and/or vertical translation. For small-scale rotation and scale changes, the Affine model can be used (6 degrees of freedom), and Perspective model can be used for full planar-perspective motion (8 degrees of freedom).
The frame window control specifies how many frames on either side of the current frame will be used to smooth out motion fluctuations. The default value is 5, which corresponds to averaging using 5 preceding frames and 5 following frames. This option will be greyed out when using Lock Motion like we will be in this tutorial.
controls sensitivity to differences in pixel intensity that are assumed to be caused by motion (the default value is 25%). Pixel intensity differences above this value are assumed to be caused by incorrect motion estimates and are therefore ignored during the stabilize calculations.
specifies the size of the border around the clip frame that is used to estimate the stabilize transformation. Increasing this value will include more of the frame (the current border area is drawn in purple in the viewer window).
is used to specify a frame that we want to lock too. For example if I go to a specific frame in the timeline and press the Target Frame button no stabilising will happen on this frame as it will be used as reference frame but all other frames will be compared and locked to it. If unselected a estimation based on all the frames will be made instead. If we no longer want to use the target frame we can click the Clear button.
Clicking the Sub Pixel button will ensure that motion estimation is performed at sub-pixel accuracy. Note that this means the final image may be re-sampled when rendering. With the sub pixel option disabled, the image will only be stabilized to the nearest pixel, meaning that no re-sampling will occur.
As well as stabilizing over a border region, specific areas can also be drawn by pressing the Add Area button and clicking and dragging with the left mouse button to place an area in the Cinema. When one or more areas are placed, the stabilizing transform will be estimated using only the pixels inside the areas.
Pressing the Marquee button will allow a selection rectangle to be drawn in the Cinema to select areas. All/none of the areas can be selected at the same time by pressing the All/None button. Selected areas can be deleted by pressing the Delete Areas button.
04. Stabilizing the Clip
For the purposes of this tutorial we will be leaving most of the parameters at the default positions. One parameter we will be changing is the Lock Motion tick box, as we do not want any motion in the frame, let’s go ahead and click in the tick box to turn it on. We will be leaving the Mode option as Translation, we could use Trans & Rotate as there is a minor amount of rotation but for the purposes of this tutorial we will use the default.
05. Adding a Tracker
Click on the Reset Motion button in the Stabilize panel to reset the Stabilize effect. We are now going to use Tracking to improve our results.
Trackers can be used to perform various tasks in PFClean, such as assisting with Stabilization or Rig Removal, animating masks and paint strokes, or providing hints to the motion analysis toolset.
We can now see a new tracker has appeared in the panel on the left showing, the name of the tracker, it’s In point and Out point, whether it’s active signified by the X tick box and finally what effect the tracker is active with. In our case it is active with the Stabilize effect. Tracking can also be used with Rig Removal, animating masks and paint strokes, or providing hints to the motion analysis toolset
We can use the trackers to help the stabilizing of a clip in the following ways.
For Translation, Horizontal and Vertical modes, a single tracker is required, although if more are present then their average motion will be used.
For Trans & Rotate mode three trackers are required. Again, if more are present then the average motion will be calculated.
For Affine and Planar Perspective modes, at least 4 trackers are required. When there are not enough trackers available, modes will appear greyed-out in the Mode menu.
We will be using 3 trackers for this clip, but let’s concentrate on a single tracker for now.
06. Positioning a Tracker
When we first add a new tracker into the effect the default position will be centered in the cinema.
The outer dotted line around the tracker represents the search area, and the inner box the actual tracking window. The contents of the tracker window are also shown in the Tracking Window panel.
The tracker displays its current X/Y position and similarity value, which corresponds to the tracking accuracy in the current frame (a value of 1.0 indicates “full accuracy”, and will appear whenever the tracker is positioned by hand).
The tracker can be positioned by clicking and dragging with the left mouse button inside the search area in the Cinema, or by clicking and dragging inside the Tracking Window panel. The edges of the search area and tracking window can also be adjusted in the Cinema using the left mouse button. Currently the center of the tracker is over the actor. If we started to track from here the tracker would lock onto and follow the actor. This might be desirable if we were creating a mask but for this tutorial we are wanting to lock the image down.
Let’s play through the clip and hunt for some static areas in the clip preferably with nothing passing in front of or obstructing the view. We want to look for areas of high contrast to provide the best chance of a solid track.
Each tracker will have it’s own set of parameters we can adjust.
The Channels buttons describe which of the red, green and blue channels will be used to track. In situations where one or more channels are significantly corrupted by noise or grain, they can be switched off to improve tracking accuracy.
The Window Update buttons are used to define how the ‘reference tracking window’ is updated during tracking. The default option is Starting Frame, which indicates that the reference window is only updated when tracking starts. Other options are Every Frame to update the window at every frame, or Above Tolerance, which will only update the reference window when the similarity is above the Window Tolerance. The Window Update options can be useful when track large trackers in situations where there are moderate perspective effects or rotation/scaling.
The Failure Mode buttons describe the behaviour of the tracker when the similarity between the ‘reference tracking window’ and the current tracking window is below the Failure Tolerance level. The available actions are Stop Tracking, or Predict Position which will attempt to predict the position of the tracker given its positions in previous frames. This can be used to help tracking features that are partially occluded in some frames.
The Tracking Offset parameter contains the current offset between the tracking point and the tracking window. It can be adjusted using the available edit boxes or by holding the Command key whilst dragging the tracker position in the Cinema window.
When the Blur Window or De-Flicker buttons are pressed, the contents of the tracking window will be blurred or de-flickered before the similarity is measured and compared against the Failure Tolerance. This can be useful in clips that contain significant noise or grain, or brightness fluctuations. Note that these changes will not be apparent in the Tracking Window panel of the interface, but are instead applied internally where necessary.
For this tutorial we are going to leave the settings at their defaults.
07. Tracking the Clip
Now we have positioned our tracker we are ready to begin tracking the clip. Below the tracker list on the left hand side of the tracking panel there are controls for tracking backwards, tracking backwards by one frame, tracking forwards by one frame and tracking forwards. When using tracking forwards or tracking backwards the tracker will continue to track until it either reaches the end of the shot or can no longer track due to tolerance failure if using Stop Tracking under the Failure Mode. You can also stop the tracking by pressing the esc key on the keyboard. If there are already keyframes in the clip the tracker will also stop before them.
Making sure we are on the first frame of the shot let’s click the track forwards button to track our clip. Once it has finished tracking let’s step through the frames using the left and right arrow keys on the keyboard and see if it has locked onto the target successfully, we can do this by watching the tracking window or zooming into the area that has been tracked in the cinema.
We can see this has been a successful track and no further adjustment needs to be made.
If we go back to the Stabilize effect by pressing the tracker panel button and click on the tracking Mode we can see some options are greyed out including the one we want to use called Trans and Rotate. As described earlier this is because we need three separate trackers in order for Trans and Rotate option to become available.
08. Adding Additional Trackers
Let’s go back to the tracker panel making sure we are on the first frame of the shot and add in two additional trackers by clicking the New button twice. In the tracker list now we should have two more trackers named Tracker0002 and Tracker0003.
Let’s select Tracker0002 in the list by clicking on it and as we did before let’s position the tracker in the frame where there should be no movement and it won’t be occluded. We want to get a fairly even spread of trackers over the frame so let’s pick an area not too close to the first tracker.
I’ve adjusted my tracker window to include an area of detail on the top left to help with the tracking. As we did before and making sure we are on the first frame let’s now click track forwards. PFClean will only ever track the tracker we have selected in the list. You can however track more than one at the same time by holding the shift button down and selecting multiple trackers.
After checking the result of Tracker0002 let’s now position Tracker0003. This time let’s place it somewhere lower in the frame. Again we are looking for something detailed and high contrast to help with the tracking.
Once placed and adjusted click the track forwards button. As soon as it is complete check through and make sure the results are good. If at any time the tracker loses its lock, simply stop the tracker by pressing esc go to the problematic frame and manually reposition it to where it should be, PFClean will auto key the manual repositioning.
09. Further Refinement With F-Curves
If you are in a situation where you have followed the above steps to stabilize your shot and yet there is still some movement left in your clip, what more can you do to fix this?
This is where we can use the F-Curve panel to make minor manual adjustments to improve your results in the case of our clip we can modify the X and Y axis and the rotation per frame.
When loaded, we will see a graph representing the movement of the frame in the Stabilize effect. Each effect we have in the stack will have a graph showing the effects parameters.
The vertical white line in the F-Curve editor shows the position of the current frame. To change this, click and drag the white line with the left mouse button. The vertical blue lines indicate the start and end of the clip’s processing range.
To pan the visible region in the F-Curve editor or Dope-Sheet, click and drag with the right mouse button in the editor window. To pan in either the horizontal or vertical direction only, hold the Command key before clicking with the right button.
To zoom in or out of the F-Curve editor or Dope-Sheet, click and drag with the middle mouse button. Pressing the Fit button will scale the viewpoint so all F-Curves are visible in the editor. Pressing the Dope-Sheet button will switch to the Dope-Sheet editor.
10. Modifying F-Curves
Let’s click on the X under Translation on the left hand side.
When we click on either axis or the value for the Rotation a number of editable points will appear in the graph.
Let’s click on the X axis on the panel on the left, with Stabilize effect the vertical numbers on the graph represent the movement in pixels. The horizontal numbers represent the frame number in the clip.
Let’s highlight the editable vertices on frame 3 by drawing a square around it in the graph to select it. Below we can see the value -2.49057, what this means is the Stabilize effect has moved the frame -2.49057 pixels to the right to compensate for the unwanted movement in the frame.
If we now click on the Y axis on the left and go to the first frame of the clip we can see there has been movement vertically up by nearly 25 pixels. By clicking the A/B button we can see the large bump at the beginning of the clip and the Stabilize effect has compensated by -25 pixels vertically to inversely match the movement and remove the bump.
You can see how useful this tool might be for correct unwanted movement in the frame after a Stabilize has been processed.
Whilst we are on the Y axis let’s go to frame 10 and highlight the vertices for that frame. Now lets drag the vertices down by around -15 pixels and let’s see what happens to the frame by pressing the left arrow key to go back a frame and then the right arrow key to go forward a frame. We can see the frame has moved vertically up. Now let’s do the inverse and move the vertices 15 pixels up, and go back one frame and forward again. We can see the frame has now gone down. If you wanted to offset the entire frame slightly you could highlight all vertices on the axis and drag them up or down.
Lastly let’s go through some of the options available to us in the F-Curve editor.
The Value edit box can be changed using the pop-up menu. Changing it to Offset will mean that the value entered into the edit box will be added (or subtracted, for negative numbers) from the current value. Scale can be used to scale all values around zero, and Filter will scale all values around their average.
To select all or none of the vertices at the same time, press the All/None button. Pressing the Add button will insert a new vertex at the current frame. The Delete button is used to delete all selected vertices from the curve.
The button can be used to apply a Gaussian filter to all selected vertices, smoothing the F-Curve. The edit box next to the smooth button can be used to enter the percentage of smoothing (0 indicates no smoothing and 100 indicates maximum smoothing).
Once we have familiarised ourselves with the F-Curve editor let’s return to the Stabilize effect by pressing the F-Curve button again on the top left.
TASK: Now that you are familiar with the different ways of stabilizing, move onto clip_004 apply what you have learnt to this clip to stabilize its motion.
In the next tutorial we will be looking at the mask panel and how to use it in combination with the Auto Dirt effect.In the next tutorial we will be looking at the mask panel and how to use it in combination with the Auto Dirt effect.