3.3 Stabilizing Footage Using Tracking

12 Jul, 2017 | PFCLEAN, TUTORIALS

In our previous extended study material Level 3.2 Creating Custom Standards, we covered using an text editor to create custom standards for our imported footage. Find out more about The Pixel Farm Training Academy and register for the next available class here.

In this tutorial we will be looking at a couple of clips and using the tracking panel in the Workbench to assist us in our restoration.

If you haven’t completed the previous level 3 tutorial, or level 2 training, I would suggest going back and starting from there as some of things we will be covering build on the knowledge we acquired in the previous level.

NOTE: To follow along in this tutorial, you will need access to the clips provided for the class. If you don’t have them to hand, below you will find the link to download the footage again.

Tutorial Footage

To learn this tutorial you will need to download and use the footage below.

Footage: Footage Name / Footage Info

Download

01. Loading the Clips

Now we have created our custom standards we can begin to work on our restoration. For this tutorial we shall be working with the Full Aperture 2K clips in Media bin 2 which should be filtered into their own Workbench. To keep our project organised let’s click in the text box in the Workbench node icon and rename this to tracking.
Now we have renamed the Workbench let’s double click to enter it. Inside the Workbench we should see two clips at the bottom named clip_003 and clip_004.

Let’s play through both clips and see what the issues are starting with clip_003. Apart from the usual dirt, dust and scratches we can see both clips suffer from a stability problem.

In clip_003 we have a snatching issue causing the frame jump then fall back into alignment. In clip_004 we have a frame bump at the beginning of the clip. Normally these might occur where there is a shot transition and the bump is the first frame after the splice.

In this tutorial we are going to look at a number of ways to fix this, firstly demonstrating the existing automatic and manual tools, then we are going to look at using tracking to assist in stabilizing clip and finally the use of the F-Curve editor to refine the results.

02. Using Auto Stabilize

With clip_003 selected let’s press the + button next to the effects stack and under the image operations add in an Auto Stabilize effect to the stack.

The Auto Stabilize effect will automatically start working the moment it is added to the stack and is used to remove high frequency fluctuations in frame position throughout the clip.

Let’s click on the playblast image6 button and the image25 and see what effect this has had on clip_003. What we can see is the Auto Stabilize has affected the clip in a positive way.  All the unwanted high frequency movement has been removed included the bump at the front of the clip, however the low frequency gate movement remains.

We could spend some time trying different settings to tune and improve the results but we still won’t be able to completely remove the low frequency movement. The Auto Stabilize effect is great for the majority clips with stability issues but in the this scenario it would be wiser to move on and try a different technique. Press esc to come out of the playblast tool and deselect the A/B view.

03. Using the Stabilize Effect

Let’s now delete the Auto Stabilize from the effects stack as we will no longer be needing it. We can do this by clicking image27 button and confirming by clicking again when the icon changes to the ?. We will be adding in a Stabilize effect this time, click the + button again next to the effects stack and under the Image Operations add in the Stabilize effect to the stack.
The major difference between the Auto Stabilize and the Stabilize effects is the Stabilize will be able to lock the frame motion throughout the clip. Let’s have a quick look at some of the more common settings we can modify before we start processing the clip. If you’re already familiar with the Stabilize parameters you can skip this section and move onto stabilizing the clip.

image15 The Mode menu specifies the type of motion that will be removed. For most situations, Translation, Translate and Rotate, Horizontal or Vertical mode can be used to remove horizontal and/or vertical translation. For small-scale rotation and scale changes, the Affine model can be used (6 degrees of freedom), and Perspective model can be used for full planar-perspective motion (8 degrees of freedom).

image34 The frame window control specifies how many frames on either side of the current frame will be used to smooth out motion fluctuations. The default value is 5, which corresponds to averaging using 5 preceding frames and 5 following frames. This option will be greyed out when using Lock Motion like we will be in this tutorial.

image13 controls sensitivity to differences in pixel intensity that are assumed to be caused by motion (the default value is 25%). Pixel intensity differences above this value are assumed to be caused by incorrect motion estimates and are therefore ignored during the stabilize calculations.

image20 specifies the size of the border around the clip frame that is used to estimate the stabilize transformation. Increasing this value will include more of the frame (the current border area is drawn in purple in the viewer window).

image24 is used to specify a frame that we want to lock too. For example if I go to a specific frame in the timeline and press the Target Frame button no stabilising will happen on this frame as it will be used as reference frame but all other frames will be compared and locked to it. If unselected a estimation based on all the frames will be made instead. If we no longer want to use the target frame we can click  the Clear button.

image28 The Lock Motion button indicates that as much motion as possible should be removed from the clip, rather than just smoothing out motion fluctuations within the frame window.

Clicking the Sub Pixel button will ensure that motion estimation is performed at sub-pixel accuracy. Note that this means the final image may be re-sampled when rendering. With the sub pixel option disabled, the image will only be stabilized to the nearest pixel, meaning that no re-sampling will occur.

image23 As well as stabilizing over a border region, specific areas can also be drawn by pressing the Add Area button and clicking and dragging with the left mouse button to place an area in the Cinema. When one or more areas are placed, the stabilizing transform will be estimated using only the pixels inside the areas.

Pressing the Marquee button will allow a selection rectangle to be drawn in the Cinema to select areas. All/none of the areas can be selected at the same time by pressing the All/None button. Selected areas can be deleted by pressing the Delete Areas button.

04. Stabilizing the Clip

For the purposes of this tutorial we will be leaving most of the parameters at the default positions. One parameter we will be changing is the Lock Motion tick box, as we do not want any motion in the frame, let’s go ahead and click in the tick box to turn it on. We will be leaving the Mode option as Translation, we could use Trans & Rotate as there is a minor amount of rotation but for the purposes of this tutorial we will use the default.

Let’s go ahead and click the image21 button to begin processing the clip.

Now click on the playblast image6 button and the image25 and see what effect this has had on clip_003. We can see immediately it has improved the results greatly Vs the Auto Stabilize effect. There is however still some minor movement in the frame, in most scenarios this would have been a perfectly acceptable result but what if we wanted to improve the results further.

05. Adding a Tracker

Click on the Reset Motion button in the Stabilize panel to reset the Stabilize effect. We are now going to use Tracking to improve our results.

Trackers can be used to perform various tasks in PFClean, such as assisting with Stabilization or Rig Removal, animating masks and paint strokes, or providing hints to the motion analysis toolset.

In the top left of the Workbench UI we are going to click the tracking image38 button to open up the tracking panel.
If we want to go back to the effect parameters we can simply click the tracking button again. With the tracking panel open we are going to click on the New button on the lower left to create a new tracker.

We can now see a new tracker has appeared in the panel on the left showing, the name of the tracker, it’s In point and Out point, whether it’s active signified by the X tick box and finally what effect the tracker is active with. In our case it is active with the Stabilize effect. Tracking can also be used with Rig Removal, animating masks and paint strokes, or providing hints to the motion analysis toolset

We can use the trackers to help the stabilizing of a clip in the following ways.

For Translation, Horizontal and Vertical modes, a single tracker is required, although if more are present then their average motion will be used.

For Trans & Rotate mode three trackers are required. Again, if more are present then the average motion will be calculated.

For Affine and Planar Perspective modes, at least 4 trackers are required. When there are not enough trackers available, modes will appear greyed-out in the Mode menu.

We will be using 3 trackers for this clip, but let’s concentrate on a single tracker for now.

06. Positioning a Tracker

When we first add a new tracker into the effect the default position will be centered in the cinema.

The outer dotted line around the tracker represents the search area, and the inner box the actual tracking window. The contents of the tracker window are also shown in the Tracking Window panel.

The tracker displays its current X/Y position and similarity value, which corresponds to the tracking accuracy in the current frame (a value of 1.0 indicates “full accuracy”, and will appear whenever the tracker is positioned by hand).  

The tracker can be positioned by clicking and dragging with the left mouse button inside the search area in the Cinema, or by clicking and dragging inside the Tracking Window panel. The edges of the search area and tracking window can also be adjusted in the Cinema using the left mouse button. Currently the center of the tracker is over the actor. If we started to track from here the tracker would lock onto and follow the actor. This might be desirable if we were creating a mask but for this tutorial we are wanting to lock the image down.

Let’s play through the clip and hunt for some static areas in the clip preferably with nothing passing in front of or obstructing the view. We want to look for areas of high contrast to provide the best chance of a solid track.

Towards the area at the top of the clip we can see a good area to position our tracker.  Let’s make sure we are on the first frame of the shot and clicking and holding the first mouse button inside the inner tracking square drag the box so that it is over an area of contrast and detail.
You could also type into the Pos: text boxes the X, Y positions shown in the image above to get the exact location. Before we begin tracking let’s looking at some of the other settings available to us.

Each tracker will have it’s own set of parameters we can adjust.

The Channels buttons describe which of the red, green and blue channels will be used to track. In situations where one or more channels are significantly corrupted by noise or grain, they can be switched off to improve tracking accuracy.

The Window Update buttons are used to define how the ‘reference tracking window’ is updated during tracking. The default option is Starting Frame, which indicates that the reference window is only updated when tracking starts. Other options are Every Frame to update the window at every frame, or Above Tolerance, which will only update the reference window when the similarity is above the Window Tolerance. The Window Update options can be useful when track large trackers in situations where there are moderate perspective effects or rotation/scaling.

The Failure Mode buttons describe the behaviour of the tracker when the similarity between the ‘reference tracking window’ and the current tracking window is below the Failure Tolerance level. The available actions are Stop Tracking, or Predict Position which will attempt to predict the position of the tracker given its positions in previous frames. This can be used to help tracking features that are partially occluded in some frames.

The Tracking Offset parameter contains the current offset between the tracking point and the tracking window. It can be adjusted using the available edit boxes or by holding the Command key whilst dragging the tracker position in the Cinema window.

When the Blur Window or De-Flicker buttons are pressed, the contents of the tracking window will be blurred or de-flickered before the similarity is measured and compared against the Failure Tolerance. This can be useful in clips that contain significant noise or grain, or brightness fluctuations. Note that these changes will not be apparent in the Tracking Window panel of the interface, but are instead applied internally where necessary.

For this tutorial we are going to leave the settings at their defaults.

07. Tracking the Clip

Now we have positioned our tracker we are ready to begin tracking the clip. Below the tracker list on the left hand side of the tracking panel there are controls for tracking backwards, tracking backwards by one frame, tracking forwards by one frame and tracking forwards. When using tracking forwards or tracking backwards the tracker will continue to track until it either reaches the end of the shot or can no longer track due to tolerance failure if using Stop Tracking under the Failure Mode. You can also stop the tracking by pressing the esc key on the keyboard. If there are already keyframes in the clip the tracker will also stop before them.

Making sure we are on the first frame of the shot let’s click the track forwards button to track our clip. Once it has finished tracking let’s step through the frames using the left and right arrow keys on the keyboard and see if it has locked onto the target successfully, we can do this by watching the tracking window or zooming into the area that has been tracked in the cinema.

We can see this has been a successful track and no further adjustment needs to be made.

If we go back to the Stabilize effect by pressing the tracker panel button and click on the tracking Mode we can see some options are greyed out including the one we want to use called Trans and Rotate. As described earlier this is because we need three separate trackers in order for Trans and Rotate option to become available.

08. Adding Additional Trackers

Let’s go back to the tracker panel making sure we are on the first frame of the shot and add in two additional trackers by clicking the New button twice. In the tracker list now we should have two more trackers named Tracker0002 and Tracker0003.

Let’s select Tracker0002 in the list by clicking on it and as we did before let’s position the tracker in the frame where there should be no movement and it won’t be occluded. We want to get a fairly even spread of trackers over the frame so let’s pick an area not too close to the first tracker.

I’ve adjusted my tracker window to include an area of detail on the top left to help with the tracking. As we did before and making sure we are on the first frame let’s now click track forwards. PFClean will only ever track the tracker we have selected in the list. You can however track more than one at the same time by holding the shift button down and selecting multiple trackers.

After checking the result of Tracker0002 let’s now position Tracker0003. This time let’s place it somewhere lower in the frame. Again we are looking for something detailed and high contrast to help with the tracking.

Once placed and adjusted click the track forwards button. As soon as it is complete check through and make sure the results are good. If at any time the tracker loses its lock, simply stop the tracker by pressing esc go to the problematic frame and manually reposition it to where it should be, PFClean will auto key the manual repositioning.

Having checked that all the trackers are correct we can now return to the Stabilize effect by clicking on the Tracker Panel button again. In the parameters for Stabilize we are going to change the Mode to Trans and Rotate and make sure the Lock Motion tick box is selected.   Now all that is left to do is click the image21 button to begin the Stabilization process with the tracking assist. Now click on the playblast image6 button and the image25 and see what effect the tracking has had to the clip. We can see this is a massive improvement. Low frequency movement has been completely removed. Although slightly more time consuming than configuring the parameters in the Stabilize effect itself we can see how powerful the combination of both can be.

09. Further Refinement With F-Curves

If you are in a situation where you have followed the above steps to stabilize your shot and yet there is still some movement left in your clip, what more can you do to fix this?

This is where we can use the F-Curve panel to make minor manual adjustments to improve your results in the case of our clip we can modify the X and Y axis and the rotation per frame.

Let’s click on the F-Curve image26 button on the top left of the UI to load the panel.

When loaded, we will see a graph representing the movement of the frame in the Stabilize effect. Each effect we have in the stack will have a graph showing the effects parameters.

The vertical white line in the F-Curve editor shows the position of the current frame. To change this, click and drag the white line with the left mouse button. The vertical blue lines indicate the start and end of the clip’s processing range.

To pan the visible region in the F-Curve editor or Dope-Sheet, click and drag with the right mouse button in the editor window. To pan in either the horizontal or vertical direction only, hold the Command key before clicking with the right button.

To zoom in or out of the F-Curve editor or Dope-Sheet, click and drag with the middle mouse button. Pressing the Fit button will scale the viewpoint so all F-Curves are visible in the editor. Pressing the Dope-Sheet button will switch to the Dope-Sheet editor.

10. Modifying F-Curves

Let’s click on the X under Translation on the left hand side.

When we click on either axis or the value for the Rotation a number of editable points will appear in the graph.

Let’s click on the X axis on the panel on the left, with Stabilize effect the vertical numbers on the graph represent the movement in pixels. The horizontal numbers represent the frame number in the clip.

Let’s highlight the editable vertices on frame 3 by drawing a square around it in the graph to select it.  Below we can see the value -2.49057, what this means is the Stabilize effect has moved the frame -2.49057 pixels to the right to compensate for the unwanted movement in the frame.

If we now click on the Y axis on the left and go to the first frame of the clip we can see there has been movement vertically up by nearly 25 pixels. By clicking the A/B button we can see the large bump at the beginning of the clip and the Stabilize effect has compensated by -25 pixels vertically to inversely match the movement and remove the bump.

You can see how useful this tool might be for correct unwanted movement in the frame after a Stabilize has been processed.

Whilst we are on the Y axis let’s go to frame 10 and highlight the vertices for that frame.  Now lets drag the vertices down by around -15 pixels and let’s see what happens to the frame by pressing the left arrow key to go back a frame and then the right arrow key to go forward a frame. We can see the frame has moved vertically up. Now let’s do the inverse and move the vertices 15 pixels up, and go back one frame and forward again. We can see the frame has now gone down. If you wanted to offset the entire frame slightly you could highlight all vertices on the axis and drag them up or down.

Lastly let’s go through some of the options available to us in the F-Curve editor.

image11 To set the value of any selected vertices, enter a number in the Value edit box and press return.

image16 The Value edit box can be changed using the pop-up menu. Changing it to Offset will mean that the value entered into the edit box will be added (or subtracted, for negative numbers) from the current value. Scale can be used to scale all values around zero, and Filter will scale all values around their average.

image18 To select all or none of the vertices at the same time, press the All/None button. Pressing the Add button will insert a new vertex at the current frame. The Delete button is used to delete all selected vertices from the curve.

image37 To display Bezier control handles for any selected vertices, press the Bezier button. To hide the handles and make the vertex linear, press the Linear button.

Pressing the image33 button will attempt to simplify the curve and remove any redundant vertices.

The image3 button can be used to apply a Gaussian filter to all selected vertices, smoothing the F-Curve. The edit box next to the smooth button can be used to enter the percentage of smoothing (0 indicates no smoothing and 100 indicates maximum smoothing).

Once we have familiarised ourselves with the F-Curve editor let’s return to the Stabilize effect by pressing the F-Curve button again on the top left.

TASK: Now that you are familiar with the different ways of stabilizing, move onto clip_004 apply what you have learnt to this clip to stabilize its motion.

In the next tutorial we will be looking at the mask panel and how to use it in combination with the Auto Dirt effect.In the next tutorial we will be looking at the mask panel and how to use it in combination with the Auto Dirt effect.

Related Posts

PFClean: The Telerack

In this tutorial you will learn how to use the Telerack node in PFClean. Telerack is PFClean’s ultra high performance video restoration engine. Each Telerack node can only process clips of the same standard format and will apply a single set of restoration parameters to…

Read More

3.4 Using Masks

In our previous extended study material Level 3.3 Stabilizing Footage Using Tracking, we covered using the stabilize effect in combination with the tracker panel to remove instability in a clip. Find out more about The Pixel Farm Training Academy and register for the…

Read More

PFClean: Colour Balance

In this tutorial you will learn how to use the Colour Balance effect in PFClean’s Workbench to correct colour shifts and restore clips to a more neutral look…

Read More

3.2 Creating Custom Standards

In our previous extended study material for Level 3 (3.1 Importing via EDL) we covered using an EDL to import footage into PFClean. Find out more about The Pixel Farm training Academy and register for the next available class here. In this tutorial we will be creating…

Read More
Share This

Share This Post On: