Z-Depth Edit

Usage  |  Controls

The Z-Depth Edit node can be used to edit or create depth maps using a variety of methods, from paint tools to automatic filling, tracking and merging.

Masks can be used to specify areas that should be excluded from the depth map, or define object boundaries across which no filling, tracking or merging will be performed.

As z-depth is calculated, it is written to disk in a floating-point file format. Typically, for a 2048x1556 image, these files occupy about 13Mb of disk space for each frame.

If left and right-eye stereo clips have been attached to the Z-Depth Edit node, and disparity has been calculated for each frame using the Disparity Solver node, enabling the Auto Stereo Transfer option will automatically apply edits made in one clip to the other

Usage

Paint

Several tools are available to edit or create depth maps by painting depth values into the image.

Fill

The Z-Depth Edit node is also able to automatically estimate depth values for empty pixels. The screenshots below show one frame from a clip where the camera is stationary, viewing an actor's face. In this situation, it is not possible to estimate depth values using the Z-Depth Solver because the shot does not contain any parallax and the actor's face is moving. However, after manually painting some depth values into the frame (left-hand image) an approximate depth map can be built for the entire frame (right-hand image).

Track

After z-depth values have been generated for one frame, they can be tracked into nearby frames. This can be useful in situations where z-depth values are required to be temporally consistent from frame-to-frame. Z-depth values can be tracked forwards or backwards from the current frame.

The Spatial and Position weights affect how depth values are tracked. When the spatial weight is non-zero, the motion of pixels in the image plane (calculated using optical flow) will be used to track depth values. Tracking with a non-zero position weight should only be done when accurate camera motion is necessary and the scene does not contain any moving objects, because it assumes that camera positions are accurate from one frame to the next and uses this information to update the depth values from frame-to-frame.

Generally, when tracking depth images for a static scene, the Position weight should be large, assuming your camera motion is accurate. For situations where camera motion is less accurate, increasing the Spatial weight may provide improved results.

When tracking z-depth values for a moving camera viewing a static scene containing no moving objects, it is often preferable to use a high Position Weight. Conversely, if depth values have been painted by hand for a scene that contains moving objects, or is viewed by a camera that is not tracked accurately, a high Spatial Weight should be used because this reduces the reliance on camera data to inform the tracking.

The image below shows a situation where depth values have been tracked using a Spatial Weight of 100 and a Position Weight of 0. The camera in this shot is stationary and the actor's face is
moving. Original image frames are shown as thumbnails.

The initial depth map for this shot was generated using the Auto-Fill tool, described above.

Note that depth tracking cannot be performed when using multiple input cameras.

Merge

When calculating z-depth values independently for each frame, it is sometimes the case that the resulting depth maps are not entirely consistent with each other. This can happen when there are many possible depth values that will satisfy the constraints that are used by the solver to estimate depth at each pixel. In situations such as these, several independent depth maps can be combined maps together to generate a result that is more consistent from frame-to-frame.

Note that depth merging cannot be performed when using multiple input cameras.

Using Masks

Masks can be used to exclude areas of the image from the depth map calculations, or to indicate boundaries between objects at different depths. Each mask that is connected to the node will appear in the Masks list, where its behaviour can be controlled. The ordering of individual masks is important, because it specifies the relative depth ordering of the objects defined by each mask. Masks at the top of the mask list define objects that are farther away from the camera than masks at the bottom of the list. The relative ordering of masks can be adjusted by clicking on the mask in the list with the left mouse button to select a mask, and the clicking either the Move Closer or Move Away buttons.

When a mask is set to Boundary, pixels covered by the mask are assumed to belong to an object. Boundary masks can prevent bleeding between areas of the image inside and outside the mask when tracking, merging or filling depth maps1.

Controls

Clear: Clear the depth map from the current frame and delete the data file from disk.

Clear All: Clear depth maps from all frames and delete the data files from disk.

Single Frame: When enabled, the depth map for the current frame will be used for all frames in the clip. Note that when this option is enabled, depth maps cannot be tracked or merged.

Auto Stereo Transfer: When enabled, edits applied to one clip will be automatically transferred to the other. Note that this will only happen if stereo disparity has been calculated using the Disparity Solver node.

Camera

Near plane: The distance of the near plane from the camera. Depth estimates will be generated between the near and far planes, so this value should be set appropriately before solving for any depth maps.

Far plane: The distance of the far plane from the camera. Depth estimates will be generated between the near and far planes, so this value should be set appropriately before solving for any depth maps.

Masks

Name: The name of each active mask.

Colour: The mask overlay colour for each active mask. Double click in this column of the selected mask to change its overlay colour.

State: The behaviour of the mask. When set to Exclude, no depth estimate will be produced for pixels covered by the mask. When set to Boundary, no smoothing will be performed across the edge of the mask. Right-click in this column to change the behaviour of each mask to either Exclude or Boundary.

Move Closer: Move the selected mask closer towards the camera. Note that the ordering of masks is important only if they overlap.

Move Away: Move the selected mask away from the camera. Note that the ordering of masks is important only if they overlap.

Paint

The brush shape, size, softness or gain parameters, will be stored separately for each paint mode. This allows to quickly change between brush shapes that are suitable for different operations (for example, a small brush with zero softness when using Paint mode, and a larger, soft brush when using Smooth mode).

Shape: Specify the shape of the brush used to paint depth values. Choices are Circle, Ellipse, Square or Rectangle.

Lasso: When enabled, a lasso can be drawn in the Cinema window by clicking and dragging with the left mouse button. Whilst the button is enabled, all paint operations will be applied to all pixels inside the lasso at the same time.

Size: The horizontal and vertical size of the paint brush (measured in window pixels, rather than image pixels).

Softness %: The softness of the brush. A value of zero will provide a hard-edged brush.

Gain %: The gain of the brush. A value of 100 will produce a brush that is full opaque at its centre, and a value of 1 will produce a brush that is almost fully transparent at its centre.

Depth: The depth value that will be painted when using Paint mode. When using Gradient mode, a second edit box is also available and this is used to specify the ending depth value of the gradient.

Sample: Allow the Depth value to be sampled from the image by clicking the left mouse button. Sampling can also be performed by holding the Shift key when painting using Paint or Gradient modes and clicking and releasing the left mouse button. When using Gradient mode, hold the Alt/Option key as well as the Shift key to sample the second gradient depth value.

Paint: Enable Paint mode and paint depth values directly into the image.

Push: Enable Push mode, which will push depth values away from the camera.

Pull: Enable Pull mode, which will pull depth values towards the camera.

Clone: Enable Clone mode, allowing depth values to be copied from elsewhere in the image. Hold the Alt/Option key and click and drag with the left mouse button to adjust the clone offset.

Gradient: Enable Gradient mode and paint gradiated depth values directly into the image. The depth values that define the start and end of the gradient are displayed in the Depth edit boxes, and the gradient orientation can be specified by holding the Alt/Option key and then clicking and dragging with the left mouse button. When using the gradient brush mode, depth can be sampled by holding the Shift key and then clicking and releasing the left mouse button in the Cinema window at the location you wish to sample depth from. This will set the starting depth value for the gradient. To sample the ending depth value, hold the Alt/Option key at the same time as the Shift key when sampling.

Smooth: Enable Smooth mode, for smoothing nearby depth values.

Erode: Enable Erode mode, which will replace nearer depth values with those that are further away from the camera.

Dilate: Enable Erode mode, which will replace further depth values with those that are nearer to the camera.

Edge: Enable Edge mode, for painting along an object edge to generate a sharper depth map that better matches the pixel colours of the object edge. The screenshot below shows smooth depth values along an object edge (left hand image). After painting along the edge with the Edge brush (middle image), the depth values have been refined to better fit the image, producing a sharper edge boundary inside the region that was painted.

Erase: Enable Erase mode, which can be used to remove depth values from the image.

Fill: Enable Fill mode, which can be used to fill depth values from surrounding areas in the image.

Denoise: Enable Denoise mode, which can be used to remove noise from depth map values.

Fill

Frame range: The processing range for z-depth filling. Options are Clip, to fill depth maps for all frame in the clip; From/To to fill depth maps for a specific range of frames; and Current to fill the depth map for the current frame only.

From: Set the From frame to the current frame when the Frame range is set to From/To. The frame number can also be adjusted in the edit box.

To: Set the To frame to the current frame when the frame range is set to From/To. The frame number can also be adjusted in the edit box.

Fill Smoothness %: The smoothness of depth values that are estimated using the Auto-Fill tool. Increasing this value will produce smoother depth maps, but may smooth depth values excessively across object boundaries. The default value is 10%.

Auto-Fill: Automatically estimate depth values for all empty pixels in the image. If the result after auto-filling is not satisfactory, click the undo button to revert to the depth map before filling and adjust the Fill Smoothness parameter or use the Paint tools to modify the initial depth map.

Track

Spatial Weight %: The weight given to the spatial motion of pixels (calculated using optical flow) when attempting to track depth values from one frame to the next.

Position Weight %: The weight given to the 3D position of the pixel as viewed from the previous or next frame's camera position when attempting to track depth values from one frame to the next.

<<, <, >, >>: Track depth maps backwards or forwards by one or more frames at a time. Tracking can be stopped by clicking the Stop button or by pressing the Esc key on the keyboard.

Merge

Frame range: The processing range for z-depth merging. Options are Clip, to merge depth map for all frame in the clip; From/To to merge depth maps for a specific range of frames; and Current to merge depth maps for the current frame only.

From: Set the From frame to the current frame when the Frame range is set to From/To. The frame number can also be adjusted in the edit box.

To: Set the To frame to the current frame when the frame range is set to From/To. The frame number can also be adjusted in the edit box.

Image Consistency %: The consideration given to the similarity of image pixel colours in adjacent frames when merging depth maps. A large value means pixels colours must be very similar before depth values can be merged.

Depth Outlier %: The difference in depth values from other frames must be before they are considered as outliers and not included when merging into the current frame.

Frame Window: The number of depth maps on either side of each frame that will be merged when generating a new depth map. The default value is 5, meaning 5 previous frames and 5 later frames will be merged together.

Merge: Merge depth maps together using the parameters described above. This operation can be run in the background by holding the Shift key when clicking on the button.

Display

Current clip: The clip currently displayed in the Cinema.

Display mode: The method that will be used to display depth maps in Viewer windows. The options are Triangle mesh, which will render a triangular mesh containing every pixel in the image, and Point Cloud which will render a point for every pixel in the image.

Viewer proxy: The resolution used to display triangular meshes and point clouds in Viewer windows. For high resolution images, selecting Half, Third or Quarter will increase rendering performance.

Grey-scale gamma: The amount of gamma correction that is used to display a grey-scale depth map in the Cinema window.

Transparency: The amount of transparency that is used to display a grey-scale depth map in the Cinema window.

Show ground: When enabled, the ground-plane will be displayed.

Show horizon: When enabled, the horizon line will be displayed.

Show depth map: When enabled, a grey-scale depth map will be displayed in the Cinema window if one is available for the current frame.

Show depth mesh: When enabled, a triangular mesh or point cloud will be displayed in both the Cinema and Viewer windows. If no depth map is available for the current frame, the depth map for the nearest available frame will be displayed.

Default Keyboard Shortcuts

Keyboard shortcuts can be customised in the Preferences.

Clear	Ctrl+L
Clear All	Shift+L
Merge	Shift+G
Auto-Fill	Shift+A
Move Closer	Shift+N
Move Away	Shift+M
Paint Brush	Ctrl+1
Push Brush	Ctrl+2
Pull Brush	Ctrl+3
Clone Brush	Ctrl+4
Gradient Brush	Ctrl+5
Smooth Brush	Ctrl+6
Erode Brush	Ctrl+7
Dilate Brush	Ctrl+8
Edge Brush	Ctrl+9
Erase Brush	Ctrl+0
Sample	Ctrl+S
Lasso	Ctrl+A
Track Backwards	[
Track Backwards Frame	Shift+{
Track Forwards Frame	Shift+}
Track Forwards	]
Show Ground	Ctrl+G
Show Horizon	Ctrl+H
Show Depth Map	Ctrl+D
Show Depth Mesh	Ctrl+P
Toggle Clips	C