Photo Mesh

Usage  |  Controls

The Photo Mesh node can be used to convert a dense point cloud (generated by the Photo Cloud node) into triangular mesh, complete with texture, normal, occlusion and displacement maps. These texture maps are attached to the mesh and passed down-stream where they can be exported using the Scene Export node.

Usage

Once a dense point cloud has been generated by the Photo Cloud node, it should be attached to the first input of the Photo Mesh node. Any additional cameras used to generate the dense point cloud can also be attached to assist with texturing operations.

The bounding box of the mesh can be adjusted in the Viewer windows using the Edit button. This may correspond to the entire dense point cloud, or can be reduced to focus on a specific area of the scene if desired.

Low and medium resolution meshes can be generated faster than high resolution meshes, and require less RAM, but may not be able to represent all the details present in the dense point cloud. The amount of RAM required to generate high resolution meshes can increase significantly. For this reason, it is recommended to free as much RAM as possible (for example, by clearing the RAM cache before processing) to avoid running out of memory.

The images below illustrate the different results obtained using different mesh resolutions (all meshes in this case were generated from the same set dense point cloud). The meshes contain between 675K and 6.1M triangles, and finer details can clearly be seen as resolution increases. Note that depending on the depth map quality, increasing the mesh resolution to High may also mean noise is mistaken for detail when constructing the mesh.

The Smoothing control can be used to adjust the amount of smoothing that is used when generating the mesh. Increasing this value will generate a smoother mesh, which can be useful if the dense point cloud contain a significant amount of noise.

Once the mesh has been generated, it will be displayed in the Cinema and Viewer windows.

Simplifying and texturing the mesh

In order to simplify the mesh and pass it down-stream for further processing, a target number of triangles must be specified in the Number of triangles edit box. UI performance can decrease if the mesh is too complex, so typically the number of triangles should be kept under 1 million depending on your CPU, GPU and RAM hardware specification.

If the mesh bound is adjusted before simplification, triangles outside the bound will be trimmed before simplification. This can be useful if only part of the overall mesh is required for simplification.

As part of the simplification process, a custom UV map is automatically generated and attached to the simplified mesh. Various types of texture map can be automatically extracted from the input images and high resolution mesh. These texture maps will be no larger in resolution than the value specified in the Texture Resolution edit box, and can be exported down-stream using the Scene Export node.

The simplified mesh can also be passed down-stream for texturing using the Texture Extraction node if required, by specifying a Custom UV projection in that node. For further descriptions of the available tools, see the documentation for the Texture Extraction node.

Colour Map

A colour map can be constructed using colour from the original input images. Alternatively, if the input images contain significant variations in exposure, a Balanced colour map can be generated. This mode will attempt to even out the exposure and brightness variations in the input images before extracting a colour value for each texel, providing a more even overall colour.

When a Balanced colour map is generated, the original input images will be exposure corrected and passed down-stream once the mesh simplification has finished, allowing them to be used for manual texture adjustments in the Texture Extraction node, or exported to other applications.

The image below illustrates the difference between a colour map captured from the original images, and the colour map obtained after balancing the exposure variations. In this case, the images were captured with auto-exposure enabled to help obtain an accurate set of depth maps.

Normal Map

When simplifying a high resolution mesh to a very low triangle count, a normal map can be applied to the simplified mesh that will greatly increase the overall level of visual fidelity. The normal map is generated by comparing the simplified mesh to the high resolution original and calculating an angle for the surface normal at each texel.

Tangent space normal maps use either World or Tangent space. It is important to set this value correctly before simplifying the mesh, depending on how the asset will be used once exported.

Tangent space uses the standard Mikk tangent space to store the normal direction values. This is the same tangent system used in many realtime engines such as Unity^TM).

World space normal maps store normal values in the world coordinate system. Applications such as Mari^TM by The Foundry Ltd. provide tools that can be used to convert world-space normal maps into their local tangent space.

Occlusion Map

An occlusion map can be generated that simulates illumination visibility at each texel, depending on the surface position and normal in the original high resolution mesh. Local occlusion maps will measure the local visibility around a surface point, depending on the angle of the surface normal. The Sky option will measure the visibility of the sky hemisphere at each point.

Note that occlusion maps can be time consuming to generate, especially when the original high resolution mesh contains several million triangles, but will take full advantage of all available CPU cores.

Displacement Map

Displacement maps are used to provide a physical displacement of mesh vertices whilst rendering, which can provide better visual accuracy along silhouette edges when combined with a surface normal map. The displacement map is generated by comparing the simplified mesh to the original high resolution version, and stores the displacement along the surface normal at each texel.

Displacement maps can be generated as a full floating-point texture that contains positive and negative values indicating the amount of displacement along the surface normal in world coordinates. Alternatively, the displacement information can be normalised and stored as a grey-scale texture, where a grey value of 0.5 (in the range 0..1) indicates zero displacement. When generating a
normalised displacement map, the absolute displacement scale is written to the application log, so it can be recorded and used elsewhere if required.

Controls

Current clip: The clip that is being displayed in the Cinema window.

Scene

Bounding Box Edit: Allow the mesh bounding box to be adjusted in the Viewer windows by clicking and dragging with left mouse button (hold the Ctrl key to adjust the furthest face of the bound instead of the nearest face).

: Toggle display of the mesh bounding box in the Cinema and Viewer windows.

: Toggle display of the dense input point cloud in the Viewer windows.

: Toggle display of the ground plane in the Cinema and Viewer windows.

: Toggle display of the cameras in the Viewer windows.

Mesh

Resolution: The resolution at which the mesh will be generated. Note that generating high resolution meshes will require significant amounts of system RAM.

Smoothing %: The amount of smoothing that is applied when generating the final mesh. Increasing this value will produce a smoother mesh, but may also remove fine details (default value is 20%).

Hole Filling %: The maximum hole size that will be filled, as a percentage of the overall mesh bound. Holes smaller than this threshold will be filled automatically during the mesh construction process (default value is 5%).

Create: Start the mesh creation process.

: Clicking this button will delete the mesh and the corresponding data file from disk.

: Toggle display of the mesh bounding box.

: Toggle display of the mesh.

: Toggle mesh colouring in the Viewer windows.

: Toggle shading of the mesh using lighting and surface normals in the Viewer windows.

: Toggle display of back-facing triangles.

: Toggle display of a wireframe triangle overlay.

Simplification

Mesh name: The name of the mesh that will be passed down-stream after simplification.

Number of triangles: The target number of triangles during simplification.

Texture resolution: The maximum resolution of texture maps generated during simplification.

Colour map: The colour map mode for mesh texturing. Options are None for no colour map; Original which will use the original images, or Balanced which will attempt to compensate for significant exposure differences between images, generating a more balanced overall colour map. The check-box will toggle display of the colour map on and off in the Viewer windows.

Normal map: The normal map mode for mesh texturing. Options are None for no normal map; Tangent which will generate a normal map in the standard Mikk tangent space, or World which will generate a normal in in world space. The check-box will toggle display of the normal map on and off in the Viewer windows.

Occlusion map: The occlusion map mode for mesh texturing. Options are None for no occlusion map; Local which will generate an occlusion map representing local hemispherical visibility at each texel, oriented according to the surface normal, and Sky which will generate an occlusion map representing hemispherical sky visibility at each texel. The check-box will toggle display of the occlusion map on and off in the Viewer windows.

Displacement map: The displacement map mode for mesh texturing. Options are None for no displacement map; Float which will generate floating-point displacement map containing both positive and negative values indicating displacement along the surface normal in world-coordinate units, and Normalized which will generate normalised displacement map where a grey-scale value of 0.5 indicates no displacement. For normalised displacement maps, the overall displacement scale will be printed to the application log.

Show simplified: Toggle between displaying the original mesh and the simplified version.

Simplify: Start the simplification process and generate the specified texture maps. Note that a mesh must be simplified before it can be passed down-stream.

Default Keyboard Shortcuts

Edit Mesh Bound	Shift+B
Show Mesh Bound	Ctrl+B
Show Input Cloud	Ctrl+I
Show Ground	Ctrl+G
Show Cameras	Ctrl+C
Create Mesh	Shift+V
Simplify Mesh	Shift+W
Show Mesh	Ctrl+V
Show Mesh Colour	Ctrl+L
Show Mesh Shading	Ctrl+S
Show Mesh Backfaces	Ctrl+A
Show Wireframe	Ctrl+R
Show Simplified	Shift+S
Next Clip	C