Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes#

Overview#

This app implements Neural Geometric Level of Detail (Takikawa et al. 2021), a pipeline for optimizing Neural Signed Distance Functions.

A large portion of the codebase overlaps with the original work. However, some differences from the original paper stand out:

This version uses a shared decoder for multiple LODs, whereas the original work used a decoder per LOD.
This version supports additional backbones in addition to the Octree, which didn’t appear in the original publication.

Diagrams#

The NGLOD app is made of the following building blocks:

As described in the original NGLOD paper, training is performed directly over the SDF samples and sphere tracing is used only at inference time.

An interactive exploration of the optimization process is available with the OptimizationApp.

When using the non-interactive renderer (e.g. OfflineRenderer, used within the trainer for evaluation), outputs can be rendered with a matcap. Otherwise, surface normals will be visualized by default.

How to Run#

Neural SDFs are trained from mesh samples, a prerequisite is to download some mesh. A sample OBJ file of the Spot cow is available here.

To run the SDF training:

NGLOD (Octree):

cd kaolin-wisp
python3 app/nglod/main_nglod.py --config app/nglod/configs/nglod_octree.yaml --mesh-path /path/to/spot.obj

Currently, the SDF sampler we have shipped with our code can be quite slow for larger meshes. We plan to release a more optimized version of the SDF sampler soon.

To run with additional backbones:

NGLOD (Hash):

cd kaolin-wisp
python3 app/nglod/main_nglod.py --config app/nglod/configs/nglod_hash.yaml --mesh-path /path/to/spot.obj

NGLOD (Triplanar):

cd kaolin-wisp
python3 app/nglod/main_nglod.py --config app/nglod/configs/nglod_triplanar.yaml --dataset.mesh-path /path/to/spot.obj

Optimization#

Note the difference between the octree / hash variants and the triplanar script.

The octree and hash variants initialize an octree blas (OctreeAS, a bottom level acceleration structure) from the mesh. That is - the mesh is used to initialize the occupancy structure at the beginning of the script. The dataset then is initialized from the octree to efficiently sample. Here, --mesh-path is short for --blas.mesh-path.

The triplanar script initializes a full occupancy structure (OctreeAS), which samples the mesh directly. Hence, --mesh-path refers to --dataset.mesh-path.

Memory Considerations#

Octree & Hash:

--num-samples-on-mesh determines how many SDF train samples are generated on the mesh surface. Lower end machines reduce the default number to avoid running out of memory.