Accomplishments
- Mesh Improvements
- Texture Improvements
Bonus
- UI Improvements
The 3D model is the ultimate visual output of the 360 to 3D reconstruction process. It’s quality is judged subjectively by the user, and quantitatively by the file’s size. Qualitatively, we were able to improve the visual look of the 3D model by editing the UVs and better distributing the image textures across the scene, with more proper fades between. Quantitatively, we were able to improve the process by reducing reconstruction time by over 85%. I also improved the reconstruction workflow, allowing for groups of photos to be posed individually and connected at joints.
Mesh Improvements:
After noticing a reconstruction process take close to 1 hour for a 52 image sequence I was determined to improve the performance. After investigation I noticed two likely improvements. One was a case where cpu and gpu were switching contexts quite often, and another was related to limiting the distance with which a camera should vote on points.
I was able to re-work the process to keep contexts focused on either cpu or gpu and swap more efficiently. I was also able to limit the valid range for each camera. The depth estimation is only valid to a certain distance, so a camera from one end of the house shouldn’t be contributing votes to the room on the other side. This massively reduced the calculations for determining the final point cloud and mesh. Combined these changes were able to increase efficiency from 51 minutes down to just 8 for a 52 image sequence.
Texture Improvements:
Balancing visual fidelity with file size is a challenging task. The level of fidelity required is determined by the final application through which the user will experience the 3D model. The current application for which intends to be environmental navigation tool for spatial tours. While last week’s result was adequate for this, sharp transitions and blotchy textures littered the scene. Seing slices of walls or pillows and sharp changes in contrast is uncanny and distracting.
I was able to setup a process which establishes an area between the seams for blending. This softens each transition and provides a gradient for color to change over. This limits the expression of sharp contrast changes or color inequalities across photos and provides an image much more appealing.
Part of this process included improvements to the UV maps. UV maps are the unfolded faces of an object, cut into islands. Just like coloring origami on a flat sheet of paper before folding it. Before the UV map improvements there were an incredible number of very small islands. These little triangles littered the scene and often had textures slightly different than the area around them, standing out quite obviously. Updates were made to grow these islands. Now more triangles are connected to their neighboring islands, and the islands overall are much larger. This allows us to better spread out the textures across surfaces for more complete and consistent visuals.
Bonus: UI Updates:
The user interface was completely overhauled to provide a simpler interface and better conform to the new workflow. Images can now be uploaded in groups and automatically process between downsample and masking operations. Masks and RGB images are visible directly from the image’s drop down in the persistent content window. Masked groups can be posed via an action and immediately display in the 3D viewport. From here a camera’s height can be provided to scale the whole cloud. And groups can be linked together at a joint to allow for more piecewise alignments. Merged groups can be modeled and when complete, again they appear directly in the 3D viewport.
With these changes I have streamelined the end-to-end process and reduced views to maintain from several to just 1.
Summary
Better visual fidelity and faster processing. Simpler workflow and less to maintain. These simple benefits are the result of persistent iteration and reflection on a process that continues to evolve and further funnels it closer to its final permutation. It’s effectiveness continues to increase, and I am grateful for the changes accomplished this week.
