Accomplishments
- Multi-Layer Research
Bonus
- Mobile App Improvements
Multi-Layer panorama images may have the potential to bring added depth to standalone 360 images. This week I dove deeper into considering what a process may look like to automate carving images from scenes and layering them for display. Alongside this, I made some improvements to the mobile application.
Multi-Layer Research
For most of us, reality exists in 3 dimensions. Photos compress this and convey the world in 2D. 360 photos expand photos from only showing a crop of an image, and make it into a captured scene. To take that scene and add depth to it so far has required multiple images. Having multiple photos of a shared space gives the opportunity to triangulate points, adding depth. This is similar to how the distance between our eyes can offer us depth perception. This depth can be guessed with modern tools, and only one image; However, this alone is insufficient.
Looking at estimated depth in 3D is still astounding, you can view the distance between an elephant and a person. Currently this comes at the price of strange movements and stretched seams. A 360 photo cannot see what is behind anything, so the disparity in depth becomes sharp lines that have no known color. This is likely to take someone out of the experience and make it feel less real. So what can we do?
My current thought is to use a feature we’ve already added elsewhere, inpainting. Just like how we remove furniture from a scene, we should be able to fill in the holes where something previously existed. If a person is blocking a wall, we should be able to hide the person and paint the wall. This necessitates layering, the person and wall will need to live on separate layers.
A variety of challenges come from this. For one, the unseen side of objects is not yet accounted for. If a person is facing the camera, how might we see the back of their head? While there is some promising research for generating this, I first want to test how it feels to have people and objects cutout of the scene, like a collage. Generating full 3d models from all image segments would be an ambitious idea, perhaps for another day.
The more pressing challenge is adequately determining what to cut out. Visually a human could identify it simply. We want to remove the objects causing sharp edges. We have information to help us with this, the depthmap shows us clearly where those edges are. And the color images show us where the things are, however, how does the computer know what things to cut, and how to cut them?
This week I have tested a variety of options to do just this, with interesting results. Across multiple test images, I have seen strong showings however no resolution has yet been consistent for all scenes. Collecting the edges is straightforward. What is less straightforward is identifying what they connect to. Using segmentation tools, we are able to assign every pixel of the image to a segment. This mostly works, the ground gets its own plane and a person theirs. This struggles with more abstract edges. Three trunks have sharp edges, while the leaves often don’t. And a boulder may be smooth against the ground and similar in color, while its edge is sharp against the forest behind it.
These options show potential; However more refinement may be needed before moving this idea to further stages, like streaming multiple files and displaying them adequately in 3D while overlayed.
Bonus: Mobile App Improvements
A few new features were added to improve the user experience of the mobile app. Two small changes with big impacts were adding a delete button, and adding the ability to change the camera’s target. When working on-site with the application you may accidentally take a photo in the wrong place, or forget to change the target node before capturing. This will allow you to delete or replace the node by removing it or connecting to another edge node in the tree. Some other changes include adding back asynchronous inference and capture, while an image is being added to the 3D scene you can continue capturing the next photo. Also, height is now changeable throughout the process, instead of just at the beginning. And a bug was fixed where the HDR setting would contantly disable itself.
Summary:
Experiments breed creativity, and the bright concept of multi-layer depth images has lead me to experiment to see what new value we can generate from segmentation tools and depth maps. Not only has this research helped inform me further of a possible new feature, it also expands the horizons of capabilities available and illuminates possibilities to augment our current features further. This, alongside an improved mobile application, will help to promote further creation and more immersive viewing of spatial media.
