Accomplishments
- Batch Match Testing
- Same Side Opening Check
- Mask Removal
- Narrow Passage Fallback
- Doorway Opening Check
- Rotation Aware Match Validation
Positioning, or matching and alignment, has likely been returned to and iterated on the most throughout this development process. The real 3D world is complex, and a 2:1 series of RGB pixels only gives us so much information about it. Upon each iteration I have squeezed more and more useful information out of these 360 photos. This week was no different. Using the refined masks and depths extracted from each photo I was able to better match each camera to the point that each dataset accurately pairs each photo to a ground truth acceptable result. To confirm each step forward for one dataset was not a step back in another, I created a tool to automate the performance of the matching process for batches of projects. And to run these during my breaks, rather than overnight, I found an opportunity to reduce fine matching time again by limiting the coarse promoted doorway masks.
Batch Match Testing:
The testing tools created last week offer a great summarized view of the results and how they compare to my expectations. However, I felt like my time could be better used rather than manually running one test, uploading a file, and running another, especially with no current notification that a test has been completed. I decided to automate the process. Now, I can select multiple defined projects and the tool will automatically go one by one, reset them to the matching stage, and perform the matching process. It will download the results files, make the comparisons, and move on to the next project.
Thanks to this, I can test my changes across multiple datasets overnight or while I step away for lunch. As each change I make has a wider impact on the process, automated tools like this are imperative to secure against hidden regressions in performance.
Same Side Opening Check:
A recent change improved match confidence by promoting more through-doorway matches, and increasing the mask availability for doorways. This greatly improved matches, while also inflating the time to complete as more pairs were promoted from the fast coarse step and into the slow fine matching. I wanted to determine how to reduce these many new, improper, doorway promoted matches in a way that had no chance to affect the good through doorway candidates.
My previous sentence had the key I needed. “through doorway” means through the doorway. I did not need to promote everything that had a meaningful number of keypoints in a doorway, I needed to promote matches that were through the doorway. This problem has already been solved in a recent week, if the keypoint lands on both images masks, they are likely on the same side of that doorway. If the keypoint lands on one mask, and not on the other’s, it is likely a trait of the other camera’s room, and the first camera sees it through the doorway.
Reducing the promoted matches to only candidates that see through a doorway drastically reduced the output and processing time for fine matching. This made automated testing possible not only overnight, but during meals and other breaks.
Mask Removal
The tool being used to detect opening masks is imperfect. From testing it seems our choices are to identify far too many masks, or far too few. We can trim data, we cannot spawn more, so the choice was made to take more masks and manage the consequences. Two consequences of this were keypoints landing on small, wrong masks and promoting improper doorway pairs, as well as hallucinated openings causing holes during reconstruction.
The small masks issue was reasonably straightforward to fix. For a 360 image any valid mask through a doorway will likely take up some portion of the height and width of the screen. If the doorway is 50 ft away, the door will be very small and likely not a reasonable pair for the camera’s to use in positioning. If the doorway is 10ft away, it will likely appear much bigger in the photo and be a much more reasonable transition from one photo to the next. Knowing this, I was able to reduce the false positive masks by removing any blob below certain thresholds. This reduced the ceiling lights, windows, shelves, and other structures that had been falsely chosen as openings.
The reconstruction piece was partly solved by removing small masks, however some bigger hallucinations did persist. Using the information about which side of a doorway an image is on, I was able to after matching construct an updated mask that only included known used masks through doorways. This stops 3D reconstruction from cutting holes in walls or cabinets. Far away cameras or ones at sharp angles that do not use a doorway may now hallucinate a flat surface where the doorway should be. Luckily, the 3D reconstruction process already downweights this information and should remove it during the process.
Narrow Passage Fallback
A strange issue has been consistently noticed when positioning pieces of the first floor balcony. Even when connected to proper images their positioning was incredibly off. After investigating, this appeared to occur because the shared keypoints used to position were very few and only on one space between the cameras.
For example, two cameras looking at eachother on a thin, long balcony. The sky is removed, and the many keypoints that remain are on buildings in the distance, or directly behind either camera. Some keypoints may be on the floor, or ceiling if present, and perhaps the wall. Featureless surfaces, like a tiled floor or flat wall, offer very few keypoints. The surfaces behind each camera on a long alley may be outside the known good range and get cut off. This leads to only using the few floor and ceiling points to align the cameras which can cause errors.
To get more points, I used this information to my advantage and when narrow corridors like this occur a fallback procedure can be performed to save the match. The alignment is then much improved. This trick alone resolved the many issues plaguing the first floor balcony.
Doorway Opening Check
An interesting and regular occurrence I saw were through doorway pairs from far away cameras occurring and leading to strange alignments. Through doorway pairs naturally have fewer keypoints due to the thin overlap of perspective. Also, further away cameras tend to be closer to walls and feature rich structures that make them more attractive for positioning. This would lead to strange events where a doorway may appear in the center of a room, even though visually I could see where the doorways meet far behind it.
To solve this problem I used a trick identified when setting up 3D reconstruction. In 3D I can take the doorway’s mask and draw a curtain from the top to bottom. This veil of points is useful as it represents the entrance to the room. I can do this for both cameras, then when I match the doorways I can check. If I draw a line from camera A to camera B, ideally it should pass through both veils. And if they are positioned correctly, it should pass through camera A’s veil before camera B’s. if it passes through camera B 3 feet before camera a’s veil, then the match is likely bad. This drastically reduced issues across floors.
Rotation Aware Match Validation
What other information could we derive from the data we have, and what problems can we apply it to? One problem has been consistent this week, the third floor balcony. This rounded glass lookout suffers from similar conditions to the first floor narrow passage. Most of the image’s content is sky, distant buildings, and featureless floors and glass walls. Plus, there are 3 different balconies, all very similar. A common issue was for them all to pair together and overlap on the same balcony rather than wrap the perimeter of the building. How could I fix this?
While reviewing the alignments I noticed something strange. In 3D I could see the same tower perpendicular to itself. When I reviewed the keypoints I noticed that the primary clusters were on a featureless wall which should not be paired, and the distant tower. It occurred to me that the yaw rotation was correct for the hallucination, but incorrect for the tower. This meant that certain keypoints did not agree on rotation. Using this information, I was able to identify outliers, and if enough existed estimate the yaw based on groups of outliers. If the yaws conflicted by a lot, I determined that the match is most likely incorrect and to try other matches.
This process reduced the hallucinated matches appearing on the third floor so that all balconies exist facing their proper cardinal directions.
Summary
This weeks’ match improvements brought every single floor closer to it’s optimal result. Each step now happens faster and more consistently. Better poses mean better structure for 3D reconstruction, and a better final visual when the process is complete.
