Web Design

Accomplishments Fine Matching Speed Reduced By 90% Bonus Improved Pose Graph Accuracy Normal Estimation Depth Confidence Voting Fused Point Cloud As the quantity and capacity of datasets grows, so does the time it takes to test them all. To test under so many conditions can be cumbersome, and even moreso when one of the steps is taking over 15 minutes to complete. This week I set out with the main goal of increasing productivity by improving the fine matching speed as best as I can. I was able to reduce this by over 90%, while also having enough time to further improve the pose graph, and begin implementing the following meshing step. Fine Matching Speed Reduced By 90% Slow Speed High Speed By far the longest step in the pipeline is currently the fine matching process. Multiple factors exist which make this process so time consuming. We discussed many of these in Week 14. We solved the N-Choose-2 comparison problem by using the lightning fast LightGlue as an early coarse refinement step, narrowing down what might need to be compared. We also ran a spherical ransac operation (at the time, also reduced by 90% speed) on the matches to further confirm them. And then we progressed into a time intensive fine-matching stage with RoMa V2. RoMa V2 provides further confidence that the images are actually matches, which are the strongest, and where in the image they match. It is very time intensive, and through a variety of steps we were able to reduce this time by 90% as well. We were able to do this through a variety of steps. First we were able to introduce threading. Some tasks needed to be handled by the CPU, and some by the GPU. Previously, the GPU was stopping while the CPU completed its tasks, leaving time on the table. Now the GPU performs all its tasks, handing off the information as it goes for the CPU to pickup and work in tandem. We also receive a huge time improvement from using the fuse-local-corr, a feature provided by RoMa’s developer that was made specifically to provide faster time on certain hardware. We also added a change to compile the libraries to work with our specific hardware. This runs on startup, and takes around 2 minutes, but once performed, each subsequent run performs much faster! And lastly, we tweaked the way we distributed our matches across tiles to be more efficient. Through these changes, RoMa 2’s time was brought down significantly. It went from taking over 17 minutes, to just over 80 seconds. This greatly improves our ability to test matches, latter improvements, while also increasing efficiency as datasets continue to grow. Bonus: Improved Pose Graph Accuracy Floor Two Update RotatioFloor One Updaten Topology RotatioFloor One Updaten Topology RotatioFloor One Updaten Topology Last week we tested 4 datasets and only half of them appeared very accurate. After reviewing the results I was able to identify a variety of improvements to be performed. One being opening match rescue. In some cases openings from the coarse matching have very low confidence and do not enter into the second round of matching. Now, regardless of confidence, if a certain fraction of the matching points appear within an opening we promote it to the following round. This is purely additive, so that this low confidence candidate does not take the place of a higher confidence value in the limited number of positions being picked for round two. Instead, the number of items for round two expands. Now our doorways are preserved through coarse matching to be confirmed in fine matching. I also made some improvements to the distance calculation formula. Some matches had still been winning competitions even if more points were further. To combat this I added a maximum limit to the point depth. This means that extremely far points should not contribute to the confidence of the matched images. Skyscrapers, or fixtures beyond a long corridor, should not outweigh the features nearby, even if there are many more of them. Bonus: Normal Estimation Normal Map With so many kinds of estimation (Depth, Pose Graph, Keypoint, etc) its nice to finally hear about a “Normal” one. To those unfamiliar with 3D graphics the concept of ‘normals’ may sound anything but. Normals are a thing, every point in a 3D scene has them. You can imagine them as a little toothpick sticking out of the point in the direction it most ‘flatly’ faces. For example, the floor beneath your feat, those normals point straight up. The ceiling, those point straight down. Your glass of water, each degree points out in each direction. To reconstruct a 3D scene every point in our point clouds must have them too. Thankfully this is a common problem, and calculations exist to help in solving for the normals based on a depthmap and the camera’s position. Now the pipeline has accurate normals for the generated 3D depthmaps to be used in later steps. Bonus: Depth Confidence Voting Individual Maps Confidence Voting Results So far we have our matches, our pose graph, and our depth maps. When posed, most of the point clouds overlap. There are overlapping walls. There are false positives, where edges stretch out or small plants or items end up in different positions. Far away data like high ceilings or distant rooms may be inconsistent over few point clouds. There are even so many loose points that you can’t even fly through the aligned space to admire it visually! What can we do to reduce the quantity of false positive data across our depthmaps? We can vote. Since each of the camera’s knows where they are in relation to eachother, and each has it’s 2D depthmap available with the distance to each point, we can use these to construct a 3D matrix and compare across images what should be where. This way, if 3 cameras next to the bed all say the bed is in the center of the room,

Accomplishments Depth Aware Alignment Precise Yaw Calculation Pose Graph Estimation Bonus Improved Opening Detection Pose Graph Accuracy Check Continuing the effort of refining the camera’s determined positions I considered what further improvements could be made to the alignment process. The current pose graph and topology continue to improve yet remain imperfect. I considered the data I have available and have confidence in. I have nearby depth data, highly accurate matching range, and accurate image segmentation. With this, I improved matching and topology by using depth to adjust match confidence. I also improved yaw rotation calculation with an epipolar calculation. And lastly I improved opening mask identification by combining offset images. Depth Aware Alignment No Cieling Floor Ceiling and Floor Pre Depth Topology Post Depth Topology Pre Depth Alignment Post Depth Alignment At first glance the topology ma look incredible, however under closer inspection issues appear. I noticed that nodes like Main Entrance 3, and Main Room 7 were connected. These are great matches about 10 feet apart; However, an even better match for the main entrance would be Main Room 2, which is about half the distance. With perfect metric depths at any distance this likely wouldn’t cause issue. However, the current depth estimation tool worsens drastically as distance grows. This means that matching points on the far end of either node may be 15-20 feet apart, and the points between them are more likely to reach the edge of the estimation’s accuracy. The closer the matches, the more likely the depths are to be accurate. I determined the best approach would be a two step process. The first step would be do run an additional match check on the ceiling and floor of each image. This is likely to further increase the number of nearby matches for each image and give us better grounding during alignment. The next step is to construct the topology based on the distance of each match. If one image has two pairs that have high visual overlap, and one has much closer overlap than the other, the closer match should prevail. With these additions, the topology improved, nearby matches became preferred, and alignment improved as well with the new floor and ceiling anchoring. Precise Yaw Calculation Rotation Visualization Rotation Topology Up until now alignment has been performed by a 9 degree of freedom (DOF) solver, constrained to 4 degrees of freedom (rotation: yaw, translation: x y z, scale: none). I believe that the data collected by the 360 camera, along with the confident matches identified, could accurately handle yaw rotation separate from translation. This decouples the rotation from the noisy depthmap data used in the previous 4-DOF alignment solving process. To achieve this I implemented an epipolar calculation process. This process is similar to the doorway recovery process described in Week 15. It takes both equirectangular images and projects them onto a sphere. The matched points are used to cast rays, or lines, out from the center of the sphere. The spheres are then rotated to determine the best rotation where both sets of rays intersect most accurately. This gives a highly accurate rotation, which can be seen by the improvements to the pose graph. Pose Graph Optimization While the yaw has grown incredibly accurate there is still potential for slight rotation inaccuracies to compound and drift with time. In an attempt to overcome this, I have implemented a step for Pose Graph Optimization. This takes the estimated rotations amongst the high confidence connections in the topology and adjusts all rotations to minimize the overall drift. Bonus: Improved Opening Detection Door Mask Door Edge Doorway Mistake Door Repaired I had noticed there were instances where images on either side of a doorway were not being connected in the minimum spanning tree (MST). Instead similar rooms were beating the doorways in the confidence rankings. This could be two bedrooms with the same sliding screen doors and tile floors, or it could be two balconies that see the same building in a distance. I began reviewing the matches and identified a variety of causes that could lead to this behavior. The first and most obvious cause was improper opening masking. For one equirect image the doorframe was split, half on the left edge and half on the right. The image recognition tool I use is not 360 aware and does not understand that what ends on the edge begins on the other side. Because of this it could not accurately identify that a doorway existed on both sides of the image. I was able to solve this by performing an additional mask step. I would offset the image, rotating it 180 degrees so that the door is in the center, and run the mask again. Then I would combine the masks to have the full picture. Then I noticed a second masking issue. Some openings were being confused for glass! This was probably due to the glossy, well kept tile floor and the cleanliness of the space. I tested a variety of solutions for this improvement with varying complexity. I learned that the image recognition model outputs individual object masks and confidence values. That means that each blob of glass gets its own level of confidence. I considered comparing confidences until I noticed that the doorways received far less confidence than the glass in this instance. Perhaps there is a more complex and interesting solution to be found with more effort. Though at this step I pivoted, deciding on the simpler solution of excluding glass from openings. Previously, glass masks took precedent over opening masks, removing features. Now, matches are preserved if they exist in an opening, and are still compared with the later processes. Some adjustments were made to the distance calculations as well. In week 15 we implemented the first attempt at depth aware matching and saw improvements for the interior of a space. This week we improved upon this implementation by drawing a hard cutoff for matches after a certain distance. This

Accomplishments Depthmap improvements Removed Sky Flattened Glass Metric Depth Estimation Alignment improvements Distributed Sample of Matched Points Restricted Alignment Vectors (Yaw only, no scale) Doorway Recovery Sharp Edge Recovery Each step of the process relies on the accuracy of the data prepared by each of the previous steps. To most accurately determine the position of each camera, we must confirm that the matches are accurate in 3D space, and that they fit together just like they would in the real world. This week, we were able to greatly improve our alignment accuracy by using available data to refine the depthmaps, and identify low-accuracy depth matches and recover with an alternative alignment process. Depthmap Improvements Sky Removal EXR Sky Removal This simple step removes all points related to the sky for a given depthmap. This reduces the visual clutter when building the process while also simplifying the process of cleaning up the point clouds down the line. The empty void of the atmosphere has no accurate depth, therefore has no value for these stages of the process. Flattened Glass Glass Mask Hole In Glass Flattened Glass Clear glass in architecture is often used to see beyond a point. When looking at glass windows or walls in 3D it’s common to expect to see the solid structure, rather than sporadic holes in the space. I added a procedure to flatten the points identified as glass output by the depthmapping tool. The depthmapping tool itself did fairly well at identifying tinted glass as a flat object. I noticed that in some cases the glass would be ignored ad the depth would be generated through the glass. This inconsistency lead to surprising holes in the wall. Glass also can cause a slight refraction, or bend in the light going through it. This caused the holes to further corrupt the depth estimation with inaccurate refracted expectations. Flattening the glass worked well and even improved the overall shape of the room in some cases. This improves the accuracy of the individual point cloud and should reduce the time spent during some future cleanup phase. Metric Depth Estimation Side By Side Alignment Comparison Top Down Alignment Comparison Kitchen Misalignment Flat Floor Point Cloud 1 Flat Floor Point Cloud 2 Real World Step In Week 12’s report I discussed the tradeoffs between two tools used for depth estimation. One offered metric accuracy (most) of the time and had warped structure, the other had accurate structure but the metric accuracy varied widely. I moved forward with the second tool for its improved geometric accuracy, believing that someday metric accuracy could be applied with a scale adjustment. Today was that day. While ideating ways to improve my image alignments I reflected on what intrinsic data is available from the included photos. Upon this reflection I realized two important values I had taken for granted. The camera’s height and being level with the horizon (more on that in a following section). During shooting the camera remains at about eye level on a stand. This stand maintains roughly the same height throughout the capture process. Because the depthmaps tend to be accurate in the floor immediately below the camera, we can scale this distance to align with the value of the camera’s stand. This drastically improved the consistency across depthmaps. While the metric depth estimation drastically improves the process, some challenges still persist. The current stand in use is adjustable. That means over time, and from small changes and gravity the camera may accumulate centimeters of change in height. Also, slight angles like those of a hill or slope may lead to improper ground estimation and a larger inaccuracy to scale. Both of these can be solved with real-world adjustments to the stand and capture process. A fixed height stand with a tilt-able base could keep the camera level and over the pole while fixing its height to the ground with great accuracy. Another challenge this scale does not yet solve is that of incorrect depth over slight changes in elevation. For example, in the related images there exists a ground truth elevation change between the hard-wood floor and the kitchen area. The depthmaps for both the kitchen and the hardwood floor ignored this change and hallucinated a level area. Because of this, the floor for the kitchen appears confused, with two estimates, one from each camera. This may be something able to be rectified during a point cloud cleanup phase, though if unchecked has potential to accumulate scale drift over time. The metric depth estimation greatly improves the accuracy of depthmaps while drastically reducing the opportunity and magnitude of scale drift. All identified challenges have potential solutions to be later implemented. Alignment Improvements Distributed Matches Sparse Keypoints Real Pixel Density Sparse Alignment Distributed Alignment Sparse Points Distributed Points This week began with only the highest confidence matches being used to align two point clouds. Sometimes the highest confidence matches may group themselves in small areas. This makes alignment difficult. Think of it like a door. With hinges it has 1 or 2 matches on the same vertical line. A door still connects at those points whether its open or closed. Now think of a dead bolt, or a lock. When the door must be locked AND on its hinges, it only has one position. Instead of taking only the highest confidence matches, I decided to use a distribution of high confidence matches. In most cases this spread the matches across a wider area. While the matches may be less perfect, the overall structure will be more perfect. For example we can look at the secondary entrance and the shoe closet. The sparse, highest confidence, keypoints cluster in a few small straight lines. Then we look at the dense matching. Here the points cluster along a much wider field of view, almost the whole horizon. By transitioning from sparse, highest confidence, keypoints to a distributed selection of high confidence keypoints I greatly improved the degree of accuracy when aligning two point

Accomplishments Improved Matching Efficiency Improved Matching Quality Minimum Spanning Tree With Maximum Confidence Feature Masking Manually curating a few known-good images to be matched and processed is quick and relatively immune to false positives. However, applying that same manual process to 50+ images grinds workflows to a halt and quickly becomes strewn with incorrect matches. Without GPS or sequential metadata, chaotic image datasets suffer from the exponential performance scaling of any-to-any (n-choose-2) comparisons. This is especially painful for CPU-bound processes. Furthermore, this expanded dataset compounds the likelihood of introducing false positives that can slip through current feature-matching algorithms. To effectively work with large datasets, we need to generate a minimum spanning tree where all images are connected, with high confidence regarding what each image matches to and exactly where those matches are located. Achieving this requires a multi-step approach: reducing CPU operations, iteratively filtering matches, and masking out corrupting environmental features like clouds and glass. Improved Matching Efficiency AnyLoc Reduced Time Lightglue Time AnyLoc islands n choose 2 graph The fundamental purpose of the matching stage is to look at every photo and compare it against every other photo to find the best connections. Historically, our pipeline handled this in three steps: ALIKED (extracting specific pixels as features), LightGlue (comparing extracted pixels across every image in an any-to-any configuration), and Spherical RANSAC (converting images to spheres to validate matches without distortion). The problem? Matching any one photo to every other photo grows exponentially as the dataset scales. A small batch of just 52 photos requires over 1,300 unique comparisons. Our primary bottleneck was Spherical RANSAC, a heavily CPU-bound operation. By switching from a standard Python library to a just-in-time (JIT) compiled version, we were able to execute the math directly on the hardware rather than pushing it through a Python abstraction layer. This single optimization cut the operation time by over 90%, bringing a 20+ minute process for 52 images down to just 2 or 3 minutes. We also theorized using AnyLoc as a pre-check. AnyLoc is a “Global Descriptor”; instead of extracting individual features, it generalizes the image (e.g., turning a photo into a mathematical representation of “Inside, big bedroom, beige walls”). Because it runs entirely on the GPU, it is blazing fast, tearing through the 52 images in just 35 seconds. However, AnyLoc tends to group high-confidence matches by type rather than topology. It creates disconnected “islands” of similar rooms (like matching two separate bedrooms together) rather than finding the connective path (like a hallway leading to a bedroom). While AnyLoc is incredibly fast and worth considering for future iterations, its tendency to falsely combine distinct but similar rooms means it requires careful handling before full inclusion in the pipeline. Ultimately, by removing the abstraction bottleneck of our CPU process, we successfully reduced our overall matching speed by 95%. Quality Improvement Lightglue False Positive Ransac False Positive RoMa2 Doorway Speed is irrelevant if the feature matching yields incorrect results. The matches generated by LightGlue and Spherical RANSAC often include false positives. Two identical doors, two sides of a symmetrical feature, or even similar moldings and furniture in completely separate rooms can trick standard matchers into drawing a connection. To solve this, we looked to RoMa V2 (Robust Matching), a dense feature matcher. What sets RoMa apart is its ability to rank every pixel in the image by confidence and match them at extreme angles. When we pass our challenging image pairs through RoMa V2, the false positive matches still appear, but they are flagged with remarkably low confidence scores and lack shared pixel density. This allows us to aggressively filter out the low-confidence noise while retaining dense, high-confidence pixel groups—even when looking through narrow doorways. The trade-off for this accuracy is compute time. Dense matching requires dense math. RoMa V2 takes significantly longer than LightGlue, scaling exponentially and taking over 5 minutes for a 52-image dataset. Because of this, RoMa V2 is not a standalone silver bullet. It cannot process the entire dataset in a reasonable timeframe on its own, meaning we still rely on LightGlue as an early-stage filter to narrow down the workload before RoMa V2 confirms the final matches. Minimum Spanning Tree & Maximum Confidence Path AnyLoc Topology LightGlue Topology LightGlue + RoMa2 If we don’t confirm that every point cloud has valid matches and belongs to a unified tree of connections, we end up with incorrect geometry and floating, disconnected islands. LightGlue is reasonably fast, but its confidence scores for a correct match through a doorway might be identical to the score of a false positive between two disconnected rooms with similar moldings. RoMa V2 is the slowest, but it is the most accurate and provides the highest confidence for tricky bridges like doorways. To get the minimum spanning tree with the highest confidence paths as quickly as possible, we re-architected the workflow to play to the strengths of both tools. First, we use LightGlue to perform the any-to-any matching and rank the pairs by confidence, narrowing down the field of likely candidates. We then intentionally skip the long Spherical RANSAC step, as RoMa V2 will handle match quality confirmation. Next, we determine the top-ranked confidence values and begin building our topological tree. If any disconnected islands or orphaned nodes exist, we expand the tree to include slightly lower confidence values until a continuous path connects everything. Finally, RoMa V2 processes this curated list of rankings. It confirms the confidence values, outputs high-confidence dense matching points for each pair, and validates the bridges through doorways. This dual-model approach yields a unified minimum spanning tree connected entirely by maximum-confidence paths. Feature Masking Bathroom Masked Matches Balcony Masked Matches Main Room Masked Matches Bathroom Mask Balcony Mask Main Room Mask Glass, mirrors, and the sky are notorious problem-causers in computer vision. To an algorithm, a mirror on a wall often looks like a doorway into a completely identical room. A glass window might not register as a solid surface, but rather a hole leading to

Accomplishments Built Docker Images Refactored Proof of Concept Bonus Depth Model Comparison Many Image Point Clouds This week’s goal was to take a rapidly built prototype and convert it into a foundation that’s simpler to deploy and adjust. I set up the files to work with a tool that packages them in a way that they can easily be run on any device. I also rewrote much of the code so that steps can happen independent of each other. As a bonus, I was able to begin modifying the components to accept more than 2 images for 3D reconstruction, and generate a combined point cloud. Built Docker Images Docker Images The goal of Docker is compatibility. It packages a program along with all the tools it needs to run on any system. It’s kind of like having a universal adapter where, no matter where you go you can plug into the wall and charge. A docker image is the output. This includes the program and all the tools needed to run it anywhere. Creating the docker image comes with some steps and considerations. This includes writing out which tools the program will need to run. Since our 360 to 3D pipeline includes multiple smaller programs, it is common for their needs to overlap. To satisfy this we use something called a “base image” which includes all of the shared tools which may be required. Setting up the base image can be tedious, as it takes time to determine the best version numbers of each tool which are compatible with all the rest. For small tools quantified by Kilobytes or Megabytes, this may not be concerning. However, multiple programs in this pipeline rely on tools that are Gigabytes in size! Reusing these is very important to preserve efficiency and save on system memory. Another consideration is shared storage. Docker images by default write within their own spaces and do not have access to each other. This lead me to creating a “shared volume”. With a shared volume, each image can contribute, read, and write their own data to a shared space. This is helpful in large data scenarios (like 360 photography) where each program may require access to the full dataset. Preparing the Docker images helps transition a rapidly produced proof of concept into something less brittle, that can be preserved and recreated on any system running Docker. Refactored Proof of Concept A rapidly made proof of concept (POC) often includes shortcuts to more efficiently determine the effectiveness of a process. Upon approval, a new design should be constructed which includes the slower developed supportive steps which improve the ability to efficiently make changes and preserve functionality going forward. I converted my POC from sequential “get the job done” steps into a more modular “accomplish this step” approach. I added application program interface (API) endpoints to the program. Instead of manually updating a file and running the program once, I can now call the program any time and submit different information at that time. This gives swift flexibility to where, when, and how I interact with it. This also gives me a better opportunity to multi-task. I set up each endpoint to begin a job. Instead of waiting for the sequential job to be completed, I can now periodically request a status update from a different API endpoint. The benefits of this will be much appreciated as processing time increases in line with the size of the data set being uploaded. Breaking down sequential processes into atomic steps decouples logic and improves cohesion. This ensures that each atomic step only relies on what it absolutely needs and only affects what’s crucial. Setting a foundation in this manner permits future steps to be added with narrow impact to the other programs. Bonus: Depth Model Comparison Model Comparison Combined Mesh Comparison Strange Projection This week I experimented with the different machine learning models in the tool which calculates the depth for each photo. One model seemed to offer improved depth detection through doorways. This will be helpful when feature matching must take place beyond a shared doorframe. Overall the newly tested model appears to offer similar results. A few small oddities have been identified, like a strange bubble appearing around the entryway table, and a broad stretching happening to a small plastic standee placed a shelf. I believe a step to clean up, reduce, and combine point clouds will be needed. While these appear to be new issues, I believe they may be solved by a clean up phase and the benefits of expanded doorway depth may be retained. Bonus: Depth Model Comparison 3 Camera Pose 3 Image Point Cloud N Choose 2 With the improved foundation afforded by the containerization (Docker) and refactoring I was able to more swiftly modify the process to remove the previous limit of 2 images for pairing. A myriad of minor tweaks and process changes were necessary for this step. The new layout made it quick to identify and apply the required changes. The point cloud result excites me. This visualization represents the processes ability to not only combine two 3D projections, but also refine and expand that combination as more are added. Making these changes also called to mind the many changes that are necessary to perform this process efficiently as the number of photos scales. Steps like downsampling and generating depth maps scale 1:1 with the number of photos uploaded. If it takes a photo 0.1 seconds (s) to generate a depthmap, each extra photo will be an extra 0.1s. This scaling is not so true for our feature matching step. Combining two photos, A and B, takes 1 step, matching A-to-B. Combining three photos takes 3 steps, matching A-to-B, A-to-C, and C-to-B. This can be written as a formula, and simplified to N * (N – 1)/2. That is the total number of matches for the N number of items. This grows incredibly fast. While 4 items only need 2 more matching steps

Accomplishments 360 -> 3D End-to-End Proof of Concept Recent weeks of research have outlined the problems that must be solved to deliver a pipeline where sparse 360 imagery can be accurately reconstructed into a 3D mesh. A variety of tools and algorithms were compared in order to support this process. Ultimately an initial pipeline was developed where with just two sparse 360 images a merged 3D mesh could be completed. 360 -> 3D End-to-End Proof of Concept Workflow The whole process of converting 360 images into a 3D model requires a variety of smaller problems to be solved. The workflow intends to solve the following problems: Image downsampling Feature extraction Feature Matching Match Filtering Depthmap Estimation + Point Cloud Generation Convert matches to 3D Align point clouds Convert Point Cloud to Mesh Image Downsampling Image downsampling means to shrink an images size. This step is required to conform to a downstream tool’s image size requirement. Downsampling is a double edged sword. In this case, it greatly reduces processing time; However, it also reduces the detail used in identifying features and calculating depth. A future iteration may improve on this by downsampling later in the process. Feature Extraction Feature Extraction 1 Feature Extraction 1 The feature extraction step looks for unique areas of pixels across an image. These coordinates are later used to compare two images and see what features may overlap. Feature Matching Matched Features The features calculated in the previous step are compared across two images. The algorithm does it’s best to find groups of pixels that appear similar across the two images. In the example you might notice that some of the matches appear incorrect. The ceiling’s light fixture is very noticeable. The camera moves to the opposite side of the light. Because the fixture is symmetrical, the same features are found on both sides and incorrectly matched by the matcher. A smaller example can be seen on the black sliding doors. Match Filtering Filtered Matches Special feature filtering techniques exist for equirectangular images which help to filter out false positives like those we found in the previous step. The result is far fewer matches, and a far higher percentage of valid matches. The density of matches on the main door and wall art largely remain, while the ceiling light and sliding door are almost entirely removed. Quality is preferred to quantity in this step, and will help us better align the output in a later stage. Depthmap Estimation + Point Cloud Generation Comparing Techniques In deciding which tool to use for depthmap and point cloud generation I began by comparing two tools. Tool A was built to accept equirectangular images and output point clouds with reliable metric depth. Tool B was built to accept equirectangular images and output point clouds with reliable relative depth. Testing Tool A Top View Scale Comparison Side View Scale Comparison Storage Closet Washroom Comparison Main Room Comparison Depthmap Main Entrance Warping I ran a variety of equirectangular images through Tool A and laid out the output to visually assess the results. From the top and side view I notice surprising consistency in scale across the images. Each room appears to be the same size compared to other projections from the same room. Take note of the two examples of the storage closet washroom, as we will compare these with the output from Tool B. There was an example where the scale of the room appeared to be almost half that of other similar images. Take a look at the living room comparison. The larger point clouds were consistent with the scale estimates across all the other point clouds. The smaller ones appear as a noticeable outlier. These photos were taken on the stairs at a different elevation from all the other photos. My best guess is that this is the result of a bias introduced in the AI model. Perhaps the AI model’s training data heavily favored images taken from the floor of a room compared to ones on a staircase. If this is true, the output may assume that the ceiling and floor are equally close and may pull them to the size of the average room. Another oddity shows if we look at the point clouds for the main entrance. The walls bow in considerably. From close up, these warped angles appear across the entire output of point clouds. Testing Tool B Side View Scale Comparison Storage Closet Washroom Comparison Depthmap Running a similar series of rectangular images through Tool B we received a series of results with qualities considerably different than Tool A. The scale across images at times appeared consistent; However, they often were not. Look at the storage closet washroom in this example. The size of one is almost half the size of the other! On the bright side, we can see across images that there is far less warping and walls appear straight. Verdict ToolA Warping Tool B Consistency The goal of the depth estimation step is to prepare an estimate of what the geometry around the camera should look like in 3D. The metric accuracy of Tool A would support a result where near-accurate measurements could be made against the 3D geometry. To get this benefit we would have to overcome two problems. We would need to resolve the warping AND ensure scale consistency even across outlier positions like the staircase. For Tool B we cannot rely on metric accuracy. We can rely on geometric consistency and would only need to solve the scale problem in order to use them. I decided to proceed with Tool B as it is able to satisfy the requirements of the step with the least added steps required. Metric accuracy would be a great feature, and can be added in the future as tool B’s output should give us a 1:x scale replica in most cases. Convert Matches to 3D EXR Depthmap Matching Points in 3D The matching points output by an earlier step represent where points match

Accomplishments Pose Estimation Testing 360 -> 3D Tool Matrix 360-to-3D Reconstruction Architecture GMAT Study Guide Architecture.  “The art and technique of designing and building” (Link) Constructing something big often begins with outlining and constructing small atomic steps. To start baking a cake, first you crack an egg. I began testing options to estimate camera positions from 360 photos and have since refined the steps related to their 3D reconstruction. I studied and reviewed dozens of tools and libraries related to this process and compiled a matrix of related details. I also outlined my goals and steps towards them for taking the GMAT assessment tests. Pose Estimation Testing Determining where each photo was taken is a foundational step for beginning 3D reconstruction. While this is trivial in many cases, efficient, affordable, and quality 3D environment reconstruction comes with constraints that make it much more challenging. I tested a variety of commercial solutions and available libraries to better understand the process and understand how my requirements affect it. Matterport Test RealSee AI Test Commercial tools exist that perform the structure from motion (SfM) step to determine the camera’s positions as part of a greater process. Similar datasets were provided to Matterport and Realsee AI. Matterport was able to take the sparse camera data and combine them, while Realsee AI seemed to fail at aligning most of the cameras. Matterport bundles the camera positions as an export with the completed mesh at a cost. This cost may be fair and affordable for many projects; However, it raises the barrier to entry and puts environmental preservation and immersive experiences out of reach for many places. Metashape Camera Poses Metashape Mesh DAP Point Cloud 1 Camera RealityScan and Metashape are commercial tools with much different associated costs. In 1 year, RealityScan will cost at most what it costs to deliver 21 scans with Matterport. Metashape is a one-time purchase and is more affordable then Matterport after 59 reconstructions. RealityScan does not yet natively support 360 photos and failed to correctly position most images. Metashape did a much better job with some test data. Every camera was correctly aligned. I tried to see if this would also lead to a usable mesh, however the result was left with many holes. Further, with the sparse test data of Supalai Place, it failed to position more than 50% of the cameras. Sphere with Tracking Points 15 / 37 Camera Accurate Result I moved on from commercial solutions and began looking at libraries. One workflow involved SuperPoint, SphereGlue, and OpenMVG. It first took each 360 photo and identified the important areas on its surface. It then sought to match those areas across the other 360 photos. This worked surprisingly well for a first attempt, getting 15/37 photos correctly oriented. Getting the following 22 photo positions was much harder. The original output used a process that attempts to triangulate positions based on multiple photos. While the 22 photos may share much overlap, the points reviewed were not sufficient for the algorithm to pair them with the other 15 photos and they were dropped. Tracking Points in Chain Large Scale Stretching Overlap with Tangent This will likely be the case for sparse photo sets indoors where each image is only guaranteed 1 line of sight to another image. To overcome this I experimented with a chain where images were oriented sequentially. This ensured all images would be positioned. This came at the cost of quality. Since the positions aren’t all compared to each other the distance between images is harder to calculate consistently. This appears to lead to overlapping images and tangents at the start or end of the chain. Rather than a proper series of two circles the output was consistently one circle with another either stretched, overlayed, or including a straight tangent line. The quality output received from Matterport’s system indicates that a meaningful reconstruction can be performed on data as sparse as mine. The varying results delivered by the other tools confirm that sparse data comes with unique qualities that if not addressed lead to missing or misplaced cameras. I will continue to identify these qualities and review other libraries towards a solution. 360 to 3D Tool Matrix 3D Tool Matrix Inputting 360 photos and outputting a 3D model can have vastly different outcomes depending on the steps taken in between. The industry is wide and new. This leads to many different takes on how to perform the process. Through continued exposure to this environment I have begun to identify a variety of important qualities in assessing the tools. I decided to build a matrix. This matrix includes information like license, commercial availability, purpose, and if it’s outdated or replaced by another tool. This is especially important in assessing the overall viability of a workflow, and keeping track of the tools that have been tested. With over 90 tools there is a lot to keep track of! 360-to-3D Reconstruction Architecture AI Architecture Diagram Understanding the goal, our requirements, and the limitations of the available tools are key to laying out a successful roadmap. Goal: Input 360 equirectangular images from a sparse dataset and generate a navigable digital twin. Requirements: Indoor and outdoor images accepted. Must work even with only single line of sight to another camera. Even through doorways. (Ex: One image in the hallway, one in the bathroom.) Must work even when rooms are repeated. (Ex: hotel or dorm may have two identical rooms.) Mirrors and reflections should appear flat and not hallucinate rooms beyond the plane. Should work with non-standard layouts like curved glass walls, large rooms and open spaces. Tools: Segment Anything Model (SAM) 3: Identify segments based on context like “mirror, window, sky” LightGlue: Identify features and match them across images. SphereSFM – COLMAP: Uses matched features to construct structure from motion and deliver camera pose estimation. Depth Anything Panorama (DAP): Generate depthmap from equirect photos. LGET-NET: Room layout estimation from 360 equirect images. Open3D: 3D Point Cloud Reconstruction Truncated Signed Distance Function

Accomplishments Hosting Architecture and Pricing Research Multi-floor Diorama Workflow Research Bonus  360 Reconstruction Research “Plans are nothing; planning is everything.” – Dwight D. Eisenhower Virtual tours must be delivered efficiently and reliably. These two requirements drove much of my research this week. While I didn’t make any direct updates to the software, the development path has become much clearer. I spent time refining the hosting and service requirements for delivering 360 tour files over the web, and outlined workflows for handling multi-floor and large-space 3D reconstructions. After refining the initial workflow and identifying its bottlenecks, I began exploring and evaluating alternative solutions. Multi-floor Diorama Workflow Research improper Occlusion Manual Reconstruction Texturing Manual Reconstruction Texturing Photogrammetry Texturing Dioramas provide an eye-catching overview of a 3D scene and offer an intuitive way to navigate a tour. In a previous post, I made my first attempt at creating a usable tour for a dense, three-story property. However, the workflow I used was time-consuming, prone to system crashes, and ultimately unscalable. This week, I challenged myself to test alternative workflows to deliver this service more effectively. As a quick refresher, dioramas are 3D models that represent a physical space. These models need to reflect the environment with enough visual accuracy so that viewers can easily figure out where they are and where they want to go. Crucially, this navigation must happen without lag. Because 3D models and their associated images can have large file sizes that take time to load over the internet, we must carefully balance visual quality with performance. In my previous workflow, I started with a flat “plane”, essentially a digital piece of paper. I would cut and stretch the edges until it matched the floor plan, then stretch the sides up to build the walls. After that, I added the ceiling and interior details like countertops, appliances, and inner walls. The result was an incredibly low-poly object with a small file size, but the manual labor required to build it was unreasonable. Furthermore, the process put incredible stress on the 3D modeling program, which struggled to load over 100 uncompressed 11K .jpg files just to calculate the model’s textures. It was a great learning exercise, but it highlighted the severe challenges of recreating spaces manually. For my second iteration, I tried to speed up the manual recreation by starting with a cube instead of a flat plane. Stretching and cutting a cube handled the floor, walls, and ceiling simultaneously, drastically increasing the speed of the build. However, this process still took considerable time and completely missed smaller, non-cubic details like couches or shelves. Without these details, the visual “hotspots” weren’t properly occluded from one another, causing their circular icons to appear cut or incomplete. Adding those missing details manually would simply add that saved time right back into the process. This second workflow also involved texturing the 3D model by projecting portions of the 360-degree photos onto the digital walls and floors. When done perfectly, the high resolution of the 360 images creates a fantastic display. In practice, however, it requires massive manual effort. The 3D surfaces are divided into sections, and the software decides which parts of the image to project based on distance and angle—not based on what looks best. This forces you to go back through the scene and manually adjust the surfaces to use the correct image projections. The smallest model I created for the three-story home had over 5,000 triangle surfaces requiring manual review. That is simply not acceptable. In my third attempt, I decided to sacrifice file size in favor of speed. I used the raw photogrammetry scan taken on location as the 3D model itself. Unfortunately, phone-generated photogrammetry scans still require significant manual intervention. The datasets needed to generate full-house scans are often too large for a phone’s memory, meaning you have to take multiple smaller scans and manually align them. The edges of these stitched pieces often feature incomplete or warped geometry—a wall might flare out unnaturally, or a room at the edge of the scan might devolve into a discolored blob. All of this requires manual cleanup. Photogrammetry also struggles heavily with reflections. This can be as obvious as a mirror or as subtle as the glossy sheen on a wooden dresser. Reflective surfaces create strange, warped artifacts in the scene that look incredibly uncanny to a viewer trying to admire a space. Furthermore, if certain angles aren’t captured perfectly, they render as large, undefined blobs. All these anomalies degrade the final output and require time-consuming manual fixes. The result is far from perfect, heavily bloated in file size, and costly to produce in terms of time. So, how can we improve? There are a few remaining options that might reduce the manual labor required for these low-poly reconstructions: The tool used for texturing the walls has a feature to texture based on vertex groups. This has the potential to quickly map individual cameras to vast areas of triangles, greatly reducing texturing time. Camera positions could be calculated using Structure from Motion (SfM) algorithms. This would eliminate the need to manually place and align each camera in the digital space. We could abandon the diorama entirely for larger scenes, using a basic, untextured mesh solely for occlusion and hotspot placement. While each of these options improves the process slightly, they still fail to overcome the biggest hurdle: the manual labor required to piece everything together. To offer 3D reconstructions at larger scales, the process must be heavily automated. It needs to produce a consistent, high-quality output that visually supports seamless navigation, and it must compress down enough to download quickly and run smoothly within the memory constraints of mobile devices. Hosting Architecture and Pricing Research AI Architecture Diagram Everything you see on the internet lives on a physical computer somewhere and travels over a network to reach you. Companies offer this service at a cost, differentiating themselves based on how they deliver that content. The virtual tours I create, along

Accomplishments Minor UX improvements and Bugfixes Research Video Playback Challenges Paginated VR Grid Menu Bonus Thumbnail Update Tool To continue improving the user experience (UX) for viewers in virtual reality (VR) I decided to create a 3D version of the image gallery menu designed in Week 2. A gallery menu option allows viewers to select the exact scene they want to view in the fewest clicks. This greatly reduces the friction when navigating dense tours or sporadic image galleries. Through this implementation, I identified edge cases affecting the existing vr menus. This includes an issue that relies on the video playback process and requires more investigation. As a bonus, I was able to produce a tool for editing the panorama thumbnails that appear in the custom grid view to allow for optional adjustments. Minor UX improvements and Bugfixes: Info spot with html popup 3D Cursor VR The increased focus on VR functionality has lead to more user journeys being tested and improvements being identified. Hotspots, the images we use in 3D, are used for a variety of cases and should appear or disappear based on specific criteria. They also animate. When using menus like the media controls menu, a viewer will be surprised if these animations are delayed, or are not synchronized between the immersive view and the 2D view. The 360 images transition too, and it’s important that these transitions handle well in VR as to not disorient the viewer. All of these challenges found resolution through my work this week. The two categories of hotspot that had incorrect visibility were the information hotspots (infospots) and the VR 3D cursor. Information hotspots were implemented in week 6 and when hovered over they spawn a 2D HTML snippet (or iframe) onto the screen. This occurs in 2D space and uses the document object model (DOM) which is not available in 3D immersive VR scenes. Because the HTML could not be displayed, I added a value to each infospot which triggers the html display. I then added check when entering and exiting VR to show or hide them. Now the infospot feature does not appear when in an immersive VR view. In VR we don’t have a traditional mouse and cursor. We often select items by pointing with our controller or finger. Usually a line is drawn from the hand to the nearest surface where you are pointing and a small circle or 3D cursor appears to show what you are about to click. This feedback tells viewers they can interact, and what they are about to interact with. While working with last week’s VR menu, I noticed that hiding the menu also hid the 3D cursor. The logic that hides the vr menu searches the scene for all hotspots with a specific tag, animates them out, and removes them from the scene. The 3D cursor in use comes from a third party tool. Unknowingly, I had provided the same tag to my vr menu items that was being used by the 3D cursor. I updated the tag on my items to be more specific to my use case, and this fixed the issue. When you turn a faucet or flick a switch, it’s expected that water or light immediately appears. That is the nature of reality. Cause begets action immediately. Virtual reality is different, it can have a cause and pause. A delayed reaction immediately signals to our brain “something might be wrong”. While interacting with the media player I noticed a variety of steps which lead to animations and reactions lagging behind or completely ignoring my input. Animating between scenes, or transitioning, in VR would freeze. Under certain conditions toggling the media center would stall or never react, and in many cases would get desynchronized from its 2D counterpart. For the media controls menu most of the cases identified were resolved by setting flags and adjusting timings. Some extra steps were added to check and confirm synchronization between 2D and 3D menus. In the case of the VR transitions I had to get a bit creative. Scenes transition differently depending on how you are moving between them. In a tour without 3D elements the 360 images can simply fade and blend from one to another. In 3D tours the camera’s position teleports to another location during the transition. During this move there are a few moments where the 3D space appears stretched and warped due to how the previous image is painted onto it. Instead we use an intermediate scene of all black. Imagine navigating a website and every time you clicked a new tab the whole screen, including the menu bar, went completely black for a moment. You may ask yourself what might be wrong. Without a menu bar we cannot return home or navigate to another page, our only option is to reset the browser or go back. In VR these options are much more costly. If you’re immersed in an environment, asking you to restart the browser or refresh the page is like asking you to get out of the pool and dry off just so that you can hop in again. It’s cumbersome and sours the experience. And further, you’re not just staring at a black screen, you become immersed in a black void. With no visual anchor points you can quickly feel disoriented and concerned, even in this brief moment of transition. To solve the concern about losing control and the disorientation of a black void I updated the transitions to retain only the vr menu. This gives a consistent reference point, and the feeling of control and ability to make a change if the black screen were to persist beyond the consistent transition period. Research Video Playback Challenges Bad Bunny Youtube Video “Andrea” Adjusting timings and setting flags solved most, but not all, challenges with animating the media controls menu. I wondered why I had not seen these issues during development, and why they were only appearing intermittently. Issues with logic and

Accomplishments Added Control Panel to the VR mode. VR Related Bugfixes Progress bars visual update The virtual tours I create are intended to be accessible across three device categories: traditional computers, touchscreen mobile devices, and virtual reality (VR) headsets. Recently, research and development has focused on 2D features accessible to the first two categories. This week, I shifted my focus to the VR user experience (UX). Through this effort, I identified and resolved bugs, improved feature parity by providing a menu in VR, and implemented shared features that also enhance the 2D experience. VR Bugfixes In the overview, I describe three categories of devices relevant for testing their unique user experiences. This neglects to mention how VR devices effectively function as two categories. VR devices often offer a traditional 2D web browsing experience, as well as an immersive 360 experience that places you inside the webpage. In this 2D experience, two issues were immediately apparent. The control center buttons were not visible, and there was no option to enter VR. These hurdles needed to be resolved. After investigating, I found both stemmed from the same source: a neglected feature from the template where VR devices display a prompt encompassing the screen, immediately offering the user the option to enter VR. At some point, the necessary calls to load and handle this screen were removed, and the action to hide the control center was built for a previous, much shorter version. I restored the prompt and adjusted the control panel animations so everything would display properly. Now the viewer can again begin on a 2D screen and choose to enter VR. VR Menu 3D VR Menu AI Dolphin& Bunny Using VR Pear Phone On Hover Changes A viewer can enter a virtual experience, but how can they control it? In some 3D tours there are hotspots, or markers, placed on the ground or throughout the space that can be clicked. These take the viewer to the next photo. But how would you control a video or an unrelated gallery? How would you hide hotspots to remove distractions and better immerse yourself in the scene? One option would be to map these actions to buttons on a controller. However, this could increase the learning curve and decrease engagement. Viewers would need to learn that “B” means hide icons, “A” means play, “X” means pause, and so on. Volume and video scrubbing would become even more complicated. Maintenance would also increase, since controllers vary across devices, and some devices have no controllers at all. Instead, we can provide a menu in VR. This emulates a 2D experience for controlling video, altering the scene, and iterating through gallery photos. Initially, I hoped to reuse the logic and design of the 2D menu within virtual reality. This cannot currently be done in modern web browsers. The 2D menu is built into the document object model (DOM). The DOM works well on 2D displays, but its structure does not translate into three-dimensional space. Since VR headsets are essentially two 2D screens positioned directly in front of your eyes, it might seem possible to place the menu directly on those screens. However, with screens only centimeters from your eyes, this would likely feel like a sign hanging from both eyelashes. The lack of depth would be disorienting and uncomfortable. A new solution was required. You may recall the previously mentioned hotspots—2D images positioned in 3D space that perform actions when clicked. These are already familiar to users and are often the primary method of interaction within 3D environments. Using this approach, we can rebuild the user interface with necessary adjustments to meet the expectations and limitations of 3D space. In 2D DOM menus, we have convenient grouping tools called containers. These allow items to be spaced evenly horizontally or vertically. Containers provide alignment controls, and spacing is defined locally within them. This means moving one container moves everything inside it. This convenience does not exist in 3D with hotspots. Each item must be individually placed relative to where the viewer is looking. This makes maintenance and future changes more complex. If you initially place a toolbar at the bottom and later move it to the top, every toolbar item must be updated individually. I was grateful to have waited until the 2D menu matured before building the 3D menu. Since the 2D design has been tested, the likelihood of major changes to the 3D version is reduced. During early testing of the 3D menu, I noticed that when scenes changed, the menu immediately disappeared. Opinions on this behavior may vary, which suggests that changing a scene does not inherently imply the menu should hide. These actions can be logically separate and therefore should be. A viewer may prefer to keep the menu positioned to their right while iterating through scenes or preparing to start a video. After brief investigation, I found a way to keep the menu visible and in its relative position while moving through scenes in both galleries and 3D tours. Another difference unique to the 3D menu is button indicators. In 2D, a highlight indicates when a button is hovered. In 3D, I initially struggled to replicate this behavior. After stepping away briefly, I considered alternative approaches. Instead of mimicking a flat highlight, I leveraged 3D space to animate movement similar to physical buttons. This animation feels responsive and clearly conveys interaction, so I chose to adopt it. Working in 3D comes both with challenges and benefits. One key benefit is the consistency of the viewer’s perspective. In 2D we have to account for devices of so many shapes, sizes, and resolutions. One menu must look good on a standard desktop monitor as well as ultra-wide screens, tall cell phones, older phones at 1080p, and newer screens beyond 4K. I put forth effort to craft one design that hopes to satisfy all viewport configurations, even ones as strange as the (fictional) Pear Phone. In VR, this challenge is reduced. Most human heads and