Web Design

Accomplishments Reconstruction Improvements Reconstruction Research Edge Curtain Fading Texture Improvements Bonus Integrated Cropping Tool Automatic Floor Grouping                 While the 3D reconstruction process continues to persist as the main point of focus, during this time a variety of adjacent improvements have been identified and implemented. These include visual distinctions like edge curtain fading and improved texturing through reorganizing how the mesh is textured. I was also able to integrate the multi-crop tool envisioned back in Week 7, and setup automatic floor separation. Reconstruction Research Improved Texture Previous Texture Edge Fade                 A variety of attempts were made this week to improve the 360 reconstruction algorithm, specifically around 3 key areas: same-side alignment, conflict resolution, and speed. One very promising option emerged that mostly solved alignment and conflicts; However, it struggled heavily in efficiency. The solve ballooned from 8 minutes for a large dataset to over 30. This was unacceptable. Overall, alignment was improved to be more consistent with higher datasets, however conflict still struggles to consistently move points outside of conflicting regions.                 A new idea emerged while determining how best to deal with the new content generated by retaining sky points. Sky points, and cubic limits were added so that 3D transitions do not look out to the infinite void. This improves transitions, but the mesh gains a lot of new information and it’s odd, unnatural, and mostly useless to the viewer. This is where edge curtain fading and transparency come in. By identifying the sky, and the sharp edges that lead to it, we can identify most of the geometry that is unlikely to be seen by the cameras; but is necessary for the enclosed structure. This geometry can be made transparent, as its only useful for avoiding voids. Hiding this also allows us to decimate it much further than the rest of the geometry, as we are not concerned with the structure. Decimating the geometry, and placing all the faces for it within the same small area of the texture map allow us to retain space for other faces and their textures, and minimize the additional data needed to store these points.                 The edges still have a consistent problem, often they are jagged, surreal, and unnatural. To fix this we can fade to transparent at these edges. Anything bordering a fully transparent face can be made to fade out, almost like the seam of a cloud. This is a lot softer on the eyes and lessens the visibility of the mesh artifacts.                 Additionally, I updated the texturing algorithm to better manage and distribute textures across the scene for higher quality viewing within the same resolution. Bonus: Integrated Cropping Tool Crop Tool                 Back in Week 7 we created a separate tool to assist with viewing and cropping 360 images with seamless transitions. This allows you to add photos to a carousel in apps that don’t support 360 photos, like Instagram, and give a pseudo 360 effect. Integrated directly in the image editing tool suite, it allows you to toggle how it’s cropped, and work on the edited or original photo. This outputs a zip file with all image crops ready to post. Bonus: Automatic Floor Grouping                 Multi-Floor tours has been a consistent goal for delivery in the 360-to-3D process. We’ve previously implemented solutions to manually group 3D floors so that they may be selected and have their transparency adjusted dynamically. This new improvement intends to automate some of that manual work through automatic floor grouping. At the press of a button a variety of criteria are used to identify and estimate which cameras belong on what floor. This simplifies the process, and is still open to manual edits for tailoring results. Intermediate cameras, like those on staircases, may go to one floor, when a desire is present for them to be on the other. In this case, the author can edit the group the cameras belong to. Summary:                 3D reconstruction remains the priority, and appears to be the final major hurdle to the finish line of a much more improved and robust process. Along this journey we were able to make meaningful other improvements this week. Integrating the cropping tool expands the functionality of the interface. Improving texturing and adding the edge fade improves the visual fidelity provided to the viewer. And automatic floor grouping improves the experience of an author and may save time.

Accomplishments Research Reconstruction User Interface Improvements Bonus Image Enhancements AI Image Editing Research Project Page Improvements Long Text Improvement                 Reconstruction research has continued to be the focus this week, identifying edge cases and discovering a pipeline that is effective and efficient has been the primary goal. While running tests, I was able to implement a variety of other improvements. These include new user interface features for organizing projects, adding structure to the individual image view and edit tools, and updating the tour to work better with long text. Reconstruction Research Full Image                 This week I further clarified the requirements needed for the final reconstructed 3D model and attempted a variety of solutions to achieve them. For a 3D model to feel more realistic while traversing it, it must conform to the real world in a few ways.                 One way is that objects rarely appear at infinite distances. Previously, some objects and the sky would appear infinitely far, and a sharp cut could be seen in this hole on the mesh, which could be jarring when traversing. To resolve this I added cubic limits to where each individual camera can address. This makes it so that sky, and the objects at its edge, do not appear infinitely far. Another issue is when one camera sees into the room of another camera. This happens due to a feature called “Backface culling”, and occurs when part of a camera’s wall gets lost while reconstructing. It can see through the back of another wall, into another room. This causes strange depth discontinuities while moving between images in a scene, and opens the opportunity for the user to mis-click and go to a space they had not prepared for. The added requirement is to ensure no camera sees through the backface of another, which in turn ensures that the camera’s walls are often present and in it’s expected positions.                 Two other cases were determined as beneficial: alignment and conflict resolution. The camera positions can be imperfect. Distortion, changes in light, and biases of the model can affect how cameras are positioned in 3D space. Even if the position of the camera is correct, the geometry it projects into 3D can be incorrectly estimated. This can happen on mirrors, walls, and sometimes doorways may appear flat, or two walls may appear at different distances. To resolve these we need to align similar walls, and resolve conflicting geometry. Aligning similar spaces is the simpler task. We already can determine confidence of specific points based on various criteria like distance from the camera and consensus from multiple cameras. This means that we can take not-so-confident points and move them along a lint until they clamp to their confident counterparts. This deforms estimated depths and establishes a consensus.                 Conflicts are a separate set of challenges. They can most easily be thought of as opposing walls. When one rooms wall goes beyond the placement of another room’s wall we must find an agreement so that the walls are on their own sides. This part is somewhat simple, each wall faces opposing directions, we can identify that and move them based on confidence. It gets more complicated at angles. When identifying how to deal with the corners near the ceiling and floor, and how to move them, or roughness along the wall. One option may be to identify that they are in different rooms. This is great for many cases, however, what about when the geometry is in the same room? Perhaps two cameras see the same waist-high fence from different sides? It would be ideal to have a robust conflict resolution for these cases. To solve conflict I considered an option where each point would be addressed within a voxel, and identify conflicting spaces based on their points. Once identified, we could move the group of voxels related to the conflicting region, and some connected vertices within adjacent voxels, until a consensus is reached. This seemed to greatly improve the edge cases (literally) and be a solution that would work even for objects in the same space. However, it was still imperfect and the time to complete was very high.                 This week came with many attempts, further refinement of the requirements, and new ideas for how to provide a meaningful solution. New options are currently being pursued. Bonus: Prompt Edits Full Image People Removed                 Another new feature for image editing is prompt edits. I identified this use case by viewing my gallery view photos. Sharp galre, scuffs on a lense, or a bit of moisture can add a lot of blur and problems to an image. While not perfect, local models have gotten incredibly capable of editing images.                 This feature allows for the whole image to be edited, or just a masked portion. Perhaps you want to remove all people, or you just want to erase a stray subject, you can do it from here. This process came with some challenges. First was identifying the best model. This process was already progressing from another task of improving the defurnishing tool. I reviewed a variety of models and found one that appeared substantial and had a version which could run on local hardware. While imperfect this was a big improvement. This model was also promptable and able to do many things well.                 The second challenge was working well with equirectangular photos. 360 photos are often not the primary type of photos included in models, which gives them a bias towards 2d images. This shows when whole images are updated and you view them in 360. The seam on the edges of the equirect are very noticeable as the image often changes color or shade at these sides. To fix this required processing edits with the whole image visible, requiring multiple steps, fading the changes, and aligning the outputs; However, this was able to be done in a reasonable time with good quality output.                 And lastly was improving the output quality.

Accomplishments Reconstruction Experimentation Automate Encryption                 Returning from vacation I began diving into a challenging problem, rethinking the reconstruction service. This is a challenging task, and may be necessary to better produce tours as new requirements are identified. Alongside this I was able to embed encryption directly into the tour export process. Reconstruction Experimentation                 The reconstruction service is the core of what brings posed point clouds into a tangible 3D surface. While reconstructing larger, more complex scenes, I began to notice many challenges. Cameras sometimes saw through walls of rooms, or saw into the infinite void in 3D space. Standing still, these are barely noticeable. However, once the camera moves or transitions it becomes jarring and hard to overlook. It also affects the 3D cursor and challenges a viewer’s ability to navigate the environment. This lead me to think differently about reconstruction, and identify new ways to make the scene watertight to better assist navigation and the user experience.                 This came with much research. Rebuilding a pipeline like this is challenging, and each step contributes to each other. So far, keeping each photo projected as a watertight space seems promising; However, it is challenging to identify how to best carve free space and consider how to resolve overlapping points on the same side of walls, or opposite. This research is ongoing. So far I have identified a variety of challenges, and mismatched solutions. Automate Encryption                 In a previous week I had setup encryption for the xml files used in viewing the immersive tour. This week I integrated that feature into the tour editor service. Now once all files are prepared they are downloaded as encrypted files. This can help protect content unique to the tour, as well as general plugins used across tours. I also identified and integrated another step for encrypting javascript files. Now the majority of functional content is able to be encrypted upon output. Summary:                 With automated encryption, output tours are more secure and the functional content is kept more private. As reconstruction research continues, that content will hopefully continue to come alongside an improved user experience. With the new needs identified, the challenge now is to identify how to resolve them.

Accomplishments New Datasets Captured Bonus Vacation                 For the 34th week of the year, I took some time off. I was blessed enough to visit the one happy island of Aruba. I am so grateful for the hospitality, sun, and sand I got to experience while travelling there.                 During my time away and recharging I still found the time to gather a few new datasets. Stay tuned for the upcoming tours of 3 new suites, as well as an immersive gallery. I captured two more tours while there, though they are outdoors and a bit more challenging. We will see how the results fair when running them through the 360 to 3D process.                 The week off was incredibly enjoyable. I am ready to get back to working towards providing a meaningful 360 to 3D tour experience for you to better experience environments from around the world.

Accomplishments Multi-Layer Research Bonus Mobile App Improvements Multi-Layer panorama images may have the potential to bring added depth to standalone 360 images. This week I dove deeper into considering what a process may look like to automate carving images from scenes and layering them for display. Alongside this, I made some improvements to the mobile application. Multi-Layer Research Depthmap Output 1 Output 2 Output 3                 For most of us, reality exists in 3 dimensions. Photos compress this and convey the world in 2D. 360 photos expand photos from only showing a crop of an image, and make it into a captured scene. To take that scene and add depth to it so far has required multiple images. Having multiple photos of a shared space gives the opportunity to triangulate points, adding depth. This is similar to how the distance between our eyes can offer us depth perception. This depth can be guessed with modern tools, and only one image; However, this alone is insufficient.                 Looking at estimated depth in 3D is still astounding, you can view the distance between an elephant and a person. Currently this comes at the price of strange movements and stretched seams. A 360 photo cannot see what is behind anything, so the disparity in depth becomes sharp lines that have no known color. This is likely to take someone out of the experience and make it feel less real. So what can we do?                 My current thought is to use a feature we’ve already added elsewhere, inpainting. Just like how we remove furniture from a scene, we should be able to fill in the holes where something previously existed. If a person is blocking a wall, we should be able to hide the person and paint the wall. This necessitates layering, the person and wall will need to live on separate layers.                 A variety of challenges come from this. For one, the unseen side of objects is not yet accounted for. If a person is facing the camera, how might we see the back of their head? While there is some promising research for generating this, I first want to test how it feels to have people and objects cutout of the scene, like a collage. Generating full 3d models from all image segments would be an ambitious idea, perhaps for another day.                 The more pressing challenge is adequately determining what to cut out. Visually a human could identify it simply. We want to remove the objects causing sharp edges. We have information to help us with this, the depthmap shows us clearly where those edges are. And the color images show us where the things are, however, how does the computer know what things to cut, and how to cut them?                 This week I have tested a variety of options to do just this, with interesting results. Across multiple test images, I have seen strong showings however no resolution has yet been consistent for all scenes. Collecting the edges is straightforward. What is less straightforward is identifying what they connect to. Using segmentation tools, we are able to assign every pixel of the image to a segment. This mostly works, the ground gets its own plane and a person theirs. This struggles with more abstract edges. Three trunks have sharp edges, while the leaves often don’t. And a boulder may be smooth against the ground and similar in color, while its edge is sharp against the forest behind it.                 These options show potential; However more refinement may be needed before moving this idea to further stages, like streaming multiple files and displaying them adequately in 3D while overlayed. Bonus: Mobile App Improvements                 A few new features were added to improve the user experience of the mobile app. Two small changes with big impacts were adding a delete button, and adding the ability to change the camera’s target. When working on-site with the application you may accidentally take a photo in the wrong place, or forget to change the target node before capturing. This will allow you to delete or replace the node by removing it or connecting to another edge node in the tree. Some other changes include adding back asynchronous inference and capture, while an image is being added to the 3D scene you can continue capturing the next photo. Also, height is now changeable throughout the process, instead of just at the beginning. And a bug was fixed where the HDR setting would contantly disable itself. Summary: Experiments breed creativity, and the bright concept of multi-layer depth images has lead me to experiment to see what new value we can generate from segmentation tools and depth maps. Not only has this research helped inform me further of a possible new feature, it also expands the horizons of capabilities available and illuminates possibilities to augment our current features further. This, alongside an improved mobile application, will help to promote further creation and more immersive viewing of spatial media.

Accomplishments Look Around UI Improvements Tour Editor Improvements General Improvements Bonus Defurnish User Experience Reconstruction Wall Consensus 360 photos and spatial scenes begin with single images visible to the camera. This week I focused on improving this foundation of the experience. This came through adding output explicitly for gallery tours, including those with no 3D model, as well as a variety of bug fixes and editor improvements. With the base improved, I began looking to take individual 360 photos to the next level with a new feature adding subtle depth to the images. Look Around Look Around Look Around is a feature where the camera is able to subtly look around corners within the 360 panorama. Instead of a camera rotating as if on a ball joint, a subtle offset is given from the camera to its initial position, and instead of pivoting, it orbits its origin. This offset allows the camera to perceive a subtle parallax in the 360 photo when paired with a depthmap. Combined, this gives a more immersive 3D effect to the scene. UI Improvements Aliasing Smooth A variety of improvements were added to the tour generator’s UI. The first is a new full resolution toggle. Most image information, like the original image or views of inpainting or blurs, up until now was shown at a reduced resolution. This feature allows for the full resolution image to be streamed. This gives the creator the opportunity to see the final tour files before generation, with the full resolution of all changes including inpaint and blur. Minor details can be caught here and edited before processing begins. Blur was also improved. The edge feathering was modified. Basic edge feathering allowed the feathered portion to overlap and extend outside the bounds of the mask, completely removing the blur. Now all items within the mask will be blurred regardless of feather position. Feather position was also updated to appear closer to the edge, reducing aliasing from the mask and ensuring a smooth transition between the blur and the image. Gallery Tour generation was also added and offers the ability to make tours without generating a 3D model. Tour Editor Improvements The tour editor and experience also came with many improvements. Infospots were updated to allow for the iframe size to be edited, as well as details about the card. This new editing ability was also extended to the VR popup menu. Up until now, an edited version of the default user interface icon image was used. This is a single image with a variety of subimages that are referenced by their location to be used in the scene. Now, each image is individual and can be uploaded and replaced from the editor. Lastly, the export option was configured. Now, the tour can be translated with all changes and exported from a single button, prepared to run. General Improvements Orthographic Artifacts A variety of general changes were identified and employed while testing the rest of the service. Gallery tours often employ previous and next buttons for scene iteration. These could sometimes land on the diorama in 3D tours and were not yet set up to cycle. Both of these improvements were addressed, as now in the tour editor specific scenes can be directly removed from the list of items to be iterated upon. The array of items is also now cycled during iteration. Some transition fades and a flicker noticed on hotspots at high orthographic views were updated and resolved. Glass masking and flattening was also reintegrated. This was reintroduced after noticing depth in glass mirrors during recent changes. Bonus: Defurnish User Experience Furnished Defurnished Defurnishing 360 photos and using them to generate a 3D mesh have been accomplished. This week, the necessary steps to view and export a tour with this feature were added. Now a new button and plugin can be added to the menu, where a chair icon is displayed. Upon clicking, the screen fades from furnished to unfurnished. Iterations and hotspot selection remain on the defurnished view, as well as the dollhouse, until furnished view is toggled back on. Bonus: Reconstruction Wall Consensus Living Room Hole West Room Hole Living Room Flat West Room Flat When depthmaps are estimated and cameras are posed, there is an opportunity for the point clouds to disagree on what goes where. if two adjacent rooms say that the walls overlap, the reconstruction process must be robust to handle this. Previously, overlapping walls could result in holes appearing and tearing through the mesh. This week I implemented a new process that first goes through the series of point clouds and identifies these overlaps. Then it intelligently determines where and how to move points that overlap incorrectly. some overlap should be destroyed, like when two point clouds occlude an area slightly differently. However, when the points have drastically different normal facing directions, it is likely that the points are from opposing locations and most likely are accurate relative to the camera, but perhaps not to the whole scene. These locations are then adjusted uniformly to give enough space to the conflicting areas in order to mesh properly.  Summary: The “flat,” 2D user experience is at the very core of spatial experiences. The 360 data captured by the camera is the most accurate data in the process. Gallery views allow this to be displayed even without a connected sequence. Depth and camera offset allow it to be upgraded and made more immersive. Creation tools and general improvements help better tailor the experience, and tools like defurnishing also add the ability to share even more information through it.

Accomplishments Reconstruction Improvements Bonus New Examples Converting point clouds and forming them into cohesive 3D models must be sophisticated enough to handle many edge cases. This week I was able to tackle some of these edge cases, as well as receive the meaningful opportunity to capture three new examples valuable for testing the end-to-end workflow. Improved 3D Reconstruction Post-Inpaint Pre-Inpaint This week I tackled an issue where, during one stage of reconstruction, floating blobs would often form in direct contradiction to the nearby camera. This can be clearly seen around the staircase in Supalai place. Point clouds and cameras at the ground level and on the staircase express a large amount of empty space. However, in the reconstruction, a large canopy would appear over the staircase. A variety of causes led to this. For one, it appears that the mathematical process which occurred over this field found a small number of stray points which remained and fused them into the final mesh, adding points to connect them. These stray points often came from missed votes in regions too close to the camera. After updating these processes, cameras across the scene better expressed their free space and reduced floating artifacts. Bonus: Hotel Examples Accommodation Balcony Lobby A recent stay at a local hotel began with an opportunity to capture the accommodation. This acted as a great opportunity to test the functionality of my mobile application on-site. Multiple test runs were performed. During each, I gained better insight into how my mobile app could be used on the job and the human factors it could better support. This led to a variety of new planned features, like changing the connected camera even after capture or inference. I admired the décor and common area and, upon checkout, requested permission to capture some photos. To my elation, it was granted. I eagerly began scanning the lobby and balcony. This allowed me to take the mobile app beyond user testing and into stress testing. It did well handling dozens of photos and consistently displayed the updated point cloud. This also allowed for an opportunity to test its adaptability. After scanning a small section, a staff member arrived for daily cleaning. I paused the project and moved on to another area. Later, when the cleaning had commenced, I reopened the project, eager to see if I could pick up where I left off. With the lighting and outdoors visually similar, and the architecture unchanged, I was able to. This was incredible and informative of the adaptability of the tool I had not yet considered. This experience also offered an opportunity to manage a dynamic environment. People entered and exited the lobby consistently. This was a great opportunity to use my cutout tool to mask and remove people from images. It was also a good opportunity to engage with others. Upon completion, a group of guests expressed interest in my activity. Our discussion appeared to excite and intrigue them, leaving me with smiles and best wishes. I am so grateful to the hotel for the accommodation and opportunity to practice and test my service in a live and well-designed environment. Summary: Focusing on reconstruction comes with challenge and patience. Many interconnected steps combine to form the final result, and changes to one area open reactions in the next. Balancing these choices and steps in a way that delivers more visually accurate results over a range of examples is difficult and rewarding. I am grateful for the progress made this week and the exciting opportunity to test my tool and gather examples to even further improve my research.

Accomplishments Live Projection Inpainting Head and License Plate Blue Tour Designer Improvements 2D Viewer Frontend Bonus Reconstruction Service Updates For many people, seeing (sensing) is believing. Coding and calculating can often be more challenging than visually adjusting something to your desires. Quickly updating visual responses can also help us brainstorm and identify what our desires are. If we want to artificially inpaint someone out of an image or blur their face, it’s important to visually see the outcome and be able to update it. For our tours, it’s important to see the changes we make to our user interface, as well as our 2D and 3D items, so it is helpful to have our visual tour editor available. These and many more integrations and improvements have been added this week to bring more visual information and ability to the creator more quickly and simply. Live Projection Inpaint Post-Inpaint Pre-Inpaint Live projection refers to what you may think of when viewing a 360 photo. This involves having it projected onto a sphere around you. This allows you to see the photo as if you were in that position, and you may want to edit the photo that way too. Drawing onto this sphere comes with challenges, as the photo itself comes from a 2D rectangle. This means that the pixels of that rectangle naturally stretch and warp over the sphere. Various calculations must be performed to effectively “reproject” this image and the manual drawings painted onto it so that it can be re-represented as a 2D image. Having solved this, a new feature is now available where small portions of the image can be edited manually using image generation tools to remove and replace them. This also includes a new step where the automated nadir inpainting from Week 29 can be removed and replaced with a manually inpainted result. Heading and License Plate Blur Face Blur License Plate Blur Privacy is important, and many people may not want to share faces in crowds, license plates, or other information in their photos and galleries. I have added a step that reviews each photo and places an optional blur over every head and license plate it identifies. This can miss faces and plates far in the distance, so I reused much of the live projection work completed during the inpainting step to improve this. The license plate blurs can now be manually updated. An undo mechanism was also introduced during this implementation, which supports undoing a sequence of actions during either license plate blurring or inpainting. Tour Editor Improvements Infospot Adding Translation Integration Up until now, the system could take in photos or mobile-captured data and generate information for the tour and 3D model; however, it did not yet build the tour. Building the tour required exporting data from the system, importing it into Blender with custom scripts, making modifications, and then exporting it again as files to overwrite autogenerated files. You also had to manually generate the tour data with another application and then host and edit it with a third. This meant four distinct applications were required to take the information generated in the frontend and bring it to a deliverable tour. This week, we reduced that to one, with an optional second. First, the tool that takes 360 photos and breaks them into tiles was set up and integrated with the process. A new “save and generate” button is available. This performs the time- and space-consuming process of applying the inpaint, nadir, and blur edits to copies of the high-resolution files, while also triggering each of those high-quality images, and the optional furniture-removed ones, to be broken into small tiles ready to stream online. This step also removes the dependency on custom Blender scripts. Now, the data can be exported directly to the files with the custom information needed to hook up each photo to its position, assign the correct 3D matrix, and set up any available gallery groups and hotspots. This sets up the basis of the eventual virtual experience, which may still need tailoring. I began integrating the customization tool developed in Week 27 with the main system. Now, the tour appears as an available tab, displays this tool in the web browser, and loads the information generated for the virtual experience with no extra work from the user. Previously, the tour designer tool only supported minor changes, such as updating the camera, diorama position, thumbnails, and gallery menus. I decided to expand this and implement a plugin templating process whereby any new plugin can be added and edited based on standard settings. This allows for tabs, groups, and subgroups, as well as custom settings. Now, each of my existing plugins can have a JSON file generated with its necessary custom settings, and based on the tags, they will appear as editable settings. Numbers will be sliders, colors will use color pickers, and text will remain text. I went through and updated each of my existing plugins to work this way, so plugins and their content can now be updated through the UI with immediate visual responsiveness. I took this plugin update a step further by creating a simple alternative to the existing workflow of using a third-party plugin for defining 3D coordinates for an info spot. We now use a first-party tool that can be edited and maintained, removes the need to manually adjust tags, and eliminates unneeded extraneous information. The info spots were further updated to support templating for the iframes they display. This way, the iframe data can also be edited visually. Previously, HTML files needed to be edited and tested. Now, this only needs to be done for the initial template, after which features within it can be adjusted through the user interface. With the benefit of having all plugins included in the tour and now templated, I used this templated information to directly integrate the translation tool into the program. Now, while updating the frontend, translation can be run from within the

Accomplishments Importing Mobile Data Automated Tripod Removal Automated Furniture Removal Bonus Benefits of the Mobile Tool Adams Family Vintage Store Update Capturing photos at eye level from all angles comes with many visual challenges. How do we hide the photographer? How do we hide the camera’s stand? Through recent developments we’ve solved how to hide the photographer with layered captures. And way back in week 1 we identified how to hide the camera stand. This week expands and rethinks those workflows to integrate more holistically with the end-to-end process. Using similar processes to camera stand removal, we can expand that concept to removing furniture from the whole room. Together these changes make the functionality of the system more robust and holistic. Importing Mobile Data Import Data The mobile app allows us to capture images, remove people, and track the sequence of images collected. The hardware in the phone currently affords us an incredible preview of the work. Transferring this to a computer allows us to perform more operations on the photos, use the sequence, and reconstruct the 3D scene much faster. A new feature to export the data was added to the mobile app. This exports information like the order of images taken, which images overlap, how they overlap, and the masks drawn to cut out the parts of images with people in them. Importing this information to the computer’s application offers many benefits. It allows the computer to cut and combine the full resolution images, rather than compressed previews. It also minimizes the need to manually combine and link sections of point clouds, saving time and focus. Automated Tripod Removal Original Removed In the very first week of these updates I shared the first workflow for automating tripod removal in 360 photos. I learned a lot from that solution, such as the tools and steps required to efficiently and effectively remove tripod feet from the nadir (bottom) of a 360 photograph. This week I took that understanding and identified a new path, using updated tools and integrating directly with the overall application. This new process works faster, with less manual input, and can be triggered automatically upon image upload. The fewer steps required to go from captured photos to shared 3D experience the better. This improvement satisfies that goal, deprecating the previous self-confined process in favor of an integrated tool that can be automatically triggered. Automated Furniture Removal Original Removed Back in Week 23 I created a proof of concept to see that furniture removal could be effectively performed locally, in a reasonable time, and be used in 3D positioning and reconstruction. This week I took those concepts and began integrating them into the service. Now, after tripods are removed from the floor, the high res image can be ran through a process which selectively removes furniture and attempts to inpaint or imagine what is most likely behind it. This often leads to furniture free rooms that can later be used for reconstruction. Bonus: Benefits of the Mobile Tool During the process of bridging the mobile information and the computer tool, I reflected upon what benefit this sibling application has. At first glance it could appear superficial. A lot of time has been spent, and work done, to accomplish estimating depth and positioning 360 photos on the mobile phone. This still takes around 30s of processing time, and the visual consists of voxel cubes which differ greatly from the final result. The positioning and point clouds also differ, because this inference is only used for immediate visualization. What benefit do we get from all of this effort, maintenance, and increased processing? The mobile application satisfies 3 major concerns: Capturing the correct visual data, ensuring valid pose-to-pose connections, and reducing manual effort to connect photos. Let’s imagine some scenarios without the mobile app. For one, let’s imagine an open field with hiding spots far from the camera’s position and wireless signal. The photographer has two options. One, set a timer, run to a hiding spot, and then return to the camera. This takes extra time depending on how far the hiding spot(s) are, and how frequent they must be returned to. And this entrusts that the shot will not change over that time. If outdoors, perhaps a deer is nearby, or the sun is free from cloud cover. The scene may change during the attempt to hide. Option two would be to take two photos, and move their body between each. Then a postprocessing workflow could be performed to cut out the person from each image and link them together. Perhaps lighting or other features changed during this process that the photographer was unaware of, and postprocessing requires extra manual time. The mobile app relieves these issues by allowing the photographer to capture multiple photos immediately, and see direct feedback as to what the combined capture will look like. This saves time and ensures that it will reflect the final result. Let’s imagine another scenario where you’re capturing a home. Doorways are incredibly tricky for positioning 360 images, as the amount of overlapping data is often very small. One great current example is the 3rd floor center balcony of my Supalai Place environment. The photos on either side of the doorway were taken approximately 2 meters apart. This was consistent with other photos taken throughout the house. However, in this case the pose-to-pose alignment was very far off. The outdoor image was sharply angled, and more distant than the ground truth. This was simple to fix with the manual adjustment too; However, it was just that, manual. This consumes my time and focus. Some instances may be even too much for a manual adjustment to fix, and in those cases the photographer may have to return to the site, where features may have changed, leading to more time, effort, and money. With the mobile app, photographers can monitor their camera-to-camera poses on-site. When a pose is too far away or crooked, this visual can inform the photographer that an

Accomplishments Mobile: Voxel Empty Space Human Detection Saved Projects Bonus UX:  Orthographic Transition Dynamic Location Hotspots Architecture Template HTML New improvements to the mobile capture tool improve its ability to capture and display environment details. Reducing noise from the output improves its visual representation, while saving projects allows scenes to be returned to and updated if pauses are necessary. And the new human detection system provides a workflow to automatically detect and remove humans while stitching multiple photos together. Beyond mobile, new improvements were made to the architecture and user experience. Now when looking at a diorama from a high angle the camera’s view smoothly translates from a natural perspective to a more architectural, or orthographic, flat look. Other changes were added too, like adjusting a hovered hotspots transparency, and hiding the current location hotspot of the camera. Changes were made to the architecture as well to introduce favorite icons and replace copy pasted html files with a reusable template. Mobile Improvements: Voxel Representation Human Detection & Edit The goal of mobile capture is to get on-site feedback for adjustments before returning to a workstation for processing. A sequence of unedited voxel clouds can become very noisy, as depth estimation often stretches and becomes incorrect at sharp edges and far distances. A voting process, similar to that used by the reconstruction phase, was implemented in order to swiftly remove noisy low confidence points. Human detection was also added as an improvement to the editing workflow. Using Apple’s lightweight ML models we can automatically detect humans in the live feed and captured images. These detections can be used to mask out parts of the image to be replaced by a following capture. In large open areas, hiding from the camera may take the phone out of range. This helps automate detection and hiding of people, and the photographer. Projects can now be saved and returned to later. Downsampled images, depthmaps, and position data are stored and ready to be returned to at later sessions. Bonus: UX Improvements Orthographic Perspective Hidden Hotspots New experiences have been made to the user experience (UX). These visual improvements relate to how dioramas can be viewed, and how location hotspots react to the viewer. The diorama was updated to slowly transition from a perspective, 3D depth, view of the environment, towards a more orthographic, 2D / Architectural, view of the environment. This change in perspective makes the visual appear more like a floorplan when viewed from up high. This view is often more familiar and interpretable than the angles visible when viewing in perspective 3D. Dynamic transparency for location hotspots was improved. Now when hovering over a hotspot, the 3D cursor hides and the hotspot transitions to a more solid color. This provides improved visual cues for which hotspot is to be clicked. Also, hotspots below the viewer now transition out, and in, over time. This prevents the user from re-selecting their current hotspot, causing an unnecessary transition. Architecture: Each experience begins with an HTML file. This file acts as the landing page for the experience. It loads basic things, like the favorite icon, and the experience’s root file. Most of the time these html files are very similar, with minor changes like the name of the file called, or the title of the page. To make this easier to maintain, I created a template html file. Now tours can access the same reused information, while injecting their specific information into it. This simplifies maintenance and allows for changes to one file to improve all experiences. Summary: Mobile capture represents the input, and the user’s experience represents the output. These improvements improve both and bring them closer together. Its important when capturing a user experience to identify problems and adjust to improve it as soon as possible. Changes like auto human detection and voxel noise removal make the capturer’s workflow faster and more clearly displays the idea of the final result at the initial stages. User experience improvements like shifting perspectives and dynamic hotspot transparencies affords the user more visual cues about the scene, and a more familiar display of information within it.