Week Reviews

Accomplishments Multi-Layer Research Bonus Mobile App Improvements Multi-Layer panorama images may have the potential to bring added depth to standalone 360 images. This week I dove deeper into considering what a process may look like to automate carving images from scenes and layering them for display. Alongside this, I made some improvements to the mobile application. Multi-Layer Research Depthmap Output 1 Output 2 Output 3                 For most of us, reality exists in 3 dimensions. Photos compress this and convey the world in 2D. 360 photos expand photos from only showing a crop of an image, and make it into a captured scene. To take that scene and add depth to it so far has required multiple images. Having multiple photos of a shared space gives the opportunity to triangulate points, adding depth. This is similar to how the distance between our eyes can offer us depth perception. This depth can be guessed with modern tools, and only one image; However, this alone is insufficient.                 Looking at estimated depth in 3D is still astounding, you can view the distance between an elephant and a person. Currently this comes at the price of strange movements and stretched seams. A 360 photo cannot see what is behind anything, so the disparity in depth becomes sharp lines that have no known color. This is likely to take someone out of the experience and make it feel less real. So what can we do?                 My current thought is to use a feature we’ve already added elsewhere, inpainting. Just like how we remove furniture from a scene, we should be able to fill in the holes where something previously existed. If a person is blocking a wall, we should be able to hide the person and paint the wall. This necessitates layering, the person and wall will need to live on separate layers.                 A variety of challenges come from this. For one, the unseen side of objects is not yet accounted for. If a person is facing the camera, how might we see the back of their head? While there is some promising research for generating this, I first want to test how it feels to have people and objects cutout of the scene, like a collage. Generating full 3d models from all image segments would be an ambitious idea, perhaps for another day.                 The more pressing challenge is adequately determining what to cut out. Visually a human could identify it simply. We want to remove the objects causing sharp edges. We have information to help us with this, the depthmap shows us clearly where those edges are. And the color images show us where the things are, however, how does the computer know what things to cut, and how to cut them?                 This week I have tested a variety of options to do just this, with interesting results. Across multiple test images, I have seen strong showings however no resolution has yet been consistent for all scenes. Collecting the edges is straightforward. What is less straightforward is identifying what they connect to. Using segmentation tools, we are able to assign every pixel of the image to a segment. This mostly works, the ground gets its own plane and a person theirs. This struggles with more abstract edges. Three trunks have sharp edges, while the leaves often don’t. And a boulder may be smooth against the ground and similar in color, while its edge is sharp against the forest behind it.                 These options show potential; However more refinement may be needed before moving this idea to further stages, like streaming multiple files and displaying them adequately in 3D while overlayed. Bonus: Mobile App Improvements                 A few new features were added to improve the user experience of the mobile app. Two small changes with big impacts were adding a delete button, and adding the ability to change the camera’s target. When working on-site with the application you may accidentally take a photo in the wrong place, or forget to change the target node before capturing. This will allow you to delete or replace the node by removing it or connecting to another edge node in the tree. Some other changes include adding back asynchronous inference and capture, while an image is being added to the 3D scene you can continue capturing the next photo. Also, height is now changeable throughout the process, instead of just at the beginning. And a bug was fixed where the HDR setting would contantly disable itself. Summary: Experiments breed creativity, and the bright concept of multi-layer depth images has lead me to experiment to see what new value we can generate from segmentation tools and depth maps. Not only has this research helped inform me further of a possible new feature, it also expands the horizons of capabilities available and illuminates possibilities to augment our current features further. This, alongside an improved mobile application, will help to promote further creation and more immersive viewing of spatial media.

Accomplishments Look Around UI Improvements Tour Editor Improvements General Improvements Bonus Defurnish User Experience Reconstruction Wall Consensus 360 photos and spatial scenes begin with single images visible to the camera. This week I focused on improving this foundation of the experience. This came through adding output explicitly for gallery tours, including those with no 3D model, as well as a variety of bug fixes and editor improvements. With the base improved, I began looking to take individual 360 photos to the next level with a new feature adding subtle depth to the images. Look Around Look Around Look Around is a feature where the camera is able to subtly look around corners within the 360 panorama. Instead of a camera rotating as if on a ball joint, a subtle offset is given from the camera to its initial position, and instead of pivoting, it orbits its origin. This offset allows the camera to perceive a subtle parallax in the 360 photo when paired with a depthmap. Combined, this gives a more immersive 3D effect to the scene. UI Improvements Aliasing Smooth A variety of improvements were added to the tour generator’s UI. The first is a new full resolution toggle. Most image information, like the original image or views of inpainting or blurs, up until now was shown at a reduced resolution. This feature allows for the full resolution image to be streamed. This gives the creator the opportunity to see the final tour files before generation, with the full resolution of all changes including inpaint and blur. Minor details can be caught here and edited before processing begins. Blur was also improved. The edge feathering was modified. Basic edge feathering allowed the feathered portion to overlap and extend outside the bounds of the mask, completely removing the blur. Now all items within the mask will be blurred regardless of feather position. Feather position was also updated to appear closer to the edge, reducing aliasing from the mask and ensuring a smooth transition between the blur and the image. Gallery Tour generation was also added and offers the ability to make tours without generating a 3D model. Tour Editor Improvements The tour editor and experience also came with many improvements. Infospots were updated to allow for the iframe size to be edited, as well as details about the card. This new editing ability was also extended to the VR popup menu. Up until now, an edited version of the default user interface icon image was used. This is a single image with a variety of subimages that are referenced by their location to be used in the scene. Now, each image is individual and can be uploaded and replaced from the editor. Lastly, the export option was configured. Now, the tour can be translated with all changes and exported from a single button, prepared to run. General Improvements Orthographic Artifacts A variety of general changes were identified and employed while testing the rest of the service. Gallery tours often employ previous and next buttons for scene iteration. These could sometimes land on the diorama in 3D tours and were not yet set up to cycle. Both of these improvements were addressed, as now in the tour editor specific scenes can be directly removed from the list of items to be iterated upon. The array of items is also now cycled during iteration. Some transition fades and a flicker noticed on hotspots at high orthographic views were updated and resolved. Glass masking and flattening was also reintegrated. This was reintroduced after noticing depth in glass mirrors during recent changes. Bonus: Defurnish User Experience Furnished Defurnished Defurnishing 360 photos and using them to generate a 3D mesh have been accomplished. This week, the necessary steps to view and export a tour with this feature were added. Now a new button and plugin can be added to the menu, where a chair icon is displayed. Upon clicking, the screen fades from furnished to unfurnished. Iterations and hotspot selection remain on the defurnished view, as well as the dollhouse, until furnished view is toggled back on. Bonus: Reconstruction Wall Consensus Living Room Hole West Room Hole Living Room Flat West Room Flat When depthmaps are estimated and cameras are posed, there is an opportunity for the point clouds to disagree on what goes where. if two adjacent rooms say that the walls overlap, the reconstruction process must be robust to handle this. Previously, overlapping walls could result in holes appearing and tearing through the mesh. This week I implemented a new process that first goes through the series of point clouds and identifies these overlaps. Then it intelligently determines where and how to move points that overlap incorrectly. some overlap should be destroyed, like when two point clouds occlude an area slightly differently. However, when the points have drastically different normal facing directions, it is likely that the points are from opposing locations and most likely are accurate relative to the camera, but perhaps not to the whole scene. These locations are then adjusted uniformly to give enough space to the conflicting areas in order to mesh properly.  Summary: The “flat,” 2D user experience is at the very core of spatial experiences. The 360 data captured by the camera is the most accurate data in the process. Gallery views allow this to be displayed even without a connected sequence. Depth and camera offset allow it to be upgraded and made more immersive. Creation tools and general improvements help better tailor the experience, and tools like defurnishing also add the ability to share even more information through it.

Accomplishments Reconstruction Improvements Bonus New Examples Converting point clouds and forming them into cohesive 3D models must be sophisticated enough to handle many edge cases. This week I was able to tackle some of these edge cases, as well as receive the meaningful opportunity to capture three new examples valuable for testing the end-to-end workflow. Improved 3D Reconstruction Post-Inpaint Pre-Inpaint This week I tackled an issue where, during one stage of reconstruction, floating blobs would often form in direct contradiction to the nearby camera. This can be clearly seen around the staircase in Supalai place. Point clouds and cameras at the ground level and on the staircase express a large amount of empty space. However, in the reconstruction, a large canopy would appear over the staircase. A variety of causes led to this. For one, it appears that the mathematical process which occurred over this field found a small number of stray points which remained and fused them into the final mesh, adding points to connect them. These stray points often came from missed votes in regions too close to the camera. After updating these processes, cameras across the scene better expressed their free space and reduced floating artifacts. Bonus: Hotel Examples Accommodation Balcony Lobby A recent stay at a local hotel began with an opportunity to capture the accommodation. This acted as a great opportunity to test the functionality of my mobile application on-site. Multiple test runs were performed. During each, I gained better insight into how my mobile app could be used on the job and the human factors it could better support. This led to a variety of new planned features, like changing the connected camera even after capture or inference. I admired the décor and common area and, upon checkout, requested permission to capture some photos. To my elation, it was granted. I eagerly began scanning the lobby and balcony. This allowed me to take the mobile app beyond user testing and into stress testing. It did well handling dozens of photos and consistently displayed the updated point cloud. This also allowed for an opportunity to test its adaptability. After scanning a small section, a staff member arrived for daily cleaning. I paused the project and moved on to another area. Later, when the cleaning had commenced, I reopened the project, eager to see if I could pick up where I left off. With the lighting and outdoors visually similar, and the architecture unchanged, I was able to. This was incredible and informative of the adaptability of the tool I had not yet considered. This experience also offered an opportunity to manage a dynamic environment. People entered and exited the lobby consistently. This was a great opportunity to use my cutout tool to mask and remove people from images. It was also a good opportunity to engage with others. Upon completion, a group of guests expressed interest in my activity. Our discussion appeared to excite and intrigue them, leaving me with smiles and best wishes. I am so grateful to the hotel for the accommodation and opportunity to practice and test my service in a live and well-designed environment. Summary: Focusing on reconstruction comes with challenge and patience. Many interconnected steps combine to form the final result, and changes to one area open reactions in the next. Balancing these choices and steps in a way that delivers more visually accurate results over a range of examples is difficult and rewarding. I am grateful for the progress made this week and the exciting opportunity to test my tool and gather examples to even further improve my research.

Accomplishments Live Projection Inpainting Head and License Plate Blue Tour Designer Improvements 2D Viewer Frontend Bonus Reconstruction Service Updates For many people, seeing (sensing) is believing. Coding and calculating can often be more challenging than visually adjusting something to your desires. Quickly updating visual responses can also help us brainstorm and identify what our desires are. If we want to artificially inpaint someone out of an image or blur their face, it’s important to visually see the outcome and be able to update it. For our tours, it’s important to see the changes we make to our user interface, as well as our 2D and 3D items, so it is helpful to have our visual tour editor available. These and many more integrations and improvements have been added this week to bring more visual information and ability to the creator more quickly and simply. Live Projection Inpaint Post-Inpaint Pre-Inpaint Live projection refers to what you may think of when viewing a 360 photo. This involves having it projected onto a sphere around you. This allows you to see the photo as if you were in that position, and you may want to edit the photo that way too. Drawing onto this sphere comes with challenges, as the photo itself comes from a 2D rectangle. This means that the pixels of that rectangle naturally stretch and warp over the sphere. Various calculations must be performed to effectively “reproject” this image and the manual drawings painted onto it so that it can be re-represented as a 2D image. Having solved this, a new feature is now available where small portions of the image can be edited manually using image generation tools to remove and replace them. This also includes a new step where the automated nadir inpainting from Week 29 can be removed and replaced with a manually inpainted result. Heading and License Plate Blur Face Blur License Plate Blur Privacy is important, and many people may not want to share faces in crowds, license plates, or other information in their photos and galleries. I have added a step that reviews each photo and places an optional blur over every head and license plate it identifies. This can miss faces and plates far in the distance, so I reused much of the live projection work completed during the inpainting step to improve this. The license plate blurs can now be manually updated. An undo mechanism was also introduced during this implementation, which supports undoing a sequence of actions during either license plate blurring or inpainting. Tour Editor Improvements Infospot Adding Translation Integration Up until now, the system could take in photos or mobile-captured data and generate information for the tour and 3D model; however, it did not yet build the tour. Building the tour required exporting data from the system, importing it into Blender with custom scripts, making modifications, and then exporting it again as files to overwrite autogenerated files. You also had to manually generate the tour data with another application and then host and edit it with a third. This meant four distinct applications were required to take the information generated in the frontend and bring it to a deliverable tour. This week, we reduced that to one, with an optional second. First, the tool that takes 360 photos and breaks them into tiles was set up and integrated with the process. A new “save and generate” button is available. This performs the time- and space-consuming process of applying the inpaint, nadir, and blur edits to copies of the high-resolution files, while also triggering each of those high-quality images, and the optional furniture-removed ones, to be broken into small tiles ready to stream online. This step also removes the dependency on custom Blender scripts. Now, the data can be exported directly to the files with the custom information needed to hook up each photo to its position, assign the correct 3D matrix, and set up any available gallery groups and hotspots. This sets up the basis of the eventual virtual experience, which may still need tailoring. I began integrating the customization tool developed in Week 27 with the main system. Now, the tour appears as an available tab, displays this tool in the web browser, and loads the information generated for the virtual experience with no extra work from the user. Previously, the tour designer tool only supported minor changes, such as updating the camera, diorama position, thumbnails, and gallery menus. I decided to expand this and implement a plugin templating process whereby any new plugin can be added and edited based on standard settings. This allows for tabs, groups, and subgroups, as well as custom settings. Now, each of my existing plugins can have a JSON file generated with its necessary custom settings, and based on the tags, they will appear as editable settings. Numbers will be sliders, colors will use color pickers, and text will remain text. I went through and updated each of my existing plugins to work this way, so plugins and their content can now be updated through the UI with immediate visual responsiveness. I took this plugin update a step further by creating a simple alternative to the existing workflow of using a third-party plugin for defining 3D coordinates for an info spot. We now use a first-party tool that can be edited and maintained, removes the need to manually adjust tags, and eliminates unneeded extraneous information. The info spots were further updated to support templating for the iframes they display. This way, the iframe data can also be edited visually. Previously, HTML files needed to be edited and tested. Now, this only needs to be done for the initial template, after which features within it can be adjusted through the user interface. With the benefit of having all plugins included in the tour and now templated, I used this templated information to directly integrate the translation tool into the program. Now, while updating the frontend, translation can be run from within the

Accomplishments Importing Mobile Data Automated Tripod Removal Automated Furniture Removal Bonus Benefits of the Mobile Tool Adams Family Vintage Store Update Capturing photos at eye level from all angles comes with many visual challenges. How do we hide the photographer? How do we hide the camera’s stand? Through recent developments we’ve solved how to hide the photographer with layered captures. And way back in week 1 we identified how to hide the camera stand. This week expands and rethinks those workflows to integrate more holistically with the end-to-end process. Using similar processes to camera stand removal, we can expand that concept to removing furniture from the whole room. Together these changes make the functionality of the system more robust and holistic. Importing Mobile Data Import Data The mobile app allows us to capture images, remove people, and track the sequence of images collected. The hardware in the phone currently affords us an incredible preview of the work. Transferring this to a computer allows us to perform more operations on the photos, use the sequence, and reconstruct the 3D scene much faster. A new feature to export the data was added to the mobile app. This exports information like the order of images taken, which images overlap, how they overlap, and the masks drawn to cut out the parts of images with people in them. Importing this information to the computer’s application offers many benefits. It allows the computer to cut and combine the full resolution images, rather than compressed previews. It also minimizes the need to manually combine and link sections of point clouds, saving time and focus. Automated Tripod Removal Original Removed In the very first week of these updates I shared the first workflow for automating tripod removal in 360 photos. I learned a lot from that solution, such as the tools and steps required to efficiently and effectively remove tripod feet from the nadir (bottom) of a 360 photograph. This week I took that understanding and identified a new path, using updated tools and integrating directly with the overall application. This new process works faster, with less manual input, and can be triggered automatically upon image upload. The fewer steps required to go from captured photos to shared 3D experience the better. This improvement satisfies that goal, deprecating the previous self-confined process in favor of an integrated tool that can be automatically triggered. Automated Furniture Removal Original Removed Back in Week 23 I created a proof of concept to see that furniture removal could be effectively performed locally, in a reasonable time, and be used in 3D positioning and reconstruction. This week I took those concepts and began integrating them into the service. Now, after tripods are removed from the floor, the high res image can be ran through a process which selectively removes furniture and attempts to inpaint or imagine what is most likely behind it. This often leads to furniture free rooms that can later be used for reconstruction. Bonus: Benefits of the Mobile Tool During the process of bridging the mobile information and the computer tool, I reflected upon what benefit this sibling application has. At first glance it could appear superficial. A lot of time has been spent, and work done, to accomplish estimating depth and positioning 360 photos on the mobile phone. This still takes around 30s of processing time, and the visual consists of voxel cubes which differ greatly from the final result. The positioning and point clouds also differ, because this inference is only used for immediate visualization. What benefit do we get from all of this effort, maintenance, and increased processing? The mobile application satisfies 3 major concerns: Capturing the correct visual data, ensuring valid pose-to-pose connections, and reducing manual effort to connect photos. Let’s imagine some scenarios without the mobile app. For one, let’s imagine an open field with hiding spots far from the camera’s position and wireless signal. The photographer has two options. One, set a timer, run to a hiding spot, and then return to the camera. This takes extra time depending on how far the hiding spot(s) are, and how frequent they must be returned to. And this entrusts that the shot will not change over that time. If outdoors, perhaps a deer is nearby, or the sun is free from cloud cover. The scene may change during the attempt to hide. Option two would be to take two photos, and move their body between each. Then a postprocessing workflow could be performed to cut out the person from each image and link them together. Perhaps lighting or other features changed during this process that the photographer was unaware of, and postprocessing requires extra manual time. The mobile app relieves these issues by allowing the photographer to capture multiple photos immediately, and see direct feedback as to what the combined capture will look like. This saves time and ensures that it will reflect the final result. Let’s imagine another scenario where you’re capturing a home. Doorways are incredibly tricky for positioning 360 images, as the amount of overlapping data is often very small. One great current example is the 3rd floor center balcony of my Supalai Place environment. The photos on either side of the doorway were taken approximately 2 meters apart. This was consistent with other photos taken throughout the house. However, in this case the pose-to-pose alignment was very far off. The outdoor image was sharply angled, and more distant than the ground truth. This was simple to fix with the manual adjustment too; However, it was just that, manual. This consumes my time and focus. Some instances may be even too much for a manual adjustment to fix, and in those cases the photographer may have to return to the site, where features may have changed, leading to more time, effort, and money. With the mobile app, photographers can monitor their camera-to-camera poses on-site. When a pose is too far away or crooked, this visual can inform the photographer that an

Accomplishments Mobile: Voxel Empty Space Human Detection Saved Projects Bonus UX:  Orthographic Transition Dynamic Location Hotspots Architecture Template HTML New improvements to the mobile capture tool improve its ability to capture and display environment details. Reducing noise from the output improves its visual representation, while saving projects allows scenes to be returned to and updated if pauses are necessary. And the new human detection system provides a workflow to automatically detect and remove humans while stitching multiple photos together. Beyond mobile, new improvements were made to the architecture and user experience. Now when looking at a diorama from a high angle the camera’s view smoothly translates from a natural perspective to a more architectural, or orthographic, flat look. Other changes were added too, like adjusting a hovered hotspots transparency, and hiding the current location hotspot of the camera. Changes were made to the architecture as well to introduce favorite icons and replace copy pasted html files with a reusable template. Mobile Improvements: Voxel Representation Human Detection & Edit The goal of mobile capture is to get on-site feedback for adjustments before returning to a workstation for processing. A sequence of unedited voxel clouds can become very noisy, as depth estimation often stretches and becomes incorrect at sharp edges and far distances. A voting process, similar to that used by the reconstruction phase, was implemented in order to swiftly remove noisy low confidence points. Human detection was also added as an improvement to the editing workflow. Using Apple’s lightweight ML models we can automatically detect humans in the live feed and captured images. These detections can be used to mask out parts of the image to be replaced by a following capture. In large open areas, hiding from the camera may take the phone out of range. This helps automate detection and hiding of people, and the photographer. Projects can now be saved and returned to later. Downsampled images, depthmaps, and position data are stored and ready to be returned to at later sessions. Bonus: UX Improvements Orthographic Perspective Hidden Hotspots New experiences have been made to the user experience (UX). These visual improvements relate to how dioramas can be viewed, and how location hotspots react to the viewer. The diorama was updated to slowly transition from a perspective, 3D depth, view of the environment, towards a more orthographic, 2D / Architectural, view of the environment. This change in perspective makes the visual appear more like a floorplan when viewed from up high. This view is often more familiar and interpretable than the angles visible when viewing in perspective 3D. Dynamic transparency for location hotspots was improved. Now when hovering over a hotspot, the 3D cursor hides and the hotspot transitions to a more solid color. This provides improved visual cues for which hotspot is to be clicked. Also, hotspots below the viewer now transition out, and in, over time. This prevents the user from re-selecting their current hotspot, causing an unnecessary transition. Architecture: Each experience begins with an HTML file. This file acts as the landing page for the experience. It loads basic things, like the favorite icon, and the experience’s root file. Most of the time these html files are very similar, with minor changes like the name of the file called, or the title of the page. To make this easier to maintain, I created a template html file. Now tours can access the same reused information, while injecting their specific information into it. This simplifies maintenance and allows for changes to one file to improve all experiences. Summary: Mobile capture represents the input, and the user’s experience represents the output. These improvements improve both and bring them closer together. Its important when capturing a user experience to identify problems and adjust to improve it as soon as possible. Changes like auto human detection and voxel noise removal make the capturer’s workflow faster and more clearly displays the idea of the final result at the initial stages. User experience improvements like shifting perspectives and dynamic hotspot transparencies affords the user more visual cues about the scene, and a more familiar display of information within it.

Accomplishments New Customization Tool Mobile Capture Tool Updates Minor UX Fixes Seeing is believing. When designing an experience, you want immediate feedback as you brainstorm and create your design. This feedback extends the width of the process, from capturing images to tailoring menus and color schemes to fit the needs of the project. New Customization Tool: 2D Updater VR Updater Thumbnail Updater Edits List The standard way for adjusting tours with certain tools involves manually describing changes in a text file, saving it, and refreshing a page to see how it looks. Experienced designers can quickly identify and update the files for the changes they’d like. For those less familiar, or looking to experiment, this can be a cumbersome and slow process. With this new customization tool, the skill barrier for adjusting experiences has been drastically decreased, and the speed to do so has been greatly improved. With this tool, changes can be made with sliders and drop downs which live update the custom grid menu’s both in 2D and VR. I have also added a feature to select and update thumbnails for each photo, so that they can be set to more visually desirable locations for the grid. This all happens live, with visual feedback for the designer to consider. And I have integrated last week’s camera position tools into this interface, and included a list of images to inform the designer which have been updated, and which have not yet been changed. Reducing the skill ceiling, improving efficiency, and simplifying the tasks of changing thumbnails and camera positions, all without manually editing fields in a file, greatly improves the process for designing a tour, shortens the time, and adds more value to the individual parts. Mobile Capture Tool Updates: Updated Visual Mobile Capture empowers designers to see the structure of the final result while on-site and during capture. This ensures that things are not missed, and gives the opportunity to reshoot or gather more information for areas that appear to need more detail. This can reduce re-shoots and manual processing later. This week, I implemented multiple new features to support this process. The first major feature is to connect the 360 camera to the application. For this case, I use an insta360 x5. Insta 360 offers a standard development kit (SDK) for interfacing with their cameras. Once approved, I was able to implement this within my application and begin viewing a live feed from the camera. The capture button now sends a signal to the camera to capture a photo, and it sends the photo back to the phone for further processing. The SDK and connection also afford the opportunity to adjust the camera’s settings remotely. Things like exposure, HDR, resolution, and more can be changed from the mobile phone without having to disconnect or manually adjust within the camera. With camera connectivity I could now build out sequential photography and point cloud representations. Each captured photo is run through the process against the previous “anchor” photo. This generates the new point cloud, and it’s pose relative to the last camera. Each new photo is then displayed as a new node, creating an interconnected tree to represent the space. Selecting an anchor node can let you return to previous areas for new branches, or refining existing ones. A big problem I face is capturing tours in open areas. Large rooms or outdoor scenes where there is no place to hide, or, the best place to hide is outside of the mobile phone’s connectivity range. This can mean spending more time hiding and returning than actually capturing the photos. For this I added an edit workflow. Now, after a photo is taken, you can choose to draw over parts of the photo you want cut out. Then, you can take a second photo with the camera, and combine them. As long as the camera does not move, the two photos should blend well together. Letting you appear in both shots, and removing you from their final result. Point clouds can be great visualization tools, but when zoomed they may appear too sparse to offer the detail needed to be seen. For this I have updated the 3D visualization on mobile to display a voxel grid. Voxels are just cubes, and these can be colored to display the scene, and update as it captures more data. These solid structures make it easier to see from close or far away, offering better visual feedback to the designer. Minor UX Fixes: A few small irregularities were identified and resolved through this week. One involving the VR menu. It would not maintain its position relative to the user’s head after a transition. Naturally, when going from one end of a house to another, we expect ourselves to feel like we have moved. However; subconsciously we see a menu as an extension of ourselves, like a watch or a tablet. When we teleport, we expect it to teleport with us. That makes it jarring when it stays in place, or changes slightly after every transition. This is now resolved, and it remains fixed to the viewer. This came alongside a minor fix, where in VR or 2D, sometimes selecting items from the menu would transition in 3D, causing a disorienting move through walls and fixtures, rather than a simple fade out and fade in. The multi-floor feature came with substantial changes to how dioramas are loaded and experienced. One consequence of this was a momentary lack of texture for the diorama view when transitioning from the depthmapped scene. I added a new step that pre-loads the textures for the diorama, so that they are immediately ready when transitioning. A minor issue was identified with the auto-sizing of the 2D menu. When entering and leaving full screen, the width of the 2D menu would change even if the browser size had not. Now the 2D menu remains consistent after exiting full screen. Summary: What you see is what you get. Giving the designer more

Accomplishments Multi-Floor Display Updated Experiences Mid-Year Review Bonus Section Generation Camera Orientation Tools Local Language Automation Halfway around the sun. That is how far we have travelled over the last 6 months. An incredible distance. And with this week, another incredible distance has been accomplished, this time in the world of spatial experiences. Each week has included accomplishments and research produced to detail and create processes and tools for efficiently delivering better spatial experiences. Many improvements along the way contribute to this. Most notably, the User Experience, and the spatial reconstruction tool. While an incredible amount of improvements went into these, a meaningful summary can be found in my H1 video linked alongside this page. This week brought H1 to the finish line, and delivered a meaningful end-to-end workflow alongside tools for delivering meaningful spatial experiences more efficiently, and affordably, than 6 months prior. To complete this chapter, I implemented a few final features. I improved the multi-floor display tool first drafted in week 2. I also defined section generation in the frontend, to pair with the responsive grid designed that same week. I created a new translator tool with a frontend that now simplifies the control over what gets translated and now automatically performs the translations using a local model, rather than calling an API, while still operating nicely with the structures defined in week 5. And to better tailor the visual experience, I created two new editors for setting the camera’s position when viewing the diorama, as well as each scene’s starting view. Each new feature has been a step closer towards the original vision. Nothing is more telling than that than the fact that many of the features in the final week still cooperate with and improve upon the concepts defined in the early weeks. Multi-Floor Display Multi-Floor Display One of the driving inspirations for this greater effort was determining how best to share large, multi-floor scenes. Many spaces have multiple levels, and the larger the spaces, the more they benefit from spatial awareness and the user being able to see where to go. Using a standard 3D model is often opaque and can be difficult to navigate. If you have a 3-floor home, and a walled-off room in the center of floor 2, it is very unlikely you can see or click it. But what if the tour were smart enough to know when you want an unhindered view of the floor? When facing a floor on the horizontal plane, you often only see its walls, as the floor and ceiling are parallel to your vision. Here is a great opportunity to display other floors. Since you can’t see the ceiling or floor, levels above and below the selected one can fade in and become clickable to change the active floor. And when you angle up, the room’s floor comes into view and the walls fade away. These angles offer the opportunity to select locations, and require the other floors to disappear for focus and an uninterrupted click. This result solves one of the bigger problems for a challenging tour, Supalai Place. This 3-story building with interiors and exteriors had many rooms, often ones which may be difficult to select if not for multi-floor display. With this feature, it is now much simpler to move quickly across the house. It reduces manually walking through, or reading the gallery, to just 2 clicks. One to the diorama, and one to the room. Bonus: Section Generation The responsive grid works great to simplify navigation and describe areas within a building alongside text descriptions. A challenge with it was manually defining a group and naming it for each scene. This is trivial for a dozen photos. It is much more cumbersome for a series of 100+ items, and much more prone to human error. To solve this, I added a feature to the frontend where multiple images may be selected and set to a selection. These selections can then be exported alongside the other data and used when automatically writing the tour.xml file. This feature replaced manually placing keys throughout a file with a GUI-based definition including drag-selecting multiple cameras for a shared selection. Bonus: Camera Placement Tools A beautiful diorama is best seen from a good angle. If the camera places itself below or far away from the diorama, it may regularly cause users to have to move and zoom the camera to find a good location. This hinders the ability to navigate and appreciate a space. Also, when selecting a position, the camera’s angle may be unexpected. Staring at a blank wall or having a screen full of leaves may be momentarily jarring. And again, this causes the viewer to have to move and zoom in order to get an idea of what the scene is. These were difficult to accept. Fixes do exist for them; manually calculating a good camera position and angle can be done, as well as manually copying coordinates for each photo’s camera to look at. This can add minutes or hours when creating an experience. With two new tools, they now take seconds per instance. Each diorama needs the camera to be placed in the desired spot, and at a click of a button the replacement data is available. The same goes for camera orientation per image. For each image, just adjust the camera, click a button, and its information is prepared. Bonus: Local Language Automation Translation Tool Multi-lingual support currently requires two steps: generating the translations and displaying them. The process was mostly formed in week 5. Displaying them has remained largely the same. The user selects a language and immediately all translated text is updated. However, under the hood, a lot has changed. Previously, translations were done by manually typing a list of keys that were to be searched for and translated. Then, a paid API call was made to a machine learning tool in order to translate the files. And then new files were generated for each translation.

Accomplishments Exporting the Experience Remade Spatial Experiences Bonus Mobile Proof of Concept This week’s focus has turned back toward the penultimate application of the 3D reconstruction: the spatial experience. New features across the frontend and Blender were built to automate the generation and transfer of information necessary to generate and navigate a 3D scene. In addition, I spent some free time preparing a proof of concept on a mobile device. Exporting to Experience Exporting Hotspot Locations.png To take the information we have in the frontend and view it in a bespoke virtual tour, we must transfer and translate it in a way that the virtual tour software can use. The 3D model and textures are simple enough to copy and export within the frontend, then download like a normal file. Extracting the hotspots was also fairly simple and improved on the process built all the way back in Week 1. Now, rather than manually running a script in Blender, the hotspots will be placed upstream as soon as the point clouds are generated. Another callback to Week 1 can be seen in the updates to exporting scenes into code that works with the tour-building software. Previously, multiple scripts were used to patch different parts of the experience into the file(s). Now, a single script can be run to prepare a directory and series of compatible files. This reduction in complexity and maintenance comes from many of the core changes and pipelines developed throughout this project. The earlier problems can be solved, the more cohesive the output becomes as it compounds throughout the workflow. Taking the content generated by the 360 to 3D process and integrating it with the spatial experience connects the tool to its first purpose and gives value to its output. Remade Spatial Experiences Supalai Remade Villa Remade Pattaya Remade The 3D models for Pattaya, Villa Korbhun, and Supalai Place have all been successfully integrated and tested in their spatial experiences. This integration proves the mesh data can integrate and display well with the tour software. More than anything, it feels incredible to see a tour as complex as the Supalai Place home displayed and navigable in 3D. Bonus: Mobile Proof of Concept Villa Mask View 360 to 3D reconstruction has been solved, though some edge cases still persist. Larger tours may need manual intervention to select and connect smaller groups. Some scenes may not pose well together and appear considerably misaligned, affecting the whole environment. Both of these could be fixed with visual feedback during capture. I decided to see if it would be possible to convert part of the current process to run on a mobile device, targeting the iPhone 14 Pro. The iPhone 14 Pro only makes around 4GB of RAM available to the user and includes a CPU, GPU, and ANU, all powerful though far less so in comparison to the 24GB 4090 GPU I have been testing on. I still persisted in trying, with the understanding that tasks may be limited and slower. I recalled recent developments in model-efficiency techniques, and my familiarity with quantization left me determined to try. After dozens of attempts, I had experienced and learned a lot about the iPhone’s architecture and processes. Most methods used in PyTorch machine-learning models translate directly into counterparts available for Apple’s ANU chip; however, many do not. These often require replacements with similar but slower code. This translation from a PyTorch model to Apple’s Core ML was fairly simple and straightforward. What made things more complex was attempting to run inference with this model. Its size began reasonable; however, attempting to run it on the ANU, a chip meant for AI models, led to RAM ballooning, loading far more than the model, and crashing the application. When attempting to use the model, it would load into memory and often load more than its size due to certain compilations of translations for the ANU. One alternative was to run it on the CPU, where the model would remain its size on disk at the cost of slower inference. This worked great, leading to my first successful point cloud from a 360 photo, all performed on an iPhone. This was a great achievement, though it was far too slow and far too little. At least two images need to be inferenced and compared in order to pose them. With more photos comes more RAM and more time. That pushed the budget close to the limit, and it was very slow. This is when I began to investigate quantization and palettization, as well as changing the precision value. These techniques help shrink the machine-learning model while preserving quality the vast majority of times. Applying combinations of these techniques to different degrees decreased the memory requirements just enough to leave room for some parts of the application, but not much. Remember, the 3D scene, user interface, connection to a 360 camera, live preview, and so much more all have to be managed in memory. And each photo means another point cloud, which can take up significant space in memory. A tour of hundreds of photos may take over 1GB. How could we squeeze more space from this machine-learning model? This is where I learned machine-learning models have different subsections that can be broken down: encoders, aggregators, decoders, and more. These parts can exist on their own, which means we do not need them all in memory all the time, only one in memory at a time. It also means I could squeeze more size out of each, which improves efficiency both in RAM and in CPU performance. Further reducing the models worked out great, with my maximum memory footprint reaching around 50% of the available RAM. This gives incredible space for point clouds and other features, even for incredibly large tours. With the models now so small, I did attempt to run them on the ANU. While certain parts could run, the memory required to run them quickly ballooned far beyond comfortable levels and would likely

Accomplishments Mesh Improvements Texture Improvements Bonus UI Improvements The 3D model is the ultimate visual output of the 360 to 3D reconstruction process. It’s quality is judged subjectively by the user, and quantitatively by the file’s size. Qualitatively, we were able to improve the visual look of the 3D model by editing the UVs and better distributing the image textures across the scene, with more proper fades between. Quantitatively, we were able to improve the process by reducing reconstruction time by over 85%. I also improved the reconstruction workflow, allowing for groups of photos to be posed individually and connected at joints. Mesh Improvements: Slow Fast After noticing a reconstruction process take close to 1 hour for a 52 image sequence I was determined to improve the performance. After investigation I noticed two likely improvements. One was a case where cpu and gpu were switching contexts quite often, and another was related to limiting the distance with which a camera should vote on points. I was able to re-work the process to keep contexts focused on either cpu or gpu and swap more efficiently. I was also able to limit the valid range for each camera. The depth estimation is only valid to a certain distance, so a camera from one end of the house shouldn’t be contributing votes to the room on the other side. This massively reduced the calculations for determining the final point cloud and mesh. Combined these changes were able to increase efficiency from 51 minutes down to just 8 for a 52 image sequence. Texture Improvements: Before UV After UV Before Texture After Texture Balancing visual fidelity with file size is a challenging task. The level of fidelity required is determined by the final application through which the user will experience the 3D model. The current application for which intends to be environmental navigation tool for spatial tours. While last week’s result was adequate for this, sharp transitions and blotchy textures littered the scene. Seing slices of walls or pillows and sharp changes in contrast is uncanny and distracting. I was able to setup a process which establishes an area between the seams for blending. This softens each transition and provides a gradient for color to change over. This limits the expression of sharp contrast changes or color inequalities across photos and provides an image much more appealing. Part of this process included improvements to the UV maps. UV maps are the unfolded faces of an object, cut into islands. Just like coloring origami on a flat sheet of paper before folding it. Before the UV map improvements there were an incredible number of very small islands. These little triangles littered the scene and often had textures slightly different than the area around them, standing out quite obviously. Updates were made to grow these islands. Now more triangles are connected to their neighboring islands, and the islands overall are much larger. This allows us to better spread out the textures across surfaces for more complete and consistent visuals. Bonus: UI Updates: Villa Mask View Supalai F1 Actions Menu Pattaya Park Whole Park Min Floor 3 Floor 2 The user interface was completely overhauled to provide a simpler interface and better conform to the new workflow. Images can now be uploaded in groups and automatically process between downsample and masking operations. Masks and RGB images are visible directly from the image’s drop down in the persistent content window. Masked groups can be posed via an action and immediately display in the 3D viewport. From here a camera’s height can be provided to scale the whole cloud. And groups can be linked together at a joint to allow for more piecewise alignments. Merged groups can be modeled and when complete, again they appear directly in the 3D viewport. With these changes I have streamelined the end-to-end process and reduced views to maintain from several to just 1. Summary Better visual fidelity and faster processing. Simpler workflow and less to maintain. These simple benefits are the result of persistent iteration and reflection on a process that continues to evolve and further funnels it closer to its final permutation. It’s effectiveness continues to increase, and I am grateful for the changes accomplished this week.