MESHParade: Zubr’s collaborative 3D-scanning service allows anyone to capture a lifelike 3D model in augmented reality, and save it to the AR Sculpture Garden.
Augmented and Virtual Reality company Zubr.co set out to create what they call a ‘Mobile Edge Scan Hive’ (MESH), a novel deployment to FLAME’s edge platform which invites participants to become contributing camera nodes for photogrammetry capture. Pairing this with a unique augmented reality user experience, the resulting lifesize models can be viewed and explored in an intriguing ‘Parade’ of frozen 3D sculptures. Welcome to the MESH Parade!
The idea is that one person volunteers to be scanned, and must hold a pose while the other participants use the smartphone app to capture imagery of them; following instructions in augmented reality. The server continuously monitors the progress of the scan via the participants’ devices, and automatically decides when the scan is complete.
Zubr’s photogrammetry solution running on the edge receives and processes every image captured from each client device. A basic 3D model of the scan is churned out rapidly as a point cloud, and transmitted to all client devices, allowing participants to view the ‘sculpture’ as quickly as possible in AR. The engine then continues to process a higher-fidelity version of the 3D model, including textures – which is transferred as soon as it is complete, replacing the previous one. The resulting user experience allows the participants to see the volunteer frozen in time, locked in its’ position in the augmented reality layer.
Our main motivation for using the FLAME infrastructure for this service is the ability to synchronise devices with very low latency. This ensures that the pictures taken by the clients are all from the same instance in time, thus improving the quality of the 3D results. In addition, we make use of storage on the edge when each new 3D scan is added to the testbed’s cumulative ‘sculpture garden’ of AR 3D scans, ready to be retrieved by participants at any time. The models are not cached locally and must be downloaded a fresh for each session, meaning that the Sculpture Garden is always dynamic, and truly located at the Edge’s position – not merely in the ‘ether’ as a cloud service. We believe this gives our deployment strong creative validity as a social place-making experiment and has exciting promise for future developments.
Furthermore, high-speed data transfer unlocks a smoother user experience, as our solution depends heavily on sending and receiving multiple images and 3D files simultaneously, and this can become a slow and cumbersome experience on slower networks. Zubr conducted a trial of the finished MeshParade service in Millenium Square, Bristol, in September 2020. We would like to have conducted public trials, and gather feedback from varied participant groups, however, this was not possible in the wake of COVID-19. Nevertheless, the trial was a success, with the creation of many new 3D models on location, added to the sculpture garden. We hope to be able to build on MeshParade in the future – one example user experience we would love to see would allow collaborators to create and share 3D scans from different testbed locations around the world in realtime. We look forward to contributing towards the next generation of user-generated content experiences.