When Marvel met Medtech.

If the last decade ended with cinematic superhero worship, then this one has begun with a reminder of the real heroes in our world, the doctors, carers and supermarket staff who are helping us pull through the 2020 crisis.

It’s rare though for these two worlds to meet outside of celebrities and fundraisers thanking medical heroes dressed as Marvel’s mightiest, but there is one instance that comes from the most unlikely of places: Cast your mind back to a world before populism and pandemic, 2015 in fact, when Avengers: Age of Ultron hit cinema screens, and we were introduced to medical scientist Dr. Cho, played by Claudia Kim, and her cutting-edge tech.

With that tech she saves Hawkeye’s life, before being forced to create a shell-body for the film’s eponymous villain, one which ultimately becomes hero The Vision. All the screens we see her use – along with the film’s other UI interfaces – came designed by London’s Territory Studio, who we last featured for their sterling work on Blade Runner 2049.

Fast forward to 2020, and its turns out the studio’s Avengers work caught the eye of a medtech brand looking to utilise AR and MR in the surgery room.

“Our work on Ultron predated holograms as the common narrative device that we see in many films today,” says David Sheldon-Hicks, co-founder and executive creative director at Territory, “but we were really keen to elevate the content of flat screens using 3D elements. So, our UI for Stark’s, Banner’s and Dr. Cho’s Labs featured richly visualised 3D elements and layered content.

“Together with clean and minimal UI, the warm and human colour palettes (gold and red for Stark, green for Banner and purple for Cho) gave the content a crispness and depth that made it feel far in advance of the UI that we’re all still stuck with in the here and now.

“Both the founders of (medtech startup) Medivis were fans of our work in Ultron, and what they hoped for was this focus on human design with clarity and depth to the detail to differentiate their products.

“Of course, the technical constraints of the various systems and AR/MR platforms necessitated a lot of compromise in the final look and feel, but the similarity (from the film) was that we sought to bring the visual impact of strong design to Medivis’s platform.”

As David tells us, Medivis was founded with a Vision to give doctors ‘superpowers’ to see the invisible structures that surgeons would otherwise be unable to see in real time while performing surgery.

“The aim – as shown in this video – was to increase diagnostic accuracy and facilitate pre-operative planning and intraoperative navigation,” he elaborates, “providing greater precision for the doctor and greater safety for the patient. But that is dependent on having a platform that is easy to learn and easy to use in a busy clinical environment where time critical procedures demand a surgeon’s full attention.”

“Territory were asked us to create a UI that was clean, uncluttered and easy to use, yet still brought strong design values to the platform. When we began to design the UI for the desktop and headsets, we worked closely with Medivis founders, radiologist Dr. Chris Morley, and neurosurgeon Dr. Osamah Choudhry, to understand the way that clinicians use medical imaging and data in a clinical environment.

“By designing a UI and iconography system that is intuitive and easy to use, clinicians are today able to quickly and easily get to grips with the platform and focus on what the product has been designed for – achieve greater diagnostic accuracy and improve (the planning and navigation needed.)”

David has been in the know on extended reality in medtech as far back as 2013, when Territory were researching for Age of Ultron.

“We did a huge amount of research into emerging bio and medtech applications, and then extrapolated from that to come up with something that felt right for the hyper-sophisticated technology at the Avengers’ disposal. 

“Google Glass was involved in R&D back then, and in 2015 HoloLens really took off.  Since then the field has really gained momentum. I feel that Medivis and their relationship with Microsoft HoloLens has been a driver in the field.”

David and Territory’s attention to detail saw more extensive research based on understanding the technical constraints and the clinical expectations of an AR/MR application, as facilitated by Medivis.

“Considerations that our team tackled included user interface requirements and conventions in a clinical setting, UI design for desktop and how to translate that into 3D space, dexterity requirements in gesture commands and manipulation and requirements for clarity and legibility from multiple viewing angles in a fast-moving clinical environment.”

Augmented Assembling

The studio worked closely with Medivis to map and design the visual language and iconography of operational settings, menu systems, navigation and displays that underpin usability. 

“We started with the UI for the desktop touchscreen application,” David tells me, “and then designed icons for the HoloLens headset display that ensured consistency of visual language across the platform.” 

Medivis partnered with several big university hospitals that gave their doctors and surgeons access to the prototype, and their experience gave Territory invaluable feedback on the UI so that the studio was able to adapt and refine it over the course of the project.

“The result is a user experience that is intuitive to the contextual expectations of clinicians, enabling the technology to facilitate rather than complicate diagnosis, surgical planning and procedure.”

Rewatching Ultron in the newfound time enabled by lockdown, I’m inspired to ask David how it feels for a studio well-known for its visual effects to suddenly be creating for the ‘real world.’ I’m also intrigued in how different the Marvel and Medivis projects felt, and how experience on the former led into the latter.

“We’ve been working with real world brands in different ways for some time now, from concepting future-facing products, designing UI for Huami’s Amazfit smartwatch to working with US automakers, but this is the first medtech project that we’ve been involved with. It’s an amazing feeling to know that our work will help make real people’s lives easier or better in some way.”

On the second point, David notes the Medivis brief was “necessarily different”, but the constraints were “useful in clarifying the framework within which we could innovate.”

“In this sense, our approach shares similarities with our work on The Martian, a science fiction film in which our screens (below) were grounded in science fact.

“The data feeds and systems that we worked with were real, so our task was to create a cohesive visual language that brought consistency and clarity to the UI, with cinematic flourishes. 

“Medivis didn’t need flourishes, but it did require a clear, well-designed visual language and UI that ensures consistency across the product family, so that a med student trained on AnatomyX™ (Medivis’ educational tool that draws on an archive of real medical imaging to train med students) can easily migrate over to SurgicalAR™ (the holographic medical imaging platform in use in hospitals and operating theatres) in clinical practice. 

“I think what impressed me about the Medivis project is that real world medical technology like SurgicalAR™ is really catching up to science fiction, and will revolutionise clinical practice. The caveat is that the technology is still in development, and until AR/MR rigs are sophisticated enough to adapt to humans, rather than requiring humans to adapt the them, we will be reliant on somewhat awkward headsets and behaviours that risk getting in the way of true transformation.”

David finishes saying that Medivis represented a challenge that resonated both personally and professionally with him.

“It was a rare opportunity to apply Territory’s expertise to a disruptive product that has the potential to make a real difference to the lives of doctors and patients.”

Related: How Territory Studio created ‘tangible’ screen graphics for Blade Runner 2049

https://www.digitalartsonline.co.uk/features/hacking-maker/how-avengers-vfx-saw-real-heroes-save-lives-with-territory-xr/