Representation of Multimodel Data – A Challenging Task

Dr Ben Lawrence, Dr Tamsin Robb, Braden Woodhouse, Faculty of Medical and Health Sciences, Dr Mike Davis, Assoc Professor Uwe Rieger, Yinan Liu, School of Architecture and Planning; Rose McColl, Sina Masound-Ansari, Nick Young,
Centre for eResearch.

Home

•

Project

•

Representation of Multimodel Data – A Challenging Task

The Tumour Evolution project (XRTEP) has received a bronze for the NZ Best Awards in the value of Design category. See the article here.

The project began with a patient’s decision to donate her inoperable cancer tissue for research. Over the years, medical monitoring has enabled scientists to gather a large amount of information on the growth of cancer as well as its distribution in the patient’s body. The documentation began with the discovery of a tumour in the lung. At the time of death, there were eventually 89 different tumours, all of which the research team sequenced. This extent, as well as the complexity of the information obtained, created new challenges for the presentation of the data.

The product of an inter-disciplinary collaboration

An interdisciplinary team from the arc/sec Lab at the School of Architecture and Planning, NETwork! at the Faculty of Medical and Health Sciences and the Centre for eResearch at the University of Auckland took up the challenge. As a result, an immersive arena has been created in close collaboration with medical professionals. At its center is an interactive, holographic model of the patient’s skeleton, organs and tumours. The associated data are arranged around the model in three concentric layers. In each case, the disciplinary specificity of the data increases with distance from the model. This allows for moments of intra- and inter-disciplinary focus at different points in space. Through the use of HoloLens, users of the application are immersed in this virtual augmented reality and interact with the model at the same time.

Usage currently still limited to education and training in the medical field

Currently, the immediate use of this research project lies in the education and training of medical professionals. However, KielSCN also sees promising potential in the use of XRTEP, or similar projects, in non-scientific target groups. This, however, requires the further development of the design features as well as the usability. In addition, a stronger guidance of the user during the exploration seems promising to expand the circle of users to a broader target group. This could be achieved, for example, by embedding it in a narrative or storytelling.

Immersive technology – human interaction focus

To realize the project, the Auckland team used the latest and most complex immersive technology: extended reality. It allows an extension of the real environment with virtual objects and links them to a new reality. Users can also interact with the digital world. They can grab organs from the skeleton to get a closer look at the tumours inside. By sliding a time scale, users can see how tumours grow, shrink or spread over time. Thus, immersive technologies not only offer the possibility of an extremely realistic representation of objects and data in virtual space, rather, they enable users to enter a virtual environment and become part of it. From the users’ perspective and experience, this is probably even more important than the highly realistic representation of content. This feeling of being fully absorbed by an environment is referred to as immersion. It can be very effective in encouraging engagement with scientific content by creating a sense of awe and fascination.

However, the basic requirements for exploiting the full potential of immersion and binding users to the content are fluid interaction with the medium and a high level of usability. In other words, a high quality of user experience. This is also what KielSCN* aims at. We want to understand how emotions and usability are related in the user experience. For this purpose, we investigate in-situ processes that reveal in which way and how successfully individuals use visualizations when exploring scientific information.

XRTEP facilitates access to cancer research and abstract data

Fundamentally, the XRTEP represents an exemplary implementation. It offers a novel and unique experience to engage with cancer research data. Not only do the developers generate attention, they also facilitate access to abstract genomic data and the contexts of an oppressive topic. To learn more about the XR Tumour Evolution Project detailed background information is available on the project website.