
Representation of Multimodel Data – A Challenging Task
Dr Ben Lawrence, Dr Tamsin Robb, Braden Woodhouse, Faculty of Medical and Health Sciences, Dr Mike Davis, Assoc Professor Uwe Rieger, Yinan Liu, School of Architecture and Planning; Rose McColl, Sina Masound-Ansari, Nick Young,
Centre for eResearch.

The Tumour Evolution project (XRTEP) has received a bronze for the NZ Best Awards in the value of Design category. See the article here.
The product of an inter-disciplinary collaboration
An interdisciplinary team from the arc/sec Lab at the School of Architecture and Planning, NETwork! at the Faculty of Medical and Health Sciences and the Centre for eResearch at the University of Auckland took up the challenge. As a result, an immersive arena has been created in close collaboration with medical professionals. At its center is an interactive, holographic model of the patient’s skeleton, organs and tumours. The associated data are arranged around the model in three concentric layers. In each case, the disciplinary specificity of the data increases with distance from the model. This allows for moments of intra- and inter-disciplinary focus at different points in space. Through the use of HoloLens, users of the application are immersed in this virtual augmented reality and interact with the model at the same time.
Usage currently still limited to education and training in the medical field
Currently, the immediate use of this research project lies in the education and training of medical professionals. However, KielSCN also sees promising potential in the use of XRTEP, or similar projects, in non-scientific target groups. This, however, requires the further development of the design features as well as the usability. In addition, a stronger guidance of the user during the exploration seems promising to expand the circle of users to a broader target group. This could be achieved, for example, by embedding it in a narrative or storytelling.

Immersive technology – human interaction focus
To realize the project, the Auckland team used the latest and most complex immersive technology: extended reality. It allows an extension of the real environment with virtual objects and links them to a new reality. Users can also interact with the digital world. They can grab organs from the skeleton to get a closer look at the tumours inside. By sliding a time scale, users can see how tumours grow, shrink or spread over time. Thus, immersive technologies not only offer the possibility of an extremely realistic representation of objects and data in virtual space, rather, they enable users to enter a virtual environment and become part of it. From the users’ perspective and experience, this is probably even more important than the highly realistic representation of content. This feeling of being fully absorbed by an environment is referred to as immersion. It can be very effective in encouraging engagement with scientific content by creating a sense of awe and fascination.
However, the basic requirements for exploiting the full potential of immersion and binding users to the content are fluid interaction with the medium and a high level of usability. In other words, a high quality of user experience. This is also what KielSCN* aims at. We want to understand how emotions and usability are related in the user experience. For this purpose, we investigate in-situ processes that reveal in which way and how successfully individuals use visualizations when exploring scientific information.
XRTEP facilitates access to cancer research and abstract data
Fundamentally, the XRTEP represents an exemplary implementation. It offers a novel and unique experience to engage with cancer research data. Not only do the developers generate attention, they also facilitate access to abstract genomic data and the contexts of an oppressive topic. To learn more about the XR Tumour Evolution Project detailed background information is available on the project website.

All images in this case study are credited to the team of Mike Davis, Uwe Rieger, Yinan Liu, School of Archotecture & Planning.
The article was featured in the Kiel Science Communication Network (KielSCN).
See more case study projects

Our Voices: using innovative techniques to collect, analyse and amplify the lived experiences of young people in Aotearoa

Painting the brain: multiplexed tissue labelling of human brain tissue to facilitate discoveries in neuroanatomy

Detecting anomalous matches in professional sports: a novel approach using advanced anomaly detection techniques

Benefits of linking routine medical records to the GUiNZ longitudinal birth cohort: Childhood injury predictors

Using a virtual machine-based machine learning algorithm to obtain comprehensive behavioural information in an in vivo Alzheimer’s disease model

Mapping livability: the “15-minute city” concept for car-dependent districts in Auckland, New Zealand

Travelling Heads – Measuring Reproducibility and Repeatability of Magnetic Resonance Imaging in Dementia

Novel Subject-Specific Method of Visualising Group Differences from Multiple DTI Metrics without Averaging

Re-assess urban spaces under COVID-19 impact: sensing Auckland social ‘hotspots’ with mobile location data

Aotearoa New Zealand’s changing coastline – Resilience to Nature’s Challenges (National Science Challenge)

Proteins under a computational microscope: designing in-silico strategies to understand and develop molecular functionalities in Life Sciences and Engineering

Coastal image classification and nalysis based on convolutional neural betworks and pattern recognition

Determinants of translation efficiency in the evolutionarily-divergent protist Trichomonas vaginalis

Measuring impact of entrepreneurship activities on students’ mindset, capabilities and entrepreneurial intentions

Using Zebra Finch data and deep learning classification to identify individual bird calls from audio recordings

Automated measurement of intracranial cerebrospinal fluid volume and outcome after endovascular thrombectomy for ischemic stroke

Using simple models to explore complex dynamics: A case study of macomona liliana (wedge-shell) and nutrient variations

Fully coupled thermo-hydro-mechanical modelling of permeability enhancement by the finite element method

Modelling dual reflux pressure swing adsorption (DR-PSA) units for gas separation in natural gas processing

Molecular phylogenetics uses genetic data to reconstruct the evolutionary history of individuals, populations or species

Wandering around the molecular landscape: embracing virtual reality as a research showcasing outreach and teaching tool
