Automating data collection and generation for the Rongowai mission

Dr Mike laverick, Senior Solutions Specialist, Centre for eResearch

CeR helps support a wide range of digital research services at the University of Auckland including software upskilling, research data storage, data transfer mechanisms, and research compute. As a result, we have significant expertise in designing and implementing research workflows at all scales: from simple analysis scripts to high-throughput automatic workflows involving several University services. We highlight two examples of research workflows we have helped to develop.

Rongowai

A world-first remote-sensing instrument hosted on board a commercial aircraft in collaboration with NASA, the New Zealand Space Agency, and Air New Zealand to support climate-change missions as relevant to New Zealand’s interests in order to build national strategic capacity in earth observing payloads and missions.

The collaboration between NASA and Air New Zealand included the design of aircraft-integrable detector and used cutting-edge simulation to generate synthetic GNSS-R data and the aircraft-mounted GNSS-R flight data.

Developing scientific pipelines

The Centre for eResearch helped develop and hosts the communication infrastructure and scientific processing pipelines. The automation was designed to stream climate data from the aircraft several times a day every day for several years.

Both the communication and science pipelines run on the Nectar Research Cloud across several virtual machines (see Diagram 1). The communication pipelines run 24/7 and were designed to be as automated and robust as possible in case of any unforeseen circumstances. CeR also helped optimise the science pipelines to handle the constant influx of data and transfer data products into public NASA repositories. The pipelines typically run for months on end without need for human intervention or maintenance and have an email alert system to catch any errors or outages as soon as possible.