Project Team

Development Team

Supervisors

Downloads

Literature Synthesis

Project Proposal

  • AstroVis Proposal - An Investigation Into Accessing and Displaying Large Astronomy Data Files

Project Report

Project Poster

Project Source Code

Data cubes are the predominant format for storing radio astronomy data, the resolution and coverage of which are growing rapidly as technology advances. Radio astronomical observations are typically three dimensional, so the raw volume of data captured is increasing equally quickly, if not more so. Despite this, applications commonly used for viewing data cubes have little support for 3D displays and do not implement a scalable approach to accessing files which exceed the capacity of main memory.

Project Overview

The project is split into three different sections. A file access component for reading and writing to data cubes, an image processing component for taking large data cubes and reducing them to a form that can be fit into system memory and a user interface for visualising the reduced data cube in 3D.

These three components combined create the following process.

Program Overview

File Access Component

When files are too big to fit into system memory, virtual memory can be used instead but is very slow. The File Access Component attempts to do this without the severe performance hit.

This is done by sorting the read requests into an order with less seek time. Seeking takes up the most time when performing file reading off the hard drive, so reducing seek time reduces overall read time.

File Access Component Sorted vs Unsorted Diagram

The FAC also uses other methods to reduce the impact of reading time. By allowing the rest of the project to run in parallel with the file reading operations, the cost of processing the data is entirely hidden by the read times. This is known as latency hiding. This technique has also been used to hide the cost of sorting the read requests. By allowing sorting to occur in parallel with reading, the cost of sorting could also be hidden.

In the evaluations it was shown that the parameters of the output data cube affected read times differently. Read times increased as the width increased, while read time decreased as the height and depth increased. However in all the tests performed, sorting requests always produced faster read times.

Image Processing Component

The objective of the IPC is to allow the system to efficiently read in large data cubes without the computer needing an abundance of RAM in a reasonable amount of time. It will also need to provide the user with feedback during this process. A down-sampled representation of the data cube shall be the intermediate and final outcome of the system.

This module focuses on ways of how to break up the data cube into smaller pieces that can be processed locally and gives a representation of the whole file. It does this by segmenting the file into smaller data cubes that takes data from different parts spread over the file. A down-sampling procedure processes the segments into a final representation of the full resolution data cube be made so that the user can view intermediate results. The system is designed in such a way that it is scalable with large data cubes, which is a problem faced by most astronomy software packages currently. Three iteratively implemented solutions were made to find the best trade off between overall time to complete, time to give user feedback, memory usage and which solution scaled the best with file size.

Image Processing Component Diagram

The sequential method, which was the last solution implemented, yielded the best results in terms of overall time to complete, giving the user fast feedback and memory efficiency. It was the only solution that would scale well with large data cube sizes. Stress tests showed that the system could completely down-sample a 120GB data cube in roughly 25 minutes without using more than 512MB of RAM.

Data Visualisation

Examining data which has exceedingly high resolution becomes difficult when working with limited screen and memory space. This visualisation implementation attempts to overcome that problem by using an approach of Overview + Detail. This approach is ideal, as extensive use of an overview visualisation reduces the dimensions of the data being worked with, as well as the amount of memory required to store it.

The overview visualisation implemented volume rendering using textured slices to display a data cube in 3D. This allows visual access to a far greater portion of the file, allowing for easier and faster navigation and analysis. This 3D display is paired with a 2D detail display which contains a small subsection of the original file in full resolution. Although the data is not reduced in size in any way, a relatively small selection keeps it manageable.

Screenshot of User Interace

The volume renderer is limited in the detail it can display at one time, so the interface uses an approach of user driven selection and refinement. This allows a user to target a specific area of the displayed data cube, and effectively magnify it, increasing its detail.

Stages of refinement for User Interface