Data-intensive research in domains such as radio astronomy, particle physics and the omics has the potential to generate an incredibly large volume of data. Some of the largest datasets on the planet are in the realm of 100’s of Petabytes!

Moving this data for analysis on high performance compute, or simply to use it for cross-institution collaborative research, requires high capacity frictionless networks that can perform over very long distances. Large single flows (dubbed “elephant flows”) have the potential to consume all available bandwidth, negatively impacting the performance of the network for other users.

IT managers are faced with the challenge of how to ensure “business as usual” functions of the many in academic and scientific institutions need to be as reliably served as the intensive demands of the few:

Much has already been done to date to enhance Australia’s National Research and Education Network, AARNet, and make this frictionless networking a reality.

• Our national backbone now operates at 100Gbps, with multiple 100Gbps services in place across the Pacific Ocean to the USA.

• 100Gbps connections architected for Science at the campus edge (called Science DMZ) have been deployed at various sites to improve data transfer while preserving site network security.

• Tools for data sharing, such as CloudStor, are contributing significantly to the daily workflows of the wider research community.

• A research virtual private network with dedicated 100 Gbps capacity and international reach is available to a small group of institutions and organisations involved in data-intensive research resulting in large, sustained data transfers between research partners, such as scientific instruments, storage and compute resources.

AARNet engineers have been able to demonstrate that a little effort put into border architecture, data transfer tools and data handling workflows can result in extremely large data flows between research infrastructure services and instruments, without impacting on a broader range of users of the entire national network.

We are continuing to build new network infrastructure and new platforms and technologies to address this challenge and to support extreme, unique and evolving research institution requirements.

Brett’s role as AARNet Senior Research Analyst is to ensure that the research community learns to expect more from the significant long-term investment in the network and services provided by AARNet, and to understand that research networks worldwide are architected to serve even the most data-intensive sciences and collaborations.

Contact us to find out how we can help your institution manage big data flows.