Home Machine Learning Saying Azure Managed Lustre on your HPC and AI workloads | Azure Weblog

Saying Azure Managed Lustre on your HPC and AI workloads | Azure Weblog

Saying Azure Managed Lustre on your HPC and AI workloads | Azure Weblog


In the present day, we’re excited to announce the final availability of Azure Managed Lustre, a managed excessive efficiency parallel file system for top efficiency computing (HPC) and AI workloads. Azure Managed Lustre delivers the time-tested Lustre file system as a primary get together managed service on Azure. Very long time customers of Lustre on-premises can now leverage the advantages of an entire HPC answer, together with compute and excessive efficiency storage, delivered on Azure. Customers profit from the excessive throughput and efficiency capabilities of Lustre as a seamless managed expertise and may now give attention to their enterprise imperatives.

Want for Azure Managed Lustre

Lustre, one of the vital widespread distributed parallel filesystems within the HPC world has lengthy been deployed on-premises serving scalable and excessive throughput storage wants of HPC workloads. As an open-source answer, it has loved a thriving ecosystem of customers and builders. With the latest explosive development seen in generative AI and the necessity for a excessive throughput storage that retains the costly GPU cores from ready on knowledge, Lustre has discovered a renewed development in its adoption. Prospects in industries leveraging excessive efficiency computing are in search of a seamless end-to-end infrastructure encompassing compute, storage, and networking that’s each quick to deploy and simple to make use of.

Azure Managed Lustre allows:

  • A managed PaaS expertise for worry-free deployment and consumption of Lustre file system.
  • Lowered TCO with pay-as-you-go pricing.
  • Throughput choices at completely different pricing ranges for various workloads.
  • Simple deployment of unbiased clusters with predictable efficiency, eliminating the noisy neighbor downside generally skilled in on-premises Lustre deployments.
  • Fast deployment in a couple of minutes, assuaging the necessity to plan months or years upfront and assembly compliance and knowledge residency necessities.
  • Area of selection for deployments throughout many Azure areas, supporting compliance and knowledge residency necessities.
  • Scalability to tons of of petabytes (PB), with the related sturdiness, availability and price advantages of tiering to Azure Blob storage.
  • Seamless integration with present Azure compute providers together with Azure HPC providers, Azure Machine Studying and Azure Kubernetes service.
  • Funding safety for patrons’ present automation and platform improvement when migrating from on-premises Lustre deployments.

In the present day, two persistent sturdy situations can be found for patrons to make use of. These are primarily based on stable state drives (SSDs) that are differentiated by their efficiency choice delivered for provisioned Tebibyte of capability.

  • AMLFS Commonplace – 125 MB/s
  • AMLFS Premium – 250 MB/s

Customers of Azure Managed Lustre can obtain the Lustre consumer packages from packages.microsoft.com for his or her desired Linux distribution and kernel model. Moreover, we could have assist for HPC pictures prebuilt with Lustre consumer packages for Ubuntu – 18.04, 20.04, 22.04, and Alma 8.7. 

Azure Managed Lustre is out there immediately from the Azure portal in a number of areas. To maintain updated on what areas you should use, please consult with our merchandise accessible by area web page. To study extra about pricing, please go to our pricing web page.

Prospects are already seeing outcomes with Azure Managed Lustre

Azure Managed Lustre serves various companies and organizations who’ve the necessity for top throughput storage entry inside power, life sciences, meteorology, automotive, healthcare, and public sector industries.

Azure Managed Lustre has you coated whether or not you’re operating instruments to simulate 3D product fashions to speed up time-to-market, serving hundreds of customers querying monetary data on knowledge analytics platforms, or constructing the following AI supercomputer on your generative AI answer. 

Petronas logo

When Petronas, a world chief in power, wanted to speed up their upstream power analysis with extremely complicated quantitative interpretation of seismic processing workloads that want massive knowledge units to be fed to the compute VMs with very excessive throughput, they selected Azure Managed Lustre.

“Azure Managed Lustre is an excellent file system that has been particularly crafted to satisfy the distinctive necessities of PETRONAS’ excessive efficiency computing (HPC) and AI workloads. It delivers distinctive efficiency, offering excessive storage bandwidth and seamlessly integrating with important Azure providers.”—Faizal B Zainol Alam, PETRONAS Digital Common Supervisor

meteomatics logo

Meteomatics, a frontrunner and disruptor in democratizing climate intelligence wanted a excessive efficiency file system for his or her climate modeling and simulations, they picked Azure Managed Lustre for his or her wants. Azure Managed Lustre’s integration with Azure CycleCloud allows Meteomatics to deploy their HPC clusters rapidly and programmatically permitting them to ship worth to their finish prospects in search of quick and constant entry to climate knowledge.

“At Meteomatics, we depend on excessive efficiency computing (HPC) to run complicated numerical climate prediction fashions, delivering correct forecasts to our prospects. By leveraging Lustre, a trusted file system within the HPC neighborhood, and the seamless deployment of LustreFS, we acquire the excessive variety of IOPS and throughput essential for dealing with large knowledge volumes. Azure Managed Lustre accelerates our climate forecast simulations with a excessive spatial and temporal decision, considerably lowering time-to-completion and enabling sooner supply of forecast knowledge to our valued prospects.“—Dr. Martin Fengler, CEO Meteomatics

Duke Clinical Research Institute logo

Duke Medical Analysis Institute was in search of a scalable answer for his or her excessive I/O must assist their analytics instruments serving medical analysis. Azure Managed Lustre addressed that want with a easy deployment mannequin permitting them to give attention to offering worth to their prospects rapidly. I see the easy simplicity and energy of AMLFS as the last word selection for high-speed shared storage, empowering your analytics instruments with unequalled simplicity and flexibility for each quick and long-term wants.“—Emilio Energy, Analysis Analytics Lead, SAS System Architect, Duke Medical Analysis Institute

Utilizing Azure Managed Lustre

Azure Managed Lustre is delivered in a hosted-on-behalf-of subscription, by means of a easy interface throughout the buyer digital community. Prospects don’t should be involved concerning the internals of deploying, managing, and working the Lustre file system together with metadata servers/targets (MDS/MDT), administration servers/targets (MGS/MGT) and object storage servers/targets (OSS/OSTs).

Diagram showing the Azure Managed Lustre File System service topology.

The Azure portal affords a easy interface to configure and deploy Azure Managed Lustre, the place prospects present some inputs like useful resource group id, subnet ID, cluster measurement, and desired efficiency choices. Customers have the flexibility to watch and observe their cluster efficiency with throughput, latency, and IOPS metrics.

Screen shot of Azure Managed Lustre preview registration page.
Screen shot of Azure Amanged Lustre in a demo state, showing performance mentrics.

Hierarchical storage administration performance in Azure Managed Lustre allows customers to seamlessly switch knowledge between Azure Managed Lustre and Azure Blob. This empowers customers to determine knowledge archival, retention, and safety methods. When coping with massive datasets, customers can import solely the info that’s at present energetic and essential for his or her knowledge processing duties into Azure Managed Lustre clusters, whereas archiving and retaining the remaining knowledge in Azure Blob. This method successfully minimizes runtime prices. Moreover, by using knowledge tiering to Azure Blob, customers can reap the benefits of its world presence and availability to create Azure Managed Lustre clusters in a number of areas as wanted. The mixing with Azure Blob additionally facilitates the accessibility of information by means of a number of protocols, similar to community file programs (NFS), hadoop distributed file system (HDFS), and REST.

We’re additionally very excited to announce the final availability of the CSI driver for Azure Managed Lustre. Now, containerized purposes operating on Azure Kubernetes Service can simply devour Azure Managed Lustre.

Subsequent steps

Be taught extra about tips on how to use Azure Managed Lustre and its numerous supported options from our documentation.

Be taught extra about HPC and AI options



Please enter your comment!
Please enter your name here