What is Amazon FSx for Lustre?
High-performance, scalable, and cost-effective storage computational tasks are provided by the fully managed solution Amazon FSx for Lustre. Fully controlled shared storage is built on top of the world's most popular high-performance file system.
Benefits of Amazon FSx for Lustre
Reduce computation workloads
Computing workloads can be accelerated by shared storage with sub-millisecond latencies, hundreds of gigabytes/second throughput, and millions of IOPS. Deploy a fully managed Lustre file system in a matter of minutes.
To access and manage data sets, use Amazon S3
You may access and manage Amazon S3 data from a high-performance file system by connecting your file systems to S3 buckets.
Make storage as affordable and useful as possible for your workload
To balance cost and performance, use a range of deployment options, including storage type, performance tier, and replication level.
How it works
Amazon FSx for Lustre provides fully controlled shared storage with the performance and scalability of the popular Lustre file system.
Use cases
Increase machine learning's (ML) speed
Optimized throughput to your computing resources and easy access to training data stored in Amazon S3 can reduce training times.
It is necessary to enable high performance computing (HPC)
Even the most demanding HPC applications may be powered by fast, highly scalable storage that is directly connected with AWS compute and orchestration services.
Launch big data analytics
Support thousands of computer instances running complex analytical workloads and petabytes of data.
Increase the media workload's agility
You may adjust to ever-tinier timelines for visual effects (VFX), rendering, and transcoding when your computer's storage expands with it.
Overview of Features for Amazon Lustre FSx
Controlled, affordable, high-performance, scalable computation storage is offered by Amazon FSx for Lustre. FSx for Lustre, which is based on Lustre, the most widely used high-performance file system in the world, offers shared storage with millions of IOPS, terabytes per second throughput, and sub-ms latency. FSx for Lustre file systems can read and execute data at the same time when connected to Amazon Simple Storage Service (S3) buckets.
Boost productivity with the workload
Summary
Terabytes per second and millions of IOPS can be handled by AWS FSx for Lustre file systems. FSx for Lustre manages thousands of compute instances' parallel access to files and directories. FSx for Lustre ensures low file operation latencies.
The most popular file system with great performance
Because it processes the world's growing data collections effectively and affordably, the Lustre open source file system is the most widely used file system for the 500 fastest computers worldwide. For genome sequencing, video transcoding, machine learning, and fraud detection, it has shown itself in the energy, life sciences, media production, and financial services industries.
Use it for any workload involving computation
Summary
FSx for Lustre is compatible with well-known Linux-based AMIs, including Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux.
Easy import/export Details about Amazon S3
For data processing activities, native S3 data access is made possible by Amazon FSx for Lustre.
With just a few clicks, you can connect one or more S3 buckets to a file system in Amazon FSx. FSx for Lustre allows you to submit results to S3 and transparently shows S3 objects as files once your S3 bucket is connected to your file system. Your connected file system is immediately updated when items are added, changed, or withdrawn from your S3 bucket. Your S3 bucket is automatically updated by FSx for Lustre whenever files are uploaded, changed, or deleted. FSx for Lustre uses parallel data-transfer algorithms to export data back to S3 in a fast manner.
Make easy use of computing services
Both on-premises computers and Amazon EC2 instances can run AWS FSx for Lustre. Once mounted, you can access the files and directories in your file system just like you would a local file system. FSx for Lustre file systems are accessible to Amazon EKS containers.
Expand the number of instructor roles for Amazon SageMaker
Amazon FSx for Lustre input data is supported by Amazon Sagemaker. By bypassing the initial S3 download phase and minimizing TCO by avoiding repeated downloads of similar items (saving S3 request costs) for iterative jobs on the same data set, Amazon SageMaker and Amazon FSx for Lustre speed up machine learning training jobs.
Deployment is made easier by compute management services
Amazon FSx for Lustre uses EC2 Launch Templates to connect with AWS Batch. ML, HPC, and other asynchronous workloads are supported by our cloud-native batch scheduler. Using the current FSx for Lustre file systems, AWS Batch starts instances and executes jobs while dynamically sizing instances to meet job resource needs.
AWS ParallelCluster is compatible with Lustre FSx. Use the open-source cluster management tool AWS ParallelCluster to deploy and administer HPC clusters. It can leverage pre-existing file systems or automatically generate Lustre FSx when creating a cluster.
Quick access to data
First-byte latency for file data access is sub-millisecond on SSDs and single-digit millisecond on HDDs.
Regardless of deployment style, storage type, or throughput performance, all Amazon FSx for Lustre file systems are supported by metadata servers with low-latency SSD storage. With sub-millisecond latency, the SSD-based metadata server provides metadata operations, which comprise the majority of file system operations.
Conserve funds
Cut down on paperwork and adjust performance and capacity as necessary.
With a few clicks, you can create and scale a high-performance Lustre file system using the Amazon FSx UI, CLI, or API. Amazon FSx file systems simplify time-consuming administrative tasks like maintaining storage volumes and file servers, updating hardware, configuring software, running out of space, and fine-tuning performance.
Different deployments
Amazon FSx for Lustre offers scratch and persistent file systems for both short-term and long-term data processing. Scratch files work well for processing and storing data temporarily. Data is not replicated or saved by a failing file server. Persistent file systems are ideal for workloads and long-term storage. A persistent file system takes the role of downed servers and duplicates data.
Amazon FSx may take incremental backups of persistent file systems automatically for further data protection and business and regulatory compliance. The durability of Amazon S3 backups is 99.999999999%.
Numerous storage options
Amazon FSx for Lustre provides SSD and HDD storage options to maximize both cost and performance for your workload. Small, random file operations and low-latency, IOPS-intensive applications can be handled by SSD storage. Large, sequential file operations and workloads requiring high throughput can be handled by HDD storage.
In an HDD-based file system, provision an SSD cache to give frequently accessed files sub-millisecond latencies and improved IOPS.
To avoid wasting capacity, storage quotas can track and restrict storage usage at the user and group levels on file systems. File system administrators who support different users, groups, or projects are subject to storage quotas.
Data compression reduces storage expenses
Data compression can minimize storage and file system backups. The LZ4 technique, which optimizes compression without compromising file system speed, is used by the data compression function. Before writing and reading freshly created files to disk, FSx for Lustre uses data compression to compress and uncompress them.
Get rid of outdated files
To optimize storage capacity, release inactive data after exporting files to Amazon S3. When a file is released, its metadata is kept on S3 and its data is deleted from the file system. When you access a released file, it loads transparently and automatically from your S3 bucket onto your file system.
Assure compliance and security
Summary
In certain locations, Amazon FSx for Lustre file systems are protected both in transport and at rest.
AWS has the longest-running cloud compliance program to assist customers in managing their responsibilities. Amazon FSx's security satisfies industry and international standards. Along with HIPAA, it complies with SOC 1, 2, and 3 as well as PCI DSS, ISO 9001, 27001, 27017, and 27018. For resources, go to our compliance website. Go to the Services in Scope by Compliance Program page to view all services and certifications.
Separate networks
You can isolate your Amazon FSx file system within your virtual network by using Amazon VPC endpoints. Set up network access and security group rules for Amazon FSx file systems.
Resource-level authorizations
AWS IAM is integrated with Amazon FSx. This connection allows you to control the creation and deletion of file systems by AWS IAM users and groups. IAM user and group actions can be restricted by tagging Amazon FSx resources.
One-stop backup and AWS Backup compliance
Fully controlled, policy-based backup and recovery for Amazon FSx file systems is made possible by integration with AWS Backup. Customer data is safeguarded and AWS service compliance is guaranteed for business continuity through integration with AWS Backup.
Compliance with regional and account backups
Business continuity, disaster recovery, and compliance requirements can be met and data protection can be enhanced by copying Amazon FSx file system backups across AWS Regions, accounts, or both.
0 Comments