Steps to get your mantle environment set up for the first time.
In order for Mantle to organize your data and run pipelines within your AWS account, you will need to set up a few things.
Determine which S3 buckets you want to use for your Mantle environment. You will need to specify these buckets within the IAM policy you create for Mantle.
Mantle uploads all output files to a single S3 bucket. It is recommend to create a new bucket for Mantle to use. This bucket should be in the same region as the Batch environment you create. Please grant both read and write permissions to Mantle for this bucket (see below).
If you want to use a bucket that you already have, we will need to specifiy a prefix for Mantle to use. This will help keep Mantle’s files organized within your bucket.
Provide the bucket name and prefix to Mantle to set up your account.
For the mantle write bucket add the following CORS policy to allow Mantle to upload files to the bucket:
For any read buckets you want to use with Mantle, add the following CORS policy to allow Mantle to read files from the bucket:
This policy allows the IAM user to read from the S3 buckets you specify and write to the Mantle output bucket. It also allows the user to submit batch jobs. Nextflow requires batch actions to be performed on "Resource": [ "*" ]
. If your pipeline relies on Docker images stored in ECR, you will need to provide Mantle with access to the images. This can be done by adding the following snippet to your IAM policy. Futher instructions for how to use containers in your Nextflow pipeline can be found here.
Note: We are testing different configurations to reduce the number of permissions we need. It may be sufficient in the future to ensure your AWS Batch role has these permissions rather than giving them to the Mantle user — we will update the documentation.
Provide the access key and secret key to Mantle to set up your account.
To use EBS auto-scaling you will need to create a role granting appropriate access.
amazon-ebs-autoscale-policy-nextflow
Each job within the batch queue has the access scoped to an IAM role, to set up a role with the access required to the batch queue follow the steps below:
For more restrictive S3 Access: 8. Click on the role you just created. 9. Click “Add inline policy”. 10. Click “JSON” and use the following policy:
In order for Mantle to run Nextflow pipelines on AWS Batch, you will need to create a custom AMI with the following software installed:
We recommend only installing these two packages on the AMI to keep it lightweight and reduce the time it takes to launch an instance. All other dependencies should be intsalled within the docker container that runs the pipeline.
Bioinformatics pipelines can process a large amount of data because of this we recommend using “EBS autoscaling”. This monitors a mount point on an instance and dynamically increases the available storage based on predefined capacity thresholds. Setting this up involves installing a few lightweight dependencies and a simple daemon on the host instance. More informtion on ebs-autoscaling can be found here.
Here are the instructions for configuring an EC2 instance with EBS autoscaling:
The Different Volumes are described below:
This script installs the aws cli and ebs-autoscale on the instance.
The ebs-autoscale daemon will monitor the /var/lib/docker mount point and dynamically increase the available storage based on predefined capacity thresholds.
This installs the aws-cli to /miniconda/bin/aws
. (see instructions on how to incorporate this into your pipeline here)
Mantle runs nexflow pipelines on AWS batch. You will need to create a queue for Mantle to use.
optimal
option).Instance Configuration
-> Additional Configuration
, select the launch template you created.Steps to get your mantle environment set up for the first time.
In order for Mantle to organize your data and run pipelines within your AWS account, you will need to set up a few things.
Determine which S3 buckets you want to use for your Mantle environment. You will need to specify these buckets within the IAM policy you create for Mantle.
Mantle uploads all output files to a single S3 bucket. It is recommend to create a new bucket for Mantle to use. This bucket should be in the same region as the Batch environment you create. Please grant both read and write permissions to Mantle for this bucket (see below).
If you want to use a bucket that you already have, we will need to specifiy a prefix for Mantle to use. This will help keep Mantle’s files organized within your bucket.
Provide the bucket name and prefix to Mantle to set up your account.
For the mantle write bucket add the following CORS policy to allow Mantle to upload files to the bucket:
For any read buckets you want to use with Mantle, add the following CORS policy to allow Mantle to read files from the bucket:
This policy allows the IAM user to read from the S3 buckets you specify and write to the Mantle output bucket. It also allows the user to submit batch jobs. Nextflow requires batch actions to be performed on "Resource": [ "*" ]
. If your pipeline relies on Docker images stored in ECR, you will need to provide Mantle with access to the images. This can be done by adding the following snippet to your IAM policy. Futher instructions for how to use containers in your Nextflow pipeline can be found here.
Note: We are testing different configurations to reduce the number of permissions we need. It may be sufficient in the future to ensure your AWS Batch role has these permissions rather than giving them to the Mantle user — we will update the documentation.
Provide the access key and secret key to Mantle to set up your account.
To use EBS auto-scaling you will need to create a role granting appropriate access.
amazon-ebs-autoscale-policy-nextflow
Each job within the batch queue has the access scoped to an IAM role, to set up a role with the access required to the batch queue follow the steps below:
For more restrictive S3 Access: 8. Click on the role you just created. 9. Click “Add inline policy”. 10. Click “JSON” and use the following policy:
In order for Mantle to run Nextflow pipelines on AWS Batch, you will need to create a custom AMI with the following software installed:
We recommend only installing these two packages on the AMI to keep it lightweight and reduce the time it takes to launch an instance. All other dependencies should be intsalled within the docker container that runs the pipeline.
Bioinformatics pipelines can process a large amount of data because of this we recommend using “EBS autoscaling”. This monitors a mount point on an instance and dynamically increases the available storage based on predefined capacity thresholds. Setting this up involves installing a few lightweight dependencies and a simple daemon on the host instance. More informtion on ebs-autoscaling can be found here.
Here are the instructions for configuring an EC2 instance with EBS autoscaling:
The Different Volumes are described below:
This script installs the aws cli and ebs-autoscale on the instance.
The ebs-autoscale daemon will monitor the /var/lib/docker mount point and dynamically increase the available storage based on predefined capacity thresholds.
This installs the aws-cli to /miniconda/bin/aws
. (see instructions on how to incorporate this into your pipeline here)
Mantle runs nexflow pipelines on AWS batch. You will need to create a queue for Mantle to use.
optimal
option).Instance Configuration
-> Additional Configuration
, select the launch template you created.