Difference between revisions of "Docking in AWS With DOCK 3.8"

From DISI
Jump to navigation Jump to search
(11 intermediate revisions by the same user not shown)
Line 2: Line 2:
  
 
btingle/dockaws:testing
 
btingle/dockaws:testing
 +
 +
For information on how to set up spot fleet permissions, see here: https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html#spot-fleet-roles-cli
  
 
=== Environment Variable Input ===
 
=== Environment Variable Input ===
Line 15: Line 17:
 
If S3_INPUT_LOCATION is a directory:
 
If S3_INPUT_LOCATION is a directory:
  
Input will be taken from the AWS_BATCH_JOB_ARRAY_INDEX'th file listed under the directory
+
Input will be taken from the AWS_BATCH_JOB_ARRAY_INDEX'th file (0-indexed, because that's how aws happens to do it) listed under the directory
  
 
  <nowiki>
 
  <nowiki>
         AWS_BATCH_JOB_ARRAY_INDEX=3
+
         AWS_BATCH_JOB_ARRAY_INDEX=2
 
         S3_INPUT_LOCATION=s3://mytestbucket/input
 
         S3_INPUT_LOCATION=s3://mytestbucket/input
  
Line 31: Line 33:
 
If S3_INPUT_LOCATION is a file:
 
If S3_INPUT_LOCATION is a file:
  
Input will be downloaded from the AWS_BATCH_JOB_ARRAY_INDEX'th line in the file
+
Input will be downloaded from the AWS_BATCH_JOB_ARRAY_INDEX'th (again, 0-indexed) line in the file
  
 
  <nowiki>
 
  <nowiki>
         AWS_BATCH_JOB_ARRAY_INDEX=2
+
         AWS_BATCH_JOB_ARRAY_INDEX=1
 
         S3_INPUT_LOCATION=s3://mytestbucket/input/inputlist_batch1.txt
 
         S3_INPUT_LOCATION=s3://mytestbucket/input/inputlist_batch1.txt
 
         ### inputlist_batch1.txt
 
         ### inputlist_batch1.txt
Line 68: Line 70:
  
 
description: The array index of the current job. Automatically supplied when submitting via aws batch.
 
description: The array index of the current job. Automatically supplied when submitting via aws batch.
 +
 +
==== NO_ACCESS_KEY ====
 +
 +
Disables fetching of access key from aws secretsmanager. See "Adding S3 Permissions to your instance role" section below.
  
 
=== Optional: AWS credentials ===
 
=== Optional: AWS credentials ===
Line 79: Line 85:
  
 
If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.
 
If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.
 +
 +
In the current configuration, any access keys supplied via environmental parameters is used to fetch access keys from aws secretsmanager, which will be used to fetch/put data in s3. If you do not have your aws access key & secret key in aws secretsmanager, (under the names "aws_access_key" and "aws_secret_key" respectively) you will also want to export a non-null value for NO_ACCESS_KEY.
  
 
=== Restartability & Spot Instances ===
 
=== Restartability & Spot Instances ===
Line 104: Line 112:
  
 
#!/bin/bash
 
#!/bin/bash
 +
yum install -y aws-cli # make sure the aws cli is installed, otherwise this next step won't work
 
docker_email=ben@tingle.org
 
docker_email=ben@tingle.org
 
docker_user=btingle
 
docker_user=btingle
 
export AWS_DEFAULT_REGION=us-west-1
 
export AWS_DEFAULT_REGION=us-west-1
docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass | grep "SecretString" | cut -d':' -f2 | sed 's/\"//g' | sed 's/,//g' | tail -c +2)
+
docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass | grep "SecretString" | cut -d':' -f2 | sed 's/\"//g' | sed 's/,//g' | tail -c +2 | sed 's/ *$//g')
 +
# aws secretsmanager returns a json object, this convoluted pipe exists to format it such that only the plain text password is retrieved
  
 
echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config
 
echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config
echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"auth\":\"${docker_auth}\",\"email\":\"${docker_email}\",\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config
+
echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"password\":\"${docker_auth}\"\,\"email\":\"${docker_email}\"\,\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config
  
 
--==DOCKERHUBINIT==--
 
--==DOCKERHUBINIT==--
 
</nowiki>
 
</nowiki>
  
In the above example, we use aws secretsmanager to fetch our dockerhub password. This is accomplished by adding the AWSSecretsManager:GetSecret permission (for our dockerhub password secret arn) to our ecsInstance role. Alternatively, you can hard-code the authentication details into the launch template.
+
In the above example, we use aws secretsmanager to fetch our dockerhub password. This is accomplished by adding the AWSSecretsManager:GetSecret permission for our dockerhub password secret arn to our ecsInstance role. Alternatively, you can hard-code the authentication details into the launch template.
  
For more information on authentication methods, try this page: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth-container-instances.html
+
For more information on authentication methods, try the following links:  
 +
 
 +
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth-container-instances.html
 +
 
 +
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html
 +
 
 +
https://aws.amazon.com/blogs/containers/authenticating-with-docker-hub-for-aws-container-services/
 +
 
 +
=== Adding S3 Permissions to your instance role ===
 +
 
 +
Currently, the DOCK 3.8 image is configured to fetch an aws secret key and access key from AWS secrets manager, then use those as credentials to access s3. Here's what that looks like in the script:
 +
 
 +
<nowiki>
 +
if [ -z $NO_ACCESS_KEY ]; then
 +
        AWS_SECRET_ACCESS_KEY=$(retrieve_aws_secret aws_secret_key)
 +
        AWS_ACCESS_KEY=$(retrieve_aws_secret aws_access_key)
 +
 
 +
        aws configure set aws_access_key_id $AWS_ACCESS_KEY
 +
        aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
 +
        aws configure set default.region $AWS_DEFAULT_REGION
 +
fi</nowiki>
 +
 
 +
Notice that this process can be averted by adding a non-null value for "NO_ACCESS_KEY" to environmental input. A better solution is to add s3 permissions (for the specific bucket you want to access) to your instance role.
 +
 
 +
Here is what the policy document for this permission might look like:
 +
 
 +
<nowiki>
 +
{
 +
    "RoleName": "ecsInstanceRole",
 +
    "PolicyName": "s3policy",
 +
    "PolicyDocument": {
 +
        "Version": "2012-10-17",
 +
        "Id": "s3policy01",
 +
        "Statement": [
 +
            {
 +
                "Sid": "s3statement01",
 +
                "Effect": "Allow",
 +
                "Action": [
 +
                    "s3:PutObject",
 +
                    "s3:GetObject",
 +
                    "s3:GetBucketLocation",
 +
                    "s3:ListBucket"
 +
                ],
 +
                "Resource": [
 +
                    "arn:aws:s3:::mybucket/*",
 +
                    "arn:aws:s3:::mybucket"
 +
                ]
 +
            }
 +
        ]
 +
    }
 +
}</nowiki>
 +
 
 +
To add this policy to your instance role, just use the following cli command:
 +
 
 +
<nowiki>
 +
aws iam put-role-policy --role-name ecsInstanceRole --policy-name s3policy --policy-document file://s3policy.json</nowiki>

Revision as of 19:47, 2 June 2021

Dockerhub Tag:

btingle/dockaws:testing

For information on how to set up spot fleet permissions, see here: https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html#spot-fleet-roles-cli

Environment Variable Input

SHRTCACHE

description: The base directory where intermediate input/output will be stored. Recommended /dev/shm, default is /tmp

S3_INPUT_LOCATION

description: Should be either a directory or file hosted on s3

If S3_INPUT_LOCATION is a directory:

Input will be taken from the AWS_BATCH_JOB_ARRAY_INDEX'th file (0-indexed, because that's how aws happens to do it) listed under the directory

        AWS_BATCH_JOB_ARRAY_INDEX=2
        S3_INPUT_LOCATION=s3://mytestbucket/input

        >>> aws s3 ls $S3_INPUT_LOCATION/ | sort -k4
        >>> 2021-03-12 17:14:26  249815112 H25.db2.tgz
        >>> 2021-03-12 17:14:26  255685176 H26.db2.tgz
        >>> 2021-03-12 11:40:43  307920266 H27.db2.tgz

In this example, s3://mytestbucket/H27.db2.tgz would be evaluated

If S3_INPUT_LOCATION is a file:

Input will be downloaded from the AWS_BATCH_JOB_ARRAY_INDEX'th (again, 0-indexed) line in the file

        AWS_BATCH_JOB_ARRAY_INDEX=1
        S3_INPUT_LOCATION=s3://mytestbucket/input/inputlist_batch1.txt
        ### inputlist_batch1.txt
        s3://mytestbucket/input/H26.db2.tgz
        s3://mytestbucket/input/H25.db2.tgz
        s3://mytestbucket/input/H27.db2.tgz
        ### EOF

In this example, the job would evaluate s3://mytestbucket/input/H25.db2.tgz

S3_OUTPUT_LOCATION

description: A directory hosted on s3 for output to be stored at.

Each job will create a subdirectory within this directory according to its array index for OUTDOCK & test.mol2.gz files.

For example, a job with S3_OUTPUT_LOCATION=s3://mytestbucket/output and AWS_BATCH_JOB_ARRAY_INDEX=5 may produce the following files after the first run:

s3://mytestbucket/output/5/OUTDOCK.0
s3://mytestbucket/output/5/test.mol2.gz.0
s3://mytestbucket/output/5/restart ### potentially, if job is interrupted

S3_DOCKFILES_LOCATION

description: An s3 directory containing dockfiles, e.g receptor grids, parameters, spheres, etc. INDOCK must be present in these files. The directory is recursively copied to local storage for each job, so make sure only the necessities are in this directory.

AWS_BATCH_JOB_ID

description: The name of the current job. Automatically supplied when submitting via aws batch.

AWS_BATCH_JOB_ARRAY_INDEX

description: The array index of the current job. Automatically supplied when submitting via aws batch.

NO_ACCESS_KEY

Disables fetching of access key from aws secretsmanager. See "Adding S3 Permissions to your instance role" section below.

Optional: AWS credentials

If you are testing outside of AWS batch (or even within aws batch, but passing credentials around like that can be dangerous), you may want to provide aws account credentials for accessing any necessary s3 buckets. You can do this by supplying the following as environmental parameters:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION

If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.

In the current configuration, any access keys supplied via environmental parameters is used to fetch access keys from aws secretsmanager, which will be used to fetch/put data in s3. If you do not have your aws access key & secret key in aws secretsmanager, (under the names "aws_access_key" and "aws_secret_key" respectively) you will also want to export a non-null value for NO_ACCESS_KEY.

Restartability & Spot Instances

This image has been configured to take advantage of AWS batch w/spot instances. When a job is interrupted by a spot termination notice, it will exit gracefully and report any partial results along with a restart marker in the output. You can also manually interrupt a job by sending signal 10 (SIGUSR1) to the dock process running within the container.

Exit Codes

When a job completes successfully, the exit code is 0.

When a failure is detected, or if an interruption notice has been caught, the exit code is 1.

In our aws batch configuration, we tell jobs to re-attempt when they exit with code 1, up to a limit of 3-5 attempts. (3 attempts is generally the highest number we observe)

Misc: Configuring Launch Template with Dockerhub credentials

When running jobs at scale with aws batch, dockerhub will likely complain about too much bandwidth being used by your jobs. In order to get around this, you need to have dockerhub premium for your account. This requires that you supply dockerhub authentication to each instance launched for your jobs. This can be accomplished with a Launch Template. Simply create a blank template, and fill in the "user data" section like so:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==DOCKERHUBINIT=="

--==DOCKERHUBINIT==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
yum install -y aws-cli # make sure the aws cli is installed, otherwise this next step won't work
docker_email=ben@tingle.org
docker_user=btingle
export AWS_DEFAULT_REGION=us-west-1
docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass | grep "SecretString" | cut -d':' -f2 | sed 's/\"//g' | sed 's/,//g' | tail -c +2 | sed 's/ *$//g')
# aws secretsmanager returns a json object, this convoluted pipe exists to format it such that only the plain text password is retrieved

echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config
echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"password\":\"${docker_auth}\"\,\"email\":\"${docker_email}\"\,\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config

--==DOCKERHUBINIT==--

In the above example, we use aws secretsmanager to fetch our dockerhub password. This is accomplished by adding the AWSSecretsManager:GetSecret permission for our dockerhub password secret arn to our ecsInstance role. Alternatively, you can hard-code the authentication details into the launch template.

For more information on authentication methods, try the following links:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth-container-instances.html

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html

https://aws.amazon.com/blogs/containers/authenticating-with-docker-hub-for-aws-container-services/

Adding S3 Permissions to your instance role

Currently, the DOCK 3.8 image is configured to fetch an aws secret key and access key from AWS secrets manager, then use those as credentials to access s3. Here's what that looks like in the script:

if [ -z $NO_ACCESS_KEY ]; then
        AWS_SECRET_ACCESS_KEY=$(retrieve_aws_secret aws_secret_key)
        AWS_ACCESS_KEY=$(retrieve_aws_secret aws_access_key)

        aws configure set aws_access_key_id $AWS_ACCESS_KEY
        aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
        aws configure set default.region $AWS_DEFAULT_REGION
fi

Notice that this process can be averted by adding a non-null value for "NO_ACCESS_KEY" to environmental input. A better solution is to add s3 permissions (for the specific bucket you want to access) to your instance role.

Here is what the policy document for this permission might look like:

{
    "RoleName": "ecsInstanceRole",
    "PolicyName": "s3policy",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Id": "s3policy01",
        "Statement": [
            {
                "Sid": "s3statement01",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:GetBucketLocation",
                    "s3:ListBucket"
                ],
                "Resource": [
                    "arn:aws:s3:::mybucket/*",
                    "arn:aws:s3:::mybucket"
                ]
            }
        ]
    }
}

To add this policy to your instance role, just use the following cli command:

aws iam put-role-policy --role-name ecsInstanceRole --policy-name s3policy --policy-document file://s3policy.json