<h1 id="automatinginstallationandremovalofcloudberrydriveforec2scaling">Automating installation and removal of CloudBerry Drive for EC2 scaling</h1> <h2 id="problem">Problem</h2> <p>If you have EC2 AutoScaling group and want to use CloudBerrry Drive inside each managed instance and have a license key for multiple CB Drive installations, you may want to automate CB Drive installation to newly created instances and properly reclaim they key when terminating them.</p> <h2 id="suggestionsandresolution">Suggestions and Resolution</h2> <p>The suggested approach is to use EC2 LifeCycle hooks that will execute routines on instances and wait for completion. LifeCycle hooks sends AWS SNS notification, and AWS Lambda function is triggered to perform appropriate action. Lambda function in turn executes a PowerShell script inside the instance. You can store your PowerShell scripts and CB Drive instance in any HTTP location. In our example they are stored in S3 <a href="https://s3.us-east-2.amazonaws.com/cb-drive/">https://s3.us-east-2.amazonaws.com/cb-drive/</a>, please replace that path with your own location. CBDrive licensing information is stored as an AWS SMS parameter.</p> <h3 id="preparingtheenvironment">Preparing the environment</h3> <p>You need to have an AutoScaling group with <strong>0</strong> instances running. If you are going to add automation to the existing ASG keep in mind that the drive <strong>will not</strong> be installed into running instances. You will need to create ASG on your own, that topic is not covered in this article. In the code below ASG name will be CBDriveASG. You will need to change it to your name.</p> <h3 id="configuringtheenvironment">Configuring the environment</h3> <p>Create SNS topic named <em>CBDrive</em></p> <pre><code>aws sns create-topic --name CBDrive </code></pre> <p>The output should look something like this:</p> <pre><code>{ "TopicArn": "arn:aws:sns:us-east-2:1234567890:CBDrive" } </code></pre> <p>Remember the ARN for the topic, you'll need it later.</p> <p>Create IAM role <em>CBDriveHooksRole</em> that will allow LifeCycle hooks to publish messages to SNS <em>topicCBDrive</em>:</p> <ol> <li>Create a file <strong>AutoscaleAssumeRole.json</strong> and enter the following content. This will allow AutoScaling to assume roles IAM roles.</li> </ol> <pre><code>{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "autoscaling.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } </code></pre> <ol> <li>Then create a policy file <strong>CBDrive.json</strong> and allow publishing to SNS topic <em>CBDrive</em>. You will need to change ARN in the example to the ARN of your topic.</li> </ol> <pre><code>{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Resource": "*", "Action": [ "sns:Publish" ] } ] } </code></pre> <ol> <li>Finally create the IAM Role</li> </ol> <pre><code>aws iam create-role \ --role-name CBDriveHooksRole \ --assume-role-policy-document file://AutoscaleAssumeRole.json </code></pre> <pre><code> Ouput: </code></pre> <pre><code>{ "Role": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "autoscaling.amazonaws.com" }, "Effect": "Allow", "Sid": "" } ] }, "RoleId": "AROAI16U3N2KRKUOCTJ4C", "CreateDate": "2017-07-13T15:56:09.043Z", "RoleName": "CBDriveHooksRole", "Path": "/", "Arn": "arn:aws:iam::1234567890:role/CBDriveHooksRole" } } </code></pre> <ul> <li>And attach the inline policy</li> </ul> <pre><code>aws iam put-role-policy \ --role-name CBDriveHooksRole \ --policy-name AllowPublishCBDriveTopic \ --policy-document file://CBDrive.json </code></pre> <p>No output is expected.</p> <h3 id="createiamrolethatallowexecutionforlambdafunctions">Create IAM role that allow execution for Lambda functions</h3> <ul> <li>Create a file <strong>LambdaRole.json</strong></li> </ul> <pre><code>{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } </code></pre> <ul> <li>And the policy file LambdaPolicy.json</li> </ul> <pre><code>{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Resource": "*", "Action": [ "logs:*", "ssm:*", "autoscaling:CompleteLifecycleAction" ] }, { "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:*", "Effect": "Allow" } ] } </code></pre> <ul> <li>Create the IAM role</li> </ul> <pre><code>aws iam create-role \ --role-name LambdaCBDRole \ --assume-role-policy-document file://LambdaRole.json </code></pre> <p>Output:</p> <pre><code>{ "Role": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "lambda.amazonaws.com" }, "Effect": "Allow", "Sid": "" } ] }, "RoleId": "AROAIZWVMVZR36MN9OXZS", "CreateDate": "2017-07-13T16:00:52.378Z", "RoleName": "LambdaCBDRole", "Path": "/", "Arn": "arn:aws:iam::1234567890:role/LambdaCBDRole" } } </code></pre> <ul> <li>And attach the policy</li> </ul> <pre><code>aws iam put-role-policy \ --role-name LambdaCBDRole \ --policy-name ExecuteCBDriveAutomation \ --policy-document file://LambdaPolicy.json </code></pre> <p>No output is expected.</p> <h3 id="putlifecyclehooks">Put lifecycle hooks</h3> <ul> <li>For pushing CB Drive into newly deployed EC2 instance:</li> </ul> <pre><code>aws autoscaling put-lifecycle-hook --lifecycle-hook-name "Install-CBDrive" \ --auto-scaling-group-name “CBDriveASG” \ --lifecycle-transition "autoscaling:EC2_INSTANCE_LAUNCHING" \ --role-arn arn:aws:iam::1234567890:role/CBDriveHooksRole \ --notification-target-arn arn:aws:sns:us-east-2:1234567890:CBDrive \ --heartbeat-timeout 1800 \ --default-result 'CONTINUE' </code></pre> <ul> <li>And for reclaiming the license:</li> </ul> <pre><code>aws autoscaling put-lifecycle-hook --lifecycle-hook-name "Remove-CBDrive" \ --auto-scaling-group-name "CBDriveASG" \ --lifecycle-transition "autoscaling:EC2_INSTANCE_TERMINATING" \ --role-arn arn:aws:iam::1234567890:role/CBDriveHooksRole \ --notification-target-arn arn:aws:sns:us-east-2:1234567890:CBDrive \ --heartbeat-timeout 1800 --default-result 'CONTINUE' </code></pre> <h3 id="createthelambdafunction">Create the Lambda function</h3> <p>Here’s the code of our Lambda function:</p> <pre><code>import boto3 import time import json import logging logger = logging.getLogger() logger.setLevel(logging.INFO) def getParameter(param_name): # Create the SSM Client ssm = boto3.client('ssm') # Get the requested parameter response = ssm.get_parameters( Names=[ param_name, ], WithDecryption=False ) # Store the credentials in a variable credentials = response['Parameters'][0]['Value'] return credentials def scaleUp(intInstanceId): ssmClient = boto3.client('ssm') license = dict(item.split("=") for item in getParameter('CBBDriveLicense').split(",")) ssmCommand = ssmClient.send_command( InstanceIds = [ intInstanceId ], DocumentName = 'AWS-RunPowerShellScript', TimeoutSeconds = 240, Comment = 'Install Cloudberry Drive', Parameters = { 'commands': [ 'Invoke-WebRequest https://s3.us-east-2.amazonaws.com/cb-drive/installDrive.ps1 -outfile c:/installDrive.ps1','C:/installDrive.ps1 -license ' + license["license"] + ' -email ' + license["email"] ] }, ) #poll SSM until EC2 Run Command completes status = 'Pending' while status == 'Pending' or status == 'InProgress': time.sleep(3) status = (ssmClient.list_commands(CommandId=ssmCommand['Command']['CommandId']))['Commands'][0]['Status'] if(status != 'Success'): logger.info("Failed to install CB Dirve with status " + status) else: logger.info("CB Drive installed") return def scaleDown(intInstanceId): ssmClient = boto3.client('ssm') ssmCommand = ssmClient.send_command( InstanceIds = [ intInstanceId ], DocumentName = 'AWS-RunPowerShellScript', TimeoutSeconds = 240, Comment = 'Remove Cloudberry Drive', Parameters = { 'commands': [ 'Invoke-WebRequest https://s3.us-east-2.amazonaws.com/cb-drive/removeDrive.ps1 -outfile c:/removeDrive.ps1','C:/removeDrive.ps1' ] }, ) #poll SSM until EC2 Run Command completes status = 'Pending' while status == 'Pending' or status == 'InProgress': time.sleep(3) status = (ssmClient.list_commands(CommandId=ssmCommand['Command']['CommandId']))['Commands'][0]['Status'] if(status != 'Success'): logger.info("Failed to remove CB Drive with status" + status) else: logger.info("CB Drive removed") return #The lambda_handler Python function gets called when you run your AWS Lambda function. def lambda_handler(event, context): s3Client = boto3.client('s3') snsClient = boto3.client('sns') asClient = boto3.client('autoscaling') ecClient = boto3.client('ec2') message = json.loads(event[u'Records'][0][u'Sns'][u'Message']) logger.info(json.dumps(event)) if message['LifecycleTransition'] == 'autoscaling:EC2_INSTANCE_TERMINATING': scaleDown(message['EC2InstanceId']) elif message['LifecycleTransition'] == 'autoscaling:EC2_INSTANCE_LAUNCHING': time.sleep(60) scaleUp(message['EC2InstanceId']) else: logger.info("Unknow state " + eventType) response = asClient.complete_lifecycle_action( LifecycleHookName=message['LifecycleHookName'], AutoScalingGroupName=message['AutoScalingGroupName'], LifecycleActionToken=message['LifecycleActionToken'], LifecycleActionResult='CONTINUE', InstanceId=message['EC2InstanceId'] ) return </code></pre> <p>You will need to save it to file ManageCBDrive.py and compress into ManageCBDrive.zip archive. Then create the Lambda function in AWS:</p> <pre><code>aws lambda create-function \ --function-name ManageCBDrive \ --zip-file fileb://ManageCBDrive-1.0.zip \ --runtime python3.6 \ --role arn:aws:iam::1234567890:role/LambdaCBDRole \ --handler ManageCBDrive.lambda_handler \ --timeout 300 </code></pre> <p>Output:</p> <pre><code>{ "TracingConfig": { "Mode": "PassThrough" }, "CodeSha256": "HuG0UIv3s2QzX4RB1Hokq5tn38EZV3LzVHCMyc+Xb9o=", "FunctionName": "ManageCBDrive", "CodeSize": 1349, "MemorySize": 128, "FunctionArn": "arn:aws:lambda:us-east-2:1234567890:function:ManageCBDrive", "Version": "$LATEST", "Role": "arn:aws:iam::1234567890:role/LambdaCBDRole", "Timeout": 300, "LastModified": "2017-07-13T16:56:07.747+0000", "Handler": "ManageCBDrivelambda_handler", "Runtime": "python3.6", "Description": "" } </code></pre> <h3 id="subscribethelambdafunctiontothesnstopic">Subscribe the Lambda function to the SNS topic</h3> <pre><code>aws sns subscribe --protocol lambda \ --topic-arn arn:aws:sns:us-west-2:1234567890:CBDrive \ --notification-endpoint arn:aws:lambda:us-west-2:1234567890:function:ManageCBDrive </code></pre> <p>Output:</p> <pre><code>{ "SubscriptionArn": "arn:aws:sns:us-east-2:988585708477:CBDrive:7527c5b4-5282-47b7-bed3-38943a78f64c" } </code></pre> <h3 id="grantpermissionsonthelambdafunctiontothesnstopic">Grant permissions on the lambda function to the SNS topic:</h3> <pre><code>aws lambda add-permission \ --function-name ManageCBDrive \ --statement-id 1 \ --action "lambda:InvokeFunction" \ --principal sns.amazonaws.com \ --source-arn arn:aws:sns:us-west-2:1234567890:CBDrive </code></pre> <p>Output:</p> <pre><code>{ "Statement": "{\"Sid\":\"1\",\"Resource\":\"arn:aws:lambda:us-east-2:1234567890:function:ManageCBDrive\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"sns.amazonaws.com\"},\"Action\":[\"lambda:InvokeFunction\"],\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:sns:us-east-2:1234567890:CBDrive\"}}}" } </code></pre> <h3 id="createawssmsparameterandstorelicensinginformationthere">Create AWS SMS parameter and store licensing information there</h3> <pre><code>aws ssm put-parameter --name CBBDriveLicense \ --type String \ --value "license=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX,email=yourame@yourdomain.com" </code></pre> <h3 id="uploadpowershellscriptandcbdrivedistrotoyourhttpshare">Upload PowerShell script and CB Drive distro to your HTTP share</h3> <p>installDrive.ps1:</p> <pre><code>param ( [Parameter (Mandatory=$True)][string]$license, [Parameter (Mandatory=$True)][string]$email ) New-Item -ItemType Directory -Force -Path c:\bootstrap $url="http://s3.amazonaws.com/cb_currentreleases/Drive/setup.exe" $output="c:\bootstrap\setup.exe" Invoke-WebRequest $url -outfile $output $install_command=$output $install_arguments= " /S" start-process -FilePath $install_command -ArgumentList $install_arguments -Wait $activate_command="C:\Program Files\CloudBerryLab\CloudBerry Drive\cbd.exe" $activate_arguments= '/activatelicense' + ' -k "'+$license+'" -e "'+$email+'"' Start-Process -FilePath $activate_command -ArgumentList $activate_arguments -Wait Remove-Item -Path $output </code></pre> <p>removeDrive.ps1:</p> <pre><code>$release_command="c:\Program Files\CloudBerryLab\CloudBerry Drive\cbd.exe" $release_arguments= " /silent /releaselicense" start-process -FilePath $release_command -ArgumentList $release_arguments -Wait $uninstall_command="c:\Program Files\CloudBerryLab\CloudBerry Drive\uninst.exe" $uninstall_arguments=" /S" Start-Process -FilePath $uninstall_command -ArgumentList $uninstall_arguments -Wait Remove-Item -Path C:\removeDrive.ps1 </code></pre>