How to Automatically Stop an EC2 Instance With No Open SSH or Session Manager Connections

Stopping an EC2 instance that you don’t need can save you money and improve security. In this post I demonstrate how to use the AWS CDK to provision an EC2 instance that is automatically stopped when there are no open SSH or Session Manager connections to it.

These days I most often use an EC2 instance as a Bastion server. I use SSH or Session Manager to connect to it and access a protected database from there. This is usually just for occasional maintenance or debugging. An “always on” instance for this purpose feels wasteful since it would just sit idle most of the time. Because I only use it with an SSH (or Session Manager) connection we can automate the process of stopping it when there are no remaining connections. The primary downside of this is that I must manually start the instance (and wait for it to boot) when I need to use it. In my case it is a tradeoff I’m happy to make.

Below I demonstrate how to use the AWS CDK to provision an EC2 instance that is automatically stopped when there are no open SSH or Session Manager connections to it. The full source code is available on GitHub at https://github.com/mpvosseller/cdk-ec2-autostop.

The approach we take is as follows:

1. Install a bash script report-metrics.sh on the instance to determine whether it is “active” or not and publish the result as a CloudWatch metric.
2. Configure a cronjob on the instance to run report-metrics.sh once a minute.
3. Configure a CloudWatch Alarm to monitor the metric and stop the instance when it is reported inactive for more than 15 minutes.

Report Metrics Script

Below is the report-metrics.sh script and it does most of the heavy lifting. It runs on the instance and determines whether it should be considered “active” or not. It then publishes a CloudWatch metric with the result. We consider the instance active if it has any open SSH or Session Manager connections to it or if the instance was recently booted.

We use various tools and services to accomplish this job:

• The EC2 instance metadata service at 169.254.169.254 is used to determine the availabilityZone, region, and instanceId of the instance.
• ec2 describe-instances is used to get the stackName to which the instance belongs.
• aws ssm describe-sessions is used to count the number of Session Manager connections.
• ss is used to count the number of SSH connections.
• aws cloudwatch put-metric-data is used to publish our custom CloudWatch metrics.
• jq is used for parsing JSON API responses.

#!/bin/bash

# Publish a CloudWatch metric to report whether this instance should be considered active or not.
# We consider it active when there are any open SSH or Session Manager connections to it or if it
# was recently booted.

availabilityZone=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
region=$(echo "${availabilityZone}" | sed 's/[a-z]$//')
instanceId=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
stackName=$(aws --region "${region}" ec2 describe-instances --instance-id "${instanceId}" --query 'Reservations[*].Instances[*].Tags[?Key==`aws:cloudformation:stack-name`].Value' | jq -r '.[0][0][0]')
uptimeSeconds=$(cat /proc/uptime | awk -F '.' '{print $1}')
uptimeMinutes=$((uptimeSeconds / 60))
ssmConnectionCount=$(aws ssm describe-sessions --filters "key=Target,value=${instanceId}" --state Active --region "${region}" | jq '.Sessions | length')
sshConnectionCount=$(/usr/sbin/ss -o state established '( sport = :ssh )' | grep -i ssh | wc -l)
((totalConnectionCount = ssmConnectionCount + sshConnectionCount))

# note that "ssh over ssm" connections are double counted

isActive=0
if [ "${totalConnectionCount}" -gt 0 ] || [ "${uptimeMinutes}" -lt 15 ]; then
    isActive=1
fi

metricNameSpace="${stackName}"
aws --region "${region}" cloudwatch put-metric-data --metric-name "ConnectionCount" --dimensions InstanceId="${instanceId}" --namespace "${metricNameSpace}" --value "${totalConnectionCount}"
aws --region "${region}" cloudwatch put-metric-data --metric-name "UptimeMinutes" --dimensions InstanceId="${instanceId}" --namespace "${metricNameSpace}" --value "${uptimeMinutes}"
aws --region "${region}" cloudwatch put-metric-data --metric-name "Active" --dimensions InstanceId="${instanceId}" --namespace "${metricNameSpace}" --value "${isActive}"

Contab File

Below is the crontab file. Nothing exciting here. It is a standard crontab file that requests the report-metrics.sh script to be run every minute.

* * * * * /home/ec2-user/report-metrics.sh

CDK Application

Below is the main CDK application code. If you are familiar with the AWS CDK it should be mostly straightforward. The most important aspects are:

• Install the jq package on the instance because it is needed by report-metrics.sh.
• Install the files report-metrics.sh and crontab on the instance at /home/ec2-user/.
• Run the crontab command to setup the cron job.
• Grant the permissions required for Session Manager and report-metrics.sh.
• Create a CloudWatch Alarm that stops the instance when it is inactive over a 15 minute period.

import * as cloudwatch from '@aws-cdk/aws-cloudwatch'
import * as actions from '@aws-cdk/aws-cloudwatch-actions'
import * as ec2 from '@aws-cdk/aws-ec2'
import * as iam from '@aws-cdk/aws-iam'
import { CfnOutput, Construct, Duration, Stack, StackProps } from '@aws-cdk/core'

export class CdkEc2AutostopStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)

    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', {
      isDefault: true,
    })

    const keyName = '' // for ssh you must enter the name of your ec2 key pair

    const instance = new ec2.Instance(this, 'Instance', {
      init: ec2.CloudFormationInit.fromConfig(
        new ec2.InitConfig([
          ec2.InitPackage.yum('jq'),
          ec2.InitFile.fromFileInline(
            '/home/ec2-user/report-metrics.sh',
            './assets/report-metrics.sh',
            {
              owner: 'ec2-user',
              group: 'ec2-user',
              mode: '000744',
            }
          ),
          ec2.InitFile.fromFileInline('/home/ec2-user/crontab', './assets/crontab', {
            owner: 'ec2-user',
            group: 'ec2-user',
            mode: '000444',
          }),
          ec2.InitCommand.shellCommand('sudo -u ec2-user crontab /home/ec2-user/crontab'),
        ])
      ),
      instanceName: 'AutoStopInstance',
      vpc,
      keyName: keyName || undefined,
      instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.NANO),
      machineImage: ec2.MachineImage.latestAmazonLinux({
        generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
        cpuType: ec2.AmazonLinuxCpuType.X86_64,
      }),
    })

    // WARNING: this opens port 22 (ssh) publicly to any IPv4 address    
    // instance.connections.allowFrom(ec2.Peer.anyIpv4(), ec2.Port.tcp(22), 'SSH access');

    // permissions for session manager
    instance.addToRolePolicy(
      new iam.PolicyStatement({
        actions: ['ssmmessages:*', 'ssm:UpdateInstanceInformation', 'ec2messages:*'],
        resources: ['*'],
      })
    )

    // permissions for report-metrics.sh script
    instance.addToRolePolicy(
      new iam.PolicyStatement({
        actions: ['ec2:DescribeInstances', 'ssm:DescribeSessions', 'cloudwatch:PutMetricData'],
        resources: ['*'],
      })
    )

    const alarm = new cloudwatch.Alarm(this, 'Alarm', {
      alarmName: `Idle Instance - ${this.stackName}`,
      metric: new cloudwatch.Metric({
        // this metric is generated by report-metrics.sh
        namespace: this.stackName,
        metricName: 'Active',
        dimensions: {
          InstanceId: instance.instanceId,
        },
        statistic: 'maximum',
        period: Duration.minutes(15),
      }),
      comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
      threshold: 1,
      evaluationPeriods: 1,
      treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
    })
    alarm.addAlarmAction(new actions.Ec2Action(actions.Ec2InstanceAction.STOP))

    new CfnOutput(this, 'InstanceId', {
      description: 'Instance ID of the host. Use this to connect via SSM Session Manager',
      value: instance.instanceId,
    })
  }
}

Once deployed I get an EC2 instance that can be manually started when needed and it will be automatically stopped when there are no remaining SSH or Session Manager connections to it. This lets me sleep just a tiny bit better at night.

The full code can be found on GitHub at https://github.com/mpvosseller/cdk-ec2-autostop.

If you found this helpful or have some feedback please let me know.