How to Automatically Shutdown Your EC2 Instances when not in use to Save Costs

Often, we need EC2 instances for brief daily tasks but find them sitting idle the rest of the time, consuming unnecessary resources. The issue is that you’re charged for the EC2 instance as long as it’s turned on, regardless of whether you’re actively using it or not. Today, I want to share a strategy I developed to automatically run an EC2 machine for the execution of a script and automatically shut it down afterwards. Oh and it gets even more specific than that, this solution will require you to have another device, like your own laptop, computer or Raspberry Pi.

For context, I needed to perform daily scraping work for apyguru.com, a task that required only about 45 minutes of computation time per day. However, keeping an EC2 instance running 24/7 for this brief task was costing me around $15 per month – a non-negligible expense for such limited usage.

In this post, I’ll walk you through the steps to optimize your EC2 usage for such scheduled tasks, assuming you already have an EC2 instance set up. By the end, you’ll see how I managed to reduce my monthly costs from $15 to just $1.46.

Steps to Optimize Your EC2 Usage

Set up your computer or Raspberry Pi or any other device you have running linux

In my case, I have a Raspberry Pi, so I’ll explain the steps for this setup. However, the same principles apply to any device you might use.

Create a script on the Raspberry Pi

1. Create a new file in your home folder named ec2_script.sh

2. Past the following into ec2_script.sh: 

# Starts the EC2 instance and logs any errors to ec2_script_errors.txt
/home/pi/.local/bin/aws ec2 start-instances --instance-ids INSTANCE_ID > /home/pi/ec2_script_errors.txt 2>&1 &

3. Replace INSTANCE_ID with the proper value. Log in to the AWS console to find out what your instance ID is.

4. In a Terminal, run the following command to make your script executable:

chmod +x ec2_script.sh

Set up a cron job on the Raspberry Pi

1. Use crontab -e to edit your cron jobs.

2. Add a line to run your script at the desired time. For example, to run it daily at 2 AM: 

0 2 * * * /home/pi/ec2_script.py

Prepare your EC2 instance

1. SSH into your EC2 instance.

2. Create a shell script that runs your required program and then shuts down the instance.

For instance, let’s create the following run_and_shutdown.sh script under /home/ubuntu:

# Execute here your main program or script
python3 scrape.py
# Then shutdown the EC2 machine
sudo shutdown now

3. Example run_and_shutdown.sh: #!/bin/bash # Run your program python3 /path/to/your/scraping_script.py # Shutdown the instance sudo shutdown now

4. Configure your EC2 instance to run your shell script on startup.

Add a command to execute run_and_shutdown.sh to /etc/rc.local. If you are not sure how to do that, make sure you read this tutorial.

Alternatively you could add the execution of run_and_shutdown.sh to the crontab.

5. Test your setup

Manually run the script on your Raspberry Pi to start the EC2 instance and monitor the EC2 instance to ensure it starts up, runs your program, and shuts down as expected. Make sure you open the /home/pi/ec2_script_errors.txt file to check if any errors occurred.

That’s it!

By implementing this solution, I was able to dramatically reduce my EC2 costs for this specific task. Instead of paying $14 per month for a continuously running instance, I now pay only $1.46 per month for the same computation work.

This approach is particularly useful for scheduled tasks that require specific hardware or software configurations not available on your local machine. It allows you to leverage the power of EC2 when you need it, without incurring costs for idle time.

Of course, this solution can be brittle, as the internet connection of your Raspberry Pi could be offline right when it needs to boot up the EC2 instance. However, we could imagine adding fancier mechanisms to guard against this. Surprisingly, it has been working incredibly smoothly for me over the last 2 months.

Happy optimizing!