Introduction to scripting with Amazon EC2

My name is Owen Evans, and I work in Esri’s D.C. Technology Center in Vienna, Virginia, where we demonstrate Esri technology to many US government customers. Most of the demonstrations we give are Web-based, so we make our GIS services and applications accessible outside the office by publishing them to a few servers that we have on premises.

Over the past few months, however, we have been supplementing our on-premises hardware resources with virtual resources in the Amazon Elastic Compute Cloud (EC2), using the recently-released ArcGIS Server on Amazon EC2. In this post, I’d like to share some of our experiences using scripting to manage our cloud resources. This post assumes you’re familiar with EC2 terminology, which you can review here.

Why use scripting with Amazon EC2?

One benefit of cloud computing is the commoditization of computing power; you can pay one simple hourly fee for resources that actually have many embedded costs associated with them (procurement, hardware, energy, building space, software installation and maintenance, etc.). Another benefit of cloud computing power is that you can turn it on and off like a water faucet and only pay for what you use.

Scripting is a simple way to automate scheduled tasks. With simple scripting tools that Amazon Web Services (AWS) provides, you can start and stop your EC2 instances during times when your apps and services are not needed. You can also automate other tasks like backups and expansion/contraction of capacity in the cloud. Scripting can help you manage your elastic computing utility bill, thus saving you time and money by conserving computing resources and automating repetitive scheduled tasks. Let’s look at a few examples of how you might use scripts to manage your cloud resources.

Use cases for scripting Amazon EC2

Scheduled automation

Like many organizations, our business operations are mainly during the work week (Monday through Friday). Since our demo servers are not typically used on the weekends, we shut them down at about 9pm ET on Friday evening (6pm PT) and start them up again at 7am ET on Monday morning. This allows us to save 58 hours of computing time each weekend (3 hrs for Friday, 24 hrs each for Saturday and Sunday, and 7 hrs for Monday). At $0.48/hr that is about $30/weekend or $1500/year for one instance. That’s a big payoff for the small effort of writing a script. Automating these repetitive processes saves time since we don’t have to remember to manually start and stop these instances, and it saves money since the resources are shut down when they are not being used.

Flexible administration

We also have EC2 instances that support in-office activities. These instances can be started and stopped by anyone using scripts that we have set up with some desktop shortcuts. When we need to give a presentation that uses these cloud resources, we start the instances. Then, when the meeting is over, we stop them. The person that turns the instances on and off doesn’t have to know the credentials for our AWS account. In this case, our scripts allow a non-administrator to manage cloud resources in a controlled way.

How to get started with scripting Amazon EC2

AWS has a set of downloadable command line tools that enable you to interact with the EC2 instances programmatically. The tools perform functions like starting/stopping instances, creating/deleting tags, and allocating/associating elastic IP addresses. Setting up these tools is not difficult. If you are familiar with the command line and environment variables, that certainly helps; but the configuration is straightforward enough that you can learn as you go.

Below is a step by step list of what you need to do to get started. The hyperlinks point you to download locations and specific steps in the documentation.

1. Install Java (version 5 or later, SDK or JRE)
2. Create a folder to store your tools, scripts, and related files (e.g., C:AWS or C:Users[user]DocumentsAWS)
3. Download the Amazon EC2 API Tools
4. Create an X.509 certificate and download the two associated .pem files
5. Set environment variables for your Java install location, EC2 tools location, and your account credentials
6. Run commands

Once you install the tools, you can run any of the EC2 commands from a command line. Scripting enables you to chain several commands together and save common workflows so that you can reuse them. One of the simplest ways to do this is with a batch file (.bat), which is a just text file containing a list of commands that run in sequence. To make a batch file, just create a new text file and change the file extension from .txt to .bat. Then type your commands in the file in the order you want them to run.

Basic Amazon EC2 scripting examples

Below are a few simple scripts for common workflows: 1) stop an instance (our Friday evening script), 2) start an instance and associate an elastic IP (our Monday morning script), and 3) stop multiple instances.

Stop an instance

Perhaps the simplest script is one that stops an already running instance. This example calls the EC2-STOP-INSTANCES command and passes the instance name that I want to stop. I’m using sample values for instance names and other parameters, you would simply substitute your information for the placeholders in the examples below.

ec2-stop-instances i-abcd1234

Easy, right? In this case we just specify the stop command and an instance name, and the instance is stopped.

Ok, now let’s try something a little more interesting.

Start an instance and associate an elastic IP

This next script starts an instance, then associates an elastic IP address with that instance. This is a common workflow because when you stop and start an EC2 instance, the IP address changes. An Amazon Elastic IP gives you a way to maintain an unchanging address for your instances. The only requirement is that whenever you stop and start the instance, you have to re-associate the address. Here’s the script:

call ec2-start-instances i-abcd1234
timeout 300
ec2-associate-address 0.0.0.0 -i i-abcd1234
ec2-reboot-instance i-abcd1234

The first line calls the EC2-START-INSTANCES command, which has a similar syntax to the EC2-STOP_INSTANCES command showed above. In this case, since we want to run several commands in sequence, we run the EC2 command in its own shell using the CALLcommand. The EC2 batch commands are actually shell scripts that run Java code, and CALL is needed to return control to the original batch file. If CALL is not used, the batch process terminates after the first EC2 command, and the rest of the commands are never executed. (Note that CALL is optional in the “stop instance” script since it only runs a single command.)

Then there is a TIMEOUT command that waits five minutes (300 seconds) to give the instance time to start. When an instance is first started, the status is “pending” until the startup process is complete. At that point, the status changes to “running.” An elastic IP can only be associated with a running instance, so we need to wait this out in our script. In my experience, five minutes has been plenty of time, but you may find that you need to adjust this.

Next, EC2-ASSOCIATE_ADDRESS is used to associate the elastic IP with the instance, and, finally, the last command in this script reboots the instance. Rebooting is required for ArcGIS Server to recognize the elastic IP. Rebooting is not needed for an enterprise geodatabase instance.

Start/stop several instances

Of course these commands can be chained together to stop or start several instances. One way is to add a separate command for each instance; however, the EC2 commands enable you to reference several instances in the same command. See the example below, which would stop three instances:

ec2-stop-instances i-abcd1234 i-efgh5678 i-ijkl9012

Scheduling scripts

Once you have some scripts to work with your EC2 instances, you can use operating system tools to schedule them to run at regular times (such as Friday nights or Monday mornings). For example, Windows Task Scheduler is available on the various desktop and server editions of Windows and gives you an easy-to-use GUI environment for scheduling a script.

You can set a BAT file to run in the Task Scheduler by creating a Basic Task (Action > Create Basic Task). Once you name your task and set the schedule, you’ll be prompted to select an action to perform. Choose Start a Program and point to your BAT file. Your script is now set to run.

Keep in mind that you can always manually launch a script by just double-clicking it, or running it from the task scheduler. An advantage of running the script from the Task Scheduler is that a history will be recorded of when the task was run. This information is accessible in the properties of any task under the History tab.

Summary

In summary, Amazon EC2 includes scripting tools that can help you automate your work. Scripts make it easier to administer your servers remotely, and, in many cases, allow you to cut costs. Once you have a useful script, you can use operating system tools to run it automatically.

This post is meant as an introduction to scripting, but going further you can use scripting to launch or destroy new instances, create security groups, attach volumes, and so on. Stay tuned to this blog for more scripting tips.

Contributed by Owen Evans of the Esri Washington, D.C. Technology Center