Get Started with Amazon Web Services

Objectives:

  • To learn about Amazon Cloud Computing Services and set up an account
  • Set up an EC2 account and launch an instance
  • Set up an S3 account, create a bucket, and add an object to it
  • Download and configure the Firefox S3 Organizer
  • Use S3 Organizer to create a bucket and transfer files between S3 and your workstation

About Amazon Web Services (AWS)

Amazon Web Services provides IT infrastructure services including Elastic Compute Cloud (EC2), Simple Storage (S3), and Elastic MapReduce. For more information on AWS, see
What is AWS?

Amazon Elastic Compute Cloud (EC2)

Amazon Elastic Compute Cloud (Amazon EC2) is a Web service that enables you to launch and manage Linux/UNIX and Windows server instances in Amazon's data centers. Amazon Elastic Compute Cloud (Amazon EC2) is a Web service that provides resizable compute capacity in the cloud. It is designed to make Web-scale computing easier for developers. For a more in-depth introduction to EC2, see Introduction to Amazon Elastic Compute Cloud.

Amazon Elastic Block Store (EBS)

Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persists independently from the life of an instance. Amazon Elastic Block Store provides highly available, highly reliable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Click here for more information on EBS.

Amazon Simple Storage Service (S3)

Amazon Simple Storage Service (Amazon S3) is storage for the internet. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. You can accomplish these tasks using the AWS Management Console. For a more in-depth introduction to S3, see Introduction to Simple Storage Service.

Setting up an Amazon Web Service (AWS) Account

You will use the same Amazon account username and password that you use to shop for books, electronics, etc.

Accessing an AWS account

  1. Open the following link in a new window or tab: http://aws.amazon.com
  2. Sign up for an AWS account using the yellow button on the upper right.
  3. Click Security Credentials
  4. For "My e-mail address is", enter the the email address you use to access your Amazon account.
  5. Select "I am an returning user and my password is:"
  6. Enter your "Amazon password."
  7. Click Sign in using our secure server. You are now logged into the Security Credentials Page.

Amazon Web Services (AWS) Security

Amazon has a barrage of security credentials that you need to use. Keeping the security credentials straight is the most difficult part of using AWS. Refer back to the AWS "Security Credentials" page if you need help. You will need to refer to this page later, especially when saving your own Amazon Machine Image.

  1. On the "Security Credentials" page, look under "Access Credentials." The Access Key ID and the Secret Access Key allow you to use external tools to access your AWS account. This information will be used with the Firefox "S3 organizer" browser plugin (discussed later) to access your Amazon S3 account. Do not create a new access key!

  2. Click the "X.509 Certificates" tab. If you ever want to save a custom Amazon Machine Image to the S3 storage service rather than Elastic Block Store (EBS) (this is useful for making public instances), you will need to know your X.509 certificate as well.

  3. Click the "Key Pairs" tab. To connect to your running instances using SSH, you will be using SSH key pairs, which Amazon refers to as "Amazon EC2 Key Pairs." You will create your own unique keypair later.

  4. Scroll to the bottom of the page to the "Account Identifiers" section. The "AWS Account ID" is sometimes needed to identify yourself when using various EC2 functions, like saving a customized Amazon Machine Image to S3.

Running Instances with Amazon Elastic Compute Cloud (EC2)

Amazon EC2 presents a true virtual computing environment, allowing you to use web service interfaces to launch instances with a variety of operating systems, load them with your custom application environment, manage your network’s access permissions, and run your image using as many or few systems as you desire.

An image is an Amazon Machine Image, a file that contains all the information required to run a virtual computer, including the operating system and any local data.  You can launch an image to create an instance.  An instance is a running virtual machine that has all the capabilities of a physical computer.

To use Amazon EC2, you:

  1. Select a pre-configured, templated image to get up and running immediately or create an Amazon Machine Image (AMI) containing your applications, libraries, data, and associated configuration settings.
  2. Configure security and network access on your Amazon EC2 instance.
  3. Choose which instance type(s) and operating system you want, then start, terminate, and monitor as many instances of your AMI as needed, using the web service APIs or the variety of management tools provided.
  4. Determine whether you want to run in multiple locations, utilize static IP endpoints, or attach persistent block storage to your instances.
  5. Pay only for the resources that you actually consume, like instance-hours or data transfer.


To sign up and get started with Amazon EC2, follow the Amazon guide located here: EC2 Getting Started Guide.
 

  

Managing Data with Amazon Simple Storage Service (S3)

 


Amazon Simple Storage Service (Amazon S3) is storage for the internet. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. You can accomplish these tasks using the AWS Management Console.

 

 

Clouds are ephemeral bodies as is data on EC2. This is one important distinction between EC2 and your desktop workstation. Whenever you shut down your workstation, all of the data on the hard drive is still there. With EC2, whenever you terminate a running instance, you lose *all* data on that instance unless you have created a new AMI with it before you pull the plug.

Creating an Amazon Machine Image (AMI) is a time- and storage-intensive process. You don't want to have to store a new AMI every time you change a little bit of data on an instance. For preserving data within AWS, Amazon provides a persistent storage infrastructure called the "Simple Storage Service" (S3).

To get the most out of Amazon S3, you need to understand a few simple concepts. Amazon S3 stores data as objects within buckets. An object is comprised of a file and optionally any metadata that describes that file.

To store an object in Amazon S3, you upload the file you want to store to a bucket. When you upload a file, you can set permissions on the object as well as any metadata.

Buckets are the containers for objects. You can have one or more buckets. For each bucket, you can control access to the bucket (who can create, delete, and list objects in the bucket), view access logs for the bucket and its objects, and choose the geographical region where Amazon S3 will store the bucket and its contents.

When using the AWS Management Console you can create folders to group objects. You can nest folders (create folders within folders). If you have used the Amazon S3 API or other utilities, you can learn some important aspects about how folders work with other grouping conventions in the Amazon S3 Console User Guide.

  • Objects - A file or other piece of data.
  • Buckets - You upload objects into buckets.

There is no limit to the number of objects that you can store in a bucket. Every bucket has a name that is unique across all of S3, so you will need to be descriptive in your bucket names. Each developer can own up to 100 buckets at a time. You own each bucket you create, and by default nobody else can access your buckets. Note that Amazon charges you money based on how much S3 storage space you use. Their charges can be found on the Amazon S3 webpage.

To sign up and get started with Amazon S3, follow the Amazon S3 guide located here: S3 Getting Started Guide Note: Ignore instructions in the Amazon S3 guide to delete the bucket you create. You will use this bucket in the following section.

Firefox S3 Organizer Plugin


The eScience Institute recommends the Firefox S3 Organizer Extension to help organize/manage/store your files on Amazon S3.

Install S3 Firefox Organizer Plugin  

  1. Follow the link to the website.
  2. Click Add to Firefox.
  3. Click Accept and Install.
  4. Click Install.
  5. Restart Firefox.
  6. Verify installation by clicking Tools and selecting S3 Organizer.

Entering your credentials into the S3 Organizer

  1. Click Tools and select S3 Organizer
  2. At the top left, click Manage Accounts.
  3. For the "Account Name", enter "eScience Test Account". The "Account Name" can be anything you want - it's just an identifier to S3 Organizer if you use it to access more than one S3 account. 
  4. The "Access key" and "Secret key" are available on the AWS "Security Credentials" page discussed previously.
  5. Verify that you can now see the bucket you created in the Amazon S3 guide under "Remote View."

You can now drag files to and from your S3 buckets just like dragging files between folders on your computer.

Create a unique S3 bucket using S3 Organizer

In S3 Organizer:

  1. At the upper right, click the blue folder. The tooltip will say "Create Bucket/Directory"
  2. Enter a name of the form escience.washington.edu.<keypair_name>

Now, you will copy your private key file into your newly created S3 bucket.

Transfer Files between S3 and your local workstation

  1. Your private key file is located in your default download directory. (You should hace created and downloaded your Key Pair while going through the Amazon EC2 Getting Started Guide). The file will be named <keypair_name>.pem
  2. Drag your private key file ending in ".pem" from you local workstation into your S3 bucket. The transfer should take a very short amount of time. You have now copied data into S3.

Summary

This guide introduced Amazon Web Services with an emphasis on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3). On completion, you should have created Amazon AWS, EC2, and S3 accounts. You should be familiar with starting and logging into an EC2 instance, creating buckets and objects with S3, and using the Firefox S3 Organizer Add-on to manage S3 buckets.

Learn More

General Amazon Web Services (AWS) information:

Amazon Elastic Compute Cloud (EC2) information:

Amazon Simple Storage Service (S3) information: