Transfer Data to Amazon S3 Quickly using AWS Import Export

If you have hundreds of gigabytes or even terabytes of data on your local network at home, you probably have it all stored on a computer, an external hard drive or a NAS (network attached storage) device. Having backups of your data is extremely important, but having them all in one place is never a good idea.

I realized this myself when I saw I have over 2 TB of photos, videos, backups, etc stored on my local NAS. Sure it has 4 hard drives and if one fails, none of my data will be lost. However, if my house burns down or gets flooded, everything will be lost along with the NAS. So I decided to backup the data to the cloud.

I checked out Dropbox, SkyDrive, Google Drive, CrashPlan and Amazon S3 and Glacier before finally settling on Amazon S3. Why Amazon? Well, they have a cool service where you can send in an external hard drive up to 16 TB in size and have it uploaded directly to their servers, thereby bypassing the massive problem of trying to upload that data over your slow Internet connection.

aws

With AT&T in my neighborhood, I get a whopping 1.4 Mbytes/sec upload speed. It would take many months to upload the 2.5 TB of data I have stored on the NAS. With Amazon Import/Export, you can pay a $80 service fee and have them upload all that data for you in one day. I ended up making a video tutorial that walks you through the whole process from signing up for Amazon Web Services to packing your hard drive and shipping it to Amazon.

Here is the full transcript of the video:

Hey everyone. This is Aseem Kishore from Online Tech Tips. I’m going to be doing something new today. I’m going to do a video tutorial on Amazon Web Services Import Export features. So what is the Import Export feature?
Well it’s basically a way to get a large amount of data into an Amazon S3 bucket or into a Glacier vault. Amazon S3 and Glacier are basically two storage options that you have for data backup and data archiving with Amazon. So why would you want to use this service from Amazon?

Well, it basically lets you move a large amount of data into the Cloud very quickly. If you are someone like me, you might have hundreds of gigabytes of photos and videos stored locally on your computer or on an external hard drive. Trying to upload 100 gigabyte or 500 gigabyte or even a terabyte of data into the Cloud will take you weeks if not months on a slow upload connection. Instead what you can do is copy that data onto an external hard drive that can be up to 16 terabytes in size and just ship that to Amazon where they will take it to their data center and upload it straight to your bucket or vault and then you can go ahead and access that from the web.

So to get started, the first thing you are going to have to do is create an Amazon Web Services Account. To do that, you’re going to go to aws.amazon.com and you’re going to go ahead and click on the Sign Up button. Go ahead and type in your e-mail address and then select, “I am a new user,” if you do not have an Amazon account already. If you do, go ahead and select, “I am a returning user,” and you can use your current account Amazon account to sign up for Amazon Web Services.

Once you created your Amazon Web Services Account, you’re going to have to download the Import Export tool. This tool is very simple to use. It does take a little configuration, which I am going to go ahead and explain. But you can see on the screen, there is a down load link which I am going to add in the caption in the bottom of this video. So go ahead and download that and then extract that into a directory on your computer.

Now that you’ve downloaded that tool and extracted it, you should have a directory that looks like this. At this point, we will need to edit a file called, “AWS Credentials.” This contains two values, Access Key ID and Secret Key. Basically, these are two values that Amazon uses to link to your account. You can get these two values from your Amazon Web Services Account by going to the following address. It’s aws.amazon.com/securitycredentials. On the Security Credentials page, you’re going to go ahead and click on Access Keys.

Now it gets a little confusing here. If you’ve already used Amazon Web Services and have already created keys in the past, then you won’t be able to see your secret key here. This is kind of a new interface from Amazon and in order to see your existing secret keys, you have to click on a Security Credentials link that takes you to the old Legacy page.

If you just created a new account, then you’ll be able to create a new root key. This button will be active. At that point you’ll get an access Key ID, and you’ll get the secret key so that it will give you both the values. And this is the Legacy Security page where you can access your secret keys if you have already created an Access Key ID for Amazon Web Services. So as you can see here, I have two access keys and if I wanted to go ahead and see my secret key, I can go ahead and click the Show button and then I can copy those two values into AWS Credentials file that I had shown you earlier. So you want to go ahead and paste the Access ID key here and paste the Secret key here.

Now at this point, if you are getting confused by the Access Key ID and the Secret Access key, that’s okay. You really don’t need to know what they are or care about them in any way whatsoever. All you have to do is sign and, get the values, and copy and paste them into that file.

The next thing we’re going to go ahead and do is create import job. Now the next two parts are the two hardest parts of this whole procedure. In order to create an import job for Amazon S3, we’re going to go ahead and create a manifest file. This manifest file basically contains some information on your device. Wherever you want to store the data and where do you want the device shipped back to.

Now the nice thing is that we don’t have to create this manifest file ourselves. It’s already created for us, we just have to go ahead and fill it out. So what you’ll want to go ahead and do is go into the directory and where you have your import export tool and click on Examples. Here you are going to go ahead and open up the S3 import manifest. As you can see here, I’ve already gone ahead and filled out the information for my import job. So let’s go ahead and take a look at this a little bit more closely.

As you can see, the first thing you have to do is type in your access key ID again. You have to get rid of the brackets, and you just go ahead and paste it directly after the colon. The next thing you’re going to want to do is type in the bucket name. You’re going to have to go ahead and create a bucket, which I’m going to go ahead and show after this, but for now go ahead and type in whatever name that you will want where you’re data is going to be stored. So if you create a folder called Back Up, than anything that you have on your device, any folders or anything in there, will go underneath that bucket name.

The next thing that you will want to go ahead and do is type in your device ID. This is basically a unique identifier for your external hard drive. This can be the serial number that’s on the back of the hard drive. If you don’t have a serial number that’s on the back of your hard drive, what you can go ahead and do is just create a number of your own or create an identifier. Just write that on something, a sticker that you can put onto your device and then just type that value here. It just has to be something that is the same on the device and in this file. Erase device, it’s already set to No, so you are going to leave that. You can leave the next one. Service level is standard, you can leave that. And the return address, you’re going to go ahead and fill out your address like I’ve done here. In the original file, there are some optional fields. You have to go ahead and remove those if you’re not going to use them. So you can just go ahead and delete those lines out.

Okay, so the next thing we’re going to do after we fill out the manifest file is save it into the appropriate directory. To do that, we’re going to go ahead and click File, Save As, and we’re going to move up back into the import export Web Services Tool directory. This is also the location of that dot property file that we filled out earlier. Here you are going to have the go ahead and name your file, “my import manifest.txt. ” Since your Save As Type is already txt, you don’t have to type that into the file name. Go ahead and click Save.

Now that we’ve edited the AWS Credentials file and credited the My Import Manifest file, we can go ahead and create a bucket in Amazon S3. This is very simple to do. What you are going to go ahead and do is go to aws.amazon.com, and you’re going to go ahead and click on My Account console and then click on AWS Management Console. Once you log in, you should get a screen that looks like this with all the different Amazon Web Services. At this point, all we care about is Amazon S3, which is down here at the bottom left. Click on that, and it’s going to go ahead and load up the S3 console. And as you can see here, there’s not really much to it other than buckets. So I have two buckets, this is my backup of my synology nas, which is the network type storage device.

What you’ll want to go ahead and do is click Create Bucket, and think you’re going to go ahead and give your bucket a bucket name. You can also choose a different region, but I suggest you just go to the region that it populates for you automatically. The bucket name just can only have dots, and it has to be unique in that entire region where it’s being stored. So if somebody else already has that bucket name, it’s going to give you an error. For example, if I say, nasbackup, and I say create, it’s going to give me an error that the requested bucket name is not available. In that case you can use dots so you can put ‘dot’, and whatever else you want and click create, and if that’s unique, and then it goes ahead and creates that bucket name. So you can go ahead and create a bucket, that is we a data on all of that external hard drive is going to be stored.

At this point, you might be wondering what else has to be done. So let’s take a look at what we have done so far. We signed up for the AWS service. We’ve downloaded and extracted the tool. We’ve edited the file and editor keys. We’ve gone ahead and created the manifest file is saved it in the import manifest in the same directory as the credentials file, and we created a bucket on Amazon S3. So there’s only a couple more things to do to get this done.

The next thing we have to do is create a job request using a Java command line tool. This is a bit technical and this is probably the most technical thing that you are going to have to do, but is really not that hard. Now in order to create this job request, we have to run a Java command at the command prompt. But in order to do that, we have to have the Java development kit installed. This is different from the Java runtime environment, which is normally installed on most computers, but it won’t let you run Java commands at the command prompt.

In order to do that, what you’ll do is go to Google and just do a search for Java SE, and this is Java Standard Edition. Go ahead and click on the first link here and this brings you to this page. Here you could scroll down, and you’ll see three options, JDK server, JRE, and JRE. We don’t need to worry about these two here. We’re going to go ahead and download the JDK. On the next page, go ahead and click, Accept License Agreement and then you can download the file that matches your system specifications. In my case, I downloaded the Windows 64 bit executable file.

Now that you’ve install the Java executable kit, we can go ahead and run the Java command, and you can go ahead and see this command here in the documentation that I have highlighted here. And by the way, if you need to get to this documentation, the easiest way is to go to Google and do a search for “AWS import export docs”. And then go ahead and click on create your import job, and then click on create your first Amazon S3 import job, and you’ll be brought to this page.

Now we can go ahead and run the command by going to the command prompt. In order to do that we click on Start, type in CMD and press Enter. Now that we have a command prompt, we need to go into the directory where the Amazon import export tool is located. In our case, it’s in Downloads, and then there’s a folder called Import Export Web Service Tool. So in order to navigate directories into the command prompt, you type in “cd”, and then I’m going to type in “downloads”, and then I’m going to type in “cd” again, and I’m going to type in “import export web service tool,” which is the name of the Directory. Now that I’m in that Directory, I’m simply going to go ahead and copy this command and paste that into the command prompt.

You may have noticed that in the command we just copied and pasted, the name of the manifest file is My S3 Import Manifest.txt. I think this is a problem with the documentation because when I tried to run it this way, I got an error saying that the file had to be named My Import Manifest.txt. So simply move your cursor and delete the S3 part, and you should be able run the command. Now I’m not going to go ahead and run the command right now because of already run it before. But when you go ahead and press Enter you should get something like this, job created, job ID, the AW shipping address, and the Signature File Contents.

The signature file contents is basically a file that’s created in the root directory here under Import Export Web Services tool call Signatures. This will be created when you run the actual command. If everything goes Okay, you can then take this file and you’re going to have to copy onto the root of your hard drive.
We’re almost to the end here. The next thing we have to do is copy the Signature File to the root of the hard drive. We can find the file called Signature in the Import Export Web Services Tool Directory after you run the Java command.

The second to the last step is printing out the packing slip and filling it out. This is what the packing slip looks like. It’s a very simple document. You go ahead and put date, your e-mail account ID, your contact number, your name and phone number, the job ID, and the identifier that you have put for your device. Again you can find this document here off of the documentation.

And finally the last step is to simply pack your hard drive and ship it to Amazon. There are few little things that you have to take note of. Firstly, you need to include the power supply and any power cables and any interface cables, so if it’s USB 2.0, 3.0, esata, you need to include the USB cable or esata cable. If not, they’ll go ahead and return it back to you. You’ll also have to fill out that packing slip that I mentioned earlier and put that inside the box. And lastly, you’re going to send the package to the address that you had gotten from that create response command that we ran.

There are two other small things to note when you are shipping. Firstly, you’re going to make sure that the shipping label has that job ID on there. If not, they’re going to return it back. So you need to make sure you have the job ID in the shipping label. Secondly, you should also fill out a return shipping address. This is going to be different than the return shipping address that we have put in the manifest file. If they do not process your hard drive for some reason, if there’s a problem or something like that, they will return the hard drive to the shipping address on the shipping label. If they process your hard drive and they are able to transfer all of the data, they’ll return the hard drive to the shipping address that you have in the man of this about. So it’s important to put in of return shipping address on the label too. You can choose whatever carrier you’d like. I chose UPS. It’s good to have the tracking number, and they can go ahead and do all of this for you without a problem.

And that’s about it. It is a few steps and it does take a little bit of time the first time you do it. But after that, it’s pretty quick and it’s a great way to save a lot of data to the Cloud, an Amazon is also cheap for storage. So if you have a ton a day to you need to store, and you want to back it up somewhere other than in your house or on your external hard drive, then Amazon Web Services S3 is a great option.

I hope you enjoyed this tutorial Online Tech Tips. Please come back and visit.

More posts from the Computer Tips Category

How to Resize a Video or Change Resolution

3 Free Rosetta Stone Alternatives

How to Edit and Fix MP3 Tags

Free Up Disk Space using TreeSize

Popular Posts This Month

How to Resize a Video or Change Resolution

clip-image002-thumb.jpg

Copy Large Files Over the Network Faster with TeraCopy

firefox home pages

How to Setup Multiple Home Pages in Your Web Browser

How to Read and Open .DAT Files in Windows

Leave a Reply