A prominent Amazon Web Services (AWS) customer has just sent me an unapologetic missive that doesn’t take ownership for service degradation. Read on.
This customer has a dependency on with its core line-of-business knowledge application. I’ve been told some of its features and functionality disappeared recently (a widget). While investigating “what the hell happened…,” as the client put it, an unsolicited excuse arrived from AWS that I have reprinted below. What concerns me is that Amazon doesn’t say “we’re sorry” and it hardly takes ownership of the malady. Here is the bottom line: Amazon has had a service failure, likely on a server farm, and it’s being less than helpful in assisting my client in restoring its LOB to full functionality. As I read it, the excuse looks as if lawyers run technical support at AWS. It deftly avoids responsibility and promotes its right to retire instances. Where is the redundancy (ummm…RAID or other approaches?)?
My advice to this loyal reader? Move to AZURE! I’m hearing more and more AWS horror stories. In the case of this reader, it should qualify for the generous BizSpark program at Microsoft whereby it can receive up to $60,000 of Azure credits as a start-up. And that’s not small potatoes, lemme tell you.
Computer guys are notorious for not reading (e.g. RTFM). The Amazon storyline below is juicy. As a special reward for reading, I have a secret below. Here is Amazon’s notification:
We have important news about your account. EC2 has detected degradation of the underlying hardware hosting one or more of your Amazon EC2 instances in the us-west-2 region. Due to this degradation, your instance(s) could already be unreachable. Running instances will be stopped or terminated after 12:00AM UTC on 2015-07-10. The affected instances are listed below:
You can see more information on your instances that are scheduled for retirement in the AWS Management Console (https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events)
How does this affect you?
* If your instance’s root device is an instance store volume, the instance will be terminated after the specified retirement date. When the instance is terminated, all data stored on the instance store will be deleted and can’t be recovered. Depending on the nature of the hardware degradation, you may be able to connect to your instance and migrate any data from instance store volumes. Unfortunately, in the case of disk failure the instance store data cannot be recovered.
* If your instance's root device is an EBS volume, the instance will be stopped after the retirement date, and you can start it again at any time.
In either case, since EC2 instance store volumes are physically attached to the host computer, any data on these volumes will be lost when the instance is stopped or terminated.
To check whether your instance’s root device is an instance store or EBS volume, go to the AWS Management console (https://console.aws.amazon.com/ec2/home?region=us-west-2#s=Instances), select your instance, and view the Root device type information in the details pane.
What do you need to do?
If you can still access the instance, we recommend the following:
* If your instance's root device is an instance store volume, launch a replacement instance from your most recent AMI and migrate all available data to the replacement instance, or migrate your data to an EBS volume, which you can back up with a snapshot.
* If your instance's root device is an EBS volume, you can replace the instance by creating an AMI of your instance, and launching a new instance from the AMI. For more information please see Amazon Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) in the EC2 User Guide. In case of difficulties stopping your EBS-backed instance, please see the Instance FAQ (http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
* If you've registered the instance in EC2 Classic with a load balancer, the load balancer will not be able to route traffic to your instance if you stop and start it, due to the new IP address associated to the instance. You must deregister the instance from the load balancer after stopping the instance, and then re-register it after starting the instance. In EC2 VPC, the IP address does not change, but the abovementioned deregister and register actions will speed up the time that it takes for the load balancer to recognize the instance. For more information, see De-Registering and Registering Amazon EC2 Instances (http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_DeReg_Reg_Instances.html) in the Elastic Load Balancing Developer Guide.
AWS may schedule instances for retirement in cases where there is an unrecoverable issue with the hardware on an underlying host. For more information about scheduled retirement events, please see Monitoring Scheduled Events in the EC2 user guide (http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html). To avoid single points of failure within critical applications, please refer to our architecture center for more information on implementing fault-tolerant architectures: http://aws.amazon.com/architecture.
If you have any questions or concerns, you can contact the AWS Support Team on the community forums and via AWS Premium Support at: http://aws.amazon.com/support
Amazon Web Services
As your reward for reading the above, I have a secret phone number that is extremely hard to get. It’s the Amazon general support telephone number: 866-216-1072. This is how you avoid e-mail hell configuring your Kindle or other Amazon products. Bon Appétit!