{"id":324,"date":"2017-12-18T23:00:54","date_gmt":"2017-12-19T01:00:54","guid":{"rendered":"http:\/\/megumijr.com\/blog\/?p=324"},"modified":"2017-12-27T12:41:08","modified_gmt":"2017-12-27T14:41:08","slug":"affordable-disaster-recovery-with-aws","status":"publish","type":"post","link":"https:\/\/megumijr.com\/blog\/2017\/12\/affordable-disaster-recovery-with-aws\/","title":{"rendered":"Affordable Disaster Recovery with AWS"},"content":{"rendered":"

MUCH MORE OF THE SAME<\/h2>\n

I\u2019ve been working with IT for more than 30 years. When I started around the 80\u2019s, it was not even called \u201cIT\u201d. Actually, my first contact with this universe was around the 70\u2019s when I was fourteen and I took my first Cobol programing language course. At that time, computers were restricted to big corporations, universities and government and my course was entirely on \u201ccoding forms\u201d (a form with the same amount of columns of the punched cards).<\/p>\n

\"\"<\/p>\n

I started to work with computers before the PCs were commercially available, so I had opportunity to work with the first personal computers (TRS-80, Apple II and ZX Spectrum), microcomputers, minicomputers, mainframes and I must admit I was getting \u201cbored\u201d with IT lately. Although it has evolved a lot over time, I\u2019ve not seen anything really new for a long time.<\/p>\n

Take a look on the storages<\/em>, for example: they increased a lot on capacity and availability but is \u201cconceptually\u201d similar to an external hard disk and makes me remember those huge \u201cwardrobes<\/em>\u201d used on the mainframes.<\/p>\n

\"\"<\/p>\n

The blue cabinet on the photo was a typical hard disk housing and the CPU used another similar cabinet (the tape units were really cool: you just had to put reel tape on one side, push a button and a vacuum system \u201csucks\u201d the tape tip to the other reel<\/em>).<\/p>\n

Talking about mainframes, the IBM’s VM-SP operating system I used to work on a 4381 mainframe on the 80\u2019s was already capable to provide several virtual machines running on a single CPU, so virtualization is not a new concept. It took more than 15 years (around 2008) for me to see the first VMWare version running on my desktop, which was very cool because I was able create test machines.<\/p>\n

During my career, I dedicated an expressive amount of time to study Information Security<\/em> and I always became concerned about Business Continuity Plan<\/em>. Even with a good backup policy, in case of a catastrophic disaster it would take several days and probably weeks to restore the IT services in most cases. Most of the companies are not prone to invest in a mirror site<\/em> with all the necessary infrastructure (air conditioning, fire prevention, data links, servers, routers, etc) that maybe will never be used. However, if you don\u2019t have a mirror site, where can you deploy the applications after the disaster? How much time will be required to configure new servers? Is there a written (and updated) procedure to configure all the applications and servers?<\/p>\n

Let’s face the reality: the majority of the companies does not have a feasible disaster recovery plan and actually rely on lucky to survive a catastrophic disaster<\/strong>.<\/p>\n

NEWS PERSPECTIVES AND FRESH AIR<\/h2>\n

I\u2019ve never had opportunity to work with Cloud Services and from occasional readings I had a misconception that it was all about Infrastructure as a Service<\/em> (IaaS). A few days ago, I started to look the Amazon Web Services (AWS) portfolio and\u2026 \u201cWhat!?? Serverless application!!?? No need to worry about hardware capacity!?? Instant server provisioning!!?? \u201cMachine learning\u201d available to ordinary developers!!?? Internet of Things ready to use platform???<\/span><\/em>\u201d<\/p>\n

I got really impressed and amazed with their huge \u201cecosystem\u201d and paradigm breaking services. My feelings says that we are in front of a new revolution on IT scenario and I got really excited to take a ride on this wave, so I decided to start playing around with EC2.<\/p>\n

I\u2019ve always read that AWS was a good solution for occasional high demanding computer power, but I always thought of contacting some vendor to ask for some virtual machines and wait to be provisioned, and bla, bla, bla\u2026 but it\u2019s not!!<\/strong> Once you have an account, you can deploy new virtual machines in seconds using web interface or you can do it programmatically<\/em>!!<\/p>\n

Let\u2019s say you have a nightly batch routine at your company and it is high demanding computer power process. The amount of data is growing and you\u2019re about to blow up the batch processing window. The usual approach would be considering buying new hardware, but you also have to consider the data growth for reasonable lifetime of the new hardware. To make decisions hard, once in a month there is the monthly processing batch with an even higher computer power demand. During daytime, however, the server stays almost idle.\u00a0So, you will invest a considerable amount of money in a new powerful hardware that will stay almost idle for most of the time?<\/p>\n

With AWS you could keep a sync copy of you database, programmatically deploy the servers before the nightly batch and decommission them right after the routine ending. This approach is very interesting because you only pay for the time of the batch routine run. You don\u2019t have to pay for idle servers<\/strong>!<\/p>\n

Let\u2019s consider the Disaster Recovery<\/em> mentioned earlier. You can setup (and test) a whole environment to support your applications, create snapshots<\/em> of servers and terminate their running instances. I case of disaster, you can bring the servers to life just running a setup script!<\/p>\n

Of course, you have to consider an update routine for the snapshots. Data can be kept sync either with database online replication or restoring from a backup stored at Glacier service, for example.<\/p>\n

You also have to take into account that it is definitively not a solution for legacy file based applications and may not be suitable for client-server applications due to network latency and bandwidth limitations.<\/p>\n

PROOF OF CONCEPT<\/h2>\n

To learn a bit of AWS, I decided to make this small proof of concept<\/em> to check how ease would be to deploy an environment programmatically. Using the web interface, I created a single LAMP server and saved as a machine image<\/em> (AMI). From that image, I could start new instances of this server but I created Launch Templates<\/em> for simple deployment.<\/p>\n

\"\"

Launch Templates<\/em><\/p><\/div>\n

I created three templates, one for each availability zone<\/em> using the same image.<\/p>\n

\"\"

Network Interfaces<\/em><\/p><\/div>\n

Each server template is tied to its availability zone<\/em> by the network interface used.<\/p>\n

The final configuration of this POC is just a small infrastructure with two LAMP servers behind a load balancer exposed to the Internet.<\/p>\n

Several different programing languages<\/a> can be used for AWS automation. I like to use Python on my scripts. It is powerful, flexible and natively available on most Linux distributions, although I have used Python for Windows<\/em><\/a> for these tests. AWS recommends BOTO3<\/a> library to provide high level API for Python on AWS management, but as far as I saw, it has a limited set of classes.<\/p>\n

To overcome BOTO3 limits, I decied to use Python\u2019s subprocess<\/em> module for shell command calls and make use of Command Line Interface<\/em><\/a> (CLI) provided by AWS. Subprocess<\/em> allows me to capture the AWS command’s output and it’s JSON format can be parsed to easily extract any information.<\/p>\n

The test script takes less than a minute to complete and show some data to access each server individually and the Load Balancer URL.<\/p>\n

\"\"

Script output<\/em><\/p><\/div>\n

To confirm that the solution is working fine, just copy the Load Balancer DNS<\/em> name and paste on the browser.<\/p>\n

\"\"

Load Balance URL<\/em><\/p><\/div>\n

 <\/p>\n

THE SCRIPT<\/h2>\n

If you wish to reproduce the test or create your own provisioning script, check the code below.<\/p>\n

import subprocess\r\nimport json\r\nimport sys\r\n\r\ndef main():\r\n   #Launch Template TestServer_a (lt-049c3074a32d98ce6)\r\n   templateId = \"lt-049c3074a32d98ce6\"\r\n   print(\"Launching template: {}\".format(templateId))\r\n   instId1 = runInstance(templateId)\r\n\r\n   #Launch Template TestServer_b (lt-04d403f7cc4f71698)\r\n   templateId = \"lt-04d403f7cc4f71698\"\r\n   print(\"Launching template: {}\".format(templateId))\r\n   instId2 = runInstance(templateId)\r\n\r\n   #Get 1st Instance Data\r\n   print(\"Get instance data for: {}\".format(instId1))\r\n   DNSName1, subnetId1, vpcId = getInstanceData(instId1)\r\n   print(\"VPC Id: {} - Subnet Id: {} - DNS Name: {}\".format(vpcId, subnetId1, DNSName1))\r\n   print(\"\")\r\n\r\n   #Get 2nd Instance Data\r\n   print(\"Get instance data for: {}\".format(instId2))\r\n   DNSName2, subnetId2, vpcId = getInstanceData(instId2)\r\n   print(\"VPC Id: {} - Subnet Id: {} - DNS Name: {}\".format(vpcId, subnetId2, DNSName2))\r\n   print(\"\")\r\n \r\n   #Create Load Balancer\r\n   print(\"Creating Load Balancer...\")\r\n   lbDnsName = createLB(instId1, instId2, subnetId1, subnetId2, vpcId)\r\n   print(\"Load Balancer DNS: {}\".format(lbDnsName))\r\n   print(\"\")\r\n\r\n#end main\r\n\r\ndef runInstance(templateId):\r\n   #Start Instance\r\n   cmd = \"aws ec2 run-instances --launch-template LaunchTemplateId=\"+templateId\r\n   js = runCmd(cmd)\r\n\r\n   return js[\"Instances\"][0]['InstanceId']\r\n#End runInstance\r\n\r\ndef getInstanceData(instId):\r\n   #Wait instance to be available\r\n   cmd = \"aws ec2 wait instance-running --instance-ids \"+instId\r\n   ret=subprocess.call(cmd) # there's no output to get\r\n   if ret != 0:\r\n      sys.exit(ret)\r\n\r\n   #Get instance data\r\n   cmd = \"aws ec2 describe-instances --instance-ids \"+instId\r\n   js = runCmd(cmd)\r\n   DNSName = js[\"Reservations\"][0][\"Instances\"][0]['PublicDnsName']\r\n   subnetId = js[\"Reservations\"][0][\"Instances\"][0]['SubnetId']\r\n   vpcId = js[\"Reservations\"][0][\"Instances\"][0]['VpcId']\r\n\r\n   return DNSName, subnetId, vpcId\r\n#End getInstanceData\r\n\r\ndef createLB(instId1, instId2, subnetId1, subnetId2, vpcId):\r\n   #Create Load Balancer\r\n   cmd = \"aws elbv2 create-load-balancer --name test-lb --type network --subnets \" + \\\r\n   subnetId1 + \" \" + subnetId2\r\n   js = runCmd(cmd)\r\n   lbDnsName = js[\"LoadBalancers\"][0][\"DNSName\"]\r\n   lbArn = js[\"LoadBalancers\"][0][\"LoadBalancerArn\"]\r\n \r\n   #Create target group\r\n   cmd = \"aws elbv2 create-target-group --name test-targets --protocol TCP --port 80 --vpc-id \"+vpcId\r\n   js = runCmd(cmd)\r\n   targetArn = js[\"TargetGroups\"][0][\"TargetGroupArn\"]\r\n\r\n   #Register tags\r\n   cmd = \"aws elbv2 register-targets --target-group-arn \"+ \\\r\n   targetArn+\" --targets Id=\"+instId1+\" Id=\"+instId2\r\n   ret=subprocess.call(cmd) # there's no output to get\r\n   if ret != 0:\r\n      sys.exit(ret)\r\n \r\n   #Create listener\r\n   cmd = \"aws elbv2 create-listener --load-balancer-arn \"+ \\\r\n   lbArn+\" --protocol TCP --port 80 --default-actions Type=forward,TargetGroupArn=\"+targetArn\r\n   js = runCmd(cmd)\r\n \r\n   return lbDnsName\r\n#End createLB\r\n\r\ndef runCmd(cmd):\r\n   try:\r\n      output = subprocess.check_output(cmd)\r\n   except:\r\n      raise\r\n   return json.loads(output)\r\n#End runCmd\r\n\r\nif __name__ == \"__main__\":\r\n   main()<\/pre>\n
 <\/p>\n
CONCLUSION<\/h2>\n
AWS has an extensive set of solutions to provide a flexible infrastructure to your company. I did not dive into the economic aspects but I\u2019m pretty sure that AWS costs are considerably lower than running on-premises<\/em> infrastructures if you consider the costs of hardware (servers, storage, network appliances, etc) and all the surrounding infrastructure (UPSs, air conditioning, fire prevention, physical security, power supply, data links, etc). Different availability zones is a plus providing high availability for unbeatable lower costs than building you own on-premises<\/em> mirror sites.<\/p>\n","protected":false},"excerpt":{"rendered":"
MUCH MORE OF THE SAME I\u2019ve been working with IT for more than 30 years. When I started around the 80\u2019s, it was not even called \u201cIT\u201d. Actually, my first contact with this universe was around the 70\u2019s when I was fourteen and I took my first Cobol programing language \u2026 Continue lendo → <\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18,17],"tags":[23,21,20,22,19],"_links":{"self":[{"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/posts\/324"}],"collection":[{"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/comments?post=324"}],"version-history":[{"count":4,"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/posts\/324\/revisions"}],"predecessor-version":[{"id":336,"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/posts\/324\/revisions\/336"}],"wp:attachment":[{"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/media?parent=324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/categories?post=324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/megumijr.com\/blog\/wp-json\/wp\/v2\/tags?post=324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}