Category: 21_days_aws_using_terraform

Posted on

21 Days of AWS using Terraform – Day 13- Introduction to NAT Gateway using Terraform

Welcome to Day 13 of 21 Days of AWS using Terraform. The topic for today is Introduction to Nat Gateway using Terraform

What is NAT Gateway

NAT gateway enables instance in Private Subnet to connect to the internet or other AWS services but prevent the internet from initiating a connection with those instances.

How NAT works

  • NAT device has an Elastic IP address and is connected to the Internet through an internet gateway.
  • When we connect an instance in a private subnet through the NAT device, which routes traffic from the instance to the internet gateway and routes any response to the instance
  • NAT maps multiple private IPv4 addresses to a single public IPv4 address.

NAT gateway doesn’t support IPv6 traffic for that you need to use Egress only gateway.

NOTE: IPv6 traffic is separate from IPv4 traffic, route table must include separate routes for IPv6 traffic.

For more info

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-comparison.html

To create a NAT gateway

Go to VPC Dashboard → NAT Gateways → Create NAT gateways

  • Make sure you select the Public Subnet in your custom VPC
  • For NAT gateway to work, it needs Elastic IP

NOTE: NAT Gateway creation will take 10–15 min

  • Once the NAT gateway is available, add it to your default Route table

The advantage of NAT Gateway

  • NAT gateway is highly available but we need it per availability zone.
  • Can scale up to 45Gbps
  • Managed by AWS

Limitation of NAT Gateway

  • You can associate exactly one Elastic IP address with a NAT gateway. You cannot disassociate an Elastic IP address from a NAT gateway after it’s created. To use a different Elastic IP address for your NAT gateway, you must create a new NAT gateway with the required address, update your route tables, and then delete the existing NAT gateway if it’s no longer required.
  • You cannot associate a security group with a NAT gateway. You can use security groups for your instances in the private subnets to control the traffic to and from those instances.
  • You can use a network ACL to control the traffic to and from the subnet in which the NAT gateway is located. The network ACL applies to the NAT gateway’s traffic

On Day 2 we wrote the AWS VPC code, now its time to modify that code

21 Days of AWS using Terraform – Day 2- Building AWS VPC using Terraform
  • As we done via AWS UI, first step is to create Elastic IP
resource "aws_eip" "my-test-eip" {
  vpc = true
}

vpc – Boolean if the EIP is in a VPC or not.

  • Let’s create NAT gateway
resource "aws_nat_gateway" "my-test-nat-gateway" {
  allocation_id = "${aws_eip.my-test-eip.id}"
  subnet_id = "${aws_subnet.public_subnet.1.id}"
}
  • allocation_id – The Allocation ID of the Elastic IP address for the gateway.
  • subnet_id – The Subnet ID of the subnet in which to place the gateway.
  • Once the NAT gateway is available, add it to your default Route table
resource "aws_default_route_table" "private_route" {
  default_route_table_id = "${aws_vpc.main.default_route_table_id}"
  route {
    nat_gateway_id = "${aws_nat_gateway.my-test-nat-gateway.id}"
    cidr_block = "0.0.0.0/0"
  }

  tags = {
    Name = "my-private-route-table"
  }
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/vpc

Posted on

21 Days of AWS using Terraform – Day 12- Introduction to CloudTrail using Terraform

Welcome to Day 12 of 21 Days of AWS using Terraform. Topic for today is Introduction to CloudTrail using Terraform

What Is AWS CloudTrail?

AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.

  • It’s enabled when the account is created(for 7 days)
  • When activity occurs in your AWS account, that activity is recorded in a CloudTrail event.
  • Entries can be viewed in Event History(for 90 days)
  • Event logs can be aggregated across accounts and regions.

NOTE

  • Historically CloudTrail was not enabled by default
  • It won’t logs events like SSH/RDP only API call.
CloudTrail Dashboard(Most Recent Events)

Reference: https://aws.amazon.com/blogs/aws/new-aws-api-activity-lookup-in-cloudtrail/

For an ongoing record of activity and events in your AWS account, create a trail. It allows us to send logs to S3 bucket and can be single or multi-region.

To create a trail

Go to AWS Console --> Management & Governance --> CloudTrail --> Trails --> Create trail
* Trail name: Give your trail name
* Apply trail to all regions: You have an option to choose all regions or specific region.
* Read/Write events: You have the option to filter the events
* Data events: Data events provide insights into the resource operations performed on or within a resource
S3: You can record S3 object-level API activity (for example, GetObject and  PutObject) for individual buckets, or for all current and future buckets  in your AWS account
Lambda:You can record Invoke API operations for individual functions, or for all current and future functions in your AWS account.
* Storage Locations: Where to store the logs, we can create new bucket or use existing bucket
* Log file prefix: We have the option to provide prefix, this will make it easier to browse log file
* Encrypt log file with SSE-KMS:Default SSE-S3 Server side encryption(AES-256) or we can use KMS
* Enable log file validation: To determine whether a log file was modified, deleted, or unchanged after CloudTrail delivered it, you can use CloudTrail log file integrity validation
* Send SNS notification for every log file delivery:SNS notification of log file delivery allow us to take action immediately

NOTE:

  • There is always a delay between when the event occurs vs displayed on CloudTrail dashboard. On top of that, there is an additional delay when that log will be transferred to S3 bucket.
  • Delivered every 5(active) minutes with up to 15-minute delay
  • All CloudEvents are JSON structure you will see something like this when you try to view any event

Terraform Code

resource "aws_cloudtrail" "my-demo-cloudtrail" {
  name                          = "${var.cloudtrail_name}"
  s3_bucket_name                = "${aws_s3_bucket.s3_bucket_name.id}"
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true
}
  • name – Specifies the name of the trail.
  • s3_bucket_name – Specifies the name of the S3 bucket designated for publishing log files.
  • include_global_service_events – Specifies whether the trail is publishing events from global services such as IAM to the log files. Defaults to true.
  • is_multi_region_trail – Specifies whether the trail is created in the current region or in all regions. Defaults to false.
  • enable_log_file_validation – Specifies whether log file integrity validation is enabled. Defaults to false.

Then we have S3 bucket policy which allows us to send logs to S3 bucket and can be single or multi-region.

resource "aws_s3_bucket" "s3_bucket_name" {
  bucket = "${var.s3_bucket_name}"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
{
   "Sid": "AWSCloudTrailAclCheck",
   "Effect": "Allow",
   "Principal": {
      "Service": "cloudtrail.amazonaws.com"
},
 "Action": "s3:GetBucketAcl",
 "Resource": "arn:aws:s3:::s3-cloudtrail-bucket-with-terraform-code"
},
{
"Sid": "AWSCloudTrailWrite",
"Effect": "Allow",
"Principal": {
  "Service": "cloudtrail.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::s3-cloudtrail-bucket-with-terraform-code/*",
"Condition": {
  "StringEquals": {
     "s3:x-amz-acl": "bucket-owner-full-control"
  }
}
  }
  ]
  }
  POLICY
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/cloudtrail

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 11- Introduction to S3 using Terraform

Welcome to Day 11 of 21 Days of AWS using Terraform, topic for today is Introduction to S3 using terraform.

What is AWS S3?

AWS Simple Storage Service(S3) provides secure, durable and highly scalable object storage. S3 is easy to use and we can store and retrieve any amount of data from anywhere on the web.

Let’s create S3 bucket using terraform

provider "aws" {
  region = "us-west-2"
}

resource "aws_s3_bucket" "my-test-bucket" {
  bucket = "${var.s3_bucket_name}"
  acl = "private"
  versioning {
    enabled = true
  }
  tags =  {
      Name = "21-days-of-aws-using-terraform"
  }
}

For more info

https://www.terraform.io/docs/providers/aws/r/s3_bucket.html

bucket: name of the bucket, if we ommit that terraform will assign random bucket name
acl: Default to Private(other options public-read and public-read-write)
versioning: Versioning automatically keeps up with different versions of the same object.

NOTE: Every S3 bucket must be unique and that why random id is useful to prevent our bucket to collide with others.

To implement that let use random_id as providers

resource "random_id" "my-random-id" {
  byte_length = 2
}

byte_length – The number of random bytes to produce. The minimum value is 1, which produces eight bits of randomness.

https://www.terraform.io/docs/providers/random/r/id.html

Now to use that in your code

provider "aws" {
  region = "us-west-2"
}

resource "random_id" "my-random-id" {
  byte_length = 2
}

resource "aws_s3_bucket" "my-test-bucket" {
  bucket = "${var.s3_bucket_name}-${random_id.my-random-id.dec}"
  acl = "private"
  versioning {
    enabled = true
  }
  tags =  {
      Name = "21-days-of-aws-using-terraform"
  }
}

LifeCycle Management

Why do we need LifeCycle Management?

  • To save cost
  • In most of the cases, data which is generated by our application is relevant for us for the first 30 days and after that, we don’t access that data as frequently.

Reference:

https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html

LifeCycle object supports the following but I am going to enable just the required parameters

  • enabled — (Required) Specifies lifecycle rule status.
  • transition – (Optional) Specifies a period in the object’s transitions
resource "aws_s3_bucket" "my-test-bucket" {
  bucket = "${var.s3_bucket_name}-${random_id.my-random-id.dec}"
  acl = "private"
  versioning {
    enabled = true
  }
  lifecycle_rule {
    enabled = true
    transition {
      storage_class = "STANDARD_IA"
      days = 30
    }
  }
  tags =  {
      Name = "21-days-of-aws-using-terraform"
  }
}
  • Here I am defining after 30 days move the objects to STANDARD_IA and after 60 days to GLACIER.

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/s3

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 10- Introduction to IAM using Terraform

Welcome to Day 10 of 21 Days of AWS using Terraform, topic for today is Introduction to IAM using terraform.

What is IAM?

Identity and Access Management(IAM) is used to manage AWS

  • Users
  • Groups
  • Roles
  • Api Keys
  • IAM Access Policies

and it provide access/access-permissions to AWS resources(such as EC2,S3..)

If we notice at the right hand side at the top of console it says Global i.e creating a user/groups/roles will apply to all regions

To create a new user,Just click on Users on the left navbar

By default any new IAM account created with NO access to any AWS services(non-explicit deny)

Always follow the best practice and for daily work try to use a account with least privilege(i.e non root user)

IAM Policies: A policy is a document that formally states one or more permissions.For eg: IAM provides some pre-built policy templates to assign to users and groups

  • Administrator access: Full access to AWS resources
  • Power user access: Admin access except it doesn’t allow user/group management
  • Read only access: As name suggest user can only view AWS resources

Default policy is explicitly deny which will override any explicitly allow policy

Let take a look at these policies

AdministratorAccess

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

We can create our own custom policy using policy generator or written from scratch

So Custom Policy where everything denies for EC2 resources

{
 “Version”: “2012–10–17”,
 “Statement”: [
 {
 “Sid”: “Stmt1491718191000”,
 “Effect”: “Deny”,
 “Action”: [
 “ec2:*”
 ],
 “Resource”: [
 “*”
 ]
 }
 ]
}
  • More than one policy can be attached to a user or group at the same time
  • Policy cannot be directly attached to AWS resources(eg: EC2 instance)
  • There is a really nice tool https://policysim.aws.amazon.comwhich we can use to test and troubleshoot IAM and resource based policies

Below is the simulation I run where I created a test user who has only Amazon S3 read only access

Now let me run the simulation,as you can see it’s a nice way to test your policies

Let’s create IAM user using terraform

resource "aws_iam_user" "example" {
  name = "prashant"
}

AWS: aws_iam_user — Terraform by HashiCorp
Provides an IAM user.www.terraform.io

Now if I want to create two IAM user

One way to achieve the same is copy paste the same piece of code but that defeats the whole purpose of DRY.

Terraform provides meta parameters called count to achieve the same i.e to do certain types of loops.The count is a meta parameter which defines how many copies of the resource to create.

resource "aws_iam_user" "example" {
  count = 3
  name = "prashant"
}

One problem with this code is that all three IAM users would have the same name, which would cause an error since usernames must be unique

To accomplish the same thing in Terraform, you can use count.index to get the index of each “iteration” in the “loop”:

resource “aws_iam_user” “example” {
 count = 3
 name = “prashant.${count.index}”
}

If we run the plan again we will see terraform want to create three IAM user, each with a different name(prashant.0, prashant.1, prashant.2)

Think of a real-world scenario, we are hardly going to see names like prashant.0–2

The solution for this issue, If we combine count.index with interpolation functions built into Terraform, you can customize each “iteration” of the “loop” even more. To achieve this we need two interpolation functions length and elementInterpolation Syntax — Terraform by HashiCorp
Embedded within strings in Terraform, whether you’re using the Terraform syntax or JSON syntax, you can interpolate…www.terraform.io

element(list, index) — Returns a single element from a list at the given index. If the index is greater than the number of elements, this function will wrap using a standard mod algorithm. This function only works on flat lists.

OR

The element function returns the item located at INDEX in the given LIST

length(list) — Returns the number of members in a given list or map, or the number of characters in a given string.

OR

The length function returns the number of items in LIST (it also works with strings and maps)

#variables.tf
variable "username" {
  type = "list"
  default = ["prashant","pankaj","ashish"]
}
#main.tf
resource "aws_iam_user" "example" {
  count = "${length(var.username)}"
  name = "${element(var.username,count.index )}"
}

Now when you run the plan command, you’ll see that Terraform wants to create three IAM users, each with a unique name

One thing to note as we have used count on a resource, it becomes the list of resources rather than just a single resource.

For example, if you wanted to provide the Amazon Resource Name (ARN) of one of the IAM users as an output variable, you would need to do the following:

# outputs.tf
output “user_arn” {
 value = “${aws_iam_user.example.0.arn}”
}

If you want the ARNs of all the IAM users, you need to use the splat character, “*”, instead of the index:

# outputs.tf
output “user_arn” {
 value = “${aws_iam_user.example.*.arn}”
}

As we are done with creating IAM user, now let attach some policy with these users(As by default new user have no permission whatsoever)

IAM Policies are JSON documents used to describe permissions within AWS. This is used to grant access to your AWS users to particular AWS resources.

IAM Policy is a json document

Terraform provides a handy data source called the aws_iam_policy_document that gives you a more concise way to define the IAM policy

data "aws_iam_policy_document" "example" {
  statement {
    actions = [
      "ec2:Describe*"]
    resources = [
      "*"]
  }
}
  • IAM Policy consists of one or more statements
  • Each of which specifies an effect (either “Allow” or “Deny”)
  • One or more actions (e.g., “ec2:Describe*” allows all API calls to EC2 that start with the name “Describe”),
  • One or more resources (e.g., “*” means “all resources”)

To create a new IAM managed policy from this document, we need to use aws_iam_policy resource

resource “aws_iam_policy” “example” {
 name = “ec2-read-only”
 policy = “${data.aws_iam_policy_document.example.json}”
}

This code uses the count parameter to “loop” over each of your IAM users and the element interpolation function to select each user’s ARN from the list returned by aws_iam_user.example.*.arn.

resource “aws_iam_user_policy_attachment” “test-attach” {
 count = “${length(var.username)}”
 user = “${element(aws_iam_user.example.*.name,count.index )}”
 policy_arn = “${aws_iam_policy.example.arn}”
}

IAM Roles are used to granting the application access to AWS Services without using permanent credentials.

IAM Role is one of the safer ways to give permission to your EC2 instances.

We can attach roles to an EC2 instance, and that allows us to give permission to EC2 instance to use other AWS Services eg: S3 buckets

Motivation

  • Give EC2 instance access to S3 bucket

Step1

  • Create a file iam.tf
  • Create an IAM role by copy-paste the content of a below-mentioned link
  • assume_role_policy — (Required) The policy that grants an entity permission to assume the role.

AWS: aws_iam_role — Terraform by HashiCorp
Provides an IAM role.www.terraform.io

resource "aws_iam_role" "test_role" {
  name = "test_role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
  tags = {
      tag-key = "tag-value"
  }
}
  • This is going to create IAM role but we can’t link this role to AWS instance and for that, we need EC2 instance Profile

Step2

  • Create EC2 Instance Profile
resource "aws_iam_instance_profile" "test_profile" {
  name = "test_profile"
  role = "${aws_iam_role.test_role.name}"
}

AWS: aws_iam_instance_profile — Terraform by HashiCorp
Provides an IAM instance profile.www.terraform.io

  • Now if we execute the above code, we have Role and Instance Profile but with no permission.
  • Next step is to add IAM Policies which allows EC2 instance to execute specific commands for eg: access to S3 Bucket

Step3

  • Adding IAM Policies
  • To give full access to S3 bucket
resource "aws_iam_role_policy" "test_policy" {
  name = "test_policy"
  role = "${aws_iam_role.test_role.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

Step4

  • Attach this role to EC2 instance
resource "aws_instance" "role-test" {
  ami = "ami-0bbe6b35405ecebdb"
  instance_type = "t2.micro"
  iam_instance_profile = "${aws_iam_instance_profile.test_profile.name}"
  key_name = "mytestpubkey"
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/iam

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 9- Introduction to Route53 using Terraform

Welcome to Day 9 of 21 Days of AWS using Terraform, topic for today is Introduction to Route53 using terraform.

What is AWS Route53?

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. You can use Route 53 to perform three main functions in any combination:

  • Domain Registration
  • DNS Routing
  • Health Checking

Key DNS Terms

  • A Record

A record is used to translate human-friendly domain names such as “www.example.com” into IP-addresses such as 192.168.0.1 (machine friendly numbers).

  • CNAME Record

A Canonical Name record (abbreviated as CNAME record) is a type of resource record in the Domain Name System (DNS) which maps one domain name (an alias) to another (the Canonical Name.)

  • NameServer Record

NS-records identify the DNS servers responsible (authoritative) for a zone.

Amazon Route 53 automatically creates a name server (NS) record that has the same name as your hosted zone. It lists the four name servers that are the authoritative name servers for your hosted zone. Do not add, change, or delete name servers in this record.

ns-2048.awsdns-64.com
ns-2049.awsdns-65.net
ns-2050.awsdns-66.org
ns-2051.awsdns-67.co.uk
  • SOA Record

A Start of Authority record (abbreviated as SOA record) is a type of resource record in the Domain Name System (DNS) containing administrative information about the zone, especially regarding zone transfers

AWS Specific DNS Terms

  • Alias Record

Amazon Route 53 alias records provide a Route 53–specific extension to DNS functionality. Alias records let you route traffic to selected AWS resources, such as CloudFront distributions and Amazon S3 buckets. They also let you route traffic from one record in a hosted zone to another record.

Unlike a CNAME record, you can create an alias record at the top node of a DNS namespace, also known as the zone apex. For example, if you register the DNS name example.com, the zone apex is example.com. You can’t create a CNAME record for example.com, but you can create an alias record for example.com that routes traffic to www.example.com.

  • AWS Route53 Health Check

Amazon Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources. Each health check that you create can monitor one of the following:

  • The health of a specified resource, such as a web server
  • The status of other health checks
  • The status of an Amazon CloudWatch alarm

Step1: Create a hosted zone

Go to https://console.aws.amazon.com/route53 --> Hosted zones --> Create Hosted Zone
* Domain Name: You must need to purchase this domain either from your Domain Registrar or you can purchase from Amazon
* Comment: Add some comment
* Type: Public Hosted Zone(if purchased by a domain registrar) OR you can set Private Hosted Zone for Amazon VPC
  • The moment you create a hosted zone four NS and SOA record will be created for you.

NOTE: Please don’t change or alter these records.

Step2: Create A record

* Name : www
* Type: A-IPv4 address
* Alias: No
* TTL: Select +1m(60second)
* Value: Public IP of your EC2 instance
* Routing Policy: Simple
  • You will see the record like this

Choosing a Routing Policy

  • When you create a record, you choose a routing policy, which determines how Amazon Route 53 responds to queries:

Simple Routing

Simple Routing Policy use for a single resource that performs a given function for your domain, for example, a web server that serves content for the example.com website.

If you choose the simple routing policy in the Route 53 console, you can’t create multiple records that have the same name and type, but you can specify multiple values in the same record, such as multiple IP addresses. If you specify multiple values in a record, Route 53 returns all values to the recursive resolver in random order, and the resolver returns the values to the client (such as a web browser) that submitted the DNS query. The client then chooses a value and resubmits the query.

  • As you can see in the above case we are using Simple Routing Policy.
  • Let’s automate this with the help of terraform code

resource "aws_route53_zone" "my-test-zone" {
  name = "example.com"
  vpc {
    vpc_id = "${var.vpc_id}"
  }
}

name – This is the name of the hosted zone.

vpc – Configuration block(s) specifying VPC(s) to associate with a private hosted zone.

resource "aws_route53_record" "my-example-record" {
  count = "${length(var.hostname)}"
  name = "${element(var.hostname,count.index )}"
  records = ["${element(var.arecord,count.index )}"]
  zone_id = "${aws_route53_zone.my-test-zone.id}"
  type = "A"
  ttl = "300"
}
  • zone_id – The ID of the hosted zone to contain this record.
  • name – The name of the record.
  • type – The record type. Valid values are A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SOA, SPF, SRV and TXT.
  • ttl – The TTL of the record.
  • records – (Required for non-alias records) A string list of records
  • Here I am using element function
element retrieves a single element from a list.

element(list, index)
The index is zero-based. This function produces an error if used with an empty list.

Use the built-in index syntax list[index] in most cases. Use this function only for the special additional "wrap-around" behavior described below.

For more info

https://www.terraform.io/docs/configuration/functions/element.html

variables.tf

variable "vpc_id" {}
variable "hostname" {
  type = "list"
}
variable "arecord" {
  type = "list"
}

Final Route53 terraform module look like this

module "route53" {
  source   = "./route53"
  hostname = ["test1", "test2"]
  arecord  = ["10.0.1.11", "10.0.1.12"]
  vpc_id   = "${module.vpc.vpc_id}"
}

GitHub link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/route53

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 8- Introduction to AWS RDS MySQL using Terraform

Welcome to Day 8 of 21 Days of AWS using Terraform. The topic for today is Introduction to AWS RDS MySQL using Terraform. With that we will be able to finish the last part of our two-tier architecture i.e MySQL DB is private subnets.

What is AWS RDS?

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks.

To create a database, go to Database section(AWS Console) and click on RDS

Click on Get Started Now

Then Select MySQL

On the next screen, choose Dev/Test -MySQL(or depend upon your requirement, as my use case is only for testing purpose)

On the next screen provide all the info

As this is for testing Purpose

  • DB instance class(db.t2.micro)
  • Skip MultiAZ deployment for the time being
  • Gave all the info like(DB instance identifier, Master username, Master password)

Fill all the details in the next screen

Mainly you need to fill

  • Database name(Don’t confuse it DB instance identifier)
  • Backup retention period(0 days, for the time being)

Then click on Launch DB instance

Wait for few mins 5–10min(or depend upon your instance type and size) and check Instance Status(It should be available)

Now lets try to create Read Replica out of this database

Ohho no Create read replica option is not highlighted for me and the reason for that

  • We don’t have a snapshot
  • We don’t have an automated backups

Read replica is always created from a snapshot or the latest backup

Let’s take a snapshot of this database

Once the snapshot creation is done, let’s try to convert this into multi-AZ. Go to Instance actions and click on Modify

These are the things you need to modify

  • Multi-AZ set to Yes
  • Under settings you need to enter the password again
  • I am enabling backup and set it to 1 day
  • On the final screen, you have the option

1: Apply during the next scheduled maintenance window

2: Apply immediately(This will cause a downtime)

To restore a database from the snapshot

and then on the next screen, give DB Instance Identifier or any other setting you want to modify while restoring

To Verify if Multi-AZ is enabled, Click on the particular DB

Now let’s try to create read-replica again, as you can see Create read replica tab is now enabled

The Important thing to remember we can create read replica in any other region

Under the Settings tab, give it a unique name

Now whatever we have done manually here, let’s try to terraformed it

provider "aws" {
  region = "us-west-2"
}


resource "aws_db_instance" "my-test-sql" {
  instance_class = "${var.db_instance}"
  engine = "mysql"
  engine_version = "5.7"
  multi_az = true
  storage_type = "gp2"
  allocated_storage = 20
  name = "mytestrds"
  username = "admin"
  password = "admin123"
  apply_immediately = "true"
  backup_retention_period = 10
  backup_window = "09:46-10:16"
  db_subnet_group_name = "${aws_db_subnet_group.my-rds-db-subnet.name}"
  vpc_security_group_ids = ["${aws_security_group.my-rds-sg.id}"]
}

resource "aws_db_subnet_group" "my-rds-db-subnet" {
  name = "my-rds-db-subnet"
  subnet_ids = ["${var.rds_subnet1}","${var.rds_subnet2}"]
}

resource "aws_security_group" "my-rds-sg" {
  name = "my-rds-sg"
  vpc_id = "${var.vpc_id}"
}

resource "aws_security_group_rule" "my-rds-sg-rule" {
  from_port = 3306
  protocol = "tcp"
  security_group_id = "${aws_security_group.my-rds-sg.id}"
  to_port = 3306
  type = "ingress"
  cidr_blocks = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "outbound_rule" {
  from_port = 0
  protocol = "-1"
  security_group_id = "${aws_security_group.my-rds-sg.id}"
  to_port = 0
  type = "egress"
  cidr_blocks = ["0.0.0.0/0"]
}
* allocated_storage: This is the amount in GB
* storage_type: Type of storage we want to allocate(options avilable "standard" (magnetic), "gp2" (general purpose SSD), or "io1" (provisioned IOPS SSD)
* engine: Database engine(for supported values check https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBInstance.html) eg: Oracle, Amazon Aurora,Postgres 
* engine_version: engine version to use
* instance_class: instance type for rds instance
* name: The name of the database to create when the DB instance is created.
* username: Username for the master DB user.
* password: Password for the master DB user
* db_subnet_group_name:  DB instance will be created in the VPC associated with the DB subnet group. If unspecified, will be created in the default VPC
* vpc_security_group_ids: List of VPC security groups to associate.
* allows_major_version_upgrade: Indicates that major version upgrades are allowed. Changing this parameter does not result in an outage and the change is asynchronously applied as soon as possible.
* auto_minor_version_upgrade:Indicates that minor engine upgrades will be applied automatically to the DB instance during the maintenance window. Defaults to true.
* backup_retention_period: The days to retain backups for. Must be between 0 and 35. When creating a Read Replica the value must be greater than 0
* backup_window:The daily time range (in UTC) during which automated backups are created if they are enabled. Must not overlap with maintenance_window
* maintainence_window:The window to perform maintenance in. Syntax: "ddd:hh24:mi-ddd:hh24:mi".
* multi_az: Specifies if the RDS instance is multi-AZ
* skip_final_snapshot: Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted, using the value from final_snapshot_identifier. Default is false

NOTE: Here we are storing mysql password in plan text, I will come up the blog shortly how to store this password in encrypted format.

variables.tf

variable "rds_subnet1" {}
variable "rds_subnet2" {}
variable "db_instance" {}
variable "vpc_id" {}
  • We need to do some changes in order to make this work, by outputting the value of private subnet in terraform vpc module which will act an input to rds aws_db_subnet_group
output "private_subnet1" {
  value = "${element(aws_subnet.private_subnet.*.id, 1 )}"
}

output "private_subnet2" {
  value = "${element(aws_subnet.private_subnet.*.id, 2 )}"
}
  • RDS Mysql Module will look like this
module "rds" {
  source      = "./rds"
  db_instance = "db.t2.micro"
  rds_subnet1 = "${module.vpc.private_subnet1}"
  rds_subnet2 = "${module.vpc.private_subnet2}"
  vpc_id      = "${module.vpc.vpc_id}"
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/rds

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 7- Introduction to CloudWatch using Terraform

Welcome to Day 7 of 21 Days of AWS using Terraform. Topic for today is Introduction to CloudWatch using terraform.

What is CloudWatch?

AWS CloudWatch is a monitoring service to monitor AWS resources, as well as the applications that run on AWS.

As per official documentation

Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real time. You can use CloudWatch to collect and track metrics, which are variables you can measure for your resources and applications.

Reference

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html

EC2/Host Level Metrics that CloudWatch monitors by default consist of

  • CPU
  • Network
  • Disk
  • Status Check

There are two types of status check

  • System status check: Monitor the AWS System on which your instance runs. It either requires AWS involvement to repair or you can fix it by yourself by just stop/start the instance(in case of EBS volumes).Examples of problems that can cause system status checks to fail
* Loss of network connectivity
* Loss of system power
* Software issues on the physical host
* Hardware issues on the physical host that impact network reachability
  • Instance status check: Monitor the software and network configuration of an individual instance. It checks/detects problems that require your involvement to repair.
* Incorrect networking or startup configuration
* Exhausted memory
* Corrupted filesystem
* Incompatible kernel

NOTE

  • Memory/RAM utilization is custom metrics.
  • By default, EC2 monitoring is 5 minutes intervals but we can always enable detailed monitoring(1 minutes interval, but that will cost you some extra $$$)

Reference

https://aws.amazon.com/cloudwatch/pricing/

P.S: CloudWatch can be used on premise too. We just need to install the SSM(System Manager) and CloudWatch agent.

Enough of the theoretical concept, let setup first CloudWatch alarm

  • Scenario1: We want to create a CloudWatch alarm that sends an email using SNS notification when CPU Utilization is more than 70%
  • Setup a CPU Usage Alarm using the AWS Management Console
  • Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  • In the navigation pane, choose Alarms, Create Alarm.
  • Go to Metric → Select metric → EC2 → Per-Instance-Metrics → CPU Utilization → Select metric
Define the Alarm as follows
* Type the unique name for the alarm(eg: HighCPUUtilizationAlarm)
* Description of the alarm
* Under whenever,choose >= and type 70, for type 2. This specify that the alarm is triggered if the CPU usage is above 70% for two consecutive sampling period
* Under Additional settings, for treat missing data as, choose bad(breaching threshold), as missing data points may indicate that the instance is down
* Under Actions, for whenever this alarm, choose state is alarm. For Send notification to, select an exisiting SNS topic or create a new one 
* To create a new SNS topic, choose new list, for send notification to, type a name of SNS topic(for eg: HighCPUUtilizationThreshold) and for Email list type a comma-seperated list of email addresses to be notified when the alarm changes to the ALARM state.
* Each email address is sent to a topic subscription confirmation email. You must confirm the subscription before notifications can be sent.
* Click on Create Alarm

Scenario2: Create a status check alarm to notify when an instance has failed a status check

Creating a Status Check Alarm Using the AWS Console

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  2. In the navigation pane, choose Instances.
  3. Select the instance, choose the Status Checks tab, and choose to Create Status Check Alarm.
* You can create new SNS notification or use the exisiting one(I am using the existing one create in earlier example of high CPU utilization)
* In Whenever, select the status check that you want to be notified about(options Status Check Failed(Any), Status Check Failed(Instance) and Status Check Failed(System)
* In For at least, set the number of periods you want to evaluate and in consecutive periods, select the evaluation period duration before triggering the alarm and sending an email.
* In Name of alarm, replace the default name with another name for the alarm.
* Choose Create Alarm.
  • Let’s write the terraform code
resource "aws_cloudwatch_metric_alarm" "cpu-utilization" {
  alarm_name                = "high-cpu-utilization-alarm"
  comparison_operator       = "GreaterThanOrEqualToThreshold"
  evaluation_periods        = "2"
  metric_name               = "CPUUtilization"
  namespace                 = "AWS/EC2"
  period                    = "120"
  statistic                 = "Average"
  threshold                 = "80"
  alarm_description         = "This metric monitors ec2 cpu utilization"
  alarm_actions             = [ "${var.sns_topic}" ]
dimensions = {
    InstanceId = "${var.instance_id}"
  }
}


resource "aws_cloudwatch_metric_alarm" "instance-health-check" {
  alarm_name                = "instance-health-check"
  comparison_operator       = "GreaterThanOrEqualToThreshold"
  evaluation_periods        = "1"
  metric_name               = "StatusCheckFailed"
  namespace                 = "AWS/EC2"
  period                    = "120"
  statistic                 = "Average"
  threshold                 = "1"
  alarm_description         = "This metric monitors ec2 health status"
  alarm_actions             = [ "${var.sns_topic}" ]
dimensions = {
    InstanceId = "${var.instance_id}"
  }
}
  • alarm_name – The descriptive name for the alarm. This name must be unique within the user’s AWS account
  • comparison_operator – The arithmetic operation to use when comparing the specified Statistic and Threshold. The specified Statistic value is used as the first operand. Either of the following is supported: GreaterThanOrEqualToThreshold, GreaterThanThreshold, LessThanThreshold, LessThanOrEqualToThreshold.
  • evaluation_periods – The number of periods over which data is compared to the specified threshold.
  • metric_name – The name for the alarm’s associated metric.
  • namespace – The namespace for the alarm’s associated metric.
  • period – The period in seconds over which the specified statistic is applied.
  • statistic – The statistic to apply to the alarm’s associated metric. Either of the following is supported: SampleCount, Average, Sum, Minimum, Maximum
  • threshold – The value against which the specified statistic is compared.
  • alarm_actions – The list of actions to execute when this alarm transitions into an ALARM state from any other state. Each action is specified as an Amazon Resource Name (ARN).
  • dimensions – The dimensions for the alarm’s associated metric. 
  • We need to modify our SNS module a little bit where the output of SNS arn will act as a input to this cloudwatch module.
output "sns_arn" {
  value = "${aws_sns_topic.my-test-alarm.arn}"
}
  • Same way output of EC2 module will act as an input to cloudwatch module
output "instance_id" {
  value = "${element(aws_instance.my-test-instance.*.id, 1)}"
}
  • There is a bug in terraform code, where I can’t specify multiple instances
output "instance_id" {
  value = "${aws_instance.my-test-instance.*.id)}"
}
  • If we try to use the above it’s will fail due to below error
Error: Incorrect attribute value type

  on cloudwatch/main.tf line 12, in resource "aws_cloudwatch_metric_alarm" "cpu-utilization":
  12: dimensions = {

Inappropriate value for attribute "dimensions": element "InstanceId": string
required.


Error: Incorrect attribute value type

  on cloudwatch/main.tf line 29, in resource "aws_cloudwatch_metric_alarm" "instance-health-check":
  29: dimensions = {

Inappropriate value for attribute "dimensions": element "InstanceId": string
required.
  • Final CloudWatch terraform code
module "cloudwatch" {
  source      = "./cloudwatch"
  sns_topic   = "${module.sns_topic.sns_arn}"
  instance_id = "${module.ec2.instance_id}"
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/cloudwatch

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 6- Introduction to Simple Notification Service(SNS) using Terraform

Welcome to Day 6 of 21 Days of AWS using Terraform. Topic for today is Introduction to Simple Notification to Service(SNS) using terraform.

What is SNS?

As per official documentation

AWS SNS is a web service that coordinates and manages the delivery or sending of messages to subscribing endpoints or clients.

In case of cloudwatch(high CPU utilization or System/Instance Status Check) when the certain event occurs and SNS is used to send a notification. CloudWatch in combination with SNS creates a full monitoring solution with notifies the administrator in case of any environment issue(high CPU, Downtime…).

https://docs.aws.amazon.com/sns/latest/dg/welcome.html

SNS has three major components

Publisher

  • The entity that triggers the sending of a message(eg: CloudWatch Alarm, Any application or S3 events)

Topic

  • Object to which you publish your message(≤256KB)
  • Subscriber subscribe to the topic to receive the message
  • Soft limit of 10 million subscribers

Subscriber

An endpoint to a message is sent. Message are simultaneously pushed to the subscriber

  • As you can see it follows the publish-subscribe(pub-sub) messaging paradigm with notification being delivered to the client using a push mechanism that eliminates the need to periodically check or poll for new information and updates.
  • To prevent the message from being lost, all messages published to Amazon SNS are stored redundantly across multiple Availability Zones.

Enough of theory, let see SNS in action

Using the AWS Console

Step 1: Create a topic

  • Create a topic

Step2: Subscribe to a Topic

  • Choose to Create a subscription.
  • The Create Subscription dialog box appears.
  • Go to your email and confirm subscription

Step3: Publish to the topic

  • Choose the Publish to the topic button.
  • The Publish a Message page appears.

Let’s check the terraform code

resource "aws_sns_topic" "my-test-alarm" {
  name = "my-test-alarms-topic"

  delivery_policy = <<EOF
{
  "http": {
    "defaultHealthyRetryPolicy": {
      "minDelayTarget": 20,
      "maxDelayTarget": 20,
      "numRetries": 3,
      "numMaxDelayRetries": 0,
      "numNoDelayRetries": 0,
      "numMinDelayRetries": 0,
      "backoffFunction": "linear"
    },
    "disableSubscriptionOverrides": false,
    "defaultThrottlePolicy": {
      "maxReceivesPerSecond": 1
    }
  }
}
EOF

  provisioner "local-exec" {
    command = "aws sns subscribe --topic-arn ${self.arn} --protocol email --notification-endpoint ${var.alarms_email}"
  }
}
  • name – The friendly name for the SNS topic. By default generated by Terraform.

variables.tf 

variable "alarms_email" {}

outputs.tf

output "sns_arn" {
  value = "${aws_sns_topic.my-test-alarm.arn}"
}

Final SNS Module will look like this

module "sns_topic" {
  source       = "./sns"
  alarms_email = "[email protected]"
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/sns

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 5- AWS Auto-Scaling Group using Terraform

Welcome to Day 5 of 21 Days of AWS using Terraform,  So far we build VPC ,EC2 and Application Load Balancer, let’s build Auto Scaling Group and rather then creating an instance via EC2 module, let auto-scaling group take care of it based on load.

What is Auto Scaling?

What auto-scaling will do, it ensures that we have a correct number of EC2 instances to handle the load of your applications.

How Auto Scaling works

  • It all started with the creation of the Auto Scaling group which is the collection of EC2 instances.
  • You can specify a minimum number of instances and AWS EC2 Auto Scaling ensures that your group never goes below this size.
  • The same way we can specify the maximum number of instances and AWS EC2 Auto Scaling ensures that your group never goes above this size.
  • If we specify the desired capacity, AWS EC2 Auto Scaling ensures that your group has this many instances.
  • Configuration templates(launch template or launch configuration): Specify Information such as AMI ID, instance type, key pair, security group
  • If we specify scaling policies then AWS EC2 Auto Scaling can launch or terminate instances as demand on your application increased or decreased. For eg: We can configure a group to scale based on the occurrence of specified conditions(dynamic scaling) or on a schedule.

Reference:https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html

In the above example, Auto Scaling has

  • A minimum size of one instance.
  • Desired Capacity of two instances.
  • The maximum size of four instances.
  • Scaling policies we define adjust the minimum or a maximum number of instances based on the criteria we specify.

Step1: The first step in creating the AutoScaling Group is to create a launch configuration, which specifies how to configure each EC2 instance in the autoscaling group.

  resource "aws_launch_configuration" "my-test-launch-config" {
  image_id        = "ami-01ed306a12b7d1c96"
  instance_type   = "t2.micro"
  security_groups = ["${aws_security_group.my-asg-sg.id}"]

  user_data = <<-EOF
              #!/bin/bash
              yum -y install httpd
              echo "Hello, from Terraform" > /var/www/html/index.html
              service httpd start
              chkconfig httpd on
              EOF

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_security_group" "my-asg-sg" {
  name   = "my-asg-sg"
  vpc_id = "${var.vpc_id}"
}

resource "aws_security_group_rule" "inbound_ssh" {
  from_port         = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-asg-sg.id}"
  to_port           = 22
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "inbound_http" {
  from_port         = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-asg-sg.id}"
  to_port           = 80
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "outbound_all" {
  from_port         = 0
  protocol          = "-1"
  security_group_id = "${aws_security_group.my-asg-sg.id}"
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}
  • Most of the parameters look similar to EC2 configuration except lifecycle parameter which is required for using a launch configuration with an ASG
21 Days of AWS using Terraform – Day 3- Creating AWS EC2 Instance using Terraform
  • One of the available lifecycle settings are create_before_destroy, which, if set to true, tells Terraform to always create a replacement resource before destroying the original resource. For example, if you set create_before_destroy to true on an EC2 Instance, then whenever you make a change to that Instance, Terraform will first create a new EC2 Instance, wait for it to come up, and then remove the old EC2 Instance.
  • The catch with the create_before_destroy the parameter is that if you set it to true on resource X, you also have to set it to true on every resource that X depends on (if you forget, you’ll get errors about cyclical dependencies).

https://www.terraform.io/docs/configuration/resources.html#lifecycle-lifecycle-customizations

Step2: Define auto-scaling group

data "aws_availability_zone" "available" {}

resource "aws_autoscaling_group" "example" {
  launch_configuration = "${aws_launch_configuration.my-test-launch-config.name}"
  vpc_zone_identifier  = "${var.subnet_id}"
  availability_zones = ["${data.aws_availability_zone.available.name}"]

  target_group_arns = ["${var.target_group_arn}"]
  health_check_type = "ELB"

  min_size = 2
  max_size = 10

  tag {
    key                 = "Name"
    value               = "my-test-asg"
    propagate_at_launch = true
  }
}
  • Next step is to create an auto-scaling group using the aws_autoscaling_group resource.
  • This autoscaling group will spin a minimum of 2 instance and a maximum of 10 instances OR completely based on your requirement.
  • It’s going to use the launch configuration we created in the earlier step 1.
  • We are using an aws_availibity_zone resource which will make sure instance will be deployed in different Availability Zone.
  • vpc_zone_identifier  A list of subnet IDs to launch resources in, we will get this value from vpc module.
  • target_group_arns  A list of aws_alb_target_group ARNs, for use with Application or Network Load Balancing, this value we will get from ALB module
  • health_check_type – Can be “EC2” or “ELB”. Controls how health checking is done. In this case we are using ELB.
  • In case if you are planning to use the default VPC, you can use data sources to look up the data for your default VPC
data "aws_vpc" "default" {
  default = true
}
  • We can combine this with another data source, aws_subnet_ids to lookup the subnets within that VPC. 
data "aws_subnet_ids" "default" {
  vpc_id = data.aws_vpc.default.id
}
  • Now to use it within your code
vpc_zone_identifier  = data.aws_subnet_ids.default.ids
  • We need to make slight changes to our VPC module outputs.tf file, as the output of subnet block, will not act as an input to autoscaling group vpc_zone_identifier
output "public_subnets" {
  value = "${aws_subnet.public_subnet.*.id}"
}
  • Same way we need to output the target arn from alb module so that it can act as an input to auto-scaling group module(target_group_arn)
output "alb_target_group_arn" {
  value = "${aws_lb_target_group.my-target-group.arn}"
}
  • Auto-Scaling Module will look like this
module "auto_scaling" {
  source           = "./auto_scaling"
  vpc_id           = "${module.vpc.vpc_id}"
  subnet_id        = "${module.vpc.public_subnets}"
  target_group_arn = "${module.alb.alb_target_group_arn}"
}
  • Final word, as we are now using auto-scale/auto-launch configuration to spin up our instance we probably don’t need EC2 module. I am leaving it for the time being but for ALB let me comment out the code
/*resource "aws_lb_target_group_attachment" "my-alb-target-group-attachment1" {
  target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  target_id        = "${var.instance1_id}"
  port             = 80
}

resource "aws_lb_target_group_attachment" "my-alb-target-group-attachment2" {
  target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  target_id        = "${var.instance2_id}"
  port             = 80
}*/

Final auto-scaling code will look like this

provider "aws" {
  region = "us-west-2"
}

  resource "aws_launch_configuration" "my-test-launch-config" {
  image_id        = "ami-01ed306a12b7d1c96"
  instance_type   = "t2.micro"
  security_groups = ["${aws_security_group.my-asg-sg.id}"]

  user_data = <<-EOF
              #!/bin/bash
              yum -y install httpd
              echo "Hello, from Terraform" > /var/www/html/index.html
              service httpd start
              chkconfig httpd on
              EOF

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "example" {
  launch_configuration = "${aws_launch_configuration.my-test-launch-config.name}"
  vpc_zone_identifier  = "${var.subnet_id}"

  target_group_arns = ["${var.target_group_arn}"]
  health_check_type = "ELB"

  min_size = 2
  max_size = 10

  tag {
    key                 = "Name"
    value               = "my-test-asg"
    propagate_at_launch = true
  }
}

resource "aws_security_group" "my-asg-sg" {
  name   = "my-asg-sg"
  vpc_id = "${var.vpc_id}"
}

resource "aws_security_group_rule" "inbound_ssh" {
  from_port         = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-asg-sg.id}"
  to_port           = 22
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "inbound_http" {
  from_port         = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-asg-sg.id}"
  to_port           = 80
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "outbound_all" {
  from_port         = 0
  protocol          = "-1"
  security_group_id = "${aws_security_group.my-asg-sg.id}"
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/auto_scaling

Complete Code

https://github.com/100daysofdevops/21_days_of_aws_using_terraform

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform
Posted on

21 Days of AWS using Terraform – Day 4- Creating Application Load Balancer(ALB) using Terraform

Welcome to Day 4 of 21 Days of AWS using Terraform,  So far we build VPC and EC2, let’s build Application Load Balancer and add two instances behind it. This is going to be a modular approach i.e we are going to get vpc id,subnet1 and subnet2 created during the VPC module and instance id from EC2 module.

21 Days of AWS using Terraform – Day 2- Building AWS VPC using Terraform
21 Days of AWS using Terraform – Day 3- Creating AWS EC2 Instance using Terraform

What is the Application Load Balancer?

The Application Load Balancer is a feature of ElasticLoad Balancing that allows a developer to configure and route incoming end-user traffic to applications based in the Amazon Web Services (AWS) public cloud.

Features

  • Layer7 load balancer(HTTP and HTTPs traffic)
  • Support Path and Host-based routing(which let you route traffic to different target group)
  • Listener support IPv6

Some Key Terms

Target Group

Target types:

  • Instance types: Route traffic to the Primary Private IP address of that Instance
  • IP: Route traffic to a specified IP address
  • Lambda function

Health Check

  • Determines whether to send traffic to a given instance
  • Each instance must pass its a health check
  • Sends HTTP GET request and looks for a specific response/success code

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html

Step1: Define the target group: This is going to provide a resource for use with Load Balancer.

resource "aws_lb_target_group" "my-target-group" {
  health_check {
    interval            = 10
    path                = "/"
    protocol            = "HTTP"
    timeout             = 5
    healthy_threshold   = 5
    unhealthy_threshold = 2
  }

  name        = "my-test-tg"
  port        = 80
  protocol    = "HTTP"
  target_type = "instance"
  vpc_id      = "${var.vpc_id}"
}
health_check: Your Application Load Balancer periodically sends requests to its registered targets to test their status. These tests are called health checks
interval: The approximate amount of time, in seconds, between health checks of an individual target. Minimum value 5 seconds, Maximum value 300 seconds. Default 30 seconds.
path: The destination for the health check request
protocol: The protocol to use to connect with the target. Defaults to HTTP
timeout:The amount of time, in seconds, during which no response means a failed health check. For Application Load Balancers, the range is 2 to 60 seconds and the default is 5 seconds
healthy_threshold: The number of consecutive health checks successes required before considering an unhealthy target healthy. Defaults to 3.
unhealthy_threshold: The number of consecutive health check failures required before considering the target unhealthy
matcher: The HTTP codes to use when checking for a successful response from a target. You can specify multiple values (for example, "200,202") or a range of values (for example, "200-299")name: The name of the target group. If omitted, Terraform will assign a random, unique name.
port: The port on which targets receive traffic
protocol: The protocol to use for routing traffic to the targets. Should be one of "TCP", "TLS", "HTTP" or "HTTPS". Required when target_type is instance or ip
vpc_id:The identifier of the VPC in which to create the target group. This value we will get from the VPC module we built earlier
target_type: The type of target that you must specify when registering targets with this target group.Possible values instance id, ip address
  • The VPC module we built earlier, we need to modify it a little built by outputting the value of VPC id(outputs.tf) which will act as an input to Application Load Balancer Module
output "vpc_id" {
  value = "${aws_vpc.main.id}"
}

Step2: Provides the ability to register instances with an Application Load Balancer (ALB)

resource "aws_lb_target_group_attachment" "my-alb-target-group-attachment1" {
  target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  target_id        = "${var.instance1_id}"
  port             = 80
}

resource "aws_lb_target_group_attachment" "my-alb-target-group-attachment2" {
  target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  target_id        = "${var.instance2_id}"
  port             = 80
}

target_group_arn: The ARN of the target group with which to register targets
target_id: The ID of the target. This is the Instance ID for an instance. We will get this value from EC2 module
port: The port on which targets receive traffic.

  • We need to modify the EC2 module(outputs.tf) by outputting the value of EC2 id which will act as an input to Application Load Balancer Module
output "instance1_id" {
  value = "${element(aws_instance.my-test-instance.*.id, 1)}"
}

output "instance2_id" {
  value = "${element(aws_instance.my-test-instance.*.id, 2)}"
}
  • element(list, index) – Returns a single element from a list at the given index. If the index is greater than the number of elements, this function will wrap using a standard mod algorithm. This function only works on flat lists.

https://www.terraform.io/docs/configuration-0-11/interpolation.html#element-list-index-

Step3: Define the load balancer

resource "aws_lb" "my-aws-alb" {
  name     = "my-test-alb"
  internal = false

  security_groups = [
    "${aws_security_group.my-alb-sg.id}",
  ]

  subnets = [
    "${var.subnet1}",
    "${var.subnet2}",
  ]

  tags = {
    Name = "my-test-alb"
  }

  ip_address_type    = "ipv4"
  load_balancer_type = "application"
}

name: The name of the LB. This name must be unique within your AWS account, can have a maximum of 32 characters, must contain only alphanumeric characters or hyphens, and must not begin or end with a hyphen. If not specified, Terraform will autogenerate a name beginning with tf-lb (This part is important as Terraform auto
internal: If true, the LB will be internal.
load_balancer_type: The type of load balancer to create. Possible values are application or network. The default value is the application.
ip_address_type: The type of IP addresses used by the subnets for your load balancer. The possible values are ipv4 and dualstack
subnets: A list of subnet IDs to attach to the LB. In this case, I am attaching two public subnets we created during load balancer creation. This value we will get out from the VPC module.
tags: A mapping of tags to assign to the resource.

  • We need to modify the VPC module and output the value of two subnet id which will act as an input to application load balancer module
output "subnet1" {
  value = "${element(aws_subnet.public_subnet.*.id, 1 )}"
}

output "subnet2" {
  value = "${element(aws_subnet.public_subnet.*.id, 2 )}"
}

Step4:  Security group used by ALB


resource "aws_security_group" "my-alb-sg" {
  name   = "my-alb-sg"
  vpc_id = "${var.vpc_id}"
}

resource "aws_security_group_rule" "inbound_ssh" {
  from_port         = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-alb-sg.id}"
  to_port           = 22
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "inbound_http" {
  from_port         = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-alb-sg.id}"
  to_port           = 80
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "outbound_all" {
  from_port         = 0
  protocol          = "-1"
  security_group_id = "${aws_security_group.my-alb-sg.id}"
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

NOTE: I already explained about Security Group in the VPC module.

Step5: Provides a Load Balancer Listener resource

resource "aws_lb_listener" "my-test-alb-listner" {
  load_balancer_arn = "${aws_lb.my-aws-alb.arn}"
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  }
}

load_balancer_arn – The ARN of the load balancer(we created in step3)

port – (Required) The port on which the load balancer is listening.

protocol – (Optional) The protocol for connections from clients to the load balancer. Valid values are TCP, TLS, UDP, TCP_UDP, HTTP and HTTPS. Defaults to HTTP.

default_action – An Action block.

type – (Required) The type of routing action. Valid values forward, redirect, fixed-response, authenticate-cognito and authenticate-oidc.

target_group_arn – The ARN of the Target Group to which to route traffic. Required if type is forward(Created in Step1)

Final Code

provider "aws" {
  region = "us-west-2"
}

resource "aws_lb_target_group" "my-target-group" {
  health_check {
    interval            = 10
    path                = "/"
    protocol            = "HTTP"
    timeout             = 5
    healthy_threshold   = 5
    unhealthy_threshold = 2
  }

  name        = "my-test-tg"
  port        = 80
  protocol    = "HTTP"
  target_type = "instance"
  vpc_id      = "${var.vpc_id}"
}

resource "aws_lb_target_group_attachment" "my-alb-target-group-attachment1" {
  target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  target_id        = "${var.instance1_id}"
  port             = 80
}

resource "aws_lb_target_group_attachment" "my-alb-target-group-attachment2" {
  target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  target_id        = "${var.instance2_id}"
  port             = 80
}

resource "aws_lb" "my-aws-alb" {
  name     = "my-test-alb"
  internal = false

  security_groups = [
    "${aws_security_group.my-alb-sg.id}",
  ]

  subnets = [
    "${var.subnet1}",
    "${var.subnet2}",
  ]

  tags = {
    Name = "my-test-alb"
  }

  ip_address_type    = "ipv4"
  load_balancer_type = "application"
}

resource "aws_lb_listener" "my-test-alb-listner" {
  load_balancer_arn = "${aws_lb.my-aws-alb.arn}"
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = "${aws_lb_target_group.my-target-group.arn}"
  }
}

resource "aws_security_group" "my-alb-sg" {
  name   = "my-alb-sg"
  vpc_id = "${var.vpc_id}"
}

resource "aws_security_group_rule" "inbound_ssh" {
  from_port         = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-alb-sg.id}"
  to_port           = 22
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "inbound_http" {
  from_port         = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.my-alb-sg.id}"
  to_port           = 80
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "outbound_all" {
  from_port         = 0
  protocol          = "-1"
  security_group_id = "${aws_security_group.my-alb-sg.id}"
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

variables.tf

variable "vpc_id" {}
variable "instance1_id" {}
variable "instance2_id" {}
variable "subnet1" {}
variable "subnet2" {}

GitHub Link

https://github.com/100daysofdevops/21_days_of_aws_using_terraform/tree/master/alb

Final ALB Module will look like this

module "alb" {
  source       = "./alb"
  vpc_id       = "${module.vpc.vpc_id}"
  instance1_id = "${module.ec2.instance1_id}"
  instance2_id = "${module.ec2.instance2_id}"
  subnet1      = "${module.vpc.subnet1}"
  subnet2      = "${module.vpc.subnet2}"
}

Final Code

https://github.com/100daysofdevops/21_days_of_aws_using_terraform

Looking forward for you guys to join this journey

In addition to that, I am going to host 5 meetups whose aim is to build the below architecture.

  • Meetup: https://www.meetup.com/100daysofdevops
  • Day1(Nov 10): Introduction to Terraform https://www.meetup.com/100daysofdevops/events/266192294/
  • Day 2(Nov 16): Building VPC using Terraform
  • Day 3(Nov 17): Creating EC2 Instance inside this VPC using Terraform
  • Day 4(Nov 23): Adding Application Load Balancer and Auto-Scaling to the EC2 instance created on Day 3
  • Day5(Nov 24): Add Backend MySQL Database and CloudWatch Alarm using Terraform