Introduction

Hello, after some time without posting we are back!. Today’s post will be focused on Terraform. For those that never heard about it, Terraform allows you to build infrastructure as if you were writing code.

I started playing with it mainly for two reasons: In the first place, I’m reading the book Test-Driven Development with Python, and at some point I had to setup my own machine with a webserver to deploy an application. I thought that having some kind of automation for it was a good idea. Secondly, I needed a good excuse to learn some AWS magic, and this was the perfect excuse for that!.

So, in this blogpost I’ll share what I learned about Terraform and my progress from zero to feeling a bit less confused with AWS and the idea of Infrastructure as Code. I’ll show you how to automate the creation of an EC2 Instance running Nginx in a non-default VPC.

AWS basic concepts

Before learning about Terraform I needed to understand what I had to “build” with it. My plan was to have an isolated non-default network (I did not want to use the default VPC, I felt that it was kind of cheating), with a webserver serving an application. All of this needed to be reachable from the Internet. Last but not least (and the most important thing!) all of this should be accomplished using AWS Free tier infrastructure.

My requirements translated to the following concepts in the AWS world:

Virtual Private Cloud

Allows you to create a network isolated from others. You can choose IP addressing scheme, routing tables, Gateways and so on. When you create an account in AWS it comes with a VPC by default. In this example as I wanted to learn more, I decided to create my own VPC.

Using a custom VPC brings some complexity that requires you to understand at least the basics of networking in AWS. We will need to create a subnet, an Internet Gateway, define a routing table and deal with some security mechanisms designed to control what traffic reaches what endpoints.

VPCs are created in one specific region (for example, “us-esast-1”) and they “cover” all the availability zones within that region.

VPCs are defined by a name and a CIDR block, for example in our case will be 192.168.0.0/16.

VPCs support netmasks between /16 and /28.

Note: There isn’t any costs for creating a VPC, however some components can have additional costs (not used in this blogpsot). For more information, you can check: https://aws.amazon.com/vpc/pricing/

Subnet

Once the VPC is created, you need at least a subnet in it. Subnets only span across one availability zone, but is possible to create one (or more) subnets in each availability zone. In this specific case I decided to keep things simple and only created one subnet. The IP addressing scheme I used for it is 192.168.2.0/24.

Internet Gateway

The previously created subnet should be reachable from the Internet to allow users to access the web server that we will deploy. For that, AWS provides “Internet Gateways”, that you can attach to your VPC and enable communications between the VPC and the Internet.

Routing table and routes

Route tables define how network packets reach the different subnets in our VPC. For our example the routing table will be simple as we only have one subnet and one gateway. There are some interesting concepts worth mentioning.

Main Routing table

It is the route table that comes by default once you create a VPC. It controls the routing for all the subnets that you create inside the VPC that DO NOT HAVE THEIR OWN Route table associated.

A subnet can only be associated with one route table, but multiple subnets can share the same route table (allowing routing between different subnets).

Custom Routing table

Additional Routing tables that you can create for your VPC that can be associated to subnets. For our example we will create a custom routing table and associate it to our own subnet.

Route table association

It’s the process of joining a Route Table with a subnet or an Internet Gateway.

AWS networking security 101

Security groups (SGs)

Security groups are a set of rules (like a virtual firewall) that apply to instances running in a VPC. These rules determine what traffic is allowed to access and leave an instance, based on the traffic’s protocol, port and source/destination address.

• Rules in the SGs are stateful. For example, if you allow inbound TCP traffic on port 22 you don’t need to explicitly add an outbound rule for that.

• Rules in the SGs are whitelist. It isn’t possible to forbid traffic with them.

• When a SG is created, by default it does not allow any inbound traffic.

• When a SG is created, by default it allows ALL outbound traffic.

Network Access Control Lists (ACLs)

Network Access Control Lists are an additional security mechanism that applies at subnet level. They work as a complement to security groups.

• Every VPC has a default network ACL that allows all inbound and outbound IPv4 traffic.

• Each subnet created in a VPC must have a network ACL associated. It’s possible to create custom network ACLs and associate them to the subnets. In case that no custom network ACL is created, the default will be used.

• Each time a custom network ACL is created it denies all traffic, until you specifically add rules to allow it.

• Network ACLs are composed of different rules that are evaluated in ascending order according to a number that each rule has. Once a rule matches the rest of them are ignored.

• Network ACLs are stateless. This means that you’ll to take care of creating inbound rules to allow connections, but also outbound rules to allow responses to go back to clients.

Terraform to the rescue

Well, after a lot of AWS theory and concepts we can start the fun part which is building our infrastructure with Terraform. Using Terraform we will create:

1. A new VPC.
2. A new subnet in the previously created VPC.
3. An Internet Gateway and associate it with the subnet from step 2 to provide connectivity to our VPC to the Internet.
4. A routing table and associate it with our subnet.
5. A security group to allow SSH and HTTP access to our web server. We will associate this SG to our instance.
6. An EC2 instance that will host our application.
7. A keypair to securely connect to SSH without any password.
8. We will use some magic from Terraform called Provisioner to execute commands once our instance is created to update software, install and run Nginx.

To install Terraform I followed Hashicorp Terraform installation tutorial. Once installed, I created a new folder learn-terraform-aws-instance. This folder will serve as the root of our Terraform project.

Inside this folder I created the following files:

• main.tf: Holds EC2 instance creation code.
• network.tf: Contains VPC, subnet and routing configuration code.
• netsec.tf: Creates the SG.
• variables.tf: An utility file where all variables used are defined.

Some comments about this:

• In my first attempt I just used one big file with everything defined there (At the end using multiple files it’s just for the sake of clarity).
• I could (and should) have used Terraform modules to do almost all of this. As I wanted to fully understand what I was doing I chose to do it with resources instead.

All files used in this project can be found in this repository: https://github.com/nahueldsanchez/terraform-simple-deploy.

Let’s start, to create a new VPC I first needed to let terraform what provider I was going to use and later init the project. For that I added the following code in the main.tf file:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = var.aws_region
}

As you can see I’m using a variable aws_region. To use a variable in a file you need to prefix its name with var.. In my case I defined them in variables.tf:

variable "aws_region" {
    description = "Amazon region used to launch the EC2 instance"
    type = string
    default = "us-west-2"
}

With this we are saying that by default we will use us-west-2 region to deploy our infrastructure. For a full explanation of variables in Terraform you can refer to this link.

Once I have that files created I needed to run terraform init. With this command I let Terraform now that I was going to use AWS as a provider and Terraform automatically downloaded the required files.

All resources created share a Tag that I defined as Environment = Dev to easily identify them. The idea is that you can use it to have multiple environments at the same time.

The next step was to define the network.tf file that creates the VPC, subnet, Internet Gateway and the route table:

resource "aws_vpc" "python-tdd-vpc" {
  cidr_block = var.vpc_cidr_block
  enable_dns_support = true
  enable_dns_hostnames = true

  tags = {
    Name = "python-tdd-vpc"
    Environment = var.environment
  }
}

resource "aws_subnet" "python-tdd-subnet" {
  vpc_id = aws_vpc.python-tdd-vpc.id
  cidr_block = var.subnet_cidr_block
  map_public_ip_on_launch = true

  tags = {
    Name = "python-tdd-subnet"
    Environment = var.environment
  }
}

resource "aws_internet_gateway" "python-tdd-internet-gw" {
  vpc_id = aws_vpc.python-tdd-vpc.id

  tags = {
    Environment = var.environment
  }
}

resource "aws_route_table" "python-tdd-routing-table" {
  vpc_id = aws_vpc.python-tdd-vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.python-tdd-internet-gw.id
  }

  tags = {
    Environment = var.environment
  }
}

resource "aws_route_table_association" "python-tdd-route-association" {
  subnet_id      = aws_subnet.python-tdd-subnet.id
  route_table_id = aws_route_table.python-tdd-routing-table.id
}

The first resource creates the VPC. Again i’m using a variable var.vpc_cidr_block to determine the CIDR block. To know what other properties you can configure for each resource (ie: enable_dns_support and enable_dns_hostnames) you can refer to Terraform’s documentation.

Secondly I created the subnet. As explained above subnets are associated with a VPC. For this reason I’m using the previously created VPC ID. It won’t be known until it is created, but we can reference it like this: aws_vpc.python-tdd-vpc.id where first we find the resource type, the name that we defined and lastly the id property.

A key detail when creating the subnet is to set map_public_ip_on_launch = true to tell AWS that we want to automatically provide public IPs to instances deployed within this subnet.

The third resource is our Internet Gateway that provides Internet connectivity to the VPC. As you can see it’s attached to it using its ID vpc_id = aws_vpc.python-tdd-vpc.id.

The latest two resources are the route table and its association to the subnet. As you can see the route table defines a route for CIDR 0.0.0.0/0 through the Internet Gateway. The last step is to associate this route table to our subnet. Remember that if a subnet does not have an explicitly associated subnet to it, will use the VPC’s default routing table.

The next thing I did was to create the netsec.tf file that defines the Security Group to be used by the EC2 instance that we are going to create. This SG needs to allow the following traffic:

• Incoming TCP traffic on port 22 to allow access to the SSH server.
• Incoming TCP traffic on port 80 to allow access to the webserver.
• Outgoing traffic from the VPC to allow updates, downloading the application and so on (this can be done better, defining a more restrictive rule).

I defined like this:

resource "aws_security_group" "allow_ssh_http" {
  name = "Allow SSH and HTTP"
  description = "Allows SSH access on port 22 and HTTP access on port 80 and all outgoing traffic"
  vpc_id = aws_vpc.python-tdd-vpc.id

  ingress {
    description = "Allow traffic from 0.0.0.0/0 to port 22"
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = [var.administrator_ip_address]
  }

  ingress {
    description = "Allow traffic from 0.0.0.0/0 to port 80"
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "Allow all traffic from VPC to 0.0.0.0/0"
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Environment = var.environment
  }
}

some notes:

• Protocol "-1" in the egress rule means all protocols.
• cidr_blocks are lists.
• I decided to use a variable var.administrator_ip_address that can be used to narrow who can connect to the SSH server. By default is defined as 0.0.0.0/0 but this can be changed.

Once all of that is defined we only need to create the actual EC2 instance, this was defined in main.tf:

resource "aws_key_pair" "python-tdd-keypair" {
    key_name = "python-tdd-keypair"
    public_key = file("./python-tdd-keypair.pub")
}


# https://www.terraform.io/docs/language/resources/provisioners/remote-exec.html
resource "aws_instance" "python-tdd-ec2instance" {
  ami = var.amis_ids_us_regions[var.aws_region]
  instance_type = "t2.micro"
  subnet_id = aws_subnet.python-tdd-subnet.id
  vpc_security_group_ids = [aws_security_group.allow_ssh_http.id]
  key_name = aws_key_pair.python-tdd-keypair.id

  provisioner "remote-exec" {
    inline = [
      "sudo apt update",
      "sudo apt upgrade -y",
      "sudo apt install nginx -y",
      "sudo service nginx start"
    ]
  }

  connection {
    host = aws_instance.python-tdd-ec2instance.public_dns
    user = "ubuntu"
    private_key = file("./python-tdd-keypair")
  }

  tags = {
    Environment = var.environment
  }
}

output "Public_IP" {
  value = aws_instance.python-tdd-ec2instance.public_ip
}

There are a few interesting things here:

First we see defined a aws_key_pair resource. This is used to tell Terraform to upload a previously created key to use with the SSH server. This keypair was created with the following command:

ssh-keygen -f python-tdd-keypair

Once prompted for the passphrase leave it blank. This will create two files in the current directory: python-tdd-keypair and python-tdd-keypair.pub.

I used Terraform function file that read a file from the filesystem and retrieves its content.

Secondly, I decided to use a Provisioner to execute a few commands once the EC2 instance is created. This is not the perfect solution as there are specific tools to provision servers such as Packer or cloud-init. But as I had to execute a few commands, this was enough for me.

With provisioner "remote-exec" you define a list of commands to execute and also you need to define a connection. In this case we are retrieving the instance’s public DNS and using the default user ubuntu with the private key previously created to connect using SSH.

Lastly, we print the instance’s public IP.

Once we have everything, we run terraform plan. This will show what Terraform will create. To apply the proposed changes we run terraform apply and wait a few minutes.

I hope that this long blogpost with a full step-by-step of how I created basic infrastructure using Terraform was useful!.

Thanks for reading.

References

• VPCs and subnets (AWS Virtual Private Cloud User Guide) - https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ug.pdf#VPC_Subnets

• Route tables for your VPC (AWS Virtual Private Cloud User Guide) - https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ug.pdf#VPC_Route_Tables

• VPCs Security Groups - https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html