Replace AWS NAT Gateway with pfSense VM

For improved security, more and more AWS cloud resources are being deployed into private subnets with no public internet endpoints. A NAT-Gateway is sometimes added to these private subnets to allow private resources to access internet (e.g. download upgrades) without otherwise being exposed.

However, there are costs associated with NAT Gateways, including an hourly charge and then NAT Data Processing Charge based on usage.

It is worth noting that unlike the standard AWS Data Transfer Cost which is free for data transfer from the internet into AWS, the NAT Data Processing Charge does not distinguishing between the direction of data flow. This makes NAT Gateway a relatively expensive solution for private cloud applications that require huge data in-take. NAT Gateway will charge for these data in-take while data transfer into AWS via Internet Gateway is otherwise free.

In this blog post, I document how I set up a pfSense VM to replace the NAT Gateway for private subnets. The calculation below shows the cost savings for 5TB data in-take over 1 month (720 hours) in the ca-central-1 region.

• NAT Gateway has a base rate of $0.05/hr, plus Data Processing Cost of $0.05/GB. So its total cost would be $0.05 x 720 + $0.05 x 5120 = $292.
• A pfSense Plus software running on a t3.nano VM costs $0.0058/hr for the VM and $0.02/hr for the software, while inbound data is free. So the total cost would be ($0.0058 + 0.02) x 720 = $18.6.

Architecture Design

In the VPC (172.29.0.0/16) there are 3 subnets.

• A public subnet in AZ ca-central-1a (172.29.0.0/20)
• A private subnet in AZ ca-central-1a (172.29.16.0/20)
• A private subnet in AZ ca-central-1b (172.29.32.0/20)

AWS EC2 virtual machines can only have network interfaces in the same Availability Zone (AZ). I demonstrate that this solution can allow both private subnets in different AZs safely access internet.

Prepare VPC and Subnets

The first step is to create the VPC and subnets as shown in the previous section.

Subnets and CIDRs

Then create two security groups, pfsense-public and pfsense-private with the following in-bound rules.

• For pfsense-public, allow port 443 from your current IP. This will allow you to connect to pfSense web portal once you created the VM.
• For pfsense-private, allow all traffic from all source IPs. This will allow any traffic on the private subnets to reach the pfSense VM. We will control firewall rules from within pfSense.

We then create two Network Interfaces in the EC2 console. Both interfaces need to be in same AZ (e.g. ca-central-1a).

• In public-subnet-1, create a public-nic network interface and assign pfsense-public security group to it.
• In private-subnet-1, create the private-nic and assign pfsense-private security group to it.

Network Interfaces and Security Groups

IMPORTANT: You must modify the “Source/Destination Check” setting of both network interfaces. Uncheck the “Enable” box to disable this check. Otherwise pfSense VM will not be able to receive traffic that is not destined to the VM. To act as a NAT device, pfSense must be able to receive all network traffic to route them via NAT.

Disable Source/Destination Check

Now heading to the Elastic IP section to allocate a new static IP. Then go to the Network Interface section, find the public-nic interface, and associate it with the static IP.

Associating Elastic IP

Now create an Internet Gateway, and attach it to the VPC.

To use this Internet Gateway, create a new Route Table and add the following routes.

Public Subnet Route Table

Associate this Route Table with public-subnet-1. This makes public-subnet-1 a routable subnet. Our public-nic will now be able to receive public internet traffic.

Public Subnet Route Table Association

Now we modify the VPC’s default Route Table as shown below and associate it with the two private subnets private-subnet-1 and private-subnet-2. Note the route for 0.0.0.0/0 points to the private-nic network interface.

Private Subnet Route Table

Private Subnet Route Table Association

Create pfSense VM

Now we are ready to fire up the pfSense VM!

Go to EC2 console and select Launch Instance. Search for “pfSense Plus” AMI. You will find it in Marketplace.

Select pfSense Plus AMI

Select an instance type (e.g. t3.nano) and specify the SSH key to use.

Specify Instance Type and Key Pair

Now, click Edit in the Network Settings section. Choose to launch the VM in public-subnet-1 and disable Auto-assign Public IP. For Firewall (security group), choose Select existing security group) but don’t actually select one. We will specify this in the Advanced Network Configuration section.

Network Settings

Click to expand the Advanced Network Configuration section. Change Network Interface from “New Interface” to public-nic. Then click “Add network interface” and add private-nic as the second NIC.

Network Interfaces

IMPORTANT: Finally, expand the Advanced Detail section, and enter password=xxxxxxxx into the User Data box. This is the initial password for the pfSense web portal. It is still possible to login without specifying the initial password here, but it would be more difficult.

Intial Password in User Data

Now Click “Launch Instance” to fire up the VM.

Update pfSense Configurations

Once the VM started, connect to its public IP address using https://xxx.xxx.xxx.xxx. Login with username admin and the password you specified in the User Data, and complete the initial setup wizard.

When the setup wizard has completed, go to “Interfaces” → “Assignments”. The WAN interface is already there, and there is another available interface waiting for assignment. Click “Add” to assign it.

Interface Assignments

The second interface is assigned as “LAN”. Click the name to edit its settings. Check “Enable interface” and set “IPv4 Configuration Type” to “DHCP” as shown below. Click “Save” and then “Apply Changes”.

LAN Interface Settings

We now need to update the NAT settings. Go to “Firewall” → “NAT” → “Outbound”. The default setting is to use “Automatic outbound NAT rule generation” with the following auto-generated rules.

Auto-Generated NAT Rules

As AWS uses DHCP to configure both interfaces, they are treated as “WAN” mode interfaces by pfSense. This means both interfaces will be assigned NAT rules. This is not what we want. We only want NAT on the public-nic / WAN interface.

To update these NAT rules, we will need to change the Outbound NAT Mode to “Manual Outbound NAT rule generation”, then click “Save”. This expands the auto-generated rules into a list of rules that we can modify.

Change NAT Mode

We will first remove all rules on the LAN interface. Select these rules and click “Delete”.

Delete Auto-Generated NAT Rules

We will now copy the two rules related to IPSec and modify them to allow NAT for our two private subnets. First, click the Copy icon on the ISAKMP rule.

Copy the ISAKMP Rule

This will ope the copied rule for us to edit. Change the source network to the IP range of private-subnet-1 (in our case it is 172.29.16.0/20). Click “Save”.

Modify the Source Network IP Range

Then copy the other rule and also modify its source network to the IP range of private-subnet-1.

Copy the NAT Rule

Repeat the above two steps for private-subnet-2. The final list should look like this. When all looks correct, click “Apply Changes”.

Completed Manual NAT Rules

That’s it! All resources on your private subnets can now use this pfSense VM to access internet!