WireGuard on AWS ECS

AWS ECS (Elastic Container Service) is a convenient way to run containers in the cloud. And it’s easy to run WireGuard as a container on ECS (with a few important caveats) — this article will show you how.

In this article, we’ll run WireGuard as an ECS task that allows remote “point to cloud” access to our internal AWS infrastructure. We’ll use the Pro Custodibus WireGuard image, and configure it and the surrounding network similar to the bastion host scenario covered by the Point to Cloud WireGuard With AWS Private Subnets article:

Point To Cloud with AWS ECS

This particular configuration allows you, from your local workstation, to access network resources in the same AWS VPC (Virtual Private Cloud) as the WireGuard container. The WireGuard container forwards traffic from your local workstation to the VPC, masquerading your traffic as if had originated on the container’s own EC2 host (in this example, having an IP address of 10.10.20.95 within the VPC).

With this configuration, because the forwarded network traffic appears to originate on the EC2 host, the security group rules that apply to the EC2 host also apply to the forwarded traffic. So to allow your local workstation access to an internal application within the VPC that’s protected by a certain security group (in this example, the app running at TCP port 8080 on 10.10.21.130, protected by the sg-0abcdef1234567890 security group), you simply authorize ingress to that security group from the WireGuard container’s EC2 host (or from a security group used by the host, like sg-1234567890abcdef0 in this example).

Preparing the Host

When preparing to run WireGuard on ECS, keep in mind these two important caveats:

  1. WireGuard can be run only on the EC2 launch-type (not on Fargate).

  2. The WireGuard kernel module must be present on the EC2 host itself.

The WireGuard kernel module is part of every Linux kernel version 5.6 and newer, so if you’re managing your own ECS container instances, and using a relatively modern kernel, you’re all set. You can check the kernel version of an EC2 host by running the following command on it:

$ uname -r
5.15.0-1005-aws

In the above example, we’re running Ubuntu 22.04, which uses the Linux kernel version 5.15. Since the kernel version is above 5.6, we don’t have to do anything to this host — it will run WireGuard containers just fine the way it is.

However, if you’re running a Linux kernel version below 5.6, you’ll have to manually install the WireGuard kernel module on the host. This often requires compiling the WireGuard kernel module from source on the host. In particular, for Amazon Linux 2 with the old kernel version 4.14 in particular, see the section dedicated to it in the Installing WireGuard on Amazon Linux article.

Deploying the Container

As long as the ECS host has the WireGuard kernel module, deploying a WireGuard container as an ECS task to it is straightforward. However, the task definition must first be configured with the Linux NET_ADMIN capability added. This cannot be done via the AWS console UI — you have to use a tool that exposes the advanced ECS task definition settings, such as the AWS SDK, or the AWS CLI, or AWS CloudFormation (or various other similar tools). We’ll use the AWS CLI in this article.

Follow these steps to deploy the procustodibus/wireguard container image as an ECS task:

  1. Set Up the WireGuard Config File

  2. Adjust Your Security Groups

  3. Define the WireGuard Task

  4. Run the WireGuard Task

  5. View the Container Logs

Set Up the WireGuard Config File

The easiest way to set up a WireGuard config file for an ECS container to use is to store it on an AWS EFS (Elastic File System) volume, and mount that volume into the container so that the config file shows up in the container’s /etc/wireguard directory. You can also take other approaches, such as to set up the EC2 host with the config file on a local volume, and map that local volume into the container; or to create a custom WireGuard image that pulls the config from some other data store (like the AWS SSM Parameter Store, AWS Secrets Manager, AWS S3, etc); but EFS is the simplest, so we’ll show it here.

We’ll assume that you already have an ECS cluster and EFS filesystem set up — if you don’t, follow the first four steps of AWS’s Using Amazon EFS file systems with Amazon ECS tutorial to do so. In this article, we’ll use fs-1234567890abcdef0 for the ID of this EFS filesystem.

With the EFS filesystem set up, copy the following WireGuard config file to the filesystem under a subdirectory dedicated to the ECS task that we’ll define in the next step:

# /prod/infra/wg-p2s/conf/wg0.conf

# local settings for the container
[Interface]
PrivateKey = ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBFA=
Address = 10.0.0.2/32
ListenPort = 51820

# IP masquerading
PreUp = iptables -t nat -A POSTROUTING ! -o %i -j MASQUERADE
# firewall wg peers from other hosts
PreUp = iptables -A FORWARD -o %i -m state --state ESTABLISHED,RELATED -j ACCEPT
PreUp = iptables -A FORWARD -o %i -j REJECT

# remote settings for my workstation
[Peer]
PublicKey = /TOE4TKtAqVsePRVR+5AA43HkAK5DSntkOCO7nYq5xU=
AllowedIPs = 10.0.0.1/32

See the WireGuard Point to Site Configuration article for a detailed explanation of each WireGuard config setting used above.

Important

Do not use the WireGuard keys shown in this article — generate your own keys, and copy and paste them into the WireGuard config file (as detailed in the Generate WireGuard Keys section of the point-to-site article). Run the following commands to generate new private keys:

$ wg genkey > the-container.key
$ wg genkey > my-workstation.key

And the following commands to compute the corresponding public keys:

$ wg pubkey < the-container.key > the-container.pub
$ wg pubkey < my-workstation.key > my-workstation.pub

In this example, we’ll use /prod/infra/wg-p2s/conf as the dedicated directory on the EFS filesystem for this WireGuard config file. If you have the filesystem mounted at /mnt/efs/fs1 on some EC2 instance, and have saved the above file locally on that instance at ~/wg0.conf, you can run the following commands from that instance to create the dedicated directory, and copy the config file to it:

$ sudo mkdir -p /mnt/efs/fs1/prod/infra/wg-p2s/conf
$ sudo cp ~/wg0.conf /mnt/efs/fs1/prod/infra/wg-p2s/conf/.

Adjust Your Security Groups

A WireGuard container using the above config file will provide connectivity from its remote WireGuard peers to anything its hosting EC2 instance can access, similar to the WireGuard server from the Point to Cloud WireGuard With AWS Private Subnets article. Like with the WireGuard server from that article, we can use VPC security groups to control access to and from the hosting EC2 instance.

At minimum, we’ll need to add a security group rule that allows inbound access to UDP port 51820 of the EC2 host, to allow remote WireGuard peers to connect. If the EC2 host was protected by a security group with an ID of sg-1234567890abcdef0, we could use the following AWS CLI command to add this rule:

$ aws ec2 authorize-security-group-ingress \
    --group-id sg-1234567890abcdef0 \
    --protocol udp \
    --port 51820 \
    --cidr 0.0.0.0/0
Tip

This allows WireGuard access from any IPv4 address. If you know that clients will only need to access this WireGuard network from a limited set of places, like just a few office locations, you may want to instead set up security group rules to allow WireGuard access from just the CIDR blocks used by those places, instead of everywhere.

We also need to set up the security group rules that will allow access from the WireGuard container’s EC2 host to the other internal network resources to which we want to connect through WireGuard (if such rules are not already in place). In our example scenario, we want to be able to access an internal web app running at TCP port 8080 on some other EC2 instance in the VPC. This other EC2 instance is protected by a security group with an ID of sg-0abcdef1234567890, so we’d run the following AWS CLI command to add a rule allowing access to that app:

$ aws ec2 authorize-security-group-ingress \
    --group-id sg-0abcdef1234567890 \
    --protocol tcp \
    --port 8080 \
    --source-group sg-1234567890abcdef0

Define the WireGuard Task

With the above WireGuard config file in place on the EFS filesystem, and our EC2 host’s security group updated to allow access to its UDP port 51820, we can define an ECS task for our WireGuard container with the following JSON:

{
    "family": "wg-p2s",
    "containerDefinitions": [
        {
            "name": "wireguard",
            "image": "procustodibus/wireguard",
            "cpu": 16,
            "memory": 16,
            "portMappings": [
                {
                    "hostPort": 51820,
                    "containerPort": 51820,
                    "protocol": "udp"
                }
            ],
            "mountPoints": [
                {
                    "containerPath": "/etc/wireguard",
                    "sourceVolume": "wireguard-configuration"
                }
            ],
            "linuxParameters": {
                "capabilities": {
                    "add": ["NET_ADMIN"]
                }
            }
        }
    ],
    "volumes": [
        {
            "name": "wireguard-configuration",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-1234567890abcdef0",
                "rootDirectory": "/prod/infra/wg-p2s/conf"
            }
        }
    ]
}

Let’s go through this task definition, setting-by-setting:

family

Name of the task. You can name it whatever you want — it just needs to be unique within your AWS account.

containerDefinitions.name

Name of the container within the task. You can name it whatever you want — it just needs to be unique within this task.

containerDefinitions.image

Name of the WireGuard image to pull.

containerDefinitions.cpu

CPU units (out of 1024) reserved for the container. Unless you are planning to send a very large amount of traffic through this WireGuard container, you won’t need to set aside very many CPU units.

containerDefinitions.memory

Megabytes of memory reserved for the container. The most minimal ECS container needs about 6 MB to run; this WireGuard container doesn’t need much more than that.

containerDefinitions.portMappings.hostPort

WireGuard port, as exposed to the VPC. This should match the Endpoint port you put in your WireGuard client configuration. (Or if, instead of allowing direct access to the EC2 host, you are going to put an NLB (Network Load Balancer) in front of it, similar to the High Availability WireGuard on AWS article, this should match the port setting of the load-balancer’s target group.)

containerDefinitions.portMappings.containerPort

WireGuard port, as exposed by the container. This should match the ListenPort from the WireGuard server configuration we set up above.

containerDefinitions.portMappings.protocol

Always udp for WireGuard.

containerDefinitions.mountPoints.containerPath

Maps the root directory of the wireguard-configuration volume (defined in the volumes section of the task definition) to the /etc/wireguard directory on the container. The root directory of the wireguard-configuration volume should contain the wg0.conf WireGuard config file we set up above.

containerDefinitions.mountPoints.sourceVolume

Name of the volume containing the WireGuard config file. You can name it whatever you want — it just needs to match the name defined in the volumes section.

containerDefinitions.linuxParameters.capabilities.add

Must include NET_ADMIN for WireGuard. This setting cannot be configured through the AWS console UI (which is why we need to define this task via JSON).

volumes.name

Name of the volume containing the WireGuard config file. You can name it whatever you want — it just needs to match the containerDefinitions.mountPoints.sourceVolume setting above.

volumes.efsVolumeConfiguration.fileSystemId

ID of the EFS filesystem containing the WireGuard config file.

volumes.efsVolumeConfiguration.rootDirectory

Path from the root of the EFS filesystem to the directory containing the WireGuard config file. In the Set Up the WireGuard Config File section above, we placed the config file in the /prod/infra/wg-p2s/conf directory.

If you save this task definition JSON as the file wg-p2s.json on your local workstation, you can then run the following AWS CLI command from the same directory as it to register the task definition with AWS:

$ aws ecs register-task-definition --cli-input-json file://wg-p2s.json

Run the WireGuard Task

Once the task has been registered, we can run it on the hosts of the cluster that we prepared above. If that cluster’s name is for example prod-infra, we can run version 1 of this task with the following AWS CLI command:

$ aws ecs run-task --cluster prod-infra --task-definition wg-p2s:1

The output of this command will include information about the task, including the full ARN (Amazon Resource Name) of the ECS host in which the task was launched (as the containerInstanceArn property):

{
    "tasks": [
        {
            "availabilityZone": "us-west-2b",
            "clusterArn": "arn:aws:ecs:us-west-2:012345789012:cluster/prod-infra",
            "containerInstanceArn": "arn:aws:ecs:us-west-2:012345789012:container-instance/prod-infra/18796a629c6c4e9595db03b074bcf34a",
            "containers": [
                {
                    "containerArn": "arn:aws:ecs:us-west-2:012345789012:container/prod-infra/ada8d3f7c14c4dc48556f1a34b745447/99071183-d5d8-458c-8d32-93821079936e",
                    "taskArn": "arn:aws:ecs:us-west-2:012345789012:task/prod-infra/ada8d3f7c14c4dc48556f1a34b745447",
                    "name": "wireguard",
                    "image": "procustodibus/wireguard",
                    "lastStatus": "PENDING",
                    "networkInterfaces": [],
                    "cpu": "16",
                    "memory": "16"
                }
            ],
            ...

We can use this container-instance ARN to lookup up information about the EC2 ID of the host:

$ aws ecs describe-container-instances \
    --cluster prod-infra \
    --container-instances arn:aws:ecs:us-west-2:012345789012:container-instance/prod-infra/18796a629c6c4e9595db03b074bcf34a \
    --query containerInstances[].ec2InstanceId \
    --output text
i-0123456789abcdef0

And with the EC2 ID of the host, we can look up its public IP address:

$ aws ec2 describe-instances \
    --instance-ids i-0123456789abcdef0 \
    --query Reservations[].Instances[].PublicIpAddress \
    --output text
203.0.113.2

Once the task is up and running, you can connect to the WireGuard container from your workstation with the following WireGuard config file:

# /etc/wireguard/wg-prod.conf

# local settings for my workstation
[Interface]
PrivateKey = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEE=
Address = 10.0.0.1/32

# remote settings for WireGuard container
[Peer]
PublicKey = fE/wdxzl0klVp/IR8UcaoGUMjqaWi3jAd7KzHKFS6Ds=
Endpoint = 203.0.113.2:51820
AllowedIPs = 10.10.0.0/16

See the WireGuard Point to Site Configuration article for a detailed explanation of each setting used in this config; but note that:

  • The PrivateKey used in this config file should correspond to the PublicKey setting for some peer listed in the container’s config file (when you generate a WireGuard key for a client in this scenario, you put the private key in the client’s local config, and put the public key in the container’s remote config).

  • The Address used in this config file should be the same as the AllowedIPs setting for the same peer in the container’s config file (in this example, we used 10.0.0.1 for my workstation).

  • The PublicKey used in this config file should correspond to the PrivateKey setting for the container’s config file (when you generate the WireGuard key for the server, you put the private key in the server’s config, and copy the public key to all the client configs).

  • The Endpoint used in this config file should be the public IP address (or public DNS entry) of the EC2 host running the WireGuard container, with the port matching the hostPort from the task’s definition. (Or if you’re using an NLB, this should be IP address and port of the NLB listener.)

  • The AllowedIPs used in this config file should be the CIDR block of the VPC in which the container is running (this directs the client to use the WireGuard connection to access any host in the VPC). If you want to use the WireGuard connection to access only a few subsets of this address block, you can specify those subsets instead (for example, 10.10.20.123/32, 10.10.21.0/24 instead of the full 10.10.0.0/16 block).

Assuming you have WireGuard installed on your local workstation, you can save the above config file as /etc/wireguard/wg-prod.conf, and run the following command on your workstation to start up the WireGuard connection to the remote container:

$ sudo wg-quick up wg-prod

In our example scenario, we have a web app running on port 8080 of an EC2 instance, with an IP address of 10.10.21.130 within the VPC. It’s protected by security group sg-0abcdef1234567890, and in the above Adjust Your Security Groups section, we granted the container’s host access to it. So now you can access that web app from your local workstation simply by navigating to http://10.10.21.130:8080 in a web browser; or by running the following cURL command:

$ curl 10.10.21.130:8080
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
...

View the Container Logs

With the minimal task definition we set up in the Define the WireGuard Task section above, the container’s logs are available only by SSHing into the container’s host, and running the following command (use the docker ps command to identify the container ID to use in place of 1234567890ab):

$ sudo docker logs 1234567890ab

 * /proc is already mounted
 * /run/lock: creating directory
 * /run/lock: correcting owner
   OpenRC 0.44.7.88ce4d9bb0 is starting up Linux 4.14.275-207.503.amzn2.x86_64 (x86_64) [DOCKER]

 * Caching service dependencies ... [ ok ]
Warning: `/etc/wireguard/wg0.conf' is world accessible
[#] iptables -t nat -A POSTROUTING ! -o wg0 -j MASQUERADE
[#] iptables -A FORWARD -o wg0 -m state --state ESTABLISHED,RELATED -j ACCEPT
[#] iptables -A FORWARD -o wg0 -j REJECT
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.0.2 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip -4 route add 10.0.0.1/32 dev wg0
 * Starting WireGuard interface wg0 ... [ ok ]

You’ll probably want to instead send these logs to some centralized logging service. If you’re using AWS CloudWatch for logs, you can do this by adding the following logConfiguration setting to the WireGuard container definition in your task definition JSON:

{
    "family": "wg-p2s",
    "containerDefinitions": [
        {
            "name": "wireguard",
            ...
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/prod/infra/wg-p2s",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ],
    ...
}

Set the awslogs-group option to whatever log group you like, the awslogs-region option to the region in which the task is defined, and the awslogs-stream-prefix to any prefix you want. If you use a log group that hasn’t been created yet, make sure you create it first; like via this AWS CLI command:

$ aws logs create-log-group --log-group-name /prod/infra/wg-p2s

Once you do that, you’ll be able to “tail” the container’s logs the next time you start it up:

$ aws logs tail /prod/infra/wg-p2s --follow
2022-05-06T00:24:05.055000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39  * /proc is already mounted
2022-05-06T00:24:05.057000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39  * /run/lock: creating directory
2022-05-06T00:24:05.057000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39  * /run/lock: correcting owner
2022-05-06T00:24:05.248000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39    OpenRC 0.44.7.10dab8bfb7 is starting up Linux 4.14.275-207.503.amzn2.x86_64 (x86_64) [DOCKER]
2022-05-06T00:24:05.248000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39  * Caching service dependencies ... [ ok ]
2022-05-06T00:24:05.344000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 Warning: `/etc/wireguard/wg0.conf' is world accessible
2022-05-06T00:24:05.349000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] iptables -t nat -A POSTROUTING ! -o wg0 -j MASQUERADE
2022-05-06T00:24:05.351000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] iptables -A FORWARD -o wg0 -m state --state ESTABLISHED,RELATED -j ACCEPT
2022-05-06T00:24:05.352000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] iptables -A FORWARD -o wg0 -j REJECT
2022-05-06T00:24:05.576000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] ip link add wg0 type wireguard
2022-05-06T00:24:05.577000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] wg setconf wg0 /dev/fd/63
2022-05-06T00:24:05.578000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] ip -4 address add 10.0.0.2/32 dev wg0
2022-05-06T00:24:05.581000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] ip link set mtu 1420 up dev wg0
2022-05-06T00:24:05.585000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39 [#] ip -4 route add 10.0.0.1/32 dev wg0
2022-05-06T00:24:05.586000+00:00 ecs/wireguard/57322a73f6a64d6d8164da05f071bc39  * Starting WireGuard interface wg0 ... [ ok ]

Troubleshooting

If you see a not supported error in the WireGuard container’s logs when it starts up, like the following:

[#] ip link add wg0 type wireguard
RTNETLINK answers: Not supported
Unable to access interface: Protocol not supported

It means the host does not have the WireGuard kernel module installed. See the Preparing the Host section at the start of this article for what to do about this.

If you see an operation not permitted error in the WireGuard container’s logs when it starts up, like the following:

getsockopt failed strangely: Operation not permitted
Unable to access interface: Operation not permitted

It means that the container does not have permission to set up a WireGuard interface. Make sure you granted the container the NET_ADMIN Linux capability, as shown in the Define the WireGuard Task section.

If the container’s logs show that its WireGuard interface started up OK, but when you try to connect the WireGuard client on your local workstation to the WireGuard container, you can’t seem to access anything in the container’s VPC, run the following command on your local workstation to check the status of your WireGuard interfaces:

$ sudo wg
interface: wg-prod
  public key: /TOE4TKtAqVsePRVR+5AA43HkAK5DSntkOCO7nYq5xU=
  private key: (hidden)
  listening port: 54918

peer: fE/wdxzl0klVp/IR8UcaoGUMjqaWi3jAd7KzHKFS6Ds=
  endpoint: 203.0.113.2:51820
  allowed ips: 10.10.0.0/16
  latest handshake: 3 minutes, 24 seconds ago
  transfer: 2.21 KiB received, 6.02 KiB sent

If your connection to the WireGuard container doesn’t show a latest handshake entry, it means that the WireGuard connection to the container has never been established. Double check that:

  1. The PrivateKey used in your local config file corresponds to the PublicKey setting for some peer listed in the container’s config file.

  2. The Address used in your local config file is the same as the AllowedIPs setting for the same peer in the container’s config file.

  3. The PublicKey used in your local config file corresponds to the PrivateKey setting for the container’s config file.

  4. The Endpoint used in your local config file is the public IP address (or public DNS entry) of the EC2 host running the WireGuard container, with the port matching the hostPort from the task’s definition. (Or if you’re using an NLB, it’s the IP address and port of the NLB listener.)

  5. The security groups used by the EC2 host (and the Network ACL of its subnet) allow inbound access from your public IP address to the UDP port matching the hostPort from the task’s definition. (Or if you’re using an NLB, the security groups used by the EC2 host allow inbound access from the VPC’s full CIDR block to the hostPort from the task’s definition; and the security groups used by the NLB allow inbound access from your public IP address to the NLB’s listener port.)

If your connection to the WireGuard container does show a latest handshake entry, it means that the WireGuard connection to the container is working. Double check that the security groups (and Network ACLs) of the resources to which you’re trying to connect allow access from the container’s EC2 host. In the Set Up the WireGuard Config File section, we configured the container to masquerade all the connections that it forwards from the WireGuard network — so the traffic forwarded by the container from your workstation to the VPC will appear to the VPC and all other AWS services as if it originated from the container’s EC2 host itself.

by Justin Ludwig
  • WireGuard
  • Cloud
  • AWS
  • Containers
  • Point to Site
    •