Securing Operational Technology With WireGuard

In this article, we’re going to take two OT devices that need to communicate with each other, and which are currently connected directly to a plant network, and put a simple Linux router in front of each. We’ll set up a firewall on each router that will prevent any malicious actor who’s gotten access to your plant network from accessing either of the devices, and secure the communication between the devices with WireGuard.

No software is perfect, but the Linux network stack has been battle-tested and hardened for 30 years. Most importantly, when security flaws are discovered, they are patched immediately; and modern Linux distributions make it safe and easy to apply those patches automatically. So you’re effectively replacing the under-scrutinized, under-maintained, and under-patched network stack of your OT devices with the fully-patched and hardened network stack of the Linux router in front of it.

And using a WireGuard VPN (Virtual Private Network) to secure the communication between your OT devices will protect you from a malicious actor being able to exploit any unpatched vulnerabilities in the application layer of those OT devices. When secured with WireGuard, nobody will be able to decrypt or manipulate the messages between those OT devices — or even probe those devices for network vulnerabilities at all.

So if you secure a pair (or larger group) of OT devices as described by this article, in order for a malicious actor to gain network access to one of those devices in order to exploit an unpatched vulnerability on the device, they would first have to either:

In other words, you’ve just reduced the risk of a remote attack on your OT devices by at least an order of magnitude.

In the example scenario we’ll use for this article, we’ll have two OT devices in two different locations at our plant that need to communicate with each other, and are currently connected via wired Ethernet network interface to the plant network. We’ll say the first device, which I’ll call Endpoint A, is an operator node that initiates connections to TCP port 1234 on the second device, which I’ll call Endpoint B, a processor node that accepts these TCP connections and actuates some equipment as a result (for example, opens and closes a bank of values).

We’ll unplug these endpoints from the network, and plug two Linux boxes in their place; set up those boxes to serve as routers for the endpoints; and plug the endpoints into a second NIC (Network Interface Card) on each of the boxes. The box in front of Endpoint A I’ll call Router α, and the box in front of Endpoint B I’ll call Router β.

The diagram below illustrates this scenario:

Router α will be attached to the plant network through its eth0 network interface, using an IP address of 192.168.1.123; and will create a separate network for Endpoint A through its eth1 network interface, using an IP address of 172.30.30.17 (as part of the 172.30.30.16/28 network). Additionally, Router α will have a wg0 WireGuard interface, using an IP address of 10.0.0.1 for it.

Similarly, Router β will be attached to the plant network through its eth0 network interface, using an IP address of 192.168.1.234; and will create a separate network for Endpoint B through its eth1 network interface, using an IP address of 172.30.30.33 (as part of the 172.30.30.32/28 network). Additionally, Router β will have a wg0 WireGuard interface, using an IP address of 10.0.0.2 for it.

Since Endpoint A always initiates connections to Endpoint B, and not vice versa, Endpoint A’s IP address does not need to be fixed — so Router α will just assign a dynamic IP address to Endpoint A using DHCP. In our example, this will be 17.30.30.18. Endpoint A needs to know Endpoint B’s IP address, however, so Router β will assign a fixed IP address of 17.30.30.34 to Endpoint B via DHCP.

These are the steps we’ll follow:

You’ll probably want to do all but the last two steps in the convenience of an office environment somewhere; and only when the routers are fully tested and ready would you move them into place at your plant, and schedule some downtime to plug the OT devices into them. If you have a remote pair of hands who can handle the switchover on one device while you switch over the other, you can probably do this in as little as 5 minutes of downtime. If possible, however, it’d be better to give yourself some extra time for troubleshooting, rebooting, and walking back and forth between the two devices, before they absolutely need to be up and in communication again.

Also, at the end of the article I’ve included some instructions for how to handle a few variants of this example scenario:

The first thing to do is find two computers that can serve as the routers. Pretty much any old PC boxes that you have laying around will work fine for this — unless you’re planning on putting a bunch of high-network-throughput OT devices behind these routers, you probably won’t be stressing their existing processor or network cards at all. Make sure you add a second NIC to each box if it doesn’t already have one.

If you really want to go the cheap route, you can even use Raspberry Pis for these boxes (and add a second network interface to each via USB dongle) — but if the performance and reliability of the connection between the two OT devices is critical, you’ll probably want to use hardware that’s purpose-built for networking applications. You can find PCs like this in the $200-$300 range these days (such as the Protectli Vault; or try searching for “mini pc dual nic” at your hardware supplier of choice).

You won’t need much in the way of processing power, RAM, or disk space for these routers. 1 GB of RAM and 8 GB of disk space will be plenty; and whatever CPU the box comes with will probably be fine. (If you do have serious network-throughput requirements, however — say anything above 100 Mbps — you should do a load test with your boxes forwarding data over WireGuard to one another in the lab to make sure they can handle your traffic when you deploy them to production.)

For this guide, I’m going to describe how to install and configure Rocky Linux 8. Rocky Linux is one of the successors to the CentOS project, which maintains packages and updates for all the free or open-source software in RHEL (Red Hat Enterprise Linux). You should be able to substitute any 8.x version of RHEL or its derivatives (like AlmaLinux) for Rocky Linux without any trouble (but note that each major version, like 6.x or 7.x, is significantly different, so the following instructions won’t necessarily work with them).

Download the x86_64 minimal ISO image for Rocky Linux from the Rocky Linux Downloads page. Once downloaded, use it to create a bootable USB drive with a tool like Etcher, Ventoy, etc (you’ll need a 3 GB USB stick or bigger).

For each router box, plug an Ethernet cable attached to your local network into its primary NIC, and also plug in a monitor, keyboard, mouse, and your bootable USB stick. Then start up the box (use its BIOS menu to boot from your USB stick if it doesn’t do so automatically).

In the Localization section of the installer, select your preferred language, keyboard layout, and timezone.

In the Software Selection subsection of the Software section, choose Minimal Install (with no “Additional software” — we’ll install the specific software packages we need later — plus even the “minimal” install actually installs and enables a few key services, such as an SSH server and firewall).

In the Installation Destination subsection of the System section, the default disk selected should be fine, as should the “Automatic” storage configuration.

In the Network & Host Name subsection of the System section, turn on your primary NIC (I’m going to call it eth0, but it’s probably actually named something else like ens3 or enp0s3 etc). Also, for your own convenience, you may want to change the hostname to a meaningful name (like say router-a or router-b etc).

Then in the same Network & Host Name subsection, turn on the secondary NIC (I’m going to call it eth1), and configure its IPv4 Settings to Manual (via the “Method” field). Add an Addresses entry to its IPv4 Settings with the following values (for Router α):

For Router β, set its IP Address to 172.30.30.33 instead of 172.30.30.17 (but don’t worry too much about the exact settings for the secondary NICs right now — we’ll fix up their settings via config file later on).

In the Root Password subsection of the User Settings section, enter a strong random password (and write it down).

In the User Creation subsection of the User Settings section, create a user for yourself. Make the user an administrator, require a password to use the account, and enter a strong random password (write it down also). Later you can create user accounts for anyone else who needs to administer this box.

Now click the Begin Installation button. When installation is finished, remove the USB stick and click the Reboot System button.

At the login screen, enter your username and password (which you just set up in the User Creation subsection of the installer).

The first thing you should do after logging in for the first time is run the following command, which will upgrade all the packages currently installed to their latest versions from the package repositories:

The second thing you should do is set up automatic package updates. Start by installing the “dnf-automatic” package:

Then edit its configuration file at /etc/dnf/automatic.conf to make sure both its download_updates and apply_updates configuration settings are set to yes:

The “minimal” install comes with a basic version of Vim installed at /usr/bin/vi; so if you’re comfortable with Vim keybindings, you can edit this configuration file with the following command:

If you prefer Emacs, install it via sudo dnf install emacs; or if you prefer neither, install Nano and use it to edit the configuration file:

Everything you need to know to use Nano is listed at the bottom of the screen when Nano is running — just navigate around the screen with the arrow keys, type to make edits, and use Ctrl-O to save your changes.

Then enable the automatic-update service with the following command:

After setting up automatic updates, make sure you have an NTP client installed (to combat the natural drift of the system’s internal clock). You can install the NTP client Chrony and enable it with the following commands:

Now install WireGuard on each router with the following commands:

(But if you’re using RHEL itself instead of one of its derivatives, substitute the following for the first command:)

Then run the following command on Router α to generate a WireGuard key pair for it:

And do the same on Router β:

The first file, router-a.key (or router-b.key) will contain the private key, and the second file, router-a.pub (or router-b.pub) will contain the matching public key. Copy router-a.pub to Router β and router-b.pub to Router α (don’t copy the private keys — they should never leave the computer on which they were generated).

For this example, let’s say the files we generated contained the following keys:

On Router α, edit /etc/wireguard/wg0.conf, and configure it like this:

On Router β, edit /etc/wireguard/wg0.conf to configure it like this:

Make sure you replace the example private and public keys in the above configuration files with the content from the router-a.key, router-b.pub, router-b.key, and router-a.pub files you generated.

Note that the configuration for each router contains its own internal WireGuard address in the Address setting (10.0.0.1 for Router α and 10.0.0.2 for Router β), but the other’s public IP address in the Endpoint setting (192.168.1.123 for Router α and 192.168.1.234 for Router β), as well as the other’s private subnet in the AllowedIPs setting (172.30.30.16/28 for Router α and 172.30.30.32/28 for Router β).

On each router, enable the WireGuard service with the following command:

See the Site to Site Configuration guide for a more in-depth explanation of these WireGuard configuration settings, as well as some troubleshooting tips (we’ll configure routing and our firewall a little differently in this article, so ignore those sections from the Site to Site guide).

You won’t be able to connect from Router α to Router β yet, however, until we replace the default firewall with a custom one that allows access to the WireGuard listen ports on the routers (UDP port 51820 on both routers). For now, this is what you should see if you run the WireGuard status command, wg show from Router α:

And from Router β:

The “minimal” install comes with firewalld installed and running by default. Firewalld is great for endpoints, but a little too minimal for routers, so we’ll remove it and use the old workhorse iptables directly:

Edit the /etc/sysconfig/iptables file to make it look like this:

Make sure you replace eth0 and eth1 in the above with the actual name of the box’s Ethernet interfaces. Then run the following command to flush the old iptables chains, and load the updated rule set:

Next, to enabling packet forwarding, create a /etc/sysctl.d/local.conf file, and set it to this content:

Then run the following command to apply it (along with all other sysctl conf files):

If you run the iptables status command, iptables-save, you should see basically the contents of /etc/sysconfig/iptables echoed back at you (plus perhaps some empty tables with no rules like *security, *raw, *mangle, *nat, etc):

Now you have a firewall up and running on the router that will:

For more examples of using iptables with WireGuard, see the WireGuard Access Control With Iptables guide (and its Site to Site section in particular).

Next, we’ll set up a DHCP server on each router. This will allow the OT devices behind the routers to configure their network settings automatically with the proper IP address, netmask, and gateway settings.

First, make sure you’ve set up the secondary network interface correctly with a static IP address on each router. On Router α, your /etc/sysconfig/network-scripts/ifcnf-eth1 file should look like this (where eth1 is the name of your secondary interface):

On Router β, your /etc/sysconfig/network-scripts/ifcnf-eth1 file should look look the same, but with an IP address of 172.30.30.33. Run the following command if you need to make changes to this file:

If you run the ip address command, you should see something like the following:

The primary network interface (eth0 in this example) has an IP address and netmask (192.168.40.11/24) assigned to it from the DHCP server on the network into which it’s plugged. The secondary network interface (eth1) has the IP address and netmask (172.30.30.17/28) which you manually specified.

Write down the MAC addresses and interface names of the two NICs in your equipment inventory list (eg 06:bd:e1:00:b8:ef (eth0) plant network, a0:79:14:11:8c:ec (eth1) private network).

Next, run the following command to install the ISC DHCP server on each router:

Then edit the /etc/dhcp/dhcpd.conf file on Router α to make it look like this:

And edit the /etc/dhcp/dhcpd.conf file on Router β to make it look like this:

Make sure you replace the MAC address above in the host endpoint-b declaration with the actual MAC address of Endpoint B. If you don’t know its MAC address, but you do know its current IP address, you can find its MAC address by running the following two commands in succession from a machine that’s plugged into to the same subnet as Endpoint B (say Endpoint B currently has an IP address of 192.168.1.222 on the plant network):

On each router, enable the DHCP server with the following command:

If you check the logs for the DHCP server, you should see something like the following:

When you plug in a device into the router’s secondary NIC, after a few seconds you should see the following in the logs, indicating the device asked the DHCP server (172.30.30.17) for an IP address, and the DHCP server gave it one (172.30.30.18):

Before deploying your routers to production, test them out in a lab environment. Plug the primary NIC of each router (eth0) into the lab network. Plug a different test machine (like a spare laptop or other PC) into the secondary NIC of each router (eth1). If the test machines support WiFi, turn off WiFi (to make sure they use the router into which they’ve been plugged as their default gateway).

Run the following command on each router to display its main routing table. One Router α, it should look like this:

And on Router β, it should look like this

The routers should both be assigned an address from the lab’s DHCP server for their eth0 network interface (192.168.40.11 for Router α and 192.168.40.12 for Router β in this example). Update the Endpoint setting in the /etc/wireguard/wg0.conf file on each router to set each peer endpoint to the other router’s current eth0 IP address:

On Router α /etc/wireguard/wg0.conf should look like this:

And on Router β /etc/wireguard/wg0.conf should look like this:

Restart WireGuard on both routers with the following command:

Now run a test TCP service, like a dummy webserver, on the test box simulating Endpoint B (the one plugged into the secondary interface of Router β). You can use Python to run a simple webserver (serving the contents of your local directory) on TCP port 1234 with the following command:

From the test box simulating Endpoint A (the one plugged into the secondary interface of Router α), try to connect to the dummy webserver at an URL of http://172.30.30.35:1234 (or whatever IP address was assigned to the box simulating Endpoint B):

You should get a response from Endpoint B (if not, try the Basic Troubleshooting steps from the WireGuard Site to Site Configuration guide).

Now move the two routers to their permanent location at the plant. Ideally they would be close to either the OT endpoints or to the switch or router into which the endpoints are currently connected, so you won’t have to run any new cable (or power); and in a place where they won’t be tripped over, smashed, splattered, etc. For each router, make sure you have the right length power cord for the location, the right length of new Ethernet cable, and that the current cable to the OT device is the right length to plug into the router (or run a new cable of the right length).

Power up each router and make sure it boots. Double check the output of wg show, iptables-save, and journalctl -u dhcpd and make sure it matches the expected output from above (bring along a small monitor and keyboard so you can check).

If you can, update the plant network’s DHCP server to give each router a fixed IP address, using the MAC address of each router’s primary NIC (eth0). If you can’t, but can plug the routers into an open port on a switch/router on the plant’s network, do so now, to at least get a dynamic IP address assigned for its primary NIC.

If you can get an IP address assigned to the primary NIC of one or both routers, update the Endpoint setting in the /etc/wireguard/wg0.conf file of the opposite router with the IP address of its pair. For example, if you assigned 192.168.1.123 to the primary NIC for Router α, and 192.168.1.234 to the primary NIC of Router β, update /etc/wireguard/wg0.conf on Router α to this:

And update /etc/wireguard/wg0.conf on Router β to this:

Then restart WireGuard on both routers with the following command:

Now they’re both ready to deploy to production!

When you deploy to production, ideally you’d have a monitor and keyboard plugged into each router, and a person standing by each to troubleshoot if something goes wrong (as well as by the OT devices, if they’re in different locations than the routers themselves).

For each OT device, unplug one end of its connection to the plant network. If you unplugged the device’s end of the connection, plug the old cable into the router’s primary NIC (eth0). Plug the new Ethernet cable into device and the into the router’s secondary NIC (eth1).

If you unplugged the network’s end of the connection, plug the old cable into the router’s secondary NIC (eth1). Plug the new Ethernet cable into the switch/router port you just unplugged, and the other end of the new cable into the router’s primary NIC (eth0).

If you hadn’t assigned an IP address to the primary NIC of one or both of the routers in the Prep for Production step above, run the following command on the router to show the IP address that’s been assigned to it (replace eth0 with the actual name of the router’s primary network interface):

Update the Endpoint setting in the /etc/wireguard/wg0.conf file of the opposite router with the IP address of its pair. For example, if 192.168.1.123 was assigned to Router α, and 192.168.1.234 assigned to Router β, update /etc/wireguard/wg0.conf on Router α to this:

And update /etc/wireguard/wg0.conf on Router β to this:

Then restart WireGuard with the following command:

Now update Endpoint A with the new IP address for Endpoint B (172.30.30.34 in our example, assigned by the DHCP service on Router β). You can check the DHCP assignments the router has made from the output of journalctl -u dhcpd (or by inspecting the file at /var/lib/dhcp/dhcpd.leases).

The control panel on Endpoint A should now be interacting with Endpoint B as normal. If it’s not, check the output of wg show, iptables-save, and journalctl -u dhcpd on the routers as described in the above sections. The output of wg show on both routers should indicate a recent handshake and a certain amount of data sent and received — if not, follow the Basic Troubleshooting steps from the WireGuard Site to Site Configuration guide.

If you need a fixed IP address for both OT endpoints (for example, both Endpoint A and Endpoint B need to be configured with the other’s IP address), configure the /etc/dhcp/dhcpd.conf file on Router α to include a host entry for Endpoint A, and give Endpoint A a fixed IP address (just like you did for Endpoint B on Router β):

Make sure you also update the range setting for the subnet entry in the same file to exclude the fixed IP address from the range of dynamic addresses that router may hand out:

Also make sure you restart the DHCP server if you change its configuration:

Then edit the /etc/sysconfig/iptables file on Router α to allow new connections to be made at whatever port Endpoint A receives connections from Endpoint B (2345 for example).

Run the following command to flush the old iptables chains, and load the updated rule set:

If one or both of the OT devices don’t use DHCP, you can skip the Set Up DHCP section for its router. Instead, configure the OT device itself with a fixed IP address, netmask, and gateway.

For Endpoint A, you’d configure the device with the following settings:

For Endpoint B, you’d use these settings:

Other network settings on the OT devices, like DNS or NTP servers, you can leave as is.

If one of the endpoints (like the operator node, Endpoint A in this example) is just a Windows PC (or running some other off-the-shelf operating system that supports WireGuard directly), you don’t need to set up a router for it at all. Just install WireGuard on the endpoint itself, and configure WireGuard on it the way you would have its router.

For example, if Endpoint A is just a Windows PC, install WireGuard on it, and configure it like this:

On Router β, you just need to change its WireGuard configuration a little:

For example, update /etc/wireguard/wg0.conf like this on Router β:

If you actually have several different Windows PCs (or tablet computers etc) that can connect to Endpoint B, generate a unique WireGuard peer pair for each, assign each it’s own WireGuard IP address, and add a separate [Peer] entry for each in Router β’s WireGuard configuration.

For example, say you have have three PCs that can connect to Endpoint B to monitor it: Endpoint A1, Endpoint A2, and Endpoint A3.

On Endpoint A1, create a new WireGuard tunnel, and configure it like this (using the private key generated for you instead of the example key):

Copy the public key generated for you (MMsBeTT9v/7XNWB8a/jSMn9O8olPVNduUwvUPJ6eB14= in this example) to Router β.

On Endpoint A2, do the same, and configure it like this (using the private key generated for you instead of the example key):

Copy the public key generated for you (kAYKIv6gpBDqueaVNOsY8ddKmIC2dnnXJQ0iKzWxfBI= in this example) to Router β.

On Endpoint A3, same thing; configure WireGuard like this (using the private key generated for you instead of the example key):

Copy the public key generated for you (af24MfZ5LzUedF5WlpK2+O8602g2fmiKO8dYdv8dUyI= in this example) to Router β.

Then on Router β, use the public key generated for each endpoint to configure a separate WireGuard [Peer] entry for each:

You may have multiple OT devices you want to put behind the same router. For example, say you have three similar processor nodes each situated nearby the others, and all three receive connections from the same operator node. In that case, it would make sense to put all three on the same private network behind Router β.

If Router β only has two NICs, that’s okay — just add an Ethernet switch to it (you can get a new 5-port switch for less than $20). Plug the Ethernet cable from Router β’s secondary NIC into the uplink port on the switch, and plug the other OT devices into the other ports.

You’ll probably also want to set up a fixed IP address for each OT device (I’ll call them Endpoint B1, Endpoint B2, and Endpoint B3). Edit the /etc/dhcp/dhcpd.conf file on Router β to add a host entry for each (do this before you plug them into the switch):

Make sure you also update the range setting for the subnet entry in the same file to exclude those fixed IP addresses from the range of dynamic addresses that router may hand out:

Also make sure you restart the DHCP server if you change its configuration:

Then edit the /etc/sysconfig/iptables file to allow new connections to be made to each of the three devices at TCP port 1234 (or at whatever port Endpoint A connects to them):

Run the following command to flush the old iptables chains, and load the updated rule set:

You’ll now be able to securely access the OT devices from Endpoint A as 172.30.30.34 for Endpoint B1, 172.30.30.35 for Endpoint B2, and 172.30.30.36 for Endpoint B3.

If you want to allow the routers to be administered via SSH when you or one of your colleagues connect a laptop to the plant network, you can do so by enabling SSH access through WireGuard.

First, add the following rule to the /etc/sysconfig/iptables file, allowing for SSH access over WireGuard:

The full /etc/sysconfig/iptables file on Router α, for example, would now look something like this:

Run the following command to flush the old iptables chains, and load the updated rule set:

Then create a new WireGuard tunnel on your laptop, and configure it like this (using the private key generated for you instead of the example key, and the actual public keys you generated originally for Router α and Router β):

Have your other colleagues who also need administrative access to the routers do the same (let’s say there are two: Alice and Bob). Copy the public key generated for your WireGuard tunnel (MMsBeTT9v/7XNWB8a/jSMn9O8olPVNduUwvUPJ6eB14= in this example), as well as Alice’s (kAYKIv6gpBDqueaVNOsY8ddKmIC2dnnXJQ0iKzWxfBI=) and Bob’s (af24MfZ5LzUedF5WlpK2+O8602g2fmiKO8dYdv8dUyI=), to both Router α and Router β. Also copy public SSH keys for yourself, Alice, and Bob to the routers.

Edit the /etc/wireguard/wg0.conf file on each of the routers to add a [Peer] entry for yourself and each of your colleagues; for example, like this on Router α:

And like this on Router β:

Restart WireGuard on both of the routers with the following command:

Create user accounts on both the routers for Alice and Bob (if you haven’t already). For Alice, you can do it like this (where you for example copied her SSH key to the router as alice_ed25519.pub):

Do the same for Bob. For yourself, do this (where you for example copied your SSH key to the router as me_ed25519.pub):

Now edit the /etc/ssh/sshd_config file (note the d in the name) to explicitly turn off root logins, turn on public-key authentication, and turn off password authentication. Find these settings in the /etc/ssh/sshd_config (uncomment them if necessary), and set them like so:

After making those changes, restart the SSH server:

On your laptop, while connected to the plant network, start the WireGuard tunnel you created to administer these routers, and try connecting to Router α with the user account and SSH key you set up for it (where the -i flag points to the private key file matching the public key you set up on Router α, and me is the username you set up for yourself):

You can do the same for Router β, using the WireGuard address for Router β: