Homelab 2022 Part 3 - Networking

Overview

While it's tempting to use a simple flat network at home, it's always more fun to set up an over the top enterprise level system with all the usual bells and whistles.

Here I'm using a virtualised core router for VMs and to provide internet to a handful of clients. I'll show the basic layout of the network and then dig into some of the more interesting features such as BGP over wireguard for redundant WAN uplinks and policy based routing.

There is some contention on the use of BGP in smaller environments, mostly from diy users or SOHO operators as it tends to get a (unwarranted) reputation for being a large complex tool fraught with peril and despair. But the reality is it's not more difficult to use than any other IGP, and has many advantages and features that traditional protocols lack.

Requirements

  • Isolation of interior services from exterior shared network
  • Make use of redundant WAN uplinks (BGP multipath)
  • Allow (near) seamless transition from faulted uplinks (BFD)
  • Allow inbound routing through static external addresses regardless of transport (CGNAT)
  • External VPN login for a handful of clients

Below is an interactive map showing the network as a whole, hopefully it helps with understanding the configuration:


The same thing in tabulated form.

Subnet Purpose
5.5.5.2/24 Our pseudo public address
10.0.0.0/20 Direct attached clients and devices (trusted network users)
10.0.8.0/20 VPN clients
10.0.16.0/20 DMZ servers and services
10.0.32.0/20 Storage network
10.251.1.0/30 WG tunnel transit network 1
10.251.2.0/30 WG tunnel transit network 2
10.251.3.0/30 WG tunnel transit network 3
DHCP WAN uplink 1 (4G)
DHCP WAN uplink 2 (4G)
DHCP WAN uplink 3 (p2p wireless)
DHCP Exterior network (provides LB over all uplinks)

Implementation

To actually implement this I am using VyOS 1.4 rolling on two routers and running a iBGP autonomous system to do dynamic route distribution and multipathing/failover. If you've not played around with VyOS before, the UX is largely inspired by JunOS and to some degree IOS, the feature set is very decent and even rolling is quite stable.

As can be seen from the map above we are using a star topology and thus losing our core router will result in complete failure of the network. For production use you would definitely want to set up a VRRP group with a backup instance.

You'll also need somewhere to put the second router. I'm using linode as they support installing and running custom operating systems without purchasing a dedicated box, which puts your running costs at $5 USD/mo.

Interfaces

We can either set up a heap of VLANs on a managed switch and bring them in as a trunk, or connect each gateway to a physical port on the server. I my case I did something in between, a trunk goes into the server where the hypervisor untags each VLAN into separate vswitches, this has the advantage that each network appears as a discrete interface but the disadvantage that's there's more places where things can break.

We have our 4 LANs listed with massively over provisioned address spaces for good measure. The two we provide internet access on (Internal & DMZ) have a route policy which will be explained later and is optional.

There is also a trio of wireguard tunnels that have a fwmark, we will use this mark to assign each tunnel to a particular link later on in our routing policy. At the other end of these tunnels is our VPS, this will be performing the link aggregation and define the true edge of our network (wg0 is our internal/external access instance, it will be used for road warrior users so skip it if you don't need that feature).

FYI you can generate your own keys for the wireguard tunnels using built in tools:

vyos@vyos:~$ generate pki wireguard key-pair
Private key: WMVVJuE3rj98l1h0oaGkj4ZvH9+aAdH3/5kYNMLq/lI=
Public key: SXBrG+WOnGN0uLipimN0UY+cjysQLWY9fbQPagFdaWI=

vyos@vyos:~$ generate pki wireguard preshared-key
Pre-shared key: FYpxSLGtaoCQ2o7UDsQbPt8V934G1MGgTMqgbC55+dc=

For completeness we'll also set up source NAT since our modems are far too cheap to implement RIP or anything else useful.

interfaces {
    ethernet eth0 {
        address dhcp
        description "Untrusted LAN"
    }
    ethernet eth1 {
        address 10.0.16.1/20
        description DMZ
        policy {
            route WAN-PBR
        }
    }
    ethernet eth2 {
        address 10.0.32.1/20
        description Storage
        mtu 9000
    }
    ethernet eth3 {
        address 10.0.0.1/20
        description Internal
        policy {
            route WAN-PBR
        }
    }
    ethernet eth4 {
        address dhcp
        description "Optus 1"
    }
    ethernet eth5 {
        address dhcp
        description "Optus 2"
    }
    ethernet eth6 {
        address dhcp
        description NBN
    }
    wireguard wg0 {
        address 10.0.8.1/20
        description "Internal Access"
        peer lappy {
            allowed-ips 10.0.8.2/32
            preshared-key ****************
            public-key ****************
        }
        policy {
            route WAN-PBR
        }
        port 50100
        private-key xxxxxx
    }
    wireguard wg1 {
        address 10.251.1.2/30
        description "Uplink 1"
        fwmark 101
        peer linode {
            address 5.5.5.2
            allowed-ips 10.251.1.1/32
            allowed-ips 0.0.0.0/0
            port 51820
            preshared-key ****************
            public-key ****************
        }
        private-key xxxxxx
    }
    wireguard wg2 {
        address 10.251.2.2/30
        description "Uplink 2"
        fwmark 102
        peer linode {
            address 5.5.5.2
            allowed-ips 10.251.2.1/32
            allowed-ips 0.0.0.0/0
            port 51821
            preshared-key ****************
            public-key ****************
        }
        private-key xxxxxx
    }
    wireguard wg3 {
        address 10.251.3.2/30
        description "Uplink 3"
        fwmark 103
        peer linode {
            address 5.5.5.2
            allowed-ips 10.251.3.1/32
            allowed-ips 0.0.0.0/0
            port 51822
            preshared-key ****************
            public-key ****************
        }
        private-key xxxxxx
    }
}
nat {
    source {
        rule 10 {
            outbound-interface eth4
            translation {
                address masquerade
            }
        }
        rule 20 {
            outbound-interface eth5
            translation {
                address masquerade
            }
        }
        rule 30 {
            outbound-interface eth6
            translation {
                address masquerade
            }
        }
    }
}
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'Untrusted LAN'
set interfaces ethernet eth1 address '10.0.16.1/20'
set interfaces ethernet eth1 description 'DMZ'
set interfaces ethernet eth1 policy route 'WAN-PBR'
set interfaces ethernet eth2 address '10.0.32.1/20'
set interfaces ethernet eth2 description 'Storage'
set interfaces ethernet eth2 mtu '9000'
set interfaces ethernet eth3 address '10.0.0.1/20'
set interfaces ethernet eth3 description 'Internal'
set interfaces ethernet eth3 policy route 'WAN-PBR'
set interfaces ethernet eth4 address 'dhcp'
set interfaces ethernet eth4 description 'Optus 1'
set interfaces ethernet eth5 address 'dhcp'
set interfaces ethernet eth5 description 'Optus 2'
set interfaces ethernet eth6 address 'dhcp'
set interfaces ethernet eth6 description 'NBN'
set interfaces wireguard wg0 address '10.0.8.1/20'
set interfaces wireguard wg0 description 'Internal Access'
set interfaces wireguard wg0 peer lappy allowed-ips '10.0.8.2/32'
set interfaces wireguard wg0 peer lappy preshared-key '****************'
set interfaces wireguard wg0 peer lappy public-key '****************'
set interfaces wireguard wg0 policy route 'WAN-PBR'
set interfaces wireguard wg0 port '50100'
set interfaces wireguard wg0 private-key 'xxxxxx'
set interfaces wireguard wg1 address '10.251.1.2/30'
set interfaces wireguard wg1 description 'Uplink 1'
set interfaces wireguard wg1 fwmark '101'
set interfaces wireguard wg1 peer linode address '5.5.5.2'
set interfaces wireguard wg1 peer linode allowed-ips '10.251.1.1/32'
set interfaces wireguard wg1 peer linode allowed-ips '0.0.0.0/0'
set interfaces wireguard wg1 peer linode port '51820'
set interfaces wireguard wg1 peer linode preshared-key '****************'
set interfaces wireguard wg1 peer linode public-key '****************'
set interfaces wireguard wg1 private-key 'xxxxxx'
set interfaces wireguard wg2 address '10.251.2.2/30'
set interfaces wireguard wg2 description 'Uplink 2'
set interfaces wireguard wg2 fwmark '102'
set interfaces wireguard wg2 peer linode address '5.5.5.2'
set interfaces wireguard wg2 peer linode allowed-ips '10.251.2.1/32'
set interfaces wireguard wg2 peer linode allowed-ips '0.0.0.0/0'
set interfaces wireguard wg2 peer linode port '51821'
set interfaces wireguard wg2 peer linode preshared-key '****************'
set interfaces wireguard wg2 peer linode public-key '****************'
set interfaces wireguard wg2 private-key 'xxxxxx'
set interfaces wireguard wg3 address '10.251.3.2/30'
set interfaces wireguard wg3 description 'Uplink 3'
set interfaces wireguard wg3 fwmark '103'
set interfaces wireguard wg3 peer linode address '5.5.5.2'
set interfaces wireguard wg3 peer linode allowed-ips '10.251.3.1/32'
set interfaces wireguard wg3 peer linode allowed-ips '0.0.0.0/0'
set interfaces wireguard wg3 peer linode port '51822'
set interfaces wireguard wg3 peer linode preshared-key '****************'
set interfaces wireguard wg3 peer linode public-key '****************'
set interfaces wireguard wg3 private-key 'xxxxxx'
set nat source rule 10 outbound-interface 'eth4'
set nat source rule 10 translation address 'masquerade'
set nat source rule 20 outbound-interface 'eth5'
set nat source rule 20 translation address 'masquerade'
set nat source rule 30 outbound-interface 'eth6'
set nat source rule 30 translation address 'masquerade'

Firewall

Firewalls are pretty straight forward, all we want to do is keep the weirdo's out while not locking ourself out of the router. VyOS supports zone based firewall rules as well as traditional per-interface ACLs, to me zones are superior and break the habit some users and router interface designers have where NAT and firewalls get melted into one thing or become a spreadsheet a mile long.

That said, in this scenario we have 5 zones which may or may not allow cross-network routing, having to type out the rules of all of these zones by hand is a bit... excessive. An alternative is to come up with our security policy ahead of time and programmatically generate the firewall rules as needed.

For this I have a short script, it will generate the boiler plate commands with a default action for every zone defined. Each zone to zone relationship can have it's default action overwritten to allow communication in a particular direction. Zone actions must be named {a}-from-{b} format in order to be effective.

package main

import (
        "fmt"
)

var (
        // Set the default action if no zone-from-zone action is specified
        defaultAction = "reject"

        // Disable connection logging
        disableLogging = false

        // Whether or not to allow cross zone ICMP ping for ipv6 addresses
        allowIPv6ICMP = true

        // Don't generate any IPv4 rules or policies
        disableIPv4 = false

        // Don't generate any IPv6 rules or policies (I sure hope you don't do this)
        disableIPv6 = false
)

// List out each zone and it's corresponding interfaces, if no interface is
// provided it will be created as a local-zone for intermediate processing
var zones = map[string][]string{
        "untrust": {"eth0", "eth4", "eth5", "eth6"}, // Untrusted LAN/WAN
        "trust":   {"wg0", "eth3"},                  // Trusted network users
        "dmz":     {"eth1"},                         // Services
        "storage": {"eth2"},                         // Shares and iSCSI targets
        "linode":  {"wg1", "wg2", "wg3"},            // Linode server and gateway for priority traffic
}

// Set the non-default actions using zone-from-zone nomenclature
var zoneActions = map[string]string{
        // Allow outbound internet access
        "untrust-from-trust": "accept",
        "untrust-from-dmz":   "accept",
        // Allow DMZ access from all but untrust zone
        "dmz-from-trust":   "accept",
        "dmz-from-storage": "accept",
        "dmz-from-linode":  "accept",
        // Allow DMZ services and trusted users to access shares
        "storage-from-trust": "accept",
        "storage-from-dmz":   "accept",
        // Allow DMZ services and trusted users to route traffic via linode
        "linode-from-trust": "accept",
        "linode-from-dmz":   "accept",
}

// Additional flags for output
var (
        genDefaultRules = true
        genZonePolicy   = true
)

func main() {
        for zone, interfaces := range zones {
                fmt.Print("# Zone ", zone, "\n")

                if genZonePolicy {
                        fmt.Print("set zone-policy zone ", zone, " description '", zone, " zone'\n")

                        if len(interfaces) < 1 {
                                fmt.Print("set zone-policy zone ", zone, " local-zone\n")
                        } else {
                                for _, iface := range interfaces {
                                        fmt.Print("set zone-policy zone ", zone, " interface ", iface, "\n")
                                }
                        }
                }

                for from := range zones {
                        if from == zone {
                                continue
                        }

                        name := zone + "-from-" + from

                        action, ok := zoneActions[name]
                        if !ok {
                                action = defaultAction
                        }
                        if genDefaultRules {
                                defaultRules(name, action)
                        }
                        if allowIPv6ICMP {
                                v6ICMP(name)
                        }

                        if genZonePolicy {
                                if !disableIPv4 {
                                        fmt.Print("set zone-policy zone ", zone, " from ", from, " firewall name ", name, "-v4\n")
                                }
                                if !disableIPv6 {
                                        fmt.Print("set zone-policy zone ", zone, " from ", from, " firewall ipv6-name ", name, "-v6\n")
                                }
                        }
                        fmt.Println()
                }
        }
}

func defaultRules(name, action string) {
        if !disableIPv4 {
                fmt.Print("set firewall name ", name, "-v4 default-action ", action, "\n")
                if !disableLogging {
                        fmt.Print("set firewall name ", name, "-v4 enable-default-log\n")
                        fmt.Print("set firewall name ", name, "-v4 rule 20 log enable\n")
                }
                fmt.Print("set firewall name ", name, "-v4 rule 10 action accept\n")
                fmt.Print("set firewall name ", name, "-v4 rule 10 state established enable\n")
                fmt.Print("set firewall name ", name, "-v4 rule 10 state related enable\n")
                fmt.Print("set firewall name ", name, "-v4 rule 20 action drop\n")
                fmt.Print("set firewall name ", name, "-v4 rule 20 state invalid enable\n")
        }
        if !disableIPv6 {
                fmt.Print("set firewall ipv6-name ", name, "-v6 default-action ", action, "\n")
                if !disableLogging {
                        fmt.Print("set firewall ipv6-name ", name, "-v6 enable-default-log\n")
                        fmt.Print("set firewall ipv6-name ", name, "-v6 rule 20 log enable\n")
                }
                fmt.Print("set firewall ipv6-name ", name, "-v6 rule 10 action accept\n")
                fmt.Print("set firewall ipv6-name ", name, "-v6 rule 10 state established enable\n")
                fmt.Print("set firewall ipv6-name ", name, "-v6 rule 10 state related enable\n")
                fmt.Print("set firewall ipv6-name ", name, "-v6 rule 20 action drop\n")
                fmt.Print("set firewall ipv6-name ", name, "-v6 rule 20 state invalid enable\n")
        }
}

func v6ICMP(name string) {
        fmt.Print("set firewall ipv6-name ", name, "-v6 rule 30 action accept\n")
        fmt.Print("set firewall ipv6-name ", name, "-v6 rule 30 protocol icmpv6\n")
        if !disableLogging {
                fmt.Print("set firewall ipv6-name ", name, "-v6 rule 30 log enable\n")
        }
}
# Zone storage
set zone-policy zone storage description 'storage zone'
set zone-policy zone storage interface eth2
set firewall name storage-from-untrust-v4 default-action reject
set firewall name storage-from-untrust-v4 enable-default-log
set firewall name storage-from-untrust-v4 rule 20 log enable
set firewall name storage-from-untrust-v4 rule 10 action accept
set firewall name storage-from-untrust-v4 rule 10 state established enable
set firewall name storage-from-untrust-v4 rule 10 state related enable
set firewall name storage-from-untrust-v4 rule 20 action drop
set firewall name storage-from-untrust-v4 rule 20 state invalid enable
set firewall ipv6-name storage-from-untrust-v6 default-action reject
set firewall ipv6-name storage-from-untrust-v6 enable-default-log
set firewall ipv6-name storage-from-untrust-v6 rule 20 log enable
set firewall ipv6-name storage-from-untrust-v6 rule 10 action accept
set firewall ipv6-name storage-from-untrust-v6 rule 10 state established enable
set firewall ipv6-name storage-from-untrust-v6 rule 10 state related enable
set firewall ipv6-name storage-from-untrust-v6 rule 20 action drop
set firewall ipv6-name storage-from-untrust-v6 rule 20 state invalid enable
set firewall ipv6-name storage-from-untrust-v6 rule 30 action accept
set firewall ipv6-name storage-from-untrust-v6 rule 30 protocol icmpv6
set firewall ipv6-name storage-from-untrust-v6 rule 30 log enable
set zone-policy zone storage from untrust firewall name storage-from-untrust-v4
set zone-policy zone storage from untrust firewall ipv6-name storage-from-untrust-v6

set firewall name storage-from-trust-v4 default-action accept
set firewall name storage-from-trust-v4 enable-default-log
set firewall name storage-from-trust-v4 rule 20 log enable
set firewall name storage-from-trust-v4 rule 10 action accept
set firewall name storage-from-trust-v4 rule 10 state established enable
set firewall name storage-from-trust-v4 rule 10 state related enable
set firewall name storage-from-trust-v4 rule 20 action drop
set firewall name storage-from-trust-v4 rule 20 state invalid enable
set firewall ipv6-name storage-from-trust-v6 default-action accept
set firewall ipv6-name storage-from-trust-v6 enable-default-log
set firewall ipv6-name storage-from-trust-v6 rule 20 log enable
set firewall ipv6-name storage-from-trust-v6 rule 10 action accept
set firewall ipv6-name storage-from-trust-v6 rule 10 state established enable
set firewall ipv6-name storage-from-trust-v6 rule 10 state related enable
set firewall ipv6-name storage-from-trust-v6 rule 20 action drop
set firewall ipv6-name storage-from-trust-v6 rule 20 state invalid enable
set firewall ipv6-name storage-from-trust-v6 rule 30 action accept
set firewall ipv6-name storage-from-trust-v6 rule 30 protocol icmpv6
set firewall ipv6-name storage-from-trust-v6 rule 30 log enable
set zone-policy zone storage from trust firewall name storage-from-trust-v4
set zone-policy zone storage from trust firewall ipv6-name storage-from-trust-v6

set firewall name storage-from-dmz-v4 default-action accept
set firewall name storage-from-dmz-v4 enable-default-log
set firewall name storage-from-dmz-v4 rule 20 log enable
set firewall name storage-from-dmz-v4 rule 10 action accept
set firewall name storage-from-dmz-v4 rule 10 state established enable
set firewall name storage-from-dmz-v4 rule 10 state related enable
set firewall name storage-from-dmz-v4 rule 20 action drop
set firewall name storage-from-dmz-v4 rule 20 state invalid enable
set firewall ipv6-name storage-from-dmz-v6 default-action accept
set firewall ipv6-name storage-from-dmz-v6 enable-default-log
set firewall ipv6-name storage-from-dmz-v6 rule 20 log enable
set firewall ipv6-name storage-from-dmz-v6 rule 10 action accept
set firewall ipv6-name storage-from-dmz-v6 rule 10 state established enable
set firewall ipv6-name storage-from-dmz-v6 rule 10 state related enable
set firewall ipv6-name storage-from-dmz-v6 rule 20 action drop
set firewall ipv6-name storage-from-dmz-v6 rule 20 state invalid enable
set firewall ipv6-name storage-from-dmz-v6 rule 30 action accept
set firewall ipv6-name storage-from-dmz-v6 rule 30 protocol icmpv6
set firewall ipv6-name storage-from-dmz-v6 rule 30 log enable
set zone-policy zone storage from dmz firewall name storage-from-dmz-v4
set zone-policy zone storage from dmz firewall ipv6-name storage-from-dmz-v6

set firewall name storage-from-linode-v4 default-action reject
set firewall name storage-from-linode-v4 enable-default-log
set firewall name storage-from-linode-v4 rule 20 log enable
set firewall name storage-from-linode-v4 rule 10 action accept
set firewall name storage-from-linode-v4 rule 10 state established enable
set firewall name storage-from-linode-v4 rule 10 state related enable
set firewall name storage-from-linode-v4 rule 20 action drop
set firewall name storage-from-linode-v4 rule 20 state invalid enable
set firewall ipv6-name storage-from-linode-v6 default-action reject
set firewall ipv6-name storage-from-linode-v6 enable-default-log
set firewall ipv6-name storage-from-linode-v6 rule 20 log enable
set firewall ipv6-name storage-from-linode-v6 rule 10 action accept
set firewall ipv6-name storage-from-linode-v6 rule 10 state established enable
set firewall ipv6-name storage-from-linode-v6 rule 10 state related enable
set firewall ipv6-name storage-from-linode-v6 rule 20 action drop
set firewall ipv6-name storage-from-linode-v6 rule 20 state invalid enable
set firewall ipv6-name storage-from-linode-v6 rule 30 action accept
set firewall ipv6-name storage-from-linode-v6 rule 30 protocol icmpv6
set firewall ipv6-name storage-from-linode-v6 rule 30 log enable
set zone-policy zone storage from linode firewall name storage-from-linode-v4
set zone-policy zone storage from linode firewall ipv6-name storage-from-linode-v6

# Zone linode
set zone-policy zone linode description 'linode zone'
set zone-policy zone linode interface wg1
set zone-policy zone linode interface wg2
set zone-policy zone linode interface wg3
set firewall name linode-from-untrust-v4 default-action reject
set firewall name linode-from-untrust-v4 enable-default-log
set firewall name linode-from-untrust-v4 rule 20 log enable
set firewall name linode-from-untrust-v4 rule 10 action accept
set firewall name linode-from-untrust-v4 rule 10 state established enable
set firewall name linode-from-untrust-v4 rule 10 state related enable
set firewall name linode-from-untrust-v4 rule 20 action drop
set firewall name linode-from-untrust-v4 rule 20 state invalid enable
set firewall ipv6-name linode-from-untrust-v6 default-action reject
set firewall ipv6-name linode-from-untrust-v6 enable-default-log
set firewall ipv6-name linode-from-untrust-v6 rule 20 log enable
set firewall ipv6-name linode-from-untrust-v6 rule 10 action accept
set firewall ipv6-name linode-from-untrust-v6 rule 10 state established enable
set firewall ipv6-name linode-from-untrust-v6 rule 10 state related enable
set firewall ipv6-name linode-from-untrust-v6 rule 20 action drop
set firewall ipv6-name linode-from-untrust-v6 rule 20 state invalid enable
set firewall ipv6-name linode-from-untrust-v6 rule 30 action accept
set firewall ipv6-name linode-from-untrust-v6 rule 30 protocol icmpv6
set firewall ipv6-name linode-from-untrust-v6 rule 30 log enable
set zone-policy zone linode from untrust firewall name linode-from-untrust-v4
set zone-policy zone linode from untrust firewall ipv6-name linode-from-untrust-v6

set firewall name linode-from-trust-v4 default-action accept
set firewall name linode-from-trust-v4 enable-default-log
set firewall name linode-from-trust-v4 rule 20 log enable
set firewall name linode-from-trust-v4 rule 10 action accept
set firewall name linode-from-trust-v4 rule 10 state established enable
set firewall name linode-from-trust-v4 rule 10 state related enable
set firewall name linode-from-trust-v4 rule 20 action drop
set firewall name linode-from-trust-v4 rule 20 state invalid enable
set firewall ipv6-name linode-from-trust-v6 default-action accept
set firewall ipv6-name linode-from-trust-v6 enable-default-log
set firewall ipv6-name linode-from-trust-v6 rule 20 log enable
set firewall ipv6-name linode-from-trust-v6 rule 10 action accept
set firewall ipv6-name linode-from-trust-v6 rule 10 state established enable
set firewall ipv6-name linode-from-trust-v6 rule 10 state related enable
set firewall ipv6-name linode-from-trust-v6 rule 20 action drop
set firewall ipv6-name linode-from-trust-v6 rule 20 state invalid enable
set firewall ipv6-name linode-from-trust-v6 rule 30 action accept
set firewall ipv6-name linode-from-trust-v6 rule 30 protocol icmpv6
set firewall ipv6-name linode-from-trust-v6 rule 30 log enable
set zone-policy zone linode from trust firewall name linode-from-trust-v4
set zone-policy zone linode from trust firewall ipv6-name linode-from-trust-v6

set firewall name linode-from-dmz-v4 default-action accept
set firewall name linode-from-dmz-v4 enable-default-log
set firewall name linode-from-dmz-v4 rule 20 log enable
set firewall name linode-from-dmz-v4 rule 10 action accept
set firewall name linode-from-dmz-v4 rule 10 state established enable
set firewall name linode-from-dmz-v4 rule 10 state related enable
set firewall name linode-from-dmz-v4 rule 20 action drop
set firewall name linode-from-dmz-v4 rule 20 state invalid enable
set firewall ipv6-name linode-from-dmz-v6 default-action accept
set firewall ipv6-name linode-from-dmz-v6 enable-default-log
set firewall ipv6-name linode-from-dmz-v6 rule 20 log enable
set firewall ipv6-name linode-from-dmz-v6 rule 10 action accept
set firewall ipv6-name linode-from-dmz-v6 rule 10 state established enable
set firewall ipv6-name linode-from-dmz-v6 rule 10 state related enable
set firewall ipv6-name linode-from-dmz-v6 rule 20 action drop
set firewall ipv6-name linode-from-dmz-v6 rule 20 state invalid enable
set firewall ipv6-name linode-from-dmz-v6 rule 30 action accept
set firewall ipv6-name linode-from-dmz-v6 rule 30 protocol icmpv6
set firewall ipv6-name linode-from-dmz-v6 rule 30 log enable
set zone-policy zone linode from dmz firewall name linode-from-dmz-v4
set zone-policy zone linode from dmz firewall ipv6-name linode-from-dmz-v6

set firewall name linode-from-storage-v4 default-action reject
set firewall name linode-from-storage-v4 enable-default-log
set firewall name linode-from-storage-v4 rule 20 log enable
set firewall name linode-from-storage-v4 rule 10 action accept
set firewall name linode-from-storage-v4 rule 10 state established enable
set firewall name linode-from-storage-v4 rule 10 state related enable
set firewall name linode-from-storage-v4 rule 20 action drop
set firewall name linode-from-storage-v4 rule 20 state invalid enable
set firewall ipv6-name linode-from-storage-v6 default-action reject
set firewall ipv6-name linode-from-storage-v6 enable-default-log
set firewall ipv6-name linode-from-storage-v6 rule 20 log enable
set firewall ipv6-name linode-from-storage-v6 rule 10 action accept
set firewall ipv6-name linode-from-storage-v6 rule 10 state established enable
set firewall ipv6-name linode-from-storage-v6 rule 10 state related enable
set firewall ipv6-name linode-from-storage-v6 rule 20 action drop
set firewall ipv6-name linode-from-storage-v6 rule 20 state invalid enable
set firewall ipv6-name linode-from-storage-v6 rule 30 action accept
set firewall ipv6-name linode-from-storage-v6 rule 30 protocol icmpv6
set firewall ipv6-name linode-from-storage-v6 rule 30 log enable
set zone-policy zone linode from storage firewall name linode-from-storage-v4
set zone-policy zone linode from storage firewall ipv6-name linode-from-storage-v6

# Zone untrust
set zone-policy zone untrust description 'untrust zone'
set zone-policy zone untrust interface eth0
set zone-policy zone untrust interface eth4
set zone-policy zone untrust interface eth5
set zone-policy zone untrust interface eth6
set firewall name untrust-from-storage-v4 default-action reject
set firewall name untrust-from-storage-v4 enable-default-log
set firewall name untrust-from-storage-v4 rule 20 log enable
set firewall name untrust-from-storage-v4 rule 10 action accept
set firewall name untrust-from-storage-v4 rule 10 state established enable
set firewall name untrust-from-storage-v4 rule 10 state related enable
set firewall name untrust-from-storage-v4 rule 20 action drop
set firewall name untrust-from-storage-v4 rule 20 state invalid enable
set firewall ipv6-name untrust-from-storage-v6 default-action reject
set firewall ipv6-name untrust-from-storage-v6 enable-default-log
set firewall ipv6-name untrust-from-storage-v6 rule 20 log enable
set firewall ipv6-name untrust-from-storage-v6 rule 10 action accept
set firewall ipv6-name untrust-from-storage-v6 rule 10 state established enable
set firewall ipv6-name untrust-from-storage-v6 rule 10 state related enable
set firewall ipv6-name untrust-from-storage-v6 rule 20 action drop
set firewall ipv6-name untrust-from-storage-v6 rule 20 state invalid enable
set firewall ipv6-name untrust-from-storage-v6 rule 30 action accept
set firewall ipv6-name untrust-from-storage-v6 rule 30 protocol icmpv6
set firewall ipv6-name untrust-from-storage-v6 rule 30 log enable
set zone-policy zone untrust from storage firewall name untrust-from-storage-v4
set zone-policy zone untrust from storage firewall ipv6-name untrust-from-storage-v6

set firewall name untrust-from-linode-v4 default-action reject
set firewall name untrust-from-linode-v4 enable-default-log
set firewall name untrust-from-linode-v4 rule 20 log enable
set firewall name untrust-from-linode-v4 rule 10 action accept
set firewall name untrust-from-linode-v4 rule 10 state established enable
set firewall name untrust-from-linode-v4 rule 10 state related enable
set firewall name untrust-from-linode-v4 rule 20 action drop
set firewall name untrust-from-linode-v4 rule 20 state invalid enable
set firewall ipv6-name untrust-from-linode-v6 default-action reject
set firewall ipv6-name untrust-from-linode-v6 enable-default-log
set firewall ipv6-name untrust-from-linode-v6 rule 20 log enable
set firewall ipv6-name untrust-from-linode-v6 rule 10 action accept
set firewall ipv6-name untrust-from-linode-v6 rule 10 state established enable
set firewall ipv6-name untrust-from-linode-v6 rule 10 state related enable
set firewall ipv6-name untrust-from-linode-v6 rule 20 action drop
set firewall ipv6-name untrust-from-linode-v6 rule 20 state invalid enable
set firewall ipv6-name untrust-from-linode-v6 rule 30 action accept
set firewall ipv6-name untrust-from-linode-v6 rule 30 protocol icmpv6
set firewall ipv6-name untrust-from-linode-v6 rule 30 log enable
set zone-policy zone untrust from linode firewall name untrust-from-linode-v4
set zone-policy zone untrust from linode firewall ipv6-name untrust-from-linode-v6

set firewall name untrust-from-trust-v4 default-action accept
set firewall name untrust-from-trust-v4 enable-default-log
set firewall name untrust-from-trust-v4 rule 20 log enable
set firewall name untrust-from-trust-v4 rule 10 action accept
set firewall name untrust-from-trust-v4 rule 10 state established enable
set firewall name untrust-from-trust-v4 rule 10 state related enable
set firewall name untrust-from-trust-v4 rule 20 action drop
set firewall name untrust-from-trust-v4 rule 20 state invalid enable
set firewall ipv6-name untrust-from-trust-v6 default-action accept
set firewall ipv6-name untrust-from-trust-v6 enable-default-log
set firewall ipv6-name untrust-from-trust-v6 rule 20 log enable
set firewall ipv6-name untrust-from-trust-v6 rule 10 action accept
set firewall ipv6-name untrust-from-trust-v6 rule 10 state established enable
set firewall ipv6-name untrust-from-trust-v6 rule 10 state related enable
set firewall ipv6-name untrust-from-trust-v6 rule 20 action drop
set firewall ipv6-name untrust-from-trust-v6 rule 20 state invalid enable
set firewall ipv6-name untrust-from-trust-v6 rule 30 action accept
set firewall ipv6-name untrust-from-trust-v6 rule 30 protocol icmpv6
set firewall ipv6-name untrust-from-trust-v6 rule 30 log enable
set zone-policy zone untrust from trust firewall name untrust-from-trust-v4
set zone-policy zone untrust from trust firewall ipv6-name untrust-from-trust-v6

set firewall name untrust-from-dmz-v4 default-action accept
set firewall name untrust-from-dmz-v4 enable-default-log
set firewall name untrust-from-dmz-v4 rule 20 log enable
set firewall name untrust-from-dmz-v4 rule 10 action accept
set firewall name untrust-from-dmz-v4 rule 10 state established enable
set firewall name untrust-from-dmz-v4 rule 10 state related enable
set firewall name untrust-from-dmz-v4 rule 20 action drop
set firewall name untrust-from-dmz-v4 rule 20 state invalid enable
set firewall ipv6-name untrust-from-dmz-v6 default-action accept
set firewall ipv6-name untrust-from-dmz-v6 enable-default-log
set firewall ipv6-name untrust-from-dmz-v6 rule 20 log enable
set firewall ipv6-name untrust-from-dmz-v6 rule 10 action accept
set firewall ipv6-name untrust-from-dmz-v6 rule 10 state established enable
set firewall ipv6-name untrust-from-dmz-v6 rule 10 state related enable
set firewall ipv6-name untrust-from-dmz-v6 rule 20 action drop
set firewall ipv6-name untrust-from-dmz-v6 rule 20 state invalid enable
set firewall ipv6-name untrust-from-dmz-v6 rule 30 action accept
set firewall ipv6-name untrust-from-dmz-v6 rule 30 protocol icmpv6
set firewall ipv6-name untrust-from-dmz-v6 rule 30 log enable
set zone-policy zone untrust from dmz firewall name untrust-from-dmz-v4
set zone-policy zone untrust from dmz firewall ipv6-name untrust-from-dmz-v6

# Zone trust
set zone-policy zone trust description 'trust zone'
set zone-policy zone trust interface wg0
set zone-policy zone trust interface eth3
set firewall name trust-from-untrust-v4 default-action reject
set firewall name trust-from-untrust-v4 enable-default-log
set firewall name trust-from-untrust-v4 rule 20 log enable
set firewall name trust-from-untrust-v4 rule 10 action accept
set firewall name trust-from-untrust-v4 rule 10 state established enable
set firewall name trust-from-untrust-v4 rule 10 state related enable
set firewall name trust-from-untrust-v4 rule 20 action drop
set firewall name trust-from-untrust-v4 rule 20 state invalid enable
set firewall ipv6-name trust-from-untrust-v6 default-action reject
set firewall ipv6-name trust-from-untrust-v6 enable-default-log
set firewall ipv6-name trust-from-untrust-v6 rule 20 log enable
set firewall ipv6-name trust-from-untrust-v6 rule 10 action accept
set firewall ipv6-name trust-from-untrust-v6 rule 10 state established enable
set firewall ipv6-name trust-from-untrust-v6 rule 10 state related enable
set firewall ipv6-name trust-from-untrust-v6 rule 20 action drop
set firewall ipv6-name trust-from-untrust-v6 rule 20 state invalid enable
set firewall ipv6-name trust-from-untrust-v6 rule 30 action accept
set firewall ipv6-name trust-from-untrust-v6 rule 30 protocol icmpv6
set firewall ipv6-name trust-from-untrust-v6 rule 30 log enable
set zone-policy zone trust from untrust firewall name trust-from-untrust-v4
set zone-policy zone trust from untrust firewall ipv6-name trust-from-untrust-v6

set firewall name trust-from-dmz-v4 default-action reject
set firewall name trust-from-dmz-v4 enable-default-log
set firewall name trust-from-dmz-v4 rule 20 log enable
set firewall name trust-from-dmz-v4 rule 10 action accept
set firewall name trust-from-dmz-v4 rule 10 state established enable
set firewall name trust-from-dmz-v4 rule 10 state related enable
set firewall name trust-from-dmz-v4 rule 20 action drop
set firewall name trust-from-dmz-v4 rule 20 state invalid enable
set firewall ipv6-name trust-from-dmz-v6 default-action reject
set firewall ipv6-name trust-from-dmz-v6 enable-default-log
set firewall ipv6-name trust-from-dmz-v6 rule 20 log enable
set firewall ipv6-name trust-from-dmz-v6 rule 10 action accept
set firewall ipv6-name trust-from-dmz-v6 rule 10 state established enable
set firewall ipv6-name trust-from-dmz-v6 rule 10 state related enable
set firewall ipv6-name trust-from-dmz-v6 rule 20 action drop
set firewall ipv6-name trust-from-dmz-v6 rule 20 state invalid enable
set firewall ipv6-name trust-from-dmz-v6 rule 30 action accept
set firewall ipv6-name trust-from-dmz-v6 rule 30 protocol icmpv6
set firewall ipv6-name trust-from-dmz-v6 rule 30 log enable
set zone-policy zone trust from dmz firewall name trust-from-dmz-v4
set zone-policy zone trust from dmz firewall ipv6-name trust-from-dmz-v6

set firewall name trust-from-storage-v4 default-action reject
set firewall name trust-from-storage-v4 enable-default-log
set firewall name trust-from-storage-v4 rule 20 log enable
set firewall name trust-from-storage-v4 rule 10 action accept
set firewall name trust-from-storage-v4 rule 10 state established enable
set firewall name trust-from-storage-v4 rule 10 state related enable
set firewall name trust-from-storage-v4 rule 20 action drop
set firewall name trust-from-storage-v4 rule 20 state invalid enable
set firewall ipv6-name trust-from-storage-v6 default-action reject
set firewall ipv6-name trust-from-storage-v6 enable-default-log
set firewall ipv6-name trust-from-storage-v6 rule 20 log enable
set firewall ipv6-name trust-from-storage-v6 rule 10 action accept
set firewall ipv6-name trust-from-storage-v6 rule 10 state established enable
set firewall ipv6-name trust-from-storage-v6 rule 10 state related enable
set firewall ipv6-name trust-from-storage-v6 rule 20 action drop
set firewall ipv6-name trust-from-storage-v6 rule 20 state invalid enable
set firewall ipv6-name trust-from-storage-v6 rule 30 action accept
set firewall ipv6-name trust-from-storage-v6 rule 30 protocol icmpv6
set firewall ipv6-name trust-from-storage-v6 rule 30 log enable
set zone-policy zone trust from storage firewall name trust-from-storage-v4
set zone-policy zone trust from storage firewall ipv6-name trust-from-storage-v6

set firewall name trust-from-linode-v4 default-action reject
set firewall name trust-from-linode-v4 enable-default-log
set firewall name trust-from-linode-v4 rule 20 log enable
set firewall name trust-from-linode-v4 rule 10 action accept
set firewall name trust-from-linode-v4 rule 10 state established enable
set firewall name trust-from-linode-v4 rule 10 state related enable
set firewall name trust-from-linode-v4 rule 20 action drop
set firewall name trust-from-linode-v4 rule 20 state invalid enable
set firewall ipv6-name trust-from-linode-v6 default-action reject
set firewall ipv6-name trust-from-linode-v6 enable-default-log
set firewall ipv6-name trust-from-linode-v6 rule 20 log enable
set firewall ipv6-name trust-from-linode-v6 rule 10 action accept
set firewall ipv6-name trust-from-linode-v6 rule 10 state established enable
set firewall ipv6-name trust-from-linode-v6 rule 10 state related enable
set firewall ipv6-name trust-from-linode-v6 rule 20 action drop
set firewall ipv6-name trust-from-linode-v6 rule 20 state invalid enable
set firewall ipv6-name trust-from-linode-v6 rule 30 action accept
set firewall ipv6-name trust-from-linode-v6 rule 30 protocol icmpv6
set firewall ipv6-name trust-from-linode-v6 rule 30 log enable
set zone-policy zone trust from linode firewall name trust-from-linode-v4
set zone-policy zone trust from linode firewall ipv6-name trust-from-linode-v6

# Zone dmz
set zone-policy zone dmz description 'dmz zone'
set zone-policy zone dmz interface eth1
set firewall name dmz-from-untrust-v4 default-action reject
set firewall name dmz-from-untrust-v4 enable-default-log
set firewall name dmz-from-untrust-v4 rule 20 log enable
set firewall name dmz-from-untrust-v4 rule 10 action accept
set firewall name dmz-from-untrust-v4 rule 10 state established enable
set firewall name dmz-from-untrust-v4 rule 10 state related enable
set firewall name dmz-from-untrust-v4 rule 20 action drop
set firewall name dmz-from-untrust-v4 rule 20 state invalid enable
set firewall ipv6-name dmz-from-untrust-v6 default-action reject
set firewall ipv6-name dmz-from-untrust-v6 enable-default-log
set firewall ipv6-name dmz-from-untrust-v6 rule 20 log enable
set firewall ipv6-name dmz-from-untrust-v6 rule 10 action accept
set firewall ipv6-name dmz-from-untrust-v6 rule 10 state established enable
set firewall ipv6-name dmz-from-untrust-v6 rule 10 state related enable
set firewall ipv6-name dmz-from-untrust-v6 rule 20 action drop
set firewall ipv6-name dmz-from-untrust-v6 rule 20 state invalid enable
set firewall ipv6-name dmz-from-untrust-v6 rule 30 action accept
set firewall ipv6-name dmz-from-untrust-v6 rule 30 protocol icmpv6
set firewall ipv6-name dmz-from-untrust-v6 rule 30 log enable
set zone-policy zone dmz from untrust firewall name dmz-from-untrust-v4
set zone-policy zone dmz from untrust firewall ipv6-name dmz-from-untrust-v6

set firewall name dmz-from-trust-v4 default-action accept
set firewall name dmz-from-trust-v4 enable-default-log
set firewall name dmz-from-trust-v4 rule 20 log enable
set firewall name dmz-from-trust-v4 rule 10 action accept
set firewall name dmz-from-trust-v4 rule 10 state established enable
set firewall name dmz-from-trust-v4 rule 10 state related enable
set firewall name dmz-from-trust-v4 rule 20 action drop
set firewall name dmz-from-trust-v4 rule 20 state invalid enable
set firewall ipv6-name dmz-from-trust-v6 default-action accept
set firewall ipv6-name dmz-from-trust-v6 enable-default-log
set firewall ipv6-name dmz-from-trust-v6 rule 20 log enable
set firewall ipv6-name dmz-from-trust-v6 rule 10 action accept
set firewall ipv6-name dmz-from-trust-v6 rule 10 state established enable
set firewall ipv6-name dmz-from-trust-v6 rule 10 state related enable
set firewall ipv6-name dmz-from-trust-v6 rule 20 action drop
set firewall ipv6-name dmz-from-trust-v6 rule 20 state invalid enable
set firewall ipv6-name dmz-from-trust-v6 rule 30 action accept
set firewall ipv6-name dmz-from-trust-v6 rule 30 protocol icmpv6
set firewall ipv6-name dmz-from-trust-v6 rule 30 log enable
set zone-policy zone dmz from trust firewall name dmz-from-trust-v4
set zone-policy zone dmz from trust firewall ipv6-name dmz-from-trust-v6

set firewall name dmz-from-storage-v4 default-action accept
set firewall name dmz-from-storage-v4 enable-default-log
set firewall name dmz-from-storage-v4 rule 20 log enable
set firewall name dmz-from-storage-v4 rule 10 action accept
set firewall name dmz-from-storage-v4 rule 10 state established enable
set firewall name dmz-from-storage-v4 rule 10 state related enable
set firewall name dmz-from-storage-v4 rule 20 action drop
set firewall name dmz-from-storage-v4 rule 20 state invalid enable
set firewall ipv6-name dmz-from-storage-v6 default-action accept
set firewall ipv6-name dmz-from-storage-v6 enable-default-log
set firewall ipv6-name dmz-from-storage-v6 rule 20 log enable
set firewall ipv6-name dmz-from-storage-v6 rule 10 action accept
set firewall ipv6-name dmz-from-storage-v6 rule 10 state established enable
set firewall ipv6-name dmz-from-storage-v6 rule 10 state related enable
set firewall ipv6-name dmz-from-storage-v6 rule 20 action drop
set firewall ipv6-name dmz-from-storage-v6 rule 20 state invalid enable
set firewall ipv6-name dmz-from-storage-v6 rule 30 action accept
set firewall ipv6-name dmz-from-storage-v6 rule 30 protocol icmpv6
set firewall ipv6-name dmz-from-storage-v6 rule 30 log enable
set zone-policy zone dmz from storage firewall name dmz-from-storage-v4
set zone-policy zone dmz from storage firewall ipv6-name dmz-from-storage-v6

set firewall name dmz-from-linode-v4 default-action accept
set firewall name dmz-from-linode-v4 enable-default-log
set firewall name dmz-from-linode-v4 rule 20 log enable
set firewall name dmz-from-linode-v4 rule 10 action accept
set firewall name dmz-from-linode-v4 rule 10 state established enable
set firewall name dmz-from-linode-v4 rule 10 state related enable
set firewall name dmz-from-linode-v4 rule 20 action drop
set firewall name dmz-from-linode-v4 rule 20 state invalid enable
set firewall ipv6-name dmz-from-linode-v6 default-action accept
set firewall ipv6-name dmz-from-linode-v6 enable-default-log
set firewall ipv6-name dmz-from-linode-v6 rule 20 log enable
set firewall ipv6-name dmz-from-linode-v6 rule 10 action accept
set firewall ipv6-name dmz-from-linode-v6 rule 10 state established enable
set firewall ipv6-name dmz-from-linode-v6 rule 10 state related enable
set firewall ipv6-name dmz-from-linode-v6 rule 20 action drop
set firewall ipv6-name dmz-from-linode-v6 rule 20 state invalid enable
set firewall ipv6-name dmz-from-linode-v6 rule 30 action accept
set firewall ipv6-name dmz-from-linode-v6 rule 30 protocol icmpv6
set firewall ipv6-name dmz-from-linode-v6 rule 30 log enable
set zone-policy zone dmz from linode firewall name dmz-from-linode-v4
set zone-policy zone dmz from linode firewall ipv6-name dmz-from-linode-v6
# Imagine typing this lol, lmao

PBR (policy based routing)

Here we will define a general purpose route-policy. This policy is assigned to all interfaces which we provide internet access on, and lets us choose the routing table (among other things) that will be used.

For example traffic from p2p download software should not be sent via our VPS as it's rude to use those address ranges for such traffic, a similar rule exists for web browsing since many sites now either block or require additional steps to access due to abuse of VPSs as bots. So in this case I am deferring this traffic to the untrust LAN router, which is running pfsense as a simple WAN load balancer. If you are worried about 3 letter agencies, you can route all your traffic via the VPS instead.

In the case of local addresses, we need to exclude that traffic and use the main routing table.

firewall {
    group {
        network-group RFC1918 {
            network 10.0.0.0/8
            network 172.16.0.0/12
            network 192.168.0.0/16
        }
    }
}
policy {
    route WAN-PBR {
        description "General ingress policy"
        rule 10 {
            description "Bypass for local networks"
            destination {
                group {
                    network-group RFC1918
                }
            }
            protocol all
            set {
                table main
            }
        }
        rule 11 {
            description "Always send priority traffic via VPS"
            destination {
                port 53,22
            }
            protocol tcp_udp
            set {
                table 20
            }
        }
        rule 12 {
            description "Web via untrust LAN (STUN and TURN also so we don't break things like WebRTC)"
            destination {
                port 80,443,3478,5349
            }
            protocol tcp_udp
            set {
                table 10
            }
        }
        rule 13 {
            description "Torrents via untrust LAN"
            protocol tcp_udp
            set {
                table 10
            }
            source {
                address 10.0.16.3/32
            }
        }
        rule 21 {
            description "DMZ via VPS"
            protocol all
            set {
                table 20
            }
            source {
                address 10.0.16.0/20
            }
        }
        rule 22 {
            description "Storage via VPS"
            protocol all
            set {
                table 20
            }
            source {
                address 10.0.32.0/20
            }
        }
        rule 99 {
            description "Default to sending via VPS"
            set {
                table 20
            }
        }
    }
}
set firewall group network-group RFC1918 network '10.0.0.0/8'
set firewall group network-group RFC1918 network '172.16.0.0/12'
set firewall group network-group RFC1918 network '192.168.0.0/16'
set policy route WAN-PBR description 'General ingress policy'
set policy route WAN-PBR rule 10 description 'Bypass for local networks'
set policy route WAN-PBR rule 10 destination group network-group 'RFC1918'
set policy route WAN-PBR rule 10 protocol 'all'
set policy route WAN-PBR rule 10 set table 'main'
set policy route WAN-PBR rule 11 description 'Always send priority traffic via VPS'
set policy route WAN-PBR rule 11 destination port '53,22'
set policy route WAN-PBR rule 11 protocol 'tcp_udp'
set policy route WAN-PBR rule 11 set table '20'
set policy route WAN-PBR rule 12 description 'Web via untrust LAN (STUN and TURN also so we don't break things like WebRTC)'
set policy route WAN-PBR rule 12 destination port '80,443,3478,5349'
set policy route WAN-PBR rule 12 protocol 'tcp_udp'
set policy route WAN-PBR rule 12 set table '10'
set policy route WAN-PBR rule 13 description 'Torrents via untrust LAN'
set policy route WAN-PBR rule 13 protocol 'tcp_udp'
set policy route WAN-PBR rule 13 set table '10'
set policy route WAN-PBR rule 13 source address '10.0.16.3/32'
set policy route WAN-PBR rule 21 description 'DMZ via VPS'
set policy route WAN-PBR rule 21 protocol 'all'
set policy route WAN-PBR rule 21 set table '20'
set policy route WAN-PBR rule 21 source address '10.0.16.0/20'
set policy route WAN-PBR rule 22 description 'Storage via VPS'
set policy route WAN-PBR rule 22 protocol 'all'
set policy route WAN-PBR rule 22 set table '20'
set policy route WAN-PBR rule 22 source address '10.0.32.0/20'
set policy route WAN-PBR rule 99 description 'Default to sending via VPS'
set policy route WAN-PBR rule 99 set table '20'

Gluing it all together with BGP

So far we've defined a pile of subnets, but have no way to determine which uplink should be used when forwarding said traffic to the VPS.

To fix this we will use BGP to dynamically distribute networks between the two routers and provide unequal cost load balancing of internet bound traffic. Specifically we'll be using iBGP and will assign a single AS number to all participating routers, this will avoid having to create additional route policies to prevent unwanted import and export of routes between autonomous systems (as would be the case with eBGP). The only downside is that iBGP prefers a full mesh topology, so every router has a transit network to every other router, but in this case with only 2 it's no problem at all.

Start by defining the networks we wish to expose to the VPS under the ipv4 unicast address family, this will tell BGP to only expose the unicast portion and will not set up routes for broadcast traffic. Next we'll assign ourselves a AS number, 65000 is part of a large range of ASNs for private use which is perfectly sufficient unless we plan to peer with other exterior networks. You'll also need a router ID to uniquely identify the router within the AS (I have chosen to use one of the transit networks addresses but it only needs to be a valid-ish ipv4 address not used by another BGP instance).

For peering we'll set up 3 neighbours pointing to the wireguard client addresses that will be set up on the VPS. Each of these have a weight, an update-source interface and a peer-group that let's us assign common parameters between them. The peer group enables soft reconfiguration which will allow frequent changes in the routing path without dropping packets, it also assigns the remote ASN and enables a BFD profile we will configure later.

protocols {
    bgp {
        address-family {
            ipv4-unicast {
                network 10.0.0.0/20 {
                }
                network 10.0.8.0/20 {
                }
                network 10.0.16.0/20 {
                }
                network 10.0.32.0/20 {
                }
            }
        }
        system-as 65000
        neighbor 10.251.1.1 {
            address-family {
                ipv4-unicast {
                    weight 10
                }
            }
            peer-group LINODE
            update-source wg1
        }
        neighbor 10.251.2.1 {
            address-family {
                ipv4-unicast {
                    weight 10
                }
            }
            peer-group LINODE
            update-source wg2
        }
        neighbor 10.251.3.1 {
            address-family {
                ipv4-unicast {
                    weight 5
                }
            }
            peer-group LINODE
            update-source wg3
        }
        parameters {
            no-fast-external-failover
            router-id 10.251.1.2
        }
        peer-group LINODE {
            address-family {
                ipv4-unicast {
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            bfd {
                profile linode
            }
            description "Linode transit network peers."
            password xxxxxx
            remote-as 65000
        }
    }
}
set protocols bgp address-family ipv4-unicast network 10.0.0.0/20
set protocols bgp address-family ipv4-unicast network 10.0.8.0/20
set protocols bgp address-family ipv4-unicast network 10.0.16.0/20
set protocols bgp address-family ipv4-unicast network 10.0.32.0/20
set protocols bgp system-as '65000'
set protocols bgp neighbor 10.251.1.1 address-family ipv4-unicast weight '10'
set protocols bgp neighbor 10.251.1.1 peer-group 'LINODE'
set protocols bgp neighbor 10.251.1.1 update-source 'wg1'
set protocols bgp neighbor 10.251.2.1 address-family ipv4-unicast weight '10'
set protocols bgp neighbor 10.251.2.1 peer-group 'LINODE'
set protocols bgp neighbor 10.251.2.1 update-source 'wg2'
set protocols bgp neighbor 10.251.3.1 address-family ipv4-unicast weight '5'
set protocols bgp neighbor 10.251.3.1 peer-group 'LINODE'
set protocols bgp neighbor 10.251.3.1 update-source 'wg3'
set protocols bgp parameters no-fast-external-failover
set protocols bgp parameters router-id '10.251.1.2'
set protocols bgp peer-group LINODE address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp peer-group LINODE bfd profile 'linode'
set protocols bgp peer-group LINODE description 'Linode transit network peers.'
set protocols bgp peer-group LINODE password 'xxxxxx'
set protocols bgp peer-group LINODE remote-as '65000'

Default routes

In this section we can add some default routes. 20 and the main table will use our VPSs gateway address directly, while table 10 will use our untrust LAN router (you could also select one of the uplinks directly if you do not wish to run a 3rd router). This where our multipathing comes in as BGP will dynamically assign a route for the network we are trying to reach. Since it knows two of our uplinks provide an equal amount of bandwidth, both will be utilised on a per flow basis.

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>  0.0.0.0/0 [1/0] via 5.5.5.1 (recursive), weight 1, 00:57:58
  *                   via 10.251.1.1, wg1, weight 1, 00:57:58
  *                   via 10.251.2.1, wg2, weight 1, 00:57:58
B>* 5.5.5.0/24 [200/0] via 10.251.1.1, wg1, weight 1, 00:57:58
  *                    via 10.251.2.1, wg2, weight 1, 00:57:58

Finally, define the gateway addresses in separate tables (101-103) for each of the uplinks, and create a local-route policy to assign the fwmark on each wireguard interface to the appropriate table, this completes the configuration needed to get traffic on to the correct physical link.

policy {
    local-route {
        rule 101 {
            fwmark 101
            set {
                table 101
            }
        }
        rule 102 {
            fwmark 102
            set {
                table 102
            }
        }
        rule 103 {
            fwmark 103
            set {
                table 103
            }
        }
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 5.5.5.1 {
            }
        }
        table 10 {
            route 0.0.0.0/0 {
                next-hop xxx.xxx.0.1 {
                }
            }
        }
        table 20 {
            route 0.0.0.0/0 {
                next-hop 5.5.5.1 {
                }
            }
        }
        table 101 {
            route 0.0.0.0/0 {
                next-hop xxx.xxx.101.1 {
                    interface eth4
                }
            }
        }
        table 102 {
            route 0.0.0.0/0 {
                next-hop xxx.xxx.102.1 {
                    interface eth5
                }
            }
        }
        table 103 {
            route 0.0.0.0/0 {
                next-hop xxx.xxx.103.1 {
                    interface eth6
                }
            }
        }
    }
}
set policy local-route rule 101 fwmark '101'
set policy local-route rule 101 set table '101'
set policy local-route rule 102 fwmark '102'
set policy local-route rule 102 set table '102'
set policy local-route rule 103 fwmark '103'
set policy local-route rule 103 set table '103'
set protocols static route 0.0.0.0/0 next-hop 5.5.5.1
set protocols static table 10 route 0.0.0.0/0 next-hop xxx.xxx.0.1
set protocols static table 20 route 0.0.0.0/0 next-hop 5.5.5.1
set protocols static table 101 route 0.0.0.0/0 next-hop xxx.xxx.101.1 interface 'eth4'
set protocols static table 102 route 0.0.0.0/0 next-hop xxx.xxx.102.1 interface 'eth5'
set protocols static table 103 route 0.0.0.0/0 next-hop xxx.xxx.103.1 interface 'eth6'

Faster convergence with BFD

As you probably already know, BGP performs health checking of it's peers on a regular interval, if the physical link fails or it's no longer possible to communicate it will close this session and stop sending traffic until a connection is re-established. This works great if your WAN is generally stable and you probably don't mind a few minutes down time each month.

However, the internet can be anything but stable and what can happen (especially over wireless connections), is where the packet loss is high enough that some of our keepalive packets make it but then are immediately dropped again once load is put on the modem.

This will manifest itself as a type of slow flap due to the slow convergence time of our routing protocol. After 4 seconds of you not being able to resolve a hostname, BGP does reconverge and starts routing traffic over one of the working links, 1 second later a keepalive packet makes it through since it's the sole thing going over that broken link right now, BGP goes "fantastic, I-I-I'M GONNA RECONVERGE" and dumps all of your freshly recovered connections onto a link that can barely handle dial up speeds.

Since we had 4 seconds downtime followed by 1 second of uptime, our packet loss is still >75% despite having an alternate path available 100% of the time.

Thankfully we can use BFD (Bidirectional Forwarding Detection) to address this, it's a protocol agnostic tool so it can be used with OSPF, BGP or whatever, and aims to perform fast detection of faulty forwarding planes. Less than 1s and in most cases 50ms is attainable even over the internet.

BFD is not a magic tool to fix your ISPs fuck ups however, it will merely mitigate the worst of it by forcing alternate path selection faster than the routing protocol would. In the above scenario the link will still flap, but it will do it quickly enough L7 protocols like DNS/HTTP/etc will tend to handle it (2-10% packet loss still likely). You will still need to diagnose and correct whatever issue is afoot, but in a more sensible situation where the link only goes down every few minutes, you won't notice the transition.

protocols {
    bfd {
        peer 10.251.1.1 {
            source {
                address 10.251.1.2
                interface wg1
            }
        }
        peer 10.251.2.1 {
            source {
                address 10.251.2.2
                interface wg2
            }
        }
        peer 10.251.3.1 {
            source {
                address 10.251.3.2
                interface wg3
            }
        }
        profile linode {
            echo-mode
            interval {
                echo-interval 100
                multiplier 3
                receive 300
                transmit 300
            }
        }
    }
}
set protocols bfd peer 10.251.1.1 source address '10.251.1.2'
set protocols bfd peer 10.251.1.1 source interface 'wg1'
set protocols bfd peer 10.251.2.1 source address '10.251.2.2'
set protocols bfd peer 10.251.2.1 source interface 'wg2'
set protocols bfd peer 10.251.3.1 source address '10.251.3.2'
set protocols bfd peer 10.251.3.1 source interface 'wg3'
set protocols bfd profile linode echo-mode
set protocols bfd profile linode interval echo-interval '100'
set protocols bfd profile linode interval multiplier '3'
set protocols bfd profile linode interval receive '300'
set protocols bfd profile linode interval transmit '300'

Setting up the remote end at last

First thing you'll notice is that the VPS configuration is notably simpler, it really only has one job and that's to aggregate our core routers network and perform source NAT for outbound connections. I have omitted the firewall since it's all implementation specific, as are the DNAT rules that allow you to see this website, but this is indeed where all that junk gets defined.

To me that's a much cleaner solution than the standard set up where absolutely everything is configured on one on-premises router, there is a clear separation of concerns, and even if I did accidentally NAT internet traffic to the motherboards BMC, the core router would drop such a request due to the zone policy constraints we generated at the beginning.

interfaces {
    ethernet eth0 {
        address 5.5.5.2/24
    }
    wireguard wg1 {
        address 10.251.1.1/30
        description "Uplink 1"
        peer primary {
            allowed-ips 0.0.0.0/0
            allowed-ips 10.251.1.2/32
            preshared-key ****************
            public-key ****************
        }
        port 51820
        private-key xxxxxx
    }
    wireguard wg2 {
        address 10.251.2.1/30
        description "Uplink 2"
        peer primary {
            allowed-ips 0.0.0.0/0
            allowed-ips 10.251.2.2/32
            preshared-key ****************
            public-key ****************
        }
        port 51821
        private-key xxxxxx
    }
    wireguard wg3 {
        address 10.251.3.1/30
        description "Uplink 3"
        peer primary {
            allowed-ips 0.0.0.0/0
            allowed-ips 10.251.3.2/32
            preshared-key ****************
            public-key ****************
        }
        port 51822
        private-key xxxxxx
    }
}
nat {
    source {
        rule 10 {
            outbound-interface eth0
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    bfd {
        peer 10.251.1.2 {
            source {
                address 10.251.1.1
                interface wg1
            }
        }
        peer 10.251.2.2 {
            source {
                address 10.251.2.1
                interface wg2
            }
        }
        peer 10.251.3.2 {
            source {
                address 10.251.3.1
                interface wg3
            }
        }
        profile linode {
            echo-mode
            interval {
                echo-interval 100
                multiplier 3
                receive 300
                transmit 300
            }
        }
    }
    bgp {
        address-family {
            ipv4-unicast {
                network 5.5.5.0/24 {
                }
            }
        }
        system-as 65000
        neighbor 10.251.1.2 {
            address-family {
                ipv4-unicast {
                    weight 10
                }
            }
            peer-group LINODE
            update-source wg1
        }
        neighbor 10.251.2.2 {
            address-family {
                ipv4-unicast {
                    weight 10
                }
            }
            peer-group LINODE
            update-source wg2
        }
        neighbor 10.251.3.2 {
            address-family {
                ipv4-unicast {
                    weight 5
                }
            }
            peer-group LINODE
            update-source wg3
        }
        parameters {
            no-fast-external-failover
            router-id 10.251.1.1
        }
        peer-group LINODE {
            address-family {
                ipv4-unicast {
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            bfd {
                profile linode
            }
            description "Linode transit network peers."
            password xxxxxx
            remote-as 65000
        }
    }
    static {
        route 0.0.0.0/0 {
            next-hop 5.5.5.1 {
            }
        }
    }
}
set interfaces ethernet eth0 address '5.5.5.2/24'
set interfaces wireguard wg1 address '10.251.1.1/30'
set interfaces wireguard wg1 description 'Uplink 1'
set interfaces wireguard wg1 peer primary allowed-ips '0.0.0.0/0'
set interfaces wireguard wg1 peer primary allowed-ips '10.251.1.2/32'
set interfaces wireguard wg1 peer primary preshared-key '****************'
set interfaces wireguard wg1 peer primary public-key '****************'
set interfaces wireguard wg1 port '51820'
set interfaces wireguard wg1 private-key 'xxxxxx'
set interfaces wireguard wg2 address '10.251.2.1/30'
set interfaces wireguard wg2 description 'Uplink 2'
set interfaces wireguard wg2 peer primary allowed-ips '0.0.0.0/0'
set interfaces wireguard wg2 peer primary allowed-ips '10.251.2.2/32'
set interfaces wireguard wg2 peer primary preshared-key '****************'
set interfaces wireguard wg2 peer primary public-key '****************'
set interfaces wireguard wg2 port '51821'
set interfaces wireguard wg2 private-key 'xxxxxx'
set interfaces wireguard wg3 address '10.251.3.1/30'
set interfaces wireguard wg3 description 'Uplink 3'
set interfaces wireguard wg3 peer primary allowed-ips '0.0.0.0/0'
set interfaces wireguard wg3 peer primary allowed-ips '10.251.3.2/32'
set interfaces wireguard wg3 peer primary preshared-key '****************'
set interfaces wireguard wg3 peer primary public-key '****************'
set interfaces wireguard wg3 port '51822'
set interfaces wireguard wg3 private-key 'xxxxxx'
set nat source rule 10 outbound-interface 'eth0'
set nat source rule 10 translation address 'masquerade'
set protocols bfd peer 10.251.1.2 source address '10.251.1.1'
set protocols bfd peer 10.251.1.2 source interface 'wg1'
set protocols bfd peer 10.251.2.2 source address '10.251.2.1'
set protocols bfd peer 10.251.2.2 source interface 'wg2'
set protocols bfd peer 10.251.3.2 source address '10.251.3.1'
set protocols bfd peer 10.251.3.2 source interface 'wg3'
set protocols bfd profile linode echo-mode
set protocols bfd profile linode interval echo-interval '100'
set protocols bfd profile linode interval multiplier '3'
set protocols bfd profile linode interval receive '300'
set protocols bfd profile linode interval transmit '300'
set protocols bgp address-family ipv4-unicast network 5.5.5.0/24
set protocols bgp system-as '65000'
set protocols bgp neighbor 10.251.1.2 address-family ipv4-unicast weight '10'
set protocols bgp neighbor 10.251.1.2 peer-group 'LINODE'
set protocols bgp neighbor 10.251.1.2 update-source 'wg1'
set protocols bgp neighbor 10.251.2.2 address-family ipv4-unicast weight '10'
set protocols bgp neighbor 10.251.2.2 peer-group 'LINODE'
set protocols bgp neighbor 10.251.2.2 update-source 'wg2'
set protocols bgp neighbor 10.251.3.2 address-family ipv4-unicast weight '5'
set protocols bgp neighbor 10.251.3.2 peer-group 'LINODE'
set protocols bgp neighbor 10.251.3.2 update-source 'wg3'
set protocols bgp parameters no-fast-external-failover
set protocols bgp parameters router-id '10.251.1.1'
set protocols bgp peer-group LINODE address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp peer-group LINODE bfd profile 'linode'
set protocols bgp peer-group LINODE description 'Linode transit network peers.'
set protocols bgp peer-group LINODE password 'xxxxxx'
set protocols bgp peer-group LINODE remote-as '65000'
set protocols static route 0.0.0.0/0 next-hop 5.5.5.1

Conclusion

VyOS is a pretty cool bit of software, it makes these kinds of set ups very straightforward and easy to debug. Certainly other router platforms have more tailored ways of achieving the same result, but there is a lot of utility in being able to just show your entire set up to see where you're at.

Some things of note

Throughput losses

It should be unsurprising that we would lose some speed by encapsulating most of our traffic in UDP tunnels. From my experience, if the fastest uplinks can manage 120mbps then the wireguard tunnel will operate at around 90mbps, there are two major factors here, the fragmentation of packets and the ISPs queue behaviour when dealing with UDP traffic. Sine we're starting with 1500 byte ethernet frames, these are going to be cut up as wireguard only allows 1440 bytes of payload, multiplying our packet rate by around 1.25x. The other problem is the ISP having queues several seconds deep and an unwillingness to drop anything, this obviously results in latencies going to the moon, so using BFD to sever the connection under such conditions is critical.

Perhaps in a later post I'll go over some QoS solutions, that's not some magic fix but let's us trade bandwidth for lower 99th percentile latencies during high congestion periods.

DHCP renewals and wireguard

One neat feature I found on some of these ISP gateways is when a DHCP lease is up, rather than issuing the same address again the modem will issue a new one each time, on the surface this seems completely benign but it will result in wireguard tunnels hanging and being unable to set up a session.

When I was trying to figure this out the packet capture clearly showed both sides sending handshakes back and forth except for the returned answer being sent to the VyOS interfaces prior IP address. The reason is that as the address changes it breaks the modems state association with the UDP flow, wireguard won't close the connection but instead keep sending handshakes (thus preventing the flow from expiring in the state table), since there is never a new "connect" event initiated from the client side the traffic is effectively black holed.

The solution for me was to just use static addresses. ¯\_(ツ)_/¯

Tools used in creating this post

In order to extract the configuration sensibly we can issue the VyOS show command directly as part of the ssh argument list. The cool thing about this wrapper is it detects when you're outside of configuration mode and automatically strips private information.

ssh vyos@xxx.xxx.0.1 "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration"

The next part was providing the command listing, since the configs were split up and edited for clarity the output from the router itself isn't much use. The good news is that of course the VyOS distro includes all of the translation tooling to go back and forth between each format.

cat my_conf.txt | ssh vyos@xxx.xxx.0.1 "/usr/bin/vyos-config-to-commands"

Finally in order to create the network map above I used the vis-network js library. You can grab a snapshot of my use of it here: https://gist.github.com/thetooth/f4dc14e66744d12472e647db31da54a7