OpenBSD as a Firewall and Reverse Proxy/Load Balancer

Use Case

I recently was tasked with replacing some aging firewall ACL management infrastructure at work, the previous system was basically just an old SSG340 with some destination NAT rules that were updated via an expect script on an authentication server.
This, obviously, has a few issues. Primarily in that ScreenOS is difficult to dynamically update quickly, especially using an expect script that connects via SSH. Additionally, without buying additional licenses, there's no way for a given address group to exceed 256 addresses, and I believe we were limited to a total of 256 address groups. While 256^2 is more addresses than we'd likely ever need, it also makes policy management more cumbersome than it needs to be. This doesn't even touch the topic of aging hardware on a platform that is dying. Converting these rules to their SRX equivalents takes a long time and a lot of thought, and still limits our ability to expand functionality without purchasing additional licenses.
So the decision was made that we should investigate something we have more control over, initially looking at PfSense and OPNsense, but deciding that the appliance-like approach made it difficult to get the flexibility and customization we needed without having to go through the API documentation and developing new tooling to ensure the expected changes are actually being made. So after a bit of deliberation, we decided to work with OpenBSD, as it has a more recent version of pf(4), and has relayd(8) available in the base system, as well as being the home of several fantastic networking and security tools and libraries, alongside the most dead-simple configuration I've come across.

Installation

If you've never installed OpenBSD before, it's got one of the best installers available, totally text-driven through simple menus with sane defaults. The only problem I have with OpenBSD is that they don't yet have a better filesystem available than ffs. Not that there's anything wrong with it, there's just better options available that would be nice to use, not that it particularly matters when the majority of the system is mounted read-only and is generally never hosting human connections.

Getting an image

First, you absolutely need to get yourself an installation image. While there are more architectures supported and available through the OpenBSD site, generally you want the minirootXX.fs or installXX.fs images for AMD64. These images are meant to be burned to flash media, like a USB drive. The site has much better installation documentation than I'm able to write out here, so use it as your install guide. The only missing piece is that if you're on Windows, or you're not confident in your ability to use `dd(1)`, Etcher or Rufus to create the install media.

PF(4) Configuration

So the basic goal here is to set up a simple set of rules that are capable of handling all your expected traffic needs. This way, you only have to really worry about managing the contents of the tables you allow through, not the rules themselves. Now, of course, your needs are likely to diverge a fair amount from what we needed, but the rules are essentially the same: drop any packet we didn't ask for and doesn't correspond to a customer-facing use case, send unknown traffic to the auth server, send known/authenticated traffic to the website. Can't get much more simple than that, at least for this usecase.
In my situation, we needed the firewalls to be in a redundant HA configuration, so if one fails, we don't start dropping everyone's requests this also has the benefit of staggered upgrades, as if one host goes down, the other picks up the slack seamlessly.

			/etc/pf.conf:
				# defining interface macros enables simplified rules, and makes it easier to 
				# swap roles on a given interface if necessary
				int_if="bge1"
				ext_if="bge0"
				sync_if="em0"
				carp_if="carp0"
				# these two are currently the same, but are defined separately to enable
				# a migration to separate relay hosts/addresses without rewriting any filtering rules
				auth_relay="192.168.1.10"
				cust_relay="192.168.1.10"

				# with most tables being static files, it becomes easier to 
				# document which addresses you allow/block and why
				table <1918> const file "/etc/tables/rfc1918"
				table <1122> const file "/etc/tables/rfc1122"
				table <martians> const file "/etc/tables/martians"
				table <customers> const file "/etc/tables/customers"
				table <admins> const file "/etc/tables/admins"
				table <allowed> persist { 10.10.10.10 }

				# drop packets we don't want
				set block-policy drop 
				# defeat OS detection
				match all scrub (no-df random-id)

				# default to dropping all packets, and log them
				block log all
				# include rules created by relayd
				anchor "relayd/*"
				# allow outbound traffic
				pass out on $ext_if
				pass out on $int_if
				# HA allowances
				pass quick on $ext_if proto carp keep state (no-sync)
				pass quick on $sync_if proto pfsync keep state (no-sync)
				# tag connections based on their origin
				# the ordering is important, this way all connections are tagged "UNAUTH"
				# but if they're in one of the two tables allowed through, they get re-tagged as "AUTHORIZED"
				match in log on $ext_if inet proto tcp from any tag "UNAUTH"
				match in log on $ext_if inet proto tcp from <customers> tag "AUTHORIZED"
				match in log on $ext_if inet proto tcp from <allowed> tag "AUTHORIZED"

				# allow icmp monitoring
				pass in quick on $ext_if inet proto icmp
				pass in quick on $int_if inet proto icmp

				# here's where we actually filter traffic using the tags AND tables
				# again, the order matters, though this time, we want the more specific rules to 
				# come before the more generic rules, so we don't end up with all traffic being routed to the wrong relay
				# for ease of troubleshooting and management, we have separate relays for both authorized and unauthed traffic
				# as well as separate relays for plaintext http and https connections
				pass in quick log on $ext_if inet proto tcp from <customers> to ($carp_if:0) port 443 divert-to $cust_relay port 15443 tagged "AUTHORIZED"
				pass in quick log on $ext_if inet proto tcp from <customers> to ($carp_if:0) port 80 divert-to $cust_relay port 15080 tagged "AUTHORIZED"
				pass in quick log on $ext_if inet proto tcp from <allowed> to ($carp_if:0) port 443 divert-to $cust_relay port 15443 tagged "AUTHORIZED"
				pass in quick log on $ext_if inet proto tcp from <allowed> to ($carp_if:0) port 80 divert-to $cust_relay port 15080 tagged "AUTHORIZED"
				pass in quick log on $ext_if inet proto tcp from any to ($carp_if:0) port 443 divert-to $cust_relay port 16443 tagged "UNAUTH"
				pass in quick log on $ext_if inet proto tcp from any to ($carp_if:0) port 80 divert-to $cust_relay port 16080 tagged "UNAUTH"

				# Allow admins to log into the system over ssh
				pass in quick log on $ext_if inet proto tcp from <admins> to ($ext_if:0) port 22
				pass in quick log on $ext_if inet proto tcp from <admins> to ($carp_if:0) port 22

		
As you can (probably) tell, we're only concerned with TCP traffic over standard web ports except when connecting for administrative tasks. One such task being that the authentication server connects over ssh to run a custom management script (one of the host_* scripts here), which in turn updates one of the tables used by pf. The primary action anticipated is that a user logs into the auth server, is added to the "allowed" table, and is then connected to the appropriate relay to continue their session.
Of course, similar rules can be added as more services are hosted through the system.

Relayd(8) Configuration

The next important part, of course, is relayd. Relayd is an extremely powerful tool allowing you to configure it to act as an SSL accelerator, reverse proxy, proxy, load balancer, deep packet inspection host, web filter, and even gives you the ability to rewrite requests.
In my use case though, it's just acting as a proxy and SSL/TLS termination point, it's configured to allow load balancing later, but for the time being, it's offloading that task to a physical loadbalancer.

			/etc/relayd.conf:
				# macro definitions similar to the pf.conf
				relayd_host="192.168.1.10"
				http_port="80"

				# global configuration options
				# start 16 relays by default
				prefork 16
				# poll relay targets every 30s
				interval 30
				# log state changes 
				log updates
				timeout 50

				# protocol configurations
				# these are basic filtration and configuration rules 
				# we can apply to each relay
				http protocol "httpfilter" { 
					# return http error codes 
					return error
					# tell the remote endpoint the actual origin 
					match header set "X-Forwarded-For" value "$REMOTE_ADDR"
					# tell the remote endpoint which host and relay is handling the connection
					match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
					# ensure we forward the timeout value
					match header set "Keep-Alive" value "$TIMEOUT"
					# anonymize/obfuscate the server identification string
					match response header set "Server" value "Exile Heavy Industries"
					# TLS info
					tls ca key "/etc/ssl/private/tls.key
					tls ca cert "/etc/ssl/tls.crt"
					tls ca file "/etc/ssl/tls.pem"
					# ensure we're only accepting a certain level of encryption
					tls { no tlsv1.0, ciphers HIGH } 
					# track sessions
					match query hash "sessid"
					# log the url requests
					pass url log
					# filter traffic based on the pf tag
					pass tagged "AUTHORIZED"
					block tagged "UNAUTH"
				}

				# of course, there's an identical one with the tags switched 
				# that exclusively handles traffic from unknown addresses
				# though the protocol will need a different name, like "screening"

				# table definitions
				# these addresses are tested 5 times each before getting dropped from the pool
				# this is one area where relayd can add loadbalancing functionality 
				# an important note is that while these tables resemble pf tables, they are not shared
				# with pf
				table <sites> { 192.168.15.5 192.168.15.10 retry 5 } 
				table <auth_server> { 192.168.1.16 retry 5 }

				# relays
				# this is where the most fun stuff comes into play, by default, a relay can be used 
				# as a simple proxy. depending on how you want each relay to behave, it can enable 
				# deep packet inspection via a MITM setup, act as an SSL accelerator by being the SSL/TLS
				# termination point and communicating to hosts behind it via HTTP. this can enable a server without
				# good SSL/TLS support to get upgraded to the latest version supported by LibreSSL in whatever branch of 
				# OpenBSD you're using. additionally, it can rewrite requests, act as an advanced web filter for environments
				# where you would need to restrict web traffic to certain sites like facebook, twitter, porn sites, gaming sites
				# and so on. 
				relay "www" { 
					# define which address and port to listen on, and only accept encrypted traffic
					listen on $relayd_host port 15443 tls
					# use our special protocol for these connections
					protocol "httpfilter" 
					# this part proxies the connection to hosts in <sites> that 
					# return the specified code when checking a given location on the remote server
					# there's several other checks, but this ensures that we send the user to a working webserver
					forward to <sites> port $http_port mode loadbalance check http "/index.php" code 302
				}

				# this relay handles our unauthorized traffic
				relay "screening" { 
					listen on $relayd_host port 16443 tls
					protocol "screening" 
					forward to <auth_server> port $http_port mode loadbalance check http "/index.php" code 200
				}
		
And Viola! You've now got a nice, simple firewall configuration forwarding connections to a TLS accelerating reverse proxy that only needs to have additional hosts added into its tables to also act as a load balancer. With such a configuration, you'll want to keep an eye on your resource utilization for a while, it may be necessary to do some performance tweaking to prevent the relays from becoming slow, or having the host handle too many connections at a time, using all your RAM.
It may be necessary to get additional hosts to offload specific tasks or configure bonded/LAGG connections to handle better bandwidth. Whatever your organization's needing to filter, it's hard to get more useful than a good OpenBSD install.