- relayd and HAProxy can both be used for load balancing on pfSense, but HAProxy is more powerful, flexible, and reliable as a true proxy. - An example setup demonstrated configuring relayd and HAProxy for load balancing multiple web servers, with HAProxy able to offload SSL and use ACLs to route requests based on hostname. - Let's Encrypt certificates can be automated for HAProxy using a Lua script and the ACME package to validate via HTTP.

1. Server Load Balancing on 2.4
July 2017 Hangout
Jim Pingle

2. Project Notes
●
pfSense 2.3.4-RELEASE-p1
– Security/Errata release with an important update for OpenVPN
– If you have not already updated to 2.3.4-p1 or updated OpenVPN as mentioned last month, update immediately
– https://www.netgate.com/blog/pfsense-2-3-4-p1-release-now-available.html
● FreeRADIUS 2.x EOL, has security issues
– Uninstall it, install the FreeRADIUS 3.x package
– Same features (and more), configuration will carry over
● 2.4 progressing
– Evaluating remaining tickets
– Release Highlights: https://www.netgate.com/blog/pfsense-software-version-2-4-release-highlights.html
● FreeBSD 11, new installer, ZFS, OpenVPN 2.4.x
●
No more NanoBSD or i386 support
●
SG-1000 ARM device support, more platforms coming (SG-3100!)
– RC very soon!
● NRDM demo during on-site training, coming soon to the France, UK, Germany, and Russia
– https://www.netgate.com/training/
● Advanced Class “pfSense Supplementals I” coming soon as well
– Higher-level topics such as snort, RADIUS, DNSBL, and HAProxy

3. About this Hangout
● Server Load Balancing using relayd and HAProxy
– Primarily focused on HAProxy as it is more capable and reliable
– Coverage of relayd will be skimmed, no major changes from last hangout
– Advantages and disadvantages
– Assumes web servers are already in place, including DNS entries
● SSL Offloading in HAProxy
● ACME Integration for Let’s Encrypt certificate automation
● Redirecting requests by hostname using HAProxy ACLs

4. About relayd and HAProxy
● relayd is built into the pfSense software base installation
– Originally from OpenBSD
– Meant to work with pf directly
– Simple service for specific tasks
● HAProxy is an add-on package for pfSense
– Very powerful/flexible
– True proxy
– More capabilities, but uses more resources

5. Comparison of relayd vs HAProxy
● How they Operate
– relayd works using NAT and pf, like a “super” port forward
– HAProxy is a true proxy, accepting client connections and making new connections to servers
● Connection Handling
– relayd does not inspect the contents of packets, it forwards based on packet headers
– HAProxy can look inside the request and can act on headers
●
For example, HAProxy can route requests to specific servers based on requested hostname
– This also works, in a limited way, with non-offloaded SSL using SNI
● Service Types
– Both can handle arbitrary TCP services
– The features in HAProxy are geared toward HTTP/HTTPS, but can work with other protocols

6. Comparison of relayd vs HAProxy
●
Client Addresses
– relayd shows the client IP address to the server, so the server sees the real address
– By default, HAProxy connections appear to originate from the proxy itself
● HAProxy can add X-Forwarded-For which may be used by the web server
– Apache, nginx, and others can easily log X-Forwarded-For instead of the proxy address
– Web applications can see X-Forwarded-For as well, but support varies by package
●
HAProxy also has a Transparent Client IP setting to pass through the actual client address, similar to relayd
●
Reporting
– relayd only reports up/down status and uptime percentages
– HAProxy tracks detailed statistics about usage and distribution of traffic/requests
● Reliability
– For more than simple/basic tasks, relayd is unreliable, especially with high loads
– HAProxy is more much robust and reliable, but does consume more resources in the process

7. Comparison of relayd vs HAProxy
● Daemon/Service Binding
– Because relayd works using NAT, relayd does NOT bind to a virtual server IP address and port
– HAProxy must bind to the IP address and port specified for a Frontend virtual server
●
If the GUI or GUI redirect is running on a port to be used by HAProxy, it must be moved
● Balancing Methods
– relayd on pfSense can only use round-robin style load balancing
– HAProxy supports several balancing algorithms
● Round robin, Static Round Robin, Least Connections, Source, and more
● Client/Server Relationships
– relayd, even using Sticky, has issues maintaining client-server relationships
– HAProxy can maintain client-server relationships in several ways, such as by source address or cookie values
● SSL Offloading
– HAProxy is capable of SSL Offloading, relayd is not, due to the fact that only HAProxy is actually a proxy

8. Example Demonstration Environment
● Test setup used for this demo of relayd and HAProxy includes:
– One firewall to act as a load balancer
– Multiple web servers
– DNS entries set so www.example.com resolves to a VIP address on
WAN, along with other test hosts (example.com, blog.example.com)
– Test client on WAN
● Virtual server for relayd will be on port 8080, HAProxy on
80/443, so they can both be run at the same time for this demo

9. Example relayd setup
● Full walk-through available on the Wiki:
https://doc.pfsense.org/index.php/Inbound_Load_Balancing
● Before setting up servers, configure monitors as needed
– This example uses basic HTTP for simplicity, but a custom HTTP
monitor can check a specific URI to ensure the web server is
operational
● Ensure the target servers are on-line and operational

10. Example relayd setup
● Setup Load Balancing Pool (actual web servers that exist on the internal network)
– Services > Load Balancer, Pools tab, Add
– Name = MyServers, or something short (no spaces or special characters)
– Mode = Load Balance
– Port = 80
●
This is the port that the ACTUAL web servers are listening on, NOT the public port!
– Retry = 5, how many times to test before declaring the server dead
– Monitor = HTTP, or whichever monitor is desired
– Enter one of the actual internal web server IP addresses, then click Add to Pool, repeat as needed
until all servers are present in the Enabled column.
● If there is a separate “maintenance” or static page server to use as a fall back if all of the
above servers are down, create a separate Pool and add that server

11. Example relayd setup
●
Setup a Virtual Server entry (Public-facing address and port for the web site)
– Services > Load Balancer, Virtual Server tab, Add
– Name = MyWebSite, or something similar (short, no special characters, etc)
– IP Address = The WAN address (typically public!) on which the site is to be hosted
● If this is not the WAN address, it may need a Virtual IP address defined
● Must be hardcoded, but can be a host alias if it needs to be dynamic
– Port = 80 (or 8080 for this example), the port on which clients will connect from the outside
● Typically 80 for HTTP and 443 for HTTPS
– Virtual Server Pool = The pool defined previously (e.g. MyServers)
– Fall Back Pool = none, or choose one if one was defined previously
– Relay Protocol = TCP

12. Example relayd setup
● Visit the Settings tab (optional)
– Timeout: Milliseconds before a health check is failed, defaults to 1000ms (1 second)
– Interval: Seconds between health checks. Default is 10 seconds.
– Prefork: Not used for TCP
– With the default timers, it could be a full minute before a down server is detected.
Tuning is strongly advised!
● Add firewall rules to pass traffic to a destination of the Internal IP
addresses of the actual web servers and their ports. In this example,
pass to 10.2.0.8, 10.2.0.9 on port 80
– Aliases are handy for this!

13. Example relayd setup
● Clients on LAN cannot access servers on LAN without manual
outbound NAT rules to mask the traffic
– Interface = LAN, Source = LAN subnet, Destination = Web Servers,
Destination Port = Server Port, Translation = Interface Address
● Status is at Status > Load Balancer
– Uncheck a server and click Save to manually remove it from service
● Aim a web browser at it, e.g. http://www.example.com:8080
● When testing, use private browsing mode, close/reopen browser, etc.

14. Example HAProxy setup
● To use port 80 on HAProxy with the GUI on another port,
disable redirect on System > Advanced
● If HAProxy will use the same port as the GUI, move the GUI to
another port on System > Advanced
● Install the HAProxy package from System > Packages,
Available Packages tab
● Once installed, HAProxy is available under Services >
HAProxy

15. HAProxy – Add Backend
● Services > HAProxy, Backend tab, Add
● Name = MyWebServers (or similar)
● Servers = Add each of the actual internal web servers along with the port on which they are listening internally
– Example: Active, serverX, Address+Port, 10.2.0.x, 80, unchecked, blank
● Balance = Round Robin
● Health Check Method = HTTP
● Transparent ClientIP = Your choice
● Stick tables – (Optional – left out of this example, to show balancing from a single client)
– Stick on Existing Cookie Value
– Cookie Name = PHPSESSID
– Length = 64
– Expire = 3h
– Size = 100k (max # of concurrent clients)
● Review other settings, set as needed

16. HAProxy – Add Frontend
● Services > HAProxy, Frontend tab, Add
● Name = MyWebSite
● Status = Active
● External Address
– Listen address = WAN address (IPv4)
● The PUBLIC facing IP address of the service, to which the DNS entries resolve
● Could use an IP Alias or CARP type VIP here
– Port = 80, the PUBLIC facing port for the service
●
To run HAProxy on an alternate port, enter it here and then use a port forward to redirect traffic as needed
● Type = HTTP / HTTPS (offloading)
● Default Backend = MyWebServers
● Use 'forwardfor' option (optional) = checked, adds X-Forwarded-For header with true client IP address
● Use 'httpclose' option (optional) = httpclose, disables keep-alive, ensures X-Forwarded-For is accurate

17. HAProxy – Global Settings & Testing
● Setup HAProxy global settings
– Services > HAProxy, Settings tab
– Enable HAProxy = checked
– Maximum Connections = 1000 per backend (tune to suit available resources and
load!)
– Internal Stats Port = 2200
– Review other options & Save
● Stats tab shows server status and stats, can manually disable servers
● Add firewall rules to pass traffic to the Frontend listen IP address and port
● Aim a web browser at it, e.g. http://www.example.com
● When testing, use private browsing mode, shift+click reload or ctrl+F5

18. SSL Offloading
● SSL Offloading means that HAProxy on the firewall will handle SSL/TLS Negotiation
and encryption/decryption
– This will greatly increase the CPU burden on the firewall!
– Use hardware with AES-NI to help with crypto operations
– Decreases CPU burden on the web servers
– Communication between HAProxy and web servers can be HTTP or HTTPS
● While using HTTPS to the backend servers will consume even more resources, the best practice is to
encrypt all communications if the setup will be handling financial, medical, or other sensitive data.
● Add the CA/Cert for HAProxy to use
– Import CA, any Intermediates, and Server Cert into the Cert Manager on pfSense
– Alternately, use Let’s Encrypt with the ACME package

19. SSL Offloading
● Frontend Settings:
– Select SSL Offloading for the External Address
– SSL Offloading Section (Appears once the SSL Offloading is checked)
● Pick the server Certificate to use
● Check to add ACL for the SAN
● If there are multiple certificates for different hostnames, use “Additional Certificates” to pick them
● Backend Settings
– Exact settings depend on preferences
– For HTTP to servers, disable SSL on Server List entries, set to port 80, etc
– To also do SSL/TLS to servers, check SSL and optionally setup additional parameters

20. ACME Integration
● Let’s Encrypt lets you obtain free domain-validated SSL
certificates
– ACME Package on pfSense handles the request/processing needed
– See the April 2017 hangout on Let’s Encrypt for details
● Using ACME with HAProxy
– DNS methods work great and need no special handling in HAProxy
– For HTTP, the best method is to use a LUA script (next slide)

21. ACME Integration
● Add LUA Script to HAProxy
– Source posted with this hangout
– Or download from https://github.com/janeczku/haproxy-acme-validation-plugin/releases
– In HAProxy, Files tab, add entry “acme-http01”, “LUA Script”, paste contents of script
● Add a simple HTTP frontend that uses the script:
– Type: HTTP
– ACL: “url_acme_http01”, “Path Starts With”, “/.well-known/acme-challenge/”
– Action: “http-request lua service”, “METH_GET url_acme_http01”, lua-function: “acme-http01”
●
In the ACME Package, create cert entry, in Domain SAN List, set it to:
– Method: webroot local folder
– Root Folder: /tmp/haproxy_chroot/.well-known/acme-challenge/

22. Using HAProxy ACLs
● As shown in the ACME example, ACLs can be used to match a
request and then take an action based on that match
● Common Examples:
– ACL to match a hostname, action that matches that ACL, directs to
another backend
– ACL to match a specific path, action that directs to a different backend
– ACL to match a specific source address, add an HTTP header, deny
access, etc.

23. Using HAProxy ACLs - Example
● One public IP address, multiple web servers with different hostnames
● DNS: www.example.com and blog.example.com both resolve to your public IP Address
●
Backend:
– www.example.com is hosted on x.x.x.2 defined as backend “www”
– blog.example.com is hosted on x.x.x.3 defined as backend “blog”
● Frontend:
– ACL: host_www, Host Matches, www.example.com
●
Additional condition: host_www, Host Matches, example.com
– ACL: host_blog, Host Matches, blog.example.com
– Action: Use Backend, host_www, backend: www
– Action: Use Backend, host_blog, backend: blog

24. Conclusion
● Questions?
● Ideas for hangout topics? Post on forum, comment on the blog
posts, Reddit, etc