How To Firewall, Failover, Load Balance, And Throttle Multiple Network Interfaces In Linux

There is a plethora of documentation on how to configure individual aspects of Linux boxes used as gateways; however, what is needed is a single document that comprehensively details ALL aspects of a gateway's configuration, particularly when multiple public and private network interface cards (NICs) are employed. The aspects to which I'm referring are:

firewall - keep the nasty stuff out and let nice stuff in/out
failover - keep your gateway users happy as long as at least one public NIC is alive
load balance - leverage the bandwidth on all public NICs
throttle - ensure public NICs aren't choked by limiting their user space upload bandwidth

Combining the aspects is crucial for a Linux router and is why I wrote fifalobath and its associated scripts.

firewall + failover + load balance + throttle == fifalobath

Each of the aspects requires multiple degrees of configuration:

standard - thwarting the "ping of death", allowing ssh on external interfaces, etc
boilerplate - port forwarding, traffic shaping, etc
custom - opening ports for specific machines and/or under certain conditions, etc

In the simplest case, that of a single public interface, fifalobath creates a highly customizable and easy to use firewall. In the most complex case, that of multiple public and private interfaces, fifalobath handles all the aspects and their degrees of configuration in a modular way. The modularity keeps things simple.

If you've already setup fifalobath and are having trouble then go to the Troubleshooting section.

Example

The following ASCII art represents a Linux router that has two internal (private) interfaces, eth0 and wlan0, in addition to two independent Internet service providers (ISPs) on eth1 and ppp0, respectively. It is this configuration that is used as an example.

                                                                      ________
        __                                     +------------+        /
    ___/  \_                                   |            |       |
  _/        \__                   +------------+ Provider 1 +-------
 /             \                  |            |            |     /
| Local network ----      +-------+--------+   +------------+    |
 \_           __/    \    |       eth1     |                    /
   \__     __/        ----+ eth0           |                   |
      \___/               |                |                   |
        __                |   Linux router |                   |     Internet
    ___/  \_              |                |                   |
  _/        \__       ----+ wlan0          |                   |
 /             \     /    |       ppp0     |                    \
| Local network ----      +-------+--------+   +------------+    |
 \_           __/                 |            |            |     \
   \__     __/                    +------------+ Provider 2 +-------
      \___/                                    |            |       |
                                               +------------+        \________

I'm most familiar with Fedora, so that's the context for the scripts. They can be applied to other distributions (Ubuntu, Suse, Gentoo, etc) by making appropriate file location/NIC configuration substitutions. Centos and RHEL should be good to go right out of the box. You need to be root to do all that is outlined in this document. I'll start with the basic configuration options and then move on to advanced options.

Setup

As root you can copy-and-paste these commands into a shell in order to complete the setup. Each command is preceded by a comment about what it does; you can cut-and-paste the comments - they will be ignored by the shell. For commands in "optional" stanzas you can either prefix them with "#" or not cut-and-paste them if you don't want their functionality.

## set the location of the fifalobath script
export FIFALOBATH=/etc/sysconfig/network-scripts/fifalobath

## get it
curl 'http://www.avaritia.com/fifalobath/fifalobath' > "$FIFALOBATH"

## make it executable
chmod a+x "$FIFALOBATH"

## add fifalobath file locations to /etc/sysconfig/network
cat <<EOS >> /etc/sysconfig/network
# http://www.avaritia.com/linux_network_firewall_failover_load_balance_throttle
# firewall, failover, load balance, throttle: fifolobath
FIFALOBATH="$FIFALOBATH"
FIFALOBATH_DYN=/var/log/dynamic_nic_info.log # dynamic interface info log
EOS

## add NIC info to /etc/sysconfig/network; eg, for the ASCII art above
cat <<EOS >> /etc/sysconfig/network
EXTIFs="eth1 ppp0"  # external (public) interfaces; primary first
INTIFs="eth0 wlan0" # internal (private) interfaces
EOS

## optional customizations to /etc/resolv.conf; comment them out if you don't want them
cat <<EOS >> /etc/sysconfig/network
RESOLV_SEARCH="search domain_with_dns_on_localhost.com some_other_domain.com"
RESOLV_DNS="nameserver 127.0.0.1\nnameserver 1.2.3.4" # \n delimited additions to DNSs
EOS

## optional logging of packets dropped by the firewall
cat <<EOS >> /etc/sysconfig/network
LOG_DROPPED=1       # log dropped packets
EOS

## optional common firewall options
cat <<EOS >> /etc/sysconfig/network
ALLOW_PING=1        # comment out to disallow pings on EXTIFs
ALLOW_SSH=1         # comment out to disallow ssh on EXTIFs - HIGHLY recommended to keep
ALLOW_DHCPD=1       # comment out ot disallow dhcpd request on INTIFs
EOS

## optional failover timeout
cat <<EOS >> /etc/sysconfig/network
ROUTE_TIMEOUT=3     # failover timeout in seconds
EOS

## optional flag for load balancing
cat <<EOS >> /etc/sysconfig/network
LOAD_BALANCE=1      # comment out for failover only, no load balancing
EOS

## throttling variables; if THROTTLE is set then ALLOT_REALTIME and ALLOT_P2P are mandatory
cat <<EOS >> /etc/sysconfig/network
THROTTLE=0.9        # maximum fraction of each external interface's UPRATE available to users
ALLOT_REALTIME=0.15 # dedicated fraction of bandwidth for skype, rtp, etc
ALLOT_P2P=0.05      # dedicated fraction of bandwidth for p2p
EOS

Alternatively, after you've obtained fifalobath, here's an example of /etc/sysconfig/network that can be edited in place and then cut-and-paste into your /etc/sysconfig/network.

NETWORKING=yes
HOSTNAME=localhost

# http://www.avaritia.com/linux_network_firewall_failover_load_balance_throttle
# firewall, failover, load balance, throttle: fifolobath
FIFALOBATH=/etc/sysconfig/network-scripts/fifalobath
FIFALOBATH_DYN=/var/log/dynamic_nic_info.log # dynamic interface info log

EXTIFs="eth1 ppp0"  # external (public) interfaces; primary first
INTIFs="eth0 wlan0" # internal (private) interfaces

RESOLV_SEARCH="search domain_with_dns_on_localhost.com" # for /etc/resolv.conf
RESOLV_DNS="nameserver 127.0.0.1\n; nameserver 1.2.3.4" # \n delimited

LOG_DROPPED=1       # log dropped packets

ALLOW_PING=1        # comment out to disallow pings on EXTIFs
ALLOW_SSH=1         # comment out to disallow ssh on EXTIFs
ALLOW_DHCPD=1       # comment out ot disallow dhcpd request on INTIFs

ROUTE_TIMEOUT=3     # failover timeout in seconds

LOAD_BALANCE=1      # comment out for failover only, no load balancing

THROTTLE=0.9        # maximum fraction of each external interface's UPRATE
ALLOT_REALTIME=0.15 # dedicated fraction of bandwidth for skype, rtp, etc
ALLOT_P2P=0.05      # dedicated fraction of bandwidth for p2p

If none of the optional configuration variables are set then the basic setup is a simple firewall that does network address translation (NAT) and forwards packets between internal interfaces. NAT allows machines on the local nets to access the Internet. The forwarding, for example, allows machines on the eth0 subnet to browse samba shares on the wlan0 subnet and vice versa - nice!

For the ASCII art example, we need to specify the load balancing weights and upload rates for eth1 and ppp0. Assuming eth1 gets 20Mbps down and 1Mbps up and ppp0 gets 3Mbps down and 512kbps up:

## specify eth1's load balancing weight and upload rate
cat <<EOS >> /etc/sysconfig/network-scripts/ifcfg-eth1
WEIGHT=20
UPRATE=1mbps
EOS

## specify ppp0's load balancing weight and upload rate
cat <<EOS >> /etc/sysconfig/network-scripts/ifcfg-ppp0
WEIGHT=3
UPRATE=512kbps
EOS

Now it's time to do make sure that all the ancillary stuff upon which fifalobath relies is available:

## check that the necessary load balancing/throttling variables are set
$FIFALOBATH ncheck # skip this if you just want a firewall

## check that the necessary entries are in /etc/ip_route2/rt_tables
$FIFALOBATH tcheck # skip this if you just want a firewall

## check that the necessary executables are available
$FIFALOBATH xcheck

## all together now
$FIFALOBATH ncheck && $FIFALOBATH tcheck && $FIFALOBATH xcheck

If you saw "all good" across the board then you're all good. Otherwise, follow the directions provided by the output to correct the problems. Re-run the checks until you get all the "all good"s.

If your Linux router is assigned IP addresses dynamically then you'll also need /etc/sysconfig/network-scripts/dhclient-exit-hooks:

## grab dhclient-exit-hooks
curl 'http://www.avaritia.com/fifalobath/dhclient-exit-hooks' > /etc/sysconfig/network-scripts/dhclient-exit-hooks

So, in Fedora, the first thing to do is chkconfig network on && yum -y remove NetworkManager\* && service network restart because NetworkManager sucks at handling sophisticated network configurations.

Next, since fifalobath will manage the firewall and uses it uses rsyslog for logging

service iptables stop && chkconfig iptables off

Network configuration information that fifalobath reads needs to be added to /etc/sysconfig/network like this:

- a lot of rules are standard; most rules are boilerplate; custum rules are custom
- rsyslog before network
- last step is cd /etc/dhcp/ && ln -s dh..exit* ; ( cd /sbin && ln -s fifalobath ifup-local && ln -s fifa ifdown-local )

Troubleshooting

If something goes wrong try

## "reset" the firewall and routing
/etc/sysconfig/network-scripts/fifalobath bootstrap

Cutting-and-pasting the above commands into a shell will flush all the firewall and routing rules. If that doesn't work then manually edit $FIFALOBATH_DYN and insert the appropriate values for any interfaces with dynamic IP addresses.

Miscellaneous

Ethernet bonding has amazing failover capabilities but only works for certain NICs - search through the kernel source code to find them - and cannot be used if you use two independent ISPs.

Files & Links

/etc/sysconfig/network-scripts/dhclient-exit-hooks

#!/bin/bash
# $Id: dhclient-exit-hooks 38 2009-08-31 10:15:00Z dave $

. /etc/sysconfig/network

if [ ! -n "$FIFALOBATH_DYN" ] || [ `touch "$FIFALOBATH_DYN"` ]; then
   echo "FIFALOBATH_DYN in /etc/sysconfig/network must be writable" && exit -1
fi

. "$FIFALOBATH_DYN"

eval "
   old_GW=\$GW${interface}
   old_DNS=\$DNS${interface}
   old_NM=\$NM${interface}
   old_IP=\$IP${interface}
"

if [ "$old_GW" != "${new_routers}" ]; then
   echo "export GW${interface}='${new_routers}'" >> "$FIFALOBATH_DYN"
   CHANGED=1
fi

if [ "$old_NM" != "${new_subnet_mask}" ]; then
   echo "export NM${interface}='${new_subnet_mask}'" >> "$FIFALOBATH_DYN"
   CHANGED=1
fi

if [ "$old_IP" != "${new_ip_address}" ]; then
   echo "export IP${interface}='${new_ip_address}'" >> "$FIFALOBATH_DYN"
   CHANGED=1
fi

if [ "$old_DNS" != "${new_domain_name_servers}" ]; then
   echo "export DNS${interface}='${new_domain_name_servers}'" >> "$FIFALOBATH_DYN"
   NEWDNS=1
fi

if [ "${reason}" != "REBOOT" ] && [ -n "${new_ip_address}" ] && [ $CHANGED ]; then
   "$FIFALOBATH" ${interface}
elif [ "${reason}" == "RENEW" ] && [ ! -z $NEWDNS ]; then
   "$FIFALOBATH" ${interface} dns
fi

/etc/sysconfig/network-scripts/fifalobath

#!/bin/bash
# Copyright Dave Puchyr 2009.  All Rights Reserved.
# $Id: fifalobath 61 2010-04-14 19:31:31Z dave $

# important vars
TC=tc
IP=ip
IPTABLES=iptables
RESOLV=/etc/resolv.conf
# uncomment these for dry-runs
#TC="echo tc"
#IP="echo ip"
#IPTABLES="echo iptables"
#RESOLV=/tmp/resolv.conf

# rock 'n' roll
. /etc/sysconfig/network

ME=$(stat $0 -c "%N" | sed -e 's/.*-> `//' -e "s/[\`']//g")
TAG=`basename $ME`
ME="$(cd $(dirname "$ME"); pwd)/$TAG"
LOG="logger -t $TAG"
ERR="$LOG ERROR: "

[ -z $1 ] && cat <<EOS && cat /etc/sysconfig/network && exit 0
usage: $ME ncheck | tcheck | xcheck | NIC [ dns ]
 bootstrap - writes firewall and routing rules without relying on cached data
    ncheck - checks that the necessary variables for load balancing and
             throttling are set
    tcheck - checks that the necessary entries are in /etc/ip_route2/rt_tables
    xcheck - checks that the necessary executables that are available
    NIC    - reads configuration variables from /etc/sysconfig/network and
             performs firewalling, failover, load balancing, and throttling
             where NIC is an interface specified in /etc/sysconfig/network.
             This script logs to /var/log/messages with the tag '$TAG'.
             See http://www.avaritia.com/linux_network_firewall_failover_load_balance_throttle
             for details.
    reload - updates $RESOLV and reloads the firewall, etc

Contents of /etc/sysconfig/network is currently
EOS

# check for necessary load balancing and throttling variables
if [ "$1" == "ncheck" ]; then
   BAD=0
   for EXTIF in $EXTIFs; do
      # reset vars
      WEIGHT=
      UPRATE=

. /etc/sysconfig/network-scripts/ifcfg-$EXTIF

if [ ! -z $LOAD_BALANCE ] && [ -z $WEIGHT ]; then
         echo "$EXTIF will be assigned a default load balancing weight of 1"
      fi

if [ ! -z $THROTTLE ]; then
         if [ -z $UPRATE ]; then
            echo "$EXTIF needs an UPRATE, eg UPRATE=512kbits"
            BAD+=1
         fi
         if [ -z $ALLOT_REALTIME ] || [ -z $ALLOT_P2P ]; then
            echo "$EXTIF needs both ALLOT_REALTIME and ALLOT_P2P to be between 0 and 1"
            BAD+=1
         fi
      fi
   done
   [ $BAD == 0 ] && echo "all good"
   exit
fi

# check for necessary executables
if [ "$1" == "xcheck" ]; then
   for i in awk basename cut date dirname egrep fgrep grep host ifconfig ip ipcalc iptables logger ls mktemp route sed stat tac tc traceroute wc; do
      which $i 2>&1 > /dev/null || exit -1
   done
   echo "all good" && exit 0
fi

# check for rt_tables entries
if [ "$1" == "tcheck" ]; then
   RT_TABLES=`grep '^[0-9]' /etc/iproute2/rt_tables | sed 's/[\t ]/ /g'` # strip the comments out of rt_tables
   ALL_GOOD=1

for EXTIF in $EXTIFs; do
      if ! `echo -e $RT_TABLES | grep -q "[0-9] $EXTIF$TAG"`; then
         echo "$EXTIF$TAG entry in /etc/iproute2/rt_tables is REQUIRED"
         ALL_GOOD=
      fi
   done
   [ -n "$ALL_GOOD" ] && echo "all good"
   exit 0
fi

# check if we're interested in the interface
if ! `echo "$EXTIFs $INTIFs bootstrap reload" | fgrep -q "$1"`; then
   $LOG "$ME ignoring $1 and exiting since it's not one of $EXTIFs $INTIFs as defined in /etc/sysconfig/network"
   exit 0;
fi

# pppd and its call to /etc/ppp/ip-up are woefully inadequate since the DNS(s) are not provided!
if `echo $1 | fgrep -q ppp`; then
   PPPHACK=`tac /var/log/messages | egrep -m 4 'pppd.*((((local|remote) +IP)|(primary|secondary) +DNS) address)'`
   DNS2=`echo $PPPHACK    | sed -e 's/.*secondary DNS address //' -e 's/ .*//'`
   DNS1=`echo $PPPHACK    | sed -e 's/.*primary DNS address //' -e 's/ .*//'`
   GATEWAY=`echo $PPPHACK | sed -e 's/.*remote IP address //' -e 's/ .*//'`
   IPADDR=`echo $PPPHACK  | sed -e 's/.*local IP address //' -e 's/ .*//'`
   INTERFACE="$1"
   cat <<EOS >> "$FIFALOBATH_DYN"
      # PPPHACK $INTERFACE
       IP$INTERFACE=$IPADDR;
       NM$INTERFACE=255.255.255.255;
       GW$INTERFACE=$GATEWAY;
      DNSs$INTERFACE='$DNS1 $DNS2';
      DNS1$INTERFACE=$DNS1;
      DNS2$INTERFACE=$DNS2;
EOS
fi

# at this point there's something more complex to do
[ -s "$FIFALOBATH_DYN" ] && . "$FIFALOBATH_DYN"
cd /etc/sysconfig/network-scripts
. ./network-functions

# functions
act() {
   $1 && $LOG "did $1" || $ERR "puked on $1"
}

actlog() {
   $1 > /dev/null 2> /dev/null ; $LOG "$1 # exited with $?"
}

commit_resolv() {
   act "chmod a+r $1"
   act "mv $1 $RESOLV"
}

throttle_single_port() {
   SDPORT=$1

for DELIMITED in $2; do
      SERVICE=`echo $DELIMITED | cut -d'/' -f 2`
      PORT=`echo $DELIMITED | cut -d'/' -f 1`
      PROTOCOLS=`echo $DELIMITED | cut -d'/' -f 3 | sed 's/,/ /g'`
      if [ ! -n "$PROTOCOLS" ]; then PROTOCOLS=tcp; fi

for PROTOCOL in $PROTOCOLS; do
         if ! `echo "tcp udp icmp" | fgrep -q "$PROTOCOL"`; then
            $ERR "invalid protocol '$PROTOCOL' extracted from '$DELIMITED'"
            continue
         fi

act "$IPTABLES -t mangle -A THROTTLE -p $PROTOCOL --$SDPORT $PORT -j CLASSIFY --set-class $CLASS"
      done
   done
}

throttle_multi_port() {
   for DELIMITED in $1; do
      SERVICE=`echo $DELIMITED | cut -d'/' -f 2`
      PORT=`echo $DELIMITED | cut -d'/' -f 1`
      PROTOCOLS=`echo $DELIMITED | cut -d'/' -f 3 | sed 's/,/ /g'`
      if [ ! -n "$PROTOCOLS" ]; then PROTOCOLS=tcp; fi

for PROTOCOL in $PROTOCOLS; do
         if ! `echo "tcp udp" | fgrep -q "$PROTOCOL"`; then # only tcp and udp allowed
            $ERR "invalid protocol '$PROTOCOL' extracted from '$DELIMITED'"
            continue
         fi

act "$IPTABLES -t mangle -A THROTTLE -p $PROTOCOL -m multiport --ports $PORT -j CLASSIFY --set-class $CLASS"
      done
   done
}

# determine which interfaces are active and their ip addresses
for INTERFACE in $EXTIFs $INTIFs; do
   if ! LC_ALL=C ip -o link 2> /dev/null | grep -q "${INTERFACE}[:@].*,UP"; then
      $LOG "$INTERFACE is down"
      continue
   fi

if echo "$EXTIFs" | fgrep -q "$INTERFACE"; then
      ACTIVE_EXTIFs="$ACTIVE_EXTIFs $INTERFACE"
   else
      ACTIVE_INTIFs="$ACTIVE_INTIFs $INTERFACE"
   fi

IPADDR=
   NETMASK=
   GATEWAY=
   DNSs=
   DNS1=
   DNS2=

need_config $INTERFACE
   source_config

# set the load balance weight
   [ -z "$WEIGHT" ] && WEIGHT=1
   eval "
      UP$INTERFACE=$UPRATE;
      W$INTERFACE=$WEIGHT;
   "

if [ -n "$IPADDR" ]; then
      eval "
           IP$INTERFACE=$IPADDR;
           NM$INTERFACE=$NETMASK;
           GW$INTERFACE=$GATEWAY;
         DNS1$INTERFACE=$DNS1;
         DNS2$INTERFACE=$DNS2;
      "
   else
      DYNAMIC_IFs="$DYNAMIC_IFs $INTERFACE"

if [ "$1" == "bootstrap" ]; then
         INFO=`ifconfig $INTERFACE | fgrep 'inet addr:' | sed 's/.*inet addr:'//`
         IPADDR=`echo $INFO | sed 's/ .*//'`;
         NETMASK=`echo $INFO | sed 's/.*Mask://'`
         GATEWAY=`traceroute -n -i $INTERFACE 74.125.127.100 -m 1 | egrep '^ 1' | sed -e 's/ 1  //' -e 's/ .*//' | egrep '([0-9]+\.){3}[0-9]'`

if [ "x$GATEWAY" == "x" ]; then
            MISSING="$MISSING $INTERFACE"
            cat <<EOS >> "$FIFALOBATH_DYN"
            # TODO: manually configure $INTERFACE
            export IP$INTERFACE=$IPADDR;
            export NM$INTERFACE=$NETMASK;
            export GW$INTERFACE=$GATEWAY;
            export DNSs$INTERFACE='$DNSs';
            export DNS1$INTERFACE=$DNS1;
            export DNS2$INTERFACE=$DNS2;
EOS
         fi
      fi
   fi
done

if [ "$1" == "bootstrap" ]; then
   # abort if we're missing info
   [ -n "$MISSING" ] && echo "manually configure $MISSING in $FIFALOBATH_DYN" && exit -1

# otherwise force dns to use current values
   RESOLV_SEARCH=`egrep -v '^(^[;#])' /etc/resolv.conf | fgrep search`
   RESOLV_DNS=`egrep -v '^(^[;#])' /etc/resolv.conf | fgrep nameserver`
   RESOLV_SEARCH="; RESOLV_SEARCH and RESOLV_DNS from $ME boostrap\n$RESOLV_SEARCH"
fi

# generate resolv.conf
RESOLV_TMP=`mktemp /tmp/XXXXXX`
NOW=`date`
echo "; written by $ME using /etc/sysconfig/network at $NOW" > "$RESOLV_TMP"
[ -n "$RESOLV_SEARCH" ] && echo -e "$RESOLV_SEARCH ; from RESOLV_SEARCH" >> "$RESOLV_TMP"
[ -n "$RESOLV_DNS" ] && echo -e "; from RESOLV_DNS\n$RESOLV_DNS" >> "$RESOLV_TMP"
for EXTIF in $ACTIVE_EXTIFs; do
   echo "; from $EXTIF" >> "$RESOLV_TMP"
   eval "DNS=\$DNS1$EXTIF"
   [ -n "$DNS" ] && echo "nameserver $DNS" >> "$RESOLV_TMP"
   eval "DNS=\$DNS2$EXTIF"
   [ -n "$DNS" ] && echo "nameserver $DNS" >> "$RESOLV_TMP"
done

# firewall stuff first
UNIVERSE="0.0.0.0/0"

# rock 'n' roll
$LOG "script: $0 $@ maps to $ME $@"
$LOG "external interface(s): $ACTIVE_EXTIFs"
$LOG "internal interface(s): $ACTIVE_INTIFs"

if echo "$0" | fgrep -q "down"; then
   $LOG "bailing on $ME since $0 matches 'down'"
   exit 0
fi

# clear existing rules
$IPTABLES -P INPUT DROP
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT DROP
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -F -t nat
$IPTABLES -F -t mangle
`$IPTABLES -L | fgrep -q SYNFLOOD` && $IPTABLES -F SYNFLOOD && $IPTABLES -X SYNFLOOD
$IPTABLES -N SYNFLOOD
`$IPTABLES -L | fgrep -q DROPIT` && $IPTABLES -F DROPIT && $IPTABLES -X DROPIT
$IPTABLES -N DROPIT
$IPTABLES -Z # reset counters

# limit syn flood attacks
$IPTABLES -A INPUT -p tcp --syn -j SYNFLOOD &&
$IPTABLES -A SYNFLOOD -m limit --limit 1/s --limit-burst 3 -j RETURN &&
$IPTABLES -A SYNFLOOD -j DROP &&
$IPTABLES -A INPUT -p tcp ! --syn -m state --state NEW -j DROP &&
$IPTABLES -A INPUT -f -j DROP &&
$IPTABLES -A INPUT -p tcp --tcp-flags ALL ALL -j DROP &&
$IPTABLES -A INPUT -p tcp --tcp-flags ALL NONE -j DROP &&
   $LOG "limited syn flood attacks"

# http://www.netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-7.html
$IPTABLES -A FORWARD -p tcp --syn -m limit --limit 1/s -j ACCEPT &&
$IPTABLES -A FORWARD -p tcp --tcp-flags SYN,ACK,FIN,RST RST -m limit --limit 1/s -j ACCEPT &&
$IPTABLES -A FORWARD -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT &&
   $LOG "limited syn flood (again), furtive port scanner, and ping of death"

if [ -z $LOG_DROPPED ]; then
   act "$IPTABLES -A DROPIT -j REJECT"
else
   act "$IPTABLES -A DROPIT -j LOG --log-level info"
fi

# make lo good to go
$IPTABLES -A INPUT  -i lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT
$IPTABLES -A OUTPUT -o lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT

# make private NICs good to go
nINTIFs=`echo "$ACTIVE_INTIFs" | wc -w`
for INTIF in $ACTIVE_INTIFs; do # loop on ACTIVE_INTIFs
   eval " \
      INTIP=\$IP$INTIF; \
      INTMASK=\$NM$INTIF; \
   "
   INTNET="$INTIP/$INTMASK"
   INTIP="$INTIP/32"

$IPTABLES -A INPUT -i $INTIF -s $INTNET -d $UNIVERSE -j ACCEPT &&
      $LOG "allowing $INTNET to connect anywhere"

if [ ! -z $ALLOW_DHCPD ]; then
      $IPTABLES -A INPUT -i $INTIF -p tcp --sport 68 --dport 67 -j ACCEPT &&
      $IPTABLES -A INPUT -i $INTIF -p udp --sport 68 --dport 67 -j ACCEPT &&
         $LOG "allowing dhcpd request on $INTIF"

$IPTABLES -A OUTPUT -o $INTIF -p tcp -s $INTIP --sport 67 -d 255.255.255.255 --dport 68 -j ACCEPT &&
      $IPTABLES -A OUTPUT -o $INTIF -p udp -s $INTIP --sport 67 -d 255.255.255.255 --dport 68 -j ACCEPT &&
         $LOG "allowing dhcpd requests from $INTIP"
   fi

# allow internal interfaces to forward to each other
   for OTHERINTIF in $ACTIVE_INTIFs; do
      [ "$INTIF" == "$OTHERINTIF" ] && continue

$IPTABLES -A FORWARD -i $INTIF -o $OTHERINTIF -j ACCEPT &&
         $LOG "allowing forwarding between $INTIF and $OTHERINTIF"
   done

# holes for the intranet
   [ -x "$ME.private" ] && . "$ME.private"

$IPTABLES -A OUTPUT -o $INTIF -s $INTIP -d $INTNET -j ACCEPT &&
      $LOG "allowing $INTNET from $INTIP"
done # loop on ACTIVE_INTIFs

# external (public) interface(s)
for EXTIF in $ACTIVE_EXTIFs; do # loop on ACTIVE_EXTIFs
   eval "EXTIP=\$IP$EXTIF"

# allow cross-traffic between the external interfaces
   for OTHEREXTIF in $ACTIVE_EXTIFs; do
      eval "OTHEREXTIP=\$IP$OTHEREXTIF"
      [ "$EXTIP" == "$OTHEREXTIP" ] && continue
      $IPTABLES -A INPUT  -i $OTHEREXTIF -d $EXTIP -j ACCEPT
      $IPTABLES -A OUTPUT -o $EXTIF -s $OTHEREXTIP -j ACCEPT
   done

# allow pings
   if [ ! -z $ALLOW_PING ]; then
      $IPTABLES -A INPUT -i $EXTIF -p ICMP -s $UNIVERSE -d $EXTIP -j ACCEPT &&
         $LOG "allowing pings on $EXTIP"
   fi

# allow ssh
   if [ ! -z $ALLOW_SSH ]; then
      $IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED -p tcp -s $UNIVERSE -d $EXTIP --dport 22 -j ACCEPT &&
         $LOG "allowing ssh on $EXTIP"
   fi

# allow related back into the box
   $IPTABLES -A INPUT -i $EXTIF -s $UNIVERSE -d $EXTIP -m state --state ESTABLISHED,RELATED -j ACCEPT &&
      $LOG "alowing establish/related on $EXTIP"

# internal (private) interface(s)
   for INTIF in $ACTIVE_INTIFs; do # loop on ACTIVE_INTIFs
      eval " \
         INTIP=\$IP$INTIF; \
         INTMASK=\$NM$INTIF; \
      "
      INTNET="$INTIP/$INTMASK"
      INTIP="$INTIP/32"

$IPTABLES -A INPUT -i $EXTIF -s $INTNET -d $UNIVERSE -j DROPIT &&
         $LOG "thwarting spoofing on $INTIF"

$IPTABLES -A OUTPUT -o $INTIF -s $EXTIP -d $INTNET -j ACCEPT &&
         $LOG "allowing $INTNET from $EXTIP"

$IPTABLES -A OUTPUT -o $EXTIF -s $UNIVERSE -d $INTNET -j DROPIT &&
         $LOG "dropping all outgoing to $INTNET on $EXTIF"

$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED -j ACCEPT &&
         $LOG "allowing related connections into $INTIF"

$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT &&
         $LOG "allowing all connections out of $INTIF"
   done # loop on ACTIVE_INTIFs

# allow everything else out
   $IPTABLES -A OUTPUT -o $EXTIF -s $EXTIP -d $UNIVERSE -j ACCEPT &&
      $LOG "allowing everything out from $EXTIP"

# do masquerading
   $IPTABLES -t nat -A POSTROUTING -o $EXTIF -j SNAT --to $EXTIP &&
      $LOG "SNATed $EXTIP"

# holes in the firewall, ie ports/services that are accessible to the public
   [ -x "$ME.public" ] && . "$ME.public"
done # loop on ACTIVE_EXTIFs

# trusted and public get a loop of their own so that they can specify host names, not just interface names
for EXTIF in $ACTIVE_EXTIFs; do # loop on ACTIVE_EXTIFs
   eval "EXTIP=\$IP$EXTIF"

# peep holes in the firewall, ie ports/services that are accessible to trusted hosts
   [ -x "$ME.trusted" ] && . "$ME.trusted"

# tunnels in the firewall, ie ports/services that are on the intranet but are accessible to the public
   [ -x "$ME.forwarding" ] && . "$ME.forwarding"
done # loop on ACTIVE_EXTIFs

# catch all on INPUT
$IPTABLES -A INPUT -s $UNIVERSE -d $UNIVERSE -j DROPIT &&
   $LOG "dropping all other incoming"

# catch all on OUTPUT
$IPTABLES -A OUTPUT -s $UNIVERSE -d $UNIVERSE -j DROPIT &&
   $LOG "dropping all other outgoing"

# catch all on FORWARD
$IPTABLES -A FORWARD -j DROPIT &&
   $LOG "dropping all other forwarding"

# balancing, failover, and routing
if (( $nINTIFs > 0 )); then
   echo "1" > /proc/sys/net/ipv4/ip_forward && $LOG "enabled ip forwarding"
fi

# failover
nEXTIFs=`echo "$ACTIVE_EXTIFs" | wc -w`
if (( $nEXTIFs > 1 )) && [ -n "$ROUTE_TIMEOUT" ]; then
   # set failover timeout
   echo "$ROUTE_TIMEOUT" > /proc/sys/net/ipv4/route/gc_timeout && $LOG "set gc_timeout to $ROUTE_TIMEOUT"
fi

# balancing: delete all routes and rules just to keep things simple
actlog "$IP route del default"

for EXTIF in $EXTIFs; do # loop on all EXTIFs, not just active ones
   eval "IPADDR=\$IP$EXTIF"
   if [ -n "$IPADDR" ]; then
      ROUTE=`ip route | fgrep -q $IPADDR` # not $IP so dry-runs work as expected
      if [ -n "$ROUTE" ]; then actlog "$IP route del $ROUTE"; fi
   fi

actlog "$IP route del table $EXTIF$TAG"

actlog "$IP rule del table $EXTIF$TAG"
done # loop on all EXTIFs, not just active ones

for EXTIF in $ACTIVE_EXTIFs; do # loop on ACTIVE_EXTIFs
   # get the "current" values
   eval "
      IPADDR=\$IP$EXTIF;
      NM=\$NM$EXTIF;
      GW=\$GW$EXTIF;
      W=\$W$EXTIF;
   "
   eval `ipcalc -p $IPADDR $NM` # sets PREFIX
   eval `ipcalc -n $IPADDR $NM` # sets NETWORK

# add EXTIF's route
   actlog "$IP route add $NETWORK/$PREFIX dev $EXTIF src $IPADDR table $EXTIF$TAG"

# add EXTIF as a default route
   actlog "$IP route add default via $GW table $EXTIF$TAG"

# add EXTIF's route to the main table
   # automagically added? act "$IP route add $NETWORK/$PREFIX dev $EXTIF src $IPADDR"

# add EXTIF's rule
   act "$IP rule add from $IPADDR table $EXTIF$TAG"

# add default route if no load balancing
   if [ "$nEXTIFs" == "1" ] || [ -z $LOAD_BALANCE ] && [ -z $DEFAULTED ]; then
      act "$IP route add default via $GW"
      DEFAULTED=1
   else
      WEIGHTS="$WEIGHTS nexthop via $GW dev $EXTIF weight $W"
   fi

# add rules for marked packets
   MARK=`fgrep $EXTIF$TAG /etc/iproute2/rt_tables | sed 's/[ \t].*$//'`
   eval "export MARK$EXTIF=$MARK" # export to make MARKs available for $ME.mark
   act "$IP rule add from all fwmark $MARK table $EXTIF$TAG"
done

# mark
[ -x "$ME.mark" ] && . "$ME.mark"

# balance
if [ -z $DEFAULTED ]; then
   act "$IP route add default scope global $WEIGHTS"
fi

act "$IP route flush cache"

# prune $FIFALOBATH_DYN if all dynamic, external interfaces are active and all went well
if echo "$ACTIVE_EXTIFs" | fgrep -q "$EXTIFs"; then
   for INTERFACE in $DYNAMIC_IFs; do
      eval "
         DYNIP=\$IP$INTERFACE;
         DYNNM=\$NM$INTERFACE;
         DYNGW=\$GW$INTERFACE;
         DYNDNS=\$DNSs$INTERFACE;
         DYNDNS1=\$DNS1$INTERFACE;
         DYNDNS2=\$DNS2$INTERFACE;
      "
      DYNINFO=`cat <<SETVAR
         $DYNINFO
         # pruned $INTERFACE
         export IP$INTERFACE=$DYNIP
         export NM$INTERFACE=$DYNNM
         export GW$INTERFACE=$DYNGW
         export DNSs$INTERFACE='$DYNDNS'
         export DNS1$INTERFACE=$DYNDNS1
         export DNS2$INTERFACE=$DYNDNS2
SETVAR`
   done

if [ -n "$DYNINFO" ]; then
      echo "$DYNINFO" > "$FIFALOBATH_DYN"
   fi
fi

# throttle
`$IPTABLES -t mangle -L | fgrep -q THROTTLE` || $IPTABLES -t mangle -N THROTTLE
for i in $EXTIFs; do # reset tc and iptables
   actlog "$TC qdisc del dev $i root"
   actlog "$IPTABLES -t mangle -D POSTROUTING -o $i -j THROTTLE"
done
actlog "$IPTABLES -t mangle -F THROTTLE"

# commit resolve.conf now
commit_resolv $RESOLV_TMP

# check if we want to throttle
[ -z $THROTTLE ] && exit 0

xint() ( IFS=*; echo "$*" | bc -l | sed 's/\..*//')

for EXTIF in $ACTIVE_EXTIFs; do
   eval "UPRATE=\$UP$EXTIF"
   if [ ! -n "$UPRATE" ]; then
      $ERR "ifcfg-$EXTIF must set variable UPRATE for throttling to work"
      continue
   fi

UPLOAD=`echo $UPRATE | sed 's/[^0-9]//g'`
   UNITS=`echo $UPRATE | sed 's/[0-9]//g'`
   THROTTLED=`xint $THROTTLE $UPLOAD`
   UPREALTIME=`xint $THROTTLED $ALLOT_REALTIME`
   UPP2P=`xint $THROTTLED $ALLOT_P2P`
   UPOTHER=$(( $THROTTLED - $UPREALTIME - $UPP2P ))
   $LOG "throttling $EXTIF from $UPLOAD to $THROTTLED$UNITS to avoid choking; realtime:p2p:other = $UPREALTIME:$UPP2P:$UPOTHER$UNITS"
   UPRATE="$THROTTLED$UNITS"

# add HTB root qdisc
   act "$TC qdisc add dev $EXTIF root handle 1: htb default 99"

# add main rate limit classes
   act "$TC class add dev $EXTIF parent 1: classid 1:1 htb rate $UPRATE"

# set classes; each class can lend and borrow bandwidth from each-other
   act "$TC class add dev $EXTIF parent 1:1 classid 1:10 htb rate $UPREALTIME$UNITS ceil $UPRATE quantum 1 prio 0" # realtime
   act "$TC class add dev $EXTIF parent 1:1 classid 1:20 htb rate    $UPOTHER$UNITS ceil $UPRATE quantum 1 prio 1" # responsive
   act "$TC class add dev $EXTIF parent 1:1 classid 1:30 htb rate    $UPOTHER$UNITS ceil $UPRATE quantum 1 prio 2" # interactive
   act "$TC class add dev $EXTIF parent 1:1 classid 1:40 htb rate    $UPOTHER$UNITS ceil $UPRATE quantum 1 prio 3" # low ports
   act "$TC class add dev $EXTIF parent 1:1 classid 1:50 htb rate    $UPOTHER$UNITS ceil $UPRATE quantum 1 prio 5" # data
   act "$TC class add dev $EXTIF parent 1:1 classid 1:99 htb rate      $UPP2P$UNITS ceil $UPRATE quantum 1 prio 6" # p2p

# attach qdisc to leaf classes
   act "$TC qdisc add dev $EXTIF parent 1:10 handle 10: pfifo limit 5" # no sfq; better to drop packets than delay them in realtime
   act "$TC qdisc add dev $EXTIF parent 1:20 handle 20: pfifo limit 30"
   act "$TC qdisc add dev $EXTIF parent 1:30 handle 30: pfifo limit 30"
   act "$TC qdisc add dev $EXTIF parent 1:40 handle 40: sfq perturb 10 limit 200" # long queue here and sqf to minimize drops
   act "$TC qdisc add dev $EXTIF parent 1:50 handle 50: sfq perturb 10 limit 200"
   act "$TC qdisc add dev $EXTIF parent 1:99 handle 99: sfq perturb 10 limit 300" # longest queue for p2p

act "$IPTABLES -t mangle -I POSTROUTING -o $EXTIF -j THROTTLE" # setup chain for filtering and marking
done

# define classifications
CLASS=1:10 # realtime
act "$IPTABLES -t mangle -A THROTTLE -m tos --tos 0x10 -j CLASSIFY --set-class $CLASS" # ensure min delay by TOS field
act "$IPTABLES -t mangle -A THROTTLE -p udp --dport 123 -j CLASSIFY --set-class $CLASS" # ntp
[ -x "$ME.realtime" ] && . "$ME.realtime"

CLASS=1:20 # responsive
act "$IPTABLES -t mangle -A THROTTLE -p icmp -j CLASSIFY --set-class $CLASS"
act "$IPTABLES -t mangle -A THROTTLE -p tcp --tcp-flags SYN,RST,ACK SYN,FIN -j CLASSIFY --set-class $CLASS"
act "$IPTABLES -t mangle -A THROTTLE -p tcp -m multiport --ports ssh -m limit --limit 10/second -j CLASSIFY --set-class $CLASS" # ssh, not scp
[ -x "$ME.responsive" ] && . "$ME.responsive"

CLASS=1:30 # interactive
act "$IPTABLES -t mangle -A THROTTLE -p tcp -m length --length 60 -j CLASSIFY --set-class $CLASS" # small packets (probably just ACKs)
[ -x "$ME.interactive" ] && . "$ME.interactive"

CLASS=1:40 # low port services
act "$IPTABLES -t mangle -A THROTTLE -p tcp --sport 0:1024 -j CLASSIFY --set-class $CLASS"
act "$IPTABLES -t mangle -A THROTTLE -p tcp --dport 0:1024 -j CLASSIFY --set-class $CLASS"

CLASS=1:50 # data
[ -x "$ME.data" ] && . "$ME.data"

/etc/sysconfig/network-scripts/fifalobath.private

#!/bin/bash
# $Id: fifalobath.private 26 2009-08-25 17:06:27Z dave $

# Slash delimited port/name/protocol(s).
# Ports can have a range, eg 137:139 or 5060,5062.
# Protocols are comma delimited; if omitted then tcp is assumed.
SERVICES=`cat <<SETVAR
   25/smtp
   53/dns/udp
   69/tftpd/tcp,udp
  137:139/samba/tcp,udp
  445/samba/tcp,udp
  993/pops
 2049/nfs
 3306/mysql
 5901/vnc
 6000:6002/X/tcp,udp
19150/gkrellm
SETVAR`

for DELIMITED in $SERVICES; do
   SERVICE=`echo $DELIMITED | cut -d'/' -f 2`
   PORT=`echo $DELIMITED | cut -d'/' -f 1`
   PROTOCOLS=`echo $DELIMITED | cut -d'/' -f 3 | sed 's/,/ /g'`
   if [ ! -n "$PROTOCOLS" ]; then PROTOCOLS=tcp; fi

for PROTOCOL in $PROTOCOLS; do
      if ! `echo "tcp udp icmp" | fgrep -q "$PROTOCOL"`; then
         $ERR "invalid protocol '$PROTOCOL' extracted from '$DELIMITED'"
         continue
      fi

$IPTABLES -A INPUT -i $INTIF -m state --state NEW,ESTABLISHED,RELATED -p $PROTOCOL -s $INTNET -d $INTIP --dport $PORT -j ACCEPT &&
         $LOG "allowing $PROTOCOL access from $INTNET to $SERVICE on $INTIP:$PORT"
   done
done