gateway failover with linux

[Abhijit Menon-Sen]

Hi.

I'm wondering if there's a good way to configure a Linux firewall box to
failover to a single backup server, while preserving connection state.

This question has been asked before, but the latest reference I can find
is from 2004, at which time Linux had no equivalent of OpenBSD's pfsync,
though Harald was said to be working on one.

Did anything come of those efforts? Or is there now another alternative?

Any examples or advice would be appreciated.

Thank you.

-- ams

[Abhijit Menon-Sen]

Post by Abhijit Menon-Sen
I'm wondering if there's a good way to configure a Linux firewall box
to failover to a single backup server, while preserving connection
state.

Looks like this is it:

http://people.netfilter.org/pablo/conntrack-tools/

-- ams

[Grant Taylor]

Post by Abhijit Menon-Sen
I'm wondering if there's a good way to configure a Linux firewall box
to failover to a single backup server, while preserving connection
state.

I'm a bit confused, are you wanting a single Linux firewall / router to
have redundant internet connections, or to route traffic to redundant
systems behind it and intelligently handle the failure of one or more of
said redundant systems? I'm also not sure how conntrackd (comparable to
OpenBSD's pfsync) is coming in to play here. Or is there more than one
Linux firewall / router that you are wanting to synchronize? Or are you
wanting the connection tracking between the multiple systems behind the
Linux firewall / router? I think that all of these are possible to
various degrees, though each uses a different method to achieve it.

Post by Abhijit Menon-Sen
This question has been asked before, but the latest reference I can
find is from 2004, at which time Linux had no equivalent of OpenBSD's
pfsync, though Harald was said to be working on one.

*nod* Conntrackd is the tool that you want to use to synchronize
connection tracking connection meta data between two systems, or the
closest thing that Linux presently has (that I'm aware of).

Post by Abhijit Menon-Sen
Did anything come of those efforts? Or is there now another
alternative?

Yes, conntrackd.

Post by Abhijit Menon-Sen
Any examples or advice would be appreciated.

Will you please clarify what you are really wanting to do per above and
I'll be more than happy to try to point you in the right direction.




Grant. . . .

[Abhijit Menon-Sen]

Hi Grant.

Post by Grant Taylor
I'm a bit confused, are you wanting a single Linux firewall /
router to have redundant internet connections, or to route
traffic to redundant systems behind it and intelligently
handle the failure of one or more of said redundant systems?

Neither.

I just want a hot standby for a single Linux firewall, such that clients
behind it are not affected by a hardware failure on the firewall. If my
configuration would allow me to someday promote the backup and run both
firewall machines in a load-balancing configuration, so much the better.

The following example looks very much like what I want:

http://people.netfilter.org/pablo/conntrack-tools/testcase.html

(Can anyone comment on whether I should stick with keepalived as
described above, or try out ucarp?)

Post by Grant Taylor
Will you please clarify what you are really wanting to do per
above and I'll be more than happy to try to point you in the
right direction.

Thanks, I'd appreciate any advice you can give me.

-- ams

[Mohan Sundaram]

Post by Abhijit Menon-Sen
Hi Grant.

Post by Grant Taylor
I'm a bit confused, are you wanting a single Linux firewall /
router to have redundant internet connections, or to route
traffic to redundant systems behind it and intelligently
handle the failure of one or more of said redundant systems?

Neither.
I just want a hot standby for a single Linux firewall, such that clients
behind it are not affected by a hardware failure on the firewall. If my
configuration would allow me to someday promote the backup and run both
firewall machines in a load-balancing configuration, so much the better.
http://people.netfilter.org/pablo/conntrack-tools/testcase.html
(Can anyone comment on whether I should stick with keepalived as
described above, or try out ucarp?)

Post by Grant Taylor
Will you please clarify what you are really wanting to do per
above and I'll be more than happy to try to point you in the
right direction.

Thanks, I'd appreciate any advice you can give me.
-- ams

In case your firewall is a proxy for some service, those connections
will fail though - unless you can use a virtual interface with the same
IP as the source for such connections.

I guess you'll use vrrp in conjunction for failover. It would make sense
to use vrrpd with status tracking of WAN gateway but AFAIK no such
feature exists as yet.

Mohan

[Grant Taylor]

It would make sense to use vrrpd with status tracking of WAN gateway
but AFAIK no such feature exists as yet.

If I understand what you are saying, you are referring to an external
T-1 router converting from the WAN circuit to the ethernet going in to
the two routers right?

If this is the case, no matter what is done, the WAN router is a single
point of failure and thus can not be avoided.

No matter what, the Linux boxen can be configured such that they will
try to reach this single point of failure and in the event that they can
not do so, they (both) will ultimately return an ICMP "no route to host"
error message back to the client. However this is out side the scope of
what VRRP is meant to do.



Grant. . . .

[Grant Taylor]

I think my point was misunderstood. Let us say each of these Linux
boxes are connected to a WAN link each. If the WAN gateway/link of a
box goes down, vrrp must flag itself down or as secondary. Some
similarity to our earlier discussions on redundant gateways. This
feature of object tracking is available in CISCO (their patent) but
only tracks the interface status and not the gateway reachability.
I'd love to have a feature where gateway reachability is tracked.

Each firewall / router / gateway having its own independent wan /
internet connection makes things a bit different. First of all, each
will have its own public IP address for the associated WAN link and as
such probably have it's own NAT configuration.

I wonder, what type of WAN connection are these? Could both be hooked
up to both gateways? In other words are they ethernet or something that
gets bridged to ethernet or are they some sort of legacy WAN link, i.e.
T1, Frame Relay, ATM? If you could connect the WAN links to both
systems, you can have even more functionality and you would be back to
what I was originally thinking except for the fact that instead of one
WAN connection, you have two to make each router aware of.

At the very least, I think you will need to make each router aware of
the other one. This way, if a router's (primary) WAN link is not usable
it can route the traffic over to the other router and have it route the
traffic out to the world. Thus each router would have a primary default
gateway of the router at the other end of its WAN link *AND* a secondary
default gateway of the other router that it is connected to.

I also think that you are very close to needing to use some sort of
monitoring utility / daemon to check the status of the WAN link *AND* to
make the other router aware of the status. This may be easily done with
a small daemon to monitor the link and update the local routing table in
conjunction with a routing protocol between the two routers to keep each
aware of the others routing table.

If you have any more information on Cisco's technology I'd be interested
in doing some reading about it if you would care to toss it my way.



Grant. . . .

[Grant Taylor]

Post by Abhijit Menon-Sen
I just want a hot standby for a single Linux firewall, such that
clients behind it are not affected by a hardware failure on the
firewall. If my configuration would allow me to someday promote the
backup and run both firewall machines in a load-balancing
configuration, so much the better.

Ok, to pull this off I would use a pair of Linux boxen with vrrpd and
conntrackd with almost identical configurations on each system (save for
host name and management IPs).

For the sake of discussion I'll presume that you are using a cable modem
or adsl connection to the internet with a static IP address. PPPoE /
DHCP should be possible, but it will get much nastier. You will really
want a minimum of two static IP addresses, three if possible, on each
side of the firewall. One IP address will be for the routing of traffic
and the other two will be for management. The one routing IP address
will be the virtual IP that is passed back and forth between the systems
and also the IP address that it looks like all your traffic passes
through. Use VRRP to make sure that one of the systems will always have
the routing IP up and functional.

Post by Abhijit Menon-Sen
http://people.netfilter.org/pablo/conntrack-tools/testcase.html

*nod* This is very much what you are wanting to do.

However I'm not sure that the direct connection between the two systems
is absolutely required. However having not done this my self I can not
say for sure. The last time I looked at this I was going to be trying
to make redundant routers for a 1 - 3 Mbps connection and thus was
planing on using VLANs to do Router-on-a-stick across two bonded 100
BaseT connections. So, if I needed to I could create a new VLAN for the
router to router communications.

Post by Abhijit Menon-Sen
(Can anyone comment on whether I should stick with keepalived as
described above, or try out ucarp?)

I would use vrrpd over keepalived. As far as ucarp, I'm not familiar
with it so I can't comment. I can say that keepalived is much more
complex and can do a lot more than what you need to do. Now if you were
running a load balancing cluster of servers behind it, keepalived is the
proper answer. However for just making sure that a router is available,
vrrpd is much closer to what you need with out overkilling. In fact, I
believe vrrpd only needs command line options and no config file where
as, last I looked, keepalived has a very complex config file for all
that it could do.

With regards to "... run both firewall machines in a load-balancing
configuration ..." I think that conntrackd will allow you to do this.
At the very least, you can have inbound traffic use one firewall as its
primary and outbound use the other firewall as its primary where each is
a failover for the other. If you want inbound and / or outbound to use
both routers at the same time, it gets very trick to have the layer 2
ethernet network know which router to use to send the traffic to.

A quick run down of how VRRP (and Cisco's proprietary HSRP) works for
those that may not know or want a refresher. VRRP uses a management IP
and a virtual routing (that may or may not be one of the management
IPs). The virtual routing IP uses a special MAC address
(00-00-5E-00-01-<virtual router ID>). This special MAC address allows
all client workstations using the virtual routing IP to be able to cache
the MAC address and not have to re-arp for the system that currently has
the IP. In the event that the backup router detects that the primary
router is down, the backup router will claim to be the virtual IP on the
virtual router MAC address, usually GARPing so that switches now see the
virtual router MAC address on the new switch port. Thus when clients
try to send traffic out through the virtual router, the layer 2 traffic
will be to the virtual MAC address which the switches now know to be
elsewhere on the functioning router. The net result is a very brief
down time while the backup router is detecting that the primary is not
functioning and switches over.

(Note: This is a very far fetch problem, but is possible.) The only
real problem with this scenario is if for some reason both routers are
up and functioning, but for some reason they can not see each other,
thus they both think the other node is down. In this case, you may have
a battle for who is up. If you want to know what to do in this
situation read about SONITH (Shoot Other Node In The Head) to make sure
that there is only one active node at a time.

You can configure VRRP to automatically switch back from the backup to
the primary when the primary comes back on line, or let the backup
remain the primary until it fails and then the real primary that is
acting as the backup will resume its role as the primary.

Post by Abhijit Menon-Sen
Thanks, I'd appreciate any advice you can give me.

Having a functioning understanding of what is going on will help make
this a success. What you are wanting to do is completely possible and
should be able to be achieved with out a lot of problems, but it is out
side of the scope of any point and click GUI that I have seen for Linux.

Good luck and let me know if there is any thing else that I can do to help.



Grant. . . .

[Abhijit Menon-Sen]

I just want a hot standby for a single Linux firewall [...]

I would use a pair of Linux boxen with vrrpd and conntrackd

OK, great. I didn't know about vrrpd. I'll check it out.

As far as ucarp, I'm not familiar with it so I can't comment.

If I have the time, I'll try out ucarp and post a summary of my
experiences for the archives.

If you want to know what to do in this situation read about SONITH
(Shoot Other Node In The Head) to make sure that there is only one
active node at a time.

("STONITH", for those asking Google.)

I have one other question. How does conntrackd interact with traffic
shaping? My firewall also uses HTB to impose various bandwidth limits
on clients. From what I've read so far, I have the impression that the
failover may lose some packets that are being delayed in a queue, but
existing connections should recover and be esentially unaffected.

Can anyone confirm that?

-- ams

[Abhijit Menon-Sen]

Post by Abhijit Menon-Sen
If I have the time, I'll try out ucarp and post a summary of my
experiences for the archives.

Not much to report. I set up ucarp as directed in the README, and it
just worked. It was simple and did what I wanted (which was to allow
two machines to share a virtual IP with heartbeat/failover).

-- ams

[Grant Taylor]

In VRRP based gateway pairs, this is normally done by partitioning
the LAN to use both gateways by different subnets. i.e G1 is primary
for subnet1 with G2 as secondary; G2 is primary for subnet2 with G1
as secondary. This is done by defining multiple vrrp groups. AFAIK,
no dynamic balancing methods/ features exist.

Agreed, this would be the easiest to implement active / active with
traffic flowing out through both routers. I was trying to state that
outbound for the network would flow through one router while inbound
would flow through the other router. This could also be extended to
mirror the multiple subnets like you are talking about too, though I
think load sharing reasoning would be lost at that point as you have
more traffic on both routers. At this point in time it would come down
to statistics of network load to see how you wanted to do it.

I'm sure there are ways that you can have all traffic spread across both
routers in either direction. Though I'm not quite sure how to go about
it. I'm going to say that VRRP / Load Balancing / Bridging / Layer 2
Filtering / and other services would be in effect to divide the traffic
across both active routers. No matter how you slice it, this is beyond
the scope of the OPs question. I was merely stating that things are
possible.




Grant. . . .