Multipath Multirouter - Can it be done?

[Tony Hagans]

Currently I have a setup that involves connections from several different providers ranging from 6 mbit DSL's to 10 mbit fractional DS3 connections tied together to provide multipath high availability internet in our area for schools and businesses. We recently extended our coverage to another area which has several other ISP's. The box that is doing multiwan right now is the only gateway for the entire network and for some sites it is several hops back to that gateway. My goal is for us to purchase bandwidth in other locations, install a linux box and have multiple gateways available, but I think this is easier said than done. I will diagram this as best I can for you in current and future setup modes.

Currently:

ISP 1 ISP 2 ISP 3 etc...
| | | |

Multiwan box

|

Area 1

/ | \

Area 2 Area 3 Area 4

Goal:

Currently:

ISP 1 ISP 2 ISP 3 etc...
| | | |

Multiwan box

|

Area 1

/ | \

Area 2 Area 3 Area 4

| |
Linux Box Linux Box
| |
ISP x ISP y

My goal is for people in area 1-4 to all be able to use any of the connections to ISP 1-z and if one fails it passes the traffic through the network to an available route. Area 1-4 don't current have any linux hardware but it has been a long standing plan to put some in place in those areas as well so if that is required it could be done. These areas are all serviced by different small ISP's and would make acceptable alternate routes say if a backhual failed for some reason so that is the goal.

Thanks for any input.

[Alex Samad]

sound like you need a routing protocol, if you have access to manage all the
routers

Post by Tony Hagans
Currently I have a setup that involves connections from several different providers ranging from 6 mbit DSL's to 10 mbit fractional DS3 connections tied together to provide multipath high availability internet in our area for schools and businesses. We recently extended our coverage to another area which has several other ISP's. The box that is doing multiwan right now is the only gateway for the entire network and for some sites it is several hops back to that gateway. My goal is for us to purchase bandwidth in other locations, install a linux box and have multiple gateways available, but I think this is easier said than done. I will diagram this as best I can for you in current and future setup modes.
ISP 1 ISP 2 ISP 3 etc...
| | | |
Multiwan box
|
Area 1
/ | \
Area 2 Area 3 Area 4
ISP 1 ISP 2 ISP 3 etc...
| | | |
Multiwan box
|
Area 1
/ | \
Area 2 Area 3 Area 4
| |
Linux Box Linux Box
| |
ISP x ISP y
My goal is for people in area 1-4 to all be able to use any of the connections to ISP 1-z and if one fails it passes the traffic through the network to an available route. Area 1-4 don't current have any linux hardware but it has been a long standing plan to put some in place in those areas as well so if that is required it could be done. These areas are all serviced by different small ISP's and would make acceptable alternate routes say if a backhual failed for some reason so that is the goal.
Thanks for any input.
_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[Grant Taylor]

Post by Tony Hagans
Currently I have a setup that involves connections from several
different providers ranging from 6 mbit DSL's to 10 mbit fractional
DS3 connections tied together to provide multipath high availability
internet in our area for schools and businesses. We recently
extended our coverage to another area which has several other ISP's.
The box that is doing multiwan right now is the only gateway for the
entire network and for some sites it is several hops back to that
gateway. My goal is for us to purchase bandwidth in other locations,
install a linux box and have multiple gateways available, but I think
this is easier said than done. I will diagram this as best I can for
you in current and future setup modes.

...

Post by Tony Hagans
My goal is for people in area 1-4 to all be able to use any of the
connections to ISP 1-z and if one fails it passes the traffic through
the network to an available route. Area 1-4 don't current have any
linux hardware but it has been a long standing plan to put some in
place in those areas as well so if that is required it could be done.
These areas are all serviced by different small ISP's and would make
acceptable alternate routes say if a backhual failed for some reason
so that is the goal.

Like Alex said, I think you need at least an interior routing protocol.
I'd also recommend that you look in to getting your own address space
from your regional provider. Ideally you would get multiple if not all
of the ISPs to BGP advertise your address space out to the world. In
addition to the world knowing the better / best routes in to you
depending on where the traffic is originating from, your network would
know the better / best routes to take out to destinations with relation
to your ingress / egress points.

Additionally as long as you are doing NATing much closer to the
customer, or the customer is doing it them selves, you could have
multiple routes to / from the customer with out worrying about changing
NAT IPs.

Post by Tony Hagans
Thanks for any input.

You are welcome.




Grant. . . .

[Tony Hagans]

Thanks for the input Grant, we have kicked around the idea of BGP or a similar method for the routing but the issue is that NAT takes place at the box doing the multiwan. We only have a few external IP addresses and all of our connections are from different vendors so they are spread across several different IP ranges. My first though was to create a tunnels between the machines at each location and put the tunnel adapters in a routing table with a prio > the local routing table. This would allow data to:

client --> main gateway --> routing table mpath to any gateway on the network --> internet as available

I have a feeling this would cause some very very bad problems with return paths and I don't know what would happen if a tunnel were to fail or something of that nature. The system is basically many many wireless repeaters, access points, and clients spread across about 60 square miles. It crosses 3 LATAs and is within range of at least 10 different ISP's who all use different providers. The idea was if we could instead of bringing the traffic all back to where it goes now be able to go wherever was closest/least busy/etc and hop on a DSL/Cable/Wireless/whatever we can pick up for cheap bandwidth it would take some of the traffic from individuals doing whatever they do off the DS3 durring the day for businesses and schools to make use of.

Tony

[Grant Taylor]

Post by Tony Hagans
Thanks for the input Grant, we have kicked around the idea of BGP or
a similar method for the routing but the issue is that NAT takes
place at the box doing the multiwan. We only have a few external IP
addresses and all of our connections are from different vendors so
they are spread across several different IP ranges. My first though
was to create a tunnels between the machines at each location and put
the tunnel adapters in a routing table with a prio > the local

You are welcome.

Just because you are doing NATing at the core egress points instead of
the edge ingress points does not mean that you can not still use some
sort of routing protocol. Do NATing where you are but use some sort of
routing protocol that will keep all your routers in sync with each
other. If you use some sort of routing protocol that includes
connection state and possibly load, you could easily shift where traffic
is routed out of your network based on load and / or availability. Your
customers would talk to the upstream router that you direct them to use,
which will then send the traffic out the appropriate way.

Post by Tony Hagans
client --> main gateway --> routing table mpath to any gateway on the
network --> internet as available

Using some sort of routing protocol that includes load / cost of a
particular connection, you could easily just let routers pick the best
route to go out based on the routing protocol. Your routers that have a
single connection would just do standard NATing. Your routers that have
multiple connections would use the basic multipath routing described in
the LARTC How-To. Use your routing protocols to pick which rout to go out.

I would also make sure to educate each router be aware of the subnets
that the provider has. There is no reason to ever go out provider B to
get to a server on provider A's network unless the link with provider A
is down.

It sounds like you don't have a bunch of IPs, say one for each client,
but could you get extra IPs from each provider, say one ip for each
router in your network from each provider. So if you had 5 providers,
get 5 IPs from each, for a total of 25 IPs. If you could do this, you
could have each router be able to connect directly to the links from
each provider and load balance out with multipath routing if you so
chose. Granted this would be dependent on your network structure and
its capabilities. One advantage of this is that you would have more IPs
to hid services behind. If one IP became black listed for some reason,
you could shift traffic off of it and use another one with out much
hassle at all.

Post by Tony Hagans
I have a feeling this would cause some very very bad problems with
return paths and I don't know what would happen if a tunnel were to
fail or something of that nature. The system is basically many many
wireless repeaters, access points, and clients spread across about 60
square miles. It crosses 3 LATAs and is within range of at least 10
different ISP's who all use different providers. The idea was if we
could instead of bringing the traffic all back to where it goes now
be able to go wherever was closest/least busy/etc and hop on a
DSL/Cable/Wireless/whatever we can pick up for cheap bandwidth it
would take some of the traffic from individuals doing whatever they
do off the DS3 durring the day for businesses and schools to make use
of.

I think this is possible to do. I think you will have better luck if
you have traffic predominantly use the closest router and not try to
load balance a given client across multiple providers but rather load
balance by picking which provider a given client uses. You could even
use something as devious as VRRP and virtual routers to on the fly
change which router was a given IP with out requiring clients to
re-configure any thing. Granted you run a chance of in progress
sessions being messed up, but that is a problem you will have with just
about any multipath setup where you are NATing at the core.

If you will provide more information on what your actual network
topology is including if things are a layer 2 or layer 3 link and what
subnets are where, I'll be glad to help.



Grant. . . .