One machine, two net feeds, outbound route selection

[Ben Scott]

Hello, world!

Sorry to subscribe to the list only to immediately ask a question,
but this one's got me scratching my head and I can't find the answer
in the archives, the HOWTO, or on the web. Maybe I'm just asking the
wrong question.

Anyway, our Internet gateway is a Linux box. We've got two Internet
feeds, a fast one from a crummy provider, and a slower one from a good
provider. The goal is to control which feed gets used on a
per-service basis. For the most part, I've been pretty successful at
this, and it's worked well. Score one for the good guys.

However, I'm having trouble trying to get Sendmail -- which is
running on the gateway box itself -- to use the good provider when it
tries to deliver mail. (If we use the crummy provider, too many
others reject us as a likely spammer.)

I found plenty of docs that tell me what to do if Sendmail is
running on a different box -- just select packets using iptables as
they come in, fwmark them, and then use a routing rule to put them
into the appropriate routing table. This is conceptually easy,
because the packets would already have well-defined characteristics.

The problem is that Sendmail is running on the gateway itself. I
don't know how to tell the system what to select. The packets won't
have IP address or interface info yet, because they haven't gone
through the kernel router yet. Right? And once they've gone through
the kernel router, it's too late to try and pick the route they'll
use. Right? Catch-22?

I can't bind Sendmail's outgoing SMTP client mailer to a specific
interface, because it has to be able to forward mail on to inside
systems, too.

I can think of all sorts of possible combinations of iptables
options I might try (table, chain, interface, TCP port, etc.), but
there are literally hundreds of permutations. Trial-and-error doesn't
seem like a good way to do this. Especially since it's a production
box. I tried a few ideas and got nowhere useful.

Does someone here already know the answer?

Environment:
- CentOS 5
- kernel 2.6.18-8.1.10.el5
- iptables 1.3.5-1.2.1
- iproute 2.6.18-4.el5
- Both feeds connect to the gateway with plain old IP-over-Ethernet
- Static IP addresses for both feeds
- LAN is NAT'ed and using a 10/8 private subnet
- Successfully using multiple routing tables, iptables, and/or
interface-bindings to select route/feed for most services

advTHANKSance!

-- Ben

[Peter Rabbitson]

<snip>
I can't bind Sendmail's outgoing SMTP client mailer to a specific
interface, because it has to be able to forward mail on to inside
systems, too.

Of course you can. Remember that the kernel knows about both networks -
the internal and external ones. Once you bind to the external IP, a
packet destined to the inside would still be routed correctly, without
regard to the source IP.

[Ben Scott]

Post by Ben Scott
I can't bind Sendmail's outgoing SMTP client mailer to a specific
interface, because it has to be able to forward mail on to inside
systems, too.

Of course you can. Remember that the kernel knows about both networks ...

Hmmm. You're right, of course. And even better, it worked when I
tried it. ;) Thanks!

Now, for the sake of knowledge, let us say that a piece of needed
software didn't have an option to bind to a specific interface. Would
it be possible to control the outgoing route/interface anyway, by
using iptables or some other mechanism external to the software? For
example, what if Sendmail didn't have a ClientPortOptions directive
(heh, I know, use Postfix, but work with me here)?

And remember: This is for something running on the gateway, not
forwarding packets received from another machine. :)

(No, I don't presently have such a piece of software, but I'd like to know.)

Thanks again, either way!

-- Ben

[Peter Rabbitson]

Post by Ben Scott
Now, for the sake of knowledge, let us say that a piece of needed
software didn't have an option to bind to a specific interface. Would
it be possible to control the outgoing route/interface anyway, by
using iptables or some other mechanism external to the software? For
example, what if Sendmail didn't have a ClientPortOptions directive
(heh, I know, use Postfix, but work with me here)?

Unfortunately not easy without doing local NAT (from the local interface
to another local interface). The problem lies in how the kernel sends
packets without a specified source. I wrote an explanation some time
ago: http://mailman.ds9a.nl/pipermail/lartc/2007q2/020941.html

[Ben Scott]

Post by Peter Rabbitson
Unfortunately not easy without doing local NAT (from the local interface
to another local interface).

I thought that might be the case. I even started to write a rule
about how the NAT might work... but then I ran into brain pain trying
to figure out how, because I didn't know when the packets get what
address/interface info assigned to them, and I didn't know how SNAT
would interact with the routing tables. Normally, I do SNAT in the
POSTROUTING chain, but by then the routing rules have already run,
right? So the packet would still be bound for the wrong interface,
even if the source address is translated. No?

In other words, let's say $DEF_ADDR is the IP address of the
interface that is going to be picked by the default routing table, but
I really want the packets to go out the $ALT_ADDR interface. So I try
this:

iptables -t nat -A POSTROUTING -s $DEF_ADDR -p tcp --dport smtp -j
SNAT --to $ALT_ADDR

But the whole point of changing the source address/interface is to
influence which routing rules match, and those have already been
applied by the time the packet transverses the POSTROUTING chain,
right? In any event, that didn't work.

So then I thought, well, maybe I can do SNAT in the PREROUTING chain
for this? But in that case, the kernel won't have assigned it an
address yet, right? So there's nothing to SNAT. And I can't do "-s
0/0" because that actually means "match all packets", right?

So then I thought, well, maybe I can mark the packet in the OUTPUT
chain of the mangle table, and match that in the routing rules, and
*also* match that in the POSTROUTING chain:

iptables -t mangle -A OUTPUT -s $DEF_ADDR -p tcp --dport smtp -j MARK
--set-mark 42
ip rule add fwmark 42 table 42
iptables -t nat -A POSTROUTING -m mark --mark 42 -j SNAT --to-source $ALT_ADDR

I think I tried that and it didn't work either. It was getting late
and my maintenance window was closing and my brain hurt.

If this is just one of those "you can't do that" situations, I'm
willing to accept that answer. But if there is a way, I'd like to
know what it is. :)

-- Ben

[Alex Samad]

Post by Ben Scott

Post by Peter Rabbitson
Unfortunately not easy without doing local NAT (from the local interface
to another local interface).

Can you use marking, mark the packet in the mangle table, us iptables to select
the which packets and then use ip rules fw mark -> routing table (sorry about
the syntax)

Post by Ben Scott
I thought that might be the case. I even started to write a rule
about how the NAT might work... but then I ran into brain pain trying
to figure out how, because I didn't know when the packets get what
address/interface info assigned to them, and I didn't know how SNAT
would interact with the routing tables. Normally, I do SNAT in the
POSTROUTING chain, but by then the routing rules have already run,
right? So the packet would still be bound for the wrong interface,
even if the source address is translated. No?
In other words, let's say $DEF_ADDR is the IP address of the
interface that is going to be picked by the default routing table, but
I really want the packets to go out the $ALT_ADDR interface. So I try
iptables -t nat -A POSTROUTING -s $DEF_ADDR -p tcp --dport smtp -j
SNAT --to $ALT_ADDR
But the whole point of changing the source address/interface is to
influence which routing rules match, and those have already been
applied by the time the packet transverses the POSTROUTING chain,
right? In any event, that didn't work.
So then I thought, well, maybe I can do SNAT in the PREROUTING chain
for this? But in that case, the kernel won't have assigned it an
address yet, right? So there's nothing to SNAT. And I can't do "-s
0/0" because that actually means "match all packets", right?
So then I thought, well, maybe I can mark the packet in the OUTPUT
chain of the mangle table, and match that in the routing rules, and
iptables -t mangle -A OUTPUT -s $DEF_ADDR -p tcp --dport smtp -j MARK
--set-mark 42
ip rule add fwmark 42 table 42
iptables -t nat -A POSTROUTING -m mark --mark 42 -j SNAT --to-source $ALT_ADDR
I think I tried that and it didn't work either. It was getting late
and my maintenance window was closing and my brain hurt.
If this is just one of those "you can't do that" situations, I'm
willing to accept that answer. But if there is a way, I'd like to
know what it is. :)
-- Ben
_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[Peter Rabbitson]

Post by Ben Scott

Post by Peter Rabbitson
Unfortunately not easy without doing local NAT (from the local interface
to another local interface).

I thought that might be the case. I even started to write a rule
about how the NAT might work... but then I ran into brain pain trying
to figure out how, because I didn't know when the packets get what
address/interface info assigned to them, and I didn't know how SNAT
would interact with the routing tables. Normally, I do SNAT in the
POSTROUTING chain, but by then the routing rules have already run,
right? So the packet would still be bound for the wrong interface,
even if the source address is translated. No?

I was not thorough enough. The NAT is necessary in order to make the
packet come back through the link/interface you want (because as I noted
previously you do not have control over the choice of a source address).
Once this is out of the way the only problem is how to make an already
routed packet to leave via a different interface. One way to do this is
the ROUTE target:
http://netfilter.org/documentation/HOWTO//netfilter-extensions-HOWTO-4.html#ss4.5
There might also be other ways to do this, but I never investigated, as
this is a mostly theoretical exercise.

Peter