Racoon/ipsec/setkey questions/bugs

[Flechsenhaar, Jon J]

I have two x86 laptops one with ipsec-utils 0.7 and one with 0.7.1. It seems that the filtering rules in setkey don't work as the man page describes. Below I have listed some combinations of what has worked and what has failed as a bi-directional pair. A to b and then b to a. For example.

For each pair combination I did a fresh ping from node A; logged the result. I then restarted the racoon daemon and flushed setkey and did a ping from node B; logged the result. 3 examples of the actual setkey configuration are below in no significant order. I actually summarized the rules in the results table below as well. Fail means the IPSEC session never extablished, usualy failed proposal section for phase 2. Pass means that the IPSEC session established and the ping went through.

Test 1:
spdadd 2.2.2.2/32 0.0.0.0/0 icmp -P out ipsec esp/transport//require ah/transport//require;
spdadd 0.0.0.0/0 2.2.2.2/32 icmp -P out ipsec esp/transport//require ah/transport//require;

Test 5:
spdadd 2.2.2.2/32 1.1.1.1/32 icmp -P out ipsec esp/transport//require ah/transport//require;
spdadd 1.1.1.1/32 2.2.2.2/32 icmp -P out ipsec esp/transport//require ah/transport//require;

Test 11:
spdadd 2.2.0.0/28 0.0.0.0/0 any -P out ipsec esp/transport//require;
spdadd 0.0.0.0/0 2.2.0.0/28 any -P in ipsec esp/transport//require;

Complete Results table:
Test # PC Rule A PC Rule B Ping from PC A Ping from PC B
1 ip 0/0 out ICMP ip 0/0 out ICMP fail fail
0/0 ip in ICMP 0/0 ip in ICMP
2 ip 0/0 out ICMP ip 0/0 out any fail pass
0/0 ip in ICMP 0/0 ip in any
3 ip 0/0 out any ip 0/0 out any pass pass
0/0 ip in any 0/0 ip in any
4 ip/32 ip/32 out ICMP ip 0/0 out any pass pass
ip/32 ip/32 ip in ICMP 0/0 ip in any
5 ip/32 ip/32 out ICMP ip 0/0 out ICMP pass fail
ip/32 ip/32 ip in ICMP 0/0 ip in ICMP
6 ip/32 ip/32 out ICMP ip/32 ip/32 out ICMP pass pass
ip/32 ip/32 ip in ICMP ip/32 ip/32 ip in ICMP
7 ip/32 ip/24 out ICMP ip/32 ip/32 out ICMP pass pass
ip/32 ip/24 ip in ICMP ip/32 ip/32 ip in ICMP
8 ip/32 ip/24 out ICMP ip/32 ip/24 out ICMP pass pass
ip/32 ip/24 ip in ICMP ip/32 ip/24 ip in ICMP
9 ip/28 ip/28 out ICMP ip/28 ip/28 out ICMP pass pass
ip/28 ip/28 ip in ICMP ip/28 ip/28 ip in ICMP
10 ip/28 0/0 out ICMP ip/28 0/0 out ICMP fail fail
0/0 ip/28 ip in ICMP 0/0 ip/28 ip in ICMP
11 ip/28 0/0 out any ip/28 0/0 out any pass pass
0/0 ip/28 ip in any 0/0 ip/28 ip in any

Summary:
Basically I have noticed when I change the mask or the protocol that I'm filtering on; a IPSEC session can fail to establish or pass. This seems like a definite bug in setkey. I am surprised that I don't see any posts regarding this anywhere though. In my search most setkey configurations are very basic though and maybe theres isn't a demand for a more complex setkey policy configuration?

If there is a better list to post this on, please advise as well.

Any thoughts/ideas/help on this would be apprecitated. Thanks!

Jon Flechsenhaar
Boeing WNW Team
Network Services Layer
(714)-372-5172
B11-F2-2B60