How to fight with encrypted p2p

Post by sAwAr
Hi
I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted?
Modern p2p clients have this ability moreover some of them have this enabled by default.
Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

David Bierce

2007-11-12 07:02:41 UTC

Some clients P2P clients are nice about there encryption and negotiate
encryption ahead of time using plain communication. I.E. Limewire,
Azureus. However, some just start TLS and that is all you can see.

Looking at ipp2ps signatures, I don't see anything that leads me to
believe they track that kind of info.

David Bierce

Hi
I believe that whole question is in topic. Is there any way to
recognize ( and then shape ) p2p traffic which is encrypted?
Modern p2p clients have this ability moreover some of them have
this enabled by default. Now I'm using ipp2p for iptables but as I
know this doesn't recognize encrypted traffic.
Thanks in advance.
Pozdrawiam
Szymon Turkiewicz

Have not tried this. An idea. P2P initiations are not encrypted
AFAIK. Thus connections can be marked and related traffic shaped. If
initiation is also encrypted, then I think we have a serious problem.
Mohan
_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

sawar

2007-11-12 11:17:13 UTC

Rtorrent which I use sometimes have ability to completely disable plain text
communication :

man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable
unencrypted handshakes), require_RC4 (also disable plaintext
transmission after the initial encrypted handshake), enable_retry (if the
initial outgoing connection fails, retry with encryption turned on if it was
off or off if it was on), prefer_plain text (choose plaintext when peer
offers a choice between plaintext transmission and RC4 encryption, otherwise
RC4 will be used).

and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a
matter of time and we will lose this "fight" very soon.

Post by David Bierce
Some clients P2P clients are nice about there encryption and negotiate
encryption ahead of time using plain communication. I.E. Limewire,
Azureus. However, some just start TLS and that is all you can see.
Looking at ipp2ps signatures, I don't see anything that leads me to
believe they track that kind of info.
David Bierce

_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Marco Aurelio

2007-11-13 16:32:03 UTC

As you might have seen, these are words from ipp2p author:

"""

I have seen some pieces of code from ipoque which can detect encypted bittorrent
and edonkey traffic. Unforunately, this code will not work with
iptables, because it needs
more information about the flow history and the history of an ip address.

Right now, I do not have the time and the money to develop a filter
like this, but
if you are interested in a developement in this direction, please contact me.

"""

I *think* that we need something like a "bittorrent helper" in the
kernel to keep this extra information about the flow history and then
an iptables plugin to match. What do you think? Maybe we could contact
him to know what kind of information is it?

Post by sawar
Rtorrent which I use sometimes have ability to completely disable plain text
man rtorrent
allow_incoming (allow incoming encrypted connections),
try_outgoing (use encryption for outgoing connections), require (disable
unencrypted handshakes), require_RC4 (also disable plaintext
transmission after the initial encrypted handshake), enable_retry (if the
initial outgoing connection fails, retry with encryption turned on if it was
off or off if it was on), prefer_plain text (choose plaintext when peer
offers a choice between plaintext transmission and RC4 encryption, otherwise
RC4 will be used).
and many other clients have similar abilities.
I'm afraid that full encrypted and enabled by default communication is only a
matter of time and we will lose this "fight" very soon.

_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

--
Marco Casaroli
SapucaiNet Telecom
+55 35 34712377 ext. 5

Klaus

2007-11-14 09:42:23 UTC

About ipp2p,

Right now, the battle against p2p is lost with l7 detection from ipp2p,
l7 filter and others.

Why ?? It is a known fact that pattern matching does not work with full
encrypted P2P handshakes based on DHT key exchange algorithms with byte
padding. You have absolutely no byte pattern and no fixed packet lengths
in the stream. So something like a flow history will fail or might have
a very high false +ve rate.

The thing is that there are proprietary solutions which can detect fully
encrypted p2p streams based on a heuristic approach. (AFAIK ipoque is
selling a proprietary library for this which is integrated in some
firewall vendors). I have not seen any open source development into this
direction.

Klaus, (former) maintainer of ipp2p

Post by Marco Aurelio
"""
I have seen some pieces of code from ipoque which can detect encypted bittorrent
and edonkey traffic. Unforunately, this code will not work with
iptables, because it needs
more information about the flow history and the history of an ip address.
Right now, I do not have the time and the money to develop a filter
like this, but
if you are interested in a developement in this direction, please contact me.
"""
I *think* that we need something like a "bittorrent helper" in the
kernel to keep this extra information about the flow history and then
an iptables plugin to match. What do you think? Maybe we could contact
him to know what kind of information is it?

_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Sébastien CRAMATTE

2007-11-14 14:32:23 UTC

Klauss,

Could you
Might be you can sponsor the development ...

Regards

Sébastien

Post by Klaus
About ipp2p,
Right now, the battle against p2p is lost with l7 detection from ipp2p,
l7 filter and others.
Why ?? It is a known fact that pattern matching does not work with full
encrypted P2P handshakes based on DHT key exchange algorithms with byte
padding. You have absolutely no byte pattern and no fixed packet lengths
in the stream. So something like a flow history will fail or might have
a very high false +ve rate.
The thing is that there are proprietary solutions which can detect fully
encrypted p2p streams based on a heuristic approach. (AFAIK ipoque is
selling a proprietary library for this which is integrated in some
firewall vendors). I have not seen any open source development into this
direction.
Klaus, (former) maintainer of ipp2p

_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Sébastien CRAMATTE

2007-11-14 14:44:56 UTC

Sorry ... I'm little bite tired ...
I mean that we might sponsor Klauss and L7 team to develop this ...

Regards

Post by SÃ©bastien CRAMATTE
Klauss,
Could you
Might be you can sponsor the development ...
Regards
Sébastien

_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Marcin Stanczyk

2007-11-13 11:58:53 UTC

As far as I am concerned there is no way to recognize encrypted p2p.
I`ve tested ipp2p, and layer7, and also Mikrotik`s 3.0 p2p mark systems
[ which in fact uses modified layer7 ], and none of them cant recognize
this traffic simply because it is enctypted.

- --

[ ... and the Slackware for All... ]

gpg_key: http://szuwar.mayhem.net.pl/keys/szuwar_mayhem-public.gpg

Grant Taylor

2007-11-13 15:09:32 UTC

Post by sAwAr
Is there any way to recognize ( and then shape ) p2p traffic which is
encrypted? Modern p2p clients have this ability moreover some of
them have this enabled by default. Now I'm using ipp2p for iptables
but as I know this doesn't recognize encrypted traffic.

Does this mean that we are down to handling traffic based on the
sustained stream(s)? I.e. how long the streams have been active, how
many packets per second, how many streams a given end point has, speed
of traffic, average size of packets?

Encrypted or not, I believe all traffic can be somewhat recognized by
its usage pattern(s). However there may be more false positives. We
may end up recognizing what we know as good and putting the rest at a
lower class of service.

Thought's / comments / objections / flame wars?

Grant. . . .

Carl-Daniel Hailfinger

2007-11-13 15:37:17 UTC

Post by Grant Taylor

Does this mean that we are down to handling traffic based on the
sustained stream(s)? I.e. how long the streams have been active, how
many packets per second, how many streams a given end point has, speed
of traffic, average size of packets?
Encrypted or not, I believe all traffic can be somewhat recognized by
its usage pattern(s). However there may be more false positives. We
may end up recognizing what we know as good and putting the rest at a
lower class of service.

Well, you can surely try. But then again, I have been doing research
(publication pending) in traffic-pattern-based detection of VoIP flows
and peer-to-peer connections. While it usually is easy to find a pattern
matching your particular traffic class very well, part of this research
has been dedicated to automatically circumvent these systems. Why that?
Simple. Application evolve to circumvent detection. If you can simulate
that evolution in the lab, you can find out where your detection
algorithms will fail. Of course, that enumeration of possible failure
modes is non-exhaustive.

Bottom line: This is an arms race. Unless you do lots of research and
testing, detection will always be trying to catch up. If detection
manages to catch up, circumvention will advance, but you may have a
small time window where you can enjoy the "win". However, winning
becomes more and more expensive. Clients can expend considerable amount
of CPU power to avoid detection. You don't have that luxury in filter
systems unless you have one filter per client.

Regards,
Carl-Daniel

Grant Taylor

2007-11-13 15:53:26 UTC

Post by Carl-Daniel Hailfinger
Well, you can surely try. But then again, I have been doing research
(publication pending) in traffic-pattern-based detection of VoIP
flows and peer-to-peer connections. While it usually is easy to find
a pattern matching your particular traffic class very well, part of
this research has been dedicated to automatically circumvent these
systems. Why that? Simple. Application evolve to circumvent
detection. If you can simulate that evolution in the lab, you can
find out where your detection algorithms will fail. Of course, that
enumeration of possible failure modes is non-exhaustive.
Bottom line: This is an arms race. Unless you do lots of research and
testing, detection will always be trying to catch up. If detection
manages to catch up, circumvention will advance, but you may have a
small time window where you can enjoy the "win". However, winning
becomes more and more expensive. Clients can expend considerable
amount of CPU power to avoid detection. You don't have that luxury in
filter systems unless you have one filter per client.

All very good points with regard to pattern based detecting P2P (and the
likes) traffic. What do you think about recognizing the traffic you do
want and treating all else as a second or third class citizen. Or is
this just a form of net neutrality? Or really is this entire discussion
such. Further does the net neutrality issue apply to companies (read:
non ISPs) wanting to filter their own internal traffic.

Additionally as an aside will you please provide more information on
your pending publication? I'd likely be curious to read (what ever)
when ever it is published. Thanks in advance.

Grant. . . .

Konstantin Astafjev

2007-11-13 16:08:03 UTC

Hello ,

Post by Grant Taylor
Encrypted or not, I believe all traffic can be somewhat recognized by
its usage pattern(s). However there may be more false positives. We
may end up recognizing what we know as good and putting the rest at a
lower class of service.
Thought's / comments / objections / flame wars?

It's true that encrypted/unencrypted ratio of p2p traffic increased
about 10 times for past 2 years. There were a lot of predictions
before about such situation will happen. I guess that we have revise
shaping policing by some usage patterns as Grant said before. I guess
I'll start to reread about all netfilter features and it helps me to
figure out how to solve this problem.

--
Konstantin

Andrew Beverley

2007-12-02 11:42:36 UTC

Post by sAwAr
I believe that whole question is in topic.
Is there any way to recognize ( and then shape ) p2p traffic which is encrypted?
Modern p2p clients have this ability moreover some of them have this enabled by default.
Now I'm using ipp2p for iptables but as I know this doesn't recognize encrypted traffic.

One way to do this is to look for the style of traffic. For example, I
look for lots of connections from one PC to port numbers above 1024.
This will also incorrectly recognise some other traffic, but on the
whole it works well for me.

The following are some examples using connlimit (now included in vanilla
kernel) and ipset (see http://ipset.netfilter.org/)

# first look for style of traffic and log that client to an ipset
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p dst

# then shape traffic above port 1024 for those detected clients
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
-m set --set p2p dst -j MARK --set-mark 60

Regards,

Andy Beverley

Gustin Johnson

2007-12-03 10:49:12 UTC

I believe "fighting" is the wrong approach. Badly shaping the wrong
traffic is just as bad, if not worse IMO. An ISP in my neck of the
woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a
result of their P2P fight. Needless to say we no longer use them, and
we encourage clients, friends, and colleagues not to as well. I don't
use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with
these services is not useful. Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

For me, if P2P does not belong at all, for instance on a corporate
network, then a default deny on the outbound works much better. We then
only allow specific connections on a case by case basis. For instances
where I am not able to block p2p, I define specific rules for high and
low priority, and leave everything else in the default. If the end user
wants to use the bulk of his or her bandwidth for P2P, so be it. Of
course in this case bandwidth accounting is far more useful.

I would also encourage your users to use software that is or can be well
behaved. Software that allows you set a proper TOS for instance. If
possible work with the end users.

I have personally found that the best solutions are not tech solutions.
Having a well defined Acceptable Use Policy, plus a constructive
dialogue with my users has been far more effective than any shaping
routine I/we could come up with.

Ask yourself, what is the problem you are really trying to solve.

One way to do this is to look for the style of traffic. For example, I
look for lots of connections from one PC to port numbers above 1024.
This will also incorrectly recognise some other traffic, but on the
whole it works well for me.
The following are some examples using connlimit (now included in vanilla
kernel) and ipset (see http://ipset.netfilter.org/)
# first look for style of traffic and log that client to an ipset
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p src
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
-m connlimit --connlimit-above 10 -j SET --add-set p2p dst
# then shape traffic above port 1024 for those detected clients
iptables -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
iptables -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \
-m set --set p2p dst -j MARK --set-mark 60
Regards,
Andy Beverley
_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Andrew Beverley

2007-12-03 19:33:12 UTC

Post by Gustin Johnson
I believe "fighting" is the wrong approach. Badly shaping the wrong
traffic is just as bad, if not worse IMO. An ISP in my neck of the
woods plays havoc with encrypted mail (SMTP + TLS as well as IMAPS) as a
result of their P2P fight. Needless to say we no longer use them, and
we encourage clients, friends, and colleagues not to as well. I don't
use P2P but I do use ssh, imaps, sftp, and https daily. Screwing with
these services is not useful.

Using the rules in the example previously given specifically steers well clear
of these services.

Post by Gustin Johnson
Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Post by Gustin Johnson
For me, if P2P does not belong at all, for instance on a corporate
network, then a default deny on the outbound works much better. We then
only allow specific connections on a case by case basis.

I have seen this work very well on corporate networks, and would
recommend this
approach where possible. Unfortunately though, on a normal home user network,
there are so many different possibilities that this isn't very practical.

Post by Gustin Johnson
For instances
where I am not able to block p2p, I define specific rules for high and
low priority, and leave everything else in the default. If the end user
wants to use the bulk of his or her bandwidth for P2P, so be it. Of
course in this case bandwidth accounting is far more useful.

Again, this depends on the circumstances. If you only have 2Mbit/s to share
between 100 users then each user cannot have their own 'share' of the
connection. Equally, people downloading in a responsible way are lumped
into the
same category as p2p users, which is not fair. Bandwidth accounting is a
possibility, and something I haven't investigated.

For those who want to fairly share bandwidth beween users, I would
recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis,
rather than per connection. This prevents the hundreds of p2p connections from
drowning out single downloads.

Post by Gustin Johnson
I would also encourage your users to use software that is or can be well
behaved. Software that allows you set a proper TOS for instance. If
possible work with the end users.
I have personally found that the best solutions are not tech solutions.
Having a well defined Acceptable Use Policy, plus a constructive
dialogue with my users has been far more effective than any shaping
routine I/we could come up with.

Agreed. However, in a situation where you have a lot of users coming
and going,
it is not easy to educate the many hundreds of users.

I guess it all boils down to your own situation. Traffic shaping on a
corporate
network or on a network where your users are static can be done using
the above
techniques. However, sharing a small connection between hundreds of regularly
changing users is difficult, and I have found the 'blunt' rules previously
described to work very well with no complaints.

Regards,

Andy Beverley

the sew

2007-12-10 13:37:07 UTC

Hi,

We had similiar problem with p2p, used ipp2p and L7filter together
before and worked well until clients( mostly clever ones) started
getting around it with encryption. We have about 700 wireless clients
hitting our network and our network was taking big knocks with guys
using couple of gigs day on entry level packages.

Was going to use Ipoque, but was quite pricy for us, Only solutions
for us to use a daily limit of eg 500MB, then they get slowed down to
slower speeds, This worked like a charm

Out of interest we used freeradius / pptpd|pppd with some custom perl
scripts and tc rules

Sew

Using the rules in the example previously given specifically steers well clear
of these services.

Post by Gustin Johnson
Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Again, this depends on the circumstances. If you only have 2Mbit/s to share
between 100 users then each user cannot have their own 'share' of the
connection. Equally, people downloading in a responsible way are lumped
into the
same category as p2p users, which is not fair. Bandwidth accounting is a
possibility, and something I haven't investigated.
For those who want to fairly share bandwidth beween users, I would
recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis,
rather than per connection. This prevents the hundreds of p2p connections from
drowning out single downloads.

Agreed. However, in a situation where you have a lot of users coming
and going,
it is not easy to educate the many hundreds of users.
I guess it all boils down to your own situation. Traffic shaping on a
corporate
network or on a network where your users are static can be done using
the above
techniques. However, sharing a small connection between hundreds of regularly
changing users is difficult, and I have found the 'blunt' rules previously
described to work very well with no complaints.
Regards,
Andy Beverley
_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Mario Antonio Garcia

2007-12-10 14:09:19 UTC

Thanks for sharing.

Just one question, how are you implementing the daily limit?

Regards,

Mario Antonio

----- Original Message -----
From: "the sew" <***@gmail.com>
To: "Andrew Beverley" <***@andybev.com>
Cc: ***@mailman.ds9a.nl
Sent: Monday, December 10, 2007 8:37:07 AM (GMT-0500) America/New_York
Subject: Re: [LARTC] How to fight with encrypted p2p

Hi,

We had similiar problem with p2p, used ipp2p and L7filter together
before and worked well until clients( mostly clever ones) started
getting around it with encryption. We have about 700 wireless clients
hitting our network and our network was taking big knocks with guys
using couple of gigs day on entry level packages.

Was going to use Ipoque, but was quite pricy for us, Only solutions
for us to use a daily limit of eg 500MB, then they get slowed down to
slower speeds, This worked like a charm

Out of interest we used freeradius / pptpd|pppd with some custom perl
scripts and tc rules

Sew

Using the rules in the example previously given specifically steers well clear
of these services.

Post by Gustin Johnson
Limiting your rules to specific ports is
pretty useless. This has been done before, and it failed miserably.

Agreed.

Again, this depends on the circumstances. If you only have 2Mbit/s to share
between 100 users then each user cannot have their own 'share' of the
connection. Equally, people downloading in a responsible way are lumped
into the
same category as p2p users, which is not fair. Bandwidth accounting is a
possibility, and something I haven't investigated.
For those who want to fairly share bandwidth beween users, I would
recommend the
ESFQ patches. These allow bandwidth sharing to be done on an IP address basis,
rather than per connection. This prevents the hundreds of p2p connections from
drowning out single downloads.

Agreed. However, in a situation where you have a lot of users coming
and going,
it is not easy to educate the many hundreds of users.
I guess it all boils down to your own situation. Traffic shaping on a
corporate
network or on a network where your users are static can be done using
the above
techniques. However, sharing a small connection between hundreds of regularly
changing users is difficult, and I have found the 'blunt' rules previously
described to work very well with no complaints.
Regards,
Andy Beverley
_______________________________________________
LARTC mailing list
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

the sew

2007-12-10 14:28:30 UTC

Post by Mario Antonio Garcia
Thanks for sharing.

No problem

Post by Mario Antonio Garcia
Just one question, how are you implementing the daily limit?

two ways of detecting them:

1st is the /etc/ppp/ip-up.local which executes the script to check
usage against the radius db and shape them on authentication,

Once they logged in , I dont want to kick everone off every few hours
to check usage, I have a "nice" cron job running every 3 hours, to
check every single user against the db and if they reached their
quotas ,they get shaped while being online., Radius stores all info
about the nas in the db, so makes it quite strait forward.

Also nicely added is our reselling guys who maintain the clients get a
report everytime of users who exceeded the limit and can be aware of
which clients is the problem if they phone to complain, Also nice is
that usually the infected pc's gets knocked off first to save alot of
bandwidth

Sew

Post by Mario Antonio Garcia
Regards,
Mario Antonio
----- Original Message -----
Sent: Monday, December 10, 2007 8:37:07 AM (GMT-0500) America/New_York
Subject: Re: [LARTC] How to fight with encrypted p2p
Hi,
We had similiar problem with p2p, used ipp2p and L7filter together
before and worked well until clients( mostly clever ones) started
getting around it with encryption. We have about 700 wireless clients
hitting our network and our network was taking big knocks with guys
using couple of gigs day on entry level packages.
Was going to use Ipoque, but was quite pricy for us, Only solutions
for us to use a daily limit of eg 500MB, then they get slowed down to
slower speeds, This worked like a charm
Out of interest we used freeradius / pptpd|pppd with some custom perl
scripts and tc rules
Sew