From owner-uuasc@uuasc.org Thu Nov 2 12:09:05 2006 Date: Thu, 2 Nov 2006 12:08:49 -0800 From: Phil Dibowitz To: uuasc@uuasc.org Subject: Cisco is evil Some info incase anyone else runs into this... So I was tasked with troubleshooting an issue yesterday that a lot of people had been having problems getting the bottom of. The problem turned out to be 3-fold: 1. Something was doing MSS-clamping and munging packets from "mss 1460" to "mss 1380" (MSS is MTU-40 - ethernet is 1500 MTU, so MSS is 1460). 2. Solaris 8 isn't honoring this munged value (which is why we never saw it linux -> linux). 3. The latest PIX (aka ASA) code (7.0) drops packets of sizes bigger than the negotiated MSS (it keeps track of said negotiations in 7.0). Fortunately I have a LOT of experience with MSS/MTU/PTMUD [1], so I had an advantage catching this. ;) Anyway, turns out, that the PIX code, by default - as far back as we've been able to find - defaults to MSS clamping (EVIL!) all packets to 1380 [2]. They get this value, according to the docs, based on what it would end up being if you were going through an IPSec tunnel - despite the fact that we have no IPSec configured - this is just default. The solution here is 'no sysopt connection tcpmss'... but seriously, I expected more from Cisco. MSS Clamping is sometimes a necessary hack... but to enable it be default is just seriously wrong. Anyway, I thought I'd share incase anyone else runs into this. [1] http://www.phildev.net/mss/ [2] See "sysopt connection tcpmss" at: http://www.cisco.com/en/US/products/ps6120/products_command_reference_chapter 09186a008073ad68.html#wp1542036 -- Phil Dibowitz phil@ipom.com Freeware and Technical Pages Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." - Benjamin Franklin, 1759 [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc] ------------------------------------------------------------------------ --General instructions-- --Administrative requests-- Post to this list, Send to Majordomo@uuasc.org, no subject needed send to uuasc@uuasc.org Include ONLY the following in the message BODY Last resort, send to To unsubscribe: "unsubscribe uuasc" owner-uuasc@uuasc.org Information or help: "info uuasc" or "help" ====== From owner-uuasc@uuasc.org Thu Nov 2 13:01:26 2006 Subject: RE: Cisco is evil Date: Thu, 2 Nov 2006 13:01:14 -0800 From: "Jeremy Boyd" To: We had a similar problem when we upgraded from 6.3 code to 7.1 code on the PIX. However, since we did have IPSEC tunnels we solved it a slightly different way. Basically we used the new class map features of the 7.x code and made a map. tcp-map mss-map exceed-mss allow class mss-traffic-map set connection advanced-options mss-map access-list mss-traffic-list extended permit tcp any any class-map mss-traffic-map match access-list mss-traffic-list The 'no sysopt connection tcpmss' command seems more elegant... But I wonder if stomp on the IPSEC tunnels. Does anyone know? Jeremy -----Original Message----- From: owner-uuasc@uuasc.org [mailto:owner-uuasc@uuasc.org] On Behalf Of Phil Dibowitz Sent: Thursday, November 02, 2006 12:09 PM To: uuasc@uuasc.org Subject: Cisco is evil Some info incase anyone else runs into this... So I was tasked with troubleshooting an issue yesterday that a lot of people had been having problems getting the bottom of. The problem turned out to be 3-fold: 1. Something was doing MSS-clamping and munging packets from "mss 1460" to "mss 1380" (MSS is MTU-40 - ethernet is 1500 MTU, so MSS is 1460). 2. Solaris 8 isn't honoring this munged value (which is why we never saw it linux -> linux). 3. The latest PIX (aka ASA) code (7.0) drops packets of sizes bigger than the negotiated MSS (it keeps track of said negotiations in 7.0). Fortunately I have a LOT of experience with MSS/MTU/PTMUD [1], so I had an advantage catching this. ;) Anyway, turns out, that the PIX code, by default - as far back as we've been able to find - defaults to MSS clamping (EVIL!) all packets to 1380 [2]. They get this value, according to the docs, based on what it would end up being if you were going through an IPSec tunnel - despite the fact that we have no IPSec configured - this is just default. The solution here is 'no sysopt connection tcpmss'... but seriously, I expected more from Cisco. MSS Clamping is sometimes a necessary hack... but to enable it be default is just seriously wrong. Anyway, I thought I'd share incase anyone else runs into this. [1] http://www.phildev.net/mss/ [2] See "sysopt connection tcpmss" at: http://www.cisco.com/en/US/products/ps6120/products_command_reference_ch apter 09186a008073ad68.html#wp1542036 -- Phil Dibowitz phil@ipom.com Freeware and Technical Pages Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." - Benjamin Franklin, 1759 ====== From owner-uuasc@uuasc.org Thu Nov 2 13:04:21 2006 Date: Thu, 2 Nov 2006 13:04:05 -0800 From: Phil Dibowitz To: uuasc@uuasc.org Subject: Re: Cisco is evil On Thu, Nov 02, 2006 at 01:01:14PM -0800, Jeremy Boyd wrote: > We had a similar problem when we upgraded from 6.3 code to 7.1 code on > the PIX. However, since we did have IPSEC tunnels we solved it a > slightly different way. Basically we used the new class map features of > the 7.x code and made a map. We added the policy map too - we only found the tcpmss option this morning. > The 'no sysopt connection tcpmss' command seems more elegant... But I > wonder if stomp on the IPSEC tunnels. Does anyone know? It should work just fine provided you're not blocking ICMP 3/4 anywhere (required for Path MTU Discovery to work properly). -- Phil Dibowitz phil@ipom.com Freeware and Technical Pages Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." - Benjamin Franklin, 1759 ====== From owner-uuasc@uuasc.org Thu Nov 2 13:48:13 2006 Date: Thu, 2 Nov 2006 13:47:53 -0800 From: Phil Dibowitz To: uuasc@uuasc.org Subject: Re: Cisco is evil On Thu, Nov 02, 2006 at 01:04:05PM -0800, Phil Dibowitz wrote: > On Thu, Nov 02, 2006 at 01:01:14PM -0800, Jeremy Boyd wrote: > > We had a similar problem when we upgraded from 6.3 code to 7.1 code on > > the PIX. However, since we did have IPSEC tunnels we solved it a > > slightly different way. Basically we used the new class map features of > > the 7.x code and made a map. > > We added the policy map too - we only found the tcpmss option this morning. > > > The 'no sysopt connection tcpmss' command seems more elegant... But I > > wonder if stomp on the IPSEC tunnels. Does anyone know? > > It should work just fine provided you're not blocking ICMP 3/4 anywhere > (required for Path MTU Discovery to work properly). BTW - my Cisco expert is telling me 'no sysopt connection tcpmss' resets it to the default of 1380, and what we actually want is 'sysopt connection tcpmss 0'. We'll be testing such a change and rolling out in the next available maintenance window. We'll probably also turn off that policy map previously mentioned. -- Phil Dibowitz phil@ipom.com Freeware and Technical Pages Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." - Benjamin Franklin, 1759