Friday, 11 August 2017

ASA : Drop-reason: (conn-limit) Connection limit reached

On a Normal day I was sitting and relaxing, Got a P1 call from one my customers, stating High urgency and business critical. We got onto the call. Below are the details

Issue Reported : Connections are getting dropped by ASA from outside and other interfaces

Initial Log Provided :  It was only a packet-tracer from ASA

pri/act# packet-tracer input dmz3 udp 10.10.21.24 2343 192.168.100.74 111 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   192.168.0.0     255.255.0.0     inside

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   10.10.21.0      255.255.255.0   dmz3

Result:
input-interface: dmz3
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (conn-limit) Connection limit reached

Hardware ASA : ASA 5520 running 8.2 version of code

Troubleshooting Steps/ Strategy
We started questioning and finally we come to a last statement that connections for specific source and destination are getting dropped.

as per the drop reason , we started with looking at resources for the ASA
1. show resources usage all

Resource               Current        Peak      Limit        Denied Context
SSH                          0           5          5            44 System
Syslogs [rate]              77        1804        N/A             0 System
Conns                     4156        9817     280000             0 System
Xlates                    2912        8400        N/A             0 System
Conns [rate]                62         984        N/A             0 System
Inspects [rate]             13         584        N/A             0 System

Could not be the reason , as all were with in the limit.

2. Checked CPU and memory

CPU was only 31%, Memory was only 61 %

3. Checked Static Nat configuration, where connection limit might have been defined for the specific connection.

There was not static nat or any nat which defines the limit , limit was only defined for UDP connections upto 5, but for TCP it was 2000.

4. Check ASP drop usage

show asp drop 

asp drop shows the packet count incremented which are getting drop because of connection limit


5. It took more than 3 hours to check all the statements , but still no clue for the connection limit. But for sure it was not a hardware-limitation.

6. Finally we found that it was because, in ASA static nat statement if we define the connection limit, it does not take both source and destination into consideration , but connection limit in NAT works only with consideration to source. 

and we found a statement where a /24 subnet was limited to number of connections. 

Conclusion : In none of documents, technical documents it has been specified that, if we have done a connection limit in static statement , it does not match both source and destination , but it only matches the source and does a connection limit.

Recommendation : Please consider MPF/ Policy-map to limit the number of connections, where it could not be included in static statement.