5 This document explains the meaning of SNMP counters.
9 All layer 4 packets and ICMP packets will change these counters, but
10 these counters won't be changed by layer 2 packets (such as STP) or
15 Defined in `RFC1213 ipInReceives`_
17 .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
19 The number of packets received by the IP layer. It gets increasing at the
20 beginning of ip_rcv function, always be updated together with
21 IpExtInOctets. It will be increased even if the packet is dropped
22 later (e.g. due to the IP header is invalid or the checksum is wrong
23 and so on). It indicates the number of aggregated segments after
28 Defined in `RFC1213 ipInDelivers`_
30 .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
32 The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
33 ICMP and so on. If no one listens on a raw socket, only kernel
34 supported protocols will be delivered, if someone listens on the raw
35 socket, all valid IP packets will be delivered.
39 Defined in `RFC1213 ipOutRequests`_
41 .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
43 The number of packets sent via IP layer, for both single cast and
44 multicast packets, and would always be updated together with
47 * IpExtInOctets and IpExtOutOctets
49 They are Linux kernel extensions, no RFC definitions. Please note,
50 RFC1213 indeed defines ifInOctets and ifOutOctets, but they
51 are different things. The ifInOctets and ifOutOctets include the MAC
52 layer header size but IpExtInOctets and IpExtOutOctets don't, they
53 only include the IP layer header and the IP layer data.
55 * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
57 They indicate the number of four kinds of ECN IP packets, please refer
58 `Explicit Congestion Notification`_ for more details.
60 .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
62 These 4 counters calculate how many packets received per ECN
63 status. They count the real frame number regardless the LRO/GRO. So
64 for the same packet, you might find that IpInReceives count 1, but
65 IpExtInNoECTPkts counts 2 or more.
69 Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
70 dropped due to the IP header error. It might happen in both IP input
73 .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
77 Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
78 scenarios: (1) The IP address is invalid. (2) The destination IP
79 address is not a local address and IP forwarding is not enabled
81 .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
85 This counter means the packet is dropped when the IP stack receives a
86 packet and can't find a route for it from the route table. It might
87 happen when IP forwarding is enabled and the destination IP address is
88 not a local address and there is no route for the destination IP
93 Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
94 layer 4 protocol is unsupported by kernel. If an application is using
95 raw socket, kernel will always deliver the packet to the raw socket
96 and this counter won't be increased.
98 .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
100 * IpExtInTruncatedPkts
102 For IPv4 packet, it means the actual data size is smaller than the
103 "Total Length" field in the IPv4 header.
107 Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
108 in the IP receiving path and due to kernel internal reasons (e.g. no
111 .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
115 Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
116 dropped in the IP sending path and due to kernel internal reasons.
118 .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
122 Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
123 dropped in the IP sending path and no route is found for it.
125 .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
129 * IcmpInMsgs and IcmpOutMsgs
131 Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
133 .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134 .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
136 As mentioned in the RFC1213, these two counters include errors, they
137 would be increased even if the ICMP packet has an invalid type. The
138 ICMP output path will check the header of a raw socket, so the
139 IcmpOutMsgs would still be updated if the IP header is constructed by
144 | These counters include most of common ICMP types, they are:
145 | IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
146 | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
147 | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
148 | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
149 | IcmpInRedirects: `RFC1213 icmpInRedirects`_
150 | IcmpInEchos: `RFC1213 icmpInEchos`_
151 | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
152 | IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
153 | IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
154 | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
155 | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
156 | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
157 | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
158 | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
159 | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
160 | IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
161 | IcmpOutEchos: `RFC1213 icmpOutEchos`_
162 | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
163 | IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
164 | IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
165 | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
166 | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
168 .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169 .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170 .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171 .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172 .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173 .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174 .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175 .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176 .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177 .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178 .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
180 .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181 .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182 .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183 .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184 .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185 .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186 .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187 .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188 .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189 .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190 .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
192 Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
193 Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
194 straightforward. The 'In' counter means kernel receives such a packet
195 and the 'Out' counter means kernel sends such a packet.
199 They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
200 ICMP type number. These counters track all kinds of ICMP packets. The
201 ICMP type number definition could be found in the `ICMP parameters`_
204 .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
206 For example, if the Linux kernel sends an ICMP Echo packet, the
207 IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
208 packet, IcmpMsgInType0 would increase 1.
212 This counter indicates the checksum of the ICMP packet is
213 wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
214 before updating IcmpMsgInType[N]. If a packet has bad checksum, the
215 IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
217 * IcmpInErrors and IcmpOutErrors
219 Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
221 .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222 .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
224 When an error occurs in the ICMP packet handler path, these two
225 counters would be updated. The receiving packet path use IcmpInErrors
226 and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
227 is increased, IcmpInErrors would always be increased too.
229 relationship of the ICMP counters
230 ---------------------------------
231 The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
232 are updated at the same time. The sum of IcmpMsgInType[N] plus
233 IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
234 receives an ICMP packet, kernel follows below logic:
236 1. increase IcmpInMsgs
237 2. if has any error, update IcmpInErrors and finish the process
238 3. update IcmpMsgOutType[N]
239 4. handle the packet depending on the type, if has any error, update
240 IcmpInErrors and finish the process
242 So if all errors occur in step (2), IcmpInMsgs should be equal to the
243 sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
244 step (4), IcmpInMsgs should be equal to the sum of
245 IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
246 IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
253 Defined in `RFC1213 tcpInSegs`_
255 .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
257 The number of packets received by the TCP layer. As mentioned in
258 RFC1213, it includes the packets received in error, such as checksum
259 error, invalid TCP header and so on. Only one error won't be included:
260 if the layer 2 destination address is not the NIC's layer 2
261 address. It might happen if the packet is a multicast or broadcast
262 packet, or the NIC is in promiscuous mode. In these situations, the
263 packets would be delivered to the TCP layer, but the TCP layer will discard
264 these packets before increasing TcpInSegs. The TcpInSegs counter
265 isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
266 counter would only increase 1.
270 Defined in `RFC1213 tcpOutSegs`_
272 .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
274 The number of packets sent by the TCP layer. As mentioned in RFC1213,
275 it excludes the retransmitted packets. But it includes the SYN, ACK
276 and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
277 GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
282 Defined in `RFC1213 tcpActiveOpens`_
284 .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
286 It means the TCP layer sends a SYN, and come into the SYN-SENT
287 state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
292 Defined in `RFC1213 tcpPassiveOpens`_
294 .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
296 It means the TCP layer receives a SYN, replies a SYN+ACK, come into
299 * TcpExtTCPRcvCoalesce
301 When packets are received by the TCP layer and are not be read by the
302 application, the TCP layer will try to merge them. This counter
303 indicate how many packets are merged in such situation. If GRO is
304 enabled, lots of packets would be merged by GRO, these packets
305 wouldn't be counted to TcpExtTCPRcvCoalesce.
307 * TcpExtTCPAutoCorking
309 When sending packets, the TCP layer will try to merge small packets to
310 a bigger one. This counter increase 1 for every packet merged in such
311 situation. Please refer to the LWN article for more details:
312 https://lwn.net/Articles/576263/
314 * TcpExtTCPOrigDataSent
316 This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
319 TCPOrigDataSent: number of outgoing packets with original data (excluding
320 retransmission but including data-in-SYN). This counter is different from
321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
322 more useful to track the TCP retransmission rate.
326 This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
332 * TCPFastOpenActiveFail
334 This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
338 the remote does not accept it or the attempts timed out.
340 .. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
342 * TcpExtListenOverflows and TcpExtListenDrops
344 When kernel receives a SYN from a client, and if the TCP accept queue
345 is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
346 At the same time kernel will also add 1 to TcpExtListenDrops. When a
347 TCP socket is in LISTEN state, and kernel need to drop a packet,
348 kernel would always add 1 to TcpExtListenDrops. So increase
349 TcpExtListenOverflows would let TcpExtListenDrops increasing at the
350 same time, but TcpExtListenDrops would also increase without
351 TcpExtListenOverflows increasing, e.g. a memory allocation fail would
352 also let TcpExtListenDrops increase.
354 Note: The above explanation is based on kernel 4.10 or above version, on
355 an old kernel, the TCP stack has different behavior when TCP accept
356 queue is full. On the old kernel, TCP stack won't drop the SYN, it
357 would complete the 3-way handshake. As the accept queue is full, TCP
358 stack will keep the socket in the TCP half-open queue. As it is in the
359 half open queue, TCP stack will send SYN+ACK on an exponential backoff
360 timer, after client replies ACK, TCP stack checks whether the accept
361 queue is still full, if it is not full, moves the socket to the accept
362 queue, if it is full, keeps the socket in the half-open queue, at next
363 time client replies ACK, this socket will get another chance to move
371 Defined in `RFC1213 tcpEstabResets`_.
373 .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
377 Defined in `RFC1213 tcpAttemptFails`_.
379 .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
383 Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
384 the 'segments sent containing the RST flag', but in linux kernel, this
385 couner indicates the segments kerenl tried to send. The sending
386 process might be failed due to some errors (e.g. memory alloc failed).
388 .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
390 * TcpExtTCPSpuriousRtxHostQueues
392 When the TCP stack wants to retransmit a packet, and finds that packet
393 is not lost in the network, but the packet is not sent yet, the TCP
394 stack would give up the retransmission and update this counter. It
395 might happen if a packet stays too long time in a qdisc or driver
400 The socket receives a RST packet in Establish or CloseWait state.
404 This counter indicates many keepalive packets were sent. The keepalive
405 won't be enabled by default. A userspace program could enable it by
406 setting the SO_KEEPALIVE socket option.
408 * TcpExtTCPSpuriousRTOs
410 The spurious retransmission timeout detected by the `F-RTO`_
413 .. _F-RTO: https://tools.ietf.org/html/rfc5682
417 When kernel receives a TCP packet, it has two paths to handler the
418 packet, one is fast path, another is slow path. The comment in kernel
419 code provides a good explanation of them, I pasted them below::
421 It is split into a fast path and a slow path. The fast path is
424 - A zero window was announced from us
425 - zero window probing
426 is only handled properly on the slow path.
427 - Out of order segments arrived.
428 - Urgent data is expected.
429 - There is no buffer space left
430 - Unexpected TCP flags/window values/header lengths are received
431 (detected by checking the TCP header against pred_flags)
432 - Data is sent in both directions. The fast path only supports pure senders
433 or pure receivers (this means either the sequence number or the ack
434 value must stay constant)
435 - Unexpected TCP option.
437 Kernel will try to use fast path unless any of the above conditions
438 are satisfied. If the packets are out of order, kernel will handle
439 them in slow path, which means the performance might be not very
440 good. Kernel would also come into slow path if the "Delayed ack" is
441 used, because when using "Delayed ack", the data is sent in both
442 directions. When the TCP window scale option is not used, kernel will
443 try to enable fast path immediately when the connection comes into the
444 established state, but if the TCP window scale option is used, kernel
445 will disable the fast path at first, and try to enable it after kernel
448 * TcpExtTCPPureAcks and TcpExtTCPHPAcks
450 If a packet set ACK flag and has no data, it is a pure ACK packet, if
451 kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
452 if kernel handles it in the slow path, TcpExtTCPPureAcks will
457 If a TCP packet has data (which means it is not a pure ACK packet),
458 and this packet is handled in the fast path, TcpExtTCPHPHits will
464 * TcpExtTCPAbortOnData
466 It means TCP layer has data in flight, but need to close the
467 connection. So TCP layer sends a RST to the other side, indicate the
468 connection is not closed very graceful. An easy way to increase this
469 counter is using the SO_LINGER option. Please refer to the SO_LINGER
470 section of the `socket man page`_:
472 .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
474 By default, when an application closes a connection, the close function
475 will return immediately and kernel will try to send the in-flight data
476 async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
477 to a positive number, the close function won't return immediately, but
478 wait for the in-flight data are acked by the other side, the max wait
479 time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
480 when the application closes a connection, kernel will send a RST
481 immediately and increase the TcpExtTCPAbortOnData counter.
483 * TcpExtTCPAbortOnClose
485 This counter means the application has unread data in the TCP layer when
486 the application wants to close the TCP connection. In such a situation,
487 kernel will send a RST to the other side of the TCP connection.
489 * TcpExtTCPAbortOnMemory
491 When an application closes a TCP connection, kernel still need to track
492 the connection, let it complete the TCP disconnect process. E.g. an
493 app calls the close method of a socket, kernel sends fin to the other
494 side of the connection, then the app has no relationship with the
495 socket any more, but kernel need to keep the socket, this socket
496 becomes an orphan socket, kernel waits for the reply of the other side,
497 and would come to the TIME_WAIT state finally. When kernel has no
498 enough memory to keep the orphan socket, kernel would send an RST to
499 the other side, and delete the socket, in such situation, kernel will
500 increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
501 TcpExtTCPAbortOnMemory:
503 1. the memory used by the TCP protocol is higher than the third value of
504 the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
506 .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
508 2. the orphan socket count is higher than net.ipv4.tcp_max_orphans
511 * TcpExtTCPAbortOnTimeout
513 This counter will increase when any of the TCP timers expire. In such
514 situation, kernel won't send RST, just give up the connection.
516 * TcpExtTCPAbortOnLinger
518 When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
519 for the fin packet from the other side, kernel could send a RST and
520 delete the socket immediately. This is not the default behavior of
521 Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
522 you could let kernel follow this behavior.
524 * TcpExtTCPAbortFailed
526 The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
527 satisfied. If an internal error occurs during this process,
528 TcpExtTCPAbortFailed will be increased.
530 .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
532 TCP Hybrid Slow Start
533 =====================
534 The Hybrid Slow Start algorithm is an enhancement of the traditional
535 TCP congestion window Slow Start algorithm. It uses two pieces of
536 information to detect whether the max bandwidth of the TCP path is
537 approached. The two pieces of information are ACK train length and
538 increase in packet delay. For detail information, please refer the
539 `Hybrid Slow Start paper`_. Either ACK train length or packet delay
540 hits a specific threshold, the congestion control algorithm will come
541 into the Congestion Avoidance state. Until v4.20, two congestion
542 control algorithms are using Hybrid Slow Start, they are cubic (the
543 default congestion control algorithm) and cdg. Four snmp counters
544 relate with the Hybrid Slow Start algorithm.
546 .. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
548 * TcpExtTCPHystartTrainDetect
550 How many times the ACK train length threshold is detected
552 * TcpExtTCPHystartTrainCwnd
554 The sum of CWND detected by ACK train length. Dividing this value by
555 TcpExtTCPHystartTrainDetect is the average CWND which detected by the
558 * TcpExtTCPHystartDelayDetect
560 How many times the packet delay threshold is detected.
562 * TcpExtTCPHystartDelayCwnd
564 The sum of CWND detected by packet delay. Dividing this value by
565 TcpExtTCPHystartDelayDetect is the average CWND which detected by the
568 TCP retransmission and congestion control
569 =========================================
570 The TCP protocol has two retransmission mechanisms: SACK and fast
571 recovery. They are exclusive with each other. When SACK is enabled,
572 the kernel TCP stack would use SACK, or kernel would use fast
573 recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
574 the fast recovery is defined in `RFC6582`_, which is also called
577 The TCP congestion control is a big and complex topic. To understand
578 the related snmp counter, we need to know the states of the congestion
579 control state machine. There are 5 states: Open, Disorder, CWR,
580 Recovery and Loss. For details about these states, please refer page 5
581 and page 6 of this document:
582 https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
584 .. _RFC2018: https://tools.ietf.org/html/rfc2018
585 .. _RFC6582: https://tools.ietf.org/html/rfc6582
587 * TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
589 When the congestion control comes into Recovery state, if sack is
590 used, TcpExtTCPSackRecovery increases 1, if sack is not used,
591 TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
592 stack begins to retransmit the lost packets.
594 * TcpExtTCPSACKReneging
596 A packet was acknowledged by SACK, but the receiver has dropped this
597 packet, so the sender needs to retransmit this packet. In this
598 situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
599 could drop a packet which has been acknowledged by SACK, although it is
600 unusual, it is allowed by the TCP protocol. The sender doesn't really
601 know what happened on the receiver side. The sender just waits until
602 the RTO expires for this packet, then the sender assumes this packet
603 has been dropped by the receiver.
605 * TcpExtTCPRenoReorder
607 The reorder packet is detected by fast recovery. It would only be used
608 if SACK is disabled. The fast recovery algorithm detects recorder by
609 the duplicate ACK number. E.g., if retransmission is triggered, and
610 the original retransmitted packet is not lost, it is just out of
611 order, the receiver would acknowledge multiple times, one for the
612 retransmitted packet, another for the arriving of the original out of
613 order packet. Thus the sender would find more ACks than its
614 expectation, and the sender knows out of order occurs.
618 The reorder packet is detected when a hole is filled. E.g., assume the
619 sender sends packet 1,2,3,4,5, and the receiving order is
620 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
621 fill the hole), two conditions will let TcpExtTCPTSReorder increase
622 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
623 3 is retransmitted but the timestamp of the packet 3's ACK is earlier
624 than the retransmission timestamp.
626 * TcpExtTCPSACKReorder
628 The reorder packet detected by SACK. The SACK has two methods to
629 detect reorder: (1) DSACK is received by the sender. It means the
630 sender sends the same packet more than one times. And the only reason
631 is the sender believes an out of order packet is lost so it sends the
632 packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
633 the sender has received SACKs for packet 2 and 5, now the sender
634 receives SACK for packet 4 and the sender doesn't retransmit the
635 packet yet, the sender would know packet 4 is out of order. The TCP
636 stack of kernel will increase TcpExtTCPSACKReorder for both of the
639 * TcpExtTCPSlowStartRetrans
641 The TCP stack wants to retransmit a packet and the congestion control
644 * TcpExtTCPFastRetrans
646 The TCP stack wants to retransmit a packet and the congestion control
649 * TcpExtTCPLostRetransmit
651 A SACK points out that a retransmission packet is lost again.
653 * TcpExtTCPRetransFail
655 The TCP stack tries to deliver a retransmission packet to lower layers
656 but the lower layers return an error.
658 * TcpExtTCPSynRetrans
660 The TCP stack retransmits a SYN packet.
664 The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
665 duplicate packets to the sender. There are two kinds of
666 duplications: (1) a packet which has been acknowledged is
667 duplicate. (2) an out of order packet is duplicate. The TCP stack
668 counts these two kinds of duplications on both receiver side and
671 .. _RFC2883 : https://tools.ietf.org/html/rfc2883
673 * TcpExtTCPDSACKOldSent
675 The TCP stack receives a duplicate packet which has been acked, so it
676 sends a DSACK to the sender.
678 * TcpExtTCPDSACKOfoSent
680 The TCP stack receives an out of order duplicate packet, so it sends a
684 The TCP stack receives a DSACK, which indicates an acknowledged
685 duplicate packet is received.
687 * TcpExtTCPDSACKOfoRecv
689 The TCP stack receives a DSACK, which indicate an out of order
690 duplicate packet is received.
692 invalid SACK and DSACK
694 When a SACK (or DSACK) block is invalid, a corresponding counter would
695 be updated. The validation method is base on the start/end sequence
696 number of the SACK block. For more details, please refer the comment
697 of the function tcp_is_sackblock_valid in the kernel source code. A
698 SACK option could have up to 4 blocks, they are checked
699 individually. E.g., if 3 blocks of a SACk is invalid, the
700 corresponding counter would be updated 3 times. The comment of the
701 `Add counters for discarded SACK blocks`_ patch has additional
704 .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32
706 * TcpExtTCPSACKDiscard
707 This counter indicates how many SACK blocks are invalid. If the invalid
708 SACK block is caused by ACK recording, the TCP stack will only ignore
709 it and won't update this counter.
711 * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
712 When a DSACK block is invalid, one of these two counters would be
713 updated. Which counter will be updated depends on the undo_marker flag
714 of the TCP socket. If the undo_marker is not set, the TCP stack isn't
715 likely to re-transmit any packets, and we still receive an invalid
716 DSACK block, the reason might be that the packet is duplicated in the
717 middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
718 will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
719 will be updated. As implied in its name, it might be an old packet.
723 The linux networking stack stores data in sk_buff struct (skb for
724 short). If a SACK block acrosses multiple skb, the TCP stack will try
725 to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
726 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
727 15 in skb2 would be moved to skb1. This operation is 'shift'. If a
728 SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
729 seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
730 discard, this operation is 'merge'.
732 * TcpExtTCPSackShifted
735 * TcpExtTCPSackMerged
738 * TcpExtTCPSackShiftFallback
739 A skb should be shifted or merged, but the TCP stack doesn't do it for
746 The TCP layer receives an out of order packet and has enough memory
751 The TCP layer receives an out of order packet but doesn't have enough
752 memory, so drops it. Such packets won't be counted into
757 The received out of order packet has an overlay with the previous
758 packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
759 packets will also be counted into TcpExtTCPOFOQueue.
763 PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
764 which is used to drop old packets. It depends on the TCP
765 timestamps. For detail information, please refer the `timestamp wiki`_
766 and the `RFC of PAWS`_.
768 .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
769 .. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
773 Packets are dropped by PAWS in Syn-Sent status.
777 Packets are dropped by PAWS in any status other than Syn-Sent.
781 In some scenarios, kernel would avoid sending duplicate ACKs too
782 frequently. Please find more details in the tcp_invalid_ratelimit
783 section of the `sysctl document`_. When kernel decides to skip an ACK
784 due to tcp_invalid_ratelimit, kernel would update one of below
785 counters to indicate the ACK is skipped in which scenario. The ACK
786 would only be skipped if the received packet is either a SYN packet or
789 .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
791 * TcpExtTCPACKSkippedSynRecv
793 The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
794 TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
795 waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
796 in the Syn-Recv status. But in several scenarios, the TCP stack need
797 to send an ACK. E.g., the TCP stack receives the same SYN packet
798 repeately, the received packet does not pass the PAWS check, or the
799 received packet sequence number is out of window. In these scenarios,
800 the TCP stack needs to send ACK. If the ACk sending frequency is higher than
801 tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
802 increase TcpExtTCPACKSkippedSynRecv.
805 * TcpExtTCPACKSkippedPAWS
807 The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
808 numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
809 or Time-Wait statuses, the skipped ACK would be counted to
810 TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
811 TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
812 would be counted to TcpExtTCPACKSkippedPAWS.
814 * TcpExtTCPACKSkippedSeq
816 The sequence number is out of window and the timestamp passes the PAWS
817 check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
819 * TcpExtTCPACKSkippedFinWait2
821 The ACK is skipped in Fin-Wait-2 status, the reason would be either
822 PAWS check fails or the received sequence number is out of window.
824 * TcpExtTCPACKSkippedTimeWait
826 Tha ACK is skipped in Time-Wait status, the reason would be either
827 PAWS check failed or the received sequence number is out of window.
829 * TcpExtTCPACKSkippedChallenge
831 The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
832 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
833 `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
834 three scenarios, In some TCP status, the linux TCP stack would also
835 send challenge ACKs if the ACK number is before the first
836 unacknowledged number (more strict than `RFC 5961 section 5.2`_).
838 .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
839 .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
840 .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
844 * TcpExtTCPWantZeroWindowAdv
846 Depending on current memory usage, the TCP stack tries to set receive
847 window to zero. But the receive window might still be a no-zero
848 value. For example, if the previous window size is 10, and the TCP
849 stack receives 3 bytes, the current window size would be 7 even if the
850 window size calculated by the memory usage is zero.
852 * TcpExtTCPToZeroWindowAdv
854 The TCP receive window is set to zero from a no-zero value.
856 * TcpExtTCPFromZeroWindowAdv
858 The TCP receive window is set to no-zero value from zero.
863 The TCP Delayed ACK is a technique which is used for reducing the
864 packet count in the network. For more details, please refer the
867 .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
871 A delayed ACK timer expires. The TCP stack will send a pure ACK packet
872 and exit the delayed ACK mode.
874 * TcpExtDelayedACKLocked
876 A delayed ACK timer expires, but the TCP stack can't send an ACK
877 immediately due to the socket is locked by a userspace program. The
878 TCP stack will send a pure ACK later (after the userspace program
879 unlock the socket). When the TCP stack sends the pure ACK later, the
880 TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
883 * TcpExtDelayedACKLost
885 It will be updated when the TCP stack receives a packet which has been
886 ACKed. A Delayed ACK loss might cause this issue, but it would also be
887 triggered by other reasons, such as a packet is duplicated in the
890 Tail Loss Probe (TLP)
891 =====================
892 TLP is an algorithm which is used to detect TCP packet loss. For more
893 details, please refer the `TLP paper`_.
895 .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
897 * TcpExtTCPLossProbes
899 A TLP probe packet is sent.
901 * TcpExtTCPLossProbeRecovery
903 A packet loss is detected and recovered by TLP.
907 TCP Fast Open is a technology which allows data transfer before the
908 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
911 .. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
913 * TcpExtTCPFastOpenActive
915 When the TCP stack receives an ACK packet in the SYN-SENT status, and
916 the ACK packet acknowledges the data in the SYN packet, the TCP stack
917 understand the TFO cookie is accepted by the other side, then it
918 updates this counter.
920 * TcpExtTCPFastOpenActiveFail
922 This counter indicates that the TCP stack initiated a TCP Fast Open,
923 but it failed. This counter would be updated in three scenarios: (1)
924 the other side doesn't acknowledge the data in the SYN packet. (2) The
925 SYN packet which has the TFO cookie is timeout at least once. (3)
926 after the 3-way handshake, the retransmission timeout happens
927 net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
928 fast open after the handshake.
930 * TcpExtTCPFastOpenPassive
932 This counter indicates how many times the TCP stack accepts the fast
935 * TcpExtTCPFastOpenPassiveFail
937 This counter indicates how many times the TCP stack rejects the fast
938 open request. It is caused by either the TFO cookie is invalid or the
939 TCP stack finds an error during the socket creating process.
941 * TcpExtTCPFastOpenListenOverflow
943 When the pending fast open request number is larger than
944 fastopenq->max_qlen, the TCP stack will reject the fast open request
945 and update this counter. When this counter is updated, the TCP stack
946 won't update TcpExtTCPFastOpenPassive or
947 TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
948 TCP_FASTOPEN socket operation and it could not be larger than
949 net.core.somaxconn. For example:
951 setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
953 * TcpExtTCPFastOpenCookieReqd
955 This counter indicates how many times a client wants to request a TFO
960 SYN cookies are used to mitigate SYN flood, for details, please refer
961 the `SYN cookies wiki`_.
963 .. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
965 * TcpExtSyncookiesSent
967 It indicates how many SYN cookies are sent.
969 * TcpExtSyncookiesRecv
971 How many reply packets of the SYN cookies the TCP stack receives.
973 * TcpExtSyncookiesFailed
975 The MSS decoded from the SYN cookie is invalid. When this counter is
976 updated, the received packet won't be treated as a SYN cookie and the
977 TcpExtSyncookiesRecv counter wont be updated.
981 For details of challenge ACK, please refer the explaination of
982 TcpExtTCPACKSkippedChallenge.
984 * TcpExtTCPChallengeACK
986 The number of challenge acks sent.
988 * TcpExtTCPSYNChallenge
990 The number of challenge acks sent in response to SYN packets. After
991 updates this counter, the TCP stack might send a challenge ACK and
992 update the TcpExtTCPChallengeACK counter, or it might also skip to
993 send the challenge and update the TcpExtTCPACKSkippedChallenge.
997 When a socket is under memory pressure, the TCP stack will try to
998 reclaim memory from the receiving queue and out of order queue. One of
999 the reclaiming method is 'collapse', which means allocate a big sbk,
1000 copy the contiguous skbs to the single big skb, and free these
1005 The TCP stack tries to reclaim memory for a socket. After updates this
1006 counter, the TCP stack will try to collapse the out of order queue and
1007 the receiving queue. If the memory is still not enough, the TCP stack
1008 will try to discard packets from the out of order queue (and update the
1009 TcpExtOfoPruned counter)
1013 The TCP stack tries to discard packet on the out of order queue.
1017 After 'collapse' and discard packets from the out of order queue, if
1018 the actually used memory is still larger than the max allowed memory,
1019 this counter will be updated. It means the 'prune' fails.
1021 * TcpExtTCPRcvCollapsed
1023 This counter indicates how many skbs are freed during 'collapse'.
1030 Run the ping command against the public dns server 8.8.8.8::
1032 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1033 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
1034 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
1036 --- 8.8.8.8 ping statistics ---
1037 1 packets transmitted, 1 received, 0% packet loss, time 0ms
1038 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
1042 nstatuser@nstat-a:~$ nstat
1048 IcmpInEchoReps 1 0.0
1051 IcmpMsgInType0 1 0.0
1052 IcmpMsgOutType8 1 0.0
1053 IpExtInOctets 84 0.0
1054 IpExtOutOctets 84 0.0
1055 IpExtInNoECTPkts 1 0.0
1057 The Linux server sent an ICMP Echo packet, so IpOutRequests,
1058 IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
1059 server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
1060 IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
1061 was passed to the ICMP layer via IP layer, so IpInDelivers was
1062 increased 1. The default ping data size is 48, so an ICMP Echo packet
1063 and its corresponding Echo Reply packet are constructed by:
1065 * 14 bytes MAC header
1066 * 20 bytes IP header
1067 * 16 bytes ICMP header
1068 * 48 bytes data (default value of the ping command)
1070 So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
1074 On server side, we run::
1076 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1077 Listening on [0.0.0.0] (family 0, port 9000)
1079 On client side, we run::
1081 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1082 Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
1084 The server listened on tcp 9000 port, the client connected to it, they
1085 completed the 3-way handshake.
1087 On server side, we can find below nstat output::
1089 nstatuser@nstat-b:~$ nstat | grep -i tcp
1090 TcpPassiveOpens 1 0.0
1093 TcpExtTCPPureAcks 1 0.0
1095 On client side, we can find below nstat output::
1097 nstatuser@nstat-a:~$ nstat | grep -i tcp
1098 TcpActiveOpens 1 0.0
1102 When the server received the first SYN, it replied a SYN+ACK, and came into
1103 SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1104 SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
1105 packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
1106 of the 3-way handshake is a pure ACK without data, so
1107 TcpExtTCPPureAcks increased 1.
1109 When the client sent SYN, the client came into the SYN-SENT state, so
1110 TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
1111 ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
1112 1, TcpOutSegs increased 2.
1118 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1119 Listening on [0.0.0.0] (family 0, port 9000)
1123 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1124 Connection to nstat-b 9000 port [tcp/*] succeeded!
1126 Input a string in the nc client ('hello' in our example)::
1128 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1129 Connection to nstat-b 9000 port [tcp/*] succeeded!
1132 The client side nstat output::
1134 nstatuser@nstat-a:~$ nstat
1141 TcpExtTCPPureAcks 1 0.0
1142 TcpExtTCPOrigDataSent 1 0.0
1143 IpExtInOctets 52 0.0
1144 IpExtOutOctets 58 0.0
1145 IpExtInNoECTPkts 1 0.0
1147 The server side nstat output::
1149 nstatuser@nstat-b:~$ nstat
1156 IpExtInOctets 58 0.0
1157 IpExtOutOctets 52 0.0
1158 IpExtInNoECTPkts 1 0.0
1160 Input a string in nc client side again ('world' in our exmaple)::
1162 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1163 Connection to nstat-b 9000 port [tcp/*] succeeded!
1167 Client side nstat output::
1169 nstatuser@nstat-a:~$ nstat
1176 TcpExtTCPHPAcks 1 0.0
1177 TcpExtTCPOrigDataSent 1 0.0
1178 IpExtInOctets 52 0.0
1179 IpExtOutOctets 58 0.0
1180 IpExtInNoECTPkts 1 0.0
1183 Server side nstat output::
1185 nstatuser@nstat-b:~$ nstat
1192 TcpExtTCPHPHits 1 0.0
1193 IpExtInOctets 58 0.0
1194 IpExtOutOctets 52 0.0
1195 IpExtInNoECTPkts 1 0.0
1197 Compare the first client-side nstat and the second client-side nstat,
1198 we could find one difference: the first one had a 'TcpExtTCPPureAcks',
1199 but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1200 nstat and the second server-side nstat had a difference too: the
1201 second server-side nstat had a TcpExtTCPHPHits, but the first
1202 server-side nstat didn't have it. The network traffic patterns were
1203 exactly the same: the client sent a packet to the server, the server
1204 replied an ACK. But kernel handled them in different ways. When the
1205 TCP window scale option is not used, kernel will try to enable fast
1206 path immediately when the connection comes into the established state,
1207 but if the TCP window scale option is used, kernel will disable the
1208 fast path at first, and try to enable it after kerenl receives
1209 packets. We could use the 'ss' command to verify whether the window
1210 scale option is used. e.g. run below command on either server or
1213 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1214 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
1215 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
1216 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
1218 The 'wscale:7,7' means both server and client set the window scale
1219 option to 7. Now we could explain the nstat output in our test:
1221 In the first nstat output of client side, the client sent a packet, server
1222 reply an ACK, when kernel handled this ACK, the fast path was not
1223 enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
1225 In the second nstat output of client side, the client sent a packet again,
1226 and received another ACK from the server, in this time, the fast path is
1227 enabled, and the ACK was qualified for fast path, so it was handled by
1228 the fast path, so this ACK was counted into TcpExtTCPHPAcks.
1230 In the first nstat output of server side, fast path was not enabled,
1231 so there was no 'TcpExtTCPHPHits'.
1233 In the second nstat output of server side, the fast path was enabled,
1234 and the packet received from client qualified for fast path, so it
1235 was counted into 'TcpExtTCPHPHits'.
1237 TcpExtTCPAbortOnClose
1238 ---------------------
1239 On the server side, we run below python script::
1246 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1247 s.bind(('0.0.0.0', port))
1249 sock, addr = s.accept()
1253 This python script listen on 9000 port, but doesn't read anything from
1256 On the client side, we send the string "hello" by nc::
1258 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1260 Then, we come back to the server side, the server has received the "hello"
1261 packet, and the TCP layer has acked this packet, but the application didn't
1262 read it yet. We type Ctrl-C to terminate the server script. Then we
1263 could find TcpExtTCPAbortOnClose increased 1 on the server side::
1265 nstatuser@nstat-b:~$ nstat | grep -i abort
1266 TcpExtTCPAbortOnClose 1 0.0
1268 If we run tcpdump on the server side, we could find the server sent a
1269 RST after we type Ctrl-C.
1271 TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
1272 ---------------------------------------------------
1273 Below is an example which let the orphan socket count be higher than
1274 net.ipv4.tcp_max_orphans.
1275 Change tcp_max_orphans to a smaller value on client::
1277 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1279 Client code (create 64 connection to server)::
1281 nstatuser@nstat-a:~$ cat client_orphan.py
1285 server = 'nstat-b' # server address
1290 connection_list = []
1293 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1294 s.connect((server, port))
1295 connection_list.append(s)
1296 print("connection_count: %d" % len(connection_list))
1301 Server code (accept 64 connection from client)::
1303 nstatuser@nstat-b:~$ cat server_orphan.py
1310 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1311 s.bind(('0.0.0.0', port))
1313 connection_list = []
1315 sock, addr = s.accept()
1316 connection_list.append((sock, addr))
1317 print("connection_count: %d" % len(connection_list))
1319 Run the python scripts on server and client.
1323 python3 server_orphan.py
1327 python3 client_orphan.py
1329 Run iptables on server::
1331 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1333 Type Ctrl-C on client, stop client_orphan.py.
1335 Check TcpExtTCPAbortOnMemory on client::
1337 nstatuser@nstat-a:~$ nstat | grep -i abort
1338 TcpExtTCPAbortOnMemory 54 0.0
1340 Check orphane socket count on client::
1342 nstatuser@nstat-a:~$ ss -s
1343 Total: 131 (kernel 0)
1344 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
1346 Transport Total IP IPv6
1354 The explanation of the test: after run server_orphan.py and
1355 client_orphan.py, we set up 64 connections between server and
1356 client. Run the iptables command, the server will drop all packets from
1357 the client, type Ctrl-C on client_orphan.py, the system of the client
1358 would try to close these connections, and before they are closed
1359 gracefully, these connections became orphan sockets. As the iptables
1360 of the server blocked packets from the client, the server won't receive fin
1361 from the client, so all connection on clients would be stuck on FIN_WAIT_1
1362 stage, so they will keep as orphan sockets until timeout. We have echo
1363 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
1364 only keep 10 orphan sockets, for all other orphan sockets, the client
1365 system sent RST for them and delete them. We have 64 connections, so
1366 the 'ss -s' command shows the system has 10 orphan sockets, and the
1367 value of TcpExtTCPAbortOnMemory was 54.
1369 An additional explanation about orphan socket count: You could find the
1370 exactly orphan socket count by the 'ss -s' command, but when kernel
1371 decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
1372 doesn't always check the exactly orphan socket count. For increasing
1373 performance, kernel checks an approximate count firstly, if the
1374 approximate count is more than tcp_max_orphans, kernel checks the
1375 exact count again. So if the approximate count is less than
1376 tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
1377 would find TcpExtTCPAbortOnMemory is not increased at all. If
1378 tcp_max_orphans is large enough, it won't occur, but if you decrease
1379 tcp_max_orphans to a small value like our test, you might find this
1380 issue. So in our test, the client set up 64 connections although the
1381 tcp_max_orphans is 10. If the client only set up 11 connections, we
1382 can't find the change of TcpExtTCPAbortOnMemory.
1384 Continue the previous test, we wait for several minutes. Because of the
1385 iptables on the server blocked the traffic, the server wouldn't receive
1386 fin, and all the client's orphan sockets would timeout on the
1387 FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
1388 10 timeout on the client::
1390 nstatuser@nstat-a:~$ nstat | grep -i abort
1391 TcpExtTCPAbortOnTimeout 10 0.0
1393 TcpExtTCPAbortOnLinger
1394 ----------------------
1395 The server side code::
1397 nstatuser@nstat-b:~$ cat server_linger.py
1403 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1404 s.bind(('0.0.0.0', port))
1406 sock, addr = s.accept()
1410 The client side code::
1412 nstatuser@nstat-a:~$ cat client_linger.py
1416 server = 'nstat-b' # server address
1419 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1420 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
1421 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1422 s.connect((server, port))
1425 Run server_linger.py on server::
1427 nstatuser@nstat-b:~$ python3 server_linger.py
1429 Run client_linger.py on client::
1431 nstatuser@nstat-a:~$ python3 client_linger.py
1433 After run client_linger.py, check the output of nstat::
1435 nstatuser@nstat-a:~$ nstat | grep -i abort
1436 TcpExtTCPAbortOnLinger 1 0.0
1438 TcpExtTCPRcvCoalesce
1439 --------------------
1440 On the server, we run a program which listen on TCP port 9000, but
1441 doesn't read any data::
1446 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1447 s.bind(('0.0.0.0', port))
1449 sock, addr = s.accept()
1453 Save the above code as server_coalesce.py, and run::
1455 python3 server_coalesce.py
1457 On the client, save below code as client_coalesce.py::
1462 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1463 s.connect((server, port))
1467 nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1469 We use '-i' to come into the interactive mode, then a packet::
1474 Send a packet again::
1479 On the server, run nstat::
1481 ubuntu@nstat-b:~$ nstat
1488 TcpExtTCPRcvCoalesce 1 0.0
1489 IpExtInOctets 110 0.0
1490 IpExtOutOctets 104 0.0
1491 IpExtInNoECTPkts 2 0.0
1493 The client sent two packets, server didn't read any data. When
1494 the second packet arrived at server, the first packet was still in
1495 the receiving queue. So the TCP layer merged the two packets, and we
1496 could find the TcpExtTCPRcvCoalesce increased 1.
1498 TcpExtListenOverflows and TcpExtListenDrops
1499 -------------------------------------------
1500 On server, run the nc command, listen on port 9000::
1502 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1503 Listening on [0.0.0.0] (family 0, port 9000)
1505 On client, run 3 nc commands in different terminals::
1507 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1508 Connection to nstat-b 9000 port [tcp/*] succeeded!
1510 The nc command only accepts 1 connection, and the accept queue length
1511 is 1. On current linux implementation, set queue length to n means the
1512 actual queue length is n+1. Now we create 3 connections, 1 is accepted
1513 by nc, 2 in accepted queue, so the accept queue is full.
1515 Before running the 4th nc, we clean the nstat history on the server::
1517 nstatuser@nstat-b:~$ nstat -n
1519 Run the 4th nc on the client::
1521 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1523 If the nc server is running on kernel 4.10 or higher version, you
1524 won't see the "Connection to ... succeeded!" string, because kernel
1525 will drop the SYN if the accept queue is full. If the nc client is running
1526 on an old kernel, you would see that the connection is succeeded,
1527 because kernel would complete the 3 way handshake and keep the socket
1528 on half open queue. I did the test on kernel 4.15. Below is the nstat
1531 nstatuser@nstat-b:~$ nstat
1536 TcpExtListenOverflows 4 0.0
1537 TcpExtListenDrops 4 0.0
1538 IpExtInOctets 240 0.0
1539 IpExtInNoECTPkts 4 0.0
1541 Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
1542 between the 4th nc and the nstat was longer, the value of
1543 TcpExtListenOverflows and TcpExtListenDrops would be larger, because
1544 the SYN of the 4th nc was dropped, the client was retrying.
1546 IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
1547 -------------------------------------------------
1548 server A IP address: 192.168.122.250
1549 server B IP address: 192.168.122.251
1550 Prepare on server A, add a route to server B::
1552 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
1554 Prepare on server B, disable send_redirects for all interfaces::
1556 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1557 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1558 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1559 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1561 We want to let sever A send a packet to 8.8.8.8, and route the packet
1562 to server B. When server B receives such packet, it might send a ICMP
1563 Redirect message to server A, set send_redirects to 0 will disable
1566 First, generate InAddrErrors. On server B, we disable IP forwarding::
1568 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1570 On server A, we send packets to 8.8.8.8::
1574 On server B, we check the output of nstat::
1579 IpInAddrErrors 3 0.0
1580 IpExtInOctets 180 0.0
1581 IpExtInNoECTPkts 3 0.0
1583 As we have let server A route 8.8.8.8 to server B, and we disabled IP
1584 forwarding on server B, Server A sent packets to server B, then server B
1585 dropped packets and increased IpInAddrErrors. As the nc command would
1586 re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1587 multiple IpInAddrErrors.
1589 Second, generate IpExtInNoRoutes. On server B, we enable IP
1592 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1594 Check the route table of server B and remove the default route::
1597 default via 192.168.122.1 dev ens3 proto static
1598 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
1599 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
1601 On server A, we contact 8.8.8.8 again::
1604 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
1606 On server B, run nstat::
1613 IcmpOutDestUnreachs 1 0.0
1614 IcmpMsgOutType3 1 0.0
1615 IpExtInNoRoutes 1 0.0
1616 IpExtInOctets 60 0.0
1617 IpExtOutOctets 88 0.0
1618 IpExtInNoECTPkts 1 0.0
1620 We enabled IP forwarding on server B, when server B received a packet
1621 which destination IP address is 8.8.8.8, server B will try to forward
1622 this packet. We have deleted the default route, there was no route for
1623 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
1624 Destination Unreachable" message to server A.
1626 Third, generate IpOutNoRoutes. Run ping command on server B::
1629 connect: Network is unreachable
1631 Run nstat on server B::
1637 We have deleted the default route on server B. Server B couldn't find
1638 a route for the 8.8.8.8 IP address, so server B increased
1641 TcpExtTCPACKSkippedSynRecv
1642 --------------------------
1643 In this test, we send 3 same SYN packets from client to server. The
1644 first SYN will let server create a socket, set it to Syn-Recv status,
1645 and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
1646 again, and record the reply time (the duplicate ACK reply time). The
1647 third SYN will let server check the previous duplicate ACK reply time,
1648 and decide to skip the duplicate ACK, then increase the
1649 TcpExtTCPACKSkippedSynRecv counter.
1651 Run tcpdump to capture a SYN packet::
1653 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1654 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1656 Open another terminal, run nc command::
1658 nstatuser@nstat-a:~$ nc nstat-b 9000
1660 As the nstat-b didn't listen on port 9000, it should reply a RST, and
1661 the nc command exited immediately. It was enough for the tcpdump
1662 command to capture a SYN packet. A linux server might use hardware
1663 offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
1664 might be not correct. We call tcprewrite to fix it::
1666 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1668 On nstat-b, we run nc to listen on port 9000::
1670 nstatuser@nstat-b:~$ nc -lkv 9000
1671 Listening on [0.0.0.0] (family 0, port 9000)
1673 On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1676 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1678 Send 3 SYN repeatly to nstat-b::
1680 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1682 Check snmp cunter on nstat-b::
1684 nstatuser@nstat-b:~$ nstat | grep -i skip
1685 TcpExtTCPACKSkippedSynRecv 1 0.0
1687 As we expected, TcpExtTCPACKSkippedSynRecv is 1.
1689 TcpExtTCPACKSkippedPAWS
1690 -----------------------
1691 To trigger PAWS, we could send an old SYN.
1693 On nstat-b, let nc listen on port 9000::
1695 nstatuser@nstat-b:~$ nc -lkv 9000
1696 Listening on [0.0.0.0] (family 0, port 9000)
1698 On nstat-a, run tcpdump to capture a SYN::
1700 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1701 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1703 On nstat-a, run nc as a client to connect nstat-b::
1705 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1706 Connection to nstat-b 9000 port [tcp/*] succeeded!
1708 Now the tcpdump has captured the SYN and exit. We should fix the
1711 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1713 Send the SYN packet twice::
1715 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1717 On nstat-b, check the snmp counter::
1719 nstatuser@nstat-b:~$ nstat | grep -i skip
1720 TcpExtTCPACKSkippedPAWS 1 0.0
1722 We sent two SYN via tcpreplay, both of them would let PAWS check
1723 failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1724 for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
1726 TcpExtTCPACKSkippedSeq
1727 ----------------------
1728 To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
1729 timestamp (to pass PAWS check) but the sequence number is out of
1730 window. The linux TCP stack would avoid to skip if the packet has
1731 data, so we need a pure ACK packet. To generate such a packet, we
1732 could create two sockets: one on port 9000, another on port 9001. Then
1733 we capture an ACK on port 9001, change the source/destination port
1734 numbers to match the port 9000 socket. Then we could trigger
1735 TcpExtTCPACKSkippedSeq via this packet.
1737 On nstat-b, open two terminals, run two nc commands to listen on both
1738 port 9000 and port 9001::
1740 nstatuser@nstat-b:~$ nc -lkv 9000
1741 Listening on [0.0.0.0] (family 0, port 9000)
1743 nstatuser@nstat-b:~$ nc -lkv 9001
1744 Listening on [0.0.0.0] (family 0, port 9001)
1746 On nstat-a, run two nc clients::
1748 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1749 Connection to nstat-b 9000 port [tcp/*] succeeded!
1751 nstatuser@nstat-a:~$ nc -v nstat-b 9001
1752 Connection to nstat-b 9001 port [tcp/*] succeeded!
1754 On nstat-a, run tcpdump to capture an ACK::
1756 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1757 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1759 On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1760 string 'foo' in our example::
1762 nstatuser@nstat-b:~$ nc -lkv 9001
1763 Listening on [0.0.0.0] (family 0, port 9001)
1764 Connection from nstat-a 42132 received!
1767 On nstat-a, the tcpdump should have caputred the ACK. We should check
1768 the source port numbers of the two nc clients::
1770 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1771 State Recv-Q Send-Q Local Address:Port Peer Address:Port
1772 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000
1773 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001
1775 Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to
1778 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
1780 Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1782 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1784 Check TcpExtTCPACKSkippedSeq on nstat-b::
1786 nstatuser@nstat-b:~$ nstat | grep -i skip
1787 TcpExtTCPACKSkippedSeq 1 0.0