What if php-fpm on centos takes up too much socket with the state of CLOSED and does not release it?

[temp@xigua ~]$ ss -s 
Total: 83678 (kernel 0)
TCP:   84982 (estab 127, closed 84812, orphaned 0, synrecv 0, timewait 1485/0), ports 0

Transport Total     IP        IPv6
*         0         -         -        
RAW       0         0         0        
UDP       4         3         1        
TCP       170       168       2        
INET      174       171       3        
FRAG      0         0         0


here you can see that the socket with CLOSED status has 8w + (closed 84812) , and look at a fpm:

.
[temp@xigua ~]$ ss -s $ lsof -p ${fpm-pid}|grep "protocol: TCP"|wc -l   
1043

there are a total of 80 fpm processes, so it is almost certain that all of the above-mentioned CLOSED socket is caused by fpm. The main reason is that I do not understand why fpm
does not release these socket. Shouldn"t the socket in CLOSED state be reclaimed by the system in an instant?
these socket with CLOSED status still take up more memory. When the upper limit of tcp (cat / proc/sys/net/ipv4/tcp_mem) is exceeded, there will be an error: out of memory-- consider tuning tcp_mem
what can I do to solve it further?

< hr >

P.S. 17:53:34, February 26, 2019 further information addition:

$ ls -l /proc/${pfm}/fd|wc -l
1059
< H2 > check sockstat: < / H2 >
cat /proc/net/sockstat 
sockets: used 84442
TCP: inuse 168 orphan 0 tw 3052 alloc 84262 mem 84093
UDP: inuse 3 mem 1
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0


$ netstat -nat | awk "{print $6}" | sort | uniq -c | sort -n

  1 established)
  1 Foreign
  8 LISTEN
 28 CLOSE_WAIT
138 ESTABLISHED
782 TIME_WAIT

    
    
< H2 > configuration of fpm: < / H2 >
pm.max_children = 80
pm.max_requests = 100000
< H2 > before a restart < / H2 >
$ cat /proc/net/sockstat
sockets: used 29901
TCP: inuse 168 orphan 0 tw 3618 alloc 29720 mem 29563
UDP: inuse 3 mem 1
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7821         824         250         418        6746        6064
Swap:           511          98         413
< H2 > after restart < / H2 >
$ cat /proc/net/sockstat  
sockets: used 387
TCP: inuse 167 orphan 0 tw 3504 alloc 207 mem 42
UDP: inuse 3 mem 1
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0   

 
 
$ lsof -p ${fpm-pid}|less     
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
php-fpm 17699 temp 71u sock 0,7 0t0 552415632 protocol: TCP
php-fpm 17699 temp 72u sock 0,7 0t0 552428529 protocol: TCP
php-fpm 17699 temp 73u sock 0,7 0t0 552450479 protocol: TCP
php-fpm 17699 temp 74u sock 0,7 0t0 552462900 protocol: TCP
php-fpm 17699 temp 75u sock 0,7 0t0 552523600 protocol: TCP
php-fpm 17699 temp 76u sock 0,7 0t0 552542925 protocol: TCP
php-fpm 17699 temp 77u sock 0,7 0t0 552613133 protocol: TCP
php-fpm 17699 temp 78u sock 0,7 0t0 552639299 protocol: TCP
php-fpm 17699 temp 79u sock 0,7 0t0 552646395 protocol: TCP
php-fpm 17699 temp 80u sock 0,7 0t0 552659030 protocol: TCP
    

found a large number of words "protocol: TCP". What does this mean? Why not show the double-ended ip and port information of the tcp connection and omit the word "TCP" instead?


closed status should be closed. From your point of view, it is estimated that socket has been closed, but the protocol stack has not released resources?
you use ls-l / proc/fpm process ID/fd | wc-l look at


is there a situation in which multiple processes share the same socket in your php-fpm?

closed is not closed, all I can think of is that kernel sock- > refcnt is not zero, so it will not release

.

cat / proc/net/tcp | awk'{print $11} 'can see the reference count of tcp socket


can you provide more information, such as

  1. the maximum number of files for the fpm process. cat / proc/$ (pidof fpm) / limits
  2. client and server address information for CLOSED status socket. lsof-n-p $(pidof fpm)
  3. whether the php application contains sockets operations, and if so, post the address of the related service.
  4. whether the connection in CLOSED state always exists without restarting the fpm process, and whether it is entered by TIMEWAIT.
  5. restart the initial phase of fpm, whether the number of TIMEWAIT is greater than the number of CLOSED.
  6. kernel parameters. sysctl-a | grep tcp

feels the need to optimize TCP.

< hr >

according to the latest information provided by the subject

cat /proc/net/sockstat 
sockets: used 84442
TCP: inuse 168 orphan 0 tw 3052 alloc 84262 mem 84093
UDP: inuse 3 mem 1
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

  1 established)
  1 Foreign
  8 LISTEN
 28 CLOSE_WAIT
138 ESTABLISHED
782 TIME_WAIT

I'm pretty sure this is a tcp optimization problem, which usually occurs on high-load servers, where the tcp creation speed (user mode) is much higher than the destruction speed (kernel).
you need to adjust the following kernel parameters as appropriate

net.ipv4.ip_local_port_range
net.ipv4.tcp_fin_timeout
net.ipv4.tcp_tw_reuse

is it possible that php-fpm is configured with static mode? Php-fpm does not release the process when pm = static is configured.


Hello, have you located this question? I have the same problem here, and there is no way to analyze it at present


Hello, I have also encountered the problem that this kind of socket in closed state has not been released. Have you solved this problem? What is the way to solve the problem?

Menu