listener报错HPUX Error: 233: No buffer space available

现象描述：每天业务连接高峰期listener.log出现报错，但业务无明显异常.平台是oracle 9.2.0.8 + hpux 11.23 IA
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
HPUX Error: 233: No buffer space available
经咨询HP工程师和oracle sr，原因为内存不足或者内核参数设置不合理。本主机内存380G，使用率60%，应该不存在内存不足的问题，初步定为内核参数设置；
kmeminfo
Buffer cache = 7037169 26.8g 7% details with -bufcache

[root@crmdb2:/crmdata#]kctune|grep dbc
dbc_max_pct 7 7 Immed
dbc_min_pct 5 Default Immed


? The underlying problem of these errors is the operating system.
? The HP UX system file /usr/include/sys/errno.h shows the following information
? for the OS error 233:
?
? # define ENOBUFS 233 /* No buffer space available */
?
? ENOBUFS means there is no sufficient memory available and the system(kernel)
? can not allocate any more.
? Application will usually retry the operation when it detects this error from a
? system call since it indicates there is a transient resource shortage.
? It is the Operating system that refuses the resource request from the listener.
? The virtual memory allocation routine of the OS will determine if a swap can be
? made to disk of a real memory segment thereby allowing the listener access to
? some more real memory

Oracle回复，可能是内核tcp_conn_request_max设置较小导致。

Oracle sr回复：

Hi customer,

1. HPUX Error: 233 is an HP O/S error is typically linked to memory and swap.

Did you check the memory and swap utilization when the issue happened?

2. We've seen a number of cases where this error can be caused by a low value of tcp_conn_request_max.

Please confirm about the value of the following " tcp_conn_request_max" with your sysadmin and confirm that it is as required 10240.

3. One-off occurrences of HP-UX 233 can be ignored and can be caused by timing issues during the 3-way handshake to establish the TCP connection on HPUX. Currently the listener just reports the error, and does not block.
No Valid connection is lost.

关于参数TCP_CONN_REQUEST_MAX的具体含义，但从netstat看，却明显没有因为该参数设置导致的tcp连接问题，令人遗憾

P-UX 11i TCP_CONN_REQUEST_MAX

Description: Specifies the maximum number of connection requests that the operating system can queue when the server does not have available threads. When high connection rates occur, a large backlog of TCP/IP connection requests builds up and client connections are dropped. Adjust this setting when clients start to time out after waiting to connect. Verify this situation by issuing the netstat -p tcp command. Look for the following value: connect requests dropped due to full queue
How to view or set: Set this parameter by using the ndd -set /dev/tcp tcp_conn_request_max 8192 command.
Default value: 4096
Recommended value: In most cases the default is sufficient. Consider adjusting this value to 8192, if the default proves inadequate.

crmdb2[/tmp]netstat -p tcp
tcp:
2276246944 packets sent
4140476808 data packets (4226717825 bytes)
99244862 data packets (1684980992 bytes) retransmitted
2136121689 ack-only packets (3450143246 delayed)
350 URG only packets
92215 window probe packets
993200 window update packets
294688412 control packets
942156182 packets received
2727143257 acks (for 3082426485 bytes)
150534360 duplicate acks
0 acks for unsent data
1273908342 packets (3261709649 bytes) received in-sequence
87 completely duplicate packets (108705 bytes)
167017 packets with some dup, data (233293514 bytes duped)
10962410 out of order packets (855066444 bytes)
64 packets (348119886 bytes) of data after window
1276460 window probes
2778087255 window update packets
3792 packets received after close
5093 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
6354564 connection requests
137037980 connection accepts
143392544 connections established (including accepts)
145193664 connections closed (including 1808075 drops)
1116496 embryonic connections dropped
2492425196 segments updated rtt (of 2492425196 attempts)
22433817 retransmit timeouts
121232 connections dropped by rexmit timeout
92215 persist timeouts
3538277 keepalive timeouts
3526225 keepalive probes sent
4182 connections dropped by keepalive
0 connect requests dropped due to full queue
1636314 connect requests dropped due to no listener
0 suspect connect requests dropped due to aging
0 suspect connect requests dropped due to rate

修改参数的方法倒是比较简单，并且可在线修改，经过评估，已在主机进行修改，观察下效果
修改方法是：
ndd -set /dev/tcp tcp_conn_request_max 10240
在线修改，无需重新启动应用和网络服务。
同时修改以下配置文件/etc/rc.config.d/nddconf：
TRANSPORT_NAME[1]=tcp
NDD_NAME[1]= tcp_conn_request_max
NDD_VALUE[1]=10240