Tuning the Cluster Interconnect - Expert Oracle RAC Performance Diagnostics and Tuning - page 466

Low MTU size causes fragmentation of data, which indicates that data is split into several smaller packets, and
it takes several transmissions to complete the operation. The following output is from IPTraf (network monitoring
utility on Linux) and illustrates fragmentation of data transmitted on an Ethernet configuration without jumbo frames:

Fragmented packets, on reaching the destination, need to be reassembled. High CPU utilization and inadequate
UDP buffer space can cause packet reassembly failures. Also for the reassemble to complete successfully, these
packets have to be available until such an operation completes. Depending on the availability of resources and the
load on the network, this may or may not be successful. Packets that are not reassembled are dropped and requested
again. All this adds to the network and user response delays.
Increasing the fragment reassembly buffers, allocating more space for reassembly, and increasing the time
required to reassemble fragmented packets may be required. The reassembly buffer space is controlled by the
following parameters. Its default values are

[oracle@prddb1 ~]# cat /proc/sys/net/ipv4/ipfrag_low_thresh
196608
[oracle@prddb1 ~]# cat /proc/sys/net/ipv4/ipfrag_high_thresh
262144
[oracle@prddb1 ~]# cat /proc/sys/net/ipv4/ipfrag_time
30

•

ipfrag_high_thresh is used set the maximum memory used to reassemble IP fragments.
When ipfrag_high_thresh bytes of memory are allocated for this purpose, the fragment
handler will toss packets until ipfrag_low_thresh is reached.

•

ipfrag_low_thresh is used to set the maximum memory used to reassemble IP fragments
before the kernel begins to remove incomplete fragment queues to free up resources. The
kernel still accepts new fragments for defragmentation.

•

ipfrag_time is used to set the interval required to reassemble before the reassembly will fail
or time out.

MTU definitions do not include the data-link header. However, packet size computations include data-link
headers. Maximum packet size displayed by the various tools is the MTU plus the data-link header length. To get the
maximum benefit from the interconnect, the MTU should be configured to the highest possible value supported.
For example, a setting as high as 9K using jumbo frames would help in improved interconnect bandwidth and data
transmission.