Database Reference
In-Depth Information
Figure
9-5
shows how the scheme of fragmentation and reassembly works. In this figure, LMS process sends
a block of 8KB to a remote process. A buffer of 8KB is fragmented to six IP packets, and these IP packets are sent
to the receiving side through the switch. A kernel thread in the receiving side reassembles these six IP packets and
reconstructs the 8KB buffer. The foreground process reads the block from socket buffers to its Program Global Area
(PGA) and copies the buffer to database buffer cache.
8
FG
LMS
8K
buffer
8K
buffer
switch
Reassembly
Fragmentation
Figure 9-5.
8K transfer without Jumbo Frames
Fragmentation and reassembly are performed in memory. Excessive fragmentation and reassembly issues can
lead to higher CPU usage (accounted to kernel mode CPU usage) in the database nodes.
In a nutshell, Jumbo Frame is an MTU configuration exceeding 1,500 bytes. There is no IEEE standard for MTU
of Jumbo Frame configuration, but the industry standard for Jumbo Frame MTU size is 9,000 bytes, with an Ethernet
frame on the wire of 9,018 bytes. If the MTU size is set to 9,000, then fragmentation or reassembly is not required.
Segments can be transferred from one node to another node without fragmentation or reassembly. As an 8K buffer
can fit in a 9,000-byte network buffer, block can be transmitted from one node to another node without fragmentation.
However, the path also must support the MTU of 9,000.
Output of the ifconfig command shows that MTU of the eth3 interface is set to 9,000. The maximum size of an IP
packet that can flow through this network interface is 9,000 bytes. An MTU setting of 9,000 is known as Jumbo Frames.
$ifconfig -a |more
...
eth3:1: ...<UP,BROADCAST,RUNNING,MULTICAST,IPv4 >
mtu 9000
index 4
inet 169.254.28.111 netmask ffffc000 broadcast 169.254.63.255
...
MTU configuration must be consistent in all nodes of a cluster and the switch also should support Jumbo Frames.
Many network administrators do not want to modify the switch configuration to support Jumbo Frames. Worse, another
network administrator might change the setting back to the default of 1,500 in the switch, inducing an unintended
Clusterware restart. In essence, setting and maintaining Jumbo Frames is a multidisciplinary function involving system
administrators and network administrators.
Still, if your database block size is 16,384 and if path MTU is configured as 9,000, then a 16K block transfer would
require a fragmentation and reassembly of two IP packets. This minor fragmentation and reassembly is acceptable,
as increasing Jumbo Frames beyond 9,000 is not supported by many switches and kernel modules at this time.
8
Not all buffers are copied to buffer cache. For example, parallel query copies buffers to its PGA.
