The error occurred when switching from MAXIMUM PERFORMANCE to MAXIMUM AVAILABILITY Data Guard mode on an Exadata system. The primary RAC database, running on version 19c, crashed with an ORA-600 error code. The error mentioned that the redo log for group 45, sequence 509, was incorrectly located on DAX storage, causing issues. The incident details pointed to an internal error code [kfk_iodone_invalid_buffer], indicating that an I/O buffer did not pass HARD checks. The error was associated with the LGWR background process.

Changes
The issue arose after changing the Data Guard mode to MAXIMUM AVAILABILITY; switching to MAXIMUM PROTECTION did not trigger the ORA-600 error. Interestingly, a 12c database within the same Exadata rack did not experience the ORA-600 error. The problem manifested when the online or standby redo log files were located on Extreme Flash Cell or High Capacity Cell nodes. The system applied the following relevant patches: Patch 30165493, fixing log file fast sync parameters for PMEMLOG, Bug 31119057 associated with the ORA-600 error, and Bug 31305624 linked to instance crashes.

Solution
When transitioning to MAXIMUM AVAILABILITY Data Guard mode on Exadata, specific steps must be taken to resolve the ORA-600 [kfk_iodone_invalid_buffer] error.

Firstly, dynamic parameters must be set on all database instances: ‘_smart_log_threshold_usec‘ should be set to 0.

For Exadata systems with PMEM, ‘_exa_pmemlog_threshold_usec‘ must also be set to 0.

Furthermore, it is necessary to download and implement patch 31305624 from support.oracle.com.

Lastly, updating the system to version 19.6.0.0.200114DBRU or a higher version that includes the bug fix is part of the solution.

Oracle Data Guard is a powerful, high-availability solution that ensures Oracle databases’ continuous availability and protection. Optimizing Oracle Net, which handles the network communication between the primary and standby systems, is essential to achieve optimal performance. This article will focus on optimizing Oracle Net for Data Guard by adjusting the TCP send and receive socket buffers, modifying session data unit (SDU) size, and disabling the TCP Nagle algorithm.

To achieve high network throughput, it is recommended to set the TCP send and receive socket buffers to the bandwidth-delay product (BDP) of the network link between the primary and standby systems. The BDP is calculated as the product of the network bandwidth and latency. For example, if the bandwidth is 622 Mbit/s per second and the latency is 30 ms, the minimum recommended size for the TCP socket buffers would be 6,997,500 bytes. This value can be set using the Oracle Net parameters RECV_BUF_SIZE and SEND_BUF_SIZE.

Oracle Net buffers data into session data units (SDUs) before sending them to the network layer. By default, the SDU size is set to 8192 bytes, which might be insufficient for Data Guard operations that involve sending redo data in larger chunks. To improve performance, increasing the SDU size to 64 KB is recommended. This ensures that Data Guard can send larger pieces of redo data without chopping them up and causing additional overhead.

To preempt delays in buffer flushing in the TCP protocol stack, it is advised to disable the TCP Nagle algorithm. This can be achieved by setting TCP.NODELAY to YES in the sqlnet.ora file on both the primary and standby systems. Disabling the Nagle algorithm helps reduce latency and improve the network communication’s responsiveness.

It is important to note that the actual values of the send and receive socket buffer sizes may be limited by the host operating system or memory constraints. The default values for these parameters are specific to the operating system. On Linux, for example, the default values are 128 KB for SEND_BUF_SIZE and 174,700 bytes for RECV_BUF_SIZE. These default values are often modified to higher when the oracle-database-preinstall-19c package is installed, but they are still lower than the recommended values for Data Guard.

To determine the default and maximum amount of receive and send socket memory, you can use the following commands on the Linux operating system:
– To check the receive socket memory:
# cat /proc/sys/net/core/rmem_default
# cat /proc/sys/net/core/rmem_max
– To check the send socket memory:
# cat /proc/sys/net/core/wmem_default
# cat /proc/sys/net/core/wmem_max

If the default values are lower than recommended, you can modify the /etc/sysctl.conf file. For example, you can set the maximum send and receive socket memory to 10 MB (10485760 bytes) with the following commands:
# echo ‘net.core.wmem_max=10485760’ >> /etc/sysctl.conf
# echo ‘net.core.rmem_max=10485760’ >> /etc/sysctl.conf

Additionally, you must set the socket memory’s minimum, initial, and maximum sizes in bytes. Use the following commands:
# echo ‘net.ipv4.tcp_rmem=10240 131072 10485760’ >> /etc/sysctl.conf
# echo ‘net.ipv4.tcp_wmem=10240 131072 10485760’ >> /etc/sysctl.conf

Once the changes are made, reload the modified settings with the command:
# sysctl -p

It is worth mentioning that starting from Oracle Database 12c Release 1 (12.1), Oracle Net supports larger session data unit (SDU) sizes, with an upper limit of 2 MB. The larger SDU size can benefit networks with high bandwidth-delay products and ample host resources, allowing for better utilization of the available network bandwidth. However, the recommended SDU size for Data Guard is 64 KB. This recommendation is subject to change based on ongoing testing and future advancements.

By optimizing Oracle Net for Data Guard, you can significantly improve the performance and efficiency of your Data Guard configuration. Implementing the recommended settings for TCP socket buffers SDU size, and disabling the TCP Nagle algorithm will ensure smoother and faster network communication between the primary and standby systems, ultimately enhancing your Oracle databases’ overall reliability and availability.

For more information, see:

https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/index.html

https://education.oracle.com/oracle-database-19c-data-guard-administration-workshop/courP_86809290