Thursday, September 22, 2011

Redundant Interconnect Usage in 11G R2 RAC

We have thought of implementing this new feature with Oracle 11G as we have frequent private NIC down events which further results in node evictions.

Redundant Interconnect without any 3rd-party IP failover technology (bond, IPMP or similar) is supported natively by Grid Infrastructure starting from  Multiple private network adapters can be defined either during the installation phase or afterward using the oifcfg.  Oracle Database, CSS, OCR, CRS, CTSS, and EVM components in employ it automatically.

Grid Infrastructure can activate a maximum of four private network adapters at a time even if more are defined. The ora.cluster_interconnect.haip resource will start one to four link local  HAIP on private network adapters for interconnect communication for Oracle RAC, Oracle ASM, and Oracle ACFS etc.

Grid automatically picks link local addresses from reserved 169.254.*.* subnet for HAIP, and it will not attempt to use any 169.254.*.* address if it's already in use for another purpose. With HAIP, by default, interconnect traffic will be load balanced across all active interconnect interfaces, and corresponding HAIP address will be failed over transparently to other adapters if one fails or becomes non-communicative. .

The number of HAIP addresses is decided by how many private network adapters are active when Grid comes up on the first node in the cluster .  If there's only one active private network, Grid will create one; if two, Grid will create two; and if more than two, Grid will create four HAIPs. The number of HAIPs won't change even if more private network adapters are activated later, a restart of clusterware on all nodes is required for new adapters to become effective

New HAIP reource

NAME           TARGET  STATE        SERVER                   STATE_DETAILS
Cluster Resources
      1        ONLINE  ONLINE       ibs-ash-sr118        Started
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118        OBSERVER
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
      1        ONLINE  ONLINE       ibs-ash-sr118
While in previous releases bonding, trunking, teaming, or similar technology was required to make use of redundant network connections between the nodes to be used as redundant,dedicated, private communication channels or “interconnect”, Oracle Clusterware now provides an integrated solution to ensure “Redundant Interconnect Usage”. This functionality is available starting with Oracle Database 11g Release 2, Patch Set One (

The Redundant Interconnect Usage feature does not operate on the network interfaces directly. Instead, it is based on a multiple-listening-endpoint architecture, in which a highly available virtual IP (the HAIP) is assigned to each private network (up to a total number of 4 interfaces).By default, Oracle Real Application Clusters (RAC) software uses all of the HAIP addresses for private network communication, providing load balancing across the set of interfaces identified as the private network. If a private interconnect interface fails or becomes non-communicative,then Oracle Clusterware transparently moves the corresponding HAIP address to one of the remaining functional interfaces

Oracle RAC Databases, Oracle Automatic Storage Management (clustered ASM), and Oracle Clusterware components such as CSS, OCR, CRS, CTSS, and EVM components employ
Redundant Interconnect Usage starting with Oracle Database 11g Release 2, Patch Set One (


# $GRID_HOME/bin/oifcfg getif
eth0 global public
eth1 global cluster_interconnect

The interfaces that are currently stored in the GPnP profile, their subnets, and their role (public or cluster_interconnect) are displayed.

2. Add the remaining LLT links to the GPnP profile:
# $GRID_HOME/bin/oifcfg setif -global \

3. Verify that the correct interface subnet is in use:
# $GRID_HOME/bin/oifcfg getif
eth0 global public
eth1 global cluster_interconnect
eth2 global cluster_interconnect

4.You must restart Oracle Clusterware on all members of the cluster when you make global changes. For local changes, you only need to perform a node restart.

Interconnect changes for the database occur at instance startup. However, the interconnect for Oracle Clusterware might be different.

11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]
How to Modify Private Network Interface in 11.2 Grid Infrastructure [ID 1073502.1]
How to Change Interconnect/Public Network (Interface or Subnet) in Oracle Clusterware [ID 283684.1]
[11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]]
Known Issues

A few bug fixes for haip has come with


Post a Comment


ORA-BLOG. Copyright 2008 All Rights Reserved Revolution Two Church theme by Brian Gardner Converted into Blogger Template by Bloganol dot com