Infiniband HOWTO

Guy Coates

1. Introduction

• 1.1 The latest version
• 1.2 What is OFED?

2. Installing the OFED Software

• 2.1 Installing prebuilt packages
• 2.2 Building packages from source

3. Install the kernel modules

• 3.1 Building new kernel modules

4. Setting up a basic infiniband network

• 4.1 Upgrade your Infiniband card and switch firmware
• 4.2 Physically Connect the network
• 4.3 Choose a Subnet Manager
• 4.4 Load the kernel modules
• 4.5 (optional) Start opensm
• 4.6 Check network health
• 4.7 Check the extended network connectivity
• 4.8 testing connectivity with ibping
• 4.9 Testing RDMA performance

5. IP over Infiniband (IPoIB)

• 5.1 List the network devices
• 5.2 IP Configuration
• 5.3 Connected vs Unconnected Mode
• 5.4 TCP tuning
• 5.5 ARP and dual ported cards

6. OpenMPI

• 6.1 Configure IPoIB
• 6.2 Load the modules
• 6.3 Check permissions and limits
• 6.4 Install the mpi test programs
• 6.5 Configure Host Access
• 6.6 Run the MPI PingPong benchmark

7. SDP

• 7.1 Configuration
• 7.2 Example Using SDP with Netpipe

8. SRP

• 8.1 Configuration
• 8.2 SRP daemon configuration
• 8.3 Multipathing, LVM and formatting

9. Building Lustre against OFED

• 9.1 Check Compatibility
• 9.2 Build a lustre patched kernel
• 9.3 Build OFED modules for the lustre patched kernel
• 9.4 Configure lustre

10. Troubleshooting

• 10.1 Genernal fabric troubleshooting
• 10.2 ib_query_gid() failed errors on mlx4 platforms
• 10.3 Missing XRC support

11. Tips and Tricks

• 11.1 Descriptive node names

12. Further Information