江南人物的博客
0%

Posted on Edited on In
  1. 多人在线游戏架构实战：基于C++的分布式游戏编程. 彭放编著. 北京：机械工业出版社，2020.11 （ISBN 978-7-111-66792-6）

Posted on Edited on In

Options

npm root - Current installation root path.
npm root -g - The global installation root path.
npm uninstall - After executing this command, you should also remove the items in the package.json file and delete the package-lock.json file and the node_modules directory.

Reference

Where does npm install packages?
Cannot Uninstall two npm Packages

Posted on Edited on In

安装

sudo apt install net-tools

手册

man 7 socket - socket options
man 7 tcp - tcp options

文件系统

/proc/sys/net/ipv4/ip_local_port_range

概念

网关（Gateway）

网关（Gateway）是一个网络节点，它充当其他网络之间的访问点或网关。网关通常用于连接不同的网络，并且可以在不同的协议之间进行转换。网关设备可以是路由器、服务器或其他网络设备。当本地网络中的设备需要与外部网络通信时，它们会将数据包发送到网关，网关再将数据包转发到目标网络。

举例： 假设你的本地网络使用的IP地址范围是192.168.1.0/24，而你的网关IP地址是192.168.1.1。当你的计算机需要访问外部网络（例如互联网）时，它会将数据包发送到网关（192.168.1.1），然后由网关将数据包转发到目标网络。

在Linux系统中，你可以使用ip route命令查看默认网关。例如：

在这个例子中，default via 192.168.1.1 dev eth0表示默认网关是192.168.1.1，你的计算机通过网络接口eth0与网关连接。

子网（CIDR表示法）

CIDR（Classless Inter-Domain Routing，无类别域间路由）是一种用于分配IP地址和路由的标准，使用斜杠（/）后跟一个数字来表示子网掩码的位数。

在CIDR表示法中，192.168.1.0/24表示一个子网，其中/24表示子网掩码为255.255.255.0。这意味着子网中有256个IP地址（从0到255），但其中两个地址是保留的：

  1. 网络地址：192.168.1.0，用于标识子网本身。
  2. 广播地址：192.168.1.255，用于向子网中的所有设备发送广播消息。

因此，有效的主机IP地址范围是从192.168.1.1到192.168.1.254，共254个可用的IP地址。

The link-local address range refers to IP addresses that are used for communication within a single network segment or link. These addresses are not routable, meaning they are not intended to be used for communication beyond the local network segment.

For IPv4, the link-local address range is 169.254.0.0/16. This range is automatically assigned to network interfaces when no other IP address is available (e.g., when DHCP fails). Devices on the same local network can communicate with each other using these addresses without requiring a router.

For IPv6, the link-local address range is fe80::/10. These addresses are automatically configured on all IPv6-enabled interfaces and are used for local network communication.

The IP address range 169.254.0.0/16 is regarded as link-local because it is reserved for link-local addresses by the Internet Assigned Numbers Authority (IANA). These addresses are used for communication within a single network segment or link and are not routable beyond that segment.

Characteristics of Link-Local Addresses:

  • Automatic Assignment: Devices automatically assign themselves an IP address from the 169.254.0.0/16 range if they cannot obtain an IP address through DHCP.
  • Non-Routable: These addresses are not meant to be routed across different network segments. They are only valid within the local network.
  • Zero Configuration: Link-local addresses allow devices to communicate without manual configuration or a DHCP server.

Example:
When a device fails to get an IP address from a DHCP server, it might assign itself an address like 169.254.1.2. Other devices on the same local network segment will also have addresses in the 169.254.0.0/16 range, allowing them to communicate directly.

命令

ip

NAME

ip - show / manipulate routing, devices, policy routing and tunnels.

1
2
3
$ ip route get 8.8.8.8
8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100 uid 1000
    cache

说明：

  • 8.8.8.8: The destination IP address for which the route is being queried.
  • via 192.168.1.1: The gateway IP address through which the destination can be reached.
  • dev eth0: The network interface (e.g., eth0) used to reach the destination.
  • src 192.168.1.100: The source IP address used for sending packets to the destination.
  • uid 1000: The user ID of the process that issued the command.
  • cache: Indicates that the route information is cached.
1
2
3
4
$ ip route show
default via 192.168.1.1 dev eth0 
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.100
169.254.0.0/16 dev idrac proto kernel scope link src 169.254.1.2

说明：该命令用于显示路由表：

  • default via 192.168.1.1 dev eth0: The default route, packets to any destination not in the routing table will be sent to 192.168.1.1 via the eth0 interface.
  • 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.100: A route for the 192.168.1.0/24 network (destination network), directly reachable via the eth0 interface, with the source IP 192.168.1.100.
  • 192.168.1.0/24: Destination network.
  • dev eth0: Interface eth0.
  • proto kernel: Added by the kernel.
  • scope link: Directly reachable.
  • src 192.168.1.100: Source IP address.
  • 169.254.0.0/16: Link-local address range.

netstat / ss

NAME

ss - another to investigate sockets.

Reference

查看进程占用的端口

COMMANDS

ss -lnpt

OPTIONS

-n, --numberic

-l, --listening

-t, --tcp
    Note: Only established (non-listening) connections.

-a, --all
    Display both listening and non-linstening (for TCP this means established connections) sockets.

-u, --udp

-p, --process

lsof

Options:

-n inhibits the conversion of network numbers to host names for network files.

-P inhibits the conversion of port numbers to port names for network files.

-i specifies the Internet addresses

    multiple addresses (up to a limit of 100) may be specified with multiple options.

    An Internet address is specified in the form (Items in square brackets are optional.):

    [46][protocol][@hostname|hostaddr][:service|port]

    where:
    46 specifies the IP version, IPv4 or IPv6
        that applies to the following address.
        '6' may be be specified only if the UNIX
        dialect supports IPv6.  If neither '4' nor
        '6' is specified, the following address
        applies to all IP versions.
    protocol is a protocol name - TCP, UDP
    hostname is an Internet host name.  Unless a
        specific IP version is specified, open
        network files associated with host names
        of all versions will be selected.
    hostaddr is a numeric Internet IPv4 address in
        dot form; or an IPv6 numeric address in
        colon form, enclosed in brackets, if the
        UNIX dialect supports IPv6.  When an IP
        version is selected, only its numeric
        addresses may be specified.
    service is an /etc/services name - e.g., smtp -
        or a list of them.
    port is a port number, or a list of them.

-s list file size

-s p:s  exclude(^)|select protocol (p = TCP|UDP) states by name(s).

Examples:

1
2
3
lsof -i:<port>
lsof -i -P -n | grep LISTEN
lsof -nP -iTCP -sTCP:LISTEN

nc/netcat/nmap

socket

Create a TCP or a UNIX domain socket and connect to stdin/out.

Installation:

sudo apt-get install socket

Posted on Edited on In

电影链接

Midsomer Murders： Season 1 Episode 2 - Written in Blood

台词

Phoenix turned and fired twice, the 34 millimetre bullets spitting through the air and splintering the brickwork two inches to the left of the Russia thug’s shoulder. A moment later came the reply of the Uzi semiautomatic and Phoenix twisted round. 菲尼克斯转身开了两枪，34毫米的子弹在空中喷射，将俄罗斯暴徒肩膀左侧两英寸砖墙击碎。过一会儿，乌兹半自动冲锋枪回应了，菲尼克斯扭过身来。

Her lips, searching in the sapphire, honeyscented darkness, found his; cool, forceful, unresisting, and as the frozen waves crashed down. 当冰冷、有力、不可抗拒的浪汹涌而下的时候，她的嘴唇在蓝宝石般散发着蜜香的夜色中寻找，找到了他的嘴唇。

-Got your meeting, have you? 你要去开会，对吗？
-Yes, I’m on my way. 是的，我正在去。

He pulled her towards him, his powerful, masculine hands tearing at the silk faric of her Gaultier nightshirt.

The repetitive beat of the innercity drum.

1887, Sebastian Lyddiard, greatgrandson of Herbert Lyddiard, who served under Sir John Jervis against Napoleon. 对抗拿破仑。

The Right Honourable William Lyddiard. 威廉·利迪亚德阁下。

-Do you ever regret marrying hime, Amy? I wonder if you ever think that but for you he might still be alive? 你后悔嫁给他吗，埃米？我不知道你是否想过，如果不是因为你，他可能现在还活着？
-(Amy) No. 没有想过。
-Well I do. 我想过。

-So we come to the question of which writer we invite to address us? 所以我们打算请哪位作家来给我们做演讲呢？
-How about Jilly Cooper?
-I think he said a writer.
-Do you think we could interest Frederick Forsythe?
-I rather doubt it.
-Oh, it’s a wonderful idea, Gerald. Especially since you’re writing a thriller. I think Frederick Forsythe would be first rate! 我认为Frederick Forsythe是第一流的。
-He’d never come. He’d just pretend he didn’t have time. They always do.
-I’d like Seamus Heaney.
-The poet?
-Not another poet. That last one was such a ghastly little man.

词汇

screenplay by 剧本由……写成
millimetre 毫米
phoenix 凤凰，这里是人名，菲尼克斯
spit 吐，唾
thug 恶棍，暴徒
twist 扭，拧，捻
brickwork 砖砌体，砖墙
splinter n. 碎片，弹片； v. 碎裂。
Uzi semiautomatic 乌兹半自动冲锋枪（Uzi submachine gun）。Uzi 冲锋枪是由以色列国防军 (IDF) 上尉（后来的少校）Uziel Gal 在 1948 年阿以战争后设计的。
sapphire 蓝宝石
honey 蜂蜜，蜜
scent 香味（a pleasant natural smell）；scented 散发香味的
resist 抗拒的；unresisted 不可抗拒的。
masculine 男性的。
tear 撕，撕破，撕毁；拆；泪水。
nightshirt 睡衣
scum 浮渣
scum-bag 渣子袋，这里应该是“渣男”、“人渣”的意思，骂人的话。
hum 哼声
greatgrandson 曾孙
Sir 爵士
Napoleon 拿破仑
The Right Honourable 阁下。其中，Right是副词，意思是’very’ or ‘fully’。从语法上说，The Right Honourable 是形容短语，直接用来称呼人名或者某人用来自称都是不正确的，应该与其引用的名称或名称一起使用第三人称。
address 演讲
first rate 第一流
pretend 假装
ghastly US:/ˈɡæst.li/ UK:/ˈɡɑːst.li/ 阴森的，可怕的，非常坏的
bestseller 畅销书

Posted on Edited on In

Specify the search path for pkg_check_modules: Solution

1
set(ENV{PKG_CONFIG_PATH} "${CMAKE_SOURCE_DIR}/libs/opencv-install/lib/pkgconfig")

find_package

Posted on Edited on In

hexo修改默认端口号

参考博客

添加目录

https://blog.ligos.net/2016-07-25/Writing-Content-On-Hexo.html
https://medium.com/employbl/create-a-custom-blog-theme-with-hexo-js-b24c82eb9271#.viurzqeac
https://hexo.io/docs/themes
https://hexo.io/docs/templates
https://hexo.io/docs/helpers

Add pages

Commands

1
2
3
4
5
# The following command generate a 'categories/index.md' file
hexo new page categories

hexo new page tags
hexo new page about

Refers to Hexo NeXT Theme categories and tags page

Plugins

Display PDF

Posted on Edited on In

find_host_queue.sh

寻找 hostname 所在的 queue ：

bsub-01.shview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#!/bin/bash

# 检查是否提供了主机名参数
if [ -z "$1" ]; then
    echo "用法: $0 <hostname>"
    exit 1
fi

HOSTNAME=$1

echo "正在查找包含主机 [$HOSTNAME] 或其主机组的队列..."

# 获取所有队列名
queues=$(bqueues | awk 'NR>1 {print $1}')

for q in $queues; do
    # 获取队列的 HOSTS 字段
    hosts_line=$(bqueues -l "$q" | awk '/HOSTS:/,/^$/')
    # 将字符串按空格拆分为数组
    host_array=($hosts_line)

    # 提取所有主机或主机组
    for entry in "${host_array[@]}"; do
        # 如果是主机组（以@开头或以/结尾）
        if [[ $entry == @* || $entry == */ ]]; then
            # if [[ $entry == @* ]]; then
            #     # 去掉前面的@符号
            #     entry=${entry:1}
            # fi
            # 去掉前面的@符号和后面的斜杠
            group=${entry#@}
            group=${entry%/}
            # 获取主机组中的主机列表
            if bmgroup $group 2>/dev/null | grep -qw "$HOSTNAME"; then
                echo "✅ 队列 [$q] 包含主机组 [$group]，其中包括主机 [$HOSTNAME]"
            fi
        else
            # 直接匹配主机名
            if [[ "$entry" == "$HOSTNAME" ]]; then
                echo "✅ 队列 [$q] 直接包含主机 [$HOSTNAME]"
            fi
        fi
    done
done

auto_submit_to_host_queue.sh

根据提供的 hostname ，自动寻找最空闲的队列提交 bsub 作业：

bsub-02.shview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
#!/bin/bash

# 该脚本会自动将 hostname 提交到延迟最小、最空闲的队列
# 处理后的命令为 bsub -q <queue> -m <host> <args>

# 用法提示
# 例如：$0 myhostname -Is bash
if [ $# -lt 2 ]; then
    echo "用法: $0 <hostname> <args>"
    exit 1
fi

HOSTNAME=$1
# JOB_SCRIPT=$2

# # 检查作业脚本是否存在
# if [ ! -f "$JOB_SCRIPT" ]; then
#     echo "错误：找不到作业脚本 $JOB_SCRIPT"
#     exit 1
# fi

echo "🔍 正在查找包含主机 [$HOSTNAME] 的队列..."

declare -A matching_queues

# 获取所有队列名
queues=$(bqueues | awk 'NR>1 {print $1}')

for q in $queues; do
    # 获取队列的 HOSTS 字段
    hosts_line=$(bqueues -l "$q" | awk '/HOSTS:/,/^$/')
    host_array=($hosts_line)

    for entry in "${host_array[@]}"; do
        # 如果是主机组（以@开头或以/结尾）
        if [[ $entry == @* || $entry == */ ]]; then
            # 去掉前面的@符号和后面的斜杠
            group=${entry#@}
            group=${entry%/}
            if bmgroup "$group" 2>/dev/null | grep -qw "$HOSTNAME"; then
                echo "✅ 队列 [$q] 包含主机组 [$group]，其中包括主机 [$HOSTNAME]"
                matching_queues["$q"]=1
            fi
        elif [[ "$entry" == "$HOSTNAME" ]]; then
            echo "✅ 队列 [$q] 直接包含主机 [$HOSTNAME]"
            matching_queues["$q"]=1
        fi
    done
done

if [ ${#matching_queues[@]} -eq 0 ]; then
    echo "❌ 没有找到包含主机 [$HOSTNAME] 的队列。"
    exit 1
fi

echo "✅ 找到以下队列包含主机 [$HOSTNAME]："
for q in "${!matching_queues[@]}"; do
    echo " - $q"
done

# 获取这些队列的等待任务数，选择最空闲的一个
best_queue=""
min_pend=999999

sorted_queues=$(for q in "${!matching_queues[@]}"; do
    pend=$(bqueues "$q" | awk 'NR==2 {print $9}')
    echo "$pend $q"
    # if [[ "$pend" =~ ^[0-9]+$ ]] && [ "$pend" -lt "$min_pend" ]; then
    #     min_pend=$pend
    #     best_queue=$q
    # fi
done | sort -n | awk '{print $2}')

# if [ -z "$best_queue" ]; then
#     echo "⚠️ 无法确定最空闲队列，默认使用第一个：${!matching_queues[@]}"
#     best_queue=${!matching_queues[@]}
# fi


# 依次尝试提交任务
for q in $sorted_queues; do
    echo "🔎 尝试队列 [$q]..."
    test_output=$(timeout 10 bsub -q "$q" -n 1 -Is /bin/true 2>&1)
    if [ $? -eq 124 ]; then
        echo "⏳ 队列 [$q] 测试超时，跳过。"
        continue
    fi
    if echo "$test_output" | grep -q "User cannot use the queue"; then
        echo "⛔ 无权限使用队列 [$q]，跳过。"
        continue
    fi

    echo "🚀 提交任务到队列 [$q]..."
    CMD="bsub -q $q -m $HOSTNAME ${@:2}"
    # exec将当前终端传递给bsub
    # exec $CMD
    $CMD
    exit 0
done

echo "❌ 所有队列都无法使用或提交失败。"
exit 1

# echo "🚀 正在将任务提交到最空闲的队列 [$best_queue]..."
# # bsub -q "$best_queue" "$JOB_SCRIPT"
# bsub -q "$best_queue" -m "$HOSTNAME" "$argv[2-]"

Posted on Edited on In

函数

ibv_fork_init

文档
ibv_fork_init

模板: int ibv_fork_init(void)

输入参数: 无

输出参数: 无

返回值: 0 on success, -1 on error. If the call fails, errno will be set to indicate the
reason for the failure.

描述: ibv_fork_init 初始化 libverbs 的数据结构来安全地处理 fork() 并避免数据损坏，不论
fork() 是被显式调用还是隐式调用（比如在 system() 中被调用）。如果所有的父进程总是阻塞直至所有的
子进程结束或使用 exec() 改变地址空间，那么 ibv_fork_init 可以不被调用。

该函数在支持 madvise 的 MADV_DONTFORK 标记的 Linux 内核（2.6.17 或更高）上可以工作。

设置环境变量 RDMAV_FORK_SAFE 或 IBV_FORK_SAFE 环境变量为任意值，有着与 ibv_fork_init 相同的效
果。

设置 RDMAV_HUGEPAGES_SAFE 为任意值，以告诉库需要检查内核为内存域（memory regions）使用的底层内存页
的大小。如果应用程序直接或通过库（如 libhugtlbfs ）间接使用大内存页（博主注：即大于 4KB）时，该环境
变量是必须的。（博主注：ibv_fork_init 将检查 RDMAV_HUGEPAGES_SAFE ）

调用 ibv_fork_init 将降低性能，因为每个内存注册都将有一个额外的系统调用和分配附加的内存以追踪内存
域（memory regions）。确切的性能损失取决于工作负载，通常不会很大。

设置 RDMAV_HUGEPAGES_SAFE 会为所有的内存注册增加更多的开销。

文档

Documentation: RDMA Aware Networks Programming User Manual v1.6

Local Documentation

Sun Network QDR InfiniBand Gateway Switch Topic Set

RDMA 知乎专栏

用户态的 Verbs API 手册跟代码在一个仓库维护，手册地址
：https://github.com/linux-rdma/rdma-core/tree/master/libibverbs/man

有很多在线的 man page 网站可以查阅这些接口的说明，比如官方的连接
：https://man7.org/linux/man-pages/man3/ibv_post_send.3.html

也有一些其他非官方网页，支持在线搜索：https://linux.die.net/man/3/ibv

查阅系统 man page 如果你使用的商用 OS 安装了 rdma-core 或者 libibverbs 库，那么可以直接用 man 命令查
询接口：

1
man ibv_post_send

查询 Mellanox 的编程手册
《RDMA Aware Networks Programming User Manual Rev 1.7》
，最新版是 2015 年更新的。该手册写的比较详尽，并且附有示例程序，但是可能与最新的接口有一些差异
。Mellanox VPI®（Virtual Procotol Interconnect）架构为同时支持 InfiniBand 和以太网语义的网络适配器和
交换机提供高性能、低延迟和可靠的方法。

Book:
Linux Kernel Networking - Implementation and Theory

Dotan’s blog: Dotan Barak, an InfiniBand Expert. Dotan is a Senior
Software Manager at Mellanox Technologies working on RDMA Technologies.

性能分析工具（profiling）

Blog:
Tips and tricks to optimize your RDMA code

libibprof

IB 简介

RDMA - Remote Direct Memory Access 远程直接内存存取。

InfiniBand 是一种高性能计算机网络通信标准，它具有极高的吞吐量和极低的延迟。如果您需要使用 InfiniBand
进行编程，您需要使用支持 InfiniBand 的编程语言（如 C++）来编写代码。

机构和组织：

OFA: Open Fabrics Alliance.

IBTA: InfiniBand Trade Association.

概念

  • CQ - Complete Queue 完成队列
  • WQ - Work Queue 工作队列
  • WR - Work Request 工作请求
  • QP - Queue Pairs 队列对（Send-Receive）
  • SQ - Send Queue 发送队列
  • RQ - Receive Queue 接收队列
  • PD - Protection Domain 保护域，将 QP 和 MR 结合在一起
  • MR - Memory Region 内存区域。一块经注册过的且本地网卡可以读写的内存区域。包含 R_Key 和 L_Key。
  • SGE - Scatter/Gather Elements 分散/聚集元素。
  • R_Key - Remote Key
  • L_Key - Local Key
  • CA - (Host) Channel Adapter, an inifiniband network interface card.
  • NIC - Network Interface Card 网卡。
  • LID - Local Identifier.
  • CM - Connection Manager.

其他常见缩写：

  • RC - reliable connected.
  • SCSI - Small Computer System Interface 小型计算机系统接口。
  • SRP - SCSI RDMA Protocol. / Secure Remote Password.

博客：https://blog.51cto.com/liangchaoxi/4044818

安装

InfiniBand 和 RDMA 相关软件包

sudo apt-get install infiniband-diags
sudo apt install ibverbs-utils

API

以下是一些支持 InfiniBand 的 C++库：

Infinity：这是一个轻量级的 C++ RDMA 库，用于 InfiniBand 网络。它提供了对两侧（发送/接收）和单侧（读/
写/原子）操作的支持，并且是一个简单而强大的面向对象的 ibVerbs 抽象。该库使用户能够构建使用 RDMA 的复
杂应用程序，而不会影响性能1。

OFED：这是一个开放式 Fabrics Enterprise Distribution，它提供了对 InfiniBand 和 RoCE（RDMA over
Converged Ethernet）技术的支持。OFED 提供了一组用户空间库和驱动程序，可用于构建支持 RDMA 的应用程
序2。

以下是使用 Infinity 库编写支持 InfiniBand 的 C++代码示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// 创建新上下文
infinity::core::Context *context = new infinity::core::Context();

// 创建队列对
infinity::queues::QueuePairFactory *qpFactory = new infinity::queues::QueuePairFactory(context);
infinity::queues::QueuePair *qp = qpFactory->connectToRemoteHost(SERVER_IP, PORT_NUMBER);

// 创建并向网络注册缓冲区
infinity::memory::Buffer *localBuffer = new infinity::memory::Buffer(context, BUFFER_SIZE);

// 从远程缓冲区读取（单向）并等待完成
infinity::memory::RegionToken *remoteBufferToken = new infinity::memory::RegionToken(REMOTE_BUFFER_INFO);
infinity::requests::RequestToken requestToken(context);
qp->read(localBuffer, remoteBufferToken, &requestToken);
requestToken.waitUntilCompleted();

// 将本地缓冲区的内容写入远程缓冲区（单向）并等待完成
qp->write(localBuffer, remoteBufferToken, &requestToken);
requestToken.waitUntilCompleted();

// 将本地缓冲区的内容通过队列对发送（双向）并等待完成
qp->send(localBuffer, &requestToken);
requestToken.waitUntilCompleted();

// 关闭连接
delete remoteBufferToken;
delete localBuffer;
delete qp;
delete qpFactory;
delete context;

以下是使用 OFED 库编写支持 InfiniBand 的 C++代码示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
// 创建新上下文
struct ibv_context *context = ibv_open_device(*device);

// 创建完成端口
struct ibv_pd *pd = ibv_alloc_pd(context);

// 创建队列对
struct ibv_qp_init_attr qp_init_attr;
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
qp_init_attr.send_cq = cq;
qp_init_attr.recv_cq = cq;
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.cap.max_send_wr = 1;
qp_init_attr.cap.max_recv_wr = 1;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;
struct ibv_qp *qp = ibv_create_qp(pd, &qp_init_attr);

// 创建并向网络注册缓冲区
char *localBuffer = (char *)malloc(BUFFER_SIZE);
struct ibv_mr *mr = ibv_reg_mr(pd, localBuffer, BUFFER_SIZE, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);

// 连接到远程主机
struct sockaddr_in remoteAddress;
memset(&remoteAddress, 0, sizeof(remoteAddress));
remoteAddress.sin_family = AF_INET;
remoteAddress.sin_port = htons(PORT_NUMBER);
inet_pton(AF_INET, SERVER_IP, &remoteAddress.sin_addr);
struct rdma_cm_id *cmId;
rdma_create_id(*eventChannel, &cmId, NULL, RDMA_PS_TCP);
rdma_resolve_addr(cmId, NULL, (struct sockaddr *)&remoteAddress, RESOLVE_TIMEOUT_MS);

// 等待连接完成
rdma_wait_event(*eventChannel, RDMA_CM_EVENT_ESTABLISHED);
rdma_ack_cm_event(cmEvent);

// 获取远程缓冲区信息
struct ibv_wc wc;
ibv_post_recv(qp, &recvWr, &badRecvWr);
do {
    ibv_poll_cq(cq, 1, &wc);
} while (wc.status != IBV_WC_SUCCESS || wc.opcode != IBV_WC_RECV_RDMA_WITH_IMM || wc.imm_data != htonl(IMM_DATA));
remoteBufferInfo.rkey = ntohl(wc.imm_data >> 8);
remoteBufferInfo.vaddr = wc.wr_id;

// 将本地缓冲区的内容写入远程缓冲区（单向）
struct ibv_send_wr sendWr;
memset(&sendWr, 0, sizeof(sendWr));
sendWr.wr_id = 0;
sendWr.opcode = IBV_WR_RDMA_WRITE_WITH_IMM;
sendWr.sg_list = &localSge;
sendWr.num_sge = 1;
sendWr.send_flags = IBV_SEND_SIGNALED;
sendWr.wr.rdma.remote_addr = remoteBufferInfo.vaddr;
sendWr.wr.rdma.rkey = remoteBufferInfo.rkey;
localSge.addr = (uintptr_t)localBuffer;
localSge.length = BUFFER_SIZE;
localSge.lkey = mr->lkey;
ibv_post_send(qp, &sendWr, &badSendWr);

// 关闭连接
ibv_dereg_mr(mr);
free(localBuffer);
ibv_destroy_qp(qp);
ibv_dealloc_pd(pd);
ibv_close_device(context);

Linux manual page

Command-Line

文档：https://docs.nvidia.com/networking/pages/viewpage.action?pageId=43719572

  • ibstat

  • ibhosts - 查看所有的 IB hosts。

  • ibnetdiscover - discover InfiniBand topology.

  • ibv_devices - list RDMA devices.

  • ibv_devinof - Print information about RDMA devices available for use from userspace.

  • ibv_rc_pingpong - Run a simple ping-pong test over InfiniBand via the reliable connected (RC)
    transport.

  • targetcli - administration shell for storage targets

    targetcli is a shell for viewing, editing, and saving the configuration of the kernel’s target
    subsystem, also known as LIO. It enables the administrator to assign local storage resources
    backed by either files, volumes, local SCSI devices, or ramdisk, and export them to remote systems
    via network fabrics, such as iSCSI or FCoE.

  • srp_daemon - Discovers and connects to InfiniBand SCSI RDMA Protocol (SRP) targets in an IB
    fabric.

  • ibsrpdm - List InfiniBand SCSI RDMA Protocol (SRP) targets on an IB fabric.

liraries

devid: device ID library. Refer to
here.

ibverbs: 使得用户空间进程能够使用 RDMA verbs（即进行 RDMA 操作）。Refer to
here.

dl: Dynamic Loader.