江南人物的博客

Specify the search path for pkg_check_modules: Solution

1

set(ENV{PKG_CONFIG_PATH} "${CMAKE_SOURCE_DIR}/libs/opencv-install/lib/pkgconfig")

find_package

hexo修改默认端口号

参考博客

添加目录

https://blog.ligos.net/2016-07-25/Writing-Content-On-Hexo.html
https://medium.com/employbl/create-a-custom-blog-theme-with-hexo-js-b24c82eb9271#.viurzqeac
https://hexo.io/docs/themes
https://hexo.io/docs/templates
https://hexo.io/docs/helpers

Add pages

Commands

1
2
3
4
5

# The following command generate a 'categories/index.md' file
hexo new page categories

hexo new page tags
hexo new page about

Refers to Hexo NeXT Theme categories and tags page

Plugins

Display PDF

文档

Documentation: RDMA Aware Networks Programming User Manual v1.6

Local Documentation

Sun Network QDR InfiniBand Gateway Switch Topic Set

RDMA知乎专栏

用户态的Verbs API手册跟代码在一个仓库维护，手册地址：https://github.com/linux-rdma/rdma-core/tree/master/libibverbs/man

有很多在线的man page网站可以查阅这些接口的说明，比如官方的连接：https://man7.org/linux/man-pages/man3/ibv_post_send.3.html

也有一些其他非官方网页，支持在线搜索：https://linux.die.net/man/3/ibv

查阅系统man page
如果你使用的商用OS安装了rdma-core或者libibverbs库，那么可以直接用man命令查询接口：

1

man ibv_post_send

查询Mellanox的编程手册
《RDMA Aware Networks Programming User Manual Rev 1.7》，最新版是2015年更新的。该手册写的比较详尽，并且附有示例程序，但是可能与最新的接口有一些差异。Mellanox VPI®（Virtual Procotol Interconnect）架构为同时支持InfiniBand和以太网语义的网络适配器和交换机提供高性能、低延迟和可靠的方法。

Book: Linux Kernel Networking - Implementation and Theory

Dotan’s blog: Dotan Barak, an InfiniBand Expert. Dotan is a Senior Software Manager at Mellanox Technologies working on RDMA Technologies.

性能分析工具（profiling）

Blog: Tips and tricks to optimize your RDMA code

libibprof

IB简介

RDMA - Remote Direct Memory Access 远程直接内存存取。

InfiniBand是一种高性能计算机网络通信标准，它具有极高的吞吐量和极低的延迟。如果您需要使用InfiniBand进行编程，您需要使用支持InfiniBand的编程语言（如C++）来编写代码。

机构和组织：

OFA: Open Fabrics Alliance.

IBTA: InfiniBand Trade Association.

概念

• CQ - Complete Queue 完成队列
• WQ - Work Queue 工作队列
• WR - Work Request 工作请求
• QP - Queue Pairs 队列对（Send-Receive）
• SQ - Send Queue 发送队列
• RQ - Receive Queue 接收队列
• PD - Protection Domain 保护域，将QP和MR结合在一起
• MR - Memory Region 内存区域。一块经注册过的且本地网卡可以读写的内存区域。包含R_Key和L_Key。
• SGE - Scatter/Gather Elements 分散/聚集元素。
• R_Key - Remote Key
• L_Key - Local Key
• CA - (Host) Channel Adapter, an inifiniband network interface card.
• NIC - Network Interface Card 网卡。
• LID - Local Identifier.
• CM - Connection Manager.

其他常见缩写：

• RC - reliable connected.
• SCSI - Small Computer System Interface 小型计算机系统接口。
• SRP - SCSI RDMA Protocol. / Secure Remote Password.

博客：https://blog.51cto.com/liangchaoxi/4044818

安装

InfiniBand 和 RDMA 相关软件包

sudo apt-get install infiniband-diags
sudo apt install ibverbs-utils

API

• introduction to programming infiniband
• RDMA Aware Programming user manual (PDF)

以下是一些支持InfiniBand的C++库：

Infinity：这是一个轻量级的C++ RDMA库，用于InfiniBand网络。它提供了对两侧（发送/接收）和单侧（读/写/原子）操作的支持，并且是一个简单而强大的面向对象的ibVerbs抽象。该库使用户能够构建使用RDMA的复杂应用程序，而不会影响性能1。

OFED：这是一个开放式Fabrics Enterprise Distribution，它提供了对InfiniBand和RoCE（RDMA over Converged Ethernet）技术的支持。OFED提供了一组用户空间库和驱动程序，可用于构建支持RDMA的应用程序2。

以下是使用Infinity库编写支持InfiniBand的C++代码示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

// 创建新上下文
infinity::core::Context *context = new infinity::core::Context();

// 创建队列对
infinity::queues::QueuePairFactory *qpFactory = new infinity::queues::QueuePairFactory(context);
infinity::queues::QueuePair *qp = qpFactory->connectToRemoteHost(SERVER_IP, PORT_NUMBER);

// 创建并向网络注册缓冲区
infinity::memory::Buffer *localBuffer = new infinity::memory::Buffer(context, BUFFER_SIZE);

// 从远程缓冲区读取（单向）并等待完成
infinity::memory::RegionToken *remoteBufferToken = new infinity::memory::RegionToken(REMOTE_BUFFER_INFO);
infinity::requests::RequestToken requestToken(context);
qp->read(localBuffer, remoteBufferToken, &requestToken);
requestToken.waitUntilCompleted();

// 将本地缓冲区的内容写入远程缓冲区（单向）并等待完成
qp->write(localBuffer, remoteBufferToken, &requestToken);
requestToken.waitUntilCompleted();

// 将本地缓冲区的内容通过队列对发送（双向）并等待完成
qp->send(localBuffer, &requestToken);
requestToken.waitUntilCompleted();

// 关闭连接
delete remoteBufferToken;
delete localBuffer;
delete qp;
delete qpFactory;
delete context;

以下是使用OFED库编写支持InfiniBand的C++代码示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

// 创建新上下文
struct ibv_context *context = ibv_open_device(*device);

// 创建完成端口
struct ibv_pd *pd = ibv_alloc_pd(context);

// 创建队列对
struct ibv_qp_init_attr qp_init_attr;
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
qp_init_attr.send_cq = cq;
qp_init_attr.recv_cq = cq;
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.cap.max_send_wr = 1;
qp_init_attr.cap.max_recv_wr = 1;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;
struct ibv_qp *qp = ibv_create_qp(pd, &qp_init_attr);

// 创建并向网络注册缓冲区
char *localBuffer = (char *)malloc(BUFFER_SIZE);
struct ibv_mr *mr = ibv_reg_mr(pd, localBuffer, BUFFER_SIZE, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);

// 连接到远程主机
struct sockaddr_in remoteAddress;
memset(&remoteAddress, 0, sizeof(remoteAddress));
remoteAddress.sin_family = AF_INET;
remoteAddress.sin_port = htons(PORT_NUMBER);
inet_pton(AF_INET, SERVER_IP, &remoteAddress.sin_addr);
struct rdma_cm_id *cmId;
rdma_create_id(*eventChannel, &cmId, NULL, RDMA_PS_TCP);
rdma_resolve_addr(cmId, NULL, (struct sockaddr *)&remoteAddress, RESOLVE_TIMEOUT_MS);

// 等待连接完成
rdma_wait_event(*eventChannel, RDMA_CM_EVENT_ESTABLISHED);
rdma_ack_cm_event(cmEvent);

// 获取远程缓冲区信息
struct ibv_wc wc;
ibv_post_recv(qp, &recvWr, &badRecvWr);
do {
    ibv_poll_cq(cq, 1, &wc);
} while (wc.status != IBV_WC_SUCCESS || wc.opcode != IBV_WC_RECV_RDMA_WITH_IMM || wc.imm_data != htonl(IMM_DATA));
remoteBufferInfo.rkey = ntohl(wc.imm_data >> 8);
remoteBufferInfo.vaddr = wc.wr_id;

// 将本地缓冲区的内容写入远程缓冲区（单向）
struct ibv_send_wr sendWr;
memset(&sendWr, 0, sizeof(sendWr));
sendWr.wr_id = 0;
sendWr.opcode = IBV_WR_RDMA_WRITE_WITH_IMM;
sendWr.sg_list = &localSge;
sendWr.num_sge = 1;
sendWr.send_flags = IBV_SEND_SIGNALED;
sendWr.wr.rdma.remote_addr = remoteBufferInfo.vaddr;
sendWr.wr.rdma.rkey = remoteBufferInfo.rkey;
localSge.addr = (uintptr_t)localBuffer;
localSge.length = BUFFER_SIZE;
localSge.lkey = mr->lkey;
ibv_post_send(qp, &sendWr, &badSendWr);

// 关闭连接
ibv_dereg_mr(mr);
free(localBuffer);
ibv_destroy_qp(qp);
ibv_dealloc_pd(pd);
ibv_close_device(context);

• 入门级文档：https://zhuanlan.zhihu.com/p/337461037
• 文档：https://docs.kernel.org/infiniband/index.html
• 文档：http://blog.foool.net/wp-content/uploads/linuxdocs/infiniband.pdf
• 文档：https://support.bull.com/documentation/byproduct/infra/sw-extremcomp/sw-extremcomp-com/g
• 文档：https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_infiniband_and_rdma_networks/index
• 博客：https://zhuanlan.zhihu.com/p/337461037
• 文档：https://access.redhat.com/documentation/zh-cn/red_hat_enterprise_linux/7/html/networking_guide/ch-configure_infiniband_and_rdma_networks
• 知乎专栏：https://www.zhihu.com/column/c_1231181516811390976
• 知乎专栏：https://www.zhihu.com/column/rdmatechnology

Linux manual page

Command-Line

文档：https://docs.nvidia.com/networking/pages/viewpage.action?pageId=43719572

• ibstat

• ibhosts - 查看所有的IB hosts。

• ibnetdiscover - discover InfiniBand topology.

• ibv_devices - list RDMA devices.

• ibv_devinof - Print information about RDMA devices available for use from userspace.

• ibv_rc_pingpong - Run a simple ping-pong test over InfiniBand via the reliable connected (RC) transport.

• targetcli - administration shell for storage targets

  targetcli is a shell for viewing, editing, and saving the configuration of the kernel’s target subsystem,
  also known as LIO. It enables the administrator to assign local storage resources backed by either files,
  volumes, local SCSI devices, or ramdisk, and export them to remote systems via network fabrics, such as iSCSI or FCoE.

• srp_daemon - Discovers and connects to InfiniBand SCSI RDMA Protocol (SRP) targets in an IB fabric.

• ibsrpdm - List InfiniBand SCSI RDMA Protocol (SRP) targets on an IB fabric.

liraries

devid: device ID library. Refer to here.

ibverbs: 使得用户空间进程能够使用RDMA verbs（即进行RDMA操作）。Refer to here.

dl: Dynamic Loader.

Multiple instances of singleton across shared libraries on Linux

Functions

https://cpp.hotexamples.com/zh/examples/-/-/lprintf/cpp-lprintf-function-examples.html

• printf
• lprintf
• fprintf
• sprintf
• vsprintf

字符串处理函数
bcopy

非打印字符

参见 Escape sequeneces.

1
char str[] = "\x1E"; // "0x1E" 是一个非打印字符

Files

/etc/services: 端口列表
/etc/protocols: 协议列表

Reference

端口0的作用

“make run” argument

See here.

@: See here.

This works fine for me:

1
2
3
4
5
6
7
8
9
10
11
12

# If the first argument is "run"...
ifeq (run,$(firstword $(MAKECMDGOALS)))
  # use the rest as arguments for "run"
  RUN_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
  # ...and turn them into do-nothing targets
  # TODO: What does the following line mean?
  # $(eval $(RUN_ARGS):;@:)
endif

# "cmd" refers to any command
run:
	cmd $(RUN_ARGS)

makefile同名目标处理方式

参考：

• 链接
• 链接

makefile将命令结果赋值给变量

Makefile中短划线

1
2

all:
	-/bin/rm -rf *.log

其中，”-/bin/rm“的短划线”-“是一个特殊前缀，表示忽略命令执行过程的错误。

为每个源文件生成一个可执行程序

1
2
3
4
5
6
7

SRCS = $(wildcard *.c)

all: $(SRCS:.c=)

# Unnecessary, as the default rules are adequate.
.c:
	gcc $(CPFLAGS) $< -o $@

最后两行其实不需要，默认规则已经足够了。

其中，$(SRCS:.c=.o)表示将变量SRCS中的每个单词（以空格分割）中的.c替换为.o。以上代码则是将所有.c都去掉。

括号

引用变量时，Shell使用大括号，Makefile则大括号和小括号都行。但是在命令中使用Shell变量就需要使用大括号。

参考

Color Scheme

Refer to the link.

Add these three lines to ~/.bashrc

1
2
3
4
5

$ vi ~/.bashrc
export LS_OPTIONS='--color=auto'
# dircolors - color set for ls
eval "$(dircolors -b)"
alias ls='ls $LS_OPTIONS'

Terms

resident set size (RSS)

Comands

File

/proc/self/statm

top

Fields

ps

Files

TIME: the cumulated CPU time in [DD-]hh:mm:ss format (time=TIME)

Reference

what-does-virtual-memory-size-in-top-mean

See the article: Prefixes for binary multiples

Table: