在RK3588上使用Gstreamer做推拉流并推理记录

基础环境

1. 视频流使用结构

视频流拉取：

video/rtsp/camera ! decoder ! videoconvert ! appsink

视频流推流或显示：

appsrc ! display

推理结构：

appsink——rknn.inference(model)——appsrc

2. 环境安装

2.1 Gstreamer安装

apt-get update
apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
apt-get install libunwind8-dev

pkg-config --cflags --libs gstreamer-1.0
# 查看效果
dpkg -l | grep gstreamer

测试，在终端输入以下正常运行则安装成功

gst-launch-1.0 filesrc location=./test.mp4  ! decodebin ! videoconvert ! ximagesink

2.2 opencv安装

不能使用python3-opencv，通过pip安装的opencv无法支持gstreamer，必须使用源码编译安装

git clone https://github.com/opencv/opencv.git
unzip opencv.zip # 解压zip压缩包
cd opencv # 切换到源码包目录
mkdir build && cd build # 创建build目录并切换进去
# 其中需要配置python目录以及对gstreamer支持
cmake -D WITH_GSTREAMER=ON -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_opencv_python2=OFF -D BUILD_opencv_python3=ON -D  PYTHON3_PACKAGES_PATH=/usr/local/python3.7/lib/python3.7/site-packages/ -D PYTHON3_LIBRARY=/usr/local/python3.7/lib/libpython3.7m.so -D OPENCV_GENERATE_PKGCONFIG=YES  ..

确保opencv开启对gstreamer与python支持的api。

 Video I/O:
    DC1394:                      YES (2.2.5)
    FFMPEG:                      YES
      avcodec:                   YES (58.54.100)
      avformat:                  YES (58.29.100)
      avutil:                    YES (56.31.100)
      swscale:                   YES (5.5.100)
      avresample:                YES (4.0.0)
    GStreamer:                   YES (1.16.2)   # 识别到gstreamer
    v4l/v4l2:                    YES (linux/videodev2.h)

Python 3:
    Interpreter:                 /usr/local/bin/python3 (ver 3.7.3)
    Libraries:                   /usr/local/python3.7/lib/libpython3.7m.so (ver 3.7.3)
    numpy:                       /usr/local/python3.7/lib/python3.7/site-packages/numpy/core/include (ver 1.21.6)
    install path:                /usr/local/python3.7/lib/python3.7/site-packages//cv2/python-3.7

编译安装

make -j4 # 编译
make install # 安装opencv库到/usr/local下

cd /etc/ld.so.conf.d/ # 切换目录
touch opencv.conf # 新建opencv配置文件
echo /usr/local/lib/ > opencv.conf # 填写opencv编译后库所在的路径
sudo ldconfig # 使配置文件生效

最后需要把cv2模块的代码目录中的cv2.so拷贝到python的site-packages目录中,替换原本python3-opencv的cv2.so。

    注：#  如果查不到opencv的版本，需要将opencv4.pc 重命名为opencv.pc

运行以下程序正常表示安装成功

root@ztl:/# python3
Python 3.7.3 (default, Jul  6 2022, 07:24:11) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.getBuildInformation())

3. 使用案例

拉流：

rtsp拉取：in_gst = ('rtspsrc location={} latency=0 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw,format=(string)BGR ! appsink  drop=true').format(url)
mp4视频拉取解码：in_gst = ('filesrc location={} ! decodebin ! videoconvert ! video/x-raw,format=BGR! appsink drop=true caps=video/x-raw,format=BGR').format(url)

推流：

本地显示：out_gst = 'appsrc is-live=true caps=video/x-raw,format=(string)BGR,framerate=(fraction)30/1 ! videoconvert !  ximagesink sync=false'
rtmp推流：后续更新

推理：

# 视频帧数据处理
cap_in = cv2.VideoCapture(in_gst, cv2.CAP_GSTREAMER)
cap_out = cv2.VideoWriter(out_gst, cv2.VideoWriter_fourcc(*'H264'), fps, (640, 320), True)
# 逐帧推理
output = model.inference(image)
# 视频流输出
cap_out = cv2.write(image)