gst:start

Gstreamer
- install
- code build on windows
- plugins
- Profiling
- Sync
- validate
- RGBD
- Dot Files
- also
other things
- Python
- brave
- debug
- output
- CheatSheet

Gstreamer

gstreamer the project with the best readme ever

gstreamer 101

a gstreamer application runs a pipeline, composed of elements. the data flow is composed of buffers and events .

By linking a source element with zero or more filter-like elements and finally a sink element, you set up a media pipeline. Data will flow through the elements. This is the basic concept of media handling in GStreamer.

A pad type is defined by two properties: its direction and its availability…
GStreamer defines two pad directions: source pads and sink pads. This terminology is defined from the view of within the element: elements receive data on their sink pads and generate data on their source pads. Schematically, sink pads are drawn on the left side of an element, whereas source pads are drawn on the right side of an element. In such graphs, data flows from left to right. more on pads

The data flowing through a pipeline consists of a combination of buffers and events. Buffers contain the actual media data. Events contain control information

cheatsheet

#launch a camera and play max fps (set in eeprom)
 
gst-launch-1.0 -v -m autovideotest ! videoconvert ! fpsdisplaysink video-sink=fakesink text-overlay=false
 
gst-launch-1.0 filesrc location=test-single-30fps-synced.mkv ! decodebin ! videoconvert ! fpsdisplaysink video-sink=glimagesink

install

approach that will compile and copy gstreamer to bare metal.
we can also run directly from docker (wip)

there is an install script to be automated in the package folder

the following data is for the know how

gst-build

UBUNTU
ubuntu 18 comes with gstreamer 14.x built it , but it is an old release. missing some nice stuff.
better build using gst-build

sudo apt install -y apt-utils bison  build-essential cmake  curl flex gir1.2-gudev-1.0 git gudev-1.0-dev libbz2-dev libcairo2-dev libclang-dev libgirepository1.0-dev libglib2.0-dev libgtk-3-dev libgtk2.0-dev libjson-glib-dev libncurses5-dev libncursesw5-dev liborc-0.4-dev libreadline-dev libsqlite3-dev libssl-dev libudev-dev libxml2-dev libxslt1-dev  libyaml-dev llvm llvm-dev ninja-build python3-pip tk-dev vainfo xz-utils zlib1g-dev 
 
pip3 install meson --user
export PATH=$PATH:~/.local/bin
 
git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git && cd gst-build
meson \
       -Dbuildtype=release \
       -Dgtk_doc=disabled \
	   -Dpython=disabled \
	   -Dugly=disabled \
	   -Dges=disabled \
	   build 
 
#see below to run gst in private mode (uninstalled) or systemwide (install)
# add to env the following in later case
export LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu/

then run 'ninja uninstalled' or install over current version using 'ninja install', make sure you dirty remove gstreamer 1.4 (see below)

to run uninstalled

ninja -C /opt/gst/gst-build/build uninstalled

to “install” gstreamer over the ubuntu package do

sudo ninja -C /opt/gst/gst-build/build install

to remove/uninstall gstreamer

sudo -s
rm /usr/local/bin/gst-*
rm /usr/bin/gst-*
rm -fr /usr/local/include/gstreamer-1.0/
rm -fr /usr/include/gstreamer-1.0/
rm -fr /usr/local/libexec/gstreamer-1.0/
rm /usr/lib/x86_64-linux-gnu/libgst*
rm /usr/local/lib/x86_64-linux-gnu/libgst*

WINDOWS

https://www.collabora.com/news-and-blog/blog/2019/11/26/gstreamer-windows/
using following script
had to disable

msdk (temp until installing Media SDK for windows)
libxml
- rename/delete the libxml2.wrap file from gst-build/subproject dir
- disable GES in meson_options.txt

Click to display ⇲

Click to hide ⇱

#https://www.collabora.com/news-and-blog/blog/2019/11/26/gstreamer-windows/

@echo off

set PY_PATH=C:\Python37
set SOURCE_DIR=c:\dev\build-me\
set GSTREAMER_1_0_ROOT_X86_64=%SOURCE_DIR%\86_64\
set GST_SRC_BUILD_PATH=%SOURCE_DIR%gst-build\build\subprojects\
set GST_PLUGIN_PATH=%GST_SRC_BUILD_PATH%gst-plugins-good;%GST_SRC_BUILD_PATH%gst-plugins-bad;%GST_SRC_BUILD_PATH%gst-plugins-base
git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git
set PATH=%PATH%;%PY_PATH%;%PY_PATH%\Scripts;%PY_PATH%\Lib\site-packages;C:\Program Files\Git\cmd;%GSTREAMER_1_0_ROOT_X86_64%bin
cd %SOURCE_DIR%\gst-build

echo SOURCE_DIR=%SOURCE_DIR%
echo .
echo GSTREAMER_1_0_ROOT_X86_64=%GSTREAMER_1_0_ROOT_X86_64%
echo .
echo GST_SRC_BUILD_PATH=%GSTREAMER_1_0_ROOT_X86_64%
echo .
echo GST_PLUGIN_PATH=%GST_SRC_BUILD_PATH%
echo .
echo %GST_PLUGIN_PATH%

echo =======================================
echo NOW RUN in Visual Studio 2019 \ x64 Native Tools Command Prompt
echo .
echo meson --prefix=%GSTREAMER_1_0_ROOT_X86_64% build
echo .
echo ninja -C build

the docker didn't work out for me
froze on last step (pip install meson)

install docker for windows, and enable windows containers.

git clone https://gitlab.freedesktop.org/gstreamer/gst-ci/
cd gstr-ci
docker build .
wait....
1607: Pulling from windows/servercore
3889bb8d808b: Downloading [=============>                                     ]  1.112GB/4.07GB                         
57e8a97eaa75: Downloading [=================================>                 ]  1.092GB/1.65GB

code build on windows

installing on windows
get windows gstreamer files, MSVC edition. link

VS2019 Template - gstsample-console.zip
- place template zip in “C:\Users\user\Documents\Visual Studio 2019\My Exported Templates\”
- remove broken source and add “real” source file- like basic-tutorial-2.c
- build change to X64
set you environment params correctly

set GSTREAMER_1_0_ROOT_X86_64=C:\gstreamer\1.0\x86_64\

watch out for cpp extension generated by default on vs2019 “new project”, should be .c

in (long)

1) C/C++ → Additional Include Directories → define your include paths such as

$(GSTREAMER_1_0_ROOT_X86_64)\lib\glib-2.0\include;$(GSTREAMER_1_0_ROOT_X86_64)\include\gstreamer-1.0;$(GSTREAMER_1_0_ROOT_X86_64)\include\glib-2.0\;$(GSTREAMER_1_0_ROOT_X86_64)\include\glib-2.0\glib;%(AdditionalIncludeDirectories)

2) Linker → General → Adding Library Directories → write your lib directory path such as

$(GSTREAMER_1_0_ROOT_X86_64)\lib;%(AdditionalLibraryDirectories)

3) Linker → Input → Additional Dependencies → Write your .lib files you want to use such as

gobject-2.0.lib;glib-2.0.lib;gstreamer-1.0.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)

plugins

vision camera gstreamer plugin gst-plugins-vision - with working readme inside for ubuntu
pipviz -the visual editor for gstreamer - github
gst-video-analytics - GStreamer Video Analytics (GVA) Plugin

DIY plugin

we actually build upon the work of others. the only plugin i work on is not a brand new one.

tldr: you first run gst-plugins-bad/tools/gst-project-maker, then in the new project plugin folder you run tools/gst-elemnt-maker then you ./autogen and make

#some dependencies
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev indent

cd ~/dev

git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad
cd gst-plugins-bad
NOCONFIGURE=1 ./autogen.sh 
#tip from ridgerun
sudo cp common/gst-indent /usr/local/bin/

./gst-plugins-bad/tools-gst-project-maker mv4dec

cd gst-mv4


cat plugin/gstmv4dec.c
/* This file should be replaced by element source generated by
 * gst-element-maker, or by your own source code.  To generate suitable
 * element source using gst-element-maker, run:
 *
 *   gst-element-maker  mv4dec BASE_CLASS
 *
 * Where BASE_CLASS is replaced by one of the base class templates,
 * such as basesrc, basetransform, audiofilter, videofilter2, etc.
 * Then copy the resulting gstmv4dec.c file over this file, and
 * gstmv4dec.h over gstmv4dec.h.
 */
/* The rest of this file is shim code to allow the project to compile */

so…. choose your base class

tools/element-templates/
audiodecoder  baseparse      gobject           sinkpad-template-video  srcpad-template-video
audioencoder  basesink       sinkpad           srcpad                  videodecoder
audiofilter   basesrc        sinkpad-audio     srcpad-audio            videoencoder
audiosink     basetransform  sinkpad-simple    srcpad-simple           videofilter
audiosrc      element        sinkpad-template  srcpad-template         videosink

run the element maker in that folder

cd ~/dev/gst-mv4/plugins
~/dev/gst-plugins-bad/tools/gst-element-maker mv4dec videoencoder
cd ..
./autogen.sh
make && sudo make install

note the location of the output, its not the usual /usr/include/gstreamer-1.0

alternative

“A really minimal Gstreamer plugin template without the libtool crap”
https://github.com/floe/gst-plugin-minimal

Profiling

Latency Tracer > Measure the time it took for each buffer to travel from source to sink (src)

in powershell

$env:GST_DEBUG="GST_TRACER:7"
$env:GST_TRACERS="latency(flags=pipeline+elemen+reported)"
gst-launch-1.0 videotestsrc ! fpsdisplaysink

in unix

GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+elemen+reported)" \
gst-launch-1.0 videotestsrc ! fpsdisplaysink

see what happens using gst_debug-viewer Debugging Gstreamer Pipelines- talk

cd /opt/gst/gst-build/gst-devtools/debug-viewer
./gst-debug-viewer /opt/mv/test/log2

test the output file with fpsdisplaysink

gst-play-1.0 test.mkv  --videosink=fpsdisplaysink

Sync

how to dump GstBuffer timestamps

You can put an 'identity silent=false' in the pipeline, and use gst-launch-1.0 -v for verbose output of all traffic passing that point

GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+elemen+reported)" \
gst-launch-1.0 videotestsrc ! fpsdisplaysink

https://fosdem.org/2016/schedule/event/synchronised_gstreamer/ slides and forum thread

static GstNetTimeProvider *
create_net_clock (guint16 *port)
{
  GstClock *clock;
  GstNetTimeProvider *net_time;

https://github.com/ford-prefect/gst-sync-server - nice client/server

validate

QrLipsync is an audio-video latency (also referred to as lipsync) measurement and validation tool.
more from Arun #1, #3 and his code
Synchronised ... with GStreamer (2015)

RGBD

Dot Files

generating-gstreamer-pipeline-graphs

install graphviz
make sure you have

GST_DEBUG_DUMP_DOT_DIR=c:\temp\dot

note: videoconvert shouldn't add cpu time if both color-formats are ok with each other (BGR is accepted by matoskamux in our case)

also

PS C:\dev> gst-launch-1.0.exe --help-gst
GStreamer Options
  --gst-debug-help                  Print available debug categories and exit
  --gst-debug-level=LEVEL           Default debug level from 1 (only error) to 9 (anything) or 0 for no output
  --gst-debug=LIST                  Comma-separated list of category_name:level pairs to set specific levels for the individual categories. Example: GST_AUTOPLUG:5,GST_ELEMENT_*:3

other things

libxcam (link) - a project for extended camera(not limited in camera) features and focus on image quality improvement and video analysis. There are lots features supported in image pre-processing, image post-processing and smart analysis. This library makes GPU/CPU/ISP working together to improve image quality.

GPU

 time gst-launch-1.0 videotestsrc num-buffers=1000 ! "video/x-raw, width=4000, height=2000" ! glupload ! glcolorconvert ! gldownload ! video/x-raw,format=BGRx ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:09.868270205
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

real	0m9.999s
user	0m9.927s
sys	0m0.110s

Python

brave

a gui/frontend web based. for gstreamer github.com/bbc/brave

curl -X PUT -d '{"type": "tcp_client", "host": "0.0.0.0", "port":13001}' http://localhost:5000/api/inputs

pip3 install --user pipenv
export PATH="${HOME}/.local/bin:$PATH"
pipenv install
git clone https://github.com/bbc/brave.git
cd brave
pipenv run ./brave.py

for pipenv you should add this to your environment vars (mote at how to)

export PATH="${HOME}/.local/bin:$PATH"

debug

fool gst to disregard a filter - src

just a nice debug tool, to find symbols

user@ubu:~/dev/ffmpeg/RELEASE/lib$ nm -DC *.so -o | grep av_rescale_q
libavcodec-ffmpeg.so:                 U av_rescale_q
libavdevice-ffmpeg.so:                 U av_rescale_q
libavdevice-ffmpeg.so:                 U av_rescale_q_rnd
libavfilter-ffmpeg.so:                 U av_rescale_q
libavfilter-ffmpeg.so:                 U av_rescale_q_rnd
libavformat-ffmpeg.so:                 U av_rescale_q
libavformat-ffmpeg.so:                 U av_rescale_q_rnd
libavutil-ffmpeg.so:0000000000021da0 T av_rescale_q
libavutil-ffmpeg.so:0000000000021d70 T av_rescale_q_rnd
user@ubu:~/dev/ffmpeg/RELEASE/lib$ readelf -a libavutil-ffmpeg.so | grep av_rescale_q
   152: 0000000000021da0    10 FUNC    GLOBAL DEFAULT   13 av_rescale_q@@LIBAVUTIL_54
   372: 0000000000021d70    33 FUNC    GLOBAL DEFAULT   13 av_rescale_q_rnd@@LIBAVUTIL_54

output

(base) ubuntu@ubu:~/dev/gst/gst-plugin-ueye$ sudo make install
Making install in src
make[1]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[2]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[2]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/usr/local/lib/gstreamer-1.0'
 /bin/bash ../libtool   --mode=install /usr/bin/install -c   libueyeplugin.la '/usr/local/lib/gstreamer-1.0'
libtool: install: /usr/bin/install -c .libs/libueyeplugin.so /usr/local/lib/gstreamer-1.0/libueyeplugin.so
libtool: install: /usr/bin/install -c .libs/libueyeplugin.lai /usr/local/lib/gstreamer-1.0/libueyeplugin.la
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/local/lib/gstreamer-1.0
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/local/lib/gstreamer-1.0
 
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'
 
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[2]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[1]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[1]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
make[2]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
make[1]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye'

media

fresh_install_script.txt
fan_on_off.txt
enc001.txt
launch.txt

live streaming

https://cloud.google.com/video-intelligence/docs/streaming/live-streaming

CheatSheet

*https://developer.toradex.com/knowledge-base/video-playback-(linux)
*https://wiki.xiph.org/index.php?title=GST_cookbook
*http://wiki.oz9aec.net/index.php/Gstreamer_cheat_sheet

Table of Contents