=====Gstreamer=====
{{ :530px-gstreamer-logo.svg.png?size=200}}
gstreamer the project with the [[https://github.com/GStreamer/gstreamer|best readme ever]]\\ \\
===gstreamer 101===
a gstreamer application runs a pipeline, composed of elements. the data flow is composed of buffers and events .
By linking a source [[https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html#elements|element]] with zero or more filter-like elements and finally a sink element, you set up a media **pipeline**. Data will flow through the elements. This is the basic concept of media handling in GStreamer.
A [[https://gstreamer.freedesktop.org/documentation/application-development/basics/pads.html#pads|pad type]] is defined by two properties: its direction and its availability... \\
GStreamer defines two pad directions: source pads and sink pads. This terminology is defined from the view of within the element: elements receive data on their sink pads and generate data on their source pads. Schematically, sink pads are drawn on the left side of an element, whereas source pads are drawn on the right side of an element. In such graphs,** data flows from left to right**. [[https://gstreamer.freedesktop.org/documentation/application-development/basics/pads.html?gi-language=c|more on pads]]
The **data** flowing through a pipeline consists of a combination of buffers and events. [[https://gstreamer.freedesktop.org/documentation/application-development/basics/data.html#buffers|Buffers]] contain the actual media data. [[https://gstreamer.freedesktop.org/documentation/application-development/basics/data.html#events|Events]] contain control information
===cheatsheet===
#launch a camera and play max fps (set in eeprom)
gst-launch-1.0 -v -m autovideotest ! videoconvert ! fpsdisplaysink video-sink=fakesink text-overlay=false
gst-launch-1.0 filesrc location=test-single-30fps-synced.mkv ! decodebin ! videoconvert ! fpsdisplaysink video-sink=glimagesink
====install====
approach that will compile and copy gstreamer to bare metal.\\
we can also run directly from docker (wip)
there is an install script to be automated in the package folder
the following data is for the know how
=== gst-build ===
**UBUNTU**\\
ubuntu 18 comes with gstreamer 14.x built it , but it is an old release. missing some nice stuff. \\
better build using gst-build
sudo apt install -y apt-utils bison build-essential cmake curl flex gir1.2-gudev-1.0 git gudev-1.0-dev libbz2-dev libcairo2-dev libclang-dev libgirepository1.0-dev libglib2.0-dev libgtk-3-dev libgtk2.0-dev libjson-glib-dev libncurses5-dev libncursesw5-dev liborc-0.4-dev libreadline-dev libsqlite3-dev libssl-dev libudev-dev libxml2-dev libxslt1-dev libyaml-dev llvm llvm-dev ninja-build python3-pip tk-dev vainfo xz-utils zlib1g-dev
pip3 install meson --user
export PATH=$PATH:~/.local/bin
git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git && cd gst-build
meson \
-Dbuildtype=release \
-Dgtk_doc=disabled \
-Dpython=disabled \
-Dugly=disabled \
-Dges=disabled \
build
#see below to run gst in private mode (uninstalled) or systemwide (install)
# add to env the following in later case
export LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu/
then run 'ninja uninstalled' or install over current version using 'ninja install', make sure you dirty remove gstreamer 1.4 (see below)
to run uninstalled
ninja -C /opt/gst/gst-build/build uninstalled
to "install" gstreamer over the ubuntu package do
sudo ninja -C /opt/gst/gst-build/build install
to **remove/uninstall** gstreamer
sudo -s
rm /usr/local/bin/gst-*
rm /usr/bin/gst-*
rm -fr /usr/local/include/gstreamer-1.0/
rm -fr /usr/include/gstreamer-1.0/
rm -fr /usr/local/libexec/gstreamer-1.0/
rm /usr/lib/x86_64-linux-gnu/libgst*
rm /usr/local/lib/x86_64-linux-gnu/libgst*
**WINDOWS**\\
https://www.collabora.com/news-and-blog/blog/2019/11/26/gstreamer-windows/\\
using following script\\
had to disable
*msdk (temp until installing Media SDK for windows)
*libxml
* rename/delete the libxml2.wrap file from gst-build/subproject dir
* disable GES in meson_options.txt
#https://www.collabora.com/news-and-blog/blog/2019/11/26/gstreamer-windows/
@echo off
set PY_PATH=C:\Python37
set SOURCE_DIR=c:\dev\build-me\
set GSTREAMER_1_0_ROOT_X86_64=%SOURCE_DIR%\86_64\
set GST_SRC_BUILD_PATH=%SOURCE_DIR%gst-build\build\subprojects\
set GST_PLUGIN_PATH=%GST_SRC_BUILD_PATH%gst-plugins-good;%GST_SRC_BUILD_PATH%gst-plugins-bad;%GST_SRC_BUILD_PATH%gst-plugins-base
git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git
set PATH=%PATH%;%PY_PATH%;%PY_PATH%\Scripts;%PY_PATH%\Lib\site-packages;C:\Program Files\Git\cmd;%GSTREAMER_1_0_ROOT_X86_64%bin
cd %SOURCE_DIR%\gst-build
echo SOURCE_DIR=%SOURCE_DIR%
echo .
echo GSTREAMER_1_0_ROOT_X86_64=%GSTREAMER_1_0_ROOT_X86_64%
echo .
echo GST_SRC_BUILD_PATH=%GSTREAMER_1_0_ROOT_X86_64%
echo .
echo GST_PLUGIN_PATH=%GST_SRC_BUILD_PATH%
echo .
echo %GST_PLUGIN_PATH%
echo =======================================
echo NOW RUN in Visual Studio 2019 \ x64 Native Tools Command Prompt
echo .
echo meson --prefix=%GSTREAMER_1_0_ROOT_X86_64% build
echo .
echo ninja -C build
the **docker didn't work out** for me\\
froze on last step (pip install meson)
install docker for windows, and enable windows containers.
git clone https://gitlab.freedesktop.org/gstreamer/gst-ci/
cd gstr-ci
docker build .
wait....
1607: Pulling from windows/servercore
3889bb8d808b: Downloading [=============> ] 1.112GB/4.07GB
57e8a97eaa75: Downloading [=================================> ] 1.092GB/1.65GB
====code build on windows====
*[[https://gstreamer.freedesktop.org/documentation/installing/on-windows.html?gi-language=c|installing on windows]]
*get windows gstreamer files, MSVC edition. [[https://gstreamer.freedesktop.org/data/pkg/windows/1.16.1/|link]]
* [[https://gstreamer.freedesktop.org/data/pkg/windows/1.16.1/gstreamer-1.0-msvc-x86_64-1.16.1.msi|gstreamer-1.0-msvc-x86_64-1.16.1.msi]]
* [[https://gstreamer.freedesktop.org/data/pkg/windows/1.16.1/gstreamer-1.0-devel-msvc-x86_64-1.16.1.msi|gstreamer-1.0-devel-msvc-x86_64-1.16.1.msi]]
* [[https://gstreamer.freedesktop.org/data/pkg/windows/1.16.1/gstreamer-1.0-msvc-x86_64-1.16.1-merge-modules.zip|gstreamer-1.0-msvc-x86_64-1.16.1-merge-modules.zip]]
*VS2019 Template - {{:gstsample-console.zip}}
* place template zip in "C:\Users\user\Documents\Visual Studio 2019\My Exported Templates\"
* remove broken source and add "real" source file- like {{:basic-tutorial-2.c}}
* build change to X64
*set you environment params correctly
set GSTREAMER_1_0_ROOT_X86_64=C:\gstreamer\1.0\x86_64\
*watch out for cpp extension generated by default on vs2019 "new project", should be .c
in ([[https://stackoverflow.com/a/50756979/184085|long]])
1) C/C++ -> Additional Include Directories -> define your include paths such as
$(GSTREAMER_1_0_ROOT_X86_64)\lib\glib-2.0\include;$(GSTREAMER_1_0_ROOT_X86_64)\include\gstreamer-1.0;$(GSTREAMER_1_0_ROOT_X86_64)\include\glib-2.0\;$(GSTREAMER_1_0_ROOT_X86_64)\include\glib-2.0\glib;%(AdditionalIncludeDirectories)
2) Linker -> General -> Adding Library Directories -> write your lib directory path such as
$(GSTREAMER_1_0_ROOT_X86_64)\lib;%(AdditionalLibraryDirectories)
3) Linker -> Input -> Additional Dependencies -> Write your .lib files you want to use such as
gobject-2.0.lib;glib-2.0.lib;gstreamer-1.0.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)
==== plugins====
* **vision camera gstreamer plugin** [[https://github.com/5shekel/gst-plugins-vision|gst-plugins-vision]] - with working readme inside for ubuntu
* pipviz -the visual editor for gstreamer - [[https://github.com/virinext/pipeviz.git|github]]
* gst-video-analytics - [[https://github.com/opencv/gst-video-analytics|GStreamer Video Analytics (GVA) Plugin]]
* {{:intro.gif}}
===DIY plugin===
we actually build upon the work of others. the only plugin i work on is not a brand new one.
**tldr**: you first run //gst-plugins-bad/tools/gst-project-maker//, then in the new project plugin folder you run //tools/gst-elemnt-maker// then you //./autogen// and //make//
#some dependencies
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev indent
cd ~/dev
git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad
cd gst-plugins-bad
NOCONFIGURE=1 ./autogen.sh
#tip from ridgerun
sudo cp common/gst-indent /usr/local/bin/
./gst-plugins-bad/tools-gst-project-maker mv4dec
cd gst-mv4
cat plugin/gstmv4dec.c
/* This file should be replaced by element source generated by
* gst-element-maker, or by your own source code. To generate suitable
* element source using gst-element-maker, run:
*
* gst-element-maker mv4dec BASE_CLASS
*
* Where BASE_CLASS is replaced by one of the base class templates,
* such as basesrc, basetransform, audiofilter, videofilter2, etc.
* Then copy the resulting gstmv4dec.c file over this file, and
* gstmv4dec.h over gstmv4dec.h.
*/
/* The rest of this file is shim code to allow the project to compile */
so.... choose your base class
tools/element-templates/
audiodecoder baseparse gobject sinkpad-template-video srcpad-template-video
audioencoder basesink sinkpad srcpad videodecoder
audiofilter basesrc sinkpad-audio srcpad-audio videoencoder
audiosink basetransform sinkpad-simple srcpad-simple videofilter
audiosrc element sinkpad-template srcpad-template videosink
run the element maker in that folder
cd ~/dev/gst-mv4/plugins
~/dev/gst-plugins-bad/tools/gst-element-maker mv4dec videoencoder
cd ..
./autogen.sh
make && sudo make install
note the location of the output, its not the usual /usr/include/gstreamer-1.0
===alternative===
"A really minimal Gstreamer plugin template without the libtool crap"\\
https://github.com/floe/gst-plugin-minimal
====Profiling====
**Latency Tracer** > Measure the time it took for each buffer to travel from source to sink ({{:fosdem_modern_tools_to_debug_gst_apps.pdf|src}})
in powershell
$env:GST_DEBUG="GST_TRACER:7"
$env:GST_TRACERS="latency(flags=pipeline+elemen+reported)"
gst-launch-1.0 videotestsrc ! fpsdisplaysink
in unix
GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+elemen+reported)" \
gst-launch-1.0 videotestsrc ! fpsdisplaysink
see what happens using gst_debug-viewer [[https://gstconf.ubicast.tv/permalink/v124cbec1db0080d67b9|Debugging Gstreamer Pipelines- talk]]
cd /opt/gst/gst-build/gst-devtools/debug-viewer
./gst-debug-viewer /opt/mv/test/log2
test the output file with fpsdisplaysink
gst-play-1.0 test.mkv --videosink=fpsdisplaysink
====Sync====
how to dump GstBuffer timestamps
You can put an 'identity silent=false' in the pipeline, and use gst-launch-1.0 -v for verbose output of all traffic passing that point
or
GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+elemen+reported)" \
gst-launch-1.0 videotestsrc ! fpsdisplaysink
*[[https://fosdem.org/2016/schedule/event/synchronised_gstreamer/]] [[https://www.slideshare.net/SamsungOSG/synchronised-multidevice-media-playback-with-gstreamer|slides]] and [[http://gstreamer-devel.966125.n4.nabble.com/Synchronizing-multiple-RTP-sources-td4672395.html|forum thread]]
static GstNetTimeProvider *
create_net_clock (guint16 *port)
{
GstClock *clock;
GstNetTimeProvider *net_time;
*https://github.com/ford-prefect/gst-sync-server - nice client/server
*[[https://gstreamer.freedesktop.org/documentation/pango/timeoverlay.html?gi-language=c|timeoverlay]]
*[[https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/clock.html?gi-language=c|clock]]
*[[https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html?gi-language=c|Clocks and synchronization in GStreamer]]
====validate====
{{:cam1-qrcode-blue-30.mp4}}
[[https://github.com/UbiCastTeam/qr-lipsync|QrLipsync]] is an audio-video latency (also referred to as lipsync) measurement and validation tool. \\
more from Arun [[https://arunraghavan.net/2016/11/gstreamer-and-synchronisation-made-easy/|#1]], [[https://arunraghavan.net/2017/01/quantifying-synchronisation-oscilloscope-edition/|#3]] and his [[https://github.com/serg-z/gst-sync-server|code]]\\
[[https://gstconf.ubicast.tv/videos/synchronised-multi-room-media-playback-and-distributed-live-media-processing-and-mixing-with-gstreamer/|Synchronised ... with GStreamer]] (2015)
====RGBD====
*[[https://gstconf.ubicast.tv/videos/3d-cameras-in-gstreamer-rgb-d-camera-support-and-depth-video-compression/|talk]]
*[[https://gitlab.com/aivero/public/gstreamer/gst-rgbd|gst-rgbd git]]
*https://i.imgur.com/wrlAowC.png
*https://i.imgur.com/IeoxlNp.png
*https://gitlab.com/aivero/public/gstreamer/gst-colorizer
*https://github.com/skaegy/gst-realsenseRGBD-streaming
==== Dot Files ====
[[https://embeddedartistry.com/blog/2018/02/22/generating-gstreamer-pipeline-graphs/|generating-gstreamer-pipeline-graphs]]
install [[http://www.graphviz.org/|graphviz]] \\
make sure you have
*[[https://github.com/anthraxx/arch-pkgbuilds/blob/master/xdot/xdot.desktop|.desktop]]
*[[https://github.com/frasercrmck/dotfiles/blob/master/howtos/dot-mimetype.md|mime type]]
*[[https://help.gnome.org/admin/system-admin-guide/stable/mime-types-custom.html.en|configure]]
GST_DEBUG_DUMP_DOT_DIR=c:\temp\dot
note: videoconvert shouldn't add cpu time if both color-formats are ok with each other (BGR is accepted by matoskamux in our case) {{:screenshot_from_2019-09-05_20-28-05.png|pic}}
==== also ====
* [[https://wiki.st.com/stm32mpu/wiki/How_to_profile_video_framerate|How_to_profile_video_framerate]]
* {{:fosdem_modern_tools_to_debug_gst_apps.pdf}}
* {{:kyrylo_polezhaiev_-_profiling_gstreamer_pipelines.pdf}}
PS C:\dev> gst-launch-1.0.exe --help-gst
GStreamer Options
--gst-debug-help Print available debug categories and exit
--gst-debug-level=LEVEL Default debug level from 1 (only error) to 9 (anything) or 0 for no output
--gst-debug=LIST Comma-separated list of category_name:level pairs to set specific levels for the individual categories. Example: GST_AUTOPLUG:5,GST_ELEMENT_*:3
\\ \\
=====other things=====
*libxcam ([[https://github.com/intel/libxcam|link]]) - a project for extended camera(not limited in camera) features and focus on image quality improvement and video analysis. There are lots features supported in image pre-processing, image post-processing and smart analysis. This library makes GPU/CPU/ISP working together to improve image quality.
*[[https://github.com/matthew1000/gstreamer-cheat-sheet|cheta-sheet]]
*[[http://blog.nirbheek.in/2018/03/low-latency-audio-on-windows-with.html| cute latency testing]]
===GPU===
time gst-launch-1.0 videotestsrc num-buffers=1000 ! "video/x-raw, width=4000, height=2000" ! glupload ! glcolorconvert ! gldownload ! video/x-raw,format=BGRx ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:09.868270205
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
real 0m9.999s
user 0m9.927s
sys 0m0.110s
==== Python ====
* [[https://brettviren.github.io/pygst-tutorial-org/pygst-tutorial.html|Python GStreamer Tutorial]]
* [[http://lifestyletransfer.com/how-to-launch-gstreamer-pipeline-in-python/|how-to-launch-gstreamer-pipeline-in-python]]
* [[http://mathieuduponchelle.github.io/2018-02-01-Python-Elements.html?gi-language=undefined|a tutorial on why its a bad idea]]
====brave====
a gui/frontend web based. for gstreamer [[https://github.com/bbc/brave|github.com/bbc/brave]]
curl -X PUT -d '{"type": "tcp_client", "host": "0.0.0.0", "port":13001}' http://localhost:5000/api/inputs
pip3 install --user pipenv
export PATH="${HOME}/.local/bin:$PATH"
pipenv install
git clone https://github.com/bbc/brave.git
cd brave
pipenv run ./brave.py
for pipenv you should add this to your environment vars (mote at [[https://github.com/bbc/brave/edit/master/docs/install_ubuntu.md|how to]])
export PATH="${HOME}/.local/bin:$PATH"
====debug====
*fool gst to disregard a filter - [[https://fhackts.wordpress.com/2018/04/16/temporarily-skipping-vaapi-decoders-in-decodebin/|src]]
just a nice debug tool, to find symbols
user@ubu:~/dev/ffmpeg/RELEASE/lib$ nm -DC *.so -o | grep av_rescale_q
libavcodec-ffmpeg.so: U av_rescale_q
libavdevice-ffmpeg.so: U av_rescale_q
libavdevice-ffmpeg.so: U av_rescale_q_rnd
libavfilter-ffmpeg.so: U av_rescale_q
libavfilter-ffmpeg.so: U av_rescale_q_rnd
libavformat-ffmpeg.so: U av_rescale_q
libavformat-ffmpeg.so: U av_rescale_q_rnd
libavutil-ffmpeg.so:0000000000021da0 T av_rescale_q
libavutil-ffmpeg.so:0000000000021d70 T av_rescale_q_rnd
user@ubu:~/dev/ffmpeg/RELEASE/lib$ readelf -a libavutil-ffmpeg.so | grep av_rescale_q
152: 0000000000021da0 10 FUNC GLOBAL DEFAULT 13 av_rescale_q@@LIBAVUTIL_54
372: 0000000000021d70 33 FUNC GLOBAL DEFAULT 13 av_rescale_q_rnd@@LIBAVUTIL_54
====output====
(base) ubuntu@ubu:~/dev/gst/gst-plugin-ueye$ sudo make install
Making install in src
make[1]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[2]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[2]: Nothing to be done for 'install-exec-am'.
/bin/mkdir -p '/usr/local/lib/gstreamer-1.0'
/bin/bash ../libtool --mode=install /usr/bin/install -c libueyeplugin.la '/usr/local/lib/gstreamer-1.0'
libtool: install: /usr/bin/install -c .libs/libueyeplugin.so /usr/local/lib/gstreamer-1.0/libueyeplugin.so
libtool: install: /usr/bin/install -c .libs/libueyeplugin.lai /usr/local/lib/gstreamer-1.0/libueyeplugin.la
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/local/lib/gstreamer-1.0
----------------------------------------------------------------------
Libraries have been installed in:
/usr/local/lib/gstreamer-1.0
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the 'LD_RUN_PATH' environment variable
during linking
- use the '-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to '/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[2]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[1]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye/src'
make[1]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
make[2]: Entering directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
make[1]: Leaving directory '/home/ubuntu/dev/gst/gst-plugin-ueye'
===media===
{{:fresh_install_script.txt}}\\
{{:fan_on_off.txt}}\\
{{:enc001.txt}}\\
{{:launch.txt}}
=== live streaming ===
* https://cloud.google.com/video-intelligence/docs/streaming/live-streaming
====CheatSheet====
*https://developer.toradex.com/knowledge-base/video-playback-(linux)
*https://wiki.xiph.org/index.php?title=GST_cookbook
*http://wiki.oz9aec.net/index.php/Gstreamer_cheat_sheet