Differences

This shows you the differences between two versions of the page.

--- raw [2020/03/29 17:12] – yair
+++ raw [2024/06/30 17:30] (current) – yair
@@ Line 1: / Line 1: @@
 ====GRAY8 vs RTP====
 **problem**: \\
-single plane raw formats are not part of [[https://tools.ietf.org/html/rfc4175|rfc4175]], and as such 2nd class citizens to gstreamer, RTSP and [[https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/pixfmt-grey.html|anywhere]].
+single plane raw formats are not part of [[https://tools.ietf.org/html/rfc4175|rfc4175]], and as such 2nd class citizens to gstreamer, RTSP or [[https://www.kernel.org/doc/html/v4.10/media/uapi/v4l/pixfmt-grey.html|anywhere]] else.
 **solution**: \\
-map the GRAY8 **one plane** source into a quarter width **four plane** RGBA output image\\
+an image format is **just a convention**, internally they are byte(octet) streams. we can re-package a datastream to another format, as long as we mind our strides.
-we will then remap it to Y8 on the receiving side. \\
-the trick is to use gstreamer built in forma spec and let it optimize the remapping.
+view the image format as just a stream of octets. here is our stream in Y8 format
+       +--+--+--+--+ +--+--+--+--+
+       |Y1|xx|xx|xx| |Y1|xx|xx|xx| ...
+       +--+--+--+--+ +--+--+--+--+
+and again, data quartered and represented in 8 bit depth and with stride of 4
+       +--+--+--+--+ +--+--+--+--+
+       |R0|G0|B0|A0| |R1|G1|B1|A1| ...
+       +--+--+--+--+ +--+--+--+--+
+note the above represents two sequential pixels.
+the trick is to use gstreamer built in [[https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c#formats|format spec]] and let that handle the conventions.
 the [[https://www.collabora.com/news-and-blog/blog/2016/02/16/a-programmers-view-on-digital-images-the-essentials/|hammer]]
@@ Line 24: / Line 35: @@
 00000000 (0x7f369c00a8c0): eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb  ................
 00000000 (0x556228700560): eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb  ................
-┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
-│00000000│ eb eb eb eb eb eb eb eb ┊ eb eb eb eb eb eb eb eb │××××××××┊××××××××│
-└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘
 </code>
 **note**: why white pixel is not FF=(255)?\\
-gstreamer uses YUV colorimetry when generating a GRAY8 test image and YUV doesnt use the entire 0-255 space. [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/738|issue]]
+gstreamer uses YUV colorimetry when generating a GRAY8 test image \\
+and YUV doesnt use the entire 0-255 space. [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/738|issue]]
+**note2**: this example is running on linux/debian, \\
+to adapt to windows using powershell replace ''\'' with ''`'' and \\
+place quotes around  ''video/x-raw, format=GRAY8, width=4, height=4''
+====FakeRawPay====
+gray8 raw video transported as RTP payload via a "fake" rgba parsing step\\
+conforming to [[https://tools.ietf.org/html/rfc4175|RFC4157]]\\
+pay
+<code dot>
+gst-launch-1.0 -v \
+  videotestsrc pattern=white is-live=1 \
+  ! video/x-raw, format=GRAY8, width=4, height=4, framerate=30/1 \
+  ! rawvideoparse width=1 height=4 format=rgba \
+  ! rtpvrawpay \
+  ! queue \
+  ! udpsink host=localhost port=5000
+</code>
+depay and back to gray8
+<code dot>
+gst-launch-1.0 -v \
+  udpsrc port=5000 \
+  ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)RGBA, depth=(string)8, width=(string)1, height=(string)4, colorimetry=(string)SMPTE240M, payload=(int)96" \
+   ! rtpvrawdepay \
+   ! queue \
+   ! rawvideoparse width=4 height=4 format=gray8 \
+   ! videoconvert \
+   ! fpsdisplaysink
+</code>
 ====strides and offsets====
-note: we dont need to dig into this as we use the capabilities as listed in format_info. \\
-but hopefully this section will advance your understanding.
+there are caveats to this approach (yuv). get the hammer and study the [[https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c#formats|design document]].
@@ Line 128: / Line 168: @@
 </hidden>
-See raw video format [[https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c#formats|design document]]
-====FakeRawPay====
-gray8 raw video transported as RTP payload via a "fake" rgba parsing step\\
-conforming to [[https://tools.ietf.org/html/rfc4175|RFC4157]]\\
-pay
-<code dot>
-gst-launch-1.0 -v \
-  videotestsrc pattern=white is-live=1 \
-  ! video/x-raw, format=GRAY8, width=4, height=4, framerate=30/1 \
-  ! rawvideoparse width=1 height=4 format=rgba \
-  ! rtpvrawpay \
-  ! queue \
-  ! udpsink host=localhost port=5000
-</code>
-depay and back to gray8
-<code dot>
-gst-launch-1.0 -v \
-  udpsrc port=5000 \
-  ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)RGBA, depth=(string)8, width=(string)1, height=(string)4, colorimetry=(string)SMPTE240M, payload=(int)96" \
-   ! rtpvrawdepay \
-   ! queue \
-   ! rawvideoparse width=4 height=4 format=gray8 \
-   ! videoconvert \
-   ! fpsdisplaysink
-</code>
-====raw parsing====
-<code c>
-gst-launch-1.0 videotestsrc pattern=white num-buffers=1 \
-! video/x-raw, format=GRAY8, width=4, height=4 \
-! identity dump=1 \
-! filesink location=white.gray8 -q
-gst-launch-1.0  multifilesrc location=white.gray8 num_buffers=1 \
-! rawvideoparse format=rgba width=1 height=4 \
-! identity dump=1 \
-! filesink location=white.rgba -q
-</code>
- [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/738|issue]] with GRAY8 and gstreamer
 <hidden more digging>
@@ Line 212: / Line 208: @@
 </hidden>
-[[https://github.com/torvalds/linux/blob/master/include/uapi/drm/drm_fourcc.h#L174|fourcc]] on linux
+[[https://github.com/torvalds/linux/blob/master/include/uapi/drm/drm_fourcc.h#L174|fourcc]] on linux\\
+[[https://docs.microsoft.com/en-us/windows/win32/medfound/image-stride?redirectedfrom=MSDN|MSDN on stride]]