Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
raw [2020/03/29 17:09] – [GRAY8 vs RTP] yairraw [2024/06/30 17:30] (current) – yair
Line 1: Line 1:
 ====GRAY8 vs RTP==== ====GRAY8 vs RTP====
 **problem**: \\ **problem**: \\
-single plane raw formats are not part of [[https://tools.ietf.org/html/rfc4175|rfc4175]], and as such 2nd class citizens to gstreamer, RTSP and [[https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/pixfmt-grey.html|anywhere]]. +single plane raw formats are not part of [[https://tools.ietf.org/html/rfc4175|rfc4175]], and as such 2nd class citizens to gstreamer, RTSP or [[https://www.kernel.org/doc/html/v4.10/media/uapi/v4l/pixfmt-grey.html|anywhere]] else. 
  
 **solution**: \\ **solution**: \\
-map the GRAY8 **one plane** source into a quarter width **four plane** RGBA output image\\ +an image format is **just a convention**, internally they are byte(octet) streams. we can re-package a datastream to another format, as long as we mind our strides.  
-we will then remap it to Y8 on the receiving side. \\ + 
-the trick is to use gstreamer built in forma spec and let it optimize the remapping. +view the image format as just a stream of octets. here is our stream in Y8 format 
 +       +--+--+--+--+ +--+--+--+--+ 
 +       |Y1|xx|xx|xx| |Y1|xx|xx|xx| ... 
 +       +--+--+--+--+ +--+--+--+--+ 
 +and again, data quartered and represented in 8 bit depth and with stride of 4 
 +       +--+--+--+--+ +--+--+--+--+ 
 +       |R0|G0|B0|A0| |R1|G1|B1|A1| ... 
 +       +--+--+--+--+ +--+--+--+--+ 
 + 
 +note the above represents two sequential pixels.  
 + 
 +the trick is to use gstreamer built in [[https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c#formats|format spec]] and let that handle the conventions. 
  
 the [[https://www.collabora.com/news-and-blog/blog/2016/02/16/a-programmers-view-on-digital-images-the-essentials/|hammer]] the [[https://www.collabora.com/news-and-blog/blog/2016/02/16/a-programmers-view-on-digital-images-the-essentials/|hammer]]
Line 24: Line 35:
 00000000 (0x7f369c00a8c0): eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb  ................ 00000000 (0x7f369c00a8c0): eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb  ................
 00000000 (0x556228700560): eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb  ................ 00000000 (0x556228700560): eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb eb  ................
-┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐ 
-│00000000│ eb eb eb eb eb eb eb eb ┊ eb eb eb eb eb eb eb eb │××××××××┊××××××××│ 
-└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘ 
 </code> </code>
-note: why white pixel is not FF=(255)?\\ +**note**: why white pixel is not FF=(255)?\\ 
-gstreamer follows YUV colorimetry when generating a GRAY8 test image and YUV doesnt use the entire 0-255 space. [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/738|issue]]+gstreamer uses YUV colorimetry when generating a GRAY8 test image \\ 
 +and YUV doesnt use the entire 0-255 space. [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/738|issue]] 
 + 
 +**note2**: this example is running on linux/debian, \\ 
 +to adapt to windows using powershell replace ''\'' with ''`'' and \\ 
 +place quotes around  ''video/x-raw, format=GRAY8, width=4, height=4'' 
 + 
 + 
 +====FakeRawPay==== 
 +gray8 raw video transported as RTP payload via a "fake" rgba parsing step\\ 
 +conforming to [[https://tools.ietf.org/html/rfc4175|RFC4157]]\\ 
 +pay 
 +<code dot> 
 +gst-launch-1.0 -v \ 
 +  videotestsrc pattern=white is-live=1 \ 
 +  ! video/x-raw, format=GRAY8, width=4, height=4, framerate=30/1 \ 
 +  ! rawvideoparse width=1 height=4 format=rgba \ 
 +  ! rtpvrawpay \ 
 +  ! queue \ 
 +  ! udpsink host=localhost port=5000 
 +</code>  
 +depay and back to gray8 
 +<code dot> 
 +gst-launch-1.0 -v \ 
 +  udpsrc port=5000 \ 
 +  ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)RGBA, depth=(string)8, width=(string)1, height=(string)4, colorimetry=(string)SMPTE240M, payload=(int)96" \ 
 +   ! rtpvrawdepay \ 
 +   ! queue \ 
 +   ! rawvideoparse width=4 height=4 format=gray8 \ 
 +   ! videoconvert \ 
 +   ! fpsdisplaysink 
 +</code> 
 +   
 + 
 ====strides and offsets==== ====strides and offsets====
-rawvideoparse let you specify the strides and offset, can we get this file to be parsed with that set ? \\ + 
-**(Syntax is <stride,ustride,vstride>)**+there are caveats to this approach (yuv). get the hammer and study the [[https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c#formats|design document]].  
 + 
 + 
 +{{ :pasted:20200328-193549.png?700 }}
  
 <blockquote>**pstride** only exist for simple pixel formats. It represent the** distance <blockquote>**pstride** only exist for simple pixel formats. It represent the** distance
Line 78: Line 123:
 </code> </code>
  
-{{ :pasted:20200328-193549.png?700 }} 
  
 <hidden extended format view> <hidden extended format view>
Line 125: Line 169:
 </hidden> </hidden>
  
-See raw video format [[https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c#formats|design document]] +<hidden more digging>
- +
- +
-====FakeRawPay==== +
-gray8 raw video transported as RTP payload via a "fake" rgba parsing step\\ +
-conforming to [[https://tools.ietf.org/html/rfc4175|RFC4157]]\\ +
-pay +
-<code dot> +
-gst-launch-1.0 -v \ +
-  videotestsrc pattern=white is-live=1 \ +
-  ! video/x-raw, format=GRAY8, width=4, height=4, framerate=30/1 \ +
-  ! rawvideoparse width=1 height=4 format=rgba \ +
-  ! rtpvrawpay \ +
-  ! queue \ +
-  ! udpsink host=localhost port=5000 +
-</code>  +
-depay and back to gray8 +
-<code dot> +
-gst-launch-1.0 -v \ +
-  udpsrc port=5000 \ +
-  ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)RGBA, depth=(string)8, width=(string)1, height=(string)4, colorimetry=(string)SMPTE240M, payload=(int)96" \ +
-   ! rtpvrawdepay \ +
-   ! queue \ +
-   ! rawvideoparse width=4 height=4 format=gray8 \ +
-   ! videoconvert \ +
-   ! fpsdisplaysink +
-</code> +
-   +
- +
-====raw parsing==== +
-<code c> +
-gst-launch-1.0 videotestsrc pattern=white num-buffers=1 \ +
-! video/x-raw, format=GRAY8, width=4, height=4 \ +
-! identity dump=1 \ +
-! filesink location=white.gray8 -q +
- +
-gst-launch-1.0  multifilesrc location=white.gray8 num_buffers=1 \ +
-! rawvideoparse format=rgba width=1 height=4 \ +
-! identity dump=1 \ +
-! filesink location=white.rgba -q +
-</code> +
- +
- [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/738|issue]] with GRAY8 and gstreamer +
- +
 <file cpp /gst-plugins-base/gst-libs/gst/video/video-format.c> <file cpp /gst-plugins-base/gst-libs/gst/video/video-format.c>
  
Line 204: Line 204:
 MAKE_GRAY_FORMAT (GRAY8, "raw video", DPTH8, PSTR1, PLANE0, OFFS0, SUB4, MAKE_GRAY_FORMAT (GRAY8, "raw video", DPTH8, PSTR1, PLANE0, OFFS0, SUB4,
       PACK_GRAY8),       PACK_GRAY8),
-</file> [[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/video/video-format.c|video-format.c]] +</file>  
- +[[https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/video/video-format.c|video-format.c]] 
-[[https://github.com/torvalds/linux/blob/master/include/uapi/drm/drm_fourcc.h#L174|fourcc]] on linux+</hidden>
  
 +[[https://github.com/torvalds/linux/blob/master/include/uapi/drm/drm_fourcc.h#L174|fourcc]] on linux\\
 +[[https://docs.microsoft.com/en-us/windows/win32/medfound/image-stride?redirectedfrom=MSDN|MSDN on stride]]