Wednesday, October 3, 2012

Video to Texture Streaming (Part 3) - i.MX6 processor

Hi There Again, sorry for this long time without posting anything, I have being running out of time to deal with cool stuffs (unfortunately), but I have something nice to share.
Continuing with the video to texture streaming with the i.MX6 processors, we saw so far how to decode the video file, now we need to write the data fast as possible into the GPU buffer in order to get a good frame rate of our texture even when using high resolution media.

In order to do that, we have an API that will make our job easier as possible, and the main functions of this API are:

1 - GL_API GL_APIENTRY glTexDirectVIV (GLenum Target, GLsizei Width, GLsizei Height, GLenum Format, GLvoid **Pixels);

2 - GL_API GL_APIENTRY glTexDirectVIVMAP (GLenum Target, GLsizei Width, GLsizei Height, GLenum Format, GLvoid **Logical, Const GLuint *Physical);

3 - GL_API GL_APIENTRY glTexDirectInvalidateVIV (GLenum Target);

what are all those parameters ???

Answer:
1 - Target, Width, Height and Format have the same meaning with glTexImage2D. Width alighment by 16 pixels is required. Example:
Target: GL_TEXTURE_2D Width: 640 Height: 480 Format: GL_VIV_YV12, GL_VIV_NV12, GL_VIV_YUY2, GL_VIV_UYVY, GL_VIV_NV21

2 - Pixels (first function) is the pointer created by the driver, which means that any data you write using this pointer you are going to write directly to the GPU buffer.

3 - Logical (MAP function), is a pointer to the logical address of the application-defined buffer to the texture.

4 - Physical (MAP function), is a pointer to the physial address of the application-defined buffer to the texture, of ~0 if no physical address has been provided.

Alright, we have all the parameters and its description, so, how do we use these functions?

First of all, we need to declare our pointers and then call the function and let the driver allocate memory to us:

GLvoid *pTexel;

The second step is to use the API and get our pointer pointing to the GPU buffer:

glTexDirectVIV (GL_TEXTURE_2D, width, height, GL_VIV_YV12, &pTexel);
GLint error = glGetError ();
if (error != GL_NO_ERROR)
{
     printf ("\nError, glTexDirectVIV 0x%08x\n", error)
}

This function should be called just once in your code! (this was my mistake for a long time.....)

Now, somewhere in your code (in a loop function), you can use the following code to write the data:

glBindTexture (GL_TEXTURE_2D, texture);
memmove (pTexel, my_texture_data, w * h * 3/2);
glTexDirectInvalidateVIV (GL_TEXTURE_2D);

This is not fully optmized yet, if you check this post, you are going to see that we are already performing a memmove to copy the frame data into our global buffer. You can play with this code and remove one of these memmoves.

Also, the Gstreamer code is capturing the frame in RGB format, and you will need YUV format, you can change the caps format to:

// RGB
//gst_element_link_filtered (pFfConv, __videosink, gst_caps_new_simple ("video/x-raw-rgb","bpp",G_TYPE_INT,16, NULL));

// YUV12
gst_element_link_filtered (pFfConv, __videosink, gst_caps_new_simple ("video/x-raw-yuv", "format", GST_TYPE_FOURCC, GST_MAKE_FOURCC ('Y', 'V', '1', '2'), NULL));

The full application can be downlodaded here.

EOF !

28 comments:

  1. Related question. I am having much trouble getting openCV on the imx6 (sabre-lite). Do you have any advice or pointers? Would love to just download the latest openCV 2.4.2 libraries for ARM. Thanks.

    Ron

    ReplyDelete
    Replies
    1. Hi Ron, I'm using OpenCV-1.1, OpenCV-2.1 or higher needs cmake and a different build to get it working with ltib. OpenCV-1.1 works pretty fine there.. I'm working to get a full compilation of the latest, will post it soon as I get it done.

      Delete
    2. I already posted how to install opencv 2.4.x library on the imx6 bsp =)

      Delete
    3. yocto also has recipes for opencv, fyi. its as simple as running 'bitbake opencv' :-)

      Delete
    4. hi,

      how do you bitbake opencv for sabrelite platform?

      Delete
  2. In your examples you are only referring to glTexDirectVIV, what about the other function >glTexDirectVIVMAP? How does it work, could it be used to accomplish the same task (streaming video to texture)?

    ReplyDelete
  3. this function is used when you are going to use a shared memory if I'm not wrong, if for example, if you are using the VPU, you can pass the VPU virtual and physical address to this function and then get the data into the GPU buffer.

    ReplyDelete
    Replies
    1. malloc() allocated memory passed to glTexDirectVIVMAP() causes error GL_INVALID_VALUE (0x00000501). However, when I allocated with glTexDirectVIV() and passed it to glTexDirectVIVMAP(), it worked fine. It appears that glTexDirectVIVMAP() requires GPU memory as its input to work properly.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. I used your above link but I am running yocto so I wrote converted your ltib based makefile into a recipe and I get the following compile error:

    | glutils.cpp: In member function 'bool GLUtils::GLInit()':
    | glutils.cpp:59:53: error: too few arguments to function '_FBDisplay* fbGetDisplay(void*)'
    | In file included from /home/blah/fsl-community-bsp/build/tmp/sysroots/imx6qsabrelite/usr/include/EGL/eglplatform.h:51:0,
    | from /home/blah/fsl-community-bsp/build/tmp/sysroots/imx6qsabrelite/usr/include/EGL/egl.h:47,
    | from glutils.h:36,
    | from glutils.cpp:21:
    | /home/blah/fsl-community-bsp/build/tmp/sysroots/imx6qsabrelite/usr/include/EGL/eglvivante.h:53:1: note: declared here
    | make: *** [glutils.o] Error 1
    | make: *** Waiting for unfinished jobs....
    | glplane.cpp: In member function 'void GLPlane::PlaneCreate(float, float)':
    | glplane.cpp:205:59: error: 'glTexDirectVIV' was not declared in this scope
    | glplane.cpp: In member function 'void GLPlane::PlaneSetTexBuf(char*, int, int)':
    | glplane.cpp:410:41: error: 'glTexDirectInvalidateVIV' was not declared in this scope
    | glplane.cpp: In function 'void PlaneGPUBufferAccess()':
    | glplane.cpp:812:38: error: 'glTexDirectVIV' was not declared in this scope
    | make: *** [glplane.o] Error 1
    | gstcontrol.cpp: In member function 'void GSTVideoControl::GSTLoopFunction()':
    | gstcontrol.cpp:169:13: error: 'usleep' was not declared in this scope
    | make: *** [gstcontrol.o] Error 1
    | main.cpp: In function 'int main(int, char**)':
    | main.cpp:144:28: error: 'gettimeofday' was not declared in this scope
    | make: *** [main.o] Error 1
    | ERROR: oe_runmake failed

    Full log is here:
    http://pastebin.com/piYiuKMh

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  6. fyi, i fixed the main.cpp error (by adding #include sys/time.h ) but the glplane.cpp errors remain. Do you have any idea about how to solve this? Thanks

    ReplyDelete
    Replies
    1. I was trying to find the error reading your log file and it seemed to be link error, maybe you receipt was linking the file wrongly.

      Delete
  7. fixed all the issues...had to add wrappers for the undefined functions in wrapper.c

    ReplyDelete
    Replies
    1. Hi Andrew,

      glad to see you fixed the issues, are you now able to run this app on yocto ?

      Delete
  8. Hi Andrew,

    Could you please share your Yocto recipe with the changes you made?

    Thanks,

    Fabio Estevam

    ReplyDelete
    Replies
    1. My recipe is not yet building. I'll share it as soon as its done

      Delete
    2. I used your simple_gpu_player binary from the above git link as well as built your code for yocto and all I get is a green blob on screen rotating very slowly (1fps).Here is the error I get when I run it:
      glTexDirectVIV: 0x00000000

      (simple_gpu_player:2110): GLib-GObject-CRITICAL **: g_object_set: assertion `G_IS_OBJECT (object)' failed

      Here is complete output I get on the serial prompt

      root@imx6qsabrelite:~# ./simple_gpu_player /home/root/Wildlife.avi

      Video file: file:///home/root/Wildlife.avi
      width: 1920
      height: 1080
      glTexDirectVIV: 0x00000000

      (simple_gpu_player:2110): GLib-GObject-CRITICAL **: g_object_set: assertion `G_IS_OBJECT (object)' failed
      Aiur: 3.0.1
      Core: AVI_PARSER_03.05.00 build on Aug 27 2012 18:12:01
      mime: video/x-msvideo
      file: /usr/lib/imx-mm/parser/lib_avi_parser_arm11_elinux.so
      Content Info:
      URI:
      file:///home/root/Wildlife.avi
      Idx File:
      /home/root/.aiur/.home.root.Wildlife.avi.aidx
      Seekable : Yes
      Size(byte): 10323978

      Movie Info:
      Seekable : Yes
      Live : No
      Duration : 0:00:30.048000000
      ReadMode : File
      Track : 2

      Track 00 [video_0000] Enabled
      Duration: 0:00:29.429400000
      Language: und
      Mime:
      video/x-h264, parsed=(boolean)true, width=(int)1024, height=(int)1024, framerate
      =(fraction)30000/1001
      [INFO] Product Info: i.MX6Q/D/S
      vpudec versions :)
      plugin: 3.0.1
      wrapper: 1.0.24(VPUWRAPPER_ARM_LINUX Build on Mar 10 2013 20:52:16)
      vpulib: 5.4.6
      firmware: 2.1.5.32515
      Track 01 [audio_0000] Enabled
      Duration: 0:00:30.048000000
      Language: und
      Mime:
      audio/mpeg, mpegversion=(int)1, parsed=(boolean)true, framed=(boolean)true, chan
      nels=(int)2, rate=(int)48000, bitrate=(int)0
      [INFO] bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0

      Process time = 1455 ms


      Process time = 1438 ms


      Process time = 1439 ms


      Process time = 1434 ms


      Process time = 1439 ms


      Process time = 1438 ms


      Process time = 1438 ms


      root@imx6qsabrelite:~#

      Could you please help me with what this is?

      Thanks

      Delete
    3. ok so I was feeding in the wrong argc values. I called the simple_gpu_player with the correct width and height parameter inputs and I can see the video rendered in a texture but the update rate is still ~1 FPS.
      Here is the full output:
      root@imx6qsabrelite:~# ./simple_gpu_player /home/root/bbb1.avi 1280 720

      Video file: file:///home/root/bbb1.avi
      width: 1280
      height: 720
      glTexDirectVIV: 0x00000000

      (simple_gpu_player:2219): GLib-GObject-CRITICAL **: g_object_set: assertion `G_IS_OBJECT (object)' failed
      Aiur: 3.0.1
      Core: AVI_PARSER_03.05.00 build on Aug 27 2012 18:12:01
      mime: video/x-msvideo
      file: /usr/lib/imx-mm/parser/lib_avi_parser_arm11_elinux.so
      Content Info:
      URI:
      file:///home/root/bbb1.avi
      Idx File:
      /home/root/.aiur/.home.root.bbb1.avi.aidx
      Seekable : Yes
      Size(byte): 53465686

      Movie Info:
      Seekable : Yes
      Live : No
      Duration : 0:01:29.792000000
      ReadMode : File
      Track : 2

      Track 00 [video_0000] Enabled
      Duration: 0:01:29.749857000
      Language: und
      Mime:
      video/mpeg, mpegversion=(int)4, parsed=(boolean)true, width=(int)1280, height=(i
      nt)720, framerate=(fraction)12000019/500000
      [INFO] Product Info: i.MX6Q/D/S
      vpudec versions :)
      plugin: 3.0.1
      wrapper: 1.0.24(VPUWRAPPER_ARM_LINUX Build on Mar 10 2013 20:52:16)
      vpulib: 5.4.6
      firmware: 2.1.5.32515
      Track 01 [audio_0000] Enabled
      Duration: 0:01:29.792000000
      Language: und
      Mime:
      audio/x-ac3, channels=(int)5, rate=(int)48000, bitrate=(int)448000, framed=(bool
      ean)true
      [INFO] bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0

      Process time = 1384 ms


      Process time = 1372 ms


      Process time = 1374 ms


      Process time = 1378 ms


      Process time = 1374 ms


      Process time = 1376 ms


      Process time = 1377 ms


      Process time = 1381 ms


      Delete
    4. so can you please help me figure out why the FPS is still ~1 FPS for this case? I am using yocto danny. Haven't tried dylan yet

      Delete
    5. Hi Andrew, I'm not into yocto yet, please, try posting your question on this list: meta-freescale@yoctoproject.org

      regards,
      Andre

      Delete
  9. Hello Andre,

    Could you please post your fps numbers for both options: with glTexImage2D and with glTexDirectVIV.

    ReplyDelete
    Replies
    1. Hi there,

      Unfortunately I couldn´t find the email with the notes =(, as I remember I was able to get something around 10fps with glTexImage2D, and using glTexDirectVIV it was around 40fps. I´m going to work on this demo again and soon as I get it working on our new bsp I let you know. But the perfomance improve with this approach is significantly.

      regards,
      Andre

      Delete
    2. Hello Andre,

      Hmmm... Your numbers look really good.

      In the latest release of vivante libs more formats are supported by glTexDirectVIV. In addition to GL_VIV_YV12, GL_VIV_NV12, GL_VIV_YUY2, GL_VIV_UYVY the following formats are now supported: GL_RGB565, GL_RGB, GL_RGBA.

      I tried to replace glTexImage2D by glTexDirectVIV for large 1280x960 GL_RGBA textures. The result was disappointing - I observed significant performance degradation: from 30fps to 0.8 fps. The bottleneck seems to be in glTexDirectInvalidateVIV function. I have not yet figured out what could be wrong.

      Delete
    3. Upd.

      I tried the latest-latest-latest vivante release available here: https://community.freescale.com/docs/DOC-94809.

      Now glTexDirectVIV works as fast as glTexImage2D on GL_RGBA textures. So no performance degradation any more, though no performance boost for GL_RGBA textures.

      Delete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Andre, do you have example code to access correct y plane texture pixels from the fragment shader when I use glTexDirectVIVMAP or glTexDirectVIV and one of planar YUV formats?

    ReplyDelete
    Replies
    1. I posted the same question in freescale forum:

      https://community.freescale.com/message/341498#341498

      Delete