world leader in high performance signal processing
Trace: » ffmpeg

FFmpeg Overview

FFmpeg is a collection of free software licensed under GNU Lesser General Public License or GNU General Public License (depending on which sub-libraries are included) the provides “a complete solution to record, convert and stream audio and video. It includes libavcodec, the leading audio/video codec library. FFmpeg is developed under Linux, but it can compiled under most operating systems, including Windows.” (source: main site)

The project is made of several components:

  • ''ffmpeg'' is a command line tool to convert one video file format to another. It also supports grabbing and encoding in real time from a TV card.
  • ''ffserver'' is an HTTP (RTSP is being developed) multimedia streaming server for live broadcasts. Time shifting of live broadcast is also supported.
  • ''ffplay'' is a simple media player based on SDL and on the FFmpeg libraries.
  • libavcodec is a library containing all the FFmpeg audio/video encoders and decoders. Most codecs were developped from scratch to ensure best performances and high code reusability.
  • libavformat is a library containing parsers and generators for all common audio/video formats.

If you are interested in building your own stand-alone software based on the FFmpeg libavcodec, libavformat, and libavutil libraries, you may wish to read Martin Böhme's original and updated “Using libavformat and libavcodec” and also Silviu Minut's code. You may also be interested in FFmpeg's Video Hook functionality.

Building FFmpeg on Blackfin

The ffmpeg community has direct support for the ffmpeg optimized blackfin codecs since April 1, 2007. The optimized ffmpeg has been integrated into uclinux-dist/lib/ffmpeg. The ffmpeg currently requires that FDPIC executable format be used, so in “make menuconfig” set the following flags

  Customise Vendor/User Settings
    Blackfin Build Options
      Binary Format
        (X) FDPIC
    Blackfin app programs
      [*] FFmpeg and FFserver

Depending on your build history, this may require you to do “make clean” prior to “make” simply because the different file formats can't be mixed within a given linkage. Also, depending on what your L1 space is currently allocated to, you may build ffmpeg then get a runtime error that indicates that there is no L1 space available. The quickest way (also using 'make menuconfig') to resolve this is to pull the ethernet packet space out of L1 as follows…

  Customise Kernel Settings
    Device Drivers
      Network Device Support
        Ethernet (10 or 100Mbit)
         [ ] Use L1 memory for rx/tx packets

Note also that some of the examples below use a live video feed from a camera. This requires that the video4linux driver be enabled. Refer to v4l_blackfin_camera v4l_blackfin_camera page for more information on how to configure that into the build.

Running FFmpeg on Blackfin

To test the ffmpeg binary you can use make image or NFS-mount a directory including the executable. This mount approach might also prove useful if you want to test ffmpeg to transcode a movie from one format to another. For example to transcode a MPEG movie you could issue the command on the Blackfin:

ffmpeg -i input_movie.mpg \
       -r [FRAMERATE, e.g. "25"] \
       -b [BITRATE e.g., "256"] \
       -s [FRAME_SIZE, e.g., "352x288", "640x480" or "cif", "vga"] \
       output_movie.mp4

Some interesting things to note here input files are sometimes better stored on a web server, the way ffmpeg interprets files is via a url so file names can be valid rtsp or http requests.

ffmpeg -i rtsp://yoda/sman.mp4 -s 176x144 -b 50k /var/a.mp4

Simple WEB CAM

To setup a streaming video web camera to a specific IP address, you can use ffmpeg directly to perform this operation. You don't need to setup any additional software on the Blackfin Side. You can use broadcast addresses as well as direct udp IP connections. MPEG-TS is a very robust av format which is used in many commerical applications including digital video broadcast systems.

  • FARSIDE is BF537
  • NEARSIDE is your host system with ip 10.64.91.26
ffmpeg -f video4linux2 -s 320x240 -i /dev/video0 -f mpegts udp:10.64.91.26:1234
ffmpeg -f video4linux2 -s cif -r 15 -i /dev/video0 -v 0 -vcodec mpeg4 -f mpegts udp:10.64.91.26:1234

This is very robust you can start vlc once udp:@10.64.91.26:1234 and run the far side blackfin side many times. Because the PAT's, and PMT's re occur in the bitstream if interruption of service occurs, the far side will resyncronize automatically.

Using ffmpeg for video capture

  • Enable ST VS6524 camera sensor. please refer to st_vs6524
  • Using ffmpeg to capture raw video frames: size: 352×288, frame rate: 20 fps, output pixel format: yuv420p.
    root:~> ffmpeg -r 20 -f video4linux2 -s 352x288 -i /dev/video0 -an -f rawvideo -pix_fmt yuv420p /var/test.yuv
    FFmpeg version SVN-r5414, Copyright (c) 2000-2007 Fabrice Bellard, et al.
      configuration: --prefix=/usr --source-path=/home/adam/workspace/kernel/svn_head/uclinux-dist/lib/ffmpeg/build/../ffmpeg-svn-8891 --target-os=Linux --cpu=bfin --arch=bfin --extra-ldflags= -mfdpic   --enable-static --enable-shared --enable-gpl --enable-pp --disable-strip --disable-v4l2 --cross-prefix=bfin-linux-uclibc-
      libavutil version: 49.4.0
      libavcodec version: 51.40.4
      libavformat version: 51.12.1
      built on Jun 21 2007 14:34:04, gcc: 4.1.2 (ADI svn)
    Input #0, video4linux, from '/dev/video0':
      Duration: N/A, bitrate: 32440 kb/s
      Stream #0.0: Video: rawvideo, yuyv422, 352x288, 32440 kb/s, 20.00 fps(r)
    File '/var/test.yuv' already exists. Overwrite ? [y/N] y
    Output #0, rawvideo, to '/var/test.yuv':
      Stream #0.0: Video: rawvideo, yuv420p, 352x288, q=2-31, 200 kb/s, 20.00 fps(c)
    Stream mapping:
      Stream #0.0 -> #0.0
    Press [q] to stop encoding
    frame=   46 fps= 21 q=0.0 Lsize=    6831kB time=2.3 bitrate=24330.2kbits/s
    video:6831kB audio:0kB global headers:0kB muxing overhead 0.000000%
  • Capture video from camera and encoding to an avi file:
    root:~> ffmpeg -r 20 -f video4linux2 -s 352x288 -i /dev/video0 -an -f avi /var/test.avi
    FFmpeg version SVN-r5414, Copyright (c) 2000-2007 Fabrice Bellard, et al.
      configuration: --prefix=/usr --source-path=/home/adam/workspace/kernel/svn_head/uclinux-dist/lib/ffmpeg/build/../ffmpeg-svn-8891 --target-os=Linux --cpu=bfin --arch=bfin --extra-ldflags= -mfdpic   --enable-static --enable-shared --enable-gpl --enable-pp --disable-strip --disable-v4l2 --cross-prefix=bfin-linux-uclibc-
      libavutil version: 49.4.0
      libavcodec version: 51.40.4
      libavformat version: 51.12.1
      built on Jun 21 2007 14:34:04, gcc: 4.1.2 (ADI svn)
    Input #0, video4linux, from '/dev/video0':
      Duration: N/A, bitrate: 32440 kb/s
      Stream #0.0: Video: rawvideo, yuyv422, 352x288, 32440 kb/s, 20.00 fps(r)
    File '/var/test.avi' already exists. Overwrite ? [y/N] y
    Output #0, avi, to '/var/test.avi':
      Stream #0.0: Video: mpeg4, yuv420p, 352x288, q=2-31, 200 kb/s, 20.00 fps(c)
    Stream mapping:
      Stream #0.0 -> #0.0
    Press [q] to stop encoding
    frame=   56 fps= 20 q=3.4 Lsize=     126kB time=2.8 bitrate= 369.3kbits/s
    video:119kB audio:0kB global headers:0kB muxing overhead 5.769797% 
  • Using “vstats” option for enconding log:
    root:~> ffmpeg -r 20 -f video4linux2 -s 352x288 -i /dev/video0 -vstats -f avi /var/test.avi
    root:~> cat vstats_030622.log
    frame=     1 q= 3.2 f_size=   6694 s_size=        7kB time= 0.050 br=  1071.0kbits/s avg_br=  1071.0kbits/s type= I
    frame=     2 q= 2.0 f_size=   6641 s_size=       13kB time= 0.100 br=  1062.6kbits/s avg_br=  1066.8kbits/s type= P
    frame=     3 q= 2.0 f_size=   6216 s_size=       19kB time= 0.150 br=   994.6kbits/s avg_br=  1042.7kbits/s type= P
    frame=     4 q= 2.0 f_size=   5959 s_size=       25kB time= 0.200 br=   953.4kbits/s avg_br=  1020.4kbits/s type= P
    frame=     5 q= 2.0 f_size=   6007 s_size=       31kB time= 0.250 br=   961.1kbits/s avg_br=  1008.5kbits/s type= P
    frame=     6 q= 2.6 f_size=   2688 s_size=       33kB time= 0.300 br=   430.1kbits/s avg_br=   912.1kbits/s type= P
    frame=     7 q= 3.2 f_size=   2531 s_size=       36kB time= 0.350 br=   405.0kbits/s avg_br=   839.7kbits/s type= P
    frame=     8 q= 3.7 f_size=   1453 s_size=       37kB time= 0.400 br=   232.5kbits/s avg_br=   763.8kbits/s type= P
    frame=     9 q= 3.8 f_size=   1256 s_size=       39kB time= 0.450 br=   201.0kbits/s avg_br=   701.2kbits/s type= P
    frame=    10 q= 3.7 f_size=   1187 s_size=       40kB time= 0.500 br=   189.9kbits/s avg_br=   650.1kbits/s type= P
    frame=    11 q= 3.6 f_size=   1247 s_size=       41kB time= 0.550 br=   199.5kbits/s avg_br=   609.1kbits/s type= P
    frame=    12 q= 3.5 f_size=   1146 s_size=       42kB time= 0.600 br=   183.4kbits/s avg_br=   573.7kbits/s type= P
    frame=    13 q= 2.7 f_size=   6744 s_size=       49kB time= 0.650 br=  1079.0kbits/s avg_br=   612.5kbits/s type= I
    frame=    14 q= 3.4 f_size=   2888 s_size=       51kB time= 0.700 br=   462.1kbits/s avg_br=   601.8kbits/s type= P
    --snip-- 
  • Using ”-v <n>” option see whether frames are duplicated or dropped:
    root:~> ffmpeg -r 30 -f video4linux2 -s 352x288 -i /dev/video0 -v 3 -f avi /var/test.avi
    FFmpeg version SVN-r5414, Copyright (c) 2000-2007 Fabrice Bellard, et al.
      configuration: --prefix=/usr --source-path=/home/adam/workspace/kernel/svn_head/uclinux-dist/lib/ffmpeg/build/../ffmpeg-svn-8891 --target-os=Linux --cpu=bfin --arch=bfin --extra-ldflags= -mfdpic   --enable-static --enable-shared --enable-gpl --enable-pp --disable-strip --disable-v4l2 --cross-prefix=bfin-linux-uclibc-
      libavutil version: 49.4.0
      libavcodec version: 51.40.4
      libavformat version: 51.12.1
      built on Jun 21 2007 14:34:04, gcc: 4.1.2 (ADI svn)
    Input #0, video4linux, from '/dev/video0':
      Duration: N/A, bitrate: 48660 kb/s
      Stream #0.0: Video: rawvideo, yuyv422, 352x288, 48660 kb/s, 30.00 fps(r)
    File '/var/test.avi' already exists. Overwrite ? [y/N] y
    Output #0, avi, to '/var/test.avi':
      Stream #0.0, 1/90000: Video: mpeg4, yuv420p, 352x288, 1/30, q=2-31, 200 kb/s, 30.00 fps(c)
    Stream mapping:
      Stream #0.0 -> #0.0
    Press [q] to stop encoding
    timestamp discontinuity 11317161983, new offset= -11317161983
    *** 1 dup!
    *** 2 dup!
    *** 3 dup!
    *** 3 dup!2 fps=  0 q=2.3 size=      41kB time=0.4 bitrate= 837.4kbits/s dup=9 drop=0
    *** 3 dup!
    *** 4 dup!
    *** 5 dup!
    *** 6 dup!1 fps= 25 q=2.8 size=      82kB time=1.0 bitrate= 647.1kbits/s dup=28 drop=0
    frame=   38 fps= 25 q=2.2 Lsize=     100kB time=1.3 bitrate= 649.9kbits/s dup=35 drop=0
    video:36kB audio:0kB global headers:0kB muxing overhead 178.194107%

Using ffmpeg for audio/video capture

To enable audio output

ffmpeg -s 160x120 -f video4linux2 -i /dev/video0 \
       -ar 48000 -f audio_device -i /dev/audio \
       -f mpegts -vcodec mpeg4 udp:yoda:1234

Currently on my 500Mhz board this consumes 36.5% of the processor mpeg4 simple profile. The single channel mono audio encode with the lavc mpeg2 audio encoder is currently about 7% of the processor load. I think its about a 1s delay when streaming to vlc. -- notes Sept 5, 2007.

BTW you can stream just audio with

ffmpeg -ar 48000 -f audio_device -i /dev/audio -f mpegts udp:yoda:1234

Performance as April 1st.

Performance analysis notes test are performed with nfs mounted disk all executed with the xx script. xx is defined as a very simple script that executes on the target machine. Another small note these tests were all performed with over a wifi link in my old house with a pretty poor link. This is old data see multimedia_optimizations wiki for upto date video performance information.

xx:
#!/bin/sh
rsh -l root mad "(cd /u/b; ./$*)"

The code is a fresh svn co from mplayer.org with the patch made for llrint.

Now I'm going to show the performance of the following:

1. video decode D1 2. video encode CIF 3. video decode CIF 4. video transcode CIF→CIF 5. DVD vob decode

Although its not perfect this is on a 500 Mhz machine running full linux. MPEG4 Simple Profile Decode of Standard Definition is currently 19 fps.

yoda:~/b mmh$ xx ./ffmpeg -i http://yoda/vid/shire.mp4 -f rawvideo /dev/null
FFmpeg version SVN-r8596, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --arch=bfin --cpu=bfin --cross-compile --cross-prefix=bfin-linux-uclibc- --target-os=linux --cc=gcc --enable-static --enable-gpl --enable-gpl --enable-pp --enable-swscaler --enable-vhook --disable-strip --disable-v4l2
  libavutil version: 49.4.0
  libavcodec version: 51.40.2
  libavformat version: 51.11.0
  built on Apr  1 2007 20:35:20, gcc: 4.1.2 (ADI svn)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'http://yoda/vid/shire.mp4':
  Duration: 00:00:38.2, start: 0.000000, bitrate: 347 kb/s
  Stream #0.0(und): Video: mpeg4, yuv420p, 720x480, 25.00 fps(r)
File '/dev/null' already exists. Overwrite ? [y/N] y
Output #0, rawvideo, to '/dev/null':
  Stream #0.0: Video: rawvideo, yuv420p, 720x480, q=2-31, 200 kb/s, 25.00 fps(c)
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  146 fps= 19 q=0.0 Lsize=   73913kB time=5.8 bitrate=103680.0kbits/s     
video:73913kB audio:0kB global headers:0kB muxing overhead 0.000000% 

MPEG4 decode of CIF at 46fps.

yoda:~/b mmh$ xx ./ffmpeg -i ../vid/cara.mp4 -f rawvideo /dev/null
FFmpeg version SVN-r8596, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --arch=bfin --cpu=bfin --cross-compile --cross-prefix=bfin-linux-uclibc- --target-os=linux --cc=gcc --enable-static --enable-gpl --enable-gpl --enable-pp --enable-swscaler --enable-vhook --disable-strip --disable-v4l2
  libavutil version: 49.4.0
  libavcodec version: 51.40.2
  libavformat version: 51.11.0
  built on Apr  1 2007 20:35:20, gcc: 4.1.2 (ADI svn)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '../vid/cara.mp4':
  Duration: 00:00:12.0, start: 0.000000, bitrate: 450 kb/s
  Stream #0.0(und): Video: mpeg4, yuv420p, 352x288, 25.00 fps(r)
File '/dev/null' already exists. Overwrite ? [y/N] y
Output #0, rawvideo, to '/dev/null':
  Stream #0.0: Video: rawvideo, yuv420p, 352x288, q=2-31, 200 kb/s, 25.00 fps(c)
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  300 fps= 46 q=0.0 Lsize=   44550kB time=12.0 bitrate=30412.8kbits/s    
video:44550kB audio:0kB global headers:0kB muxing overhead 0.000000%

MPEG4 Transcoding CIF at 19fps, its all about bandwidth here we are using http to serve the video instead of nfs.

yoda:~/b mmh$ xx ./ffmpeg -i http://yoda/vid/cara.mp4 c.mp4
FFmpeg version SVN-r8596, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --arch=bfin --cpu=bfin --cross-compile --cross-prefix=bfin-linux-uclibc- --target-os=linux --cc=gcc --enable-static --enable-gpl --enable-gpl --enable-pp --enable-swscaler --enable-vhook --disable-strip --disable-v4l2
  libavutil version: 49.4.0
  libavcodec version: 51.40.2
  libavformat version: 51.11.0
  built on Apr  1 2007 20:35:20, gcc: 4.1.2 (ADI svn)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'http://yoda/vid/cara.mp4':
  Duration: 00:00:12.0, start: 0.000000, bitrate: 450 kb/s
  Stream #0.0(und): Video: mpeg4, yuv420p, 352x288, 25.00 fps(r)
File 'c.mp4' already exists. Overwrite ? [y/N] y
Output #0, mp4, to 'c.mp4':
  Stream #0.0: Video: mpeg4, yuv420p, 352x288, q=2-31, 200 kb/s, 25.00 fps(c)
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  300 fps= 19 q=31.0 Lsize=     635kB time=12.0 bitrate= 433.5kbits/s    
video:632kB audio:0kB global headers:0kB muxing overhead 0.488395%

CIF video encode currently 23 fps @ 500 MHZ, note all local traffic.

yoda:~/b mmh$ xx ./ffmpeg -s 352x288 -f rawvideo -i /var/carosel352x288.eyuv c.mp4     
FFmpeg version SVN-r8596, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --arch=bfin --cpu=bfin --cross-compile --cross-prefix=bfin-linux-uclibc- --target-os=linux --cc=gcc --enable-static --enable-gpl --enable-gpl --enable-pp --enable-swscaler --enable-vhook --disable-strip --disable-v4l2
  libavutil version: 49.4.0
  libavcodec version: 51.40.2
  libavformat version: 51.11.0
  built on Apr  1 2007 20:35:20, gcc: 4.1.2 (ADI svn)
Input #0, rawvideo, from '/var/carosel352x288.eyuv':
  Duration: N/A, bitrate: N/A
  Stream #0.0: Video: rawvideo, yuv420p, 352x288, 25.00 fps(r)
File 'c.mp4' already exists. Overwrite ? [y/N] y
Output #0, mp4, to 'c.mp4':
  Stream #0.0: Video: mpeg4, yuv420p, 352x288, q=2-31, 200 kb/s, 25.00 fps(c)
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  153 fps= 23 q=31.0 Lsize=     405kB time=6.1 bitrate= 541.6kbits/s    
video:403kB audio:0kB global headers:0kB muxing overhead 0.469420%

DVD decode 12 fps, this particular vob is 10MB/s.

yoda:~/b mmh$ xx ./ffmpeg -i http://yoda/vid/mi3c0.vob -f rawvideo /dev/null
FFmpeg version SVN-r8596, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --arch=bfin --cpu=bfin --cross-compile --cross-prefix=bfin-linux-uclibc- --target-os=linux --cc=gcc --enable-static --enable-gpl --enable-gpl --enable-pp --enable-swscaler --enable-vhook --disable-strip --disable-v4l2
  libavutil version: 49.4.0
  libavcodec version: 51.40.2
  libavformat version: 51.11.0
  built on Apr  1 2007 20:35:20, gcc: 4.1.2 (ADI svn)
Input #0, mpeg, from 'http://yoda/vid/mi3c0.vob':
  Duration: 00:00:00.3, start: 0.191167, bitrate: 2147483 kb/s
  Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x480, 9800 kb/s, 29.97 fps(r)
  Stream #0.1[0x80]: Audio: 0x0000, 48000 Hz, stereo, 192 kb/s
  Stream #0.2[0x20]: Subtitle: dvdsub
  Stream #0.3[0x21]: Subtitle: dvdsub
File '/dev/null' already exists. Overwrite ? [y/N] y
Output #0, rawvideo, to '/dev/null':
  Stream #0.0: Video: rawvideo, yuv420p, 720x480, q=2-31, 200 kb/s, 29.97 fps(c)
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  345 fps= 12 q=0.0 Lsize=  174656kB time=11.5 bitrate=124291.7kbits/s