A new screenshots comparator and loads of new x264 x265 vp8 vp9 tests

hobbit_1-756kbps--x264_x265_vp8-ssim_analysisI bought the Blu-ray of “The Hobbit: An Unexpected Journey” (which is considered to be a reference for video quality) and I decided to do more codec tests. If you want to know something about my methodology please read my previous article. I ripped a scene from the Blu-ray and I cropped it to 1920×800, then I compressed it using x264 LOSSLESS compression. Except for the crop it is bit identical with the Blu-ray.

I will not go in the details because I already explained everything, also I coded a new screenshot comparator which eases the job: use it to find the details about sources, codecs, etc.. like the exact command line with the parameters I used. It also allows you to download the sample videos. It took quite a while to code it, it uses XML to store data and it quickly grown to ~200 lines of code so please let me known if something’s wrong. I know, the graphic sucks but I hate web programming :)

Just some considerations… First of all: where is VP9? There isn’t. Why? Because after a whole week it still didn’t finish encoding. Seriously, using a wooden abacus would be faster! If someone wants to waste his cpu cycles to encode it, please send me the resulting files. You can download the source video using the screenshot comparator and encode it with this command:

ffmpeg -v 0 -i hobbit_lossless.mkv -threads 4 -vsync 0 -an -pix_fmt yuv420p -f yuv4mpegpipe - | ./vpxenc - -o hobbit.vp9 --codec=vp9 --cpu-used=0 --best --target-bitrate=756 -p 2 --pass=1 --fpf=vp9.stat; ffmpeg -v 0 -i hobbit_lossless.mkv -threads 4 -vsync 0 -an -pix_fmt yuv420p -f yuv4mpegpipe - | ./vpxenc - -o hobbit.vp9 --codec=vp9 --cpu-used=0 --best --target-bitrate=756 -p 2 --pass=2 --fpf=vp9.stat

Also, someone ranted because of lack of PSNR/SSIM graphs. Who needs SSIM when you can use your eyes? Anyway, here is the shiny graph and here is how I created it:

./qpsnr --ignore-fps -a avg_ssim -o blocksize=16:fpa=24 -r hobbit_lossless.mkv hobbit_crf28_756kbps.h265 hobbit_756kbps.h264 hobbit_756kbps.vp8 > ssim-fpa24.csv

Then you can use Libreoffice Calc to make the graph.

Some notes about the graph: yeah this is a completely different scenario from the previous test and x265 wins hand down. But beware, this is just with a VERY low bitrate. I encoded the source with x265 using CRF 28 which resulted in a 756 kbps bitrate, then I two pass encoded using x264 and vp8 with a 756 kbps target bitrate: this is the resulting SSIM graph. If you raise the bitrate there are no such big advantages using x265 over x264 which is a pity. Link. Source vs x265.

I did another encoding starting with x265 CRF 23 which resulted in a 1733 kbps bitrate. Link. Source vs x265.

Finally I did two more tests with x265 16 bit variables CRF 28 and 23: 943 kbps and 1186 kbps. I used x264 hi10p to match the file size. I just discovered “16bpp” doesn’t mean high bit depth in x265, anyway it’s bugged and shouldn’t be used right now.

Here is the SCREENSHOT COMPARATOR. Press F11 to switch your browser to the full screen mode.

15 comments to A new screenshots comparator and loads of new x264 x265 vp8 vp9 tests

  • Smarter

    Don’t use –best with vp9, use –good instead. –best is _extremely_ slow and not recommended because it won’t help very much.

  • darkbasic

    Since someone asked: yes, I will release the screenshots comparator sources but not right now. It’s a quick and dirty hack, it really needs some cleanings and some parts should be rewritten from scratch with a completely different implementation. In the menatime if someone really wants the code ask me and I will send you an e-mail, but please do not post that shit until it’s ready :)

    P.S.
    I hate dialog boxes too, I will remove them don’t worry ;)

  • Oche

    Great work here man!

    I think I saw yout ‘wooden abacus’ comment on the VP9 devel list as well. I’m guessing it was you on there too. Cracked me up. I have a quick question, how were you able to get qpsnr to work with h265 and vp9. I clone the git repository at https://github.com/hsanson/qpsnr.git. Is this the same one you used? Can you point me in the direction of how to ge it to work. Thanks

  • Oche

    Thanks, still can’t get it to work, My guess is the versions of libavcodec, libavutil etc. that come with Ubuntu 13.10 are old and can’t handle hevc and vp9 files.

  • Oche

    I have a recent development versiion of ffmpeg which I had to build on Ubuntu. I’ve tried manually editing qpsnr’s makefile to use FFMPEG’s libraries instead but i get strange errors about linking that I dont understand as my knowlegde of C and C++ is limited at best.

    Please can you tell me what Linux distro and version you use.

  • Oche

    Thanks, finally got it to work. I kept getting linker errors so I had to rebuild ffmpeg with -enable-shared and –enable-static configure flags. Then I put an -L/path/to/ffmpeg/libs in the Makefile for QPSNR and that got it to work.

    Thanks a lot again.

  • Kitcat490

    So, I got bored and had some spare cpu cycles, so here is the encoded vp9 file and log :D https://dl.dropboxusercontent.com/u/5180892/hobbit.tar.xz

    Encode time was about 20 hours on my cpu using vpxenc 1.3.0 and ffmpeg 2.2.2.

    I should mention that I have had success encoding full length movies with vp9 using somewhat lower settings while still maintaining pretty nice quality in a pretty reasonable amount of time. If you would like an example you can just email me at kitcat490@gmail.com (don’t want to post publically for obvious reasons).

  • Jackal

    This updated comparison is still very inaccurate for both x265 and vp9, you are still comparing different rate factor methods instead of creating an encoding with each encoder at the same bitrate. This is where you will actually notice a difference with x265 or vp9. Using crf is quick and dirty, it will give you a video of the quality you selected but the file size and bitrate aren’t predictable.

    For example you should use something like the following for x265 and similar settings for x264 and vp9 as well.

    ffmpeg … | x265 –y4m –fps=23976/1000 –ctu=64 –cutree –bitrate=xxx –vbv-bufsize=31250

    and you can add more fine tuning from there.

    Also if you use crf, a crf of 20 will give you similar subjective quality to the source, although not perfect, so at 28 you lose a lot of quality.

  • Jackal

    x265 has for a very long time (pre 0.6 at least), if not always has supported single pass abr with variable bitrate options. Although currently support for 2 pass abr has not been implemented. If you read the documentation it is all there, and I wouldn’t trust the credibility of half the people that frequent that irc channel despite some of them being developers, or at least claim to be.

    When I initially started these encodings I didn’t realize I hadn’t reset the bitrate on my encoding cluster to a higher value, but this also makes for a good example of x265’s ability to produce watchable video at low bitrates.

    From this comparison you can see that encoding with identical bitrates in both x264 and x265 yields similar file sizes (not enough deviation to matter). However it becomes apparent at such a low bitrate that x265 is able to maintain perceived quality, whereas the x264 encoding suffers from blocking artifacts to the point it isn’t even watchable at times.

    I was planning to include a vp9 encoding, but it will not be complete for about a week at the settings I am using.

    Constants:
    Bit Depth: 8
    FPS: 24000/1001
    Average Bitrate: 881 kbps (vbr options added as well)
    CPU Threads: 1
    Motion Estimation: Uneven Multi Hexigonal
    Motion Estimation Range: 24
    Variable Bitrate Buffer Size: 31250
    Variable Bitrate Max Rate: 31250
    B-frames: 3
    B-frame Bias: 0

    Versions Used:
    x264-0.142.2431-ac76440
    x265-1.1+192-fe370292c232ec9a

    Included is the source video and the subsequent encodings there of (x264-dbr is double the bitrate at 1762 kbps as a comparison for quality to show that x265 does achieve twice the perceived quality at the same bitrate, even though it isn’t preserving anymore detail). I chose this clip because it has a few things most encoders struggle with, retaining quality and definition in dark scenes as well as preserving the detail of facial features and this clip does well to exaggerate both. I will have to look for a clip that shows screen tearing to show the difference in motion estimation of the encoders (something with a lot of camera panning is where it would show the most)

    https://mega.co.nz/#!YM1GBDjI!5SKnXjgIt-gUgoIs59LThXdQUPDj7HeZs5pmHkqntVw (~902mb)

    SSIM Comparison:
    They are very close which makes sense because x264 preserves quality uniformly across the frame, whereas x265 preserves only portions of the frame with higher detail. For the lack of a better description, it follows along the lines of the Law of Conservation of Matter. You aren’t changing the bitrate, so therefore the filesize is constant and you have the same amount of bits in which store data; as well you aren’t gaining any compression from x264 -> x265, so therefore it is just rearranging bits to better preserve portions of the frame with more detail through the use of “Coding Units”. (look at the background vs. foreground of the screenshots)

    http://i.imgur.com/DjOE58Z.png (changed fpa to 96)

  • Jackal

    That statement is incorrect, single pass vs. two pass will not yield any difference in file size (with both you are still using a fixed bitrate), two pass merely allows for better optimizations for quality. Single pass x265 is already great, there obviously will be an improvement when two pass is complete though.

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>