Monday, November 21, 2011

ffmpeg - x264 - video libs -avconv

Links: libav - avconv, lavf   forum post identifying lavf circular problem   ffmpegsource   ffmpeg usage
Edit: Libav is the starting point, inside which lavf is the key. It's a catch-22 to install this as you will learn when installing x264, or can read many places, eg here. It's apparently available only inside libavcodec, not just x264. Ffmpeg is being deprecated by avconv. Once you've located an x264 with lavf, the order that works is 1) x264, 2)xvidcore 3) ffmpeg, and 4) finally recompile and install x264 again. "FFmepgsource" supposedly includes lavf, but appears to be windows-only code.
If you're going to eventually install avidemux, suggest doing so prior to any of the above. Avidemux (without warning) overwrites one's ffmpeg install in such a way that gives MUCH less ffmpeg functionality afterwards (as seen in "ffmpeg -formats". For commiseration about ffmpeg dependencies, see my post earlier today venting indignant rage. But in this post, let's make it all work. I did a relatively extensive post a year ago updating ffmpeg, but at that time, I didn't realize how much I had lucked-out by having a current version of x264 in place. This time, I found out the hard way, as I noted earlier today, about a circular lavf dependency.

x264 install

$ ./configure
Platform: X86
System: LINUX
cli: yes
libx264: internal
shared: yes
static: no
asm: yes
interlaced: yes
avs: no
lavf: no
ffms: no
gpac: no
gpl: yes
thread: posix
filters: crop select_every
debug: no
gprof: no
strip: no
PIC: no
visualize: yes
bit depth: 8

The unfortunate process due to SHIATTY DESIGN is going to be 1) x264 without lavf, 2 ffmpeg , 3) uninstall x264, 4) recompile and reinstall x264 with lavf-enabled. NOTE: also updated to the latest libtheora (v.1.1.1) and libmp3lame (v.3.99.2). Lame went in standard:
$ ./configure --prefix=/usr --enable-nasm
$ make
# rm /usr/lib/libmp3*
# make install
Now on to the big jobs.

1 - x264 (v.20111120 build)

$ ./configure --prefix=/usr --enable-lavf --enable-visualize --enable-shared
Platform: X86
System: LINUX
cli: yes
libx264: internal
shared: yes
static: no
asm: yes
interlaced: yes
avs: no
lavf: no
ffms: no
gpac: no
gpl: yes
thread: posix
filters: resize crop select_every
debug: no
gprof: no
strip: no
PIC: no
visualize: yes
bit depth: 8

2 - ffmpeg (v. 0.8.10)

$ ./configure --prefix=/usr --enable-libx264 --enable-shared --disable-static --enable-pthreads --enable-x11grab --enable-swscale --enable-libfaac --enable-libmp3lame --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libxvid --enable-gpl --enable-postproc --disable-ssse3 --enable-yasm --enable-nonfree --arch=i686 --cpu=i686
make, #make install. This was without incident, though many warnings during make.

3 - x264 uninstall and recompile (fail)

$ ./configure --prefix=/usr --enable-lavf --enable-visualize --enable-shared
Platform: X86
System: LINUX
cli: yes
libx264: internal
shared: yes
static: no
asm: yes
interlaced: yes
avs: no
lavf: no
ffms: no
gpac: no
gpl: yes
thread: posix
filters: resize crop select_every
debug: no
gprof: no
strip: no
PIC: no
visualize: yes
bit depth: 8
So nothing has changed yet. Let's try ffmpegsource

4 - ffmpegsource (v.2.16)

As if things are not unhelpful enough in ffmpeg installation, these dumbasses who created ffmpesource decided to package their files in 7z compression. So, instead of using some universally installed compression like .tgz or bz2, they know you will have to install p7zip just to get their files.

5 - p7zip

But wait, there's a problem, what a surprise. P7zip, just like ffmpegsource, also has a non-standard installation process and it doesn't work. It's one of those installations where they have premade makefiles and you copy over "makefile" and then use "make". The problem with their masterplan is, following their (shiatty) directions to the letter, it results in nothing but
make: *** No rule to make target `all2_test'. Stop.
The entire process appears headed toward 40 hours simply for the portion of the work updating. I looked at the code, and it's not configured properly. Accordingly, I just ran
$ make -f makefile.oldmake all2
This is just a generic bullshiat build, but at least the code in this makefile is more than just a set of processor optimizations without any instructions.

6 - back to ffmpegsource (v.2.16)

Three hours later, now that it's extracted, I see that this source is nothing but Windows crap; a .dll and an .exe. It's advertised as "cross platform", but that's something I have yet to see. So I was finally able to unpack FFmpegSource 2.16. This began to compile, but then threw errors during make:
src/core/lavfaudio.cpp: In constructor 'FFLAVFAudio::FFLAVFAudio(const char*, int, FFMS_Index&, int)':
src/core/lavfaudio.cpp:40: error: 'avcodec_open2' was not declared in this scope
So it could not successfully build. This was likely a problem in the FFLAVFAudio definition inside header src/core/audiosource.h, since when I commented out the lines in lavfaudio.cpp, the same fail moved to lavfindexer.cpp, which included the same header. Indeed, grepping, we can see that the variable avcodec_open2 is nowhere used, let alone defined, in audiosource.h. So appears we're out of luck again. I'm no C programmer and the Linux version of ffmpegsource isn't even maintained anymore. After all of this, I'm really backed into a situation where my x264 is just not going to be compiled with lavf support. Very very very annoying.

7 - x264 w/out lavf

In

8 - tests

No xvid encoding. I have libxvidcore installed, but it appears a bit outdated. Will attempt to update libxvid. I was sure I had it in my configure line, but it doesn't appear in ffmpeg -options either. Encodes to mp4. The latest download is xvidcore 1.3.2. Weird that it didn't pick it up in the ffmpeg ./configure. Conundrum and annoying, and I bet it comes back to the lavf issue again. Just have a feeling that's pivotal. For now though, will update xvidcore.

9 - xvidcore 1.3.2

The source was difficult to locate on the Xvid.org site as the "downloads" tab only led to descriptions. Sourceforge had no files. Eventually I was able to Google into the latest 1.3.2 source. When unpacked, it has a lot of Windows garbage in there, so had to drill down into "build, generic", where I found normal configure options. Did a standard...
$ configure --prefix=/usr
$ make
# make install
...and it went in without incident. Now back to another attempt at ffmpeg with xvid in there.

library note

During ./configure, I often run across libraries not being found. It's good to make sure they're all in $LD_LIBRARY_PATH which you can just check with an "echo", like any other variable. Sometimes, I do this to be sure, if it looks funky
export LD_LIBRARY_PATH=/usr/local/lib:/lib:/usr/lib
Also, it's good to make sure any special paths are in /etc/ld.so.conf and then run a quick # ldconfig to update the list of libs in the system.

ffmpeg - x264 (FAIL)

Links: forum discussion of circular dependency for lavf
Full disclosure: This post is not a solution - it's a vent about program dependencies, one of those items where you're forced to waste a DAY or TWO for no good reason other than developers didn't use their heads around installation. Anyone who uses ffmpeg knows it's linked to libx264 like a hand-in-glove. Yesterday, I came across a very unappetizing twist in this relationship. I was deep into shrinking some videos into qvga format for a NAXA device (see previous post) and found that my 1 yr old ffmpeg release was concatenating video and audio streams into an AVI file in ways giving flaky playback. Video(libxvid) and audio(MP3) were stable individually but, once concatenated, could not reliably playback on every device. With MP2 audio, there was no problem, so that it appeared possible the problem was how MP3 streams were being packaged by ffmpeg. To rule this out, I started an update on ffmpeg.

Catch-22

After deleting my old version of ffmpeg, I downloaded ffmpeg version 0.8.6 and found during ./configure that my version of x264 needed updating. I blasted my old x264 and got on that, whereby I learned that I wanted yasm, of course, for an assembled (faster) version of x264. Yasm in, I returned to x264, and found that lavf support was not showing during ./configure. WTF? This is where I learned of a motherfarking CATCH-22 circular dependency that x264 developers had implemented. X264 LAVF support requires a version of ffmpeg already installed. But ffmpeg requires that x264 be installed. Each requires the other for lavf support, so where can you start? This means I could not install x264 or or ffmpeg with lavf support or, in other words, that they would both be USELESS CRIPPLEWARE. WTF?

The only solution possiblity I can see for this GARBAGE is to install x264 without lavf support, subsequeently install ffmpeg without lavf (because it's not in the x264, because it's not in the ffmpeg, because it's not in the x264, because...), then turn around and blow out the x264 and reinstall it with the crippled ffmpeg already in place (and no LAVF support within) and HOPE that the x264 can somehow provide LAVF when it doesn't see any in the ffmpeg. Finally, blow out the ffmpeg again, and reinstall it with x264 support enabled. Altogether, this would be 4 installations, but I don't see any other way to get the LAVF in there. Thanks, guys.

All I wanted was to get some videos down to qvga size, which was already days, now I see I'm going to have to fix the thing that i need to fix the other thing, etc etc etc, in order to, days later, have a working ffmpeg again, in order to get back on the long road to making qvga sized videos. What a bunch of shiat.

Saturday, November 19, 2011

[solved] naxa nmv-155 sized playback (QVGA)


Let's see if we can get anywhere with this inexpensive media player. I find players like these NAXA's in bargain bins for $15, (probably $5 worth of parts in the unit), but the day I wanted to toy with one, I was only able to find one as cheap as $25. Bought it anyway.

In forums about these knock-off players, the two most common problems are USB recognition and media formatting. It's got a 4GB internal, and room for a card as well. But it's picky about file formats... it's cheap. Secondly, I was excited to read in its manual, that the NAXA can display PDF's (Edit: this was misadvertised; it reads only basic text files), in addition to MP3's, movies, and photos. Battery life is reasonable.

unboxing

Unit comes without any cover but fits into a G1 holder (remaining after I had a G1 stolen). A USB -> micro-USB connector cable, a 300mA AC->micro-USB charger, earbuds, stylus (a plastic dental pick also works), Windows disk, and a small manual.

USB detection/connection

Threw out the Windows disk and plugged in the USB, and no auto-recognition took place in any file managers. Nothing in lsusb. Seeing nothing in udevmonitor either, I swapped USB cables with a known good USB cable (for a different device) and suddenly the NAXA appears in the file manager, no problems. It's identified there as "GENERIC USB DISK DEVICE Music Player" (3.5GB). We're already at less than the advertised 4GB. In lsusb, it appears as
10d6:1101 Actions Semiconductor Co., Ltd D-Wave 2GB MP4 Player / AK1025 MP3/MP4 Player
This 2GB reading is probably the most accurate -- I've likely been ripped off: appears I have a Chinese 2GB device hacked to indicate twice its real capacity. Happens. A different system check (below) shows 3.7 GB, but I'd bet against it and with the 2GB reading.
# df
/dev/sdc 3709840 2084 3707756 1% /media/disk
Summary: Ditch the included USB->micro-USB cable and the Windows disk. Storage expectations should be reduced to a little less than 2GB for this (likely) piece of shiat.

video

Let's upload something less than 2GB and see if it plays.
  • MKV - First-up, a 1.4GB sports vid coded in MKV. Result: the file is not even detected in the device file manager. Can't even be seen there.
  • AVI - Second, we'll do a standard AVI, let's go. Here's the encoding and resolution
    Stream #0.0: Video: mpeg4, yuv420p, 608x336
    Stream #0.1: Audio: mp3, 48000 Hz, stereo, s16, 96 kb/s
    This file appeared in the device. I attempted to play it, but the NAXA displayed a "file format error".

QVGA = 320x240

Next step: record a short vid on the device, make sure it plays on the device (it did), and then download to the laptop and determine its settings and codecs.
Duration: 00:00:17.93, start: 0.000000, bitrate: 551 kb/s
Stream #0.0: Video: mpeg4, yuv420p, 320x240
Stream #0.1: Audio: mp3, 16000 Hz, stereo, s16, 64 kb/s
So 320x240 (ie, QVGA) and 551k bitrate, and only 64 k on the sound. This is what the NAXA makes and plays.

Let's approximate that. I'll take a file taxi_s01e01.avi and, not worrying about B frames or whatever, split it into audio and video, and then render it back together, but in QVGA with MP3:
$ ffmpeg -i taxi_s01e01.avi -vn -ar 44100 -ac 2 s1e1.wav
$ ffmpeg -i taxi_s01e01.avi -vcodec libxvid -b:v 500k -s qvga -an s1e1.avi
$ ffmpeg -i s1e1.avi -i s1e1.wav -acodec libmp3lame
-ar 44100 -ab 192k -ac 2 -vol 330 -vcodec copy -b:v 504k qvga_taxi_s01e01.avi
This encoded without errors and played without errors.

the email angle

Most email attachments won't take more than a 20 second video clip without maxing out the 20Mb attachment limit. Try the 320x240 as a solution. It makes files roughly 30 x smaller than normal WXGA resolution. Not perfect, but a workable email attachment in lieu of those crappy MMS crippled vids. Use something like Ocenaudio to clean up your audio and put the video back together and email it.

$ ffmpeg -i s1e1.mp4 -i s1e1.wav -acodec libmp3lame
-ar 44100 -ab 192k -ac 2 -vol 330 -s qvga taxismall.mp4

Sunday, November 13, 2011

pdfedit - pdf's etc (FAIL)

Links: sourceforge - pdfedit   www.boost.org   Xournal

Like most blog posts, this one is born from annoyance. My current rage was with PDF books retreived from Project Gutenberg. Typical PDF book files should be a few hundred K and fast to load. Some are. Some are several MB and also open quickly. But a few are several megabytes and, while loading, push one's CPU to an unhealthy 100% for minutes, instead of for a few seconds. This subset of larger PDF's are of course impossible to open on portable devices. The problem is the Gutenberg volunteers make a normal PDF, but then add a 1200 lines of resolution photo of the book's cover to the first page of the PDF. It takes a lot of CPU and memory for PDF software to simultaneously render a huge photo down to a tray icon, display the huge photo full-screen, and load the first few pages of text. The fix is to edit such PDF's initial page, reducing the first page photo to a typical 75 or 150 lines of resolution photo.

So these PDFs need repairing or else one's CPU will need replacing, but is there a Linux program out there which does this? We can say "yes" definitively if we want to spend hundreds on the Adobe Acrobat solution. And there are well-proven Linux tools like pdftotext that quickly extract all the PDF's text unformatted. But what about a Linux program that just opens the PDF, allows us to edit, and then close the file? Based on this ideal, I decided to give PDFedit a shot.

installation (v.0.4.5)

Comes as a .bz2 because they have decided to pander to the Windows crowd, apparently. The README indicates "Boost" is the dependency. Boost is just a set of C++ libraries, so I ran configure before doing any checks to see if they might already be installed. Nope:
checking for boostlib >= 1.20.0... configure: error: We could not detect the boost libraries (version 1.20 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in . See http://randspringer.de/boost for more documentation.

boost (v.1.47) installation

PDFedit's pretension of requiring Boost is annoying. For example, 1) C++ libraries sufficient for compiling are already on people's systems, we don't need a redundant set, 2) installing them means bloating one's system for no reason and, worst of all, 3) they are on Sourceforge servers so add an hour to the installation timeline. (Edit: indeed, the first download was 30MB and was a set of PDF documents mislabeled as source.) A half hour was already wasted, but it's a dependency, gotta get it in. Let's go directly to the boost site to get the libraries. And...the Boost site bounced me back to Sourceforge for another 53 frigging MB at Sourceforge 60K "speeds". Installing PDFedit is starting to look like a 2 hour operation.

bootstrap

So, opened the PDFedit source. No configure file, no README. Great. Noting there are some bootstrap files however, so we're apparently dealing with frigging bootstrap. Now we have bad choices by both the Boost and the PDFedit developers. Also Boost appears to require Python. So the real dependency tree is apparently: PYTHON-->BOOST-->XPDF-->PDFEDIT
$ ./boostrap.sh
$ ./b2 --prefix=/usr
This doesn't work. I finally located some installation instructions. They're on the Boost website instead of in a simple README in the source. They appear partially inaccurate since they are without root. Let's start over and change it to a way it will work.
$ ./boostrap.sh --prefix=/usr
# ./b2 install

back to pdfedit (hours later)

I've almost forgotten why I needed to install PDFedit in the first place, but here we go. Did a mostly standard configure -prefix=/usr, however the results showed me that no tools or kernel tests would be included. Start over.
$ configure -prefix=/usr --enable-tools --enable-kernel-tests
This went well except that kernel checks couldn't be configured due to some missing package apparently called Cppunit for which it wants version 1.10 or later. Let's see if we can get that in.

Cppunit (v.1.12.1) installation

This was a standard configure -prefix=/usr, make, # make install. No problems.

back to pdfedit

Attempted 3 ways
$ configure -prefix=/usr --enable-tools --enable-pdfedit-core-dev --enable-kernel-tests
$ configure -prefix=/usr --enable-tools --enable-pdfedit-core-dev
$ configure -prefix=/usr
All of these resulted in fatal errors during make
make[2]: *** [cpagecontents.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/home/foo/Download/pdfedit-0.4.5/src/kernel'
make[1]: *** [kernel] Error 2
make[1]: Leaving directory `/home/foo/Download/pdfedit-0.4.5/src'
make: *** [source] Error 2
Apparently the PDFedit source has design and documentation flaws much deeper than one can suss out in the time required for reasonable installation. On the first account, it should run with normal kernel settings. On the second account, they left the little detail of kernel recompiling out of their hard-to-locate documentation, when it should be the first thing noted. Further, the documentation neglects any information regarding which kernel switches would need to be set. So really, users would have to guess among 2,600 kernel options in order to use PDFedit. In short, PDFedit will either work on one's PC or it won't, dealer's choice. Troubleshooting using strace and finding the needle in the haystack of the entire PDFedit source, goes far beyond the investment most people should have to make to simply install a program. I certainly have more appealing things to do with two weeks.

I wasted half a day on the shiatty PDFedit product and was unable to install it or edit my PDF's. In the end, I ran pdftotext on the particular PDF I wanted to fix. I'll format that basic text file with LaTeX as I read it, and then recompile when finished -- the resultant PDF will be easily read on a portable device. This is extra work I'll first have to do with a desktop, so I guess I'll read the book at home.