Monday, November 27, 2023

xdg mime, usb mounts

If we have to jettison our physical CD's and DVD's in the name of space, we unfortunately must back them up first. At that point, we lose the convenience of...

  1. easy playback (pull the CD/DVD off the shelf, put it in a player, and press 'play')
  2. in lieu of pressing 'play' in a player, how do we play an entire set of MP3's from a CD with a single click?
  3. global selection is lost (how do we easily observe our entire collection of CD's/DVD's, as we used to on on a shelf?)
  4. the portability of a bookshelf CD player is gone, and we now require a device with an interface to select and play the music

solution

Turns-out that 1, 2, and 4 are related questions. We can create an M3U for each CD (not a trivial task), then create an HTML page with hyperlinks to the M3U's. So when we click the HTML link from our browser, the M3U is opened by the default application (eg. VLC) which plays the CD's MP3's in the order they used to be on the CD.

This fundamentally solves problems 1 and 2. And since HTML pages open on nearly any device with a web browser, we have a good start on solving problem 4.

To solve problem 3, perhaps we can also eventually add thumbnails -- a thumbnail for each CD -- to our HTML page, and then embed an M3U link into the thumbnail: see a thumbnail for a CD, click the thumbnail. Since we can place infinite thumbnails on an HTML page, we can likely see our entire collection on a single webpage. At that point, we'd only need to consider what device and speakers to connect, the hardware.

This is a fairly simple schema, and attainable, but it's a significant investment of work: we must create an intuitive HTML page, and multiple M3U's. The CD's song order and file locations cannot be determined by the application (eg, VLC) without an M3U, so an individual M3U must be created for each CD, and for any mixes.

nested additional problem

We want to open our HTML file in a browser to click on a link to the CD's M3U. However links to M3U's have no default application and thus do not natively work when clicked in browsers. So now our job is two-fold.

  • We must create functional M3U files
  • We must configure our browser or OS to make hyperlinks to M3U's click-to-play. That is, we must associate an application with the HTML link. The OS uses XDG to manage these associations.

xdg 'desktop' files

The XDG system is a script which connects file types and applications. Suppose our browser is Chromium and we click on a website link to a PDF. Chromium makes a call to the XDG system (xdg-open). If we've registered an app for our PDF fileswith XDG, the application (eg. Evince) opens the PDF.

It's a chain, so if we haven't registered a default for PDF's in XDG, Chromium's call to XDG produces no information. In these circumstances, Chromium simply downloads the PDF. XDG itself has its own files and file types with which it makes these connections. We'll configure XDG to connect the M3U to VLC, the same way it connects a PDF to Evince.

This seems simple, but later we will find out that Chromium refuses to open M3U's even when XDG is properly configured for it. See "troubleshooting" further down the page.

m3u xdg registration

Our clickable schema depends on M3U's being played from the browser. However, XDG does not typically have a default application for M3U's. Until we configure one, browsers that contact XDG get no information. As noted above, browsers typically just download the M3U file. In order for the browser to process a click on an M3U hyperlink (without downloading), we must create an association between M3U's and an application. XDG manages this.

add a file type (arch) scroll down to xdg-open and perl-mime-types. Perl mime types is straightforward, and this worked IME. informative arch page. see also their additional page.
add a file type (stack) add an existing file type.
add a file type (stack) the most thorough description. Includes syntax for any file type.
add a file type (superuser) another method, slightly more superficial, for existing file types. Create a desktop file then add to mimeapps.list or run xdg-register.
add a file type (askubuntu) have an existing file type and need to associate it with an application.
list of associations (unix exchange) how to get a list of default file apps.

configure m3u

Verify M3U is already defined within the XDG system.

$ xdg-mime query filetype foo.m3u
m3u: audio/x-mpegurl

...or...

# pacman -S perl-mime-types [incl mimetype]
$ mimetype foo.m3u
m3u: audio/x-mpegurl

...then, to associate it to vlc, or whatever player....

$ mimeopen -d foo.m3u

...verify that (in this example) vlc was associated with it...

$ xdg-mime query default audio/x-mpegurl
vlc.desktop
# update-desktop-database
# update-mime-database

...or...

$ update-mime-database ~/.local/share/mime

verify file opens natively via xdg

$ xdg-open foo.m3u

it should open with vlc.

thumbnails

We need thumnails of CD insert/booklet art for our omnibus music page. Imagemagick is our friend for processing an entire directory of photos to provide us with thumgnails. NB: not sure which of its commands is destructive or additive resize, mogrify, or convert.

$ mogrify -format gif -path /home/foo/thumbs -thumbnail 100x100 *.jpg

troubleshooting

1. M3U access through browser

Install a browser such as Falkon which respects XDG settings for M3U's


Chromium will not open an M3U. Probably a DMCA protection, since M3U's can be built to do streaming, not simply play local files the way I use them. Priority (top of foodchain) is supposed to be the ~/.config/mimeapps.list, but Chromium does not honor any XDG M3U settings or files.

IME, the simplest, fastest solution to this Chromium problem is to install a browser such as Falkon, which respects xdg-open settings. For our music schema to work, we need a browser to open our HTML files.

$ cat .config/mimeapps.list
[Added Associations]
application/pdf=org.gnome.Evince.desktop;
image/jpeg=geeqie.desktop;
text/plain=org.gnome.gedit.desktop;
image/png=geeqie.desktop;
image/gif=vlc.desktop;geeqie.desktop;
video/mp4=xplayer.desktop;
video/mpeg=xplayer.desktop;
application/octet-stream=org.gnome.gedit.desktop;

[Default Applications]
application/pdf=org.gnome.Evince.desktop
image/jpeg=geeqie.desktop
text/plain=org.gnome.gedit.desktop
image/png=geeqie.desktop
image/gif=geeqie.desktop
video/mp4=xplayer.desktop
video/mpeg=xplayer.desktop
audio/x-mpegurl=vlc.desktop;

2. browser path requirements lead to permanent mount point naming

Create a mountpoint and identical /etc/fstab entry. Put it on all devices that need access to USB external drive. All links in our music setup will use these links.


Seems impossible but, when a browser opens an HTML page, the links to M3U's cannot be just the file, eg. "foo.m3u", even if the M3U is in the same folder with the HTML file. We're not used to this. HTML files easily display photos in the same directory or in a subfolder such as 'images'. But for the M3U to open, it must be called with the complete path to the file starting from its mount point eg, "/run/media/[USER]/[LABEL]/music/foo.m3u".

This poses a problem for the user. Each computer has a different username, and the "run" mountpoint is temporary. Gvfs or fusermount inserts the USER and partition LABEL when it mounts the drive, eg, /run/media/[USER]/[LABEL]/. But we can't change the HTML links to our 100+ M3U files every time we mount the USB back-up drive in a different system.

To pass the environmental variable of '$USER' into our URL is also not easy due to security problems with URL's on non-local systems that connect to internet. I tried USER, $USER, %USER%, 'USER', '$USER', '%USER%', `USER`, `$USER`, and `%USER%`. None worked.

To obtain USER, we simply whoami or a larger list of environmental variables with printenv. To determine LABEL, we can of course use 'lsblk', or the more complete...

$ lsblk -o name,mountpoint,label,size,uuid

The next level is a udev rule or fstab configuration that I would place on any machine I use with the backup drive. But GVFS is extremely powerful and udev, fstab, etc may only unreliably/unpredictably override GVFS.

I decided to try an fstab addition since this post (scroll down) made it seem the simplest solution. If I had done the udev rule, the persistent naming setup would have been from kernel detection.

In either case, we basically want to override gvfs when the UUID or LABEL of the backup USB is detected. Unfortunately, we will never be sure GVFS might be fickle on some system and disallow being overriden by /etc/fstab. But we must attempt it, otherwise we cannot use HTML and a browser to manage the media collection. The process is from this post.

  1. Create a permanent mount point. "run/media" is a temporary file system used by GVFS. I decided to create /mnt/[label], where 'label' is the label of the partition. In this case...
    # mkdir -p /mnt/bigdata
  2. update /etc/fstab, then do systemctl daemon-reload
    # nano /etc/fstab
    # UUID=ba60a72e-0db3-4a5f-bea5-c3be0e04cda1 LABEL=bigdata
    UUID=ba60a72e-0db3-4a5f-bea5-c3be0e04cda1 /mnt/bigdata xfs rw,auto,user 0 0
    # systemctl daemon reload
    # mount -a
  3. With the "mount all", the device should load and at that directory with proper permissions. We can verify...
    $ cat /etc/mtab
    /dev/sdd1 /mnt/bigdata xfs rw,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
    ...and of course try a test write and erase to the drive to verify user permissions.
  4. Now whenever we create a hyperlink in our music oversight HTML file, we can use a persisting, cross-platform, link. Eg, for the M3U, we might have an address of /mnt/bigdata/foo.m3u in the link. If we connect to any other systems, 1) create /mnt/bigdata, and 2) modify their fstab. All links to music and M3U's in our HTML page should then work.
  5. The USB drive will *not* appear in our temporary drive list in our file manager. We'll have to navigate to /mnt/bigdata to see or edit the drive's contents.

Saturday, November 25, 2023

crucial x6 (Micron - 0634:5602)

This is a $199 (early 2023)/$150 (late 2023) 4TB USBA (3.2) <-->USBC (Android) external SSD drive. We're capped at about 800MB of data transfer, but being external obviates annoying NVMe (PCI only) issues. With SSD's, only writing, not reading, decreases their lifespan. I'm thinking of these in terms of 10 year lifespans.

$ lsusb
Bus 003 Device 009: ID 0634:5602 Micron Technology, Inc. CT4000X6SSD9
$ lsblk
sdc 8:32 0 3.6T 0 disk
├─sdc1 8:33 0 128M 0 part
└─sdc2 8:34 0 3.6T 0 part /run/media/foo/Crucial X6
$ cat /etc/mtab
/dev/sdc2 /run/media/foo/Crucial\040X6 exfat rw,nosuid,nodev,relatime,uid=0500,gid=0500,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro 0 0

FS considerations

The X6 comes with an extended FAT (exfat) 3.6TB partion and thus syncs to MSoft devices with an added file management area "System Volume Information" of about 128Mb. Should we keep the exfat? Extended FAT has an upper recommended partion limit of 512Tb, so the drive is well-within it's useful partition size, and it's also compatible with Apple devices. YMMV but, in my systems, I prefer zero MSoft either in applications, file systems, or anything else.

A person might next consider formatting to ext2. I've found it reliable for more than a decade. However 2038 is the final time-stamp date available for ext2 files -- ext2 was released in 1993 -- so ext2 is unfortunately nearing some expiration considerations.

Luckily, any FS we want will do fine: external drives don't need to boot, and we only need a single partition to store data. BtrFS supposedly has flexible inode size (so can manage smaller and larger files side by side), but Oracle was involved in its development. Currently, I believe xfs is worth a try, in absence of ext2. Xfs is the default on RedHat systems so it has active development. As for ext4, this Reddit post compares ext4 and xfs. We may also want to soon consider zfs, since it's gaining momentum with large stakeholders.

XFS

Pretty sure xfs will not natively mount in a Windows or iOS systems and has the option of encryption, but it also has a drawback: appropriates journaling space. RedHat has info, of course, and the Arch guide is also good. XFS.org also has its own FAQ.

# umount /dev/sdc
# gparted /dev/sdc [eliminate present stuff, add xfs partition]
# mkfs.xfs -L "bigdrive" /dev/sdc1
# xfs_repair /dev/sdc1
# chown 0500:0100 /run/media/foo/bigdrive

The reason for the xfs_repair is to verify it formatted successfully.

The reason for the chown: for some reason, gvfs mounts xfs as root:root (0:0) instead of the standard user:group setup. Since gvfs mounts other USB drives to group 100, I chown-ed the drive to UID:group, that is to 0500:0100. Can also chown to 0500:0500 if desired, that is user:user (supposing, eg one's UID were 500).

I only had to chown it once and gvfs subsequently auto-mounted the drive with the correct permissions.

If stripe and strip parameter warnings appear, read this and ignore them. That's for RAIDs.

Monday, November 20, 2023

media back-up

This post deals with audio CD's and DVD video -- I have no BluRay media. And it's sort of a worst-case scenario, one where a person can't physically save their media. I've learned an immense amount because it looks like an straighforward project, but it's not. I've written another blog post covering most of the remainder of it.


I recently cleared-out my storage area. I hadn't been in there in 17 years. A few boxes of CD's and DVD's were part of the contents, and the memories of the times when I purchased the media surged in an instant. They welled-up so quickly and powerfully that I choked-up. How had I become so weak and disfigured? Uncomfortable insights.

In spite of such anguish, a person is unlikely to trash the media. It's natural for a person to honor their history. At some point, I want to light a cigar, take a keepsake media disc off a shelf, and put it in a player for a rewatch/relisten.

If no home with a shelf, then very small storage areas can still be rented for what we used to pay for large ones. Ie, rent for what used to be an 8x10', might now rent a 4x4'. A 4x4 can hold a few CD's and a few papers, etc. Maybe $45.

Minus a millionaire's floorplan or an affordable storage area, a person realizes their options are down to one: back the media up, probably in some mediocre fashion/bitrate without cover art, and probably without individual tracks or chapters. Is it even worth it at all? As in every other post on this blog, the answer is "it's entirely up to you", and "the info here pertains to Linux". I found it was worth it for some discs, and others I just threw away. It's about 20 mins per CD, and about X mins per DVD.


TLDR

audio: 20 mins per CD

  • abcde to backup the files and to import online CDDB data for ID3 tags. Verify the fetching CDDB URL inside abcde.conf
  • Easytag (or some ppl prefer picard) to clean up ID3 tags.
  • Sometimes groups of MP3's will require renaming. Gprename can hasten this.
  • Cover art files do not always download and will not be the complete inner notes. While abcde is working, scan the inner liner at 200-300 dpi as close as possible to a square (1:1) crop
  • Later, can make thumbnails for the HTML oversight page.
$ abcde -d /dev/sr0 -o mp3:"-b 320" -c ~/.config/abcde.conf

dvd: handbrake to backup the files.


playback note

See my post, which suggests an HTML/M3U file solution. Some may wish to install LibreELEC and Kodi in an old laptop to work as a "CD player". All of the engineered "media manager" products are bullsh*t IMO -- more concerned with phoning home your collection than any other task. A simple HTML page works much faster, and more reliably and configurably.

If we formatted our external back-up drive with XFS, then it will not work with a Windows or Mac device. So we will at least need some device to access the files. And a 3.5mm jack in the device, to move the audio to a computer speaker system or some headphones. Alternatively, we could design some static web pages and make thumbnails of of the CD covers with built-in links to playlists or which open the folders and play the contents. The specs for m3u's are here.

Although I run icewm, my MIME settings are handled by xdg. So it will play an MP3 link, but download an M3U link.

$ xdg-settings get default-web-browser
chromium.desktop

Applications work as .desktop files. Not sure.

audio 90-100Mb per CD

Time and discwise, a CD like Fleetwood Mac's Rumours, about 40 mins long, takes about 12 min to rip and transcode at 320k. The 320K bitrate leads to a 90Mb folder of files, or roughly 2.3Mb per minute (at 320Kb). I find 90Mb per CD unproblematic at today's drive prices.

Quality wise, I prefer a full 320Kb MP3, esp if the music is nuanced with flanging or orchestration. IMO, 192K can be OK, but 128K is definitely not OK unless just a speech or some such.

Probably abcde is still the easiest attack on a stack of backup CD's. We need some way to save id3 information, and probably edit it (eg, easytag). Installing abcde will also pull-in python-eyed3. I installed glyph out of an abundance of caution for photo embedding.

$ yay -S abcde id3 python-eyed3
# pacman -S glyph id3v2

audio - abcde use and config

Abcde has an old school text conf file. Get the skeleton from /etc and copy it to wherever to edit it, eg ~/.config, then do edits like the CDDB URL and save. Next, adjust the URL where abcde finds ID3 information to populate the files.

$ cp /etc/abcde.conf ~/.config/abcde.conf
$ nano ~/.config/abcde.org
CDDBURL="gnudb.gnudb.org/~cddb/cddb.cgi"

When executing, the config file is called with the "c" flag and path.

$ abcde -c "~/.config/abcde.conf"

Abcde does not rip directly to MP3; it rips the entire CD to WAV's (10 mins), then converts each WAV to whatever number of other formats we want. From rip to conversion to one other format is about 12 mins per CD. Native MP3 conversion bit rate is 128k. Of course, if a person simply wants the WAV's, then use the defaults, eg $ abcde -d /dev/sr0.

For this project I'm likely to to specify MP3's at 320kb, though a person can specify what they like (or leave off the ':"-b 320"' for 128k).

$ abcde -d /dev/sr0 -o mp3:"-b 320"

audio - ID3 info

ID3 is no longer reliably being developed and there are different versions which might conflict. Of the versions, I consider ID3v2 compatible with nearly any player and therefor a reliable ID3 format for my MP3 files. ID3v2 is also supposedly the ID3 version EasyTag apparently uses. So after first tagging some MP3 files with EasyTag, I verified the ID3 version by installing id3v2. Then I ran its utility in a terminal, $ id3v2 -l [name].mp3, to verify my ID3 tags were version 2, ie, ID3v2.

$ id3v2 -l 02.Dreams.mp3
id3v2 tag info for 02.Dreams.mp3:
TIT2 (Title/songname/content description): Dreams
TPE1 (Lead performer(s)/Soloist(s)): Fleetwood Mac
TALB (Album/Movie/Show title): Rumours
TYER (Year): 1977
TRCK (Track number/Position in set): 02/11
TCON (Content type): Rock (17)
02.Dreams.mp3: No ID3v1 tag

Even with a functional CDDB URL in our ~/.config/abcde.conf , we often need to do the ID3 and art manually.

# pacman -S easytag
$ yay -S gprename

audio - ID3 art

It appears EasyTag only recognizes photos if gdk-pixbuf2 is installed.

# pacman -S gdk-pixbuf2

I arrange the scanner (xsane) for 300dpi, and preview the image close to a square. The resulting scan I then then scale. Get the WxH to about 1400x1400. This makes about a 500Kb file to add to each music file in that folder, so if 10 files in the CD, it adds 5MB to the entire folder. The 1K x 1K pic will display pretty sharp if running the file in VLC. Not sure other players. If I have an internal booklet in the CD, I scan it all and make a PDF to put in the folder.

Adding the art: put the image in the directory with the MP3's, open EasyTag in that directory, selecting all the files I want to receive that image (typically cover image). Then I go to the right and select "images" tab, add the pic with the "+" icon, and saving it to the files. A person will figure it out fast.

Removing art is fickle. I can do it in EasyTag, but it seems to stay in the VLC cache so I can never be sure. A sure way is to 1) Delete the directory for VLC album art: ~/.cache/vlc/ and, 2) go nuclear via "cd"ing into the music files directory and then...

$ id3v2 -r "APIC" *.mp3

audio - manual tasks

If I find old directories full of WAV's, like CD's I made back when, or whatever, I can batch convert the WAV's into 320k cbr MP3's.

$ for f in *.wav ; do lame -b 320 $f ; done

If I find a track that I want to slow it down (I don't mind pitch) and put a 0.8 speed version in the CD directory, perhaps to use with mixes, then...

$ sox foo.mp3 -C 320 foo8.mp3 speed 0.8

If I want to combine all the tracks in the CD into a blob, so I can play easily when driving or whatever...

$ sox foo1.mp3 foo2.mp3 foo3.mp3 output.mp3

There's a lot more about sox here.

Then I can get the art for it online or design something. Either way, put it in the folder with the files and add the art and info via EasyTag.

dvd 1.3G each

dvd - art

Use at least 300x300 ppi and some similar dimension. Give it the same name as the film file, but a JPG extension. When company arrives, they can easily peruse a collection with a file manager, or by browsing the photos in something like Geeqie.

dvd - content

Most movies (1.7 hrs) are 1.1GB (480P) and about 20 mins to back-up, assuming a person continues their other computer tasks throughout the encode. Episodes of TV shows are typically. If I have a higher-res recent DVD, I use the Fast(not Super-Fast)720p setting in Handbrake. It's 2 passes. Otherwise I used 480p Fast. run Fast The HandBrake software install (see picture below) is about 80Mb.

The rip is two parts, extraction and transcoding, and DVD structure is described here.


First, don't forget the obvious...

# pacman -S libdvdcss libdvdread libdvdnav

...since literally *none* of the players will do anything but spawn errors without explanation if these are lacking.

One fairly simple Arch solution is HandBrake.

# pacman -S handbrake

To execute, ghb which, along with icon, a person can find with ye olde...

$ pacman -Ql handbrake

...and there are flags of course, not sure where to locate.

I start HandBrake from a terminal (again, "ghb") even though it's a GUI. This is because the CSS keys take a minute or two to propagate and I can't see when they're finished within the application. Once I see the keys have propagated, I can return to GUI selection options.

dvd - storage

Probably need to think about file structure. At least like "movies, bbc, adultswim, sitcoms, detective". Or some such. Then include the scan of the cover art with the file

Wednesday, November 8, 2023

portable (usb) disk recovery

We've nearly all dropped an important HDD on the way to the airport or work and know the (sinking) feeling later that day when data can't be accessed. It's expensive, slow, and intrusive to have data recovered professionally.

So we've mostly switched to SSD's, but even these are of questionable1 reliability.

levels of loss

  • level of destruction -- unable to work at all, unable to be detected, unable to boot
  • type of drive SSD, USB, SDC, HDD
  • file system (ext2, ntfs, reiser, etc)

The easiest thing, the thing we'd like to do, is reset the dirty bit, or re-establish a corrupted MBR, and be able to plug our drive back in for normal use. Then we find that each crash can have a different level of complexity -- the dirty bit is often possible, the corrupted boot sector typically not.

Some of our luck in a crash also has to do with the file system. If a person has an HDD formatted in NTFS, then it needs a bootable NTFS sector even if it's just a portable data disk.

expectations: think data first, access second

The data crash reveals the drive to be unreliable; if I greedily aim to re-establish a boot sector, I might lose the data if the boot sector attempt fails. Check to see if the data is intact and shoot for that first, even if it's more time consuming than restoring access through a repaired boot.

software: testdisk

At the start of the fail I cycled through all my disk software, fdisk, cfdisk, gparted, ntfsfixbootand best so far seems to be testdisk. Also might take a look at wipefreespace on the AUR.

Data Recovery with Test Disk (18:22) oswyguy, 2018. This poor guy needs sunlight, but a superb TestDisk video beginning about 7:30.

WD Passport 1000GB, (1058:0748)

I employed the usuals to gather info and attempt repairs, cfdisk, fdisk, gparted, and from the AUR, . None of them did anything

pacman -S testdisk

This retrieved files, but didn't repair the MBR index.

nope

Zero
pacman -S ntfsffixboot
No
pacman -S ntfsfix

1Consider this SSD drive article, from which I quote:

According to the Western Digital's firmware update page, the impacted products are from the SanDisk Extreme Portable SSD V2, SanDisk Extreme Pro Portable SSD V2, and WD My Passport SSD line, and lists the models as follows:

SanDisk Extreme Portable 4TB (SDSSDE61-4T00)
SanDisk Extreme Pro Portable 4TB (SDSSDE81-4T00)
SanDisk Extreme Pro 2TB (SDSSDE81-2T00)
SanDisk Extreme Pro 1TB (SDSSDE81-1T00)
WD My Passport 4TB (WDBAGF0040BGY)

And these drives are still best-sellers on Amazon, so suppliers are being shady AF. SSD's rely on firmware but still must be properly constructed, of course. HDD's were mostly just a hardware situation.