sample video edit (less than 1 minute)

Side Note: TIME CALCULATOR. This post is FFMPEG, but there's a UAV guy who has a YT channel, who works nearly exclusively with SHOTCUT. Some of the effects are amazing, eg his video on smoothing. There's also an FFMPEG webpage with pan tilt and zooming info not discussed this post. For smoother zooming, look here at pzoom possibilities. Finally, for sound and video sync, this webpage is the best I've seen. Sync to the spike.

---

Suppose I use a phone for a couple short videos, maybe along the beach. One is 40 seconds, the other 12. On the laptop I later attempt to trim away any unwanted portions, crossfade them together, add some text, and maybe add audio (eg. narration, music). This might take 2 hours the first or second attempt: it takes time for workflows to be refined/optimized for one's preferences. Although production time decreases somewhat with practice, planning time is difficult to eliminate entirely: every render is lossy and the overall goal (in addition to aesthetics) is to accomplish editing with a minimum number of renders.

normalizing

The two most important elements to normalize before combining clips of different sources are the timebase and the fps. Ffmpeg can handle most other differing qualities: aspect ratios, etc. There are other concerns for normalizing depending on what the playback device is. I've had to set YUV on a final render to get playback on a phone before. But this post is mostly about editing disparate clips.

Raw video from the phone is in 1/90000 timebase (ffmpeg -i), but ffmpeg natively renders at 1/11488. Splicing clips with different timebases fails, eg crossfades will exit with the error...

First input link main timebase (1/90000) do not match the corresponding second input link xfade timebase (1/11488)

Without a common timebase, the differing "clocks" cannot achieve a common outcome. It's easy to change the timebase of a clip, however it's a render operation. For example, to 90000...

$ ffmpeg -i video.mp4 -video_track_timescale 90000 output.mp4

If I'm forced to change timebase, I attempt to do other actions in the same command, so as not to waste a render. As always, we want to render our work as few times as possible.

separate audio/video

Outdoor video often has random wind and machinery noise. We'd like to turn it down or eliminate it. To do this, we of course have to separate the audio and video tracks for additonal editing. Let's take our first video, "foo1.mp4", and separate the audio and video tracks. Only the audio is rendered, if we remember to use "-c copy" on the video portion, to prevent video render.

$ ffmpeg -i foo1.mp4 -vn -ar 44100 -ac 2 audio.wav
$ ffmpeg -i foo1.mp4 -c copy -an video1.mp4

cropping*

*CPU intensive render, verify unobstructed cooling.

This happens a lot with phone video. We want some top portion but not the long bottom portion. Most of my stuff is 1080p across the narrow portion, so I make it 647p tall for a 1.67:1 golden ratio. 2:1 would also look good.

$ ffmpeg -i foo.mp4 -vf "crop=1080:647:0:0" -b 5M -an cropped.mp4

The final zeroes indicate to start measuring pixels in upper left corner for both x and y axes respectively. Without these, the measurement starts from center of screen. Test the settings with ffplay prior to the render. Typically anything with action will require 5M bitrate on the render, but this setting isn't needed during the ffplay testing, only the render.

cutting

Cuts can be accomplished without a render if the "-c copy" flag is used. Copy cuts occur on the nearest keyframe. If a cut requires the precision of a non-keyframe time, the clip needs to be re-rendered. The last one in this list is an example.

• no recoding, save tail, delete leading 20 seconds. this method places seeking before the input and it will go to the closest keyframe to 20 seconds.

  $ ffmpeg -ss 0:20 -i foo.mp4 -c copy output.mp4

• no recoding, save beginning, delete tailing 20 seconds. In this case, seeking comes after the input. Suppose the example video is 4 minutes duration, but I want it to be 3:40 duration.

  $ ffmpeg -i foo.mp4 -t 3:40 -c copy output.mp4

• no recoding, save an interior 25 second clip, beginning 3:00 minutes into a source video

  $ ffmpeg -ss 3:00 -i foo.mp4 -t 25 -c copy output.mp4

• a recoded precision cut

  $ ffmpeg -i foo.mp4 -t 3:40 -strict 2 output.mp4

2. combining/concatentation

Also see further down the page for final audio and video recombination. The section here is primarily for clips.

codec and bitrate

If using ffmpeg, then mpeg2video is the fastest lib, but also creates the largest files. Videos with a mostly static image, like a logo, may only require 165K video encoding. A 165K example w/125K audio, 6.5MB for about 5 minutes. That said, bitrate is the primary determiner of rendered file size. Codec is second but important, eg, libx264 can achieve the same quality at a 3M bitrate for which mpeg2video would require a 5M bitrate.

simple recombination 1 - concatenate (no render)

The best results come from combine files with least number of renders. This way does it without rendering... IF files are the same pixel size and bit rate, this way can be used. Put the names of the clips into a new TXT file, in order of concatenation. Absolute paths is a way to be sure. Each clip takes one line. The one here shows both without and with absolute path.

$ nano mylist.txt
# comment line(s)
file video1.mp4
file '/home/foo/video2.mp4'

The command is simple.

$ ffmpeg -f concat -safe 0 -i mylist.txt -c copy output.mp4

Obviously, transitions which fade or dissolve require rendering; either way, starting with a synced final video, with a consistent clock (tbr), makes everything after easier.

simple recombination 2 - problem solving

Most problems come from differing tbn, pixel size, or bit rates. TBN is the most common. It can be tricky though, because the video after the failing one appears to cause the fail. Accordingly comment out files in the list to find the fail, then try replacing the one after it.

1. tbn: I can't find in the docs whether the default ffmpeg clock is 15232, or 11488, I've seen both. Most phones are on a 90000 clock. If the method above "works", but reports many errors and the final the time stamp is hundreds of minutes or hours long, then it must be re-rendered. Yes it's another render, but proper time stamps are a must. Alternatively, I suppose a person could re-render each clip with the same clock. I'd rather do the entire file. As noted higher up in the post, raw clips from a phone usually use 1/90000 but ffmpeg uses 1/11488. It's also OK to add a gamma fix or anything else, so as not to squander the render. The example here I added a gamma adjustment

   $ ffmpeg -i messy.mp4 -video_track_timescale 11488 [or 15232] -vf "eq=gamma=1.1:saturation=0.9" output.mp4

combine - simple, one file audio and video

I still have to specify the bitrate or it defaults too low. 3M for sports, 2M for normal person talking.

$ ffmpeg -i video.mp4 -i audio.wav -b:v 3M output.mp4

combine - simple, no audio/audio (render)

If the clips are different type, pixel rate, anything -- rendering is required. Worse, mapping is required. Leaving out audio makes it slightly less complex.

ffmpeg -i video1.mp4 -i video2.flv -an -filter_complex \ "[0:v][1:v] concat=n=2:v=1 [outv]" \ -map "[outv]" out.mp4

Audio adds an additional layer of complexity

ffmpeg -i video1.mp4 -i video2.flv -filter_complex \ "[0:v][0:a][1:v][1:a] concat=n=2:v=1:a=1 [outv] [outa]" \ -map "[outv]" -map "[outa]" out.mp4

combining with effect (crossfade)

*CPU intensive render, verify unobstructed cooling.

If you want a 2 second transition, run the offset number 1.75 - 2 seconds back before the end of the fade-out video. So, if foo1.mp4 is a 12 second video, I'd run the offset to 10, so it begins fading in the next video 2 seconds prior to the end of foo1.mp4. Note that I have to use filter_complex, not vf, because I'm using more than one input. Secondly, the offset can only be in seconds. This means that if the first video were 3:30 duration, I'd start the crossfade at 3:28, so the offset would be "208".

$ ffmpeg -i foo1.mp4 -i foo2.mp4 -filter_complex xfade=transition=fade:duration=2:offset=208 output.mp4

If you want to see how it was done prior to the xfade filter, look here, as there's still a lot of good information on mapping.

multiple clip crossfade (no audio)

Another scenario is multiple clips with the same transition, eg a crossfade. In this example 4 clips (so three transitions), each clip 25 seconds long. A good description.

$ ffmpeg -y -i foo1.mp4 -i foo2.mp4 \
-i foo3.mp4 -i foo4.mp4 -filter_complex \
"[0][1:v]xfade=transition=fade:duration=1:offset=24[vfade1]; \
[vfade1][2:v]xfade=transition=fade:duration=1:offset=48[vfade2]; \
[vfade2][3:v]xfade=transition=fade:duration=1:offset=72" \
-b:v 5M -s wxga -an output.mp4

Some additional features in this example: y to overwrite prior file, 5M bitrate, and size wxga, eg if reducing quality slightly from 4K to save space. Note that the duration mesh time increases the total offset cumulatively. I touched a TXT file and entered the values for each clip and its offset. Then just "cat"ted the file to see all the offset values when I built my command. Suppose I had like 20 clips? The little 10ths and so on might add up. Offset numbers off by more than a second will not combine with the next clip, even though syntax is otherwise correct.

$ cat fooclips.txt
foo1 25.07 - 25 (24)
foo2 25.10 - 50 (48)
foo3 25.10 - 75 (72)
foo4 (final clip doesn't matter)

multiple clip crossfade (with audio)

This is where grown men cry. Have a feeling if I can get it once, won't be so bad going forward but, for now, here's some information. It appears some additional programs beide crossfade and xfade.

fade-in/out*

*CPU intensive render, verify unobstructed cooling.

If you want a 2 second transition, run the offset number 1.75 - 2 seconds back before the end of the fade-out video. Let's say we had a 26 second video, so 24 seconds.

$ ffmpeg -i foo.mp4 -max_muxing_queue_size 999 -vf "fade=type=out:st=24:d=2" -an foo_out.mp4

color balance

Recombining is also a good time to do even if just a basic eq. I've found that general color settings (eg. 'gbal' for green) have no effect, but that fine grain settings (eg. 'gs' for green shadows) has effects.

$ ffmpeg -i video.mp4 -i sound.wav -vf "eq=gamma=0.95:saturation=1.1" codec:v copy recombined.mp4

There's an easy setting called "curves", like taking an older video and moving midrange from .5 to .6 helps a lot. Also, if bitrate is specified, give it before any filters; bitrate won't be detected after the filter.

$ ffmpeg -i video.mp4 -i sound.wav -b:v 5M -codec:v mpeg2video -vf "eq=gamma=0.95:saturation=1.1" recombined.mp4

Color balance intensity of colors. There are 9 settings - 1 each for RGB (in that order) for shadows, midtones, and highlights, separated by colons. For example, if I wanted to decrease the red in the highlights and leave all others unchanged...

$ ffmpeg -i video.mp4 -i -b:v 5M -vf "colorbalance=0:0:0:0:0:0:-0.4:0:0" output.mp4

A person can also add -pix_fmt yuv420p, if they want to make it most compatible with Windows

FFMpeg Color balancing(3:41) The FFMPEG Guy, 2021. 2:12 color balancing shadows, middle, high for RGB
why films are shot in 2 colors  (7:03) Wolfcrow, 2020. Notes that skin is the most important color to get right. The goal is often to go with two colors on oppposite ends of the color wheel or which are complementary.
PBX - on-site or cloud  (35:26) Lois Rossman, 2016. Cited mostly for source 17:45 breaks down schematically.
PBX - true overhead costs  (11:49) Rich Technology Center, 2020. Average vid, but tells hard facts. Asteriks server ($180) discussed.

adding text

*CPU intensive render, verify unobstructed system cooling.

For one or more lines of text, we can use the "drawtext" ffmpeg filter. Suppose we want to display the date and time of a video, in Cantarell font, for six seconds, in the upper left hand corner. If we have a single line of text, we can use ffmpeg's simple filtergraph (noted by "vf"). 50 pt font should be sufficient size in 1920x1080 video.

$ ffmpeg -i video.mp4 -vf "[in]drawtext=fontfile=/usr/share/fonts/cantarell/Cantarell-Regular.otf:fontsize=50:fontcolor=white:x=100:y=100:enable='between(t,2,8)':text='Monday\, January 17, 2021 -- 2\:16 PM PST'[out]" videotest.mp4

Notice that a backslash must be added to escape special characters: Colons, semicolons, commas, left and right parens, and of course apostrophe's and quotation marks. For this simple filter, we can also omit the [in] and [out] labels. Here is a screenshot of how it looks during play.

Next, supposing we want to organize the text into two lines. We'll need one filter for each line. Since we're still only using one input file to get one output file, we can still use "vf", the simple filtergraph. 10pixels seems enough to separate the lines, so I'm placing the second line down at y=210.

$ ffmpeg -i video.mp4 -vf "drawtext=fontfile=/usr/share/fonts/cantarell/Cantarell-Regular.otf:fontsize=50:fontcolor=white:x=100:y=150:enable='between(t,2,8)':text='Monday\, January 18\, 2021'","drawtext=fontfile=/usr/share/fonts/cantarell/Cantarell-Regular.otf:fontsize=50:fontcolor=white:x=100:y=210:enable='between(t,2,8)':text='2\:16 PM PST'" videotest2.mp4

We can continue to add additional lines of text in a similar manner. For more complex effects using 2 or more inputs, this 2016 video is the best I've seen.

Ffmpeg advanced techniques pt 2 (19:29) 0612 TV w/NERDfirst, 2016. This discusses multiple input labeling for multiple filters.

PNG incorporation

If I wanted to do several lines of information, an easier solution than making additional drawtexts, is to create a template the same size as the video, in this case 1980x1080. Using, say GiMP, we could create picture with an alpha template with several ines that we might use repeatedly, and save in Drive. There is then an ffmpeg command to superimpose a PNG over the MP4.

additional options (scripts, text files, captions, proprietary)

We of course have other options for skinning the cat: adding calls to text files, creating a bash script, or writing python code to call and do these things.

The simplest use of a text files are calls from the filter in place of writing the text out each filter.

viddyoze: proprietary,online video graphics option. If no time for Blender, pay a little for the graphics and they will rerender it on the site.

viddyoze review (14:30) Jenn Jager, 2020. Unsponsored review. Explains most of the 250 templates. Renders to quicktime (if alpha), or MP4 is not.~12 minute renders

adding text 2

Another way to add text is by using subtitles, typically with an SRT file. As far as I know so far, these are controlled by the viewer, meaning not "forced subtitles" which override viewer selection. here's a page. I've read some sites on forced subtitles but haven't yet been able to do this with ffmpeg.

audio and recombination

Ocenaudio makes simple edits sufficient for most sound editing. It's user friendly along the lines of the early Windows GoldWave app 25 years ago. I get my time stamp from the video first.

$ ffmpeg -i video.mp4

Then I can add my narration or sound after being certain that the soundtrack is exactly a map for the timestamp of the video. I take the sound slightly above neutral "300" when going to MP3 to compensate for transcoding loss. 192K is typically clean enough.

$ ffmpeg -i video.mp4 -i sound.wav -acodec libmp3lame -ar 44100 -ab 192k -ac 2 -vol 330 -vcodec copy recombined.mp4

I might also resize it for emailing, down to VGA or SVGA size. Just change it thusly...

$ ffmpeg -i video.mp4 -i sound.wav -acodec libmp3lame -ar 44100 -ab 192k -ac 2 -vol 330 -s svga recombined.mp4

$ ffmpeg -i video.mp4 -i sound.wav -acodec libmp3lame -ar 44100 -ab 192k -ac 2 -vol 330 -vcodec copy recombined.mp4

I might also resize it for emailing, down to VGA or SVGA size. Just change it thusly...

$ ffmpeg -i video.mp4 -i sound.wav -acodec libmp3lame -ar 44100 -ab 192k -ac 2 -vol 330 -s svga recombined.mp4

For YouTube, there's a recommended settings page, but here's a typical setup:

$ ffmpeg -i video.mp4 -i sound.wav -vcodec copy recombined.mp4

ocenaudio - no pulseaudio

If a person is just using alsa, without any pulse, they may have difficulty (ironically) using ocenaudio, if HDMI is connected. A person has to go into Edit->preferences, select the ALSA backend, and then play a file. Keep trying your HDMI ports until you luck on the one with EDID approved.

audio settings for narration

To separately code the audio in stereo 44100, 192K, ac 2, some settings below for Ocenaudio: just open it and hit the red button. Works great. Get your video going, then do the audio.

Another audio option I like is to create a silent audio file exactly the length of the video, and then start dropping in sounds into the silence, hopefully in the right place with audio I may have. Suppose my video is 1:30.02, or 90.02 seconds

$ sox -n -r 44100 -b 16 -c 2 -L silence.wav trim 0.0 90.02

Another audio option is to used text to speeech (TTS) to manage some narration points. The problem is how to combine all the bits into a single audio file to render with the audio. The simplest way seems to be to create the silence file then blend. For example, run the video in a small window, open ocenaudio and paste at the various time stamps. By far the most comprehensive espeak video I've seen.

How I read 11 books(6:45) Mark McNally, 2021. Covers commands for pitch, speed, and so on.

phone playback, recording

The above may or may not playback on a phone. If one wants to be certain, record a 4 second phone clip and check its parameters. A reliable video codec for video playback on years and years of devices:

-vcodec mpeg2video -or- codec:v mpeg2video

Another key variable is yuv. Eg,...

$ ffmpeg -i foo.mp4 -max_muxing_queue_size 999 -pix_fmt yuv420p -vf "fade=type=out:st=24:d=2" -an foo_out.mp4

To rotate phone video, often 90 degree CCW or CW, requires the "transpose" vf. For instance this 90 degree CCW rotation...

$ ffmpeg -i foo.mp4 -vf transpose=2 output.mp4

Another issue is shrinking the file size. For example, mpeg2video at 5M, will handle most movement and any screen text but creates a file that's 250M for 7 minutes. Bitrate is the largest determiner of size, but also check the frame rate (fps), which sometimes can be cut down to 30fps ( -vf "fps=30") if it's insanely high for the action. Can always check stuff with ffplay to get these correct before rendering. Also, if the player will do H264, then encoding in H264 (-vcodec libx264) at 2.5M bitrate looks similar to 5M in MPEG2. Which means about 3/5 the file size.

google cloud services initialization (cloud, colab, sites, dns, ai)

Some of Google's web services have their own names but are tied together with GCP (Google Cloud Platform) and/or some specific GCP API. GCP is at the top, but a person can enter through lesser services and be unaware of the larger picture. Again, GCP is at the top, but then come related but semi-independent services, Colab, Sites, AI. In turn, each of these might rely on just a GCP API, or be under another service. For example, Colab is tied into GCP, but a person can get started in it through Drive, without knowing its larger role. When a person's trying to budget, it's a wide landscape to understand exactly for what they are being charged, and under which service.

Static Google Cloud site (9:51) Stuffed Box, 2019. Starting at the top and getting a simple static (no database) website rolling. One must already have purchased a DNS name.
Hosting on Cloud via a Colab project (30:32) Chris Titus Tech, 2019. This is a bit dated, so prices have come up, but it shows how it's done.
Hosting a Pre-packed Moodle stack (9:51) Arpit Argawal, 2020. Shows the value of a notebook VM in Colab
Hosting on Cloud via a Colab project (30:32) Chris Titus Tech, 2019. This is a bit dated, so prices have come up, but it shows how it's done.

Google's iPython front-end Colab takes the Jupyter concept one-further, placing configuration and computing on a web platform. Customers don't have to configure an environment on their laptop, everything runs in the Google-sphere, and there are several API's (and TensorFlow) that Google makes available.

During 2020, the documentationi was a little sparse, so I made a post here, but now there are more vids and it's easier to see how we might have different notebooks running on different servers, across various Google products. This could also include something where we want to run a stack, eg for a Moodle. If all this seems arcane, don't forget we can host traditionally through Google Domains. What's going to be interesting is how blockchain changes the database concept in something like Moodle. Currently, blockchain is mostly for smart contract and DAPPs.

Colab

Notebooks are created, ran, and saved via the Drive menu, or go directly to colab.research.google.com. Users don't need a Google Cloud account to use Colab. Easiest way to access Colab is to connect it to one's Drive account, where it will save files anyway. Open Drive, click on the "+" sign to create a new file and go to down to "More". Connect Colab and, from then on, Colab notebooks can be created and accessed straight from Drive.

There's a lot you can do with a basic Colab account, if you have a good enough internet connection to keep up with it. The Pro version is another conversation. I often prefer to strengthen Colab projects by adding interactivity with Cloud.

GUI Creation in Google Colab (21:31) AI Science, 2020. Basics for opening a project and getting it operational.
Blender in Colab (15:28) Micro Singularity, 2020. Inadvertently explains an immense amount about Colab, Python, and Blender.

Colab and Google Cloud

Suppose one writes Python for Colab that needed to call a Google API at some point. Or suppose a person wanted to run a notebook on a VM that they customized? These are the two added strengths of adding Cloud: 1) make a VM (website) to craft a browser project, 2) add Google API calls. Google Cloud requires a credit card.

Cloud and Colab can be run separately, but fusing them is good in some cases. Gaining an understanding of the process allows users to know when to rely on either Colab or Google Cloud or interdependently.

Colab vs. Google Cloud (9:51) Arpit Argawal, 2020. Shows the value of a notebook VM in Colab
Hosting on Cloud via a Colab project (30:32) Chris Titus Tech, 2019. This is a bit dated, so prices have come up, but it shows how it's done.

Note the Google Cloud platform homepage above. The menu on the left is richer than the one in the Colab screenshot higher above. We run the risk of being charged for some of these features so that Google will display potential estimated charges before we submit our requests to use Google API's.

API credentials

We might want to make API calls to Cloud's API's. Say that a Colab notebook requires a Google API call, say to send some text for translation to another language. The user switches to their Cloud account and selects the Google API for translation. Google gives them an estimate of what calls to that API will cost. The user accepts the estimate, and then Google provides the API JSON credentials, which are then pasted into their Colab notebook. When the Colab notebook runs, it can then make the calls to the Google API. Protect such credentials because we don't want others to use them against our credit card.

Cloud account VM details

In the case of running notebooks and you update something, did it update on your machine or googles. its more clear on Google Cloud.

API dependencies

When a person first opens a Colab notebook, it's on a Google server, and the path for the installed Python is typically /usr/local/lib/python/[version]. So I start writing code cells, and importing and updating API dependencies. Google will update all the dependencies on the virtual machine it creates for your project on the server. ALLEGEDLY.

Suppose I want to use google-cloud-texttospeech. Then the way to updates its dependencies (supposedly):

% pip install --upgrade google-cloud-texttospeech

Users can observe all the file updates necessary for the API, unless they add the "-quiet" flag to suppress it. However, no matter that this process is undertaken, when the API itself is called, there can be dependency version problems between Python and iPython.

Note that in the case above the code exits with a "ContextualVersionConflict" listing a 1.16 version detected in the given folder. (BTW, this folder is on the virtual machine, not one's home system). Yet the initial upgrade command AND a subsequent "freeze" command show the installed version as 1.22. How can we possibly clear this since Google has told itself that 1.22 is installed, but the API detects version 1.16? Why are they looking in different folders? Where are they?

problem: restart the runtime

Python imports, iPython does not (page) Stack Overflow, 2013. Notes this is a problem with sys.path.

You'd think of course that there's a problem with sys.path, and to obviate *that* problem, I now explicitly import the sys and make sure of the path in first two commands...

import sys
sys.path.append('/usr/local/lib/python3.6/dist-packages/')

... in spite of the fact these are probably being fully accomplished by Colab. No, the real problem, undocumented anywhere I could find, is that one simply has to restart the runtime after updating the files. Apparently, if the runtime is not restarted, the newer version stamp is not reread into the API.

what persists and where is it?

basic checklist

• login to colab.
• select the desired project.
• run the update cell
• restart the runtime
• proceed to subsequent cells
• gather any rendered output files from the folder icon to the left (/content folder). Alternatively, one can hard code it into the script so that they transfer to one's system:

  from google.colab import files
  files.download("data.csv")

  There might also be a way to have these sent to Google Drive instead.

toolbox :: voice-over

contents
1. voice: a) detection* b) levels and record [alsamixer, audacity] c) cut and polish [goldwave]	3. sync: solution is slowdown video (to about .4) for clap, then work pieces
2. tts	4. notes

* typically requires ~/.asoundrc authorship

---

This post concerns two voiceover methods -- one human, the other using Text-To-Speech (TTS). Both of course have to be synced to video. Hopefully, the post also helps those doing audio w/o video.

1. Voice

input

I currently use a USB connected, non XLR, microphone, the Fifine K053 ($25, USB ID 0c76:161f).

Cleaner options are available at higher price points, eg phantom power supplies and so on to prevent hum, or XLR pre-amping. Either way, what we'd prefer is to see our mic's input levels as we're speaking, and to be able to adjust those levels in real time. Of course, the first step is its detection.

detection (and .asoundrc)

Links: mic detection primer :: asoundrc primer :: Python usage

---

If using ALSA, we need to know how the system detects and categorizes a mic within an OS. Then we'll put the information in ~/.asoundrc, so it can be used by any sound software. However, if you have the dreaded two sound card situation, which always occurs with HDMI, you've got to straighten that out first, and reboot. The HDMI situation works best when I unplug the mic before booting, and then plug it in after login. Typically, the mic is Device 2, after Device 0 and HDMI 1. Everything else with Audacity and alsamixer below is the same.

$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: SB [HDA ATI SB], device 0: ALC268 Analog [ALC268 Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0

...indicating we have plughw:1,0 for the input, double check and...

$ cat /proc/asound/cards
0 [SB ]: HDA-Intel - HDA ATI SB
HDA ATI SB at 0xf6400000 irq 16
1 [Device ]: USB-Audio - USB PnP Audio Device
USB PnP Audio Device at usb-0000:00:12.0-3, full speed

...appears good. If we'd also like to know the kernel detected module...

$ lsusb -t
[snip]
|__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 3: Dev 3, If 0, Class=Audio, Driver=snd-usb-audio, 12M
|__ Port 3: Dev 3, If 1, Class=Audio, Driver=snd-usb-audio, 12M
|__ Port 3: Dev 3, If 2, Class=Audio, Driver=snd-usb-audio, 12M
|__ Port 3: Dev 3, If 3, Class=Human Interface Device, Driver=usbhid, 12M
[snip]

... so we know it's using snd-usb-audio in the kernel on

Test the hardware config for 15 seconds

$ arecord -f cd -D plughw:1,0 -d 15 test.wav

real-time monitoring

When recording, we need to be able to see the waveform and adjust levels, otherwise it wastes a halfhour with arecord test WAV's and adjusting alsamixer, just to set levels. Audacity is the only application I currently know of that shows real-time waveform during recording. I open AUDACITY in the GUI, and then overlay ALSAMIXER in a terminal, from which I can set mic levels while recording a test WAV.

There are typically problems during playback. Sample rate may need to be set to 48000 depending on chipset. Audacity uses PortAudio, not PulseAudio or ALSA. I've found it's critical to select accurately from perhaps 12 playback options -- "default" almost never works. I go to this feature...

... often the best playback device will have some oddball name like "vdownmix", but it doesnt' matter, just use it.

TTS

tts - colab account

Create a notebook in Colab, then go to Cloud and authorize a text-to-speech API, and get a corresponding JSON credential. When running the TTS python code in Colab, the script will call to that API, using the JSON credential. Of course, it's the API which does the heavy work of translating the text into an MP3 (mono, 32Kbs, 22000Hz).

Voiceover in Collab (1:39) DScience4you, 2020. Download and watch at 40% speed. Excellent.
The Json API process (8:14) Anders Jensen, 2020.
Google TTS API (page) Google, continuously updated. Explains the features of API, for example the settings for language and pitch.

tts - local system

TTS workflow has four major steps. Some pieces are reuseable: 1) producing the text to be read, 2) writing the code translating the text to speech, 3) in the code, selecting translation settings for that project and, 4) editing output WAV or MP3's. All four must be tweaked to match formats, described in another post:

• script :: any text editor will do.
• code :: typically Python, as described here (11:25), and using gTTS.
  • # pacman -S python-pip python-pipreq pipenv to get pip. Pipenv is a VE sandbox, necessary so that pip doesn't break pacman's index or otherwise disturb the system. Also, pip doesn't require sudo. Eg: $ pip install --user gtts.
• name one's script anything besides "gtts.py" since that will spawn errors -- it's one of the libraries. It is the gtts library which interfaces with the Cloud Google TTS API.
• The "=" is used for variables. The output file is not a variable, so just put the parens near the method: output.save("output.mp3")

• TTS engine :: varies from eSpeak to the Google API gtts, from which we import gTTS.
• audio :: levels, pauses, file splittingm=, and so on. WAV is easiest format, eg with SoX, GoldWave, Jokosher, etc.

hardware

Some current and planned audio and video hardware

video

• GoPro Hero
• various old cell phones,if they charge up and have room for a SDC, i can use.

public domain films - archive.org
Video editing. The first problem are the inputs. Video clips from any of, say, 10 devices, has meant that the organization of video, audio, software, and hardware have seemed impossible to me. For years. Appropriating UML toward this chaos became the first relief, then other pieces eventually fell into place.

A second problem with TTS, is it relies on Python for something with as much speech as a script. And this means doing a separate, non-Arch, Python install and pip to get packages - or - sticking with Arch and avoiding pip in favor of yay. Blending pip updates with Arch means disaster the next time you attempt a simple pacman -Syu. The problem with yay/AUR is it may not have the modules needed to run TTS.

I do separate installs on anything that has its own updaters: Python, Latex, GIT. It's a slight pain in the ass making sure all paths are updated, but I can keep them current without harming my Arch install and updates.

UML

Universal Modeling Language ("UML") is for software, but we know its various diagrams work well for, eg. workflow, files, and concepts. Derek, per usual, has a concise conceptual overview (12:46, 2012).

We can create UML in UMLet or Dia, though we will eventually want to do them in LaTeX. Lists, esp with links and so on, are best created in HTML (Geany/GTK). It's good to keep backups of these overarching UML and HTML documents on Drive.

• Dia: # pacman -S dia/$ dia :: select the UML set of graphic icons, switch to letter-size from A4. Files are natively saved with DIA extensions, but can be exported to PNG.
• UMLet: $ yay -S umlet/$ umlet :: UMLet is the software in Derek's video above. Sadly, allows Oracle into one's system by requiring Java headers and some other files. UMLET extension is UXF (their XML schema), but can be exported in graphical formats.

audio

(UML on-file) Like video, audio arrives in any number of formats and must be standardized prior to remixing with video. Some say to sample 24 bit, some say 16 is just fine. Capturing old CD's, maybe 24/192kbv, but it's also fine to do 16/192kbv. For video or backup DVD, 16/192kb at 44100 seems to be a good output for remixing.

• GoldWave: (proprietary) :: easy wine bottle. Does well with WAV.
• Removed audio: audio source. audio separated from raw video for further processing
• TTS: audio source. creating a script and using some method to change to audio for the video.
• Mixxx: combine music, TTS, and anything else at proper levels.
• Qjackctl: # pacman -S qjackctl/$ qjackctl :: JACK manager to work alongside ALSA.

pulseaudio notes

Links:PulseAudio configuration :: PulseAudio examples

---

Go to /etc/pulse/ for the two main configuration files: default.pa, and client.conf. Any sink you add in ALSA will be detected in PulseAudio, eg. # modprobe snd_aloop will turn on at least one loopback device.

Multiple Audio Collection (11:29) Kris Occhipinti, 2017. How to do multiple audio sources.

Text-to-speech, multiple card

Overview: Single sound card setup isn't always easy, but this post has more than enough info for single card text-to-speech. As for multiple cards in ALSA, things can become complicated. My prior espeak post on multiple cards is beneficial if all the answers aren't here. Also a great multiple card site for the /usr/share/alsa/alsa.conf file. The deepest problem with two cards and HDMI is that, the cards can be re-ordered for default and less ALSA errors, but then HDMI errors may occur from conflicts of this re-ordering. What does the user want, HDMI errors, or ALSA errors? I prefer ALSA errors, given how difficult HDMI is to re-configure. I mostly take the easy way out and simply comment the lines inside /usr/share/alsa/alsa.conf.

---

I used this hack, and subbed espeak for festival, since I typically use espeak. Espeak works fine, but there are several potential problems once one gets to the point of xbindkeys. Not mentioned in these tutorials is that both ~/.xbindkeysrc AND ~/.xbindkeysrc.scm may be required for consistent operation.

1. Verify espeak, eg...

   $ espeak "this is a test"

   This site has loads of options for the line.
   
2. Install xsel. Xsel takes any selected text (stdout) and pipes it to any subsequent process we want, in this case, espeak.
3. Select some text and then:

   $ xsel | espeak -s 100
   

   ... and whatever other espeak modifiers you want.
4. Make a script from the two actions.

   $ nano talk.sh
   #!/bin/bash
   xsel | espeak -s 110
   eof

5. Chmod talk.sh to activate it.

   $ chmod +x talk.sh

6. Test it. Select/highlight some webpage or document text and then, in a terminal,

   $ ./talk.sh

   This speaks the selected text as many times as it's executed, that is, it's not a "one off" of the selected text.
   
7. Here is where it becomes difficult: binding the script to a hotkey, eg, with xbindkeys. We first need to detect the numbers associated with pressing keyboard keys.

   # pacman -S xbindkeys
   $ xbindkeys -k

   xbindkeys -k allows a person to detect the numeric codes needed inside the ~/.xbindkeysrc file. One difficulty was the super-key, aka "windows" key. Eventually, I gave-up on this key and opted for the CTRL key in combination with the letter "k". I was then able to get numbers which worked within the configuration ~/.xbindkeysrc file (see below).
   
8. Tie talk.sh to the key combo by creating an ~/.xbindkeysrc file.

   $ nano .xbindkeysrc
   "./talk.sh"
   m:0x4 + c:45

9. Now start xbindkeys

   $ xbindkeys

10. When I tried CTRL + k the selected text was spoken only once; the text had to be selected with the mouse each time I wanted it to speak. Somehow the text was being "lost" between usage.
11. I troubleshot using

    $ xbindkeys -n -v

    which runs it in the foreground and provides info. Here, I received a message that there was no ~/.xbindkeysrc.scm. Accordingly, I next ran xbindkeys_config, the GUI front-end (obtained via yaourt xbindkeys_config-gtk2). Through the GUI, I reran the configuration, which tweaked the ~/.xbindkeysrc file. After that, I had consistent repetitive behavior from the hotkey combination.
12. (Optional) I could have continued to call the script with the hotkey combination but, since the script was only one line, I didn't truly need a script in this case. I deleted the script. Within .xbindkeysrc, I replaced the script call with the script's contents, "xsel | espeak -s 100".
13. Once consistent, I added xbindkeys to my windows-manager startup file.

    $ nano .icewm/startup
    xbindkeys &

14. Exit and restart X, (or do, $ killall xbindkeys and $ xbindkeys) to be certain xbindkeys initializes using the ~/.xbindkeysrc configuration.
15. Testing any of the configurations above is always the same: select text with the mouse, b) hit your key combination (eg., CTRL + k), and verify it repetitively audibles the text.

other

If there are audible delays for a word or two and you get errors like these...

$ xsel | espeak -s 100
ALSA lib pcm_dsnoop.c:638:(snd_pcm_dsnoop_open) unable to open slave
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.HDA-Intel.pcm.front.7:CARD=1'
ALSA lib conf.c:4554:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5033:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2501:(snd_pcm_open_noupdate) Unknown PCM front
ALSA lib pcm.c:2501:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2501:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2501:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side

...then your speech datastream is probably moving along as ALSA attempts unsuccessfully to connect to JACK and so on. I have never been able to stop these JACK calls -- they seem embedded in the binary lib files, I've searched with grep throughout the ALSA file kludge. There's also speculation SPDIF sleep modes can do this. I think in my case it's a multiple card issue since I never have the delay on single soundcard issues. Secondly, HDMI validation with the display monitor also handshakes with the sound card and must have the same device numbers named in ALSA, as by UDEV. Think of the HDMI process as one which cascades from the ALSA setup, but has a dotted line to the kernel through the ELD verification. ALSA can rule it: scroll down to the ELD verification process in this prior post.

more about audible delays
Whatever delays from the speaker, the most important delays are to WAV files. We should be able to write...

$ espeak -s 100 "cut back" -w cutback.wav

...without delays. If we have ALSA speaker delays however, we typically have them writing to files also.

Add an extra word. I find the word "delay" takes the right amount of time to initialize SPDIF or perhaps multiple card JACK issues, so that...

$ espeak -s 100 "delay cut back" -w cutback.wav

...will create the file with only "cut back" spoken.
I renamed the unresponsibe ALSA libs for PulseAudio -- I don't ever use PulseAudio anyway...

# cd /usr/lib/alsa-lib/
# mv libasound_module_ctl_pulse.so zz_libasound_module_ctl_pulse.so
# mv libasoud_module_pcm_pulse.so zz_libasound_module_pcm_pulse.so

...this caused error of them not found, but better than not responding.

espeak - alsa and hdmi environment

Overview: Speech synthesis has become nearly trivial, but a typical ALSA setup relies on sound card definitions from the config files ~/.asoundrc or /usr/share/alsa/alsa.conf (I typically delete the former and modify the latter). A complex hardware situation will almost always require some modifications to the config file. Installing espeak into such an environment often requires yet further config file modifications. I often wish OSS were still developed. Note: espeak uses the PortAudio library, specifically relying upon libportaudio. Verify installation in Arch with $ pacman -Ss portaudio.

symptom

Attempted espeak statements provide pcm error messages and no audio. Environment: Two sound "cards" -- one Intel MB chip, one NVidia graphics card sound chip. NVidia card selected b/c HDMI cable. ALSA HDMI configuration working flawlessly with all input and output apps (except espeak). ALSA espeak errors:

$ espeak "hello world"
ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.HDA-Intel.pcm.front.7:CARD=1'
ALSA lib conf.c:4371:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4850:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM front
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.HDA-Intel.pcm.surround51.7:CARD=1'
ALSA lib conf.c:4371:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4850:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM surround21
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.HDA-Intel.pcm.surround71.7:CARD=1'
ALSA lib conf.c:4371:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4850:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM surround71
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.HDA-Intel.pcm.iec958.7:CARD=1,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.HDA-Intel.pcm.modem.7:CARD=1'
ALSA lib conf.c:4371:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4850:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2450:(snd_pcm_open_noupdate) Unknown PCM phoneline
ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
connect(2) call to /dev/shm/jack-1000/default/jack_0 failed (err=No such file or directory)
attempt to connect to server failed

At least two issues jump out: espeak sound is (inexplicably) attempted through the Intel card instead of the NVidia card, so that's probably why all the other subassemblies are also unreachable and spawning errors. "Dsnoop" may also be an issue.

To solve the first problem, we can easily change the values inside /usr/share/alsa/alsa.conf further than my previous post on ALSA HDMI agreement. That is, per this post, comment the line for pcm.front, and each additional line of fail as follows:

# nano /usr/share/alsa/alsa.conf
# pcm.front cards.pcm.front
pcm.front cards.pcm.default

This reroutes dynamic loading on a per command basis to the default card, eliminating the error. But this means lines and lines of corrections. Note that it appears all of these are due to HDA-Intel (instead of NVidia), being called as during dynamic loading. Let's see if we can just rewrite one line to fix the called card to NVidia, instead of doing line-by-line edits on downstream results from calling the HDA-Intel.

First let's try to specify the hardware device, as we can do in:

$ aplay -D plughw:1,7 /usr/share/sounds/alsa/Front_Center.wav

We know ALSA is thus working. So we can do a two-step method using ALSA to be sure espeak is working:

$ espeak "Hello world" --stdout |aplay -D plughw:1,7

If nothing comes out of this, or if there is no content inside a WAV produced with

$ espeak "Hello world" -w somefile.wav

... then it's time to strace.