slowing video and audio

If possible, concatenate the entire video project MP4, and then do a final pass for speed and color changes.

concatenation method

To complete by simple concatenation of clips, with hard cuts, a list of clips in a TXT file and the "concat" filter will do the trick without rendering.

$ ffmpeg -f concat -safe 0 -i mylist.txt -c copy output.mp4

Put the text file in the directory with the clips or do absolute paths to them. The file is simple; each video is one line and takes the syntax:

$ nano mylist.txt
# comment line(s)
file video1.mp4
file '/home/foo/some-other-video.mp4'

Obviously, transitions which fade or dissolve require rendering; either way, starting with a synced final video, with a consistent clock (tbr), makes everything after easier.

concat codec

If using ffmpeg, then mpeg2video is the fastest lib, but also creates the largest files. Bitrate is the number one file-size determiner and what libx264 can do at 3M, takes 5M in mpeg2video. Videos with a mostly static image, like a logo, may only require 165K video encoding. A 165K example w/125K audio, 6.5MB for about 5 minutes.

biggest problems

If I'm going to change the speed of the video, and I only need the audio to stay synced, without a change in pitch, then it's trivial. But if I want the audio to drop or rise in pitch and still stay synced, the video typically must be split apart and the audio processed in, eg. sox. That said...

• video artifacts: sometimes revealed. Interlace lines or tearing. ffmpeg yadif filter
• playback: will recoding still playback on the original device? For ffmpeg, mpeg2video is the most reliable.
• complexity and time investment: speed ramps, portions, complex filters -- all difficult with linux software
• audio pitch: easiest is to change speed without changing audio sound. Pitch is more complicated. sox changes pitch when slowing, apts and atempo do not. Which will you need?

verify the streams in containers

We need this step to sort which streams our nav packets, video, and audio, reside on. In Linux, ffmpeg is nearly unavoidable, so assuming its use. Solciting ffmpeg information results in two numbers, the file number input, and the stream number. For example, ffmpeg -i result "0:0", means first file, first stream, "0:1", means first file, second stream, and so forth. MP4 files can contain multiple streams. Examining an AVI will only result in a single audio and video stream, per file.

deinterlace and tear solutions

Deinterlacing is the horizontal lines problem, tearing is the chunks of stuff. There are extremely good, but extremely slow deinterlace filters, but for me, the low-CPU filter yadif is typically fine. For tearing, I increase the bitrate to at least 4M. Here's a way to do both. For simplicity, not showing audio here.

$ ffmpeg -i input.mp4 -f dvd -vf "setpts=1.8*PTS,yadif" -b 4M -an sloweroutput.mp4

Although multiple filters, I didn't need 'filter complex' because only one input.

single command slowing

The decision point here is about audio pitch -- do you want the pitch to change when changing the audio? If no pitch change, either asetpts or atempo work fine. I like atempo. If need to recode, the basic mpeg2video video codec is the fastest, lightest lib going.

• pitch change: So far, I can't do it. To get it right with a pitch change, I have to split-out audio and video, use sox on the audio, and recombine. I can never get ffmpeg filter, "atempo", to accmplish the pitch change. Eg. I slow the video to half speed using "setpts=2*PTS", then attempt to drop the audio pitch in half, "-af atempo=0.5". It processes without errors, and syncs but with zero pitch change.
• no pitch change: "asetpts". It will adapt the audio, sometimes with strange effects, to the new speed, but the pitch will still be whatever.The timebase and so on will still be perfect sync with audio.

pitch change with sox (audio only)

Sox default is to change pitch when slowing or accelerating. Speeding-up audio is easier to match with video, since slowing the video is a repeating number. Sox can do four decimal places of precision(!)

17% slowing, w/pitch change.

$ ffmpeg -i input.mp4 -vf "setpts=1.2*PTS,yadif" -b:v 5M -vcodec copy -an slowedoutput.mp4
$ sox input.wav slowedoutput.wav speed 0.8333
$ ffmpeg -i video.mp4 -i audio.wav -vcodec copy recombined.mp4

30% slowing and pitch change.

$ ffmpeg -i input.mp4 -vf "setpts=1.5*PTS,yadif" -b:v 4M -vcodec copy -an slowedoutput.mp4
$ sox input.wav slowedoutput.wav speed 0.667
$ ffmpeg -i video.mp4 -i audio.wav -vcodec copy recombined.mp4

An output made 20% faster, and pitch change, that syncs perfectly.

$ ffmpeg -i input.mp4 -vf "setpts=0.8*PTS" -b:v 4M -vcodec mpeg2video fasteroutput.mp4
$ sox input.wav fasteroutput.wav speed 1.25
$ ffmpeg -i video.mp4 -i audio.wav -vcodec copy recombined.mp4

bitrate C flag

Sox defaults to 128Kb, so need the "C" flag. Eg to get 320K and slow it to 92%...

sox foo.wav -C 320 foo90.mp3 speed 0.92

pulling from DVD

Be sure to install libdvdcss

single title, "VTS_01_2.VOB".

vobcopy -i /run/media/foo/SOMETITLE -O VTS_01_2.VOB -o /home/foo/SOMEFOLDER

entire disc:

problems

On occasion, if you backup a DVD, you don't need the nav stream anymore and you'll have extra work on filters if you leave it in.

$ ffmpeg -i input.mp4
Input #0, mpeg, from 'input.mp4':
 Duration: etc
 Stream #0:0[0x1e0]: Video: mpeg2video (Main), yuv420p(tv, etc) 
 Stream #0:1[0x80]: Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s
 Stream #0:2[0x1bf]: Data: dvd_nav_packet

Delete the audio and data streams:

ffmpeg -i input.mp4 -map 0:v -c copy output.mp4

local machine (localhost) LAPP, email

Both of these must be entered for the monitor to remain on during inactivity. Neither command by itself is enough to keep the monitor alive.

$ xset -dpms
$ xset s noblank

With these two entered, the screen will still be receiving a signal, but it's just for the backlight, not for any display conent. If we want the display content to remain during inactivity, we must do the two above AND add the following.

$ xset s off

Do we even need this project? Re the LAPP: "no, but helpful". Re the email: probably "yes, for understanding dashboard alerts". LAPP or any other monolithic CMS (XAMP, LAMP) that require learning PHP might be a waste if we can chain cloud services and so on (eg. read comments under this video).

Since LAPP elements are servers, they typically require user switching, individual configuration, or other permission issues. A separate post will deal with production configuration like that. I wrote this one aiming for a localhost light development environment (besides Docker). Additionally, I've attempted to view each LAPP element independently, in case we learn of an app that requires only one element, eg the PHP server, or just a relational database. I also subbed out the Apache "A" for another "L", lighthttp: LLPP. More commonly though, even browser based apps (eg Nextcloud - go to about 8:30) still use a CMS LAMP, LAPP, LNPP, etc. Electron, Go, Docker, won't be covered here.

LAPP

Linux (L)

For both LAPP and email, verifying /etc/hosts is properly configured for IPv4 and 6 for localhost smoothness. Otherwise we typically already login as some user, so we shouldn't have permission issues if everything else is well configured.

PostgreSQL (P)

# pacman -S postgreqsl
# pacman -Rsn postgresql

Lunatic designers really really really don't want a person running this on their own system under their own username. There's no security advantage to this, just poor design. This is almost more difficult than the design of the database itself.

After installing with pacman, add one's username to the postgres group in /etc/group, then initialize the database. Use...

$ initdb -D /home/foo/postgres/data

...instead of the default:

$ initdb -D /var/lib/postgres/data

Attempt startup.

$ postgres -D '/home/foo/postgres/data'

If starting-up gives the following error...

FATAL: could not create lock file "/run/postgresql/.s.PGSQL.5432.lock": No such file or directory

...run the obvious

# mkdir /run/postgresql
# chown foo:foo /run/postgresql

Just do the above, don't believe anything about editing service files or the postgresql.conf file in /data. None of it works. Don't even try to run it as a daemon or service. Just start it like this:

$ postgres -D '/home/foo/postgres/data'

Now that the server is up and running, you can open another terminal and run psql commands. If you run as user postgres (see below), you'll have admin authority.

$ psql postgres

If a lot of data has been written, a good exit phrase to make sure all is safely written:

$ psql -c CHECKPOINT && pg_ctl stop -m fast

PHP (P)

Link: CLI flags ::

# pacman -S php php-pgsql
# pacman -Rsn php php-pgsql

PHP server setup (8:04) Dani Crossing, 2015. Old, but not outdated.
php.ini basics (7:25) Program With Gio, 2021.

If we've got a dynamic website, we typically need PHP (server) and then some JavScript(browser/client) to help display it. The less JavaScript the better (slows browser). I consulted the Arch PHP Wiki .

standalone PHP - start/stop

Standalone PHP is easier to troubleshoot than when starting Apache simultaneously. Stopping is CTRL-C in the terminal where it was started. Startup with web-typical TCP port 80, however leads to a permission issue.

$ php -S localhost:80
[Fri Oct 23 11:20:04 2020] Failed to listen on localhost:80 (reason: Permission denied)

Port 80 works when used with an HTTP server (Apache, NGINX, etc), but not standalone PHP (not sure why). So, use any other port, eg port 8000, and it works.

$ php -S localhost:8000
[Fri Oct 23 11:20:04 2020] PHP 8.0.12 Development Server (http://127.0.0.1:8000) started

See this to determine when best to use index.html, or index.php. But there are at least 3 ways for PHP to locate the index.html or index.php file:

1. before starting the server, CD to the folder where the servable files are located
2. specify the servable directory in the startup command

   $ php -S 127.0.0.1:8000 -t /home/foo/HTML

   $ php -S localhost:8000 -t ~/HTML

3. edit /etc/php/php.ini to indicate file locations...

   # nano /etc/php/php.ini
   doc_root = "/home/foo/HTML"

   ... however this strategy can lead to mental illness and/or lost weekends: sometimes the ini file will not be parsed. Good luck.

First, take a breath. then verify which ini file is being used.

$ php -i |grep php\.ini
Configuration File (php.ini) Path => /etc/php
Loaded Configuration File => /etc/php/php.ini

If you have modified the correct ini file, hours and hours of finding the correct syntax for

standalone PHP - configuration and files

Links: PHP webserver documentation :: Arch PHP configuration subsection

PHP helpfully allows us to configure a document root, so I can keep all html files (including index.htm) inside my home directory. The open_basedir variable inside the configuration file (/etc/php/php.ini) is like a PATH command for PHP to find files. Also, when pacman installs PHP dependent programs like phpwebadmin or nextcloud it default links them to /etc/webapps, because this is a default place PHP tries to find them. Even though they are installed into /usr/share/webapps. So if I had a folder named "HTML" inside my home directory, I'd want to at least:

# nano /etc/php/php.ini
open_basedir = /srv/http/:/var/www/:/home/foo/HTML/:/tmp/

email local

Link: simple setup :: another setup :: notmuch MTA setup

$ mkdir .mail

We also need a mail server on the machine but which only sends to localhost and then only to one folder in /home/foo/.mail/ inside localhost. However, instead of setting up local email alerts, why not skip all of that (for now), and run a log analysis program like nagios?

We want to configure the lightest localhost system setup for email that we can read on a browser. Once we can reliably do alerts on, eg. systemd timers, or some consolidating log app (eg,Fluentd), or some Python/Bash script triggers, other things are possible. Our simplest model takes timer script outputs and sends email to the local system (out via SMTP port 25).

CLI suite

Linux has some default mail settings: mail to root is kept at /var/mail/root, user typically in ~/mail. It appears we'll need a webserver, no small email systems have this built-in. Appears Alot can be our MUA. It is available on the repositories, and can be configured with other light localhost services as described here.

cli configuration

browser MUA

Once the above configured, we can use a browser MUA. For Chromium, the Chrome store, has an extension for notmuch, called Open Email Client. configuration information is provided.

database options

Right tool for the right job. An easy case might be a password database, essentially a flat file database, Excel 2-D type of thing. A more difficult case is a video edit. You've got clips, which might expand in number, you've got recipes or methods for various transitions and combinations, and you've got an overall project view which pulls from these. Whatever we have we'd like it to be ACID compliant. Try to use umlet or some such to do the UML.

7 Database Paradigms (9:52) Fireship, 2020. Seven different ways of looking at data, with suggested applications.
issues univ of utah (19:49) Mark Durham, 2018. Has a great chart.

relational algebra, calculus

Relational Algebra is just a form of logical operators (most are shown below), easy to understand for those who've taken logic courses.

Linguistically, Relational Algebra imparted its first name, "relational", to databases constructed around its principles, ergo "relational databases". In other words, relational databases are databases built around Relational Algebra.

Relational Algebra itself was created by Edgar Codd. His main book on this is The Relational Model for Database Management, and his Turing Award paper on relational algebra (1981) is available online (PDF). The most important skill level to reach is the capacity for fast database normalizing. This takes practice, similar to the logic skill for taking statements, text or otherwise, and symbolizing.

Relational Algebra Lecture 1 (50:21) Database Psu, 2015. Penn State. The first in their amazing free online course. Lecture begins about 19:12.
Relational Algebra Pt 3 lecture (51:18) Database Psu, 2015. The various operators and their uses.

Relational calculus is a way of making queries using relational algebra to get what we want from our queries. It's less atomic than relational algebra.

relational keys (primary, foreign, composite, blank)

Link: postgres constraints ::

Relationals are a bit of a pain. See my later post on how to configure PostgreSQL for local access. Restrictions make searches easier. The main idea is a unique column (Cust ID), can then access a unique row in the next table. In the pic below, the primary is green, the foreign red. A column with a "references constraint" on it is a foreign key.

Primary and foreign keys (2:08) Udacity, 2015. Primary and foreign keys explained. Foreign key is column with references constraint. No lookup table is used, but it might be possible by indexing somehow.
Primary and foreign keys (5:24) Prescott Computer Guy, 2011. This has an intermediate lookup table. He wants every table to have an ID column, to be sure to have a unique value for that table.
Postgres version (5:42) Trent Jones, 2020. Unfortunately uses PgAdmin not DBeaver.
SQL from teaching perspective (10:12) Mike Dane, 2017. Exam configuration

• primary key: Postgres/SQL constraint equivalent - NOTNULL, UNIQUE
• foreign key: Postgres/SQL constraint equivalent - none. Verifies that a column from keyed table always has a value in another table (is linked).

constraints
PostgreSQL	MySQL
PRIMARY KEY	PRIMARY KEY
FOREIGN KEY	FOREIGN KEY
UNIQUE (column)	UNIQUE
NOT NULL/NULL (null allowed)	NOT NULL
CHECK	CHECK
DEFAULT	DEFAULT
CREATE INDEX	CREATE INDEX
EXCLUDE

start-up/shutdown (postgresql)

See my other post for how to configure as user. That done, it's a pretty simple....

$ postgres -D '/home/foo/postgres/data'

With the server is up and running, you can open another terminal and run psql commands. If you run as user postgres (see below), you'll have admin authority.

$ psql postgres

If a lot of data has been written, a good exit phrase to make sure all is safely written:

$ psql -c CHECKPOINT && pg_ctl stop -m fast

dashboard options

We'll first want to run a dashboard on our local system, using a light webserver and PHP server, before attempting it over a LAN or WAN. Step one is to inventory all our databases (including browser's IndexDB), logs, mail, and daemons (units/services) on our systems prior to attempting the dashboard. That's because a dashboard will add to what's already present. We need a system baseline. Because for example, the simplest MTA's add a database of some sort. SMTP4dev adds an SQLite.db in its directory, same with notmuch (or maybe an XML datatabase). If we go so far as Postfix, it requires a full relational database. So we need a pre-dashboard inventory of databases, logs, and mail, written on a static HTML page.

why dashboard

We may want to monitor certain applications or system parameters, financial counts, student attendance, anything. If we start with our own system as a model there are 4 things which regularly change: logs, timers/cronjobs, system values (temp, ram usage, hdd usage, net throughput, etc), and application information. We might even want to craft a database to store some values. Postgres is best, but we can get general theory from MySQL models.

MySQL table basics (14:26) Engineer Man, 2020. Great overview for the time invested.

Prior to a real-time dashboard, a slower process of daily emails with a summary are a good start, even if we just mail them to localhost disconnected from Web.

Once we can update a daily localhost email, we can attempt to expand that to internet email. Or/and, we can add a light local webserver + PHP. We need this to have dynamic webpages; opening an HTML webpage from a file directory is "load once", not updatable.

configuration models

Security camera configurations were created around updating pages, and these configurations might be adaptable to system monitoring. More than security models, the DevOps/BI models seem glean-able. These servers might be, eg Fluentd, Prometheus, and Grafana. Production server software, but localhost possibilities. Most are in the AUR. Prometheus is sort of a DAQ for software -- with email alerts possible -- and then Grafana is typically the display for it. But neither require the other. Grafana can display from any data source, or take its info from Prometheus. For Postgres built DB's, TimescaleDB has a lot of videos that might apply. We might even be able to modify a Moodle setup, now that we can upload quizzes using the Aiken model.

services

We can attempt several daemons on a local machine, and see which ones are too resource intensive. Also timer scripts to execute those services only as needed, stopping after.

local system dashboard

logs and timers

Nearly all logs are inside /var/log, but we need to evaluate our system carefully at least once for all relevant log locations. Some logs are ASCII, others are binaries that require an application or command to obtain their info. Once tallied, systemd timers and scripts are the simplest, with possible output via postfix. If we then add a webserver and PHP, we could run a systemd timer script every hour on logs and which updates a localhost webpage. To see running timers...

# systemctl list-timers

Fluentd is a log aggregator available in the AUR, but may also need a DB to write to.

Timer files (18:16) Penguin Propaganda, 2021. This guy makes one to run a script. However, what about running a program and then shutting down the service after? 10:00 restarting systemctl daemons and timers.
Systemd start files. (7:52) Engineer Man, 2020. How to make unit files, which we would need anyway for a timer file.
Dealing with system logs (10:00) Linux Learning, 2018. Mostly about /var/log. Explains some are ascii others binary.

email inside our system (mua + mta)

Link: localhost email setup - scroll down to email portion.

For a graphical MUA, I used (in Chromium) an extension called Open Email Client. Some configuration information is provided by the developer.

various

ISP Monitoring (8:04) Jeff Geerling, 2021. Jeff (St. Louis) describes typical frustrations and getting information on power and internet usage. Shelly plugs ($15 ebay) are one answer. However there are excursions into several dashboard options.
How Prometheus Works (21:30) Techworld with Nana, 2020. Why and how Prometheus is used.
Grafana Seminar (1:02:50) TimescaleDB, 2020. Avthar Sewrathan (S.Africa) Demonstation and some of his use cases
Grafana Seminar for DevOps (1:01:59) Edureka!, 2021. Grafana half of a Edureka seminar on DevOps with Prometheus and Grafana. Thorough description including how to create a systemd service file to run it locally.
Prometheus and Grafana (54:06) DevOpsLearnEasy, 2020. Adam provides description of a server deployment of Prometheus and Grafana. This guy even shows the setup of a VM on AWS. He seems confused but we learn, same as strike football penetrates the veneer.
Grafana Seminar (1:02:50) TimescaleDB, 2020. Avthar Sewrathan (S.Africa) Demonstation and some of his use cases
Grafana Seminar for DevOps (1:01:59) Edureka!, 2021. Grafana half of a Edureka seminar on DevOps with Prometheus and Grafana. Thorough description including how to create a systemd service file to run it locally.