statistics -- some foundation

Statistics overlaps into Math of course. We can record samples into a CSV or spreadsheet and do histograms, and (Algebra 1) regressions. Yet CSV data are arrays. Arrays are matrices, and thus vector, evaluable (Calc 3), as well as Python and R friendly. IMO, because it spans both simple and complex Math, and CS, Stats are a fun side project during one's Math progression, at any time.

Basics are Algebraic if one uses tables for area under curves. If not, a person could do them using simple Calc 1 calculations. TI-84's -- if available -- are also great for checking that kind of calculation (eliminating tables and Calc) and for offloading the drudgery of long lists.

HS AP Stats is essentially an Algebra-only college Stats1 class, and slightly more user friendly. A review book for AP Stats, the Princeton Review, Table of Contents is a list of roughly 35 subjects, split over 4 categories. I've used it below with some modifications to describe what a person might want. Beyond basic or AP, I've added a "next level" section at the bottom, since matrices connect Math to Statistics and arrays/data structures connect Stats to Data Science.

Introductory data data collection tabular methods graphical methods - qualitative graphical methods -quantitative numerical methods - continuous boxplots add/mult a constant comparing mult. groups bivariate data: covar,regression categorical frequency Sampling and experiments plan a study data collection plan a survey bias in surveys plan an experiment Anticipating outcomes probability

random variables probability dist. - discrete random var's probability dist. - continuous random var's normal distribution combining independent random var's sampling distributions Statistical inference confirming models parameters point estimation interval estimation confidence interval inference: significance tests hypothesis: testing and accepting estimation and inference: population proportion estimation and inference: population mean estimation and inference: 2 population proportions estimation and inference: 2 population means inference: categorical data

For a comprehensive course, there's Khan Academy. On YouTube, I prefer Brandon Foltz, M.Ed video compilation, esp at 0.5x speed. They were made 2010-ish, but ahead of their YouTube time¹. Below there are some from Foltz and from many great teachers or vids which include nuggets in some way. As one goes along in review, it's a chance to relearn the annoying Greek symbols for parameters. The same Greeks as in finance, but different usage.

¹Another brilliant info-sharing hero with early YT chops was Derek Banas. One of his best might be his investing video, which crosses data science with financials and Python. And of course Barry Brown for programming any of it in C.

---

distributions of data obtained

degrees of freedom, clt

To Z or t (38:16) Brandon Foltz, 2012. The notion behind this choice, without calculation. Almost all stats are samples, not entire population, ie, "the prevalence of depression of those over 65" - how to give all questionnaire? Can't. The smaller the number, the more chance of error. Avg temp in NYC, what if we just used a day, or 5 days? The larger the sample the more confidence we have captured reality.16:00, 33:00 decision. 19:00,30:00 degrees of freedom
Buying land (32:06) Brantley Blended, 2018. listing, PLAT recency, survey recency (due diligence period), boundaries,

bivariate data: linear regressions, covar

covariance (5:55) Ben Lambert, 2013. intuition behind covariance. Positive, negative, and none. Some formulas, but conceptual.
Variance vs. covariance (Webpage) Investopedia.

combinatorics, permutations, probability

nCr and nPr. These are the denominators of probability, since they give the universe by which we determine our odds. If "statistics" are the collected data, nested inside we have probability (based on prior events, statistics), and inside this is the "combinatorics and permuatations" which create the denominators of probability fractions.

Permutations and Combinations (17:40) Organic Chemistry Tutor, 2017. Simple review of when to use nCr, nPr, or just the factorial by itself. 14:00 How many ways can we arrange the letters in the word "Alabama".
problem - probablility (replace/don't replace) (10:11) Amy Krusemark, 2020. a slight political note at start, AP question. example of non-replacement probability, example of understanding permutation/combinatorics effecting denominator. 2:40 without other notice, 0.05 alpha is threshold for a reason to doubt.

---

next level

KL Divergence

KL Divergence (18:13) ritvikmath, 2023. Non-negative comparison of two distributions.

vectors and matrices in statistics

As my friend Bart texted...

Matrices are really just notation for a list [of] equations. Not so profound, not for a long time. Like if I have ten data points then i have that equation ten times. Ten y values. Ten x1 values, ten x2 values, etc. Thirty diff x values, so x could be written as a 10x3 grid. Ok enough [of] that
Beta is 1x3 and and x is 3x10. Then when multiplied out gives a 1x10 vector. To say that vector equals the 1x10 y vector is to say each component of y equals the corresponding for RHS. ie ten equations

And of course wherever we have a matrix, we have a potential vector.

Comp Sci v. Data Sci Matrix (10:34) ritvikmath, 2019. Reveals some clear differences between computer matrix use and math/datasci use and how they overlap (eg. CompSci efficiency can work on any math app).

data structure

necessary for data science. Data Science will evaluate these further, sometimes using Calculus, but we at least need to know what they are, IMO. Not on the AP test. The terminology transform is Statistics and Math use the term "matrix", but Data science/Computer science use the term "array". Depending on what we need to do with the matrix, computing will perform some function on a data array. Usage: imagine an R2 scatter plot, but where we have a third dimension with error information attached to each data point.

Linear Algebra: Transformational Matrices Part I (15:43) Computer Science, 2021. transformations on R2 matrices, using geometric examples. There's an entire playlist that's valuable.
Linear Algebra: Transformational Matrices Part II (9:23) Computer Science , 2021. transformations on R3 matrices, as we might do with data structures.
Trig Functions (9:15) PatrickJMT, 2011. this hero scores yet again. Just in case you need them again for the stuff above. So rotten.
1,2, and 3d structures (8:32) GridoWit, 2017. basic terminology and location tracking within different types of arrays and data structures. The inuition that arrays solve a storage problem: we don't want to have a new variable for each piece of data we have collected. C syntax is also provided.

aur - git encryption (arch)

So far, it appears affected: standardnotes-desktop.

I don't know if this will happen again, but could be helpful. Apparently, git recently changed to an authenticated SSL (encrypted) connection. AUR actions which attempt to use an unencrypted git download fail with the following.

The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
==> ERROR: Failure while downloading desktop git repo

One of the reported fixes is to simply change http to https, however this is not possible within AUR actions performed by yay -- yay will make the call to git, without any option for the user to change the URL.

solution

As time goes on, AUR package builders will write calls to "https" instead of "http". For the time being, it's a difficult problem.

colord - freedesktop.org severe device conflict (printer)

solution

NB: This takes the TN-630 toner cartridge.The TN-660 will work. If the entire drum is replaced, it's the DR-630.

Before describing the problem, here's a process for installing this (04f9:0092) Brother printer. This method is faster than the 3 days I spent understanding how colord had undermined the old reliable install method from a previous post. It's still an hour or two but compare that with salvaging configurations from older setups and losing days.

1. physically connect the printer and lsusb to verify detection.
2. pacman to install colord and cups. At the Brother site, download the RPM and xarchive to extract the PPD (and filter, if you want to go old skool).
3. if it won't archive the RPM, then be sure to install its software # pacman -S cpio.
4. new: $ yay -S brother-hll2315dw. This saves having to harvest, install and do permissions in several directories on executable scripts and data files. This is a 2 day time-saving driver.
5. can rename (or not rename) the PPD to anything "BrotherA.ppd" for example. Open it up and change the default size to 'letter' from 'A4' and/or any other such changes. Save, and put it into /usr/share/model/. Next, do either 6 or 7.
6. (CLI, more fun, less reliable)

   # lpadmin -p BrotherA -E -v [insert the USB URI from lpinfo -v] -m BrotherA.ppd

   The USB URI gained from lpinfo -v may be lengthy and will include a serial number.
7. (GUI, more reliable) Go into CUPS, http://localhost:631/, and "add printer"
   
8. inside the admin page, select "have PPD" and go to it in /usr/share/model to install the PPD from step 4 above.
9. still in the admin page, "maintenance" has the menu to set the device as the default printer.
10. # systemctl restart cups.service

---

the reason this conflict is a 3 day kludge

The solution is above, but here is the days-long struggle. We used to just install the printer with the CLI in about 10 minutes.

freedesktop: systemd and now colord

Freedesktop.org developed systemd of course. It's fairly comprehensive and complex, compared with the old Init.d flexibility of simple config files. Recently freedesktop began taking control of session color options too, and by device. Questionable. But in a horrible design decision, their colord daemon was given control not just of colors, but the DBus, apparently to make device queries/directives for color options. In practice, this means simple software conflicts about colors can lead to Dbus failures. Enter CUPS, attempting to set printer color profiles. Disaster.

CUPS circularity

It's a vicious circle. CUPS needs to write a PERL printer profile which contains (among many other settings) color settings. Meanwhile colord retains absolute control of color by controlling the DBus connection. When CUPS detects that colord has already created a device profile, it aborts its entire profile creation attempt, the one with all the settings for the printer. The result of no CUPS profile/filter is a half-installed printer. The printer doesn't print, and the error message "filter error" appears in the CUPS dashboard.

These errors can also be found in the error_log. If we attempt to install a printer, when colord is operating, we'll see the following:

$ cat /var/log/cups/error_log
W [16/Dec/2022:01:50:02 -0800] CreateProfile failed: org.freedesktop.ColorManager.AlreadyExists:profile id \'BrotherA-Gray..\' already exists

...and we can observe the details of the conflicting profile which colord had already installed...

$ colormgr get-devices
Object Path: /org/freedesktop/ColorManager/devices/cups_BrotherA
Owner: root
...
Colorspace: gray
Device ID: cups-BrotherA

colord's phony paths

It's hard to work with colord because it creates phony paths. The path above, /org/freedesktop/ColorManager/[etc], does not exist. No file is there. This is a "path" only in the sense of it being a line in an XML file or an ICC file. Colord knows how to evalute the line, but there's no file.

As for the XML files which contain the phony paths, these are apprently...

/usr/share/dbus-1/interfaces

... and two homes for the ICC files defining colors, for colord and ghostscript...

/usr/share/color/icc/colord
/usr/share/ghostscript/iccprofiles

order of operations circularity

When we install the printer, colord moves more quickly than CUPS. Colord detects the printer on the dbus and creates a device color profile faster than CUPS can write a CUPS filter. So, CUPS detects an already-made ICC colord color file, logs this color conflict as an error, and exits its entire filter creation. Accordingly we cannot fix the necessary filter later from some other CUPS process: it was never created and thus cannot be modified.

It seems we must either 1) remove colord, 2) configure colord not to create a profile for the printer, 3) modify CUPS to not to create its own color profile or to accept colord's profile, 4) modify the PPD (if possible) not to seek to create a color profile, or possibly to accept colord's profile, 5) place a filter in opt from a duplicate working setup. It might also require some combination. This is a kludge.

As a short preliminary to the 4th option, the simple act of commenting out a PPD's color settings doesn't stop the conflict.

1. remove colord

So let's pull colord, install the printer, then reinstall colord after CUPS has written a filter. It seems that we can do this: CUPS claims to only use colord ICC profiles "optionally".

# pacman -Rsn colord
checking dependencies...
:: cups optionally requires colord: for ICC color profile support

Packages (2) libgusb-0.4.2-1 colord-1.4.6-1

Total Removed Size: 8.49 MiB

:: Do you want to remove these packages? [Y/n]

...meaning CUPS should be able to install the printer without colord installed. Let's delete the error log, reinstall the printer and check for new errors...

# rm /var/log/cups/error_log
# lpadmin -p BrotherA -E -v usb:/dev/bus/usb/lp0 -m brother_HLL2315.ppd
lpadmin: Printer drivers are deprecated and will stop working in a future version of CUPS.
$ cat /var/log/cups/error_log
cat: /var/log/cups/error_log: No such file or directory

No errors -- seems we're home free. Nope

colord/dbus integration overreach

The printer installed without errors, but attempts to print failed.

FindDeviceById failed: org.freedesktop.DBus.Error.ServiceUnknown:The name org.freedesktop.ColorManager was not provided by any .service files

Yes, CUPS apparently relied on color management from colord as "optional", however reliance on DBus is in no way optional for any application. And, since colord manages both colors and DBus connections to devices, if we take out color management, our printer loses DBus access. The printer cannot then communicate, eg print. A person with a similar problem seems to confirm this. Colord must apparently remain.

2. configure colord

I could find no colord configuration file. Using colormgr, we can delete a colord device profile. But... that's not the problem. The problem is when the color profile (ICC) is created -- which is conflicting with the printer installation. Again, the only time CUPS creates a print filter in /opt/ is its initial read of the PPD file.

3. configure CUPS

The /etc/cups folder has several configuration files.

• /etc/cups/printers.conf: don't edit when CUPS is running.
• /etc/cupsd.conf: I know this file is read during systemctl restart cups.service. I've changed log levels from Warn to Debug before and seen its effects inside /var/log/cups/error_log. It also has permissions. It does not look effective for device install conflicts.

4. modify the PPD

I could find no way to modify the PPD that solved the conflict and created a device. It *is* worthwile to go into the PPD and switch the defaults to "Letter" from "A4" if a person is in the US. Saves having to worry about updates to the device file later on which may disrupt other settings.

5. duplicate a filter

During install, the PPD wants to write a PERL filter to the directory specified in the wrapper shebang. This is the filter that is not written and which causes the print failure.

$ ls /opt/brother/Printers/HLL2315DW/cupswrapper
total 76
drwxr-xr-x 2 0 0 4096 May 24 2020 .
drwxr-xr-x 5 0 0 4096 May 24 2020 ..
-rw-r--r-- 1 0 0 18351 Apr 18 2020 Copying
-rw-r--r-- 1 0 0 15010 Apr 18 2020 brother-HLL2315DW-cups-en.ppd
-rwxr-xr-x 1 0 0 24436 Apr 18 2020 brother_lpdwrapper_HLL2315DW
-rwxr--r-- 1 0 0 7650 Apr 18 2020 paperconfigml1

We can see that the PPD copied itself over and that an executable (755) PERL script (brother_lpdwrapper_HLL2315DW) was written. "Copying" is a license but should also be copied. I put the entire HLL2315DW directory - as is - on a USB key and copied it to the non-printing system. I saved to my installation directory.

HL-L2315DW old-skool steps

Somewhere between my old post and the newest way at the top.

• install colord, cups, do all the download and xarchive of the PPD and filter from the Brother site, described in this post, repeated here in brief
• can rename (or not rename) the PPD to anything "BrotherA.ppd" for example, but the filter (must) keep its full given name. Its only content will be the shebang and the /opt location.
• Attempt to install the printer normally.

  # lpadmin -p BrotherA -E -v usb:/dev/bus/usb/lp0 -m brother2315.ppd

  But now the problem is the URI. This has become extremely finnicky with colord running the dbus. So instead of usb:/dev/[etc] above, use lpinfo -v to get a better USB descriptor. Copy and paste it into the command.
• install and unpack the entire HLL2315DW directory described above and set 755 permissions on appropriate files in the subdirectories lpd, inf, and cupswraper, as well as chown them to "0" (root). This is time consuming so for Arch users, the AUR package will handle this step

ldd - libraries for deficient applications

As noted , there are times when an app is missing a necessary library, which is a PITA.

The process is to LDD the for the missing lib, and then pacman -F to find which package has the lib. In this case, the poster used.

$ ldd /usr/lib/cups/filter/rastertortlabel

...

libcrypt.so.1 => not found

...
$ pacman -F libcrypt.so.1 # to find which package to install

stack developer

Basically want to be able to deal with front and back end

Let's tear into this supposed meme below...

Imagine numbers in front of each step and that's how I've ordered this page. At the bottom of the page is sandbox information, definitely needed to practice each item. Incidentally, it might be worth it to some to also learn COBOL, so I've added that beneath sandbox info, at the very end.

Languages (16:41) Fireship, 2022. Must watch at .5 speed. Learn Python, JS, and PHP, already know Bash, HTML and CSS. SQL seriously importante.
One person's journey (webpage) the path one person took.
Programming typification (18:54) Continuous Delivery , 2021. History of languages in a thorough format. Object vs function, vs polymorphism. "messaging". Structured programming prevents unstructured Goto's.
Language rankings (15:16) Clement Mihaelescu, 2022. Front-end dev self-legend (Romanian?) type PHP hater walks us through a ranking. Comments.

7. MongoDB and MySQL (20 days)

First of all, Barry Brown. That said, databases are more like 20 years than 20 days. Just learning normalization methods is a PITA and a relief (ohhhh) at the same time, for example.

Normalization (5:41) Crack Concepts, 2019. Thick accent: 0.5x speed. Simple explanation describing why to delete redundancy and use links (keys). CC has an entire series on normalization.
One person's journey (webpage) the path one person took.

8. GitHub and Git (18 days)

This might be more like a week, if it's a 7 day 56 hour week. 18 days would certainly allow more file check-outs and lazier pace of absorption, but let's at least cede back 9 days of time to other stages, if needed.

Git tutorial (25:14) Derek Banas, 2014. Typical Banas everything but the kitchen sink. 8:00 using a directory to be monitored. 14:00 what's changed. 18:00 add changes, delete files from monitoring.
One person's journey (webpage) the path one person took.

$ git config --global user.name "Foo Fooslap"
$ git config --global user.email katyua@free.com
$ git config --global core.editor "nano"
$ git config --list
$ git help
$ git diff
$ git commit -a

9. REST API or AJAX (11 days)

Here's where the real joke exists in this meme. This is 365 days, not 11.

SANDBOX

serving pages (localhost)

Typically will need to put in Apache, but rudimentary server is in PHP if just checking basic HTML and CSS setup

# pacman -Ss
# systemctl reload apache2

Localhost SSL (14:13) Ambar Hasbiyatmoko, 2017. Necessary to have a localhost Apache running, but how to do HTTPS? Of course, will need an SSL certicate. Has a few ommisions as to how to properly configure localhost so chromium won't complain but otherwise...
PHP without Apache (4:39) Jonah Lawrence, 2020. Uses a Bitnami cookie cutter. No SSL run through : can't do that with PHP's simplistic built in server but avoids Apache.

COBOL .COB and C

Don't really have a way to get it written to a page.

COBOL (2:13:58) Derek Banas, 2020.Comments indicate this is a sensible tutorial.

log management

The first step is to locate and comprehend all log activities, second to determine triggers for undesirable events and conditions, third to text if this/these happen, and fourth to send email summaries of log changes.

I doubt I'll ever complete this post, as there's so many ways to skin this cat, both in CLI and GUI. Overall it's part of SIEM and should be accomplished with some thought.

directories

At the simplest level, this is the local directory:

/var/log

And of course to see how much use:

$ du -sh /var/log

And of course to limit the largest offender journalctl, in the first place:

# nano /etc/systemd/journald.conf
SystemMaxUse=200K

log programs

1. Linux log apps (webpage), 2022. Several log apps show information.

timestamped video: androids 12, 7 (nougat)

Date and time stamping on video is the number 1 reason we shoot video. Why can't we do this on our phone?

Ninety-five percent of the PlayStore video time stampers add text to the video after the video is complete, in a second pass. Altering the file after the fact still provides an accurate time on the video picture, but obviously provides an inaccurate date/timestamp on the file's metadata. Further, since the file timestamp varies with what is seen on the video, it's diffiicult to call it reliable evidence.

I've only found a few solutions

1. Android 12

Timestamp Camera Pro $4.99 (2022). Brian Di, the creator, made 6 versions of this program, which is confusing. The versions range from Free to $9.99 for the "Pro Enterprise" edition. Most use retroactive stamping, but this version does real-time.

2. Android 7 (nougat)

Be sure to turn on "System UI Tuner": hold down the gear, not on the dimmer, but more to upper R, until UI Tuner is activated.

The only one I found was Timestamp Camera Pro $3, which retroactively adds the date and time. In Android 7, I could not find a real time

3. other

screen recorder: I found the Xrecorder. Free to install but I upgraded from within the app -- that may have been $5.

stop motion: plenty of tutorials for Stop Motion Studio Pro $4.99 (2022). Eg, this sorta corny Kendra Fleischman video shows the process. She uses a closet shoe rack for her camera stand. Don't forget that half of the work will be "post" on a desk/laptop after taking the file off the phone via USB.

phone provider oddities

Phone company interaction has become complex, at least at T-Mobile. However, outside of the idiosyncracies below, 5G reception is better than 4G (eg, inside buildings) and 5G hotspotting carries good speed, eg 6-7Mb/sec during software updates (pacman). There are some idiosyncracies...

1. unlock - hard 40 days

It's necessary to use an unlocked phone if traveling overseas and wanting to purchase a SIM for local service. IME, especially if planning for a trip, buy any new devices 2 months ahead of trip. New devices have a hard 40 day hold (postpaid plans - see below) before T-Mobile will unlock the phone.

Secondly, probably because I purchased my device on EBay, there was no T-Mobile unlock application any longer in the device apps. This means a call was necessary to T-Mobile to accomplish unlocking. Have the IMEI and an email address handy. They will ask "Why? International? Leaving T-Mobile?"; it's almost as bad as KYC in banking. The agent notes unlocking takes 72 hours and that an email will be generated once complete. So 40+3 = 43 days from purchase, best case.

2. misc

• call/chat - chat is preferable. Calls are recorded, but there's nothing in writing and no way to attach screenshots. Talking also means KYC questions "do you plan to travel?"
• prepaid, postpaid plan - postpaid plans require credit checks and report to credit agencies, so this is higher level account. Prepaid plans are lower end and have overage charges. For best service, get all account numbers postpaid, since pre/post can vary by the number on the same account.
• unlock phones - postpaid account: 40 days on the network. Prepaid accounts: 365 days on the network. Website declaring a device is "eligible" for unlock and/or being purchased independently is irrelevant.
• temporary and permanent phone unlock: which is which and why?
• gigabyte accounting: extrememly liberal. I'd wager T-Mobile tallies Gb usage at 1.5:1 or greater. Eg, 1Gb actual usage will be declared as 1.5Gb. Also there's delayed accounting -- account might show 3Gb immediately after use and gradually rise to 4.5 over the next 12 hours of non-use.
• throttling/overage. device dependent. On my dedicated hotspot, I get throttled back to 126 K. On my phone account, I get throttled to 56K. Does the account automatically add gigabytes of data for some charge.
• sim swap - there's typically some network confusion or throttling with a recently swapped SIM or a new SIM. Usually apps will require a new login for an "unrecognized device" when the SIM changes. Google apps can work without the SIM if Wi-Fi is available.

3. usage

Day to day use -- emails and so on, blogging, browsing, is about 1Gb per day, maybe 2 if several videos (typically 360p in phone). Some problems:

• Surfline (app): no way to adjust/lower quality. Immense AJAX and ads. Seems to use 500Mb each time opened, up to 1GB if on there 4 mins.
• NFL GamePass (hotspot): no way to adjust/lower quality . Apparently, server takes over and streams at 1080 or greater. This means a game is is 15Gb, or about 3.5 per quarter of a full replay.
• Criterion Channel (hotspot): about 3-5Gb per film, 520-720p, adjustable.

4. phone app

Say a person has unlimited 5G phone and 40G hotspot. When they look on the T-Mobile app, they see screenshot below. Turns-out 17.6Gb at the top would be the combined number of both phone data and hotspotting usage.

hotspot the phone :: 4G->5G transition

"...I would forgo a separate hotspot device unless I needed if for group internet access. Buy more data for the phone and USB-tether."

NB: for usb-tethering a camera, either in a phone or as an independent cam, go here.

---

5G rolled out in July for T-Mobile. At that time, my phone's data plan began to include a free 40G of 5G "hotspot" data. This additional feature allows a person the option to use their phone as a WiFi hub. They can still access the Web through their phones' browser (the old way), but they can also broadcast the signal like a hub. A software toggle in the phone's settings (see below) turns it on or off.

Transmitting WiFi creates a hot phone, and heat is the number 1 electronics killer. When I saw how hot the phone was getting, I purchased another device (plus another number), that's solely for hotspotting. I have too much contact info and so on in my phone for me to deal with it failing due to hotspotting heat.

Of course, as soon as I purchased the dedicated hotspot device, I found-out that it wasn't really necessary. I learned I could use the phone without WiFi heat problems... if I physically tethered it using a USB cable (pic below) instead of running WiFi. In addition to less heat (a fifth of the heat), the cable charged the phone during internet use. The connection also is less hackable than WiFi.

The drawback is that a physical connection only can be attached to one device. But could a person allow multiple connections by tethering to a switch, or a router? Haven't yet attempted.

data expense on separate hotspot

I had already purchased a hotpsot device -- basically a data connection device with no calling, text or apps on it -- so I went on to purchase a $10 a month data plan for it and configured it (see further down). Data is quintuple the price of phone plans. It's possible providers don't receive enough intel from that device to sell/provide to advertisers/LEA's, or that it's considered generally a business device, or that they don't receive subsidies from Google, etc? Whatever the cause, $10 per month bought me 2GB and no rollover.

---

configuration

First though, three questions answered that I wish I could have easily found on Google.

1. Is a separate SIM needed for the dedicated mobile device? Yes. Although a person could physically move their phone SIM into the dedicated hotspot, both the phone and the dedicated hotspot device have unique IMEI's. So the phone provider will have to register and unregister the device each move of the SIM. This is liable to lead to interruptions, in addition to the liklihood of damaging the SIM during repeated transfers. However, a separate SIM *does* mean a separate phone number with its own data plan. More below.

2. With 2 SIMS, do hotspot minutes from one device merge with the other device? No. Even though users receive one bill, it has two numbers with two separate data plans. Overage charges would apply if I went over on one device but still had gigabytes left on the other device. Further, dedicated hotspot device minutes are expensive: $10 per month on the device got me 5Gigs of 5G speed data and unlimited 126K data, LOL.

3. Which dedicated hotspot is a good choice? Don't buy one: USB tether from your phone. If buying anyway, check the phone provider's website for compatible devices, then buy one inexpensively refurbished from say, EBay. Example: I'm on T-Mobile and the T-Mobile website sells Inseego and Franklin hotspots. The Franklin T10's are $30 refurbed, but they transmit data at 4G speeds. This is a waste of $$. The 5G Inseego M2000 (pictured) is $340 new from T-Mobile. I bought one for $110 off EBay. Works and charges fine.

configurations

Four sections below: phone WiFi, phone tether, and dedicated hotspot. Each requires a configuration file: an /etc/udev/rules.d/ file (tether) , and a wpa_supplicant.conf file (WiFi).

Part 1: phone hotspotting

It may overheat, but the phone can provide internet 3 ways. bluetooth, wifi, or USB cable tethered. The bluetooth is too insecure to bother with. That leaves:

1. Wifi hotspot: enable/disable in phone Settings as needed. Connect several devices with ESSID and password. If neighbors or wardialer squat it, can WAP it and/or hide the ESSID. /etc/wpa_supplicant/
2. USB tether: enable/disable in phone Settings as needed. Connect as ethernet (no pass) to single device. Cooler than WiFi, more secure than WiFi, charges while connected. Set this up, even if it's not needed. /etc/udev/rules.d/

BTW, USB tethering can also be configured for reverse-tethering - taking internet from one's connected laptop and sending it to the phone. Not sure when this would be useful, but it's available.

1a. Wifi hotspot

This is the simpler of the two, but more open to hacking. No configuration of the laptop/desktop is required -- the phone becomes another Wifi connection point, and the user connects their laptop to it like they would any other Wifi router/source.

1a. use (WiFi)

• Settings --> Connection & sharing --> Mobile hotspot (slider). Once enabled, a hotspot icon appears in the upper R, near the 5G reception bars.

• # wpa_supplicant -D wext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf &

• # dhcpcd wlan0

1a. setup (WiFi)

• Settings --> Connection & sharing --> Mobile hotspot (slider). Tapping on Mobile Hotspot will allow a person to change the ESSID name, hide it, change the password, the security, etc.
  
  Save after all are set

• $ wpa_passphrase [ESSID] [password]

• # nano /etc/wpa_supplicant/configfile.conf

  Put in the info from passphrase. The line scan_ssid=1 is necessary if hiding the ESSID. For security, key_mgmt=WPA-PSK is standard.

• # chmod +x /etc/wpa_supplicant/configfile.conf

1b. use (USB tether)

Android tether page (Arch). Tethering is slightly more complicated than Wifi for the initial setup, because configuring on the desktop/laptop is needed. Worth it.

tether use

• Connect USB cable
• On the Android, doesn't matter USB mode, but move slider below to right to enable tethering
  
• On the laptop/desktop:

  # dhcpcd enp0s21d0u2

1c. setup (USB tether)

Linux desktops/laptop network connections include loopback, ethernet id, and wifi id. This USB-tethered phone will act like a second ethernet card. The connection becomes added to the list inside /sys/class/net.

• # pacman -S usb_modeswitch android-udev android-tools

  ... can also install openvpn if desired for other options.
• group "adbuser" will have been created, so add the user...

  # nano /etc/group [add user to adbuser]

• phone: attach USB cable and ENABLE USB TETHERING (!) in Settings.
• desktop: check network units already in place. each device will have 3 lines eg, the ethernet:

  $ systemctl list-units |grep net
  sys-devices-pci0000:00-0000:00:1b.6-net-enp0s21b6.device
  loaded active plugged Ethernet Connection I219-LM
  sys-subsystem-net-devices-enp0s21b6.device

• $ lsusb
  Bus 001 Device 012: ID 22d9:276a OPPO Electronics Corp. SM8150-MTP _SN:E18D827E

  the phone contains a different product ID for its internet connection than a basic USB connection; USB TETHERING MUST BE ENABLED in the phone Settings to get a correct product ID. (see next step)
• The kernel will not create a net connection with only a Vendor ID, it must also have the (correct) Product ID:

  # nano /etc/udev/rules.d/90-android-tethering.rules
  # Execute pairing program when appropriate
  ACTION=="add|remove", SUBSYSTEM=="net", ATTR{idVendor}=="22d9", ATTR{idProduct}=="276a", ENV{ID_USB_DRIVER}=="rndis_host", SYMLINK+="android"

• # udevadm control --reload

• now our device identifier should appear on the list. It's unauthorized for a connection yet, but should exist:

  $ systemctl list-units |grep net
  sys-subsystem-net-devices-enp0s21d0u2.device loaded active plugged
  SM8150-MTP__SN:E18D827E

• now that we have its name, I just wrote it down, unplugged and replugged the USB cable and re-enabled tethering and:

  # dhcpcd enp0s21d0u2

  ...then I pinged to verify. Could also do a speed test.

extra: adb command setup

To issue commands into the phone from the connected device using "adb" commands (and not just connect to internet), which is necessary for more complicated things like port forwarding and AziLink...

• get the USD VID from lsusb and then

  # nano /etc/udev/rules.d/51-android.rules
  SUBSYSTEM=="usb", ATTR{idVendor}=="22d9", ATTR{idProduct}=="2765", MODE="0666", OWNER="foo", GROUP="plugdev"

• # udevadm control --reload

• plug USB into phone, enable the file transfer/auto option

• $ adb devices
  List of devices attached
  143cfd76     device

Having this connection for ADB commands seems to heat up a laptop and use memory. Accordingly, I keep the ADB connection turned-off by commenting out the defining line in /etc/udev/rules.d/51-android.rules unless I am going to use adb commands.

Part 2: dedicated device hotspotting

Using the 5G Inseego M2000, we can connect:

1. Wifi hotspot: enable/disable in menu. Connect several devices with ESSID and password. /etc/wpa_supplicant/wpa_supplicant.conf
2. USB tether: enable/disable in menu as needed. Connect as ethernet (no pass) to single device.

Use the embedded website to change the name of the network and its password. When it first arrived, was missing SIM, then SIM arrived by mail from T-Mobile. However, even though device was T-Mobile a call still had to made to unlock it. They never were able to unlock it, even after letters to T-Mobile.

2a. Wifi hotspot

This is the simpler of the two, but more open to hacking. No configuration of the laptop/desktop is required -- the phone becomes another Wifi connection point, and the user connects their laptop to it like they would any other Wifi router/source.

2b. use (WiFi)

• Settings --> Connection & sharing --> Mobile hotspot (slider). Once enabled, a hotspot icon appears in the upper R, near the 5G reception bars.

• # wpa_supplicant -D wext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf &

• # dhcpcd wlan0

1a. setup (WiFi)

• Settings --> Connection & sharing --> Mobile hotspot (slider). Tapping on Mobile Hotspot will allow a person to change the ESSID name, hide it, change the password, the security, etc.
  
  Save after all are set

• $ wpa_passphrase [ESSID] [password]

• # nano /etc/wpa_supplicant/configfile.conf

  Put in the info from passphrase. The line scan_ssid=1 is necessary if hiding the ESSID. For security, key_mgmt=WPA-PSK is standard.

• # chmod +x /etc/wpa_supplicant/configfile.conf

2c. use (USB-tether)

Android tether page (Arch). Like the phone, we can USB-tether with the dedicated device.

• Connect USB cable
• Unlike the Android, software selects tethering automatically. No adjustments are needed.
• On the laptop/desktop:

  # dhcpcd enp0s16f1u2

2d. setup (USB tether)

Linux desktops/laptop network connections include loopback, ethernet id, and wifi id. This USB-tethered phone will act like a second ethernet card. The connection becomes added to the list inside /sys/class/net.

• # pacman -S usb_modeswitch android-udev android-tools

  ... can also install openvpn if desired for other options.
• group "adbuser" will have been created, so add the user...

  # nano /etc/group [add user to adbuser]

• phone: attach USB cable and ENABLE USB TETHERING (!) in Settings.
• desktop: check network units already in place. each device will have 3 lines eg, the ethernet:

  $ systemctl list-units |grep net
  sys-devices-pci0000:00-0000:00:1b.6-net-enp0s21b6.device
  loaded active plugged Ethernet Connection I219-LM
  sys-subsystem-net-devices-enp0s21b6.device

• $ lsusb
  Bus 001 Device 008: ID 1410:b020 Novatel Wireless M2000

• Edit the udev rule to add the VID information. We can simply add another entry to the file we created for the phone.

  # nano /etc/udev/rules.d/90-android-tethering.rules
  # Execute pairing program when appropriate
  ACTION=="add|remove", SUBSYSTEM=="net", ATTR{idVendor}=="1410", ATTR{idProduct}=="b020", ENV{ID_USB_DRIVER}=="rndis_host", SYMLINK+="android"

• # udevadm control --reload

• now our device identifier should appear on the list. It's unauthorized for a connection yet, but should exist:

  $ systemctl list-units |grep net
  sys-subsystem-net-devices-enp0s16f1u2.device loaded active plugged M2000

• we have its kernel name and can connect with dhcpcd:

  # dhcpcd enp0s16f1u2

  ...then I pinged to verify. Could also do a speed test.

more configuration

This isn't necessary, but a person can visit the manufacturer's site (http://my.mifi) and create an account to access additional functions on the dedicated device. This probably also provides them with additional information to sell advertisers.

Part 3 troubleshooting

m2000 - invalid SIM

3 days after SIM installation, it was still spawning errors. In my online T-Mobile account, device information indicated it was "locked but available for unlock". A call to T-Mobile revealed the m2000 must be on their network 45 days, then corrected and said he could do it. They never did. They devices "locked".

m2000 - dhcpcd errors, no ping.

Strangely, browsing worked OK, but seemed to buffer.

# dhcpcd enp0s16f1u2
dhcpcd-9.4.1 starting
DUID 00:03:42:4D:45:44:01:5E:4C:10:8A:30:C6:C0:4B:46:4A:31
enp0s16f1u2: waiting for carrier
enp0s16f1u2: carrier acquired
enp0s16f1u2: IAID CC:03:11:AC
enp0s16f1u2: adding address C2D0::C6BF:7F13:14C2:1315
enp0s16f1u2: rebinding lease of 192.168.1.2
enp0s16f1u2: probing address 192.168.1.2/24
ps_bpf_recvmsg: Invalid argument
ps_bpf_recvmsg: Invalid argument
ps_bpf_recvmsg: Invalid argument
enp0s16f1u2: leased 192.168.1.2 for 86400 seconds
enp0s16f1u2: adding route to 192.168.1.0/24
enp0s16f1u2: adding default route via 192.168.1.1
ps_bpf_recvmsg: Invalid argument
forked to background, child pid 31249

Is data unlimited, albeit slow? Which one? Where do plans work?

5G - OnePlus Nord200 - Oppo N1 - DE2118

Links: Wiki page

model

Like all smartphones, manufacturing is murky and obsfucated, and can best be determined by its USB VID

$ lsusb
22d9:2765 OPPO Electronics Corp. Oppo N1

"22d9" is Oppo Electronics, so that's our real indicator. Interestingly, the OS is Cyanogen Mod, which they call Oxygen. $125 refurb off EBay. For travel it's GSM and

phone history

The best I've had was the LG-D520, best text feel, fast, least distracting

5G Nord N200 2022/08	$125 Refurb unlocked Ebay, originally on T-Mobile. Android 12, upgraded from original Android 11/Oxygen. Type C USB cable
4G Droid Turbo 2 2020/05	$50 XT 1585 Refurb unlocked from Verizon. Android 7 (Nougat) Reliable Motorola phone. Could not achieve VoLTE after 5G upgrade due to Verizon firmware. Type B ("micro usb") USB cable. Battery about 45 minute turnaround. Keep unrooted as a good cam phone.
3G Optimus F3Q 2015/05	$50 LG-D520. Refurb unlocked from T-Mobile. Android 4.4 (KitKat) Easy battery swap. This phone has slightly better-yet touch for texting than the G1. The G1 was good for texting. The LG-P659 was as smaller, glass screen (non-slider) version.
3G LG C195N 2012	$100 new. T-Mobile. Slider. Android 1.5 (Cupcake). Great texting touch and pocket and hand feel. Wide enough not to slip out of pocket. Simple, reliable, great battery life. GSM. Might still be good for Europe. SMS/MMS, no Internet.
3G LG G1 2010	$100 new. T-Mobile. Slider. Android 1.5 (Cupcake). Slider has good touch for texting. Stolen at work.
2G Moto. Razr V3 2006	$? T-Mobile. Favorite phone. Black metal case, reliable, battery features, easy to use, heft even better than Nokia, easy battery.
2G Nokia E.6010 2002	$? Cingular. Candy bar, GSM 850/1900. Also called NPM-10. Superb heft and single hand key access. Easy battery (NK3310), dedicated charger. FCC:GMLNPM-10X
Gx Alcatel 2002	$30,new. Price part of a Deutsche-Telocom package which included a number of minutes and other features.

Settings

Android 12. One of the key things to do is disable "Quick Device Connect", a persistent problematic app that will ask for one's location. Settings -> Apps -> App management -> (three dots in upper R) -> Quick Device Connect. Force stop and turn off all notifications.

• awake: double tap, then swipe up.
• usb setup: Settings -> System settings -> Developer options -> Select USB configuration
• nav bar / 3 buttons: there's no back button in the Chrome browser, so the buttons are helpful; triangle is a back button. Settings -> System settings -> System navigation select buttons or gestures (gestures has Navigation Bar).
• screen timeout: Settings -> Display & brightness -> Auto screen off (set mine to 1 min)
• notifications: apps like Uber, EBay often interrupt with announcements. Settings -> Notifications & Status bar -> Apps are in a column; toggle the notifications on/off as desired for each app.

Glass/LCD Replacement ($38) and Battery ($18)

Had the phone a couple months and then was at a meeting with some guys and dropped it on concrete. First time broken screen experience. For this phone a significant job no matter glass or glass+frame. Why not also replace (LiPo) battery since it has to be removed to do the glass. Comments on that a little further down.

Replacing only the glass is easiest fix, but requires bending-out the frame to remove the glass. I considered a smooth press-fit would be more durable and stock-appearing, so I purchased a frame/glass combo. An extra $6. Another $5 for the toolset with B-7000 glue. With tax, call it $38 for all of this, non-OEM. I also read the seller's customer feedback comments for additional tips from buyers. Delicate job and, again, not OEM.

Beyond this, having or jerry-rigging a heat-pad or gun, and clamps for re-gluing, are all required. This is true also for a straight battery job. The video below has the "with-frame" process I did, but can do just the glass, as noted above, after battery step. See video commenets.

Screen w/frame replace (17:09) Geardo, 2022. Latter half of video -- reassembly portion -- valuable to watch prior to initial disassembly. Comments valuable.

The battery was an extra $18 incl. tax. I got a LiPo version, like the OEM. Li-Ion requires an overcharge protection circuit that's likely not in the phone since it's LiPo OEM. I don't want the phone burning up on the charger.

4G -> 5G WTF

NB: The least expensive (T-Mobile) 5G entry with enough features (Android 11, Snapdragon 480 vs. Snapdragon 805 and Android 7 in my 2015 Droid2), is probably the Nord N200 5G. ~$200, or about $140 (including tax) refurbished.

Links: EBay.com :: New in box older phones :: T-Mobile

---

5G is definitely here and governments are rejoicing (all digital! AI packet inspection!). But as part of this rollout, providers began shutting down 3G networks. This meant the new fallback for voice calls became digital 4G, no more analog 3G. Also, different providers use different frequencies.

1. VoLTE functions (15:54) Telecom Tutorial info, 2018. Pre-5G video provides history of 2G, 3G, 4G and thorough description of VoLTE services including important note that the provider may not enable VoLTE on all handsets. 11:00 particularly applied to me.
2. VoLTE functions turn/on off (6:52) Make Knowledge Free, 2021. Thorough description of ways to enable VoLTE, assuming provider, handset, SIM, and handset firmware are provisioned to do so.

part 1 - 4G begins to malfunction

I have a reliable 4G phone. In March, a T-Mobile store checked the device and said, "it will work after the July 5G rollout". In August however, 4G data was reliable, but the phone dropped voice calls. Obviously, the phone was attempting to switch voice traffic to analog 3G, which no longer existed.

1st diagnostic

I checked several screens, but will only repeat the most relevant one below

analysis

1. 4G data is not a problem in this phone. The 4G icon is illuminated in the upper right, and there had been no interruptions to texts or internet.
2. my device is probably white-listed by T-Mobile, just as the store clerk said.
3. Hardware is provisioned for VoLTE or the greyed out "VoLTE provisioned" switch would not be present

Given 1-3, the disfunction must either be the device's firmware, or the device's SIM card. If it's the SIM, T-Mobile can send me a VoLTE provisioned SIM. If it's firmware... I'm out of luck and will have to purchase a new device. The phone (though reliable) is an unlocked former Verizon phone with Verizon firmware. Verizon firmware is impossible to update through T-Mobile.

2nd diagnostic - t-mobile

I recorded the IMEI of the phone but, as we can see from the screenshot above, SIM numbers were not provided to the user in Android 7. I physically pulled the SIM to record the ICCID number. With the ICCID and IMEI, I opened a laptop, went to the T-Mobile website and intiated a text chat with a tech. He took the numbers and determined the problem was likely SIM provisioning. He orded a new SIM overnighted to me. It arrived 2 days later.

3rd diagnostic - SIM install

After installing the new SIM (see below), there's a 2 hour delay for T-Mobile to fully detect and provision it.

I waited 3 hours and resarted the phone twice, however the VoLTE option was still greyed-out as previously. Verified also that voice calls were dropping. SIM was no solution. Failure.

2nd analysis

The remaining unverified problem was firmware. We can see from the phone info that both baseband and system firmware are Verizon. If I'd had unlimited patience, I might have written Verizon to see if they had a more recent version that was VoLTE enabled -- there was nothing on their website which looked like a solution.

conclusion

Appears I need to purchase a 5G phone; the Verizon firmware limitation cannot apparently be remedied. Lesson: if buying a cheap unlocked phone, it's smart to purchase a device which was previously used on my current provider's network. Accordingly, I found a 5G OnePlus Nord 200 on Ebay for $130, unlocked from T-Mobile.

---

part 2 - related notes

Home: SIM installation (T-Mobile)

Home installation relies partly on one's online T-Mobile account. The old SIM needs to be operating to log-in (2FA) to that account, and a person must complete the SIM swap before the online account times-out. There's no way to 2FA back into the online account without the new SIM being registered. Here's what I needed:

• paper; copy down the new SIM number from the package
• a tool to quickly pop the SIM from the phone
• laptop to get on the T-Mobile website

1. login to T-Mobile. T-Mobile sends a 2FA text to my phone
2. Go to Account->Lines and Devices ->Change SIM.
3. Work fast before you're logged out.
   • turn-off the phone
   • pop-out old SIM and put in new SIM
   • power-on the phone (all kinds of error messages will appear -- disregard them)
4. enter the new SIM number in the box for it on website (laptop) and then "continue"
5. completion message appears
   
6. After about 20 minutes of the phone being inoperable, a text from T-Mobile arrived asking for new SIM verification.
7. re-establish accounts in phone. Most accounts (google play, voice, phone) are tied to a SIM card, not to a phone, so all of these accounts logged out and needed to be re-established.

Vendor: possible store visit

SIM config can only be done by the provider, so if the phone is still not properly registering in the network, take it to the provider. More about it here. Also the phone probably has to be registered for IMS, unlike 3G. And if a person has a perfectly configured 4G phone, the provider might also be banning that device model from the 5G network. With so many variables, it's good to be patient with oneself if a person eventually must visit the vendor shop.

VoLTE functions (23:15) Telecom Tutorial info, 2018. Pre-5G video provides in-depth PowerPoint of how 4G VoLTE connects from User Equipment(UE) with voice in an IP Multimedia System (IMS). Covers congested network scenarios also.

another sample - short vid (24 seconds)

Note: TIME CALCULATOR, TEXT ANIMATOR (fade-in/out).

This post is how I manipulated a short vid, which is easy. But if a video is, say, 70 mins long, maybe I only want to keep the first 55 minutes of one of these...

$ ffmpeg -i foo.mp4 -t 55:00 -c copy output.mp4

Alternatively, perhaps I have a 4 hour and 20 minute video, from which I want to keep the final 20 minutes...

$ ffmpeg -ss 4:00:00 -i foo.mp4 -c copy part5.mp4

example

Got a couple minutes of funny video when slowed down to half speed. The audio needed to be gained up some, that was its only flaw.

Plan: cut it down to 30 while convert to MP4, slow audio to half speed, split audio to clean and gain. Recombine. Observe closely and cut down to 24. Add fade-in, fade out. Upload and laugh.

1. cut down to 30 and convert to MP4

Trivial to convert containers -- do it when cutting.

$ ffmpeg -i original.mkv -t 00:30 -c copy short.mp4

2. slow audio and video

Halve both the audio and video speeds, as seen in the video beneath the command.

$ ffmpeg -y -i short.mp4 -filter_complex "[0:v]setpts=2.0*PTS[v];[0:a]atempo=0.5[a]" -map "[v]" -map "[a]" halfspeed.mp4

half-speed example (3:28) The FFMPEG guy, 2021. Does audio, video, then both audio and video. Reveals filter complex mapping.

3. split audio and video

Separation was necessary in this case b/c the audio was a little dirty. I needed to work on it independently.

$ ffmpeg -i halfspeed.mp4 -vn -ar 44100 -ac 2 audio.wav
$ ffmpeg -i halfspeed.mp4 -c copy -an video.mp4

4. recombine audio and video

I force the bitrate -- ffmpeg's default is too low. I like 3M for sports, 2M for interview or person just talking.

$ ffmpeg -i video.mp4 -i audio.wav -b:v 3M combined.mp4

5. review and cut to 24 secs

Watch in slow motion to obtain a cut time, in this case 24 seconds. Then a repeat of #1 (except no container change).

$ ffmpeg -i combined.mp4 -t 00:24 -c copy short24.mp4

6. fade-in and fade-out (1 second)

There are several ways to do this. The fade filter can be used by frame, or by seconds. So if I want to use the frame number, I need to find the FPS and take that time the number of seconds to the effect. I use "st" for seconds and "s" for the frame. This one uses seconds.

$ ffmpeg -i short24.mp4 -vf fade=in:st=0:d=1,fade=out:st=23:d=1 -b:v 3M faded24.mp4

fade-in fade-out example (3:42) The FFMPEG guy, 2021. Does fade-in(frames), fade-out (seconds), and then both (frames)

7. upload to YT

I've had good luck with....

• 30 FPS
• MP4
• 1920x1080 (FHD/2K) or 1280x720 (WXGA/HD)
• 3M or 2M b:v
• h264 high (c:v libx264)
• aac audio

audio note: if using a WAV as an input, ffmpeg defaults the audio to AAC, 124Kb. Change the bitrate with b:a 192k, and the encoder to MP3 with c:a libmp3lame. I usually upload as with AAC as YouTube does some converting that "seems" to make MP3 uploads slightly less crisp than AAC uploads, not sure.

Basic $ ffmpeg/ffprobe -i command to get info on a media file cannot be grepped to find, say, various libs. It's annnoying but ffmpeg/ffprobe sends to stderr not stdout. Of course error can't be can't be piped. Error must be rerouted. Gotta use 2>&1. Eg, to verify libmp3lame...

$ ffmpeg -i foo.mp4 2>&1 | grep mp3

...and if there's any result from that, it's in there somewhere.

video -- text

Side note: to turn off annoying inactivity blanking, say when using a non-VLC player...

$ xset s -dpms

And then to restore it (if desired)....

$ xset s dpms

Some ppl find they have to add "$ xset s" entries "noblank" and/or "off", that is to use all three. In vanilla Arch for example, all three are required

---

This post is a trail of crumbs for incorporating graphic, moving text into video -- a long term, continually evolving project. There are also some screenwriting notes at the bottom, since scripts are sometimes scrolled or portions used for graphics, etc.This top portion reviews subtitles, and a previous post (1/2021), wherein I addressed the basics of static titles or labels. The subtitle information includes both user-selected, and forced subtitles. So there's a lot to cover this post: subtitles, basic labeling, graphic text.

subtitles

Subtitles can be added to videos in an optional or forced capacity. Here's a page.

There are many subtitle formats, however we want our player to be able to use them. I see these most often and they have somewhat different applications.

1. SRT from the SubRip days. This is the most common but appears to only have bold, italic and underline.
2. ASS the most expressive, according to the last post in this thread. Fonts colors, etc. I've never used it.

extract subtitles

Get the subtitle (.SRT) file from the video and look it over, update it, re-imbed it, etc

$ ffmpeg -i foo.mp4 somename.srt

To extract them from YouTube videos, where a person might have seen a foreign film can require more reading. Typically they're just in English and will come in a VTT file, received with the following:

$ youtube-dl --all-subs --skip-download [URL]

from prior post*

*CPU intensive render, verify unobstructed system cooling.

For one or more lines of text, we can use the "drawtext" ffmpeg filter. Suppose we want to display the date and time of a video, in Cantarell font, for six seconds, in the upper left hand corner. If we have a single line of text, we can use ffmpeg's simple filtergraph (noted by "vf"). 50 pt font should be sufficient size in 1920x1080 video.

$ ffmpeg -i video.mp4 -vf "[in]drawtext=fontfile=/usr/share/fonts/cantarell/Cantarell-Regular.otf:fontsize=50:fontcolor=white:x=100:y=100:enable='between(t,2,8)':text='Monday\, January 17, 2021 -- 2\:16 PM PST'[out]" videotest.mp4

Notice that a backslash must be added to escape special characters: Colons, semicolons, commas, left and right parens, and of course apostrophe's and quotation marks. For this simple filter, we can also omit the [in] and [out] labels. Here is a screenshot of how it looks during play.

Next, supposing we want to organize the text into two lines. We'll need one filter for each line. Since we're still only using one input file to get one output file, we can still use "vf", the simple filtergraph. 10pixels seems enough to separate the lines, so I'm placing the second line down at y=210.

$ ffmpeg -i video.mp4 -vf "drawtext=fontfile=/usr/share/fonts/cantarell/Cantarell-Regular.otf:fontsize=50:fontcolor=white:x=100:y=150:enable='between(t,2,8)':text='Monday\, January 18\, 2021'","drawtext=fontfile=/usr/share/fonts/cantarell/Cantarell-Regular.otf:fontsize=50:fontcolor=white:x=100:y=210:enable='between(t,2,8)':text='2\:16 PM PST'" videotest2.mp4

We can continue to add additional lines of text in a similar manner. For more complex effects using 2 or more inputs, this 2016 video is the best I've seen.

Ffmpeg advanced techniques pt 2 (19:29) 0612 TV w/NERDfirst, 2016. This discusses multiple input labeling for multiple filters.

PNG incorporation

If I wanted to do several lines of information, an easier solution than making additional drawtexts, is to create a template the same size as the video, in this case 1980x1080. Using, say GiMP, we could create picture with an alpha template with several ines that we might use repeatedly, and save in Drive. There is then an ffmpeg command to superimpose a PNG over the MP4.

additional options (scripts, text files, captions, proprietary)

We of course have other options for skinning the cat: adding calls to text files, creating a bash script, or writing python code to call and do these things.

The simplest use of a text files are calls from the filter in place of writing the text out each filter.

viddyoze: online video graphics option. They reender it on the site, but it's not a transparent overlay, but a 720p MP4.

viddyoze review (14:30) Jenn Jager, 2020. Unsponsored review. Explains most of the 250 templates. Renders to quicktime (if alpha), or MP4 is not.~12 minute renders

screenwriting

We of course need a LaTeX format, but then...

Answer these 6 Questions  (14:56) Film Courage, 2021. About, want, get it, do about it, does/doesn't work, end.
PBX - on-site or cloud  (35:26) Lois Rossman, 2016. Cited mostly for source 17:45 breaks down schematically.
PBX - true overhead costs  (11:49) Rich Technology Center, 2020. Average vid, but tells hard facts. Asteriks server ($180) discussed.

paperhater - classification issues (minus database)

We want to organize electronic media as much as possible, without a database, for at least two reasons: 1) if we can reach file organization and "findability" goals without the database, then we've saved the expense, 2) if our situation ultimately requires a database, the need for one becomes more cleary defined. For this reason, the first steps are the same, database or no ultimate database.

overview

We begin with an entirely non-homogenous mess of files, a deck of scattered cards but without names or suits. Our pile includes everything from receipts, code, diplomas, research articles, manuals, correspondence, old emails, and on. Some of these files are active, some are reference, some are family history, and on.

first cut

Our first cleavage is between all the articles, texts, and manuals which might be used as a reference, or cited in a thesis, etc. Call this our library. The second pile (receipts, forms, photos) is... everything else. We'll overlay a universal file and folder naming convention to both categories. However libraries require an additional handling step due to standardized citation requirements.

There's help for both categories. 1) for library items, ISBN/ISSNs, Dewey Decimal(since 1876), and/or a Library of Congress classifications, already exist. We can organize these using BibTeX. 2) for everything else, think back to the paper era. Records management had proven ways to organize these. This process has become electronic records management (ISO 15489), and is just as helpful. See the first video below.

records basics (25:31) US National Archives Records Management, 2009 Doubling info every two years. 11:00 paper's been lost -- we used to know how to do it. sample basic structure. 17:00 She recommends a file plan.
file management (1:23:57) Nicholas Andre 2013. Windows-based lecture, but clear thinking fellow gives good context for what we're after. Backgrounder. Corning (NY) Community College course.

naming - files

3 part naming convention. Subject, date, code. These vary in order depending on what's most important to the user of that file.

file naming conventions (10:00) Simpletivity, 2018. Ad first 1:36. Uses 3 part naming, succinctly described. Probably best at .75 speed. Comments excellent.

naming - folders(directories)

Mostly the naming is the same as the files, but arranged vertically. 3 layer naming convention. Function, subfunction, action. Thinking of the deck of cards, we can arrange by suit, by number, by color. What's the fastest way to find a card if they're in folders? Might depend on my style of play. NARA notes that granularity of folders depends on number of docs for that folder.

folders - website (8:19) John Morris, 2018. Standard website folder organization.

plan - file & folder

This step is the combination of the decisions on naming of files and folders. Government suggested in General records schedule from the NARM website.

records basics (25:31) US National Archives Records Management, 2014. Donna Read. informatioin doubling every two years. 11:00 paper's been lost -- we used to know how to do it. sample basic structure. 17:00 She recommends a file plan.
file plan basics (47:44) US National Archives Records Management, 2013. Jeff Benson. staff consensus, record retention,
holding to center (10:46) Luke Smith, 2020. Part of sticking to a file plan is understanding that what works for a person can be useful.
federal social media (43:37) US National Archives Records Management, 2013. Bethany Cron. Federal records, con

A. non-research example

This is the thing for say, billings or other saved items.

B. research/library example

We rarely seem to get New Yorker articles directly related to Bay Area unless it's a controversial topic: presumably editors don't want to research on their home NYC turf and alienate locals. Let's say I want to keep one such article to cite later. The information panel.

Step 1 - gather info

Very difficult without a physical copy. There's no online index for magazine volume and numbers cross-referenced with date, at least that I've found. With a physical copy of the New Yorker, I can get the info -- ISSN 0028792X, Volume 98, No 4, Mar 14, 2022. NYC, NY. The article of interest in this example: "The Access Trap", is from Nathan Heller, pgs 34-45. I've scanned these pages into a PDF, as yet unnamed. If online, we may also find other control numbers we want to include. Eg, if we had a dissertation to cite, we know that, "There is no single source for a comprehensive dissertation search." We might encounter a different control number at different sites and want to include them.

Step 2 - PDF name - first cut

Revisit the PDF name and evaluate the 3 part name. It should include a date, name, and subject code, in a way that at least hints at the file contents. In this case, we might, eg

20220314_NY_LowellEquity.pdf

Step 3 - BibTeX it

Information for storage, retrieval, and citation (chicago, mla, apa) could come from a bibtex BIB file. Bibtex files can be massaged in the document for any citation format. The question we'll want to ask later is how many BIB files ? since we can't have just one immense BIB file for ever article we have -- and of course we need to create a custom one for any document we create -- we're going to run into meta-problems. In this case we'd "@article" template, minimally...

@article{uniquecitekey,
author = "Heller, Nathan",
title = "The Access Trap",
journal = "The New Yorker",
year = 2022,
volume = "98",
number = "4",
month = "Mar",
pages = "34--45",
issn = "0028-792X",
doi = "20220314_NY_LowellEquity.pdf",
note = "Lowell HS equity clause"
}

We probably don't need 11 fields in a database, but BibTeX has these readymade. Review the DOI and see that it corresponds with the filename -- make any adjustments.

apa style bibtex (2:53) Charles Clayton, 2016. Also does IEEE. Important about making certain filenames match. Probably best at .75 speed. Comments excellent.
latex review (59:42) Derek Banas, 2019. Typical Banas killer review. He uses "TexShop", which appears to have autocomplete
bibtex citation (7:38) Center of Math, 2015 Have to run it twice, like a table of contents.

Step 4 - filename and folder again

Review the filename, folder, and BibTex information. Or, if a random receipt, review retreival issues again, such as the folder depth (no more than 3) and the filename (gives some info).

C. database or not?

ISSN number searchable here, topic, Somtimes we have subtopics