Thursday, July 30, 2020

blender 2.8 notes - vse, video

contents
clip matchingplugins necessary plugins
setup and render
keyframeswatermark

NB: set rendering output confinguration prior to editing, esp FPS


Blender or, as I call it, "Death from 1000 Cuts", is vast, almost an operating system. It is Python + FFmpeg based. KDenLive, the other major Linux GUI editor, is MLT based. Whether using Blender for animation or video (this post concerns video), a reasonably good understanding of keyframes goes a long ways. The reason is that, portions of edits which don't require keyframes can be done with a CLI editor. Blender is likely to be used for sliding text on or out of a frame, etc.

what has to match?

  • frame rate(in ffmpeg: avg_frame_rate) must match in final edit. Having a uniform frame rate, dimensions, bitrate, and so on, makes for easier editing. So if just using a few varied clips, it's worthwhile to recode them to match, prior to importing. Obviously, frame rate will ultimately be forced in the final render, and it can be jerry-rigged during editing if desired...
    Dealing with Variable Framerates (7:03) Michael Chu, 2019. Audio unaffected by framerate, but we want the video to correspond. This is a manual solution when many clip-types are present.
  • tbn, tbc, tbr These are ffmpeg(Blender's backend) names for timing information beyond the fps. The time_base (tbc) is simply the inverse of the time_scale (tbn), but there is not necessarily one frame for each tick of the timebase (see 2nd link below). Timescale (tbn) is often 90K.
    Variable names w/descriptions (page) GitHub, 2020. What each variable pertains to. Inaccurately substitutes "timebase" for "timescale".
    Mismatch between clips (page) 2013. An example timing mismatch between clips. Timebase vs. Codec Time base (page) 2013. Variations between these two can lead to problems.
    Container vs. Codec (page) Stack Oveflow, 2013.
    Ffmepg timebase (page) Alibaba, 2018. Description of how obtained.
    **NB: 90,000 is typical tbn because divisible by 24, 25, and 30, although 600 will work, a la Quicktime.
  • bit rate helpful if matched, varies in quality if not. Along with the i-frame interval, bit rate determiines the quality of a clip. It's OK for bit rate to differ across project clips -- they can still be fused -- as long as one understands that each clip's quality will vary accordingly.
  • i-frame interval these can vary from clip to clip and change throughout editing as keyframes are needed. However, attention should be paid to this again at the final render to see that an efficient, hopefully dynamic, setting is selected. In ffmpeg itself, the i-frame interval is defined by "Groups of Pictures".
  • PTS

setup (also render)

I strongly suggest configuring one's output render settings as the first step of any Blender project. A consistent framerate, codec, and other details set to the desired output, forces clips into alignment from the start. As a bare minimum, set the final rendering FPS when getting started. That being said, the final render requires i-frame interval adjustments. The newer codecs will do this dynamically, so that if there are periods of zero action, i-frame intervals can expand to, say, one every 10 seconds, etc.

Dealing with Variable Framerates (7:03) Michael Chu, 2019. Audio unaffected by framerate, but we want the video to correspond. This is a manual solution when many clip-types are present.
Blender 2.8 Setup (19:27) Mikeycal Meyers, 2019. Render settings begin at 5:00.

The directory structure follows a pretty standard setup with a user file in ~/.config/blender/[version]/startup.conf, but there are also files inside /usr/share/blender/, which is where add-ons seem to live. So it may be that there are configurations to save from both of these sources.

Keyframes (and other) in 2.8 (9:48) Code, Tech, and Tutorials, 2019. Solid inadvertent tour around the VSE via a simple edit. Shows how to do transitions simply without inserting them, pressing "i" for keyframe.
Blender 2.8 Setup (19:27) Mikeycal Meyers, 2019. Render settings begin at 5:00.
Ffmpeg blur box (14:48) Thilakanathan, 2019. Fast, but covers nearly everything. Rendering in last 5 minutes. Comments have a lot of tips.

render troubleshoot

I've put these rendering fixes at the top of the blog to help determine how to preventatively configure settings.

  • grey timeline is rendered but appears to have a sheen of gray over entire playback, like through a dirty window. Inside the little camera icon, go all the way to the bottom "Color Management" and change it from Filmic to Standard.

keyframes

Video keyframes are a large editing topic I cover separately, but a few notes here. Fundamentally, keyframes are a hand-entered subset of i-frames, added by users. Since they are a sub-type of i-frame, all keyframes are i-frames, but not all i-frames are key frames.

  • "i" to add, "Alt+i" to delete
  • keyframes can be edited in Timeline, Dope Sheet, or Graph Editor. Only manually added keyframes appear in these editors, not generic i-frames.
    Keyframe manipulation (6:18) Michael Chu, 2020.
  • the final render should eliminate as many keyframes as possible, to decrease the size of our output file

How to Delete all Keyframes (page) TechJunkie, 2018. This is in the older version of Blender but has loads of solid keyframe information.
Keyframes (and other) in 2.8 (9:48) Code, Tech, and Tutorials, 2019. Solid inadvertent tour around the VSE via a simple edit. Shows how to do transitions simply without inserting them, pressing "i" for keyframe.

plugins

Plugins appear to live in /usr/share/blender/[ver]/scripts/addons. There are some key ones to have

proxy encoding.

If a person has a gaming system, this is not a concern. However, many systems edit smoother if the video is proxied. The system is lags and jumps during playback. Proxying does not work well if a person has elaborate transitions that require exact keyframes and so on.

Proxy a clip or clips (4:17) Waylight Creations, 2019. How to deal with a slower system.

sound effects

Occasionally, we'd like a sound effect to this or that in blender without a video clip coming in and so on.

Inserting Sound effects in Blender(11:11) Oracles Stream School, 2020. OBS based tutorial, using the computer, not a capture card.

Sunday, July 26, 2020

rebuild ASUS Sabertooth Z77 ATX

A buddy has a 22lb Cooler Master RC-692-KKN2 CM690 ($120) from 2012 and inside we found a Sabertooth Z77 (LGA 1155 Slot) system Mobo ($350). The Z77 is a proven Mobo c. 2012: a firm foundation for a budget upgrade, or whatever.

 
CM RC-692 ($120)
Sabertooth Z77 ($350)
Corsair CMPSU-750TXv2 ($140)

 
 
Corsair CMPSU-750TXv2 Manual :: generic, no voltages, schematics, pinouts.
 
Sabertooth Z77 Manual :: missing voltages for fans or the pinouts on board.

ASUS Sabertooth Z77 (9:37) Linus Tech Tips, 2012. Full rundown plus some useful comments underneath
Using Cloud Storage (7:31) Wolfgang's Channel, 2019. Be sure to pick a provider that uses Xen or KVM, rather than OpenVz-based virtual machines.
ECC Memory considerations (9:47) Linus Tech Tips, 2021. ECC explained and shows the value of error checking in AMD vs. Intel, where it costs more.

baseline

Here's some of the other system features we started with. I got this info executing dmidecode, lshw, and smartctl -a /dev/sda (following# pacman -S smartmontools).

  • CPU Intel i7-3770 3.4G clockable to 3.8 ($140)
    LGA 1155 slot accepts newer i7-9700 or 10700, for roughly $300 or $400 respectively
  • BIOS AMI v.2104 8/13/2013
  • cooling All are 12VDC. 5x 140mm ball bearing, 3 pin (Cooler Master A14025-10CB-3BN-F1) $20 each, 2 x 35mm auxiliaries (Sunon EF35101S2-Q010-G99).
  • RAM - 16GB 1332MHz DDR3 synchronous SDRAM 240 pin (Corsair CMZ16GX3M4A1600C9 ~$140)
    array capacity is 32 GB. Currently 4 x 4GB sticks onboard. "Upgrade" to 32GB A-Tech RAM ~$140.
  • GPU  
  • Storage 1 x Seagate Barracuda HDD IDE (not detected by BIOS), 1 x SanDisk SDSSDHP256G in ATA mode on SATA 3. These are laptop SSD's. (~$120 used)
    Capacity: 2 x SATA3 and 4 x SATA2
  • Power Corsair CMPSU-750TXv2. No problems ($140)
  • Optical

first inspection

Dust is too prevalent inside and the fans are worn. Several unfiltered openings. The previous owner wired the fans undirected with air from the bottom, exiting top. The dust should be solved, and one might also seek a lighter box, eg the Q500L Midtower (magnetic filter).

fans - motherboard (9)

The motherboard seems to have 5 x 12VDC directed fans, 1 x undirected (3-pin) 12vdc fan, and 1 directed CPU fan.

We also have these oddballs:

  • Sunon EF35101S2-Q010-G99 x 2. Location: motherboard. These are both 35 mm MoBo 3-pin (unmanaged) "Assistant Fans", 12vdc (pg 2-33/35 manual below). Apparently the fitting can take up to 40 mm. The Motherboard Manual provided no pinout voltages, so that there was no way to determine if 5VDC or 12VDC without a teardown. The diagram isn't clear if these are installed as 35 or 40mm, but found the 35's installed and measured they will take up to 40mm.

The Sabertooth has a fan controller, the AI Suite II. However, this is somehow a software controller from Windoze. If not running Windoze, we might want an independent fan controller. Check out the one in the video below.

Fan types (17:32) Jayz Two Cents, 2020. Other information about hydraulic, mag-lev, and ball bearings. Hydraulic is more susceptible to dust.

The EZDIY-FAB 5 pack (5 VDC - use SATA connector) is a good $50 120mm horizontal (they are hydraulic) solution

fans - case fittings (7)

As noted above, the case has too many unfiltered openings, so that filters or jerry-rig pantyhose must be installed. These are the 5 listed in the case manual...

... and I found these two additional (undocumented) locations.

  • side panel directly beneath CPU. space for 1 x 70mm 3-pin 12vdc fan. This would blow directly onto the bottom of the CPU.
  • front 1 x 140mm currently blowing into the case. This appears controlled.

temp sensors

To understand the fan settings, we need temperature information.

S/PDIF

I noticed this was entirely disconnected, even to my GPU or Optical Drive. Seeking information, I learned that I was not alone. It seems the best is to get a small case outlet that sends it to an RCA jack if one is going to exploit it.

Sunday, July 19, 2020

Eachine E58 -- Drone X Pro scam

I wanted to inspect the chimney on the roof with a drone and thus fell prey to the Drone-X Pro scam. I trusted a YouTube ad without doing research, and paid $100 for a $25 drone. Now that I have this Eachine E58 (the only thing missing is the brand name on the controller), could I make the best of it? Nope. Failure.

The camera is only 2 megapixels, can only be seen through an app, and flight time is given as 7 minutes. Reviews note that even the slightest breeze will take it out of range. On mine, I had a common problem of the wifi connecting to my phone, but I could not work the controller or see video. The E58 apparently does not work with all phones. Review: dronedeliver.uk.

wifi - piloting - video - app

These four are tied together because the drone controller has no video screen. Suppose I wanted to inspect my roof. As the drone passes from my direct line of sight, I can no longer see where the drone is going, and I lose the ability to pilot the aircraft. This is remedied by a complicated solution with several possible failure points

  • the manufacturer inserted a wifi transceiver into the drone; it appears as a hotspot to wifi systems
  • users connect their cellphone to the hotspot using whatever wifi functionality is present in the phone
  • after connecting, users open a pre-downloaded app to view through the drone's video camera and pilot the craft
There is no suggested application in the directions, at least in English, however after an hour of YouTube videos, and forum posts, this one appeared most likely:

 

...and don't forget Google Play needs port 5228.

failure

My Droid Turbo connected to WiFi, but with a warning that I had no Internet connection, thus I think (just a guess) disabling the http transport necessary for viewing the video. The drone app noted that I needed to "connect" in spite of the phone showing I was wi-fi connected, as noted

Following my hunch, I went to this site and learned that I could potentially enable http transport on a phone which normally doesn't do so with non-internet LANS, however I would have to root my phone, which I didn't want to do.

sd video

Video is supposedly saved onto a micro SD, viewable after flight. However, the apps could never connect to the drone, and the hand controller had no function to start video recording. It appears that the video recording was never initialized in the drone -- there was no video on the SD card after the device was flown.

aftermath

I'll give the drone away to one of my friends who has a compatible cell phone, for $10 and a chimney inspection. It appears compatible means phones allowing HTTP transport on wifi connections, even when on a local LAN without DNS, eg mDNS.

Tuesday, July 7, 2020

rclone details

In a prior post, I'd found that using rclone to upload RESTful (rclone uses REST, not SOAP) data had become more complex -- by at least three steps -- than two foundational videos from 2017:
1. Rclone basics   (8:30) Tyler, 2017.
2. Rclone encrypted   (10:21) Tyler, 2017.
These videos are still worthy for concepts, but additional steps --choices actually -- must be navigated for both encrypted and unencrypted storage, whichever one desires. Thus, a second post. Unlike signing in and out of one's various Google and OneDrive accounts, all are accessed from a single rclone client. Rclone is written in Go (500Mb), so that immense dependency must be installed.

across devices

To install rclone on multiple devices, including one's Android phone (RCX), save one's ~/.config/rclone/rclone.config. For each installed client, simply duplicate this file and one can duplicate the features of the original installation. If one has encryption, losing this file would be very bad.

deleted configurations

  1. ~/.config/rclone/rclone.config (client). If this file is lost, duplicate it from another device. If lost entirely, access must be re-established entirely from scratch, and the encrypted files will be lost permanently.
  2. scope (google) Google requires authentication for access details for which Google keeps. Documentation is difficult to find, other than the OAuth info in the prior sentence. It appears that users cannot directly edit any of the 11 access scopes (files) defined, but rather only through a Google dialog screen. When installing rclone, 5 of the 11 scopes are available, for which I typically like "drive.file".

command usage

For simplest use, to the root directory...

$ rclone copy freedom.txt mygoogleserv:/

Not all commands work on all servers, so use...

$ rclone help

instead of...

$ rclone --help

The former will display only those commands on the installed version of rclone. The latter shows all commands, but not every compilation has these.

$ rclone about mygoogleserv:
Total: 15G
Used: 10.855k
Free: 14.961G
Trashed: 0
Other: 40.264M
Of course, there's also the GUI, rclone-browser.

encryption notes

Rclone documentation notes strong encryption, especially if salt is used. Minimally, we're talking 256-bit. Of course governments can read it, but what can't they read?
  • unencrypted accounts must be established first. Encryption is an additional feature superimposed onto unencrypted accounts.
  • remember the names of uploaded encrypted files; even the names of files are encrypted on the server and the original filename is necessary for download.
  • keep the same encryption password on all devices on which rclone is installed.

glossary

  • application data folder (Google) a hidden folder in one's Drive (not in one's PC). The folder cannot be accessed directly via a web browser, but can be accessed from authorized (eg OAth) apps, eg rsync. The folder holds "scope" information for file permissions.
  • authorization (OAuth, JWT, OpenID) protocols for using a third party REST app (rclone) to move files in and out of a cloud server (Google, AWS, Azure, Oracle), there's an authorization process between them, even though you are authenticated in both.
    What is OAuth (10:56) Java Brains, 2019.
    What is JWT (10:34) Bitfumes, 2018.
  • scope (Google). the permissions granted inside Drive to RESTful data uploaded by users using, eg, rclone.
  • REST Representational State Transfer API for server to client data transfer. Wikipedia notes this as an industry term and not a copyrighted concept by Oracle or Google. It refers to data exchanged by user-authorized third party apps between applications or databases and applications. This is as opposed to data directly entered by users, or data that is not authorized by users between servers.

    REST API concepts and examples (8:52) WebConcepts, 2014. Conceptually sound on this HTTP API, even though dated with respect to applications. Around 7:00 covers OAuth comprehensibly.

  • SOAP Simple Object Access Protocol. This is the older API for server to client data transfer.

    SOAP v. REST API (2:34) SmartBear, 2017. Very quick comparison.


Google 15GB

Users can personally upload and save files in Google Drive through their browser as we all know. However, Google treats rclone as a third party app doing a RESTful transfer and uses OAth to authorize it. Additional hidden files are created by Google and placed into one's Drive account to limit or control the process.
Within that process, there are two ways to rclone with Google Drive, slower or faster. The faster method requires Google Cloud services (with credit card) and a ClientID (personal API key). The slower way uses rclone's generic API connection.

1. Slower uploads

Faster to set-up, but slower uploads. Users regularly backing-up only a few MB of files can use this to avoid set-up hassles. It bypasses the Cloud Services API, and uses the built in rclone ID to upload as directed
  1. $ rclone config
    ... and just accept all defaults. For scope access, I chose option "3", which gives control over whatever's uploaded.
  2. verify function by uploading a sample file and by looking in ~/.config/rclone/rclone-config to see that the entry looks sane

2. Faster uploads

This method requires a lengthier set-up but, once configured, rclone transfers files more quickly than the generic method above. Users need a credit card for a Google Cloud Services account, which in turn supplies them with a ClientID or API key for rclone or other 3rd party access into Drive.
  1. get a Google email
  2. sign-up for Google Cloud services
  3. register one's project "app". In this case it's just rclone) with the Google API development team
  4. waiting for their approval -- up to 2 weeks
  5. receiving a Client ID and Client Secret which allow faster uploading and downloading through one's Drive account

These two videos move very quickly however they have the preferred Client ID and Client Secret method that supposedly speeds the process over the built-in ID's.

Rclone with Google API (6:38) Seedit4me, 2020. The first four minutes cover creating a remote and the 5 steps in creating the Client ID and Secret.
Get Client ID and Secret (7:29) DashSpan.me, 2020. Download and watch at 40% speed.

OneDrive 2GB

This primer is probably the best for OneDrive, however it also applies to many of the other providers

metadata and scope

These are hidden files within one's Google Drive. is is part of the Google Drive API v.3, which is what rclone uses to connect and transfer files. In particular, you will want to know about the Application Data Folder
Google API v3 usage (5:28) EVERYDAY BE CODING, 2017.
Get Client ID and Secret (7:29) DashSpan.me, 2020. Download and watch at 40% speed.
RESTFUL resources and OAuth (55:49) Oracle Developers, 2017.