Tuesday, June 30, 2020

backup - texts (sms/mms)

Unlike years prior to Patriot Act, BSA, and DMCA, both criminal and civil agencies seem to operate assuming anything on citizens' cell phones or computers will eventually be discoverable, if not somehow prosecutable. These forces are much larger than us and there's little way we can protect ourselves from their creepiness.

In the face of this, we'd prefer to delete all of our data every day but our lives would be even more negatively impacted by these forces if we do. We'd lose track of birthdays, anniversaries, important receipts, and so forth. So we still need to retain some data for our daily affairs in these absurd times.

Understanding that protecting data is impossible for less than a team of experts, what can we individuals do to retain some data, and somewhat mitigate its exposure? If we can do some selective capture, we might be able to maintain our activities without gov't and info-agency stalkers into 100% of our private affairs. Texts are possibly a good test-case.

  • application my two primary considerations are 1) include all conversations in a single back-up, 2) have a clear text, non-propietary format. There was a great app, "Email My Texts" which collated a selected period of texts into TXT and included attachment file-names. Google mysteriously removed this app from the PlayStore. It seemed like a killer app that customers were choosing over all alternative$, and maybe Google disapproved, not sure. The remaining apps have awkward formats which parse conversations or use proprietary XTML, PDF's (immensely inefficient), and so on. The best option I've found of the remaining PlayStore apps in 2020 is...
    ... which is worthy of upgrading to Pro, for something like $4.
  • format CSV the only option in text apps 2020. If a TXT app returns that can select all conversations in a single file and names the attachment, I'd take it over CSV, but that's 2020 for you.
  • storage run rclone to encrypt CSV files to cloud storage. I cover this thoroughly in next month's rclone post. Meanwhile...
    $ rclone listservers
  • searching obviously, the reason to back-up texts is the same reason to back-up emails. You might need some information downstream. How can we search encrypted CSV files in such a way that we can easily find keywords, and then print all the date and parties to the interaction? Not easily. Perhaps a Python script which displays the results in a browser, sequentially as it processes a CSV file.

storage

Assuming this is encrypted via Watch these videos first...

1. Rclone basics (8:30) Tyler, 2017.
2. Rclone encrypted (10:21) Tyler, 2017.

...which can be followed verbatim. There are some new details since these videos, discussed in one of my other blog entries, but the core of the setup is the same. Additionally, this video (8:19, 2017) has some good basic commands.

search

In order to find information in a haystack of encrypted CSV files, each file must be decypted and greped individually. Since we have many encrypted CSV files, this is unlikely to be efficient. It's probably worthwhile to have a passle of encrypted CSV files on the Cloud, and then a local backup for parsing with one's Python script.

emails

A second question arises from text retention which is how to save emails which we want and to encrypt these. How to display them?

Sunday, June 14, 2020

toolbox :: appslist

contents
cli-misc a lot of the install stuffmedia playback, creation, editing
coding Python, misc APInetwork basic connectivity
documentssafety
math/stats minus calculator

Arch :: list of applications


Items marked with an asterik "*" below should be accomplished during install prior to leaving arch-chroot. Remember that $ strace > catchit.txt 2>&1 [command] is our friend, and that there can be some pacman judo required. To safely uninstall without dependency breaking, # pacman -Rs [app] is typically enough. Our old post on cleaning up orphan space is also useful.

system configuration

During install, a few details prior to downloading apps makes life easier. I went over some of these in 2015. Limiting journal size...

# nano /etc/systemd/journald.conf
Storage=auto
SystemMaxUse=200K

CLI (misc)

  • diff: 1MB, # base :: typical: $ diff -s file1.txt file2.txt You can also send it to a file a,s,c.see here. If the result is 180c180 means that we would have to copy line 180 from file1 to line 180 in file2 to make file 2 be like file 1.
  • fdupes: 1MB, # pacman -S fdupes :: cleans ridiculous amounts of duplicate files prior or after backups.
  • htop: 1MB # pacman -S htop :: color informative version of top
  • jmtpfs: 1MB, $ yay -S jmtpfs :: support to plug move files on and off an Android.
  • lshw: 1MB, # pacman -S lshw :: provide info on MB components better than lspci.
  • * nano: 1MB, # pacman -S nano :: light text editor
  • strace: 1MB, # pacman -S strace :: typical: $ strace > catchit.txt 2>&1 [command]
  • * usbutils: 1MB, # pacman -S usbutils :: lsusb, some others
  • archiving utilities. In addition to xarchiver, install all of the various formats or you're just going to get pissed on the occasion when you need a manual or something and have to update your entire system just to get one zip library. Among these are bzip, qzip, lrzip, lz4, lzip, lzop, xz, zstd, zip, unzip, p7zip, and unarchiver, among others.
  • yt-dlp: 1MB, $ yay -S yt-dlp :: seems lately to work more reliably than the parent youtube-dl from which it's forked.

coding

With the advent of Google Collab (essentially Google version of Jupyter notebook but with TensorFlow and so on for deep), this seems less important, however we still occasionally need a sane offline environment.

  • pipenv: 3MB, # pacman -S pipenv :: critical so can do projects (eg TTS) requiring special pip updates and so on without hurting the OS install
  • geany: 1MB, # pacman -S geany :: coding editor.
  • umlet: 1MB, $ yay -S umlet :: a light UML editor. Filetype UXF
  • rclone: 2MB, # pacman -S rclone :: encrypted cloud backups. GUI rclone-browser

documents

LaTeX needs to be installed from, eg, TexLive in a separate directory in ~ somewhere.

  • cups: 12MB, # pacman -S cups, necessary evil if printing. Get the PPD files from the AUR.
  • evince: 14MB, # pacman -S evince basic PDF reading. This comes with the price of gvfs, so another option is okular, which doesn't have the problem of gvfs.
  • xournalpp: 4MB, # pacman -S xournalpp. editing and creating PDF with Huion
  • xsane: 5MB, # pacman -S xsane scanning documents

A problematic aside is what to do with old DOC and DOCX documents possibly on one's system. LibreOffice can do the conversion with a command line , but a person has to install LibreOffice writer (300MB) to get this one feature.

# pacman -S libreoffice-fresh
$ lowriter --convert-to pdf somefile.doc

math/stats

  • PSPP: $ yay -S pspp. GNU version of SPSS. Does most of the functions. GUI: psppire
  • gretl: $ yay -S gretl. Econometrics
  • octave: # pacman -S octave GNU version of MatLab
  • RStudio: R-specific IDE

media

playback

  • vlc: 100MB, # pacman -S vlc :: necessary for speed variance, some obscure filetypes. Also plays playlists and reaches to non-drm streams.
  • xplayer: 7MB, 15 min compilation $ yay -S xplayer :: much lighter than anything else, plays clean, loops.
  • libdvdcss (backup) 8MB # pacman -S libdvdcss :: any kind of backup off a DVD (eg. an old Newhart episode on a 2000 DVD) requires this. Unintuitive errors result without this. Possibly also consider $ yay -S vobcopy, which will decrypt as it copies it over.
  • pipelight, widevine - TBD. support for DRM protected media a la silverlight

creation & editing

  • audacity (sound) 20MB # pacman -S audacity :: necessary for voice recording to view levels in real time. uses PortAudio
  • ffmpeg (sound, video) CLI 20MB # pacman -S ffmpeg:: screencast audio and video capture. ffplay to precheck video.
  • flowblade (video) 20MB $ yay -S flowblade:: does cross-fades of multiple files far too complex in ffmpeg. Open a project, then import MP4's. Avidemux no good for cross-fades. Openshot crashes. Pitivi crashes. However, all of the Pitivi dependencies.... 
    ... are also good for Flowblade. Install them before getting Flowblade off the AUR. Of these, the only critical item is python-cairo. The AUR Flowblade install does not always check for python-cairo and Flowblade will no-start with errors without python-cairo.
  • gimp (JPG, PNG) 20MB # pacman -S gimp:: Swiss army knife
  • goldwave.exe (WAV) 5MB 1 hr due to WINE installation. This 90's app still the easiest and most thorough for polishing sound. See Wine configuration vid below.
  • mlt (video)3MB # pacman -S mlt:: needed for melt command as as well as Flolwblade. Be sure to add # pacman -S rubbberband if using melt commands. melt FAQ.
  • obs-studio (video) 20 MB # pacman -S obs-studio :: mixing media on the fly if want live productions and saving to file.
  • shotcut (slideshow) 20 MB # pacman -S shotcut :: supposedly this is good for making slideshows though I have never tried it. This would be the only reason to install as it's (per Linux usual) worthless for cross-fades (can't adjust overlap consistently). MLT-based.

Wine Configuration (19:58) Chris Titus Tech, 2019. WINE converts Windows system calls to POSIX system calls. Make a bottle for every Windows app.

network

  • * dhcpcd: 3MB, # pacman -S dhcpcd :: important to add before leaving chroot or no internet after post-install reboot. Also disable it in systemd or possible boot hangs.
  • umlet: 1MB, $ yay -S umlet :: a light UML editor. Filetype UXF
  • * ntp: 1MB, # pacman -S ntp :: typical: # ntpdate pool.ntp.org
  • rclone: 2MB, # pacman -S rclone :: encrypted cloud backups. GUI rclone-browser
  • * wpa-supplicant: 1MB, # pacman -S wpa-supplicant ::

graphic

Wine: Google SketchUp. The old
Wine: ConceptDraw ($99)
Wine: VideoMeld64: fonts-corefonts,tahoma DLL's-none,

audio

Wine: GoldWave: fonts-arial,corefonts,tahoma DLL's-none, however errors on ntdll

safety

  • glasswire :: monitor net usage
  • zoneminder :: security camera management

safety - security devices, rules

Our phones and yubico keys are security devices. A cell phone's primary function is a security device, however its communication functions have been conveniently conflated and incorporated. These should be separated of course, and a security device provided for free by the government, since they are the ones who access and benefit from these functions. The phone itself should return to a secure, non-traceable way (unless search warrant) to communicate. Obviously this would inconvenience security agencies, and the collaboration aspects of government and immensely profitable communication agencies. So it will never happen. That is to say, it will happen just as soon as education is reformed for national benefit.

TLDR: These are security devices so that 1) PAM is involved, therefore 2) Udev rules must be written or they will not even be detected by the kernel in lsusb

phone example

  1. #pacman
  2. # nano /etc/udev/rules.d/90-android-tethering.rules
    # Execute pairing program when appropriate
    ACTION=="add|remove", SUBSYSTEM=="net", ATTR{idVendor}=="22d9", ATTR{idProduct}=="276a", ENV{ID_USB_DRIVER}=="rndis_host", SYMLINK+="android"
  3. # udevadm control --reload

yubico example

It's arguably worthwhile to know the model of one's yubikey -- there are perhaps 40 versions. Let's take an older one, though still FIPS compliant. Nowadays we'd want one that's FIDO compliant. The problem is we can't use these FIDO compliant ones on older computers that only have USB-A ports, so it's good to have an older FIPS key and a newer FIDO key. They of course make one to use with a phone as well. When they stop working, it's a PITA to determine. Use of udevadm monitor was my friend. Then I bout a $50 Yubikey 5. I think the old one was UV sensitive somehow. After it had been in the heat it stopped working.

But back to our rule configuration. According to their website...

  1. # pacman -S yubico pam
  2. # nano /etc/udev/rules.d/70-u2f.rules
    # Execute pairing program when appropriate
    ACTION=="add|remove", SUBSYSTEM=="net", ATTR{idVendor}=="22d9", ATTR{idProduct}=="276a", ENV{ID_USB_DRIVER}=="rndis_host", SYMLINK+="android"
  3. # udevadm control --reload

Tuesday, June 9, 2020

system - server - hosting

We want a system for learning management (LMS), and another for general usage. I like the Moodle LMS and Nextcloud. The problem is that, for years, both of these should be done locally (VPN), you can't really webface them. New solutions are making it possible to do both. I've previously had webhosting, and I think that's been part of the problem. This time around I want to do a VPS. I would still put Nextcloud on a VPN, but I think Moodle can reasonably be done on a VPS at this point with TOTP. So we can host Moodle on Google, but the question is which Tech Stack (see below). The idea is there re 3 layers: the hosting (Google), the http server (Apache), and the system (Moodle, NextCloud).

  • VPS - Virtual Private Server. Cloud server. Google, UpCloud
  • VPN - Virtual Private Network. Home server. Unlimited storage, only limited by HDD space. I am uninterested in the typical web usage of VPN's for anonymity and so on. These are mostly useless (see vid from Wolfgang's Channel below). Thinking here of the much more prudent usage of a home network for a VPN. It's possible to make it web-facing also, but this should not be done without 2FA and SSL.
  • Backup Critical files need this. Probably anything paper that's irreplaceable, eg, DD214, grades, etc. This shouldn't need to be more than about 1-5 GB anyway, but critical. Chris Titus uses BackBlaze. BackBlaze however relies on Duplicity, which in turn relies upon the dreaded gvfs, one of the top 5 no-no items (pulse audio, gvfs, microsoft, oracle, adobe). Use some other with rclone, rsync, remmina, cron.

plan

Current A-Plus costs: $5 month x 2 sites ($120) + annual 2 x domain w/privacy ($30), one site only MySQL.

  1. DNS - Google ($12 yr x 2 incl.privacy)
  2. rclone some criticals to Drive
  3. Moodle VPS on Google LXC
    • $ yay -S google-cloud-sdk 282MB
    • go to Google Cloud and provide credit card
    • follow Chris Titus' instructions in video below

    Host on Google (30:32) Chris Titus Tech, 2019. Do an inexpensive, shared kernel setup. Uses Ubuntu server and Wordpress in this case.
    Moodle 3.5 Install (22:47) A. Hasbiyatmoko, 2018. Soundless. Steps through every basic setup feature. Ubuntu 18.04 server.

  4. Nextcloud VPS on Skysilk ($60)

1. transfer DNS to Google

Chatted with old provider and obtained the EPP's for both domains, began registration in the new domain. Once these are established, we'll have to change the A-records, and pehaps "@" and "C" records to point to current hosting. Each possible VPS provider handles their DNS in different ways. Some providers manage the entire process under the hood, at others a person must manually make any changes to their A-records.

Rsync Backup on Linux (9:19) Chris Titus Tech, 2019. Great rundown plus excellent comments below.
New DNS Update (7:18) Without Code, 2018. Proprietary, but a transparent example of what is involved in the process.

server blend

Nextcloud is not an actual server itself, the underlying server should be something like Apache or Nginx. Nextcloud then overlays these and serves files via the server underlying it. The logins and so forth are accomplished in Nextcloud in the same way we used to do so with, eg. Joomla or Wordpress (optimized for blogs).

Nextcloud: Setting Up Your Server (17:43) Chris Titus Tech, 2019. Uses Ubuntu as underlying server on (sponsored) Xen or Upcloud. Rule of thumb $0.10 month per GB, eg $5 for 50G.
What are Snaps for Linux (4:47) quidsup, 2018. These are the apps that are installable across distros.

2. existing storage for backup

We can use free storage such as Drive or Dropbox to backup data. They key is it should be encrypted on these data mining, big tech servers.

RClone encryption (10:21) Tyler, 2017. Methods to encypt with rclone. Also good idea to download rclone-browser, for an easy GUI.
Rsync Backup on Linux (9:19) Chris Titus Tech, 2019. Great rundown plus excellent comments below.
Using Cloud Storage (22:55) Chris Titus Tech, 2019. Easy ways to encrypt before dropping into Google Drive, etc. (sponsor:Skysilk)

choosing a VPS

One can of course select Google, but what virtualization do they typically employ? Skysilk uses LXC containers via ProxMox.

Rsync Backup on Linux (9:19) Chris Titus Tech, 2019. Great rundown plus excellent comments below.
Using Cloud Storage (7:31) Wolfgang's Channel, 2019. Be sure to pick a provider that uses Xen or KVM, rather than OpenVz-based virtual machines.

tech stack

I used to use a LAMP stack, but I am trying to avoid MySQL (proprietary RDBMS), and use PostgreSQL (OODBMS), as a minimum update (LAPP), and have looked at some other stuff (see below). I may try a PERN stack if I can get it going with Moodle. Post

Various Tech Stacks (48:25) RealToughCandy, 2020. Decent rundown plus large number of comments below. Narrator skews "random with passion" over "methodical presentation", but useful. PostgreSQL around 38:00.
Using Arch as Server (33:11) LearnLinuxTV, 2019. He's running on Linode (sponsor), but the basics the same anywhere. Arch is rolling, but just keep it as the OS for one app.