rclone details

In a prior post, I'd found that using rclone to upload RESTful (rclone uses REST, not SOAP) data had become more complex -- by at least three steps -- than two foundational videos from 2017:

1. Rclone basics   (8:30) Tyler, 2017.
2. Rclone encrypted   (10:21) Tyler, 2017.

These videos are still worthy for concepts, but additional steps --choices actually -- must be navigated for both encrypted and unencrypted storage, whichever one desires. Thus, a second post. Unlike signing in and out of one's various Google and OneDrive accounts, all are accessed from a single rclone client. Rclone is written in Go (500Mb), so that immense dependency must be installed.

across devices

To install rclone on multiple devices, including one's Android phone (RCX), save one's ~/.config/rclone/rclone.config. For each installed client, simply duplicate this file and one can duplicate the features of the original installation. If one has encryption, losing this file would be very bad.

deleted configurations

1. ~/.config/rclone/rclone.config (client). If this file is lost, duplicate it from another device. If lost entirely, access must be re-established entirely from scratch, and the encrypted files will be lost permanently.
2. scope (google) Google requires authentication for access details for which Google keeps. Documentation is difficult to find, other than the OAuth info in the prior sentence. It appears that users cannot directly edit any of the 11 access scopes (files) defined, but rather only through a Google dialog screen. When installing rclone, 5 of the 11 scopes are available, for which I typically like "drive.file".
   
   
   

command usage

For simplest use, to the root directory...

$ rclone copy freedom.txt mygoogleserv:/

Not all commands work on all servers, so use...

$ rclone help

instead of...

$ rclone --help

The former will display only those commands on the installed version of rclone. The latter shows all commands, but not every compilation has these.

$ rclone about mygoogleserv:
Total: 15G
Used: 10.855k
Free: 14.961G
Trashed: 0
Other: 40.264M

Of course, there's also the GUI, rclone-browser.

encryption notes

Rclone documentation notes strong encryption, especially if salt is used. Minimally, we're talking 256-bit. Of course governments can read it, but what can't they read?

• unencrypted accounts must be established first. Encryption is an additional feature superimposed onto unencrypted accounts.
• remember the names of uploaded encrypted files; even the names of files are encrypted on the server and the original filename is necessary for download.
• keep the same encryption password on all devices on which rclone is installed.

glossary

• application data folder (Google) a hidden folder in one's Drive (not in one's PC). The folder cannot be accessed directly via a web browser, but can be accessed from authorized (eg OAth) apps, eg rsync. The folder holds "scope" information for file permissions.
• authorization (OAuth, JWT, OpenID) protocols for using a third party REST app (rclone) to move files in and out of a cloud server (Google, AWS, Azure, Oracle), there's an authorization process between them, even though you are authenticated in both.
  What is OAuth (10:56) Java Brains, 2019.
  What is JWT (10:34) Bitfumes, 2018.
• scope (Google). the permissions granted inside Drive to RESTful data uploaded by users using, eg, rclone.
• REST Representational State Transfer API for server to client data transfer. Wikipedia notes this as an industry term and not a copyrighted concept by Oracle or Google. It refers to data exchanged by user-authorized third party apps between applications or databases and applications. This is as opposed to data directly entered by users, or data that is not authorized by users between servers.

  REST API concepts and examples (8:52) WebConcepts, 2014. Conceptually sound on this HTTP API, even though dated with respect to applications. Around 7:00 covers OAuth comprehensibly.

• SOAP Simple Object Access Protocol. This is the older API for server to client data transfer.

  SOAP v. REST API (2:34) SmartBear, 2017. Very quick comparison.

---

Google 15GB

Users can personally upload and save files in Google Drive through their browser as we all know. However, Google treats rclone as a third party app doing a RESTful transfer and uses OAth to authorize it. Additional hidden files are created by Google and placed into one's Drive account to limit or control the process.
Within that process, there are two ways to rclone with Google Drive, slower or faster. The faster method requires Google Cloud services (with credit card) and a ClientID (personal API key). The slower way uses rclone's generic API connection.

1. Slower uploads

Faster to set-up, but slower uploads. Users regularly backing-up only a few MB of files can use this to avoid set-up hassles. It bypasses the Cloud Services API, and uses the built in rclone ID to upload as directed

1. $ rclone config

   ... and just accept all defaults. For scope access, I chose option "3", which gives control over whatever's uploaded.
2. verify function by uploading a sample file and by looking in ~/.config/rclone/rclone-config to see that the entry looks sane

2. Faster uploads

This method requires a lengthier set-up but, once configured, rclone transfers files more quickly than the generic method above. Users need a credit card for a Google Cloud Services account, which in turn supplies them with a ClientID or API key for rclone or other 3rd party access into Drive.

1. get a Google email
2. sign-up for Google Cloud services
3. register one's project "app". In this case it's just rclone) with the Google API development team
4. waiting for their approval -- up to 2 weeks
5. receiving a Client ID and Client Secret which allow faster uploading and downloading through one's Drive account

These two videos move very quickly however they have the preferred Client ID and Client Secret method that supposedly speeds the process over the built-in ID's.

Rclone with Google API (6:38) Seedit4me, 2020. The first four minutes cover creating a remote and the 5 steps in creating the Client ID and Secret.
Get Client ID and Secret (7:29) DashSpan.me, 2020. Download and watch at 40% speed.

OneDrive 2GB

This primer is probably the best for OneDrive, however it also applies to many of the other providers

metadata and scope

These are hidden files within one's Google Drive. is is part of the Google Drive API v.3, which is what rclone uses to connect and transfer files. In particular, you will want to know about the Application Data Folder

Google API v3 usage (5:28) EVERYDAY BE CODING, 2017.
Get Client ID and Secret (7:29) DashSpan.me, 2020. Download and watch at 40% speed.
RESTFUL resources and OAuth (55:49) Oracle Developers, 2017.