Sunday, August 30, 2015

python and sqlite

A previous post discussed the huge (250MB) installation of PostgreSQL for something as robust as a LAMP. However, suppose I just need a small database to keep track of business cards or some such?


Unlike PostgreSQL, SQLite has no server: everything is contained in a local "db" file you create. You can easily back it up by just backing up the file. To "connect" to the database, just go in the directory with that database and find the database, let's say it's "sample.db". Then...
$ sqlite3 sample.db
... and just do your business. Once complete, it's ".exit". Much easier for this kind of thing is a simple GUI, like sqliteman (Qt based).

Python access - APSW

But in our Python app, we'll want to access this database directly from the app, so we need a class of Python commands. If we want we can use SQLite commands and just import SQLite into the our code...
import os,sys,time
import sqlite3
... and use SQL language, intermixed with Python.

But if we want a wrapper with a set of Python classes, the easiest Arch module to install is APSW (# pacman -S apsw) which is technically a wrapper, not a module, but still provides a class for interacting with SQLite databases.

Once we have that, we can start our code with, say...
import os,sys,time
import apsw
...and and go on from there. However, I don't need any SQLite wrappers for the level of programming that I do --- I just write the database related commands in SQL. It's not that hard.

Python application

Remember that standalone Python programs are relatively simple to accomplish in Linux, but it depends on whether one compiles against a Python release and set of libs, or attempts to include these all in the package. Since most of what you run on your machine are scripts, like Bash scripts, the only thing you need to worry about among Python releases is what modules are natively available. The way to get to this is to first determine the highest version of Python on your system, and then see if you have modules you need to run your scripts.


Suppose I'm building a script that I want to create a GTK window for (or Qt, but lets use GTK in this example), and so I've got gtk3 and pygtk installed. I'm going to check the version of Python and then list its modules and see if it has gtk and pygtk modules.
$ python --version
Python 3.4.3
$ python3
Python 3.4.3 (default, Mar 25 2015, 17:13:50)
[GCC 4.9.2 20150304 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')
[all modules are listed]
>>> exit()
I check the modules list and note gtk and pygtk are in the Python 2 modules list, but are not in the Python 3 modules list. Some Googling shows that Python 3 has a compatibility module, pygtkcompat, which emulates pygtk. All I need to do then is import pygtkcompat and configure it. Following that, I treat my code as gtk and pygtk existed. For example, the script might begin...
#! /usr/bin/python3

import os, sys, time
import pygtkcompat

# enable "gobject" and "glib" modules
pygtkcompat.enable_gtk(version='3.0') #matches shebang
import gtk
import glib
...and then whatever code came subsequently. Normal GTK "WINDOW" commands are available for example.

managers: system, session, window, volume, preference (and some use a display)

Links: using systemd as a session manager ::
I miss the SysV days when system, session, window (eg. "compositing"), volume, preference, and display managers were all cleanly defined. In fact, we never even needed the preference manager, gconf, because configurations were done with simple text files for each app. I'm just a regular user, and it takes me a lot of work these days to keep these managers separate or find their configuration files. They also seem to intertwine on occasion, requiring detangling time, as well.

Display Manager Disclaimer

Logins, startx, and shutdown are the only terminal commands needed in a normal session. Display Managers are only needed to make these more aesthetic. Accordingly, I haven't used a display manager in ten years. Why have an aesthetic login if it costs memory and potential fails? Second, not using it removes one variable from the kludge of 6 managers just listed above. Display Managers inadvertently obscure any start-up fails by keeping text log access out of sight. That's a problem during an install. In fact, during installs, I obtain a solid Runlevel 2 before downloading any X elements.

Session manager sussed-out

I attempted to tweak Evince's default zoom level the other day without gvfs (gnome's volume manager) installed. Sadly, Evince does not use old-skool config files; users have to set "schema keys", which are probably xml files. At any rate, one starts at the top, the correct schema, and then one modifies its key: find the schema, then its possible keys. To shorten the codeblock below, here were the two Evince-related schemas using the list-schema flag :
$ gsettings list-schemas
And then the keys for those schemas.
$ gsettings list-keys org.gnome.Evince

$ gsettings list-keys org.gnome.Evince.Default
It appears the "zoom" key is the one to set, which is in schema Evince.Default.
$ gsettings set org.gnome.Evince.Default zoom 1.0
$ gsettings get org.gnome.Evince.Default zoom
We can see the zoom was set correctly (to 100%). Now let's run Evince:
$ evince
GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.gnome.SessionManager was not provided by any .service files
Evince starts, but spawns the error above and only the default zoom factor, instead of the 100% zoom I entered.

Was gnome-session not running, sharing with systemd, or what? A simple setting can often means days of frustration in Linux. If I recall correctly, it's further the case, that gnome-session can only be started after dealing with PAM, like via gdm-password. It's unlikely any gnome sessions are running.
# ps -A |grep -i g
614 ? 00:00:00 gconfd-2
3602 pts/2 00:00:00 grep
31658 ? 00:00:00 gnome-pty-helpe
Now it's clear: no gnome-session is running. The gsetting commands were accepted. That is because, for the settings to be entered, only gconfd, gnome's preference manager, had to be running. However, there was no way to implement those preferences, because the session manager, gnome-session, was not running: an error message was generated. Meanwhile, Arch recommends against any gconf schemas, and notes they will soon be replaced with "dconf". Currently then, Evince is probably caught in the middle of these. We really have no way to configure Evince without getting a session manager going, which in turn probably requires the dreaded gvfs by the way.


In spite of the immense KDE libraries necessary for Okular, I have to look. I mean if I can just set preferences at this point, I'd be happy.

Saturday, August 15, 2015

[solved] first run of zotero back-up

I had a chance to recently attempt my first Zotero back-up and resoration. I have the stand-alone version, which I then link to a few browsers with browser plug-ins (see bottom of page). In other words, Zotero installs as a regular application, available on or offline. Zotero browser-plugins (typically XPI) install into the desired browser independent of whether Zotero is installed on one's system at all.

other bibliography/citation-related notes

BibTex tutorial :: exporting cite keys from Zotero :: helping cite keys match database


From what I could tell from Zotero's instructions, all I would need to do is backup my entire Zotero directory, and then put it in the new drive. Helpful, but experience showed it could be broken down more explicitly:
  1. Back-up the Zotero database directory (usually a few hundred MB; mine fits on a CD).
  2. Install a blank version of Zotero, on the new drive. Be sure Zotero is stable (open and close it a couple times).
  3. Add a browser connector, if you want or need one (optional).
  4. Take the backed-up data directory and overwrite the fresh install's data directory.
  5. Open Zotero and all backed-up citations should be there.
  6. Associated back-up issue: suppose one has a folder with several Ebooks or PDF's on some subject. Zotero can apparently store local HDD links to these documents, as well as metadata about the them. These links will also be in the Zotero database, just like web links, and using links uses less space in the database than the full documents would use. However, after backing up Zotero on a new install, you'll also need to duplicate the directory structure where these documents are kept. In this way, Zotero's links will point to the documents.


Initially, I made a mistake. I backed-up the Zotero directory, burning it onto a DVD with other data. This directory and all files within it were suspiciously only about 10MB. Eventually I determined I had only backed-up the application directory. The data directory is what should be backed-up. But the data directory is not in plain sight within the directory structure. To find the data directory, I searched for one of its files, zotero.sqlite:
$ find -name zotero.sqlite
This is the correct Zotero directory to back-up, the data directory. That is, ~.zotero/zotero/[hashnumber].default/zotero/*. This directory was a few hundred MB. As for the application directory, why back it up at all? Download and install the latest version of Zotero instead.


First, install the Zotero app, either the old one or the latest version. If it's a new install it will create a new folder in your home directory: ~.zotero/zotero/[newhashnumber].default/zotero/. Open and close Zotero a couple times to be sure the install is clean. Then just take the old data directory (you backed-it up above) and overwrite the new data directory. Your new structure will be something like ~.zotero/zotero/[newhashnumber].default/zotero/[old backed-up data]. The next time I opened Zotero, my backed-up citations were there.

Connect to browsers (optional)

You can manually enter Zotero entries. Browser plug-ins make it easier for some online sources. But after a back-up, the connector between stand-alone Zotero and its browser plug-ins, may or may not be broken. Once the standalone is working and up-to-date, start Zotero and then go to the links for the connector that matches the browser you like. Here's a couple of them.

Chromium Zotero Connector

Opera Zotero Connector

screencast and tablet notes

These are a few mathematics screencast notes.

Huion 1060 Pro +

A surface to write equations. Against the cost of a Wacom, or the price of a full-blown pressure sensitive tablet PC, I think I saved money with this $65 Huion. It plugs in works with Xournal out of the box. I haven't yet played with programmable keys on its side -- I'll do a separate post downstream. Meanwhile, Xournal is friendly with it, and no problems with basic functions.

The second cable charges the pen, and then there are extra pen nibs in the bottom of that penholder. You can see those programmable keys to the left side of the unit and the pen has two programmable keys. The penholder keeps the nib from pushing against anything -- that uses battery juice. I put mine, nib up, in a jar. A light touch does the trick when writing.

Ardesia - Composite Manager requirement

Ardesia seemed worthy of checking-out against Xournal. Ardesia's dependencies install OK, using pacman, and then Ardesia itself is found using yaourt. All went well, but first run revealed:
$ ardesia
In order to run Ardesia you need to enable a composite manager
Of course, I don't want to install or enable any compositing windows managers, who does? More importantly I don't understand the need for one here. For example, Xournal functions just fine without a composite manager, and I can't see the need for multiple windows in a drawing app the simple ways I will use it. Perhaps Ardesia's coders somehow expected graphics-card intensive usage for superstar effects, or perhaps they're lazy, but I'm not going there. Xournal remains my solution.

VLC recording

As described here, VLC has the capacity to capture, not simply play. I mostly use RecordMyDesktop for this, but have taken to at least experimenting with VLC, for the simple fact of rendering. RecordMyDesktop only produces OGV's and, for editing, I'm pretty much limited to OpenShot. But OpenShot is extremely CPU intensive when rendering OGV's. VLC seems to provide MP4 formats, which are much easier when rendering the final edit.

Recordmydesktop - XGrabKey error

$ recordmydesktop
X Error: BadAccess (attempt to access private resource denied)
Bad Access on XGrabKey.
This appears to be an audio permission or configuration issue, since video was perfect. Usually when sound is a problem, PulseAudio has surreptitiously been enabled by some application. Check with "$ arecord -L", which lists capture devices. If PulseAudio is anywhere on that list, get busy killing it to the last byte. I go over this in a previous post, and
there's also a great link here. But be sure that PulseAudio is well and truly dead on your system.

quick edit

If you don't have to cut stuff out of the middle, just maybe from the start or end, you can edit using ffmpeg, and whatever your audio editor for WAV's is. Just clean it up, put it back together, and then snip the ends. I use mencoder for the snipping step: simpler command than ffmpeg.
$ ffmpeg -i screencast.ogv -vn -ar 44100 -ac 2 sound.wav
[clean your sound]
$ ffmpeg -i screencast.ogv -vcodec libxvid -b:v 500k -an video.avi
[convert ogg to xvid]
$ ffmpeg -i video.avi -i sound.wav -acodec libmp3lame -ar 22050 -ab 192k -ac 2 -vol 330 -vcodec copy -b:v 500k screencast.avi
[reassemble sound and video]
With this all clean, just trim whatever seconds off. In this case, I'm trimming the first 10 seconds off:
$ mencoder -ss 10 -oac copy -ovc copy screencast.avi -o screencastout.avi
YouTube is a 16:9 format, incidentally, another story.

OpenShot (300MB)

Ridiculous size, but it's the most straightforward timeline editor I've seen.