Saturday, February 27, 2010

more data fun - PaperHater

Links: Common MySQL Commands  WizzyWeb   VistA database

I've worked on going paperless since about 2002. It's a maddeningly dull, time consuming affair if one is not a computer programmer capable of designing a database. I am not. Just a person, a computer user. I call this project "PaperHater".

schema

critical, tax, code, projects, auto,

categories

Can it be done without a database? We have the Dewey Decimal (since 1876) and Library of Congress system for books and mags (ISBN, ISSN), so we can use their system to an extent, possibly for our library and citations grouping. But at home I have receipts from vehicles, medical lab reports, HS diploma, and on. This is also not exactly business style records management, since the categories include personal areas. Probably benefits from electronic records management (ISO 15489). File naming conventions, folder conventions and metadata conventions. Yet still might end up requiring a database.

file naming conventions (10:00) Simpletivity, 2018. Ad first 1:36. Uses 3 part naming, succinctly described. Probably best at .75 speed. Comments excellent.
Ffmpeg advanced techniques pt 2 (1:23:57) Nicholas Andre 2013. Windows-based lecture, but clear thinking fellow gives good context for what we're after. Backgrounder. Corning (NY) Community College course.

clutter issues

By mid-2002, I was regularly scanning. Everything from bank statements to letters from my deceased grandmother. I learned that the number and variety of the generated files grew rapidly and could not easily be organized; I was spending significant time categorizing and renaming the files using customary file-naming and folder-naming conventions. In spite of this effort, and with only a couple thousand files (albeit growing in number), I had to admit to myself that I was wasting more time arranging and not finding files than I previously had spent hassling with the inconvenience of paper documents. What good was the computer accomplishing for me?

A proper database sitting between thousands of stored electronic files, and the user attempting to retrieve one of those files, appeared to make all the difference. However, since an immediate solution looked neither likely nor affordable at that time, I reverted to storing documents in bankers boxes. I did this for several years, but with began to slowly acquaint myself with database programs.

By 2009, I had become routinely frustrated with paper accumulation again. Stuck between a computer and a pile of papers, there appeared to be no easy way out. Since early 2009, alternating Saturdays have been spent reading and learning installation processes, webserver configurations, and the like -- LAMP stuff. It seems to have paid off. I'm roughly at the 2/3 mark on this project. Over the next several months, I intend to add an entry here and there about PaperHater's progress to summarize for myself and to possibly help other home users who might be attempting to design and implement something similar on their own system(s). Good luck to all of us. It would be helpful if expert organizations with CMS-type experience could release reasonably priced PC/Mac database solutions instead of gouging us.

step 1

Design the database. I mostly use localhost for this, since it's fast and secure, but sometimes run things on the cloud server. In other words, I spent the many weekends required to learn how to configure reliable LAMPs on both my localhost and server site, and to have them running with the permissions and tweaks (think, eg. php.ini, config.inc.php, etc) I wanted. Following that portion came additional reading, head-scratching, and back-of-envelope sketches; then downloading and installing working applications. Finally, using phpMyAdmin and MySQL commands, I reviewed and modified table structures until I could determine a desirable schema. I also worked on php scripts. Eventually, I had a set of working databases, some copied to .sql scripts, and some partially completed php scripts. That's mostly where it stands now, but I'll move on to describe what's going to come next.

step 2

Copy the structure of a localhost or online database I've designed and tested and want to use for interaction. We put this into an .sql script.
(localhost)$ mysqldump --no-data database >/home/foo/database_template.sql

(online)$ mysqldump -h whateversite.com -uuser -ppassword --no-data -D database >database_template.sql

Then go to whatever web address I keep databases on, create a new database there (thanks to this site), and give it the structure we want from the .sql script.

$ mysql -h whateversite.com -uuser -ppassword
mysql> CREATE DATABASE newdatabase;
mysql> USE newdatabase;
mysql> SOURCE /home/foo/database_template.sql;
mysql> quit;


The new database is now ready to accept data-entry and to run queries.

step 3

Upload some php scripts. I've written a large percentage of the interfacing php scripts but, today, I learned about WizzyWeb, a $99 product which could help tune my scripts or create similar ones quickly. This seems like a very reasonable price if it does what's advertised. That is, if it saves me two hours, I've made-back my money.

step 4

Enjoy. But remember: the PHP, Postgresql/Mysql,Apache, etc. are all on the server side. Users still must create some Javascript additions for their served pages to influence the browser client. However, it is possible to write such skillful PHP code on the server side, that a client needs very little browser-side code to have a nice experience on the site. And remember that server-side is usually more secure since it can't be hacked without hacking the site, not just a webpage. Here's a list of some pre-built server-side PHP "UI Frameworks" written in PHP (as opposed to Java or .Net). Edit: Also, Sitepoint's poll results from 2015 for PHP frameworks.

PHP vs Python vs Java vs C#

Java no way, since the 2011 Oracle purchase. C# no way, since it's more or less locked to Microsoft's .NET framework. But Python now has modules which allow it to work on a server (of course it can still be used to code stand-alone programs too). PHP is designed to work on a webserver inherently, and so is more naturally integrated into HTML, but if one wants to be consistent, they could just do everything with (open-source) Python, using its server side add-on modules (scroll down to "Compared as Web Development Frameworks"). Even low-end ISP's like APlus, which has a terrible lag implementing anything, has PHP4/5 and Python capacity.

Thursday, February 25, 2010

zenwalk - corrupted disk

Links: helpful command explanations

A couple of Christmases ago I installed Zenwalk 6.0 on one of my parents' HP systems. It remained mostly reliable until this week, when it froze and subsequently stopped booting properly. Ultimately, we found that their system contained one or more inode conflicts. We fixed these cross-links with two commands and a great number, roughly 1,000, confirmation keystrokes. All this via an hour-long telephone call. My mother is a patient and persistent soul.

Part one
My folks told me they recalled being online arranging bookmarks in Iceweasel (Zenwalk's rebranded Mozilla-Firefox). While arranging, the system froze entirely, apparently remaining unresponsive to the last-resort X11 Ctrl+Backspace exit. Mom said she next attempted a hard reboot but this also failed. During the attempt, error messages describing cross-indexed inodes and a corrupt superblock were scrolling. This was no doubt fsck scanning /dev/sda1. Following fsck, the script eventually exited to a single-user root login. No action was taken at the root login, and the system eventually rebooted itself and again initiated fsck. And so on. Then my phone rang ;)

Part two
It appeared the most direct route was to reboot the system and let it exit to the point where it allowed the single-user root login. From that point, we could enter commands directly, such as to locate back-up copies of the superblock on /dev/sda1. With a back-up copy located, we could run a clean fsck and resolve any doubly referenced inodes. Accordingly, the first step was to retrieve a list of the backup superblocks.

# dumpe2fs /dev/sda1 | grep superblock

This provided us with a list of about 12 back-up superblock locations. The first of these was "32768".

Part three
Used the back-up superblock to reestablish order in the system files and clear-up the multiply- referenced inode conflict.

# fsck -b 32768 /dev/sda1

Fsck eventually resolved the inode conflicts and some directory problems. During the process, my mother had to enter confirming "yes"'s nearly 1,000 times.

success
The problem was solved for the time being. Since I could not access the system, it remains unclear whether the problem was related to Iceweasel, to a possible hard drive problem, or to some combination of these.

Wednesday, February 17, 2010

accounting...and php

Links: IonCube loader instructions

Thought I'd try that Nola Pro software and see what it's all about to put something enterprise level on a localhost, or what kind of security holes it opens. Blah blah blah. Along the way, I fell prey to a php problem I was embarrassed to have overlooked, and got to try-out the IonCube code obfuscator w/associated configuration challenges.

Figure 1

Ion Cube's product is a code obsfucator, one that makes it nearly impossible for visitors to determine the underlying code of a website's pages, sort of like Zend. NolaPro requires the IonCube loader as a prerequisite for NolaPro's installation. The IonCube loader is free, the complete IonCube package is not. This business model reminded me of the early PDF era when a free Adobe Reader was provided to read PDF files, but Adobe required that Acrobat be purchased, at a significant price, to produce them. At any rate, I navigated to Ion Cube's main site, then to "Products->Free Loaders for Encoded Files" and downloaded the latest .tgz. Instructions were to untar it and move its entire folder into /var/www/htdocs. The folder appeared to contain a group of libraries.

To install NolaPro itself, I untarred the NolaPro .tgz in my home directory. From there, the instructions were to move the entire folder into /var/www/htdocs. It also suggested full "777" permissions on this folder, which I don't like to do. I made sure Apache was on and pointed my browser to http://localhost/nolapro. From there NolaPro's installer displayed the necessary php.ini settings. Here's a summary of its php complaints.

Figure 2

I thought that mbstring was properly configured in my php.ini, so that was weird, along with the gd thing. It seemed an odd diagnostic at first....

php embarrassment
A check with http://localhost/phpinfo.php soon solved one problem. How could I have overlooked it not loading my php.ini file? Idiot.
Figure 3
I remedied this problem by copying php.ini to /usr/local/lib/php.ini. But, about this time, I also recalled I had initially installed php v.5.28 on my system, and had subsequently upgraded to 5.3. Although it can't be seen in the the photo clip above, http://localhost/phpinfo.php noted version 5.3 was what Apache was reporting. However checking $ php -v yielded the following:
PHP 5.2.8 (cli) (built: Jan 9 2009 16:26:32)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
Nice -- a mixed-up set of installations. Additionally, when I compiled the module last year, I apparently overlooked gd and mbstring support:
'./configure' '--with-apxs2=/usr/sbin/apxs' '--with-pgsql=/usr/local' '--with-mysql=/usr/share' '--with-libxml-dir=/usr/lib' '--with-curl=/usr/bin/curl' '--with-zlib' '--with-gettext' '--with-gdbm' '--enable-inline-optimization' '--enable-track-vars'

Accordingly I went back to my php source folder, did a $ make clean and reconfigured, made, and #made installed the php module. This time, used all of the above and added --with-gd --enable-mbstring into the configure command. Success.

more IonCube
With the php problems resolved, NolaPro was ready to install except for the IonCube loader (Figure 2). IonCube provides a diagnostic script, ioncube-loader-helper.php , but this was displaying a blank page when I attempted to load it. I then copied the IonCube folder to both the nolapro folder and the htdocs folder as possible solution. No change. In the end, ensuring I had loader files corresponding to my processor was the answer. I thought I could run loader files that that noted 64-bit processing on my 32-bit system, and that these loader files would be back-compatible. They weren't. Once I dropped to the 32-bit version, ioncube-loader-helper.php displayed a successful test screen. Just to be certain, I also updated /usr/local/lib/php.ini with

zend_extension = /var/www/htdocs/ioncube/ioncube_loader_lin_5.3.so

I used loader 5.3 which corresponded to my php release; others should change it to the loader for their php release.

snagware
NolaPro was finally ready to install and did so routinely. At first look, NolaPro seems pleasing. It also appears to be a foot in the door to lure users into purchasing premium add-ons. If that's right, it's not freeware, but more like crippleware. I'll take a closer look over the coming week. It was a good experience insofar as I gained additional experience with php and IonCube configurations.

Friday, February 12, 2010

cdrecord/wodim woes

Links: Bug w/wodim info  Firmware info


Apparently there has been a fork in what previously was cdrtools. What's apparently happened is the previously robust cdrecord may have had licensing issues so that what's now called cdrecord is actually an alias for wodim. Wodim appears to be Jörg Schilling's freely licensed forefather to his more developed offspring cdrecord. It seems that, once the licensing problems with cdrecord became evident in 2006, the freely licensed, and more rudimentary, wodim was resurrected and included in cdrkit nee cdrtools.

This came up in an attempt to upgrade firmware on an old Pioneer drive that, for some reason, was only burning at about a 1/10th of its rated speeds. When I ran the reliable and familiar
$ cdrecord -scanbus
just to get a part number, I found results under the command wodim, which caused concern. On a web-facing PC, any time I enter one command and another appears, I'm going to ponder the worst scenario. Ah, but it's just a licensing issue.

But a real concern are any limitations with wodim not present in cdrecord. For example, in the same box that I used to have good results with cdrecord, I get these results with wodim.

$ wodim -scanbus

wodim: No such file or directory.
Cannot open SCSI driver!
For possible targets try 'wodim --devices' or 'wodim -scanbus'.
For possible transport specifiers try 'wodim dev=help'.
For IDE/ATAPI devices configuration, see the file README.ATAPI.setup from
the wodim documentation.


$ find -name README.ATAPI

./usr/doc/cdrkit-1.1.9/READMEs/README.ATAPI

Monday, February 8, 2010

Inspiron 7000 - Linksys WPC54G w/BCM4306

Links: Discussion on 43b module   b43 module installation (doesn't work)   b43 v. ndiswrapper  b43legacy issues  ndiswrapper solution   linksys WPC54G support


This is an install I'm working on with a friend's older Inspiron 7000. We dropped Zenwalk 6.0 (Slackware light) into the system and everything configured out-of-the box, except his PCMCIA Linksys WPC54G (version 1.2) card. This entry is meant as a trail of crumbs for how we solved it.

The legacy Linkys WPC54G card employs a Broadcom 4306 chip. I've never understood the impulse behind closed-source drivers for hardware. There are millions of Linux users; if Broadcom only produces a driver for Microsoft, at least make it open-source. Linux users can then easily design a good driver. They will want to buy Broadcom-based hardware.

Since the Broadcom driver was proprietary, any drivers/modules for it needed to be reverse engineered for Linux with some predictable results. For example, the b43 module described here didn't work. The fwcutter program appeared to properly extract info from bcmwl5.sys and install it into /lib/firmware; the card was detected by the kernel and apparently was semi-configured by the b43 module; but yet the card never fully initialized. This meant it came down to either the b43legacy module or, as a last resort, to using ndiswrapper on the driver coded for MSoft by Broadcom. You know, like back in 2004. But before we capitulated to ndiswrapper or b43legacy, we wanted to try fwcutter again with a more researched approach.

environment
$ uname -r
2.6.28.7

# lspci -vnn
06:00.0 Network controller [0280]: Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller [14e4:4320] (rev 03)
Subsystem: Linksys WPC54G [1737:4320]
Flags: bus master, fast devsel, latency 64, IRQ 11
Memory at 1c000000 (32-bit, non-prefetchable) [size=8K]
Capabilities: [40] Power Management version 2
Kernel driver in use: b43-pci-bridge
Kernel modules: ssb


software
Understanding from above that we're dealing with the "4320" Broadcom chip, the b43 module should have worked previously; b43 is the recommended driver for the 4320 id. I decided to recompile fwcutter, this time making sure I had version "012" of fwcutter and version 4.150.10.5 of Broadcom's proprietary driver.

$ wget http://bu3sch.de/b43/fwcutter/b43-fwcutter-012.tar.bz2
$ wget http://mirror2.openwrt.org/sources/broadcom-wl-4.150.10.5.tar.bz2


success
Untarred and compiled fwcutter -- "$ make". Didn't even have to configure. Then, after untarring the driver, put the driver files in the folder with the fwcutter program and ran:

# ./b43-fwcutter -w /lib/firmware wl_apsta_mimo.o

Rebooted and the card came right up.