Showing posts with label mysql. Show all posts
Showing posts with label mysql. Show all posts

Saturday, February 27, 2010

more data fun - PaperHater

Links: Common MySQL Commands  WizzyWeb   VistA database

I've worked on going paperless since about 2002. It's a maddeningly dull, time consuming affair if one is not a computer programmer capable of designing a database. I am not. Just a person, a computer user. I call this project "PaperHater".

schema

critical, tax, code, projects, auto,

categories

Can it be done without a database? We have the Dewey Decimal (since 1876) and Library of Congress system for books and mags (ISBN, ISSN), so we can use their system to an extent, possibly for our library and citations grouping. But at home I have receipts from vehicles, medical lab reports, HS diploma, and on. This is also not exactly business style records management, since the categories include personal areas. Probably benefits from electronic records management (ISO 15489). File naming conventions, folder conventions and metadata conventions. Yet still might end up requiring a database.

file naming conventions (10:00) Simpletivity, 2018. Ad first 1:36. Uses 3 part naming, succinctly described. Probably best at .75 speed. Comments excellent.
Ffmpeg advanced techniques pt 2 (1:23:57) Nicholas Andre 2013. Windows-based lecture, but clear thinking fellow gives good context for what we're after. Backgrounder. Corning (NY) Community College course.

clutter issues

By mid-2002, I was regularly scanning. Everything from bank statements to letters from my deceased grandmother. I learned that the number and variety of the generated files grew rapidly and could not easily be organized; I was spending significant time categorizing and renaming the files using customary file-naming and folder-naming conventions. In spite of this effort, and with only a couple thousand files (albeit growing in number), I had to admit to myself that I was wasting more time arranging and not finding files than I previously had spent hassling with the inconvenience of paper documents. What good was the computer accomplishing for me?

A proper database sitting between thousands of stored electronic files, and the user attempting to retrieve one of those files, appeared to make all the difference. However, since an immediate solution looked neither likely nor affordable at that time, I reverted to storing documents in bankers boxes. I did this for several years, but with began to slowly acquaint myself with database programs.

By 2009, I had become routinely frustrated with paper accumulation again. Stuck between a computer and a pile of papers, there appeared to be no easy way out. Since early 2009, alternating Saturdays have been spent reading and learning installation processes, webserver configurations, and the like -- LAMP stuff. It seems to have paid off. I'm roughly at the 2/3 mark on this project. Over the next several months, I intend to add an entry here and there about PaperHater's progress to summarize for myself and to possibly help other home users who might be attempting to design and implement something similar on their own system(s). Good luck to all of us. It would be helpful if expert organizations with CMS-type experience could release reasonably priced PC/Mac database solutions instead of gouging us.

step 1

Design the database. I mostly use localhost for this, since it's fast and secure, but sometimes run things on the cloud server. In other words, I spent the many weekends required to learn how to configure reliable LAMPs on both my localhost and server site, and to have them running with the permissions and tweaks (think, eg. php.ini, config.inc.php, etc) I wanted. Following that portion came additional reading, head-scratching, and back-of-envelope sketches; then downloading and installing working applications. Finally, using phpMyAdmin and MySQL commands, I reviewed and modified table structures until I could determine a desirable schema. I also worked on php scripts. Eventually, I had a set of working databases, some copied to .sql scripts, and some partially completed php scripts. That's mostly where it stands now, but I'll move on to describe what's going to come next.

step 2

Copy the structure of a localhost or online database I've designed and tested and want to use for interaction. We put this into an .sql script.
(localhost)$ mysqldump --no-data database >/home/foo/database_template.sql

(online)$ mysqldump -h whateversite.com -uuser -ppassword --no-data -D database >database_template.sql

Then go to whatever web address I keep databases on, create a new database there (thanks to this site), and give it the structure we want from the .sql script.

$ mysql -h whateversite.com -uuser -ppassword
mysql> CREATE DATABASE newdatabase;
mysql> USE newdatabase;
mysql> SOURCE /home/foo/database_template.sql;
mysql> quit;


The new database is now ready to accept data-entry and to run queries.

step 3

Upload some php scripts. I've written a large percentage of the interfacing php scripts but, today, I learned about WizzyWeb, a $99 product which could help tune my scripts or create similar ones quickly. This seems like a very reasonable price if it does what's advertised. That is, if it saves me two hours, I've made-back my money.

step 4

Enjoy. But remember: the PHP, Postgresql/Mysql,Apache, etc. are all on the server side. Users still must create some Javascript additions for their served pages to influence the browser client. However, it is possible to write such skillful PHP code on the server side, that a client needs very little browser-side code to have a nice experience on the site. And remember that server-side is usually more secure since it can't be hacked without hacking the site, not just a webpage. Here's a list of some pre-built server-side PHP "UI Frameworks" written in PHP (as opposed to Java or .Net). Edit: Also, Sitepoint's poll results from 2015 for PHP frameworks.

PHP vs Python vs Java vs C#

Java no way, since the 2011 Oracle purchase. C# no way, since it's more or less locked to Microsoft's .NET framework. But Python now has modules which allow it to work on a server (of course it can still be used to code stand-alone programs too). PHP is designed to work on a webserver inherently, and so is more naturally integrated into HTML, but if one wants to be consistent, they could just do everything with (open-source) Python, using its server side add-on modules (scroll down to "Compared as Web Development Frameworks"). Even low-end ISP's like APlus, which has a terrible lag implementing anything, has PHP4/5 and Python capacity.

Monday, January 18, 2010

MySQL - GUI's - Workbench

links
MySQL Gui's   Common MySQL commands   MySQL Workbench issues   MySQL field types

Website maintenance can be considered in two main halves, the administration of files portion and administration of databases (if any) portion. For file administration, adding and deleting files, I use a secure ftp program and an ssh tunnel to my provider. But what about database administration? If I am given MySQL databases, and if that provider has phpMyAdmin installed, then I use a mix of command line and phpMyAdmin database commands. As a side note, I like PostgreSQL more than MySQL, but most webspace providers only install MySQL.

command line
Making a command-line connection to the database is a good place to start, even if we will use a GUI later. Through the command line, we can easily verify connection is possible, and we can also run MySQL installation scripts, etc.

Let's imagine we had a database at Google that was web-facing w/remote access permissions. This would never happen, but this is the way ISP's often provide database access to you and me. We could connect to our imaginary Google database in the following way:
$ mysql -h google.com -u pietro -D mystuff -pwiggetystop

(note: no space between the "-p" command and the password itself "wiggetystop") So, if I've done this correctly, I'll be logged into my Google database and have a cheery "mysql >" prompt waiting for mysql commands.

phpMyAdmin
By default, many distros that have package management programs install phpMyAdmin into the Apache area. It's understood that Apache is used to parse the php files in phpMyAdmin, but Apache is another annoying level of overhead which also brings on security concerns if used outside of localhost. I only use phpMyAdmin when it's provided on an ISP's server. What is the solution for accessing a database remotely from one's local machine using a GUI? We don't want to pay for Navicat, now do we?

mysql workbench
It turns-out MySQL makes a GUI developer suite called "Workbench", which includes administration, navigation, and design tools. After checking dependencies, I downloaded, compiled, and installed the program, actions which appeared to have proceeded successfully. The start-up command is $ mysql-workbench, and it might have a few options for that command.


The reason this screenshot has no administration is because Workbench couldn't load the administration and nav modules, which left only the design module (pictured). There is a documented bug when compiling in Linux, even with all the Python libraries I could think of installed. The Python program pexpect, was considered helpful in the bug report, so I installed that as well. No improvement. Currently, I only have the design interface, as shown in the image above.

After digging around, filing a bug report (incidentally, they don't even want to answer within the bug forum where they will be useful to all, they direct one instead to their IRC channel where only the current people in the room will see the solution - ridiculous), all of the usual stuff, I dug in deeper with strace. It appears that it's the usual problem: Java, the most ridiculous run-time library since Visual Basic. This garbage occurs from each object MySQL Workbench attempts to retrieve:

**Message: WARNING: MetaClass db.maxdb.Catalog is registered but was not loaded from a XML


This seems to be the Java getDocument call. And there is no way to fix this I know of except to once again spend hours checking every exported path and poorly designed JRE directory search requirement. I've seen this kind of thing in other Java based programs. It's not Java itself that sucks so badly, it's their (apparently) lazily designed afterthought of a Run-Time Enivironment. And Java's inability to tolerate errors slewing off its own runtime environment, certainly doesn't help. This particular thwart experience is probably related to their "MetaClass" requirments.

navicat
I don't want an application with Wine built-in. It conflicts with the Wine I already have installed and is an incredibly boggy duplication of effort.

Wednesday, September 16, 2009

layman data III

Helpful links: **Google MySQL Primer CERT Bulletins Webmaster World Forum Simulate foreign keys - MyISAM Cascading and key constraints - MyISAM, InnoDB, NDB Create tables using PHP script
This is the third in the series, though not meant as a coherent progression. A random collection of tidbits or crumbs to follow. Recently: * Cascading and foreign key constraints with different engines. My webhoster provides only the MyISAM engine, so no foreign keys. Foreign keys are the "relation" in an RDBMS, auto-updating child relations when a parent is updated, cascading inserts and deletes, and so on. This apparently can be approximated in a number of ways in MyISAM. TRIGGERs can be created, loops which do multiple inserts, etc. The InnoDB engine makes this process native from the time of creating the tables. Much easier. To switch between engines in existing tables, we use: ALTER TABLE tablename TYPE = MyISAM; * Added CERT link above. The CERT bulletin link above quickly reveals the many injection threats arising each week. It appears one has to lock-down the code of a production server which, in turn, apparently requires time and patience to learn and implement. * Scripts to install tables. Appear to format as .sql dump files but without the data inside. * Proper documentation, once this is more focused and defined. So far, a simple RTF file using underline for primary, and italic for foreign key, has been helpfully direct. Seen it elsewhere too, but read it in Welling, L., Thomson, L. (2008). PHP and MySQL® Web Development, Fourth Edition. Addison-Wesley Professional. pg 208-209 informit link ~$50. REFERENTIAL INTEGRITY