Mythical works database (1 Viewer)

mjp

Founding member
Rather than revive the old database thread, I'll just post an update here.

When I decided to revamp the database and the database interface back in 2/06, it seemed like a simple task. Designing the new database was pretty simple, even though it went from one big table to nine smaller - and more inclusive - tables.

The interface was more complicated, but then I figured it would be. I have the interface just about ready though. Only thing is, it can't be finalized until all the data is migrated from the old to the new database, and therein lies the rub, as the kids say.

The data in the old database is a mess. So over the last year and a half I have been slowly migrating data to the new database manually. What the hell does that mean? Well it goes something like this:

- Look up the entry in the old database
- Take all the information in that record and split it into the proper tables in the new database
- Check it for accuracy and duplication while you do that
- Look up the same entry in the old database to make sure it didn't appear in more than one place. If it did, start all over again

What all that consists of is a lot of cutting and pasting of text, formatting the text and doing the new database insertion(s).

This task, when you have over 6,000 rows of data, is very depressing. Imagine trying to untangle a bathtub full of linked paper clips. It's something like that, only less fun. I would do a few hundred entries, then set it aside and try not to think about it for a couple of months. So you can see why progress has been slower than you might expect. ;)

I looked at the project a few days ago and had to decide whether to abandon it, or continue to cut paste cut paste cut paste cut paste cut paste for the rest of my life.

I decided to do neither. Instead I've been working on a set of scripts that convert the old entries to the new database. The only problem is the checking for accuracy and duplication bit. That prevents me from making a big script, hitting "run" and doing the whole thing in one bite. So I have these scripts that effectively move the data and allow me to get rid of duplicates, standardize spellings, etc. But - it still does it one entry at a time.

Which means it still takes some time, but it is about 10,000 times easier now, as all the lookup and copying shit is done by the scripts. I just click links and make the decisions that have to be made by a human.

All this long-winded post really means is that completion of the new db and interface will in fact be done some time before my 90th birthday! In fact, probably within the next couple of months.

What is funny is that as far as the data returned, you probably won't see a damn bit of difference. Ha ha ha ha. But the data can be matched and cross referenced in a number of different ways now, and by the time I finish I will be relatively certain that the data in there is clean, which is something that database geeks dream about. Not that I'm a database geek, you know...

Included in the new interface will be a little form to add items to the db that aren't in there (not directly, of course), or to report errors if you find them.

So, back to clicking I go. I just finished all the Hearse magazine entries. That means that I am up to "h" in the magazine entries. Then I have to do about half of the book entries, various other crap, clean up the interface, and viola, new db.

See? Nothing to it.
 
that sounds like a shitload of thankless work.
so thank you, really. I'm always amazed by the amount of "free" labour people put into a project just for the love of it. I promise I'll notice the difference ;).
and thank you for making me realize how lazy I am.
 
MJP, I have absolutley no idea of what this task involves. well I kind'a get the paperclip/bathtub thing. but I sure do appreciate all of your damn hard work and dedication to this sight. I'm sure ol' BUK would have been honored. thanks man.
 
Well, since we're in a new thread I'll say again that I did this (and continue to do this) for selfish reasons. I wanted to be able to look these things up, and no one else had a public database, so there you go.

There may be great private databases, but that's kind of a shame, really. I don't see any point in doing all this and keeping it hidden away just for my own use. So that's why it's here. By making it public I wind up getting more data anyway, so you see, back to selfish motivations. ;)

I love you guys too, don't get me wrong, I just love myself more.

Yeah.
 
What are you going to do with all the paper clips?

:D
 
Thanks for the update, mjp! That's a lotta work you got in front of you. Respect! - as the kids say nowadays. I'm sure ol'Buk sits on his cloud and nods his head in approval and smiles...
 
Last edited by a moderator:
It seems that you could have got to the letter i had you not wasted the time above procrastinating -get that finger out and put that nose back on the stone.


And when do you want our 2008 contributions?
 

Users who are viewing this thread

Back
Top