Tuesday, December 16, 2008

not my fault

Over the past half year or so, I have expanded my repertoire of skills and made myself important to the lab in ways I wasn't hired for. I have campaigned for the organization of our DNA constructs in electronic form and have implemented a database and web interface to take care of that. This done, I kicked a wiki into life and started harassing my colleagues about contributing their wisdom, about making their experiences and knowledge available to everyone. You can always ask, but only as long as someone remains a member of the lab. Whoever leaves – and academic labs have a dynamic turnaround – takes what he has learned and improved in the lab with him, irretrievably. Lastly, I designed and coded the lab's new website, an effort that's not complete yet because people drag their feet about adding content. But my part is done and I'm happy with it.

I know there's no reason for complacency, though. An IT project is never done when the last modification is saved and the site goes live. It requires diligent supervision, constant maintenance and painstaking error prevention. That's why I'm equipped with administrator and root passwords and have turned into some sort of mini system administrator.

Our lab's web and database servers sit on a computer that's used for regular work. In fact, it's a student's primary workstation. Today, this computer went down. Its juice was mysteriously cut and it just blanked. When the power supply was restored, it came back to life as quickly as it had faded, and everything seemed to work. Everything, that is, but the wiki. Accessing its start page only gave a cryptic php error. Something wrong with the database, apparently. And something was very wrong indeed. The database server was fine. The plasmid database worked as it should. But the wiki database hadn't shut down cleanly when the computer zonked. Data had been compromised beyond repair.

It was then that two thoughts flushed through my brain almost concurrently. First: Damn, I don't have a backup. Why did it never occur to me to be prudent? Second: I'm so damn glad I'm not professionally responsible for this database. Imagine I worked in some commercial operation that would stand lose a million pounds every minute their databases are not operational. Someone would find his way to my desk very quickly and start yelling at me as if had I gnawed through the power cable with my own teeth.

There is very little yelling in academia, thankfully. Were my boss to hear about this meltdown, he would ask with honest concern, 'Will you get it back up? Will get the data back?' And I would reply that all would be easier if I had read all the pages of the MySQL bible I bought many years ago, and not just those immediately relevant for what I was doing. We would share a laugh.

The server wasn't critical, but the database would need to come back to life eventually, and I'd like to have the data back. I went down to the department's system administrator who, professional that he is, assured me that he had done weekly dumps of the database's content and he would send me the latest version he has on backup. It will be up to me to restore it and to make sure that future backups will be more frequent. What a nice environment for learning this is.

No comments: