HOLY CRAP: Cellar loses a WEEK of messages

Undertoad • Aug 9, 2004 9:13 am
Well it's not been a good night.

Same problem as several weeks ago, except that this time my incremental backups were hosed too - which meant that the only recourse was to restore from a full backup from slightly over a week ago.

Only a few hours from when I would have done another full backup too.

It was filesystem corruption that suddenly happened; I thought I had located the exact cause of it, but apparently not!

Goddamn! Well I have some work to do. I'm looking around for more things to restore. I will try to redo the IotDs.

SORRY SORRY SORRY SORRY SORRY!

By the end of August I will be moving the Cellar to its own system so that my business and the Cellar are isolated from each other. It's not that either one particularly caused this problem but I would rather they were independent...

Argh
richlevy • Aug 9, 2004 9:16 am
I just logged on and thought something was weird. Sorry to hear about it.
Undertoad • Aug 9, 2004 9:18 am
Updated posts and threads for *another* bulletin board I run here (for a client) were found in the lost+found of the blown filesystem... but of course not the Cellar posts and threads.... dammit dammit goddammit
BrianR • Aug 9, 2004 9:18 am
Sorry to hear that UT... Makes my minor problems seem pale in comparison.

I hope you get to the bottom of the problem!

Brian
Undertoad • Aug 9, 2004 9:45 am
The really annoying thing is not knowing why it happened. This time I kicked off a really rigorous test of the drive, which does the usual testing... it writes different patterns to every track on the drive, and then reads it back to compare it. Neither affected partition had any bad sectors.

Last time I thought the monitor, which was known to be bad, might have fed voltage back into the system and crashed it mid-write or something. This time there was no monitor! So what caused a perfectly good, freshly-built filesystem to suddenly go corrupt for no obvious reason? Why did it fail and why does it work now?

Hackers? Can't rule it out.
elSicomoro • Aug 9, 2004 9:53 am
Like, say...PETA or ELF types?
glatt • Aug 9, 2004 10:16 am
That's a drag. But it makes me appreciate all the behind the scenes work that UT does. I had a very small webpage for a while. It's amazing how much work there is in maintaining it.

Thanks UT!
Trilby • Aug 9, 2004 10:19 am
Aw--that's a pain in the ass, UT. Sorry.
NothingButNet wrote some rocking haiku's--like the celery one.

Good luck, UT!
lulu • Aug 9, 2004 10:24 am
It stinks that your server crashed. I'm a new user that registered on Saturday, had to go through the process again this morning. Quickly and painlessly I was able to reregister with the same user name. :D
xoxoxoBruce • Aug 9, 2004 11:56 am
Anything I can do to help, say so. :(
Kofi Annan • Aug 9, 2004 12:28 pm
sycamore wrote:
Like, say...PETA or ELF types?



(on the phone with CrashTheirForumGuy@UN.1worldgov)

"Hey......you can relax.....they think the animal whackos did it."
lookout123 • Aug 9, 2004 2:31 pm
hey UT - thanks for all the work you do. we appreciate it. and don't worry about the lost messages, they pretty much consisted of cellarites telling a PETA head to F himself. :D
xoxoxoBruce • Aug 9, 2004 2:40 pm
Herself. :cool:
lookout123 • Aug 9, 2004 2:54 pm
herself? huh, its not really all that humorous to tell a female to F herself... hell some guys pay for that vision on the internet.
TheLorax • Aug 9, 2004 2:54 pm
seriously dude what do you think we're paying you for here

wait

scratch that
Cyber Wolf • Aug 9, 2004 3:06 pm
We still love ya, UT!...in..a..purely platontic sense, you understand. Aheh... :blush:
Undertoad • Aug 9, 2004 3:46 pm
Naw, this one deserves a red face... I suck. I should have been on top of stuff more, since the problem happened earlier. I'm blaming my poison ivy... can't think straight when you're infected with the stuff.

On the bright side, I now understand filesystem recovery very, very well.

Thankfully it was only the Cellar that was impacted... everything else business-related that changed in the last week was on other systems.
lookout123 • Aug 9, 2004 3:48 pm
think of what this may have done for LJ's post count though. it is like adding years back to a normal person's life.
xoxoxoBruce • Aug 9, 2004 4:14 pm
I'm blaming my poison ivy... can't think straight when you're infected with the stuff.
You been romping in the moonlight again, UT? :lol:
slang • Aug 9, 2004 4:20 pm
The rumor going around is that UT has been getting more tail than a bicycle mechanic lately.

Go ahead and stick with the poison ivy story if you want though.
Troubleshooter • Aug 9, 2004 4:40 pm
Were crashes this common in the previous incarnation of The Cellar?
Undertoad • Aug 9, 2004 5:03 pm
No, but there were other problems, such as the modems getting stuck and not answering the phone when you called it. And busy signals - that didn't happen much though.
Troubleshooter • Aug 9, 2004 5:06 pm
I should have been more specific. :)

Were crashes more often in version 2.x?
garnet • Aug 9, 2004 9:01 pm
Kofi Annan wrote:


"Hey......you can relax.....they think the animal whackos did it."


I admit it--I did it. :smashfrea
Undertoad • Aug 9, 2004 10:23 pm
There were no crashes before version 3, but there's nothing in v3 that should create this level of havoc. Never say never, but if there was that level of problem other people should have seen it by now!
Elspode • Aug 10, 2004 12:01 am
Garnet, I am in need of your services. I have a couple of electronic scores that need settling! :ninja:
garnet • Aug 10, 2004 12:12 am
Elspode wrote:
Garnet, I am in need of your services. I have a couple of electronic scores that need settling! :ninja:


Oh, so you NEED me now, eh? Listen to how your tune has changed. :eyebrow:
Crimson Ghost • Aug 10, 2004 2:53 am
Perhaps, UnderToad, are you using Windows? As we all know, crashes are a program in the OS. So well hidden are they that Yoda himself cannot find them. Bill Gates should be crucifi....

[center][font=Arial Black][size=6][color=red]WARNING!!! THIS USER HAS BEEN TERMINATED FROM THIS FORUM!!![/color][/size][/font][/center]
[center][font=Arial Black][size=6][color=red]MICROSOFT PROTECTION SQUAD DELTA 451[/color][/size][/font][/center]
crossfire • Aug 10, 2004 11:14 am
sorry about that UT, when do you think everything will be fixed?
Undertoad • Aug 10, 2004 11:20 am
Everything IS fixed at this point, and my backup schemes have changed so I will guarantee we will not lose a single message from here on out. No matter what.
Clodfobble • Aug 10, 2004 11:30 am
Perhaps I'm doing something wrong... but I can't find any of the older threads. Specifically, I started out looking for the Book of the Month thread on "Imajica," but I can't seem to bring up anything older than around July 11th. Was that when the first crash happened?
Undertoad • Aug 10, 2004 11:36 am
It might be when the upgrade happened. I know that posts are inserted into the search differently, but it was supposed to have rebuilt the search engine for old posts.
crossfire • Aug 10, 2004 12:27 pm
beautiful
marichiko • Aug 11, 2004 1:27 pm
I just got back from several days out of town, so now I feel like I didn't miss anything when I was gone! Seriously, sorry to hear of your woes, UT, and I hope you get to the bottom of the problem soon. I for one am very greatful that you go to all the trouble you do to make the celler available to us all!
crossfire • Aug 11, 2004 1:48 pm
yeah, thanks again UT.
marichiko • Aug 11, 2004 4:40 pm
Hey, UT, a while ago today I couldn't get on at all. I got that stupid little message about url not found. What was up with that? :confused:
richlevy • Aug 11, 2004 4:56 pm
Undertoad wrote:
Everything IS fixed at this point, and my backup schemes have changed so I will guarantee we will not lose a single message from here on out. No matter what.

Uh-oh, are you sure? I was thinking of running for President someday and was counting on an *ahem* accident to erase my posts, not to mention the fact that I inhabit the same virtual space as Radar.

I can see it now [COLOR=Red][SIZE=6][FONT=Comic Sans MS]Cellar Vets Against RICHLEVY[/FONT][/SIZE][/COLOR]
Undertoad • Aug 11, 2004 5:19 pm
There was a power outage this afternoon. Which may help explain the goofy system behavior - the UPS didn't hold the system up. It should have. So perhaps the UPS is at fault here.
xoxoxoBruce • Aug 11, 2004 5:52 pm
Whip those hamsters. :)
marichiko • Aug 11, 2004 7:40 pm
Damn that UPS! And damn those lazy hamsters![FONT=Comic Sans MS][SIZE=4][COLOR=DarkRed]And, no! Rich Levy will NEVER have my vote! We at the cellar know what he's truely like! (Gonna have us all erased, Rich?)[/COLOR] [/SIZE] [/FONT] :eyebrow: :D
zippyt • Aug 11, 2004 9:26 pm
ALL HAIL UT KEEPER OF THE CELLAR !!!!!!!
Seriousley , sunday nitei tried to sign on to see what was up , nothing , nadda , my wife said " well i guess the cellar flooded " .
Thanks Ut for fixing stuff , we didn't need those stinking posts any way !!!!
crossfire • Aug 11, 2004 11:34 pm
UT, you are greatly loved and will go down in history has THE KEEPER OF THE CELLAR, THE GREATEST FORUMS SITE ON THE INTERNET.
Nothing But Net • Aug 11, 2004 11:45 pm
Well let's not all go sucking each other dicks just yet. Aside from losing some brilliant poetry, and pics that would make Ansel Adams envious, I had posted the proof to Fermat's Last Theorem here. Now it's gone forever, or at least for another few centuries.
crossfire • Aug 12, 2004 1:12 am
but still, UT is the best
Undertoad • Aug 12, 2004 1:33 am
In reality I suck and I know it so all of this is just going right over my head.
marichiko • Aug 12, 2004 12:34 pm
Pooh, UT. This is one of the coolest sites on the net, and we all love you for keeping it going for us! :grouphug:
crossfire • Aug 12, 2004 12:51 pm
well, do what you want with it, but we all know you rule and face it, "two heads are better than one" a forums heads are more powerful than one head.
SteveDallas • Aug 14, 2004 1:08 am
Undertoad wrote:
Everything IS fixed at this point, and my backup schemes have changed so I will guarantee we will not lose a single message from here on out. No matter what.


Ahhhh a challenge!!! :biggrin:
lookout123 • Aug 14, 2004 1:11 am
Undertoad wrote:
In reality I suck and I know it so all of this is just going right over my head.

some of it has to hit you right between the eyes - i mean didn't your head get a little bigger now that you're playing out?

*damn bassists get all the chicks, just 'cuz they have rhythm*
xoxoxoBruce • Aug 14, 2004 8:03 am
Don't let him fool you, lookout123. The cord to the amp is really coming from an Ipod in his pocket. :biggrinje
lookout123 • Aug 14, 2004 10:39 am
xoxoxoBruce wrote:
Don't let him fool you, lookout123. The cord to the amp is really coming from an Ipod in his pocket. :biggrinje


what? did someone say modern-day Sid? where the hell is nancy?
marichiko • Aug 14, 2004 8:20 pm
Nancy? She hangs out with Sluggo, right? Or was that Little Lulu? :3eye: