Latest Cellar crash better
Holy crap, people.
This one was a killer, but I think the machine is now entirely much better, thankyouvery much. What follows are the crappy details that I had up here instead of a real Cellar for a few hours:
---
The power supply fan seized. OK, how many times have I mentioned how
important the power supply is? Now there's a very expensive Enermax
running the Cellar machine.
It first appeared that the system was suffering from the same trouble
that hosed it last time - a failing drive. But once I realized how hot
the machine was, and that the power supply was particular hot, I
figured it might have something more serious. I replaced the power
supply but the system was still failing. And then the motherboard
suddenly died.
The process of rescuing a Linux system that won't boot, involves
booting from "rescue" CD, fixing what's wrong with your system, and
then booting from the system again. This means hitting that DEL key,
going into the BIOS, and telling it to boot from the other device.
After doing that 20 times, suddenly the system just wouldn't boot at
all. I may have blown another BIOS setting, but... thing wouldn't
even beep.
That was last night, and at that point I had no recourse because the
stores were closed. This morning I bought a new motherboard (and a
new processor just in case). Installed those, they run great...
But what THEN turned up was a slowly failing hard drive or further
filesystem confusion.
So! I added another drive, and moved one filesystem to it, leaving
the root system on the questionable drive. I'll upgrade and move the
whole thing to another new drive over the weekend.
This corrected almost everything wrong, but the Cellar still wouldn't
fly. In fact, starting it would cause the system to crash again!
I found that the database tables that run the Cellar had been really
messed up, requiring very complete rebuilds. The system lived well
through database reconstruction work, which was like a stress test on
everything except the web and database. But that added another couple
hours of downtime for the Cellar. That's where it is now, the tables
aren't working right for some reason...
This stuff is getting expensive and so I'm going to add the tip jar back.
Future tip jar donations will go to devoting an entire system solely to
the Cellar, a move that's well overdue. It used to have its own system
when its was dial-up.
And why did all this happen? One reason I can think of. I washed the
fan filter, and I might have put it back WET. It didn't feel wet but
you know how those things are. Well, in a dusty environment (your
basic house), dust + water = glue. It might have just been too humid.
The thing died a week after replacing that filter so this may be a
reach, but my next house will include a clean room.
In the end, it was the long hours of trying to get everything back up and running, that lead to a few additional hours of downtime. I knew I had a good set of data, but was going to kinda heroic efforts to save about 6 hours worth of messages -- which is how many were lost this time. A handful.
I was so frustrated working all this time, additionally with no Cellar -- that I started making dumb mistakes. Jacquelita told me to step away from it for a while and she was right.
Now, on the good side, this system is almost entirely new. New mainboard, new power supply, new CPU and cooler, one out of two new disks. My goal is to upgrade it, maybe some time off hours this weekend or something, and then get rid of the last disk still in use. At that point the system will be almost entirely new hardware and entirely new software, and at that point it should only be another few months until the next crash.
Tony - you are an absolute stud! thanks for all the energy (and money) you expend for the community.
Congratulations UT
I knew you'd get this %$#@ figured out! Now you can rest easy tonight... Tomorrow night - Martinis - on me! :browhappy
Yikes.
Good work on the recovery.
oh and just in case you have some spare time on your hands... i still don't see a tip jar. even if we can't contribute millions, every little bit has to help, right?
The house looked intact so I figured it was something you could handle, given enough time and money. Thanks, UT. :band: Thanks Jax.
I can only imagine what a frustrating pain in the ass this must have been for you.
Glad to see it up and running again.
Thanks for the effort you put into this. :)
And it's still a frustrating pain in the ass -- kernel just panicked a minute ago, and the system had to be rebooted.
Looks like this'll continue until I can get the root disk changed out... dammit
It stayed up overnight so things look good. Still expecting to upgrade/move this weekend, if I can manage an after-hours timeframe.
The Paypal donate button is back on the front page. Do consider donating. I know there's a small chance I could become a multi-multi billionaire next year, but it really is a small chance. Plus if that did happen I would give the money back to you, if I could. I have to pay for all this new hardware next month and I think I will be selling lint on eBay to do so.
Genuine Cellar lint?! Man, I'll start the bidding off at $5.00!
Thanks, as always, UT. I don't know why you are so obsessed with keeping The Cellar a living and breathing entity, but I'm mighty glad that you are.
You're going to sell lint on ebay???
Be careful, man, I think AT&T's Unix patents still cover that. You don't want to be mixed up in this SCO mess...
Seriously, that bites. Once again thanks for all you do to keep this place running.
Oh, and,
Still expecting to upgrade/move this weekend, if I can manage an after-hours timeframe.
Fuck that. This isn't some high-availability corporate environment. Do the move when you feel like it. Don't feel obligated to do it in the middle of the night just to avoid inconveniencing the rest of us.
well.
jinx and i just made a small donation, and going by the number in parentheses ( which i believe to be the total number of transactions to a payee) there had only been 17 donations to the tip jar previously. i sure hope that that is not the TOTAL ever number. you can spare $25 or $50. get off of it. it's important. Tony and bruce can;t pay for this all by themselves.
(I think that's the number this year but includes about 12 T-shirt payments. Most of the year, the tip jar was not on the page, my fault.)
OK - the system is still sick, just having gone through a half-hour of downtime while I put new memory in it. It had been crashing every hour or so...
Some of the memory in it is new (fresh from my other desktop system) and the system seems to be staying up longer than five minutes. Good...
What are the specs of the current machine?
What are the desired specs for a standalone Cellar?
It doesn't require anything powerful at all. In fact I routinely UNDERclock the systems by about 5%. The main problem is that it has to be on and responding 24x7x365, and so any weakness in such a system will inevitably show up. [seriously geeking now] In fact I now think a fan on the MB's chipset also failed, and this probably wreaked havoc. Not all systems require fans for the MB chipset, and now I think they should be avoided. More moving parts only means more trouble and in this situation, trouble that can't be replaced easily. My video card now has a fan and I am starting to wonder whether it would be cheaper just to install a wet-dry vac blower to run air through all the systems and vent to the outside.
It lives on a big UPS and should only power-cycle about twice a year (but probably more often, for other stuff).
Where's the damn tip jar? I can't live w/out my shot of cellar each day. BB
Front page, bottom right. :)
Found it, but my email addy has changed. So having a small problem w/paypal. Said my card was bla bla, etc. guess it'll get worked out soon BB
UT .. thanks for your efforts. I suffered withdrawal symptoms for an least an hour .. then I found another BB .. :D
Seriously .. I know the frustration. My video card fan fried and died last week .. and I had to try source a new one ..
I ended up, I got the one with the bright blue LED light in it. It would be even better if I had a perspex case now, to show off my new video card fan light .. but by twisting my head to the left and lowering it to desk level, I can just glimpse the blue light thru the vent holes in the LH side of the case ..
Ahhh .. the satisfaction of current level high-tech ... ;)
P.S. .. hope you got the EXTREME SERVICE fan, with the pollution filters .. :D
Tony, you had me freaked out for a day or so. When I saw your page up about the crash, I was wondering if you would accept a paypal donation. I did not remember seeing the Paypal button on the bottom before, although I don't scroll down to the bottom of the home page very often (hint, move the button to the top).
I sent in a few bucks, since I've been on and off the Cellar since it was a dial up BBS.
*shudders, still getting over the horrific events*
Man, I was Jonesin' for the Cellar. It was horrible! On the second day I even emailed Bruce to see if it was just me or if it was really down. I didn't know what to do! It was AWFUL!!
I'll be tipping as soon as Steve's check comes in...
Status: I have turned off X (Linux graphical console) and the system is more stable but still not completely stable. I've changed the swap partition out for a different one on the new disk and the system still crashes. If it's not answering for a minute, just wait and I am probably pushing the reset button.
I don't think I have time to address this properly until maybe Tuesday! But I keep bringing the system back up and, I promise this time, no messages will be lost. I'm backing up the database regularly.
[QUOTE=OnyxCougar]*shudders, still getting over the horrific events*
Man, I was Jonesin' for the Cellar. It was horrible! On the second day I even emailed Bruce to see if it was just me or if it was really down. I didn't know what to do! It was AWFUL!!QUOTE]
I did the same thing with Wolf. Maybe I *am* still addicted, just not to booze and drugs.
Ok. I got the row hoed out w/paypal. Now just who is Anton? :-) BB
Momma gave me some fancy moniker at birth, and some people still like to call me by it!
I kinda suppected that BB
I went into real cellar withdrawals. It was AWFUL! I had my debit card stolen (GRRRRR!), but when its replacement arrives, I'll be sure to send you something. Don't have much $$$$, but I'll do what I can.
Latest Cellar crash better
I'm sorry, but all I can think of when I read that header is:
Latest Cellar crash better, Latest Cellar so hawny, Latest Cellar crash you long time joe.
Maybe I go sleep now. ok bye.
Status:
If you've noticed more downtime for the system, that's right; it's still sick. Tuesday two new sticks of memory are arriving, and I'm waiting to do anything else until they get here. Until then, the system crashes every 3 hours or so and unless I'm here to reboot it, it stays crashed.
But again, no more messages will be lost, this I promise; I'm backing up the database that contains them, all the time.
What was all that about City 1 cigarettes that popped up about an hour ago? That really annoyed me. Haven't they anything better to do?
The spam problem is getting noticeably worse... eventually we'll have to change how we operate to deal with it.