Forums incident (slowdown or error)

Post by **Chrisax** » Wed Nov 14, 2007 3:17 pm

Last night GMT and today, you maybe experienced very slow AP forums or even no access at all.

The reason is simple: the whole RAID system (!!!) died for a still unknown cause (possible EMP -Electro Magnetic Pulse- despite the protections). The electronic was replaced but the hard disks remained dead. This affected Apache/PHP/FTP/filers systems. The SQL (database) system is separate and was not affected, so data is safe. Routers were affected to some degree.

The AP bots server is also a separate machine and was not affected at all.

The whole web system switched to an emergency cluster that provided slow access due to the fact it was at the same time verifying files and restoring some backups. Also routers were being fixed/replaced.

We reached a disastrous packet loss of 81% average.

The RAID system is being rebuilt on new machines. Operations should resume to normal conditions in a few.

The power supply system is being investigated; techs are searching for hacker hamsters....

Somewhere among these UPS and power transformers a hamsters commando is hiding....

Mud · Post by **Mud** » Wed Nov 14, 2007 3:41 pm

the hamster is in sneak!

Alphacenta · Post by **Alphacenta** » Wed Nov 14, 2007 6:52 pm

Noes.. must be this:

Hyde · Post by **Hyde** » Wed Nov 14, 2007 8:42 pm

Can I post again?

Hyde · Post by **Hyde** » Wed Nov 14, 2007 8:44 pm

Hyde wrote:Can I post again?

Ok, I can post again. I was starting to take it as a hint.

(when I logged on last night and tried to send messages I could read them, I could see other people posting, but it kept telling me I couldn't post ... though ... I'm not on that computer so I may not be able to post from home, will test that by quoting myself again tonight)

Mud · Post by **Mud** » Wed Nov 14, 2007 9:58 pm

Hyde wrote:Can I post again?

no!

Post by **Chrisax** » Wed Nov 14, 2007 10:57 pm

Hyde wrote:
Hyde wrote:Can I post again?

Ok, I can post again. I was starting to take it as a hint.

(when I logged on last night and tried to send messages I could read them, I could see other people posting, but it kept telling me I couldn't post ... though ... I'm not on that computer so I may not be able to post from home, will test that by quoting myself again tonight)

Yes it's a hint: I need my favorite (except me) forum admin back

Seriously, nothing had change in your forums account/permissions and nothing will ever change.

I assume your issue came from some malfunction of the network at that time although I don't see what one. Please let us know if it works when you are back at home. If it doesn't, please provide info and I'll look into the problem.

Hyde · Post by **Hyde** » Wed Nov 14, 2007 11:35 pm

I was kidding. Sheesh, people think I have no sense of humor ... its just called "dry"

Post by **Chrisax** » Thu Nov 15, 2007 12:25 am

Hyde wrote:I was kidding. Sheesh, people think I have no sense of humor ... its just called "dry"

OMG he can post AGAIN.... the whole operation is a failure....

Hyde · Post by **Hyde** » Thu Nov 15, 2007 1:26 am

test from home

Hyde · Post by **Hyde** » Thu Nov 15, 2007 1:26 am

Hyde wrote:test from home

I'm afraid you're screwed now.

Mud · Post by **Mud** » Thu Nov 15, 2007 3:15 am

Hyde wrote:
Hyde wrote:test from home
I'm afraid you're screwed now.

pfft, I ain't scared!

Hexy · Post by **Hexy** » Thu Nov 15, 2007 3:19 am

Mud wrote:
Hyde wrote:
Hyde wrote:test from home
I'm afraid you're screwed now.
pfft, I ain't scared!

But I am!!!! :lol:

Post by **Chrisax** » Mon Nov 19, 2007 1:45 pm

We are back to normal conditions.

The average packet loss is less than 1% (Still a peak at 40+ % for some reasons being investigated)

The servers automatically restored our files (almost) without any human intervention.

The whole servers cluster is being redesigned in order to prevent such an exceptional accident, especially the way RAID is organized and the way auto backups are organized.

(If you are a Geek, you'll be happy to learn that a whole new storage system is in the making -and in fact has been in the making for 6 months- based on Solaris/Open Solaris and Sun ZFS systems.

<---- Geek)

Ah... OK... you want the truth and not tech bullshit... well... the hamsters are not on strike anymore!