Cellar outage Friday 10/31
 

Well here I sit the only one who can get to the site... it appears to be a major frame relay problem, and it's not clear right now how long the outage will be.

Two hours in total. That's not so bad. Could have been worse. I'll just put my intestines back into my body now, while I email my clients that everything's cool.

Outage was due to router problems at dca.net. A traceroute report during the outage:

3 111 ms 100 ms 120 ms 66.236.204.5.ptr.us.xo.net [66.236.204.5]
4 110 ms 100 ms 110 ms p4-3-0.MAR2.Philadelphia-PA.us.xo.net [207.88.87.45]
5 120 ms 120 ms 120 ms p5-1-0-1.RAR2.Washington-DC.us.xo.net [65.106.3.153]
6 120 ms 110 ms 121 ms p1-0.IR1.Ashburn-VA.us.xo.net [65.106.3.138]
7 120 ms 131 ms 110 ms 206.111.0.42.ptr.us.xo.net [206.111.0.42]
8 130 ms 130 ms 130 ms pos2-0.cr01.phl02.pccwbtn.net [63.218.30.33]
9 130 ms 130 ms 130 ms dcanet.fe7-3.cr01.phl02.pccwbtn.net [63.218.31.2]
10 140 ms 131 ms 140 ms core-1-pos0-1-0.oldcity.dca.net [216.158.30.65]
11 121 ms 130 ms 130 ms core-1-atm1-1-0-100.hq.dca.net [207.245.123.62]
12 150 ms 140 ms 140 ms core-1-atm0-0-0-100.oldcity.dca.net [207.245.123.61]
13 150 ms 140 ms 141 ms core-1-atm1-1-0-100.hq.dca.net [207.245.123.62]
14 221 ms 310 ms 401 ms core-1-atm0-0-0-100.oldcity.dca.net [207.245.123.61]
15 160 ms 140 ms 151 ms core-1-atm1-1-0-100.hq.dca.net [207.245.123.62]
16 180 ms 260 ms 330 ms core-1-atm0-0-0-100.oldcity.dca.net [207.245.123.61]
17 130 ms 140 ms 130 ms core-1-atm1-1-0-100.hq.dca.net [207.245.123.62]

They could just be looping because the routers are trying to find a good path while the frame relay routes are down. (Or something -- I am not Cisco certified or anything)

It could be more confusing to those routers because my routes were moved from one office to the other at one point, and maybe some of the old routers still have routing information for my addresses.

And my outgoing traceroutes don't even go via hq.dca.net, so they must have multiple paths through their whole network.

It's all geek to me.;)

Now that the system is working, this is how trace route reports:
9 150 ms 180 ms 180 ms dcanet.fe7-3.cr01.phl02.pccwbtn.net [63.218.31.2]
10 131 ms 140 ms 130 ms core-1-pos0-1-0.oldcity.dca.net [216.158.30.65]
11 140 ms 150 ms 131 ms core-1-atm1-1-0-100.hq.dca.net [207.245.123.62]
12 160 ms 160 ms 150 ms dcanet-gw.catalystinternet.com [216.158.21.162]
13 160 ms 160 ms 150 ms cellar.org [207.245.113.69]

In the meantime, why do they have you on a so expensive technology called frame relay?

Probably because nobody uses it so it's cheap now... even though there's no practical difference at all between a standard T1 and frame relay at T1 bandwidth.