03/21/16 Monday 4:26pm EST
adore2 locked up, rebooted adore
01/29/16 Friday 6:51pm EST
locked up hard, had to power cycle
12/09/15 Wednesday 04:20am EST
Locked up, had to reboot
02/13/12 Monday 10:39pm EST
sd6 died, replaced somehow I nuked the FATS on the other drives,
had to replace with mirrors.
11/03/11 Thursday 02:33am EST
load high, locked up, rebooted
07/31/11 Sunday 7:09pm EST
load high or locked up probably from incoming, didn't check, rebooted.
06/26/11 Sunday 02:23am EST
load high due to hits from 173.70.24.22 fw.block rp1
05/30/11 Monday 06:44am EST
www Locked up, out of mbufs.
05/25/11 Wednesday 09:57am EST
backbone locked up, cl -> clrp
telebooted.
03/31/11 Thursday 5:35pm EST
Replaced root drive
12/08/10 Wednesday 10:54am EST
Clarity backbone locked up.
06/16/10 Wednesday 11:26pm EST
Power outage, offline for about 30 minutes
04/26/10 Monday 4:14pm EST
Upstream is down
04/23/10 Monday 4:14pm EST
Core router taken off line by accident. Down for about 15 minutes.
04/19/10 Monday 1:57pm EST
Crashed from bad swap drive sd13e
03/26/10 Friday 1:17pm EST
Locked up on /w10 freezing I think
11/20/09 Friday 10:52pm EST
Upstream down for 6 hours from 3pm to 9pm
10/26/09 Monday 12:44am EST
Upstream down for 45 minutes or so.
10/07/09 Wednesday 6:25pm EST
Web server taken down to clear up confusions on disk mounting.
10/05/09 Monday 03:12am EST
Adore2 is down and will be down for a while, we will try to
get it back up again as soon as possible.
09/21/09 Monday 07:18am EST
Network outage from 2:30am until 7am due to bad ethernet card in
core router.
Bypasswed eth2 as eth1, and will replace.
07/31/09 Friday 12:03pm EST
Power outages in Ithaca this morning interrupted service a number
of times from 10 to 12pm, in particular South Hill Business campus and
those conected to West Village.
07/16/09 Thursday 9:02pm EST
Upstream is down.
05/06/09 Wednesday 10:49am EST
Core router had to be rebooted, did not go smoothly.
All is well now.
07/04/08 Friday 4:26pm EST
System renumbering continues.
Light is now www.lightlink.com.
Two main DNS servers are still light and majesty but are
at 64.57.176.2 and 3.
02/18/08 Monday 11:29pm EST
Pleassant Grove is presently offline, Verizon is working
on the T1.
02/18/08 Monday 1:14pm EST
bond0 widened, pulling 19 megs
01/27/08 Sunday 06:52am EST
locked up, rebooted.
10/14/07 Sunday 05:31am EST
light locked up, first time in forever. fan dead, perhaps
that's why.
10/03/07 Wednesday 07:06am EST
UPS on fvr died, taking CMC, watkins glen, WV and Axiohm off line.
New one replaced, a big one this time.
That one lasted for 10 years. RIP
08/28/07 Tuesday 2:55pm EST
Mail pop server locked up this morning about 10am.
Rebooted, all is well.
08/08/07 Thursday 7:06pm EST
Mail servers under tremendous load from bot storm, no available
sockets for real e-mail to come in. Google for 'bot storm'.
Mail delayed but not lost.
06/08/07 Friday 3:47pm EST
Wvbr link is down
06/02/07 Saturday 3:26pm EST
T3 down again. Ticket 0602128
Back 17:23:03
05/31/07 Thursday 08:33am EST
T3 down. Calling Paetec. Ticket 53140
Outage in NY area.
05/31/07 Thursday 09:42am EST
05/24/07 Thursday 7:36pm EST
Numerous outages this morning on pleasant Grove T1,
kionix and pfinc links were caused by faulty router installed
last night during DDOS attack.
05/23/07 Thursday 11:50pm EST
T3 backbone went off line, called into PAETEC.
Turns out it was a DDOS attack from one of our own colo machines.
03/10/07 Saturday 5:50pm EST
Power outage at 4pm took all RP customers off line.
03/10/07 Saturday 07:20am EST
dialup down from midnight on, for second time. This time
bug fixed.
01/11/07 Thursday 9:48pm EST
Memory error on adore, crashed. Reseated all memory, rebooted
Next time will start to replace chips that report bad.
01/08/07 Monday 2:12pm EST
Memory error on J0303 on adore. Will replace.
01/05/07 Friday 11:43pm EST
Major power outage on aurora street taking all of Roy parks and
Downtown Ithaca and Collegetown off line.
01/04/07 Thursday 01:12am EST
Upstream off line at 1:00am for scheduled maintenance.
12/18/06 Monday 12:05pm EST
Adore locked up.
12/08/06 Friday 3:30pm EST
All of lightlink off line over night for about an hour
due to upstream problems.
12/05/06 Tuesday 2:15pm EST
adore locked up
12/04/06 Monday 6:22pm EST
Light, the legacy web server, was crashed out for about an
hour.
This has been fixed with netprotect.
11/21/06 Tuesday 12:00am EST
Conn Hill radio tower has been down for most of today, we tried
to fix it but failed. Will try again tommorow.
10/26/06 Thursday 1:56pm EST
adore crashed from asynch fault errors.
rebooted
09/24/06 Sunday 05:13am EST
le0 went down. ifconfig le0 down, ifconfig le0 up brought
it back.
That's a new one.
07/31/06 Monday 04:58am EST
light crashed. async memory errors errors. rebooted
j0203 removed.
07/24/06 Monday 11:55am EST
T3 locked up. Rebooted router, came back. That's a first.
07/04/06 Tuesday 03:01am EST
light locked up, rebooted.
06/22/06 Thursday 06:09am EST
Outgoing smtp server lockedup. Rebooted.
06/13/06 Tuesday 01:09am EST
Adore locked up
05/17/06 Wednesday 6:57pm EST
Had to reboot light to gain control over spammers using
formmails to send spam.
04/25/06 Tuesday 11:55pm EST
Multiple short term lockups, hard to diagnose. swap drive failing
I believe from errors I saw during a reboot. Removed the drive.
04/11/06 Tuesday 3:58pm EST
Short outage this morning due to and old ethernet termination
turned bad connecting DNS servers to world.
04/07/06 Friday 1:10pm EST
Upstream service died for about an hour at 12 noon. All
customers off line.
USLEC says it was hardware failure at their end.
03/04/06 Saturday 7:48pm EST
Scheduled downs at 6pm and 7pm to upgrade routers at both end
of Fairview to Roy parks link.
02/26/06 Sunday 11:14am EST
vpn router locked up from disk errors, took all radio vpn customers
off line.
02/25/06 Saturday 3:26pm EST
web server light crashed, taking out web and DNS.
tried to format a dying drive, didn't like it I guess.
02/13/06 Monday 01:55am EST
eth0 locked up on light. ./net restarted it
01/24/06 Tuesday 8:50pm EST
fvr router locked up, didn't reboot smoothly.
Watkins glen, hospital, therm, and most radio users connected to
chair, thair and vhair were off line for the count.
Took router off line, cleaned up configuration, rebooted
smoothly.
01/18/06 Wednesday 5:50pm EST
Major outage caused when our stream US LEC lost 4 Ip ranges
that feed our ISP. Those not on those ranges were not affected.
All is well again.
11/09/05 Wednesday 12:28pm EST
Last night core router at Roy Parks gave up the ghost, took
a bit to get it back on line. All RoyParks customers were down
including hotspots fed by Roy parks.
This morning the mail server gave up the ghost, lost /dev/hda,
replaced.
10/28/05 Friday 12:04am EST
Adore locking up slowly, rebooted.
09/29/05 Thursday 11:14am EST
T3 down from unknown reasons. Rebooted core router and
everything came back.
09/22/05 Thursday 3:06pm EST
Barracuda not working right, delaying mail, taken off line. Lots
of spam will get through.
09/03/05 Saturday 3:59pm EST
Light load high, many 'D's in ps aus on syslog and httpd
All drives check out fine.
Rebooted
06/18/05 Saturday 2:04pm EST
Various things off line for 5 minutes while replacing burnt UPS
batteries.
06/14/05 Wednesday 10:43pm EST
Blown transformers down the street caused Nyseg to pull all power
from Lightlink at about 1pm until 3:30pm. Working on getting a
generator, really we are... :)
06/13/05 Monday 07:52am EST
adore crashed, rebooted
06/11/05 Saturday 2:19pm EST
Changed UPS on T3 router, everyone down for 2 minutes.
06/11/05 Saturday 1:25pm EST
Lightning storms damaged the watkins glen link over the past few
days. It is mostly restored. Conn Hill radio customers are still
down, should be up by today.
05/17/05 Tuesday 1:19pm EST
Yesterday's network outage was caused by a cable cut at
Fairview. Hospital, Therm and Watkins Glen were taken off line
as was everyone connected to them.
05/16/05 Monday 12:14pm EST
adore crashed, rebooted
04/20/05 Wednesday 2:13pm EST
web server taken off line by accidentally by setting root
perms to rw-r--r--
04/18/05 Monday 12:50am EST
light crashed from data exception, possibly from trying to
fromat the bad drive sd12.
04/17/05 Sunday 10:24pm EST
web pages moved from sd12 to sd6
04/13/05 Wednesday 11:37pm EST
Primary web pages on light /dev/sd6 moved to /dev/sd12. Original
drive failing.
/dev/sd6 replaced by /dev/sd12
Some NFS links to adore did not come back properly preventing
uploads of new web page material. Rebooted adore this morning.
4/14/2005
03/28/05 Monday 11:20am EST
pm1 rebooted bouncing some dialup customers.
03/27/05 Sunday 11:18am EST
vpn0 rebooted to clear errors on hda. All radio vpn connections
reset.
fw3 rebooted to add hdc. Babcock, casaroma, fairview and
all DHCP out to Watkins Glen offline for a short while.
bond0 rebooted to add hdc. All of Roy Parks off line for
a short while.
wvbrair mis configured at 15:08pm taking various customers
off line until about 9pm.
03/26/05 Saturday 3:14pm EST
vpn0 taken down to replace hard drive and ide cable.
radio vpn connections down for 10 minutes or so.
radius dialup authentication also down for same period.
03/12/05 Saturday 7:13pm EST
Roy Parks line taken of line to upgrade. Did not go smoothly,
multiple problems. Its working presently (in general :) )
03/10/05 Thursday 11:21am EST
Light, main web server, off line for 20 minutes due to homer's
stupidity.
03/05/05 Saturday 09:56am EST
Dialup server died on mal functioning radius daemon at midnight.
Restarted daemon at 8:52am.
02/21/05 Monday 11:25am EST
fvradio/cmc1 upgraded to 3.88 on saturday.
wg0 upgraded to 3.88 on saturday.
rpradio upgraded 3.88 on Sunday.
02/02/05 Wednesday 2:09pm EST
Incoming mail server crashed, lost /dev/sdc1 /var/spool/mqueue
01/23/05 Sunday 3:58pm EST
Amp taken off of rpair.
01/19/05 Wednesday 10:37pm EST
Mail delays today and yesterday from multiple problems
with barracuda and our own mail server. Mail server rebooted
tonight, and barracuda set to 10 incoming connections max.
12/09/04 Thursday 05:55am EST
Mail down from failed root partition. Replaced.
11/24/04 Wednesday 11:36pm EST
Major power outage took all of lightlink off line this afternoon
for about 2 hours.
After power came back, the Fairview to Roy Parks backbone did
not come back properly due to unexpected problems with high speed
modems forming the bonded line.
09/13/04 Monday 10:47am EST
Watkins link at Conn Hill went down again at 11pm last night.
Blown amp replaced this morning.
09/12/04 Sunday 09:43am EST
Watkins Glen was down from Friday morning to Saturday afternoon.
First problem was corruption of a radio cable that destroyed signal
quality, and once that was fixed, a problem with usage of channels 1 2 and 3,
something we still don't have a cause for.
System at conn hill was completely rebuilt with LMR 600 wire, new
amps, new antenna and new radio. But the channel problem continued.
Presently running on 4.
09/04/04 Saturday 2:34pm EST
Adore lost its root drive sd0a. Rebuilt.
08/21/04 Saturday 07:41am EST
fvair locked up. Rebooted.
07/24/04 Saturday 2:41pm EST
hssi card replaced, T3 network down for 1 minute.
07/05/04 Monday 04:00am EST
cmc radio locked up at cmc end. telebooted.
06/17/04 Thursday 8:43pm EST
HSSI card went off line again, replaced.
06/17/04 Thursday 12:44pm EST
Adore crashed, swtch
06/16/04 Wednesday 10:40pm
HSSI card went down again, rebooted.
Should be replaced or put in new slot
06/13/04 Sunday 8:01pm EST
T3 card HSSI 5/0 went down, all lightlink off line.
Rebooted, did not come back.
Reseated, rebooted came back.
05/21/04 Friday 8:39pm EST
forward and reverse DNS brought into sync, all IP's have reverse
DNS now but may have lost some specific names that had been set by
hand.
05/15/04 Saturday 7:17pm EST
Home directories separated between light and adore.
This should not cause any problems, elob and moeorg were
copied from adore over to light.
05/13/04 Thursday 4:39pm EST
Again lost the primary drive on mail, /var/spool/mail and
/var/spool/spam.
This is annoying, don't trust drives any more.
04/28/04 Wednesday 3:37pm EST
Problems receiving e-mail from nyroc.rr.com mail servers proved
to be faulty code in new spam trap code. This is fixed and queued
e-mail should be delivered shortly.
04/24/04 Saturday 2:55pm EST
Modems are down due to failures in our upstreams upstreams Level
3.
No ETA
04/18/04 Sunday 6:39pm EST
News downs, fan locked up, oiled and rebooted
04/18/04 Sunday 5:33pm EST
Incoming mail server pop.lightlink.com taken down to oil
power supply fan.
Mail server power supply failed totally a short while later,
had to replace it.
02/27/04 Friday 11:41am EST
Watkins Glen link down.
02/26/04 Friday 11:40am EST
Watkins Glen link went down again.
Nick and John G, went to CH and replaced everything, nothing
worked, put back in the original and it started working again.
Suspect bad wire/amp/antenna.
02/23/04 Monday 1:23pm EST
Watkins Glen link is down, CH <-> TH link finally
went into total failure.
Replaced radio card at CH. Link came back.
02/21/04 Saturday 5:53pm EST
Choice One modem lines are giving busy signals.
Portmaster 0 rebooted, seemed to clear the problem.
01/30/04 Friday 12:30pm EST
modem banks at 277 1228 were upgraded total non functionality
last night. Working fine now...
01/18/04 Sunday 3:47pm EST
primary mail spool drive lost, replaced with new drive.
backups from last night are damaged, so taking from Saturday night.
users sent logs of mail received so they can ask for important
stuff to be resent.
Need to reconsider raid.
01/17/04 Saturday 12:25pm EST
outgoing e-mail and webmail interface down while changing UPS that
went out.
01/10/04 Saturday 1:28pm EST
Net is damaged out in the middle of the US somewhere, google, netscape,
yahoo, have only intermittent access.
01/07/04 Wednesday 4:23pm EST
wvbr1 locked up, telebooted.
12/31/03 Wednesday 12:14pm EST
cmc radio linked locked up, rebooted fvradio.
12/24/03 Wednesday 11:50am EST
rp3 link feeding DSL modems at Roy Parks locked up. Router tweaked,
and rebooted. If it does it again, we will replace it.
12/13/03 Saturday 6:00pm EST
fvradio -> CMC1 and all related backbones are down while we fix
the cabling at fairview.
12/06/03 Saturday 1:58pm EST
SSID on fvair changed to fvair from lightlink
SSID on wvbrair changed to wvbrair from lightlink
Users who have not registered their MAC address with us may be
blocked from using these two radios.
11/28/03 Friday 8:10pm EST
Adore crashed panic on cpu 0 swtch
11/25/03 Tuesday 10:48am EST
Light locked up at around 3 am causing failures on web and dialup.
Beeper did not go off, apparently Arch Wireless is down also.
Serious.
11/23/03 Sunday 3:19pm EST
Major outage at a level3.net router in New York City is
making connectivity between RR customers and lightlink flakey at
best. Been bad since Friday night. No ETA
11/10/03 Monday 10:23am EST
POP mail server crashed this morning.
VFS No free inodes ask Linus
09/16/03 Tuesday 10:03am EST
Adore locked up for unknown reasons. Rebooted
09/04/03 Thursday 2:18pm EST
rp2 router feeding paradyne and pairgain DSL out of Roy Parks
was rebooted a few times before we found a routing error causing
network confusion.
We also want to get rid of the private IP's being given out
to people because infected machines fill up the masquerading
tables making it impossible for anyone on a private Ip to get out.
I goofed that change up but good, so things are back where they
were for the moment.
08/28/03 Thursday 10:52pm EST
Upstream down for about about an hour.
08/23/03 Saturday 2:48pm EST
mail server taken down at 2pm per scheduled down to replace
failing hard drive. Drive was not replaced, but root partition
was moved to a different place on the drive.
Went well.
08/15/03 Friday 11:41am EST
Radius authetication failed for dialup modems this morning
because the authentication server (light) ran out of swap space
because the web server was not being hit upon probably due to outages
across the state.
We were also not using our full battery of swap space, have added
2gig, so we should be fine :)
07/17/03 Thursday 1:07pm EST
adore crashed on panic on cpu 0: swtch
07/14/03 Monday 10:47am EST
adore crashed from panic on cpu 0: swtch
06/17/03 Tuesday 10:18am EST
Mail server 'mx' locked up for unknown reasons. Rebooted.
This is a rare event, hope this is not a premonition of things to
come.
05/30/03 Friday 3:24pm EST
Apparently gem was failing to do web logs properly starting
from May 19th, which may or may not have had anything to do with its
failure on the 24th.
05/24/03 Saturday 11:27pm EST
Gem lost is primary hard drive tonight while replacing
a CDROM drive. Turned machine off, put in DVD drive, turned it on,
and drive was totally dead.
Web e-mail and USER AREA were off line. Some web hits
will be permanently lost for Saturday.
05/06/03 Tuesday 2:20pm EST
Mail crashed from errors on /dev/sdc. /dev/sdd wouldn't
boot when rebooted. Cold booted everything and reseated all the
drives, and it came back fine with mininmal damage.
I hate this job.
04/30/03 Wednesday 9:37pm EST
Alpha test modems will be taken off line for a short time every
12noon pm bouncing those on it. These are the numbers starting with
216 0008 for Ithaca adn 387 7110 for Elmira. Those using our normal
modems will not be affected.
04/20/03 Sunday 10:14pm EST
adore crashed this morning at 8am from panic on cpu 2 swtch. Seems
to be doing this about once a month.
rebooted the new modem banks tonight bouncing everyone, testing
the second set of modems in the bank.
04/19/03 Saturday 12:22pm EST
Massive DDOS attack started 1pm Thursday afternoon taking
our webserver and primary DNS off line. Intermittent attacks
followed until late Friday afternoon.
IP numbers of non IP domains have been moved to get out from
under attack. Some pages may not work properly again until client
machines are rebooted.
04/10/03 Thursday 5:40pm EST
isdn2 modem bank in ithaca locked up, rebooted.
03/20/03 Thursday 2:47pm EST
adore crashed from swtch condition.
03/17/03 Monday 7:58pm EST
isdn4 bad modem line 1/2
03/16/03 Sunday 7:08pm EST
isdn2 modem bank rebooted bouncing 16 users.
permanent bad modem at isdn2-8
03/14/03 Friday 12:21pm EST
Bad ISDN lines on modem bank 2 caused dropped connections.
03/08/03 Saturday 1:56pm EST
pop server taken off line at 12 pm to upgrade motherboard to 900MHz.
Went smoothly.
03/01/03 Saturday 9:00pm EST
Lost the power strip that powers the core router. Replaced
Lost connectivity for about 5 minutes.
02/21/03 Friday 9:30pm EST
Power outage around 5pm took down entire system. All systems
running again by 9pm.
02/18/03 Tuesday 08:42am EST
Our firewall fw1 machine locked up this morning for unknown
reasons blocking traffic to all legacy services, mail, web, modem
authentication and news.
Reboot fixed it. Don't you just hate that? Or maybe you
gotta love it because it wasn't anything worse.
That's its first lockup ever in about half a year of service.
02/10/03 Monday 11:50pm EST
Adore crashed, panic on cpuX swtch
02/03/03 Monday 11:54am EST
Recent problems with our upstream resulted in periodic outages to
various parts of the net, google.com and others. Fastnet claims to
have fixed the situation. It has been reoccuring on and off for the
last week or so. 01/25/03 Saturday 11:01pm EST
Fastnet, our upstream, is doing upgrades to their core routers to
protect against the latest SQL worm. Outage should last about 30
minutes.
01/21/03 Tuesday 04:08am EST
Lightlink was down starting at 12 midnight until 4am due
to a power failure.
01/04/03 Saturday 09:04am EST
Power out at 5am or so. Came back around 8am. Gave it some time
to prove stable., everything backup around 9:30.
Watkins Glen link down until Sunday due to power outage
at Conn Hill.
12/29/02 Sunday 3:39pm EST
Gem, webmail and mailing list machine, out of control, sendmail's
taking 2 cpu seconds to verify. Rebooted to clear out.
12/27/02 Friday 04:06am EST
Light again crashed mbuf map full
Did a netstat -na vmunix.6 vmcore.6 from /var/crash and found
multiple sendmail connects. Have firewalled off the spammers. This
may break something else, too tired to think straight at the moment.
I hate spammers.
12/27/02 Friday 02:49am EST
Light crashed, mbuf map full
12/18/02 Wednesday 9:59pm EST
Mail system died this morning from a spam attack. 7000 pieces of
bounced mail accumulated in the mail queue and the mail server ran
out of process space.
We have taken measures to prevent this in the future.
Also later in the afternoon the webmail interface web server died
for unknown reasons, so webmail was down until late afternoon.
12/12/02 Thursday 7:07pm EST
Wireless link to Watkins Glen went down from ice at about 2pm.
Terry Hill Yagis were ice coated.
12/08/02 Sunday 12:14pm EST
Modem 14 on 272 2284 not answering.
12/06/02 Friday 11:12pm EST
Modem 8 on 272 2284 not answering.
11/26/02 Tuesday 11:39pm EST
Another modem on the 5026 banks giving ring no answers.
11/26/02 Tuesday 1:16pm EST
Modems on bank one of the V90 group locked up, had to reset the
modem bank a few times. Something or someone is causing ping times to
isdn.lightlink.com to periodically get huge.
11/19/02 Tuesday 12:37pm EST
Web server locked up from scsi errors at 11:30 or so.
Lost /dev/sd9a on web server housing /usr/local and /w5 and /var
Have temporarily replaced with mirror.
Will rebuild the master from the tape backups.
11/09/02 Saturday 10:49am EST
Outgoing mail is down cleaning up after a spam attack that
took out the mail server.
10/25/02 Friday 12:14pm EST
All three mail servers were hosed this morning from a spam
attack. User Area was off line, as was webemail and outgoing smtp
server. All should be fine now.
Really have to figure out some way to stop this particular kind
of attack, they are really deadly.
10/22/02 Tuesday 6:48pm EST
UPS replaced, knocking off all Ithaca dialup users.
10/21/02 Monday 7:18pm EST
UPS died on modem banks, everyone booted.
10/17/02 Thursday 11:08am EST
News server died on own determinism, no idea why.
Restarted.
10/16/02 Wednesday 3:46pm EST
Jammed modem line today refusing to let people connect, got it
reset and its working fine now.
10/06/02 Sunday 1:23pm EST
Light web server taken down to add more disk space.
10/04/02 Friday 10:46am EST
277 0356 rebooted to clear out lan failures.
09/29/02 Sunday 2:47pm EST
Light, web server, and adore, shell server, taken off line
for 2 hours to upgrade tables and power that hold them.
Dialup was unavailble during this period due to no DNS nor
radius authentication.
09/21/02 Saturday 2:10pm EST
Majesty, light and adore taken down for 10 minutes to upgrade
resolver libc's.
09/16/02 Monday 7:15pm EST
Mail taken down to increase mail and spam partitions to 4.5 gigs
each.
09/14/02 Saturday 9:03pm EST
Incoming pop mail and news offline at 12pm to move computer tables.
Outgoing smtp server offline this evening to replace dying
ethernet card.......
09/13/02 Friday 12:50am EST
Adore shell server crashed Panic on CPU 2: swtch
Becoming more often...
09/04/02 Wednesday 8:13pm EST
Mail server mx taken down to increase memory to 512Megs.
08/31/02 Saturday 5:07pm EST
Adore, the shell machine, crashed on panic on cpu2, swtch.
Seems to enjoying doing this every couple of weeks.
08/26/02 Monday 11:47am EST
There was a failure this morning in the program that
authenticates remote users to our smtp server. Some users were not
able to send mail, got 'Lightlink Relaying Denied' error messages.
New and improved program has bugs... :)
08/10/02 Saturday 6:59pm EST
Ithaca dialup modem banks taken off line to change IP range
from 205.232.34.x to 216.7.30.x. All static IP's changed also for
dialup users only.
08/04/02 Sunday 12:34pm EST
Name service was messed up last night during upgrades.
Intermittent failures to get out to the net would have been caused by
this.
07/16/02 Tuesday 12:06pm EST
web mail interface down over night due to a failed upgrade.
mail itself not affected.
During the periods when gem was actually down, logins to adore
were not available to due to nfs mounts timing out.
06/23/02 Sunday 11:59am EST
newsfeeds down for 12 hours due to running out of disk space
06/06/02 Thursday 1:42pm EST
Adore rebooted on panic on cpu 0 swtch
05/25/02 Saturday 12:10pm EST
Light taken down at 12:00pm to add second CPU.
No problems.
05/18/02 Saturday 6:39pm EST
Adore taken down at 6pm to install second CPU on its motherboard.
So far so good...
05/13/02 Monday 09:20am EST
Adore/shell crashed for unknown reasons, cpu panic swtch
Seems to be doing it every month or so.
04/27/02 Saturday 6:21pm EST
All Ithaca V90 modems were successfully rebooted using automated
script which will run at 3:30 every Sunday morning.
04/23/02 Tuesday 11:10pm EST
isdn rebooted due to instability. Users were being disconnected
on lines 1 and 3 repeatedly.
04/13/02 Saturday 3:23pm EST
12pm started scheduled down.
Light taken down, cpu upgraded to 150 MHz, second ethercard
added, power supply vaccuumed.
Adore taken down, added 4 64mb memory sticks power supply
vaccuumed.
Light's UPS had its battery replaced.
Modems taken off line to replace their UPS battery too.
Wheels placed on table holding light and adore.
Total down 3.5 hours.
04/06/02 Saturday 1:18pm EST
smtp outgoing mail server off line for 10 minutes emergency upgrades.
It's back now.
03/23/02 Saturday 11:29am EST
web hits are off line for a while, while we practice an upgrade
to light on majesty.
03/19/02 Tuesday 3:13pm EST
Elmira was down for a few minutes due to an EMI frame outage.
02/03/02 Sunday 7:47pm EST
Mail server taken down at 1pm to upgrade memory.
Didn't go well, machine turned to glue with 1024Megs, had
to remove memory and go back to old configuration.
After much playing around mail was back up by 3:30pm.
Gonna have to do this again next weekend.
01/28/02 Monday 4:16pm EST
Elmira modens offline again from about 12 noon until 4pm due
to a mistake made by Verizon while doing upgrades on the circuits.
Virtual circuit DLCI's were lost :)
01/26/02 Saturday 3:27pm EST
Elmira modems were off line from about 2am until morning due
to repeated power outages.
Public radio static IP addresses should be working properly now,
subnet range was 'shared' by another machine offering bogus mac
address for default route 205.232.177.65
01/24/02 Thursday 7:07pm EST
Harmony rebooted itself at 4pm for unknown reasons.
This affected dialup users on 277 5026 only.
In may of 2001 the config file was corrupted or copied over
and was no good, so users were able to dial up but not go anywhere
beyond local IP numbers.
We rebooted again at 6:10pm and again at 7:10pm
All should be well now.
12/09/01 Sunday 5:38pm EST
Light rebooted last night to get rid of /usr/local/main
Adore rebooted tonight for same reason.
12/04/01 Tuesday 2:09pm EST
Last night ftp daemon was upgraded to 2.6.2 due to a root exploit,
unforunately the new version did not work right and prevented the
uploading of pages. This has been fixed.
10/29/01 Monday 12:24am EST
Adore crashed, panic on cpu 0 swtch.
Been up long time!
09/19/01 Wednesday 6:37pm EST
Major web outages caused by Nimda worm attack on our webserver.
Things are mostly under control at this time.
Outgoing mail server rebooted to replace UPS.
08/02/01 Thursday 08:38am EST
Mail server mx and pop locked up at 8:38 from process table full,
not sure why. Rebooted. Could have been a lock file problem with
the spam trap.
07/26/01 Thursday 12:45pm EST
Adore, shell machine, locked up at 11:58am from memory chip
going bad. Removed, not yet replaced.
07/19/01 Thursday 2:54pm EST
DDOS against iron and seal from 213.148.3.210 using known
problems with web interface. Turned off interface and blocked IP
range at router.
07/11/01 Wednesday 1:49pm EST
Adore lost its root drive swap partition to errors. In process
of installing new drive.
06/30/01 Saturday 6:53pm EST
Mail taken down to increase disk space. Gives us some time
to decide what to do about abandoned but growing mailboxes.
Gem rebuilt with new mother board.
06/30/01 Saturday 05:03am EST
Gem locked up again, multiple ethernet cards did not work,
probably pci bus gone. Will replace motherboard today.
smtp server moved over from gem to emerald.
06/29/01 Friday 09:39am EST
Gem (outgoing smtp server) locked up ether port. Reseated
card in different slot, if continues will replace card.
06/24/01 Sunday 1:34pm EST
Rain storms over the past few days have taken their toll on the
Fairview to CMC link. It is presently down for repairs.
06/09/01 Saturday 09:16am EST
Shell machine adore locked up due to failure of spamtrap to clear
lock. Basically I think bad locking finally caught up with us, ran
out of processes. Cleared them out and cleared the spamtrap lock and
all was well.
05/25/01 Friday 8:03pm EST
All web pages restored. End of Event
05/25/01 Friday 10:50am EST
Main web drive locked up on light, needs to be replaced.
Presently running on mirror drive, do not upload web pages.
05/12/01 Saturday 1:31pm EST
All majordomo mailing lists hosted by lightlink have been
down since about a day ago. An upgrade to the sendmail.cf on gem
broke the majordomo.aliases. Mail sent to the mailing lists
would have been returned with User Unknown.
My apologies.
05/07/01 Monday 2:46pm EST
two modems on isdn2 were giving busy signals, reset seems
to have cleared them out.
05/04/01 Friday 8:58pm EST
Power strip blew out tonight from old age apparently, knocking
all modems off line, along with gem, emerald and majesty. emerald is
the smtp server, so outgoing mail was interrupted. Things should be
working again.
05/01/01 Tuesday 11:35am EST
Between Friday and Monday there was a significant outage on the
net between lightlink and earthlink. This was caused by a
malfuntioning router in our upstream's network. Due to this network
outage, much mail may have been delayed getting to Earthlink, and some
mail from earthlink may have been returned to sender as undeliverable.
03/31/01 Saturday 6:48pm EST
I crashed the outgoing smtp server accidentally by
hitting the wrong button on the UPS, trying to test it, I
turned it off. Sigh.
03/18/01 Sunday 6:09pm EST
Outgoing smtp server locked up its ethernet port for about 20 minutes
this afternoon for unknown reasons.
Majesty filed to properly beep me when smtp went due to incorrect
settings. This has been fixed terminatedly.
03/07/01 Wednesday 7:52pm EST
mail (mx) was taken down at 6pm for scheduled upgrades.
Back up at 7pm.
Scsi wiring was cleaned up on /dev/sda.
spam filter was installed, with no filtering.
03/01/01 Thursday 3:56pm EST
gem was rebuilt. web mail was down for a short while.
02/27/01 Tuesday 8:15pm EST
Adore load 120.0, large amounts of mail waiting in queue
for 5 days was suddenly dumped on me as postmaster as undeliverable.
Procmail running 5000 lines of spamtrap drove load high.
02/23/01 Friday 3:12pm EST
Got mail bombed again by same guy from sympatico.
This time we put in the DUL data base, and it is keeping it out.
Still lots of sendmail's opening up. Not ideal.
02/20/01 Tuesday 7:29pm EST
Two large mail bombs came in this afternoon bringing all
mail serves to their knees. About 3:00 to 5:00pm.
No mail lost, just some temper.
While putting in filters for the spam I managed to block
out the smtp server from everyone for about 10 minutes at 18:14pm
02/11/01 Sunday 4:27pm EST
Mail was down for 20 minutes to make emergency changes to disk
arrangements. /var/log given own partition so it doesn't 'wear out'
the root drive.
02/05/01 Monday 10:03am EST
Well after 340 days of uptime, adore finally locked up. Had to
reboot.
10/20/00 Friday 6:52pm EST
Light rebooted. Error in rc.local required multiple reboots.
10/06/00 Friday 11:28am EST
ftp www.lightlink.com locked up, for unknown reasons
killed off inetd and restarted. now its fine.
That's a first.
10/05/00 Thursday 9:22pm EST
DSL modems back on line at about 10:30 am this morning. Another
customer's traffic through same switch was triggering a bug in the
DSLAM firmware. Moved customer to another switch and DSLAM
started to behave again. Will upgrade firmware shortly.
Sheesh, only 20 hours of down.
10/04/00 Wednesday 7:06pm EST
At 12:39pm this afternoon our Paradyne DSL modem rack went into
continuous reboot mode. We have spent 7 hours so far trying to debug
it to no avail. Problem is even new chassis, cards and control cards
are doing the same thing!
This is probably going to be a long down.
09/29/00 Friday 10:34am EST
light load went high from SSL attack.
09/23/00 Saturday 8:43pm EST
6:00pm Scheduled down, ftp server rebuilt with new disk drives
raising space from 2 gig to 8 gig and adding a 40 gig backup mirror
drive.
09/20/00 Wednesday 11:29am EST
web server locked up from onslaught of ssl requests from rogue
site, blocked at border router.
09/12/00 Tuesday 11:41pm EST
Lightning took out the Roy Parks network for 3 hours tonight
ending around 10:30pm. Power came and went many times before
stabilizing. The Roy Parks router is running at 96 percent cpu, and
showing signs of smoking. Will replace the student backbone shortly
with dedicated line back to Fairview.
08/23/00 Wednesday 11:19am EST
Out of control hits on the secure server locked up light with
load 131. Modem dialups were blocked during this time.
Gonna have to do something about this, but not sure what.
08/17/00 Thursday 11:33am EST
Romance had a hard drive partition crash last night, /dev/hda11.
When rebooted, it didn't mount majesty's web data logs, so they
have been unavailable. In trying to get them to mount, majesty locked
up and had to be rebooted. Majesty had been up for 300 days.
Machines get old and rickety if left up too long.
Sort of like me.
08/15/00 Tuesday 10:23pm EST
Light rebooted at 10:15pm, been up for 137 days, but was beginning
to show signs of irrepairable corruption in core. tty's that could
not be erased, and virtual domains that went to our home page rather
than their own, twice in as many mornings.
Also some confusion on the UMG weg page, we will see if all this
clears up.
07/24/00 Monday 4:37pm EST
5026 modem bank:
modems 67-69 reburned with firmware and reseated in card cage.
They were rejecting connections for at least 2 months.
07/18/00 Tuesday 1:15pm EST
Load on light went to 110 this morning at around 11am from
a flood of secure server hits, probably an errant client. Failures
of various warning systems, buddy in particular, caused a delay
in finding out about the condition.
While load was high, dialup authentication and some DNS services
also failed.
07/09/00 Sunday 9:45pm EST
news was down for most of today while we copied over
the entire news spool to the new machine. It seems
to working fine, little or no news should have been lost.
aurora went from 200MHz 12 gig to 500MHz 120Gig.
07/07/00 Friday 10:13am EST
news was down for about 2 hours yesterday afternoon preparing
for a new server.
06/25/00 Sunday 11:10pm EST
ftpd upgraded to 2.6.0 plus patch to handle root exploit.
06/07/00 Wednesday 9:48pm EST
A major network snafu on the morning of June 6th, caused
the net to slow down and become unreachable in many cases.
Mail backlogs caused waves of incoming mail when the net
opened up again, causing all three of our mail servers to crash
from lack of process space.
Some mail was lost.
06/02/00 Friday 10:16pm EST
Power outages took out Pleasant Grove pop T1 at about 5pm, lasted
for about 1 hour.
ftp server taken down for emergency repairs, disk upgrades
not completed. Power supply replaced due to stuck fan. Motherboard
needs to be upgraded to handle 40 gig drives.
04/05/00 Wednesday 7:05pm EST
Pleasant Grove T1 suffered an event, becoming sticky and not
allowing pings to go through. Went out there but found nothing wrong,
rebooted router, but it cleared it self up before I did that.
04/02/00 Sunday 6:30pm EST
Light take down to reinstall /dev/sd2, the log drive.
Went without incident.
03/31/00 Friday 01:53am EST
At about 1:30am, light lost its primary log drive /dev/sd2
panic on cpu 0 brought the machine down.
Replaced drive with existing emtpy partitions, will put in
new drive later.
I do not love this job.
It's making me old before my time.
03/30/00 Thursday 4:34pm EST
277 5026 modem bank was rebooted a few times to stop corruption
caused by new monitoring program.
03/20/00 Monday 9:27pm EST
news server hosed, lost news spool, had to rebuild from scratch.
All extant news lost.
03/08/00 Wednesday 11:50am EST
pop server mx rebooted to install linxu 2.0.38
03/01/00 Wednesday 8:55pm EST
At about 19:17pm this evening, lightlink suffered a distributed
denial of service attack that saturated our bandwidth for about 30
minutes. It was directed at a colo machine on our network.
02/29/00 Tuesday 9:18pm EST
Earlier today at about 5pm we suffered an attack on our
secure server, address 192.248.252.133 was opening repeated
connections to port 443 on light. Blocked at T3. Load
on light went to 150, took about 30 minutes to find out what
was going on, penetrate the machine and block it.
Tonight at about 9:15pm, harmony stopped autheticating properly,
people calling up with weird errors. Rebooted harmony and the
erpcd's running on light. No idea what happened.
02/28/00 Monday 11:48am EST
mx, mail server, spawned multiple crond's this morning at about
10am. This caused multiple buddy's to fire up, flooding our system
with buddy messages, and finally filling the process table on mx.
Although there are no logs indicating mail bounced, it is
possible that some did. Mail was also queued on the back up servers
and delivered later, but it is not clear that all mail was caught
properly.
A monitor has been placed on crond on mx, if it goes over 5, I will
get beeped immediately.
01/23/00 Sunday 10:54am EST
Due to an incorrect /etc/syslog.conf setting, news logs were
being dumped on wrong hard drive, resulting in failure to rotate them
and the hard drive filling up. News has not been propagating for
maybe a day. News was coming in, but not going out.
01/20/00 Thursday 6:06pm EST
17:15pm
pop server suffered more scsi errors this afternoon. Took her
down to replace entire computer, case, power supply motherboard, cpu
and memory. Running on pentium II now.
No mail lost.
01/18/00 Tuesday 6:26pm EST
17:10pm
pop server crashed completely, main mail drive showing corrupted
FATS. Attempts to save it were in vain. Mail Spool saved to temp
directory first, should be little damage to most mailboxes.
Swapped out entire bay of scsi drives with new one.
01/17/00 Monday 8:53pm EST
pop server taken down to replace scsi cable 8:29pm
01/15/00 Saturday 1:16pm EST
pop server taken down to swap out SCSI terminator
Also taken down last night at about 5pm to
Replace scsi jumpers
Replace CPU fan
Remove Tape drive
Get rid of all real time mirrors
Rebooted again at 6pm to install new hard drive partitions for
nightly backups. We should be able to get about 24 days of full mail
spool backups with the present system.
01/03/00 Monday 6:22pm EST
Light web server preemptively rebooted to avoid slow down.
Every 30 days or so light becomes sticky, there is some
evidence this is caused by repeated ssh's into light.
01/03/00 Monday 2:39pm EST
Mail server, mx, locked up again, this time we think
we caught the hard drive causing it, sdc1 running /var/spool/mqueue.
Damage was minimal, all mail should be intact.
12/27/99 Monday 10:04am EST
9:00am
Harmony our primary terminal server lost its power supply,
replaced with backup.
12/24/99 Friday 12:21pm EST
fvradio locked up, needed rebooting.
12/20/99 Monday 6:14pm EST
Web mail interface demo version expired today, have installed
a new demo, and ordered the real thing. Apparently *LOTS* of
people like it! :)
12/19/99 Sunday 6:30pm EST
Mail taken down to replace scsi card, and create new disk
partitions on /dev/hda and /dev/hdc
Popper also changed to server mode. Really cuts down on the file
copying.
12/18/99 Saturday 2:49pm EST
Mail suffered a severe hard drive crash early this morning and
had to be taken down at about 12:30pm to rebuild the mail spool
drives.
Apparently one of the mirror drives went bad and the mirror
software did not disable it as might have been expected. This caused
corruption in the file allocation table, which resulted in corrupted
mailboxes, and some bounced mail for some people.
During the rebuild, some mailboxes were probably totally lost.
We will go through the logs and inform those that had bounced
mail or lost mail boxes privately.
12/08/99 Wednesday 1:56pm EST
External net looks down, can't ping rahul.net
12/08/99 Wednesday 07:20am EST
Light load very high 100 or so.
Rebooted, got trace back error during the core dump.
Rebooted again.
Hundreds of radius and httpsd's running.
Httpsd seem to be coming from 208.150.209.130, filtered
at T3
Rebooted all NetServers. Radius chilled out.
Only bad taste is the traceback we got during the first core
dump. Could be coincidental, could be something more serious. New
swap partition is in place if that matters.
12/05/99 Sunday 6:58pm EST
Light and adore upgraded to Y2K patches and libc.
Down time 6pm to 7pm
12/05/99 Sunday 2:07pm EST
Testing backup dialup server in preparation for tonights down
of light and adore, found that password files were not being updated,
so some people could not sign on. That has been fixed.
Elmira customers are having problems with a burnt out modem which
is being fixed shortly.
12/04/99 Saturday 6:10pm EST
Mail was offline for 10 minutes tonight to upgrade is primary
spool drives from mirrored IDE's to mirrored fast wide SCSI
differentials.
12/04/99 Saturday 3:09pm EST
Outgoing mail service interrupted for a few minutes, needed to reboot
the smtp server due to jammed processes.
12/02/99 Thursday 1:59pm EST
Light rebooted.
Getting out of memory errors, cpu %0, topp sticking, other
anomalies. halt locked up machine, had to cold boot.
Adore was locked up while light was being rebooted.
11/25/99 Thursday 12:17pm EST
Elmira modems were down for a number of hours this morning
due to a power cable cut in the neighborhood.
Mail server, mx, was rebooted twice around 12:15 to install new
hard drives to hold mail spool. It may be rebooted again through out
the day.
11/21/99 Sunday 7:07pm EST
Starting about 5:30pm this afternoon news was down for a short
while as we moved the machines to a new room. At about 6:30pm, mail
was down for the same reason. No mail was lost.
The changes put both news and mail on a 100 megabit full duplex
switched backbone, to help improve performance.
News will be down again in a day or two when we move everything
over to scsi drives from IDE which really can't take the I/O.
11/20/99 Saturday 12:00pm EST
mx ethernet locked up. Did an eth0 down and up, and
it started up again.
no idea why
11/19/99 Friday 9:48pm EST
At approximately 17:28 this afternoon our network had an 'event',
data slowed down, people couldn't sign on, things came to a
standstill.
Although it is still unclear what happened, at the same time we
were being massively overrun by spam coming into our system from one
of our own users, which caused our mail server to run out of process
room and crash. Bringing this under control took about 2 hours,
during which time incoming mail may have been queued, and pop accounts
would have been slow to respond.
Some mail may have been lost during the crash.
11/05/99 Friday 10:05pm EST
secure shell keys changed across all machines.
mx lost its trust to smtp (emerald) so pophash.db was not being
transfered, causing people to be unable to relay mail through
lightlink from remote places to remote places. Started at about 17:00
until 22:00.
11/05/99 Friday 08:17am EST
mx mail rebooted to reinstall 2.0.36 OS
Addition of mqueue partition seems to have helped in the overload.
11/04/99 Thursday 7:38pm EST
Light rebooted to clear out dying OS.
top locking up rather than running smoothly, happens
every 30 days or so.
11/02/99 Tuesday 4:36pm EST
New kernel on mail has made the situation worse.
Have added a new partition for /var/spool/mqueue.
10/30/99 Saturday 12:12pm EST
Mail was taken down to install new kernel, to see if we can
improve the performace issues. However the new kernel refused to boot
properly, being unwilling to use one of the mirror drives (hdc) in DMA
mode. Booted back to 2.0.36.
10/29/99 Friday 5:50pm EST
Our T1 has been decomissioned.
At 5pm the T1 was disconnected interrupting service to modems,
shell, web and incoming mail.
A few minutes later connectivity was restored through our T3.
10/28/99 Thursday 09:49am EST
Ethernet card on pop server mx locked up, machine unresponsive to
network requests. Rebooted. If it happens again, will replace the
ethernet card.
10/24/99 Sunday 2:22pm EST
Romance lost its root hard drive. This caused emerald (smtp
mail), gem (majordomo) and mx (pop mail) to lock up certain functions
and they needed to be rebooted.
Nothing was lost, and romance's hard drive was replaced by its
mirror.
10/23/99 Saturday 1:40pm EST
Majesty patched for Y2K and Libc.so/sa
Rebooting may have caused lockups on light and adore.
10/15/99 Friday 8:14pm EST
Adore locked up, out of swap space. Don't know which
process. Brought under control without rebooting.
10/10/99 Sunday 8:44pm EST
V90 modems still giving busy signals every other call.
Dial once, get a modem. Dial again, it rings once, then turns
busy.
Put in another trouble report.
10/07/99 Thursday 8:33pm EST
V90 phone lines are returning random busy signals, looks like a
problem with Ma Bell.
Just keep trying to get in.
Trouble ticket opened.
0141031
10/05/99 Tuesday 11:25pm EST
mx, pop mail server rebooted. High load, sticky controls,
multiple processes running, in particular crond spawning timely jobs,
but many of them. Will have to watch for this.
Also yesterday a dialup V90 NetServer 16-I was swapped out of the
Elmira pop due to failing modems. This caused some problems when
customers were moved to another modem bank with incorrect settings.
The hunt group remains down in Elmira for one of the v90 banks.
10/05/99 Tuesday 04:43am EST
Denial of service attack from dannyboy.easynet.co.uk against port
443 httpsd caused load on light to go 80, jamming it completely. Took
me a while to figure it out, and get it blocked.
Radius server was also jammed from USR.lightlink.com, creating
hundreds of radius daemons.
People were not able to sign on from about 2am to 5am.
Things should be stable now.
10/03/99 Sunday 2:19pm EST
Adore crashed and rebooted itself for unknown reasons.
panic cpu0 swtch
09/28/99 Tuesday 8:08pm EST
Adore rebooted by me, for system upgrades.
/usr/local split into /usr/local and /usr/local/main
/dev/sd2g released
09/25/99 Saturday 2:23pm EST
Never rains but pours.
Romance lost its root drive last night, has been replaced with
the mirror drive.
Majesty just lost its root swap partition. Swap is presently on
the mirror, will probably move the mirror to root and toss the present
root drive. /etc/mtab and /etc/dhcpd.leases were lost.
09/07/99 Tuesday 10:18pm EST
Lost a modem card in the Multitech racks, caused no answers
on 3 modems and interrupted the hunt group. Will busy out
and get card replaced.
Nysernet had troubles installing a new OC3 fiber ring that hosed
some of their network over the past few days. This has caused
intermittent connectivity problems to the net.
09/06/99 Monday 8:18pm EST
A modem was giving busy signals earlier this afternoon on isdn2.
I rebooted the bank bouncing people who were on isdn2.
The net seems to have a major outage going on, some sites are not
accessible, other's are very slow.
Some other ISP's are not affected, so it may be a problem with
Sprint.
09/01/99 Wednesday 2:45pm EST
ftp daemon was upgraded to 2.5.0 today on adore, and promptly
broke uploads. Reverted to old version until fixed.
08/21/99 Saturday 6:23pm EST
Scheduled down time at 6pm.
Adore upgraded to Sparc 20, 256Meg.
08/20/99 Friday 6:19pm EST
Adore locked up around 10am this morning from uninterruptible
wait states on the mail partition. I had to crash it and reboot.
During the process the password file was corrupted and some could not
sign on.
Later in the afternoon, I hot swapped /dev/sd1a back into adore
so we could get a good root mirror going for tomorrow. This also
crashed adore again because the second root drive steals the SCSI
target from the first root drive and bye bye system.
08/18/99 Wednesday 6:27pm EST
Adore crashed at 17:43pm from swtch panic.
It's Sparc 5 will be replaced by a 20 on Saturday.
07/31/99 Saturday 09:24am EST
Adore crashed at 3:33am from asynch memory faults, pretty much
proving its not a memory problem as all chips are new. So its time to
replace the mother board. At 9am this morning I pulled sd1a root
mirror drive in order to bring up a new Sparc 20 which will replace
adore for a while, and forgot that swap was on the drive, so adore
crashed again.
07/18/99 Sunday 12:55pm EST
On Saturday night at about 5pm Fairview Square was hit directly
by lighting on the power pole that leads into the main power plant of
Fairview. The fuses were burnt out on the power pole, and the main
circuit breaker inside of the Fairview Complex was blown to
smithereens. Power was restored by about 10pm at which point we
brought up all systems only to find that the T3 router would not boot.
It was replaced by a backup router which took about 1.5 hours and
everything was up and running by about 11:30pm. The router seems to
have a flakey flashrom card, perhaps a result of the strike, but also
perhaps simply a mechanical problem.
07/06/99 Tuesday 11:43pm EST
Adore went load hight at about 11:15pm. I tried to play with it
extensively to see what might be causing it, no luck. pff -a locks
up. ps didn't show procmail this time as the first in line but login.
07/05/99 Monday 7:58pm EST
Adore went load high again. Changed procmail to latest
version 3.13.1
07/05/99 Monday 6:46pm EST
Adore went load high with locked mail partition. Took
a core dump and rebooted, will send to Sun.
07/04/99 Sunday 6:29pm EST
6:00pm Tried to switch the bad UPS with the new one without
bringing the system down, needless to say it didn't work. Adore
knocked off line, along with many modems.
07/01/99 Thursday 12:54pm EST
Adore went load high at about 12:30pm this afternoon from
a problem we thought was only with the popper. But the popper
was not running.
This time we got a core dump, and will be sending it to
Sun for analysis.
06/26/99 Saturday 10:32am EST
Power outage at 10:15am or so. UPS decided to go bad at
just that moment, adore, and modems taken off line.
06/24/99 Thursday 11:30pm EST
Shell users found fetchmail broken this morning after
an upgrade to imapd on mx last night around 12 midnight.
In general people should not be using fetchmail, except once
after they first enable their shell mail with enableshellmail.
MX has been stable, and adore has been stable with the popper
off.
06/21/99 Monday 4:05pm EST
About 2pm we started a test version of the popper on adore,
one that ran as a daemon rather than through inetd. About
two hours later adores load started to climb.
Every drive partition was listable except /var/spool/mail.
This has happened on two different drives in two different
partitions, in two different drive trays in two different drive
bays. This indicates it is not a drive problem. Also Iwas
able to write to another partition on sd9 which holds
/var/spool/mail, so the drive was working fine.
Since this follows the popper it seems to be a kernel/popper
problem. One has to ask why this never occured on light.
MX
Earlier today mx started to refuse popper connections because the
inetd daemon that listens on port 110 was hard coded to turn off if
too many connections came in at once. Max Parke hacked the code to
get rid of that check, and things have been fine ever since.
06/20/99 Sunday 8:12pm EST
Adore locked up from disk drive failure. Always seems like it is
the new mail drive. This time in sd9e.
10:00pm
Adore locked up again, this time no sign of what was wrong,
totally jammed, no cursor no nothing. When I went to reboot, the
monitor wouldn't come on.
I took out 4 pieces of the new memory and it would boot, I put
number 5 back in, and it gave asynch memory fault.
So we are running on 4 for the moment.
06/20/99 Sunday 6:21pm EST
Adore taken down for emergency service.
All memory replaced and upgraded from 145M to 256Meg with
Sun bar code memory.
06/20/99 Sunday 12:35pm EST
Saturday 6/19/1999 mail was moved from adore to mx starting
at 2pm. Most systems were online again at 4pm, but some remote
users were not able to send through our smtp server until about 9pm
due to failure to copy the pophash.db file over the smtp.lightlink.com
Adore locked up at about 1am 6/20/99, and didn't notify
us of the failure until 4am when it was rebooted. Crash was
asynch memory fault again. Memory will be place tonight.
06/15/99 Tuesday 3:40pm EST
Major net outage for most of 6/14/99
Adore locked up at 8:45am this morning, then freed itself, popper
unavailable.
06/13/99 Sunday 9:03pm EST
Adore locked up again at about 8:30pm. Load 128.0
Seems to be the mail drive getting stuck in a wait state and
everything piles up on top of it. Everything else was running fine, but
anything accessing that drive failed, like ls or df etc.
Mostly poppers were building up. Couldn't kill them off with
killer.
So...
sendmail killed, inetd killed, everyone bounced.
Mail partition moved from sd8a to sd9e and popdrop from sd8g to
sd9g.
Sendmail restarted, inetd restarted, and logins allowed.
Hopefully that will be the end of this.
06/11/99 Friday 4:15pm EST
A mirror drive on adore locked up this morning at about 3am,
causing the load on adore to go to 630.0, preventing people
from getting their mail and jamming out shell users.
This also locked up light, which shares adore's home directories,
which then prevented others from signing on. This is in part why
light and adore must be separated.
The drive is undergoing testing, it was the mail mirror drive,
and locked up during the nightly copy of the mail directories to the
mirror.
A few users had their pop mail stuck in a half way state when
adore came back up, and could not get their mail. These have all been
cleaned up.
06/05/99 Saturday 9:51pm EST
Scheduled down at 6pm.
Mail was moved from light to adore with little problem.
Both MX and popper functions were moved.
Spool directories were also moved. Light no longer supports
mail.
06/04/99 Friday 6:31pm EST
Scheduled down at 6pm.
/var was moved from /dev/sd8a to /dev/sd9f
/var/spool/mail was moved from /dev/sd8a to itself as an outer directory
Everything went flawlessly.
05/23/99 Sunday 10:05pm EST
Scheduled down at 6:00pm, news2, ftp, majordomo were off line
until about 7:30pm to put rollers on the tables that hold them. Then
troubles with a hub resulted in news not drawing new news, so hub was
replaced.
04/29/99 Thursday 12:18pm EST
SMTP server (emerald) went out of control at about 11:15am, no
known reason at the moment. Load was at 15, process table full,
sendmail requests were being denied.
04/21/99 Wednesday 10:42pm EST
Adore crashed hard. New memory is on order.
04/11/99 Sunday 11:18am EST
Adore rebooted itself this morning at 9:36am.
04/05/99 Monday 9:40pm EST
Modem 3 causing ring no answers all day long.
Also modem 54.
Sigh
04/01/99 Thursday 12:14am EST
admiral (news) was down since 22:00 3/30/99 due to a smoked root
drive. pingers did not catch it because admiral had been removed from
the ping list for reasons lost to antiquity.
No one reported that news was down probably because it did not
affect our main news reading machine, however no new news was coming
in for over 24 hours.
Main drive has been replaced by the mirror, and more stringent
monitoring programs will be written to make sure this and other
machines do not go down without my finding out about it.
03/28/99 Sunday 10:37pm EST
ftp was taken down to fix a recalcitrant ethernet card on the
private backbone.
03/28/99 Saturday 9:23am EST
Remote relaying through our smtp server was intermittenly broken
again for short whiles. This was caused by two sepearate machines
trying to update the authorized IP database, thus wiping out each
others work.
03/26/99 Friday 11:00pm EST
FTP and MAILING list machine were taken down for upgrades.
CPU fan was replaced on emerald (ftp) and power supply was replace
on gem (mailing list).
03/25/99 Thursday 1:33pm EST
In preparation for upgrades to gem and emerald, smtp.lightlink.com
was moved to mx. The anti spam database that updates legal IP
numbers for remote users to use our smtp server was not updating
properly on mx, so from about 8am until now, remote users have
not been able to send e-mail.
03/23/99 Tuesday 11:00pm EST
Adore rebooted. Telnet was not working, OS beginning to die?
Light was still jammed, so rebooted also.
03/08/99 Monday 4:57pm EST
Nysernet lost the routes to the T3, so although the
routers were up, our T3 customers were unable to get anywhere.
207.127.235.0/24
207.127.234.0/24
207.127.233.64/26
03/08/99 Monday 10:02am EST
External net was down from 4:55am til 10am. Nysernet
replaced a major router downtown.
03/06/99 Saturday 7:28pm EST
Momentary bug in new account creation programs caused /etc/passwd
permissions to be not world readable causing the id command on adore
to fail, causing shell users errors in the prompt script.
03/03/99 Wednesday 6:17pm EST
adore crashed from asynch memory errors. Probably needs
to be replaced.
02/28/99 Sunday 3:20pm EST
web server was locked up, possibly due to not being restarted
properly.
02/28/99 Sunday 12:28pm EST
Adore suffered a major down at 11:45am this morning causing light
to lock up. It was a partial crash which did not set off the alarms,
so we didn't find out about it until 12 noon. Normally adore reboots
itself after such events, but this time it just crashed and stayed
crashed. Had to cold reboot both adore and light causing minor disk
damage, which took about 20 minutes to rebuild.
It's just this kind of thing that demands that we separate adore
and light and all the machines from each other.
02/19/99 Friday 02:23am EST
Web logs were lost for Feb 16 and 17 due to a program bug.
02/17/99 Wednesday 4:07pm EST
Light locked up for unknown reasons, millions of popper and
sendmail processes all waiting to run, couldn't do top or anything.
Rebooted.
02/16/99 Tuesday 11:00pm EST
Single modem giving ring no answer on 5026. Its been cleared.
02/15/99 Monday 11:40pm EST
Light rebooted to rearrange ftp. Mysterious failures in ftp have
occured since the newest version was installed. The newest version
did not work at all, and then the old version stopped doing lists
properly.
We have light working properly now, although we still have no
idea why the original software started to fail as it is still failing.
Been running dynamically loaded ls's forever, now they don't work
and have to use a static ls. Don't ask me.
02/02/99 Tuesday 7:28pm EST
Light rebooted, getting sluggish perhaps from long term
memory leaks. 71 days is too long to go without rebooting.
Apparently radius authentication server did not start
properly causing people to not be able to sign on for
a while after the boot, until 8:01pm to be exact.
01/31/99 Sunday 6:00pm EST
Scheduled down. Adore rebooted with new tape drive in place.
Added 3 9.1 gig drives to adore and 1 to light.
01/29/99 Friday 5:26pm EST
Bad routing at AOL has caused ICQ and Instant Messenger to
file across the Nysernet backbone.
01/23/99 Saturday 8:29pm EST
ISDN2 modem bank rebooted to clear out bad modem, everyone
on that bank was bounced.
12/14/98 Monday 12:03pm EST
Adore crashed this morning at 9:42am for unknown reasons.
12/13/98 Sunday 5:39pm EST
We had momentary but major network outages today due to a failing
ethernet link and a hub. Changes to the network settings made things
worse, and it took about 2 hours to undo the damage that we did
trying to fix the original problem.
Momentary outages on adore and light would have been experienced.
12/06/98 Sunday 12:23am EST
All modem banks have been changed to allow ascii users to
sign on, they are rlogined to adore at the password prompt.
This means all ppp users *MUST* use non scripted PAP.
11/17/98 Tuesday 6:45pm EST
We were spammed last night, and I set the mail limit
to 12 connections at a time, which was too low. Today a number
of people were not able to send mail because of this, it has
been raised to 24.
11/15/98 Sunday 4:15pm EST
gopher/help system on adore was left down after last nights
system down time. It is working now.
11/14/98 Saturday 6:12pm EST
Scheduled down lasted 6:00pm - 6:15pm.
Light upgraded to kernel jesw which supports vifc.
Adore also down during upgrade, no changes made.
11/13/98 Friday 11:07pm EST
Web hit log files are being moved to majesty, there will
be some interruption in availability but we are trying to not
lose hits. They should be functional again in a day or two.
11/08/98 Sunday 8:15pm EST
We got isdn and isdn2 flashed for V90, took much longer than
anticipated. All X2's are now X2/V90 capable.
10/27/98 Tuesday 3:32pm EST
16 new modems in, hunt group seems ok, 2 modems were not set
properly, gave busy signals, all should be working properly now.
10/05/98 Monday 7:34pm EST
Jammed modem on isdn3 cleared. Netserver rebooted, bouncing
everyone on isdn3.
09/30/98 Wednesday 12:17pm EST
Nysernet is having significant problems at various routing
points to the main backbone. They are aware of the problem, and
are working on it. Network slows have been happening for a few days,
and may continue to happen for a while.
09/29/98 Tuesday 1:04pm EST
There appears to be a net outage of some kind, stopping
communications from going out or coming in.
09/20/98 Sunday 9:17pm EST
System taken down for scheduled maintenance at 6pm until 7pm.
Lights root drive was replaced by its mirror and a new mirror put
in place.
Adore's mirror drive was replaced by a new mirror as big as the
root drive.
Tape drive would not work with adore, tried multiple different
tape drives and cables and terminators. It seems to be a problem with
the one drive that is on the scsi bus, it seems to screw up the
termination when the tape drive is on there. Put the tape drive
before and after the disk drive, it made no difference.
Adore is still without its own tape drive.
One UPS powering top Multitech modem bank is definitely gone,
battery will be replaced next time around.
09/20/98 Sunday 1:15pm EST
We are getting fatal disk errors on light's root drive.
This is a very bad sign and may involve replacing the root
drive with its mirror.
09/19/98 Saturday 08:02am EST
At 3:07am light started to get fatal errors on the swap partition
of its root drive.
This apparently caused the failure of named and primary name
service.
At the same time majesty seems to be unpingable from our elmira
routers, so secondary name service for those people failed also,
preventing Elmira customers from getting on line.
For some reason lost to antiquity the code that monitors named
was set to NOT try and restart it automatically, I have turned this
back on.
I was not pinged because the pinger modem was off due to system
work on majesty last night.
Sheesh.
The fatal disk errors on the root swap partition are not a good
sign, and may result in a crash in the near future. We have root
mirrors in place and fully updated in case we need to do a swap.
09/09/98 Wednesday 6:51pm EST
Looks like all system jobs ran twice last night, causing
double entries in webstats and other areas. Not sure why.
08/31/98 Monday 11:03pm EST
Upgraded our news reading server to dnews 46r. The upgrade went
smoothly, but an incorrect directory entry resulted in the server using
default files for expiration. Thus all articles were expired.
The spool will rebuild as time goes by.
08/29/98 Saturday 4:36pm EST
News has been suffering a massive planet wide denial of service
attack involving an overload of newgroup and sendsys messages. This
has caused the load on news servers to skyrocket, including ours, and
caused the flow of normal news to come to a crawl.
Patches to the news server have helped to alleviate the problem,
but the attack continues.
08/20/98 Thursday 7:04pm EST
Downtime 1 hour.
Scheduled down for light and adore did not go well.
64 meg was put in adore.
Two new root mirror drives would not boot properly, on either
machine, possibly due to a jumper error which I hadn't accounted for.
New Tape drive on adore seemed to cause scsi time outs on
sd0 which is real strange.
Presently both systems are as they were except adore has 64 meg
more memory.
08/07/98 Friday 8:12pm EST
Scheduled down for light and adore starting at 6:00pm.
Home drive on light was moved to adore, and mirror home drive on
adore was moved to light.
This broke cgi's which reside in the home directory stupidly,
which can not execute setuid from adore on light.
This also broke listproc for the same reason.
These are now fixed.
/home/www -> /homewww
Listproc moved to adore.
08/07/98 Friday 11:39am EST
Link to Elmira was down from 8:35am until now. Problem
with the Frame Relay line was called into EMI and fixed at about
11:30am.
07/19/98 Sunday 2:39pm EST
X2 modems rebooted, kicking everyone off
Apparently routing started to fail for some users at about 2am
Sunday morning. Tech support got about 20 calls on it. It may not
have affected everyone.
One user reported trying all 3 X2 modem banks, failing on all of
them, which is real weird as they are independent machines.
No idea what happened.
07/19/98 Sunday 2:14pm EST
News was effectively down from about 11pm last night until now
due to a routing error during a reboot that went unseen. Not related
to the X2 problem above. (I think).
News was coming into news1, but not being transferred to news2,
readers were able to read from news2, but weren't getting any new news
during this time. Expiry on news1 has probably removed some articles
which are now lost.
07/05/98 Sunday 01:14am EST
gopher and the help system did not restart properly during
last reboot. Should be working now.
07/05/98 Sunday 12:27am EST
isdn3 was off line due to a network lock up, perhaps for a few
days. Everyone was bounced off of isdn2 accidentally in an effort
clear this out.
07/04/98 Saturday 7:26pm EST
Raw web access logs were lost last night due to a long standing
but subtle programming bug.
Logs for 7/1 7/3 and 7/4 have been recovered, the rest are lost.
Only raw hit logs were affected.
07/04/98 Saturday 6:13pm EST
Light taken down at 6pm to replace failing backup tape drive.
07/01/98 Wednesday 18:23pm EST
adore crashed for unknown reasons, rebooted itself.
06/30/98 Tuesday 10:00pm EST
world read perms turned back on ftp due to failure
of cd command to print directory name properly. What a pain.
06/28/98 Sunday 4:24pm EST
shell help and gopher were broken by last nights ftp move. These
should be working now.
It is not immediately clear that running files across NFS between
two different platforms (SunOS and Linux) is going to work properly.
At present the SunOS executables for ftp are on the Linux box, which
probably was not intended, but seems to work anyhow as they are
being read across the network before executed on the Sun.
Probably what we want is for the /ftp/pub directory to be exported
and leave the rest as it is.
06/27/98 Saturday 5:56pm EST
ftp on light disabled to move it to emerald
Both light and adore needed to be rebooted to let go of
the original ftp directory. 7:30pm
06/26/98 Friday 4:50pm EST
Harmony locked up, had to be cold booted, bouncing all users.
06/22/98 Monday 9:14pm EST
Light was put through an emergency reboot to see if it would
clear out a persisting problem with excessive syslogs.
06/17/98 Wednesday 2:17pm EST
Sendmail stopped allowing remote mail through our system to
remote users at about 10:11am this morning due to a system screwup.
It should be working fine now.
06/16/98 Tuesday 11:57am EST
NetServer I isdn2 had a jammed modem. Rebooted both isdn1 and isdn2,
bouncing everyone off.
06/13/98 Saturday 12:19pm EST
Harmony bank 1 and usr and isdn modem banks were rebooted,
due to a jammed UPS which had to be rebooted.
06/11/98 Thursday 12:50pm EST
2 modems on harmony were found to not be picking up. They were
causing ring no answers for a few days.
06/06/98 Saturday 9:42pm EST
We suffered a slow but catestrophic failure of majesty over the
past week. It began crashing routinely, every few hours. This has
caused news server to be spotty although news2 which is the reader
machine did not go down, it was not able to have all the articles
available that it would other wise have.
Presently majesty is in sick bay, and news.lightlink.com has been
replaced with a pentium 333 machine. Things may be rough for a while,
but it probably won't crash.
06/03/98 Wednesday 7:30pm EST
Router to Elmira rebooted, interruption of service for 3 minutes.
06/02/98 Tuesday 10:37pm EST
Sendmail was rebooted improperly and started to fail
to let remote users post through lightlink to remote spots.
06/02/98 Tuesday 9:26pm EST
Upgraded news software today on news.lightlink.com to make things
faster. In process reduced size of active file getting rid of empty
groups. This cut the size from almost 2 meg to less than 1 meg which
should make news reading faster.
05/30/98 Saturday 07:47am EST
Harmony seems to have rebooted itself for unknown reasons.
This kicked everyone off of the multitechs.
05/27/98 Wednesday 2:47pm EST
news1 crashed at 2:19pm. Probably from too many news feeds,
or a failing drive. Haven't figured out which yet.
In any case a new machine is being built which can handle
the load.
news2 was rebooted in the process.
05/25/98 Monday 12:58pm EST
News1 crashed again at about 12:30pm. No warnings went off
because it seems to have half crashed leaving its ethernet port operational.
All drives have been reseated including power sockets. If it crashes
again, we will reseat the memory, and if it crashes again, we will replace
it with a new super news machine that we are considering in the wings.
News2 was not affected except some news may have been lost.
05/24/98 Sunday 4:35pm EST
News1 suffered a major crash at 3am in the morning, wiping out
the history file. The entire news system was rebuilt this afternoon.
News2 was not affected, except that incoming news will have been sparse.
News is running properly again.
It is possible that news1 is loosing a hard drive, so this
may happen again.
05/18/98 Monday 11:18pm EST
News2 crashed this afternoon and needed to be rebuilt. The
history file was lost and rebuilt, and all binary files were lost.
Most other news was not lost.
Posting was not working properly until now due to an incorrect
permission on one of the news drives.
05/14/98 Thursday 02:30am EST
Adore's ethernet locked up tonight for unknown reasons.
This tied up light because of the nfs mounted drives.
Adore was rebooted but still would not telnet to itself,
indicating hardware failure in the ether port. Turned it off and on,
and it started working again.
Worse comes to worse, adore can have another ethercard put in it,
or people can use light for shell if adore dies off completely.
05/13/98 Wednesday 6:57pm EST
Shell users were locked out of mail today, possibly for 2 hours.
The permissions on the password file were set to non world readable
for reasons that we suspect but don't know for sure, nothing serious,
more like a bug in the account creation process which has to write to
the file to create the new account.
The problem occured we believe starting at 2:41pm and lasted
until about 5:00pm
Monitors have been put in place to beep us within one minute
should this happen again.
05/11/98 Monday 01:35am EST
isdn1 277 0356 rebooted to clear out jammed lines.
04/25/98 Saturday 8:46pm EST
news1 and news2 were taken down this afternoon to install second
ether cards in them. This will allow news2 to take news from news1
over a private internal network releiving some of the load on our main
network.
news1 is being moved to the T3 shortly
04/20/98 Monday 3:12pm EST
Modem 3 locked up on isdn1 2770356, picks up but doesn't answer.
Reset bouncing modems 3 and 4.
04/16/98 Thursday 6:17pm EST
Scheduled Down. Harmony modems and power supplies were
reseated a number of times bouncing everyone at least twice.
Reseating needs to be done periodically to assure that the
contacts are clean to avoid spurious drops or modem failures.
04/16/98 Thursday 02:53am EST
Light crashed during nightly mirroring due to out of mbuf errors.
This will probably go away with scheduled changes in which machine has
which drives. Presently the mirroring takes place over the net and
stresses SunOS's ability to handle the traffic.
04/15/98 Wednesday 08:54am EST
Mail loop in the ccounsel mailing list caused a crash
of gem at about 8am which was not caught until about 2pm.
Have installed monitoring to restart mail and beep
if it dies.
04/04/98 Saturday 3:07pm EST
isdn1 lost its ethernet address, causing people to not be
able to go anywhere on the net. Rebooted, bouncing everyone.
04/04/98 Saturday 12:21am EST
eggbots and irc servers killed on adore to track down source of
DNS hits coming in on light. DNS installed on adore, so it can
handle its own name service now.
04/01/98 Wednesday 4:06pm EST
Web server taken off line for 5 minutes to track down what's
driving light out of control.
03/23/98 Monday 3:06pm EST
Modems 20 and 26 on Harmony have been giving ring no answers for
a few days. Have busied them out. The entire box needs to have all
its cards reseated and rebooted. This will happen next down time.
03/22/98 Sunday 6:22pm EST
Light and adore taken down from 6:00pm to 6:25pm to add memory
Memory increased from 128M to 320M
03/14/98 Saturday 5:56pm EST
News2 was unscheduled down for about 15 minutes to allow rewiring
of the machine room.
03/11/98 Wednesday 12:36am EST
cisco router rebooted, internet connection lost for 2 minutes.
Router had a bad arp entry for 205.232.34.91 which is modem 9 on
the USR 277 1076 modem banks. People signing onto that modem could
not get out into the internet. Bad arp entry came from a typo I made
over a week ago, so this has been going on for a long time.
03/08/98 Sunday 1:03pm EST
277 1076 modem bank rebooted, bouncing everyone
03/05/98 Thursday 12:29am EST
News2 was down from 9pm to 12am due to a faulty ether connection.
The ether connections were changed to add in a new hub.
I have no idea why the system didn't beep me.
It looks like the monitor program 'pinger' was locked up, not a
good sign.
What shall monitor the monitor program?
02/23/98 Monday 5:00pm EST
Nynex installed 4 new modems which are now on line.
Part of modem bank 1 was knocked off line due to a loose cable
twice bouncing everyone on that section.
T1 was physically moved to a better location causing a 4 minute
network outage.
02/22/98 Sunday 8:30pm EST
Cisco router upgraded to 11.1(17). Network down from about 6pm
to 6:45pm.
02/21/98 Saturday 2:28pm EST
Momentary net outage for unknown reasons.
Rebooted cisco router and kentrox CSU/DSU.
That didn't clear it up, but then it cleared itself up shortly after.
02/20/98 Friday 09:00am EST
X2 NetServer was rebooted by accident, every one bounced.
02/18/98 Wednesday 5:17pm EST
Adore was under attack today from someone at 130.238.203.192.
Rebooting the system did not end the attack.
Used tcpdump to sniff his packets and then blocked him at the border
router.
02/17/98 Tuesday 9:27pm EST
The X2 NetServer was producing busy signals even though there
were modems free. I rebooted the first X2 NetServer and the
USR V34 Server bouncing everyone off.
It seems to have cleared up.
02/14/98 Saturday 6:31pm EST
277 0356 X2 lines locked up for unknown reasons this afternoon.
It may have had to do with a test Cisco 2501 we put on the network to
test RIP2, a routing protocol. I presently do not know why that would
have cuased the problem, particularly since the new router was on line
since last night with no reported troubles. Rebooting the X2 banks
did not fix the problem, but taking the Cisco off line did.
02/11/98 Wednesday 12:19am EST
Light rebooted itself last night at around 3am in the morning.
This caused authentication services to switch over to majesty. When
light came back up, I forgot to reset authentication back to light.
For unknown reasons, majesty is refusing to authenticate some people,
so for much of the morning many people were not able to sign on.
Mystery solved. A newer version of the authentication code was
running on majesty, that was never supposed to be put in service, and
it was dying on config files from the older version that are copied
over from light to majesty every night to keep the password files in
sync.
02/07/98 Saturday 10:52pm EST
Adore crashed again. Took out another memory chip.
Gonna keep doing this until we find out which one is bad.
02/05/98 Thursday 11:36pm EST
Modems 49-72 rebooted, bouncing everyone off. Bad modem 68
is working again.
02/04/98 Wednesday 12:46pm EST
Adore locked up but didn't crash.
This caused the modem authentication software to fail on light so
some people were not able to get on.
Sun tells us its possibly a bad memory chip. I have replaced
the chip in slot 0.
01/05/98 Monday 9:25pm EST
Majesty was down from 2:41pm this afternoon. News2 was not affected,
but new news coming in was stopped until now. Various cross checking
pingers failed to go off, new code with bugs etc. My fault.
01/01/98 Thursday 3:05pm EST
A number of people have not been able to get signed on to 277
5026. When light crashed this morning, authentication shifted to the
back up server, which for reasons still unknown are rejecting certain
people's passwords.
01/01/98 Thursday 04:10am EST
Light crashed due to mbuf map full error.
12/08/97 Monday 10:51pm EST
Light was crashed by a user playing with an exploit. The
exploit has been closed on all three machines.
The anti relaying code was not restarted when light rebooted
so some remote users were not able to send mail.
My apologies.
12/08/97 Monday 3:25pm EST
Bad modem was causing ring no answers, taken off line.
12/05/97 Friday 5:37pm EST
X2 rebooted itself.
12/05/97 Friday 02:20am EST
X2 modems rebooted, bouncing everyone about 12 midnight.
11/28/97 Friday 2:41pm EST
pm2-elmira is down. Frame connection to elmira rebooted, no
change.
pm-elmira is fine.
11/24/97 Monday 01:26am EST
Aurora (new2) rebooted a number of times at around 11 to 12am,
in order to test various things.
Apparently it hasn't been posting properly to usenet for about 3
days since the 20th. This was a result of starting it from the
startup profile /etc/rc.d/rc.local. For unknown reasons it runs but
won't post.
11/22/97 Saturday 7:39pm EST
Aurora (news2) taken down for 3 minutes to replace ethernet
card. If this doesn't stop the crashing, we will replace the mother board.
11/21/97 Friday 7:15pm EST
Tested an attack against the Cisco border router, and brought
it down for about 2 minutes.
11/21/97 Friday 12:38pm EST
Aurora locked up at 8am. Because of the nature of lock up, no
warnings went off. I believe it is caused by bad ethernet code for
the card we are using. The card will be replaced shortly.
11/20/97 Thursday 10:38am EST
Aurora (news2) locked up its ether port again. Various beeping
mechanisms in place to warn me and reboot the system automatically
failed. Will need to do more testing on this. Don't know why the
etherport is locking up. Have replaced the ethernet card, the mother
board may be going bad.
11/17/97 Monday 2:01pm EST
Overview database on news2 has been erased. It will rebuild slowly
as people download articles. Downloads may be slower for a while. Hopefully
this will fix the missing news article problem.
The Overview database had headers in it for articles that were
expired so they show as available in netscape but return ARTICLE
EXPIRED when customers hit on them.
11/16/97 Sunday 10:14pm EST
News2 rebooted. Was running with 64meg instead of 128meg due to
improper boot last time around. This is fixed.
11/16/97 Sunday 9:27pm EST
News2 suffered a few crashes over the past few days from ethernet
lockup. Apparently some of its history, index and overview files got
corrupted. This may have resulted in headers being downloaded without
bodies. We have rebuilt the data base and run an expire, this
hopefully will clean it up. If not, it should clean up by itself as
time progresses and things get naturally expired.
11/14/97 Friday 6:25pm EST
Modems and news stopped for 5 minutes to reset a warning light on
a UPS. Everyone bounced. The UPS seems to have cleared out its bad
battery indicator light, but it may come back.
11/14/97 Friday 4:16pm EST
News2 down for about 15 minutes to replace a bad ethernet card.
11/06/97 Thursday 11:57pm EST
X2 rebooted itself.
11/04/97 Tuesday 4:48pm EST
I had to take light down to clear out a module that was
misbehaving badly.
I was trying to trace a hacker who was trying out passwords on
our system, and in loading the tracing software, it took over the
console and wouldn't give me control back.
10/31/97 Friday 12:24pm EST
Majesty locked up from about 5:45am from scsi disk failure,
eventually dying from process table full.
10/22/97 Wednesday 1:27pm EST
Aurora locked up at 5:30am this morning and did not beep me because
the modem was off.
News restored at 1:30pm
10/20/97 Monday 11:29pm EST
Aurora (news2) locked up for unknown reasons, rebooted.
10/19/97 Sunday 7:02pm EST
Majesty (news) crashed from mbuf map full.
10/17/97 Friday 5:47pm EST
X2 rebooted again to clear polluted arp cache.
10/17/97 Friday 3:41pm EST
USR modems down, box not responding.
X2 modems rebooted by accident.
10/12/97 Sunday 10:23am EST
Looks like there was an external network outage at around 8:30am.
10/12/97 Sunday 10:19am EST
Romance's network locked up for unknown reasons, for about 3
minutes. Rebooting cleared it out.
10/12/97 Sunday 05:23am EST
Internal network was locked up by unknown causes from 3:15am
to now. Incoming traffic went to 100 percent and outgoing went to 0.
Shutting down all machines did not clear the condition although
during the final reboot of all of them it managed to clear itself.
Nysernet and Sprint are looking into it.
10/08/97 Wednesday 10:39pm EST
X2/ISDN server totally locked up, rebooted bouncing everyone
Talked to USR today, they have escalated the service call.
10/07/97 Tuesday 9:48pm EST
External network seems to be partially down.
10/07/97 Tuesday 7:41pm EST
Modem 12 on NetServer locked up.
10/06/97 Monday 8:11pm EST
X2/ISDN server locking up routinely. Have asked USR to escalate
the complaint.
10/04/97 Saturday 12:52pm EST
News crashed at 6am from disk full errors.
10/03/97 Friday 11:41am EST
X2 modem server rebooted to clear out jammed lines.
10/02/97 Thursday 11:58pm EST
Main router locked up a few times tonight for unknown reasons.
All access to the net was blocked out.
10/01/97 Wednesday 4:30pm EST
News2 was down for about 30 minutes at 2pm. Loose cable on
a hard drive caused it to crash, cable is damaged but working.
09/29/97 Monday 9:13pm EST
News2 rebooted by accident.
09/29/97 Monday 6:28pm EST
Modem 12 on the X2 server is locked up.
Evidence indicates that the NetServer rebooted itself at 16:05
this afternoon, bouncing everyone.
Called USR, they said modem is picking up on digital calls but
not analogue calls, that this is a Nynex problem. USR said the ISDN
line is presenting an analogue call as a digital call which is
incorrect.
Called Nynex, reported the line that is not picking up.
Gotta leave the modem locked up so Nynex can see it fail.
09/26/97 Friday 3:46pm EST
Modem 2 on the *NEW* Netserver locked up.
09/21/97 Sunday 12:19am EST
Yesterday when light crashed and was rebooted, an older
version of the apache web server was started which caused
some anomalies including failures of freebie domains to go
to their proper pages.
This is fixed.
X2's seem stable.
X2's rebooted to install newest code.
Everyone bounced 12:24am
09/20/97 Saturday 9:16pm EST
News has been jammed most of the day. It was running but
throttled due to a binary fill up this morning. When I restarted
it this morning, it didn't restart properly, but kept running anyhow,
so I never got beeped.
09/19/97 Friday 9:47pm EST
Called Nynex about the 8 ISDN lines going up and down all night.
09/19/97 Friday 8:48pm EST
Light crashed. I quit out of Xwindows and got a cpu panic.
Been up way too long.
09/19/97 Friday 6:50pm EST
Nynex ISDN seems to have gone down. All X2 channels got dropped
and gave busy signals.
NetServer won't resynch up.
Nynex ISDN seems to be going down every 10 minutes for about 5
minutes.
09/18/97 Thursday 9:38pm EST
X2/ISDN server replaced. Down for about 3 hours.
Took it back out because it wasn't working, but it was
my problem, so the new one is back in again. Looks like its working.
09/16/97 Tuesday 2:33pm EST
X2 server locked up. Worked with USR on it for a number
of hours. Doing one last test before we replace it.
Please continue to report modem failures.
09/13/97 Saturday 4:17pm EST
NetServer Modem 2 locked up, call into USR to study it.
09/05/97 Friday 7:39pm EST
X2/ISDN NetServer locked up on modem 1, giving no answers.
Rebooted.
09/04/97 Thursday 1:32pm EST
External net is down.
08/31/97 Sunday 2:58pm EST
Nynex Frame Relay line died at 11:50am They are working on it.
08/25/97 Monday 5:02pm EST
Harmony password file was lost at 3:50pm for unknown reasons.
Signons failed until about 4:10pm when we noticed it.
08/22/97 Friday 12:14am EST
X2 modems rebooted twiced for system work.
Everyone was bounced
08/19/97 Tuesday 02:13am EST
Password file for X2 modems was destroyed again at 12 midnight. This
time I found out what did it.
08/18/97 Monday 11:23am EST
Password file for X2 modem banks was corrupted last night,
Down since 12 midnight.
08/16/97 Saturday 4:57pm EST
X2 port 4 is apparently hung, I did a hard reset on it, which
booted 3 too.
08/11/97 Monday 4:18pm EST
Majesty crashed from out of memory errors. This caused light
to stick during shell logins from NFS mounts.
08/09/97 Saturday 4:48pm EST
Majesty crashed, out of mbufs.
08/07/97 Thursday 12:09am EST
I tried to regather articles from the last day to make up for the
0 day expire of this afternoon. news2 promptly went and filled all 4
disks all the way and bombed, something I don't quite understand
because there wasn't 12 gig of news to steal from news1.
So I had to expire that too, but by this time all those articles
were in the history file, even though they were now expired, so I
couldn't go back and reget them again without starting from scratch
which is what annoyed everyone last time.
So, right now everything looks stable and is working properly.
Most of this was my unfamiliarity with the new software, and its lousy
manual.
The articles will fill in again as new ones come in.
08/06/97 Wednesday 8:06pm EST
news2 was put through a 0 day expire this afternoon after
it filled up one drive leaving the rest empty. This bug is
fixed but to restart it properly we had to empty the one drive.
All four drives are filling up now.
08/04/97 Monday 6:29pm EST
news2 rebuilt from scratch using latest, greatest 4.2g
Articles will refill in.
08/02/97 Saturday 10:54pm EST
news seems to be stablized.
news2 however got hopelessly corrupted.
It was been wiped clean, and groups will have to reload
themselves.
07/26/97 Saturday 8:08pm EST
inn1.6b2 has destroyed the news history file. news will be down
for a number of hours. news2 is fine.
news will probably be up again around 9am plus or minus.
07/25/97 Friday 10:43pm EST
Getting ring no answer on modem 43. Taken off line.
07/24/97 Thursday 4:42pm EST
Apparently we suffered some kind of SYN attack causing the
web server to lock up and name service to freeze.
07/23/97 Wednesday 8:56pm EST
News has been down for a few hours while we install inn 1.6b1.
Not going smoothly, tin doesn't seem to want to read. rn works fine.
07/23/97 Wednesday 5:32pm EST
NetServer rebooted itself.
07/22/97 Tuesday 2:38pm EST
Significant net outage.
07/22/97 Tuesday 12:44am EST
ISDN/X2 rebooted multiple times. Uploaded newest code,
Courier I works worse than before.
07/20/97 Sunday 4:51pm EST
The system password file was corrupted at 5:10am this morning due
to a bug in our own security software. Logins to light and e-mail
retrieval were prevented until about 6:00am when we finally figured
out what had happened.
07/19/97 Saturday 1:24pm EST
NetServer ISDN cold booted to clear out two jammed channels, s11 s12
07/13/97 Sunday 2:25pm EST
News2 will be slow today, we are rebuilding the history database.
While the rebuild is taking place, many articles will show up as
there while in fact not being there, this may cause problems with your
newsreaders.
Lots of spam in alt.sex is rendering the groups totally useless.
Expiration on alt.sex* has been set to 0 days, expiration on all
other groups has been set to 7. Binaries are still 4.
0 days doesn't mean we aren't carrying them, it means they
get expired fully every night.
Filtering of postings crossposted to more than 10 groups has been
turned off on news2. News hasn't been filtering.
07/09/97 Wednesday 01:03am EST
ISDN/X2 server rebooted at about 11pm or so. Then again a while
later. It first locked up with endless radius daemons sending majesty
to load 40, then after it was rebooted, it wouldn't do ISDN any more,
X2 worked fine, and *V2=0 worked fine, but *V2=5 wouldn't work at all..
Turning it off and on cleared it out.
07/08/97 Tuesday 12:20am EST
ISDN/X2 server rebooted.
It was in some terrible loop with the radius server on majesty,
causing MILLIONS of radiusd's to be spawned driving the load on
majesty up to 50.
06/30/97 Monday 9:01pm EST
Net Server IDSN rebooted to set speeds to 115200.
Testing slow speeds.
06/28/97 Saturday 2:14pm EST
Nynex stopped by to check errors on the T1, trouble report
originated by Nynex monitoring. T1 was down for about 5 minutes causing
interruption in connection to the net.
06/27/97 Friday 12:40am EST
An experimental version of sendmail died silently at around
11:17pm for unknown reasons.
Outgoing mail was down for about 30 minutes until I was
informed of the down.
I will place sendmail in the system monitor so this won't
happen in the future.
06/21/97 Saturday 10:27pm EST
Light taken down due to network instability. Getting le0 out of
mbuf errors, tremendous activity coming in on le0 or going out, not
sure. ciscoin and ciscoout don't show anything special.
Getting mbuf errors even while in single user mode, that
indicates tremendous hits on light coming from outside.
Installed new kernel while we are at it, jesv. Light booted
3 times without incident, pray hard.
Looks like it was a ping attack, we will get the joker the
next time it happens.
06/15/97 Sunday 10:27pm EST
Light crashed again at 18:57pm and would not reboot at all.
After multiple failures to reboot, I turned it off, pulled the
memory and reseated it, and did the same with all cables and internal
disks. Then it rebooted, but it may simply have changed its mind.
It is possible the crashes are from a dying root drive, there is
some indication that there were problems with the swap space, and boot
sectors. There is another mirror drive ready to go if it crashes
again.
If that's not it, it's going to be rough finding out what it is.
We have a second server being prepared which can take over
the functions of light if necessary, but it is far from ready to
do this on short notice at this time.
06/15/97 Sunday 3:41pm EST
Light crashed for unknown reasons.
Watchdog reset.
Came back automatically and tried to reboot, said bad boot block
or something, I stupidly didn't write it down.
Then I rebooted it by hand and it booted and ran a *LOT* of
fsck's on all drives. When it went to reboot, it failed with MEMORY
OUT OF ALIGNMENT, no tracebacks this time.
Then it booted by hand fine.
We may have a bad boot block on the main root disk, but this
doesn't explain why light crashed in the first place.
06/12/97 Thursday 6:35pm EST
Light and related systems taken down to replace UPS.
Light did not boot smoothly, it gave MEMORY OUT OF ALIGNMENT followed
by a traceback. This is new and not good.
06/11/97 Wednesday 7:55pm EST
Network interference from romance being down caused aurora to
lock up due to NFS mounts. News2 down for about 20 minutes.
06/06/97 Friday 11:31am EST
Elmira link was down for about 45 minutes for unknown reasons.
Rebooting the frame relay router at our end seems to have
reestablished the link.
06/02/97 Monday 3:15pm EST
Light crashed from running out of mbufs, a bug in the apache
web server.
It crashed at 15367/15744 mbufs.
finwait was 2474
05/24/97 Saturday 11:50pm EST
Majesty (news.lightlink.com) was down for about 6 hours from
a subtle news crash. I did not notice and no one called to inform me.
Aurora (new2) was not affected.
Some articles may have been lost.
05/20/97 Tuesday 6:14pm EST
Light taken down to install new kernel and make security patches.
Running Kernel jess
6:00pm -> 6:15pm
05/18/97 Sunday 6:40pm EST
Light taken down at 6:00pm to rewire part of machine room.
05/18/97 Sunday 3:00pm EST
News (majesty) was down for about an hour to patch the OS.
Majesty is theoretically fully patched at this point.
05/07/97 Wednesday 01:01am EST
Sendmail and web taken off line at about 12:30am for a few minutes
to deal with an incoming relay spam.
05/05/97 Monday 02:09am EST
Harmony modem bank one reset, bouncing everyone.
04/30/97 Wednesday 11:32am EST
Light crashed.
04/29/97 Tuesday 6:20pm EST
Nynex has taken down the huntgroup to fix an earlier problem.
Everyone is getting busy signals on 5026.
04/28/97 Monday 12:54pm EST
Harmony reset bouncing everyone
04/25/97 Friday 12:24am EST
Nynex has destroyed the hunt group again. Line is
giving busies. Dial in on 277 4940.
04/22/97 Tuesday 6:10pm EST
Load driven to 30 by errant cgi on the part of local user.
04/21/97 Monday 3:51pm EST
Elmira POP was offline for about 3 minutes, for unknown reasons.
04/20/97 Sunday 12:29am EST
I locked up light by accident by bringing up adore with same
IP number!
Sorry.
04/19/97 Saturday 7:02pm EST
Light taken down to replace failing boot drive.
Harmony rebooted with new operational image 11.2.3
6:00pm -> 6:30pm
04/19/97 Saturday 3:20pm EST
Netserver 16-I rebooted at 3:10pm.
04/17/97 Thursday 01:47am EST
Cisco router got locked up around 12:30am, process table full.
Rebooted at 01:45am
Apparently we suffered a syn flood attack from a disgruntled irc
user.
04/16/97 Wednesday 6:02pm EST
Majesty (news) is down for upgrades.
6:00pm -> 7:00pm
04/15/97 Tuesday 3:03pm EST
Spam filter is causing more damage than its worth. Have removed
it. sendmail was down for 10 minutes due to an error restarting the
non spam filter version.
04/08/97 Tuesday 2:36pm EST
Light is loosing is main root drive. There is a backup mirror
drive if the main drive fails totally. A new drive is on order.
Light rebooted at 2:30pm to determine which drive was failing.
04/08/97 Tuesday 11:03am EST
Elmira off line for about 30 minutes due to network lockup
at elmira end.
04/05/97 Saturday 11:57pm EST
Elmira Frame Relay to Nynex is down, lights are dead. Called into
Nynex Repair.
Nynex T1 Box was burnt out, replaced and line came back up.
Downtime 24 hours.
04/05/97 Saturday 12:09pm EST
Rebooted light by accident. Sorry.
04/05/97 Saturday 01:00am EST
Binary partitions on aurora (news2) rearranged to provide more
space. All articles lost in alt.binaries.pictures.* and a number of
warez and games groups.
04/04/97 Friday 8:45pm EST
Light rebooted at 6:00pm to install new 9 Gig drive.
Rebooted a second time to clear out jammed format session on drive.
6:00pm -> 6:07pm
8:45pm -> 8:50pm
04/01/97 Tuesday 5:40pm EST
News (majesty) rebooted with new kernel with statd security patch.
04/01/97 Tuesday 5:30pm EST
Light rebooted with new kernel with statd security patch.
04/01/97 Tuesday 3:29pm EST
News2 taken down to install 128Meg of memory.
15:22 -> 15:30
03/31/97 Monday 5:47pm EST
Light went out of control, load went to 20. I halted the
system and rebooted.
Earlier had 'out of mbuf' errors.
Probably caused by Apache 1.2b4, we are now back 1.1.1
03/28/97 Friday 1:40pm EST
Network rewiring may have caused intermittent freezes between
11am and 1:00pm.
03/28/97 Friday 11:05am EST
News died silently at 4:42am probably from an errant control
message with too long a header.
11:44am news rebooted
03/27/97 Thursday 2:15pm EST
News and news2 off line for 15 minutes at 2:00pm to rewire.
Failed, have to do it again.
03/27/97 Thursday 09:15am EST
Network locked up this morning at 8:12am necessitating a reboot.
It then locked up again at 8:26am needing anothe reboot.
There was very strange activity on one of the network hubs (BNC
connector to downstairs lan) even though there should have been no
traffic at all going through it.
Unfortunately when the network is locked up, network sniffers to
view traffic don't show much!
We will be separating the 4 ethernet hubs into their own areas,
making it easier to tell which line may be in trouble. Presently they
are all stacked in one central site making it impossible to quickly
trace the wires from blinking lights to machines they are connected
to.
03/26/97 Wednesday 9:32pm EST
Majesty (news) locked up the ethernet at 9:11pm causing all machines
to stop responding.
Light, majesty and aurora rebooted.
03/26/97 Wednesday 4:01pm EST
Nynex had a major event last night, causing our roll over
to go off line. All modems were working, but were not rolling over
on busy. People calling in on 277 5026 were getting busy signals,
because that modem was busy, although many others were not.
This fixed around 1am, but then broke again this afternoon around
2pm. All modems work, and most roll over properly, but the roll over
on 5026 has gone off line again. Dial in on 277 4940.
03/19/97 Wednesday 5:50pm EST
Aurora down: 4:00 -> 5:50pm
News2 (aurora) down for memory upgrades. 512K Cache installed,
128Meg of memory still not being recognized by Linux.
03/19/97 Wednesday 12:59am EST
USR Couriers rebooted with new kernel image, bouncing everyone.
03/11/97 Tuesday 1:54pm EST
Aurora (news2) rebooted.
03/11/97 Tuesday 12:14pm EST
Aurora (new2) taken down to install 128Meg. Rebooted fine but
wouldn't recognize the memory. Still running on an effective 64Meg.
03/10/97 Monday 10:45am EST
Light semi jammed at about 8:46am for unknown reasons, causing
various network malfunctions.
Rebooted at 10:45am.
News was down for the period of the reboot.
03/08/97 Saturday 12:17am EST
News outsend queue was jammed, some articles were not being
posted to the 15 swap sites. They were going out the main feeds.
The queue has been freed, if articles are still on our system,
they will be sent, if not please resend.
03/06/97 Thursday 12:20am EST
News taken down because of full partitions. Expire set to
1 day to clear out space so that we can clean up the mess.
News2 running fine, expire is 14 days.
02/26/97 Wednesday 08:40am EST
Light became crippled at 5:19am with a Process Table Full error.
Beepers did not go off because it was still responding to pings.
Rebooted at 8:35am.
02/24/97 Monday 8:21pm EST
Majesty (news) rebooted at 8:06pm to install new kernel.
02/23/97 Sunday 10:15pm EST
Light was taken down this afternoon for about 20 minutes due to a
jammed process that drove the load to 8.
News was rebooted twice at 22:00pm, to install new kernels.
02/23/97 Sunday 2:47pm EST
News crashed from filled up warez partition.
02/22/97 Saturday 9:53pm EST
Amphenonal connector pulled out of the modem banks taking modems
33 through 48 off line for a while. Don't know how long it was going
on.
02/20/97 Thursday 9:13pm EST
Majesty (news) rebooted with new kernel.
02/20/97 Thursday 05:12am EST
Majesty's etherport crashed for the second time. Took
about an hour to get majesty and light disentangled, had to reboot
both. I hate NFS.
Have no idea what is wrong with Majesty.
02/17/97 Monday 9:29pm EST
Light was taken down at 6:00pm to install new lib.c and
security patches. It did not go smoothly, and we are still
getting bad signal stack errors every once in a while.
Came back on line at 7:00pm and had to reboot a few times
after.
02/15/97 Saturday 4:31pm EST
News2 was down for an hour to add new disk space. Entire news spools
was wiped clean. It will refresh as you hit on groups again.
02/14/97 Friday 12:03am EST
Majesty (news) rebooted to install new security upgrades.
02/13/97 Thursday 1:53pm EST
Majesty (news) taken down for an hour for maintenance of news
spool.
02/10/97 Monday 11:59pm EST
Majesty (news) locked up and wouldn't communicate over the ether
port. Routes were all right, but it wouldn't telnet in or out or even
to itself. I had a hard time getting control back. Finally able to
do a clean reboot. It's working now.
02/08/97 Saturday 5:26pm EST
News2 restarted with clean spool, replicate true is on.
Expect it to bomb.
02/07/97 Friday 01:19am EST
News2 taken down for 1 hour to add new drive. Have 12 gig on
board.
02/06/97 Thursday 6:42pm EST
News2 was taken down for about 15 minutes to install
another hard drive. One more to go in later.
No news was lost.
02/05/97 Wednesday 9:59pm EST
news2 got completely munged with the replicate true setting. I have
turned it off and had to rebuild the news spool.
02/04/97 Tuesday 4:28pm EST
News2 taken down to move machine to machine room. News spool
was wiped clean to install "replicate true" which will force news2
to follow article numbers on news.
02/02/97 Sunday 6:29pm EST
News was down for one hour due to a mistake.
01/30/97 Thursday 2:14pm EST
Cisco router locked up.
Had to reboot it.
Down for 10 minutes.
Turns out Jane had set a wrong IP in her work machine! Not a
fault of the Cisco, easy to replicate.
01/26/97 Sunday 10:37pm EST
Light rebooted at 10:00pm with corrected version of new
kernel. SOMAXCONN set to 127 in header.h files and binary edit
of uipc_socket.o
01/25/97 Saturday 5:33pm EST
Light rebooted with new kernel. jesi contains SOMAXCONN 127
which may help web server jams.
01/21/97 Tuesday 03:34am EST
Couriers rebooted to clear out jammed modem.
01/14/97 Tuesday 2:03pm EST
Light taken down to install new virtual domain code. I screwed
up on taking light down (forgot to disengage majesty) and what should
have been a 5 minute cycle turned into a 20 minute cycle.
Installation when smoothly and it seems to work at first glance.
Light may crash.
01/09/97 Thursday 8:32pm EST
Modem rack 1 has a bad power supply. It is presently running on
its spare. I found this out by accident when I went to reseat both
power supplies alternately. The modems died when set to power supply 1.
Everyone on rack one got bounced.
Will replace.
01/06/97 Monday 10:26pm EST
Sprint has taken over management of our primary router to the
internet. I don't know if this is a good thing or not. I get to not
have the write password, but apparently they also changed the read
password, so the monitoring software is registering throughput as
zero. Not.
01/03/97 Friday 2:51pm EST
Courier modem number 4 was producing ring no answer.
The rack has been reset.
12/30/96 Monday 7:22pm EST
Light crashed. I have reinstalled the original kernel using
the older virtual interface code, kernel jesf. We started crashing
with jesg.
12/28/96 Saturday 4:51pm EST
Light crashed. Time to get rid of the new vif code.
Don't know if the old code supports more than 128 virtual domains.
12/28/96 Saturday 11:12am EST
Connection to external network was down from 4:00 to about 10:30.
This is usually caused by problems external to lightlink, but
this time for some reason the cisco router had locked up.
Sprint is "looking into it".
I have placed a pinger on the cisco from majesty so that if it
happens again I will be beeped immediately.
12/27/96 Friday 6:11pm EST
Light crashed.
12/26/96 Thursday 3:40pm EST
News taken down for upgrades.
12/26/96 Thursday 1:41pm EST
I crashed light installing new software.
12/23/96 Monday 11:40am EST
Light crashed.
Probably caused by new virtual domain code.
12/20/96 Friday 5:00pm EST
Modems 49-72 upgraded to flash prom 28MR113D.HEX
Modems 04-12 upgraded to flash prom 28MR113D.HEX
All modems upgraded.
12/20/96 Friday 07:41am EST
Light crashed, cpu panic on mget.
12/17/96 Tuesday 6:17pm EST
Vif 1.1 installed, virtual domain code. Running kernel jesg
Light taken down at 6:00pm to install code.
USR's taken off line to switch from coax to twisted pair, then
rebooted.
12/12/96 Thursday 7:16pm EST
Virtual domains moved to 205.232.88.xx
Web was down from about 6:00pm until now.
12/05/96 Thursday 5:49pm EST
Majesty was accidentally halted by one of our staff.
News was down from 4:15pm until about 5:00pm. This caused
anomolies in the mail server and shell accounts due to NFS mounted
drives between light and majesty.
12/02/96 Monday 12:42pm EST
Web server was down for 10 minutes due to a bad domain name.
12/01/96 Sunday 6:23pm EST
Light was down for scheduled maintenance starting at about 6:10pm.
/tmp was cleaned out, and various things checked to make sure
they work properly (rootenv etc.)
11/26/96 Tuesday 2:42pm EST
I crashed light by accident with an incorrect kill command.
11/26/96 Tuesday 1:20pm EST
One line fixed, one to go.
11/26/96 Tuesday 12:34pm EST
2 lines are still bad, one is producing ring no answers at around
modem 45.
Just keep trying, you will eventually hop over someone getting
the ring no answer and get in.
11/25/96 Monday 7:51pm EST
Well there were 6 dead lines. Now there are 8. Making
progress.
Nynex just showed up, we are down to 2 bad lines, one can
be busied out, the other is still ring no answer.
11/25/96 Monday 3:14pm EST
Presently the rotary seems to be working properly except we have
found 6 dead lines in the middle of it that were working fine
a few days ago. Nynex has been informed.
11/25/96 Monday 1:12pm EST
Hunt group is in process of being repaired. In mean while 18
modems are off line.
ftp filled up over night due to failure of maintenance program
over the past few days. This has been fixed to beep me.
web server did not restart smoothly this morning after web stat
runs. No reason determined. Results are cgi failures, and other
anomalies.
11/22/96 Friday 6:40pm EST
Nynex installed the remaining 9 lines for new modems and managed
to stick the new numbers in the middle of the hunt group. Because the
numbers were not working and had no modems on them if they did, people
were unable to dialup after the first 30 modems were filled up.
This produced long term ring no answers all afternoon.
11/18/96 Monday 9:06pm EST
News crashed due to disk full errors, and then later again
due my own errors.
11/17/96 Sunday 11:47pm EST
News taken down for security enhancements.
11/16/96 Saturday 7:01pm EST
News was down for about 45 minutes. A simple reboot to install
new security software turned into a nightmare when I couldn't sign on
at all. A bit too much security I would say.
Actually the cause was a bug in some commented lines in rc.local
that prevented it from running fully during boot up, and the password
authentication daemon was not getting started.
Its a very erie feeling not being able to sign on to your own
machine.
11/03/96 Sunday 2:44pm EST
All shell access has been closed off to accounts that have not
used shell in the past month.
10/31/96 Thursday 8:59pm EST
I erased the password file accidentally while cleaning up for
security sweeps.
e-mail and shell were unavailable for about 10 minutes.
10/25/96 Friday 5:54pm EST
Light and majesty both rebooted to add security features.
Running jesf and jojof
10/24/96 Thursday 12:24pm EST
Light rebooted at 11:pm last night to clear out remains
of intruder. Forgot to install new named, so at 12 midnight
when named was reset it bombed out. Name service was defective
from about 12 midnight to 11am this morning.
10/24/96 Thursday 12:24pm EST
A trogin login program was found on light recording shell
passwords. Trojan seems to have been installed Aug 16th.
Not doing too well.
10/18/96 Friday 11:15pm EST
We are under the influence of a *VERY* bad mail loop between
Cornell listproc, the department of mathematics and ourselves, namely
my own mail. A listserve I run at cornell is sending error messages
to my account at the math department which was just closed. The
messages are bouncing back to the listproc which is then bouncing them
back again to math. Each time the files get bigger. They started off
about 20K, now they are 600K. I am going to have to disable my mail
lest the mail spool fill up. Cornell has been notified, but no
response yet.
10/17/96 Thursday 10:18pm EST
Problems with named have been fixed.
10/17/96 Thursday 10:17pm EST
Cornell has asked me to remove all cornell.* groups from
our newsfeeds. They are meant for cornell only.
10/17/96 Thursday 7:04pm EST
At 2:07pm the web server suffered an unclean restart which
resulted in it rejecting all cgi execution for a few hours, before I
was notified.
Don't know what caused the problem, except that the server was
restarted by hand and then the error logs show cgi's failing left and
right.
10/16/96 Wednesday 12:48pm EST
External net is down.
10/12/96 Saturday 11:46pm EST
External net is down from about 11pm to 2am
They are working on fixing the router problems
that have plagued Ithaca for a long time.
10/10/96 Thursday 11:19pm EST
Light rebooted to install corrected named files.
10/09/96 Wednesday 11:59pm EST
I screwed up and killed a whole mess of processes that should have
been left running. Rebooted light to restabilize system.
10/09/96 Wednesday 2:07pm EST
External net is going up and down while they work on a bad
primary Ithaca router.
10/07/96 Monday 11:08pm EST
Light rebooted to clear out named. We will be installing a new
version of named and the vif code shortly. Please bear with us, presently
evey time I add a new virtual domain I have to reboot the system.
10/05/96 Saturday 9:08pm EST
Light rebooted to clear out named. Named is due for replacement
soon.
10/02/96 Wednesday 10:18pm EST
The root partition filled up this morning causing the loss of web
hit stats from about 4:20 in the morning to about 11:00am.
10/02/96 Wednesday 1:03pm EST
Light was rebooted to clear out a full root paritition.
The root system filled up during the mirror run. This occured
because the mirror drives are not mounted during normal reboots such
as the one last night, for reasons as yet undetermined. Since the
mirroring is to directories attached to the root partition, when the
mirror partitions are not attached, it merely fills up the very small
root partition.
10/02/96 Wednesday 12:49am EST
Light was down for two hours starting at about 11pm.
We ran into a problem with named (NAME-D) and virtual domains.
If named is loaded first and then /etc/rc.vif is run, it will
work properly. vifs are virtual interfaces, we need one for each
virtual domain. They assign the virtual IP number to the ethernet
port so it can respond to more than one IP number.
But after the vifs are loaded, if named is killed outright and
restarted, it runs out of file descriptors and complains about bad
file numbers and refuses to load any name service records.
There seems to be a limit of about 40 to 50 virtual domains,
before named starts to go bad. We have over 100.
Most of the down time was spent trying to corner the problem and
determine the max number of vifs.
I was playing with sendmail on majesty and had killed off named
and restarted it a few times on both machines. This is what started
the problem. When I restarted it on light, it died and it took me a
good half hour to figure out what was going on.
It's also really hard to think straight with fear running through
your stomach.
09/29/96 Sunday 6:53pm EST
Light was down for scheduled maintenance from 6:00pm to 6:50pm.
Disk drive bay filters were cleaned.
A broken SCSI cable feeding the tape backup drive was replaced.
Harmony was extensively tested for name service and signon
authentication while light was down. The RX11.1 code works much
better than the old R9.0.
Trumpet winsock 2.0B seems to ignore the second DNS number in ethernet
mode, but works fine during dialup.
Timing differences however seem to trip up the script. This will
need more study.
Win95 dialup worked fine.
Next scheduled down time will be next Sunday at 6:00pm to move
the web and log directories to their own partition.
09/27/96 Friday 7:21pm EST
News is down to repair a disk drive bay.
09/24/96 Tuesday 1:59pm EST
External net is not fully up.
13:45pm external net seems back up.
09/23/96 Monday 2:26pm EST
Web stats for 19960922 and 19960923 were munged and had
to be redone. They should be correct now.
09/23/96 Monday 10:52am EST
Light's /home directories were filled up today by a user who had
a 600 meg file in his home directory. This was going on since 5:45am
until now.
09/21/96 Saturday 10:18pm EST
News was restarted due to system weirdnesses with the w command.
09/20/96 Friday 7:54pm EST
News rebooted to find which fan is going bad. It is going to have
to be replaced soon.
09/19/96 Thursday 12:48am EST
News was rebooted a few times around 11:00pm to clear out jammed
serial ports from uucp experiments.
09/15/96 Sunday 9:57pm EST
News rebooted to clear out jammed processes on serial ports.
09/15/96 Sunday 11:38am EST
External net has been down since
06:43am -> 3:00pm
Ithaca 1 Router was down at Nysernet.
09/12/96 Thursday 11:25am EST
External net is down.
09/11/96 Wednesday 10:32pm EST
The web server went out of control at 8:20pm. We spent about 30
minutes trying to do an autopsy on the dying processes, only to find
that some could not be killed off. Since one remaining process was
tying up port 80 in an (D) uninterruptible I/O wait, we had to reboot
light to clear it out.
09/09/96 Monday 12:59pm EST
News was down for about 30 minutes to remove an air conditioner
with a blown compressor.
09/08/96 Sunday 8:48pm EST
Ithaca-1 router is off line (external network).
09/06/96 Friday 11:05am EST
Web server has been stabilized to a large degree. Source of load
spikes has been found, and hung children and root servers are being
worked on.
Web stats are being moved to 6:30am once a day so that the server
is not killed every hour interupting ftp downloads etc.
09/03/96 Tuesday 4:31pm EST
Power outage at about 4:10pm -> 4:42pm
09/03/96 Tuesday 01:10am EST
Web hit logs program is causing the server to die. I will
be running web hits by hand periodically until I figure out why.
09/02/96 Monday 3:43pm EST
Web server logs from 7:00pm or so last evening until now have
been lost due to a programming error on my part.
09/02/96 Monday 12:08am EST
Modems were reset bouncing everyone.
08/31/96 Saturday 1:01pm EST
Working on the web server, it will be up and down repeatedly.
13:01 -> 15:00
08/30/96 Friday 9:47pm EST
Light crashed due to a mistake I made.
08/23/96 Friday 12:29am EST
Perl5 was down until now from earlier upgrades today.
Links in /usr/local/bin were nuked, and permissions were set wrong.
08/22/96 Thursday 9:55pm EST
We are presently running vanilla apache 1.1.1. The web stats are
still be accumulated but they are not being distributed to users.
This will start happening again when things stabilize.
The Secure server is working again (the documentation was wrong)
but is presently down.
08/22/96 Thursday 6:54pm EST
ApacheSSL 1.3 replaced with vanila apache 1.1.1
18:55 ->
Keep Alive turned off to see if it effects spiking.
08/22/96 Thursday 10:08am EST
The Apache web server is not working perfectly. It is causing
load spikes on the system which make light stick momentarily every
once in a while. It may be related to cgi's. Apache is aware
of the problem and is 'working on it'.
1.1.1. however so far seems to have fixed the main problem
with 1.05 which is chronic freezing.
The Secure server is not responding yet, probably due to a
configuration error from the upgrade.
08/21/96 Wednesday 2:22pm EST
Apache web server is down for upgrades.
14:22 -> Running 1.1.1 -> 16:52
16:52 -> Running 1.05 -> 18:00
18:00 -> Running 1.1.1 -> *****
08/19/96 Monday 09:24am EST
News crashed at 6;07am from a news posting collision between
two groups alt.binaires and alt.binaries.
08/15/96 Thursday 6:11pm EST
The following newsgroups have been moved to a new partition. All
articles in these groups have been temporarily lost.
/alt/binaries/multimedia
/alt/binaries/games
/alt/binaries/mac
/alt/binaries/sounds
/alt/binaries/misc
/alt/fan
/alt/mac
08/13/96 Tuesday 5:34pm EST
News is down for upgrades
17:33 -> 19:07
Big 7 news groups now belong on their own partition which will
allow longer expire times for them and the world wide groups that
remain on the original partition.
soc sci comp rec talk misc news
08/13/96 Tuesday 11:23am EST
Last night at about 1:00am, there was a strange anomaly in the
harmony password authentication process. A number of people were
blocked from getting on. Causes are as yet unknown and the anomaly
has not been repeated.
08/10/96 Saturday 12:37pm EST
Majesty/News are going to be up and down all afternoon for
upgrades.
Down:
12:37 -> 12:43
16:18 -> 20:16
I tried to move the rec group from the root news partition to its
own partition pending moving all of the Big 7 to their own space.
The rec partition has 400 meg in it, and it took hours and hours
and hours to copy it over. I finally cut it short and proceeded to
erase the original files on the root directory, and THAT took hours
and hours and hours.
So, this is not going to work.
I took this course of action because I wanted to save the present
spool of news in rec (and the big 7 when I go to move them) but it
just takes too long. So probably what I am going to do is just nuke
the news spool, reformat and rearrange it the way it ought to be, and
then let it rebuild itself with new articles over the ensuing days.
We either suffer long down times and preserve articles (although
losing new articles coming in during the down times), or we nuke the
whole spool and get it running again in about 1 hour, losing
everything on it, but having little down time and losing few new
articles.
08/10/96 Saturday 01:33am EST
news reset twice to clear out expiration confusion resulting from
earlier upgrade.
08/09/96 Friday 1:49pm EST
news is down for about 2 hours for upgrades.
13:49 -> 15:41
The overview data base was moved from /sd4a to /sd5h to even out
the load on our news drives.
08/02/96 Friday 10:42pm EST
The external net is down.
07/26/96 Friday 2:52pm EST
The external net is down.
07/25/96 Thursday 4:44pm EST
Jason,
Light suffered an event a few moments ago, the load went to 50
and people started calling in thinking we were down.
During such high load periods, cron kicks in and starts taking
snap shots of the system resulting in the following top display.
You will notice mogrify is running at 99 percent CPU, using 781
Meg of swap space, 41meg of which was resident.
You must put limits on this puppy as it is highly destructive in
the hands of your users.
Homer
Thu Jul 25 16:26:39 EDT 1996
last pid: 29724; load averages: 47.44, 36.88, 20.52 16:27:09
186 processes: 160 sleeping, 24 running, 1 zombie, 1 stopped
Memory: 111M available, 53M in use, 58M free, 8984K locked
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
29114 nobody 102 10 781M 41M run 1:09 99.64% 18.75% mogrify
29709 sysdiag -12 -20 1704K 1428K run 0:00 0.00% 0.00% top
29168 sysdiag 25 0 652K 212K run 0:00 0.00% 0.00% <swapped>
28724 nobody 25 0 448K 328K run 0:00 0.00% 0.00% httpd
63 sysdiag 25 0 88K 104K run 43:39 0.00% 0.00% portmap
17674 docdoo -5 2 5780K 1412K run 17:07 0.00% 0.00% parse
12119 sysdiag -5 0 352K 200K run 7:01 0.00% 0.00% erpcd
28828 nobody -5 0 448K 176K run 0:00 0.00% 0.00% httpd
28778 gila -5 0 2532K 164K run 0:00 0.00% 0.00% pine3.95
22690 maxho -5 4 1296K 128K run 3:36 0.00% 0.00% bawt
455 root 25 0 11M 136K run 63:01 0.00% 0.00% Xsun
29153 nobody -5 0 436K 312K run 0:00 0.00% 0.00% httpd
29079 nobody -5 0 436K 236K run 0:00 0.00% 0.00% httpd
29109 nobody -5 0 440K 220K run 0:00 0.00% 0.00% httpd
7979 slam -5 0 1080K 216K run 3:30 0.00% 0.00% eggdrop
Homer
07/24/96 Wednesday 4:41pm EST
htaccess passwords for web pages is presently not working.
We are looking into it.
This is fixed. There is a new module called unix_auth_module,
that is supposed to use /etc/passwd passwords. It conflicts
with the normal dbm passwords. The server was recompiled without
the unix_auth_module and dbm passwords started working again.
07/22/96 Monday 6:27pm EST
Web stats were down for the past day. No web logs were lost,
but they weren't being collated every hour as usual. Things should
be back to normal and all past web logs should be in todays files.
07/22/96 Monday 6:25pm EST
News was down for about an hour to do routine maintenance
on disk drives and to install a new mirror drive.
07/19/96 Friday 12:53pm EST
News and Majesty were down from 11:12am to 12:52pm for
maintenance and upgrades.
07/15/96 Monday 1:44pm EST
Web server was down for about 15 minutes due to DNS failure on
a virtual domain. My fault.
07/04/96 Thursday 10:47pm EST
News server crashed from full partition on alt.binaries.
06/27/96 Thursday 7:44pm EST
Last night at 12:00am or so the web server crashed and was
down all night.
Since there were no new hits, the web stat accumulation program
simply kept adding the last batch of stats to the log files every hour
on the half hour, producing multiple entries for the time between
11:30pm and 12:00am when the server crashed.
I have cleaned up all the log files for 19960627, so there should
be no duplicates in them.
It is unknown why the server crashed, but it may have been
related to a domain name change I made, failing to inform the virtual
domain server of the change. The apache server is very sensitive to
dns failures, it will crash the whole web server if it fails to get
dns on any of its virtual domains.
I have fixed the bug in the stats program that wrote duplicate
stats to people's log files, and the apache server is due for an
upgrade shortly.
06/27/96 Thursday 12:05pm EST
Light was rebooted to install jese kernel, which includes patch
102264-02 ufs_lockf.o to fix the crash we suffered yesterday.
06/27/96 Thursday 11:20am EST
Harmony was rebooted to install fix for failure to reprompt.
06/26/96 Wednesday 11:46am EST
Light crashed and rebooted itself for unknown reasons, probably not
related to scsibus errors.
Lot's of changes have been made to the kernel recently so some
instability may have been entered into the system.
06/25/96 Tuesday 4:18pm EST
Harmony reset to set auto_detect_timeout to 10 from 30.
This will prevent some ring no answers.
06/25/96 Tuesday 3:47pm EST
Harmony was rebooted at around 12:30pm to install a new setting
to help fix some of the scripting failures the new code is causing.
This fixed a lot of scripts and promptly broke a few more.
06/24/96 Monday 10:08pm EST
News/Majesty was down for 15 minutes at 10:00pm or so to
add 64 meg of memory bringing total to 160meg.
06/23/96 Sunday 4:52pm EST
News/majesty was down for about 20 minutes at 4:10pm, for
upgrades.
/usr/local software drive on light was copied over to majesty
/sd7. Majesty is now running on identical software to light.
06/22/96 Saturday 11:32pm EST
Alt.binaries filled up the news partition and bombed out the
news server. I have nuked the alt.binaries group, starting with a clean
slate. They will fill back up again over the next few days.
Someone is flooding the net with binaries, possibly in a.b.misc,
and this is filling up partitions and using up extreme amounts of
T1 bandwidth.
06/22/96 Saturday 7:13pm EST
Harmony was taken down at 6:00pm until 6:30pm to install
RX11.1.3.
First indications are that this has NOT fixed the sign on problem.
Further testing will be necessary to see if it helps the upload problem.
Homer
06/19/96 Wednesday 10:14pm EST
News was down for 10 minutes. Rebuilt kernel on majesty for
127 max users instead of 200. Told that 200 wraps around at 128!
Rebooted.
06/19/96 Wednesday 12:30pm EST
A number of user perl scripts went out of control at 10:00pm last
night producing high loads and sticky response. The situation
lasted for about an hour before I caught it and killed them.
The alt.binaries groups have been flooded recently, tying up both
our incoming bandwidth and disk space. I cleaned out the alt.binaries
groups to one day (although expire is still 3) and I have upped our T1
to 768K effective in about a week.
06/16/96 Sunday 11:47pm EST
Light was quarantined for 2 hours from 6:00pm to 8:00pm or so to
move the ftp/web directories to a new 4 Gig drive.
The web server was down for this time, and shell usage was locked
out. E-mail, news, dialup and web surfing should all have been
working fine.
The upgarde went with out incident.
06/11/96 Tuesday 12:59pm EST
Incremental backups as of 8am are in /backup.
06/11/96 Tuesday 12:19pm EST
/etc/motd was corrupted. I replaced the kernel and motd and
rebooted.
06/11/96 Tuesday 12:06pm EST
We lost the home drive sd6 at about 8:30am this morning. Light
was down until now restoring the home directories to a new drive.
sd6 is the drive that has been having so many errors over the
past year, I think it finally decided to die for good.
Home directories are current as of 12 midnight or so.
Incrementals will be made available shortly if the system proves
stable.
06/10/96 Monday 5:52pm EST
Web server pages were messed up for about 30 minutes due to
a typo I made.
06/09/96 Sunday 8:24pm EST
ftp brought back on line. Everything looks like it is working.
06/09/96 Sunday 7:13pm EST
Light was taken offline for scheduled maintenance at 6:45pm.
/home/ftp was moved to /ftp
A link was placed in /home/ftp -> /ftp, so nothing should break
from this.
During this time I was able to sign on to harmony using winsock,
and surf the net although www.lightlink.com responded with a socket
error. I also was not able to read or send mail, Eudora responded
with "Connection refused by lightlink.com".
This means that majesty was working as both nameserver and
security server for those using Harmony to sign on.
By changing the definition of smtp.lightlink.com and
mail.lightlink.com from light to majesty, it might be possible to keep
outgoing mail functional during these down times. But this is
possibly more dangerous than it is worth.
06/09/96 Sunday 11:17am EST
News throttled itself from a full news partition over night, at
least that is what it says it did. The expire run has apparently
cleaned out the over full articles but did not restart news.
Recent spams and cross postings between news heirarchies have
caused strain on the various news partitions, I am cutting back all
news to 4 days expire time to clear things out, and will open them
back up again according to our disk space availability.
06/07/96 Friday 10:37am EST
ftp daemon jammed from too many failed uploads. I have increased
the permissible jobs to 100 and installed a monitor that will warn
me when it gets above 20.
06/05/96 Wednesday 10:40am EST
Light crashed from cpu panic after /sd6a went out of control.
First look shows no damage.
06/04/96 Tuesday 11:35pm EST
Web server killed and restarted. It was beginning to stick
again.
06/03/96 Monday 12:42pm EST
Ciscoout shows that external net was down for a short while
at 12 noon.
05/29/96 Wednesday 12:55am EST
web server is acting up. keeps jamming for unknown reasons.
This started tonight, but has been reported sporadically in the
past.
05/23/96 Thursday 01:13am EST
Unscheduled reset of Harmony bouncing everyone.
There is a setting called forwarding_timer that is supposed to be
set to 5 (25ms) but is off instead. This is supposed to be the cause
of much upset, perhaps even ftp upload problems. However I was unable
to turn the setting on.
05/16/96 Thursday 4:37pm EST
beta6 ftpd jammed with failed sesssions. Rebooted with beta11.
05/16/96 Thursday 12:00am EST
Unscheduled downtime.
All modem ports were reset at 12 midnight, to reset the subnet
mask of each port from 255.255.255.0 to 0.0.0.0.
Xylogics says the prior setting would cause trouble.
05/14/96 Tuesday 11:52pm EST
News server was shut off from Clarity tonight for a few hours
while we tried to determine where the bandwidth is going.
(It's not Clarity. 6/4/96)
05/14/96 Tuesday 11:49pm EST
The web server shut itself down tonight for a few minutes after I
deleted an account that had moved on. The account had a virtual
domain with us, and when the directories were deleted, the httpd
server decided it didn't want to run any more.
Not sure if that is a feature or a bug.
05/14/96 Tuesday 1:34pm EST
News should be fully back on line.
05/14/96 Tuesday 12:08pm EST
News is down for testing. Back at 12:45pm.
Remote domain reading is still down.
05/13/96 Monday 3:06pm EST
News is down for testing.
You will notice from the sharp fall off on the ciscoout numbers,
that news is somehow using up all our bandwidth.
News is presently back up, but various feeds are locked out,
until we find which one is causing the problem.
Problem seems to have been in part all the new micro feeds
we have added. We were sending too much to all of them at once.
Things are more in control now.
05/10/96 Friday 11:44am EST
Harmony rebooted itself at 8:36am under the new operational image.
So much for that.
05/10/96 Friday 01:19am EST
Harmony rebooted itself. It is now running the new operational
image which Xylogics says won't reboot itself any more.
05/07/96 Tuesday 3:42pm EST
Harmony rebooted itself at 7:25am this morning bouncing everyone
off.
Xylogics says that if we install the new R9.2.21 operational
image, it will stop doing this. I have installed this in the proper
directory, but it involves taking harmony down in order to load it.
Probably I will simply let harmony fail again and reboot it self with
the new image.
05/06/96 Monday 7:08pm EST
News was down for about 45 minutes due to a bug in an upgrade
that I did not spot. I have reverted the code to the original until
a fix is forthcoming.
Xylogics has acknowledged that the recent reboots of harmony are
the result of a bug in the R9.2.7 code we are running, and has offered
a fix in R9.2.21, which will not compile. They are aware of that too.
05/04/96 Saturday 12:04am EST
Harmony rebooted itself at 2:46pm on Friday, probably bouncing
everyone off.
05/03/96 Friday 8:23pm EST
News will be up and down for short periods as we try to work the
bugs out of new news software. We are trying to install software that
will allow us to feed remote sites at 10 times our present rate, thus
making a fair exchange for our 5 redundant newsfeeds (that are
presently not enabled because we couldn't feed them as fast as they
were feeding us by a factor of 10.)
04/29/96 Monday 4:37pm EST
The Sprint network was apparently down from 4:30am to about 7:30am
this morning.
04/29/96 Monday 11:43am EST
Light/web/mail/ftp was taken down for scheduled down time from
11:00am to 11:32am.
Installations went without incident.
04/28/96 Sunday 11:23pm EST
Majest/news taken off line for about 45 minutes at 10:23pm for
hardware upgrades.
16 gig of mirror drives were installed on the fast wide bus, and
4 gig of news space was reinstalled.
Installation went without incident.
04/25/96 Thursday 7:26pm EST
Harmony rebooted itself at 7:02pm for unknown reasons. booting
everyone off.
04/23/96 Tuesday 2:59pm EST
Scheduled Down Time
Light was taken down at noon for 15 minutes to install a new
mirror root drive.
Majesty was taken down at 12:15pm for 45 minutes to install a new
mirror root drive, and a fast wide scsi card.
Both installations went without incident.
04/22/96 Monday 12:36pm EST
System taken down for scheduled down time at 11am.
Could not get light to fail to boot no matter how I reset the
kernel to the way it was during the crash. This is not a good sign.
Harmony will boot properly from Majesty, but will not
authenticate. Probably a simple configuration error.
Root mirror drives were not installed, not enough time.
System will be down on Tuesday during Scheduled down time from
11am to 1pm.
04/20/96 Saturday 8:58pm EST
Presently playing with mirroring name service on majesty/light
for both forward and reverse lookups. Name server may be sticky or
momentarily non existent during these tests.
04/20/96 Saturday 4:43pm EST
Earlier today while Light was down, out of the corner of my eye I
saw Harmony reset the modem banks. Apparently during this time it
also lost its default route, preventing dialup users from going out
onto the net.
The mean time to failure around here is getting ridiculous.
Update: Apparently for whatever as yet undetermined reason,
Harmony rebooted itself while light was down. Since the new
experimental Harmony code WAS resident on Majesty, Harmony rebooted
itself from Majesty using a vanilla config.annex file which did not
have the default route to the cisco in it. Thus we lost our default
route.
Amazing. These machines are talking to each other behind my back.
Anyhow the upshot of all this is Harmony is now runnint X9.2.21 rather
than X9.2.7.
04/20/96 Saturday 2:47pm EST
Crashed again. This time out of the blue for no good reason.
Panic on CPU 0, almost definitely on sd5.
Some damage to web hit files.
Perhaps the drive is bad. I do have another one, but it would
take significant down time to swap them over.
I see its going to be a hard life.
However I also see that people were signing on while light was
down, so the harmony software is working on majesty.
Also placed the kernel back into asynch mode which is
usually more stable.
04/20/96 Saturday 2:03pm EST
At about 1:50pm I walked in on light to find it generating
repeated errors on sd5. These were not sufficient to crash it, nor
for anyone to report it, but the tape backup had failed and it was
clearly going down.
I halted light gracefully and rebooted.
I took the opportunity to remove the second drive bay, which is now
destined for Majesty as mirror, and I installed a new $100 gold plated
radium cored forced perfect terminator on the fast wide bus.
There was no damage and light rebooted flawlessly twice.
We may have fixed the 'refusing to boot more than 3 drives' problem,
but we have not fixed the scsi buss errors problems at all.
The procedure I used to bring light down gracefully was:
1.) halt command which would not complete because so many errors were
happening on sd5 it couldn't sync the drives.
2.) Stop-A which forcibly halts the processor in mid stride.
3.) Turned the drive bay off and then on again and let them come up
to speed which apparently cleans out the jammed buss.
4.) Go, which starts the processor again allowing it to complete
the syncing and halting process.
04/19/96 Friday 10:57pm EST
Light crashed while formatting a recalcitrant new disk.
"CPU panic on 0"
Not sure why. Possibly the disk was not properly registered with
the kernel, it had been complaining about read errors prior to trying
to format it. Normally I would have rebooted the system at this
point, but we have a mortorium on rebooting in place.
The kernel complaint was possibly caused by having pulled the
disk to get its serial number and repluging it. They are supposed to
be hot plugable though.
Light rebooted just fine by the way, no damage. Moment of terror
though.
Most people probably can't conceive of being in fear when their
computer crashes. But I go through it every time Light burps.
Some times I just have to walk around until the adrenaline
ceases.
I can't continue living this way.
Panic on cpu 0 indeed.
Panic on operator 0 is more like it.
04/19/96 Friday 7:51pm EST
Getting intermittent ring no answers. On Monday I am going
to have call forwarding on ring no answer installed. It's going to
cost $3/line per month or about $210/month for our 72 modems.
************************************************************************
****************** CRASH OF 4/17/96 ***********************************
RECENT EVENTS of 4/17/96
During the scheduled down time of 6pm to 7pm, after I had swapped
in the new scsi bus, the tape drive and sd8 failed to come on line
during the first boot attempt.
There was some other impropriety in how I had configured the
system which I do not remember now, and I thought the booting failure
was possibly related to it (NOT). I fixed the minor error, and
rebooted and it booted fine and the system came up at 7pm.
Then I started a tape back up, which promptly jammed.
So I rebooted Light, clearing out the jammed tape process, and
started another backup. This one bombed out with a serious scsi
error.
Then I reset the drives to asynch mode in the kernel and rebooted
again. Again the tape backup got a scsi error.
Then I figured well maybe the cable has gotten ruffled, so I took
the system down, and replaced the $100 gold plated radium cored cable
with a $10 cheapie that had been working fine on Majesty. When I went
to reboot, the tape drive wouldn't come on line at all, and neither
would sd8d.
Now this is exactly what happened two weekends ago during the
crash, only that time I was not smart enough to realize what was going
on, and whatever I did in my hurry, I managed to wipe out the FATS on
all of the drives. This time I was very careful not to do anything
stupid, and the drives so far have remained whole through out all
this.
OK, so at about 11pm the system would not boot at all, the tape
drive and sd8 wouldn't come on line no matter what I did.
I then started thinking it might be various things and set about
to test them.
First thing I did was put the old scsi card back in. No change.
The next thing I did was swap the drive bay which holds the 4
drives. We just happened to have a new one lying around which is
destined to hold the web log stats and more drives. That was
convenient. No change.
Then I took all drives out of their drawers (these are hot
plugable drives) and checked and reseated their scsi jumpers and
internal cables, all to no avail.
Then I swapped the CPU board from Majesty to see if it was the
CPU card. No change.
Then I noticed that if I booted with one of the drives off, the
missing drive would come on line. Thus we have sd5 sd6 sd7 sd8, and
sd8 was not coming on line, but if I turned off either 5 6 or 7, it
would come on line just fine. (sd = scsi drive).
Then I stuck both drive bays on the system and put in a fifth
drive that was destined for web stats.
Again, only three drives would come on line, I could turn off any
two and the other three would come on line.
Something has to be working over time to do this.
I saw that probably we could bring the system back up, if I moved
the contents of sd8 to the single wide bus, and left only 5 6 and 7 on
the fast wide bus.
At this time it was about 2am, I was getting very tired and
beginning to make mistakes (deadly mistakes), so I called it quits and
went to bed for 4 hours, leaving the system off.
When I got up at 6:45am, I went in to start putting together the
new configuration, and it just booted all the way just fine.
Somehow my getting some sleep, made the system work.
Now this in fact has been the story of this ISP, and I am
beginning to think that we have something REALLY FLAKEY going on here
that has not been spotted. Time and time again, over the past 9
months, I mean repeatedbly, when the system was rebooted it was hell
to bring back up, then it would suddenly start working again for no
reason, particularly after having been left off for a while.
Then it would run fine for a while, even days, then suddenly
start getting scsi errors again, causing numerous crashes etc.
OK, so in summary this is what has been swapped.
1.) The Sparc 20 mother board (months ago). No change.
2.) The scsi card. No change.
3.) The CPU card. No change.
4.) The drive bay. No change.
5.) The drives themselves, one at a time. No change.
6.) ALL cables, many times. No change.
7.) All terminators. No change.
Presently, light is running, but I am afraid to take it down lest
it not come up again. A tape job is fully jammed so backups can't be
done without rebooting. Of course if I reboot, the tape drive won't
come on line at all. Majesty is down because for some reason it won't
boot with Light's CPU card in it.
So things are a mess.
04/18/96 Thursday 10:43am EST
I called the makers of the scsi card PTISP, and told them how the
bus was only finding 3 drives out of 4. The guy told me to twiddle a
number in the driver and recompile the kernel that would allow the
driver to recognize more than 3 drives (which apparently it was set to
recognize.) I asked him, well how come we have been running on 4
drives all these months (albeit flakily) and he said he didn't know.
I made the change and rebooted. All 4 drives were already on
line from when I woke up, and they stayed on line during the reboot,
so not much is proven there yet. However the full system tape backup
completed which is a major improvement over 5 failues in a row.
I had asked the guy, how could your fast wide bus software affect
the single wide bus that the tape drive was on, and he said he didn't
know. However I must point out that the tape drive is backing up
drives on the fast wide bus and thus there might be some interaction
between them.
We did however get on minor scsi error during the backup, but not
enough to bomb it out. So things are not working perfectly.
04/18/96 Thursday 11:49am EST
Light is back up with its own CPU card, and Majesty and News are
back up.
Tape backups are working.
Gonna skip tonight's down time.
Been down enough.
Homer
************************************************************************
************************************************************************
04/18/96 Thursday 08:46am EST
Made change to scsi driver as suggested by tech support.
scsi_ncmds_per_dev from 3 to 32.
It is supposed to allow recognition of more than 3 drives. This
doesn't explain why we have been running 4 drives for months.
04/18/96 Thursday 07:34am EST
Light was down all night, and may be up and down all day.
It is presently running in a crippled mode, tape backups are NOT
working.
Majesty and news are also down as it won't boot with Light's CPU
board.
I can't take Light down to swap the boards just yet because it
probably won't reboot.
The SCSI errors have finally reared their ugly head and are no
longer playing patsy with me.
More to come but I have to get on with debugging the system.
04/17/96 Wednesday 9:19pm EST
Got a scsi bus error on single wide bus within 5 minutes of
starting tape job. Taking light down to reset kernel to ansychronous
mode.
04/17/96 Wednesday 8:52pm EST
Light rebooted to clear out a jammed tape process.
04/17/96 Wednesday 7:41pm EST
There is some evidence that there was an 'event' of some sort
around 5:30pm this afternoon. We were out and when we came back, the
number of people on the modems was real low, like everyone had been
kicked off or Harmony had rebooted itself for no reason. The modem
stats chart shows when this happened. I have no idea what it was.
Some reports are coming in that lines are disconnecting and then
people are getting ring no answers, then it clears up. Until we track
this down I can't say much about it except keep on trying and
reporting the problems.
04/17/96 Wednesday 7:10pm EST
Lightlink was taken down at 6pm for scheduled hardware upgrades.
There will be further down times this week at the same time period.
1.) New fast wide scsi card was swapped in for the original card
to see if this effects the prevalence of scsi bus errors that have
plagued us from the beginning. They are mostly occuring on sd5 now,
which is the first of the fast wide drives.
When the single wide drives for news were on light, most of the
scsi errors were happening on the single wide bus, so we never tested
a new double wide bus card. But as soon as news was moved to Majesty,
the errors moved over to the double wide bus.
If swapping the card fixes the scsi bus errors, then the old card
will be sent back for repair or burial. If scsi bus errors continue
to happen, adjustments will be made to the scsi programming interface,
to see if that changes anything.
1a.) New kernel (jesb) was installed which allows for additions
of more fast wide drives, and increases number of supportable virtual
domains from 64 to 128. Scsi drives are presently in synchronous
mode, which generally exacerbates scsi erros, and precipidates
crashes from those that occur.
Full tape backups are being done at least once and sometimes
twice a day, and incremental backups are being done hourly.
There are presently no hard drive backups which will start when
we get the next 4 Gig drive on line.
2.) Drive sd4 was taken off the single wide scsi bus and returned
to Majesty for use as another 4 gig News drive. Majesty now has its
full 16 gig news complement back in order. sd4 was recruited from
Majesty during the crash of two weekends ago to hold crash remains.
The crash remains are now on tape and will remain so for a long time.
3.) Harmony was reburned with new configuration settings, one of
which will hopefully allow Majesty to act as a password server for
Harmony when Light is down. This will allow users to signon to
Harmony from whence they can surf the web and read news. If Light is
down they will not have access to their home, ftp or web directories,
nor their e-mail.
4.) A new $100 gold plated radium cored SCSI cable was installed
on Majesty for the news drives. Majesty by the way has never had a
scsi error, but it doesn't have any fast wide's either.
5.) Secondary name service was not working properly on Majesty
while Light was down, this will need to be fixed.
04/16/96 Tuesday 12:57pm EST
Light rebooted to clear out jammed /dev/ttyp1's
04/11/96 Thursday 5:19pm EST
Scsi bus went out of control at about 1:30pm. Light was halted
and rebooted. No damage.
Scsi card is scheduled for replacement.
04/10/96 Wednesday 10:00pm EST
ftpd daemon reset
04/10/96 Wednesday 2:47pm EST
Modem banks reset themselves for unknown reasons bouncing
everyone.
I was playing with modem 40 (14A) 277 9530 trying to see why it
was giving a ring no answer when suddenly the whole modem rack when
off line. I was merely dialing in.
When I first stuck the phone in 14A it was dead. Then when I
tried it again, I had a dial tone. I am going to repunch it just to
make sure the wires are in tight.
There is also an odd occurance in the modems.stat home page where
every once in a while the stats page registers zero people on line for
one 15 minute cycle. Is it possible that Harmony is down during these
times and rebooting itself?
04/10/96 Wednesday 12:40am EST
Got a ring no answer today, and few reports of ring no answer or
pickup after 3 or 4 rings which is really strange.
04/09/96 Tuesday 8:04pm EST
News server reset with rebuilt install.
04/09/96 Tuesday 12:40am EST
ftp daemon reset
04/08/96 Monday 5:03pm EST
Majesty and News were taken down for 30 minutes to oil a squealing
fan cooling our news disk drive bay.
04/07/96 Sunday 11:42pm EST
ftpd daemon reset.
04/06/96 Saturday 12:03pm EST
fptd daemon reset
news server reset
04/05/96 Friday 3:31pm EST
I crashed the modem bank (I think) looking for the bad modem.
Anyhow the whole bank went down, and harmony jammed. Harmony was
rebooted, and modem banks 1 through 8 have been reburned with correct
settings. To my chagrin the MultiTech management software does not
load the modems with config files dependably. It leaves settings out
randomly as it does all the modems. So each one has to be checked
carefully by hand and set again if it didn't take hold on the first
pass.
04/05/96 Friday 2:10pm EST
Getting reports of a bad modem not letting people on. Will
try to hunt down.
04/04/96 Thursday 6:03pm EST
Light was rebooted to clear out a jammed tape process.
I have been playing the tape backup machine extensively to get a
good feeling for it while we program a system to automate the handling
and recording of tape backups.
I did a tape erase just to see what it would do. It started to
erase the tape (very slowly) so I decided to kill it. Turns out it is
an unkillable process. Rather than wait 5 hours for the job to
finish, I rebooted the system. As light was halting it said,
"Some processes wouldn't die."
No sh*t Sherlock.
04/03/96 Wednesday 11:09am EST
News should be fully functional at this time.
Going to swap out the fast wide scsi board with a new one.
If that don't work, we are going to swap out the CPU card.
If that don't work, we are going to get rid of the fast wide bus
and see if a Sun can work with its own hardware.
04/03/96 Wednesday 02:56am EST
Crashed again.
04/03/96 Wednesday 02:34am EST
Light taken down to clear out jammed scsi bus.
/dev/sd5a home directories caught in endless error loop.
04/02/96 Tuesday 9:01pm EST
ftp daemon reset.
04/02/96 Tuesday 1:02pm EST
Deer Park router is still down, some parts of the world
may not be accessible.
04/02/96 Tuesday 12:38pm EST
Nysernet Ithaca-1 router was down from 10:57 to 11:23am this morning.
04/01/96 Monday 12:28pm EST
Sprint is accepting mail again.
04/01/96 Monday 12:15pm EST
News is still flakey. Sporadic reports of people not being able
to access server. Also Sprint is refusing to accept postings, they
are looking into it. None are being lost, but none are being sent.
Postings are being sent to uunet so we are not totally locked out.
News is receiving fine.
Same for secondary news feed, receiving fine, refusing to accept
outgoing. They have been informed.
New backup drives for light are on order.
Hourly tape backups are being done.
03/31/96 Sunday 5:20pm EST
Light rebooted to bring drives back up in asynch mode.
03/31/96 Sunday 3:06pm EST
cron on light was down, web stats weren't being compiled, although
none were lost (except when we were down). They are now up to date
running every hour.
uucp was also down, it should be working again.
03/31/96 Sunday 11:37am EST
News is presently up and down as we try to fix various
things with the history file.
03/31/96 Sunday 11:37am EST
News was down over night due to incorrect permissions on alt.sex
03/31/96 Sunday 11:36am EST
Lightlink was down from 7:30am to about 10pm on Saturday due to
loss of File Allocation Tables across 4 main drives.
03/29/96 Friday 2:44pm EST
The external network connection to nysernet was down due to their
hardware failure for a few minutes between 2:00pm 2:30pm. They called
and said it was fixed and apologized for the outage.
03/29/96 Friday 12:10pm EST
Mail was down for a few hours due to /var filling up from an
incoming spam of a user.
03/28/96 Thursday 12:25pm EST
Light crashed during news expire.
News will be moved off of light today some time.
03/28/96 Thursday 11:32am EST
ftp jammed, filled up with interrupted processes.
preventing people from logging on to ftp.
This has to be fixed!
03/22/96 Friday 1:49pm EST
/var ran out of disk space. news throttled, was down until now.
03/20/96 Wednesday 6:51pm EST
Modem 3 producing ring no answer, now off line.
03/20/96 Wednesday 10:19am EST
ftp server reset.
03/19/96 Tuesday 12:44pm EST
Light crashed at 12:25pm during news expire.
03/19/96 Tuesday 09:00pm EST
ftpd daemon jammed with broken jobs. Reset
03/14/96 Thursday 09:39am EST
Light crashed and rebooted itself at 1:10am last night.
News was down for the night.
03/14/96 Thursday 12:48am EST
ftp daemon reset.
03/12/96 Tuesday 2:25pm EST
News was throttled from alt/binaries filling up from 12:00pm
this afternoon.
03/11/96 Monday 7:47pm EST
/etc/nologin removed.
03/11/96 Monday 7:27pm EST
Modem bank was down from 6:00pm til now.
Preparing for installing of new modem bank.
03/09/96 Saturday 7:22pm EST
Light reboot to clear out jammed tape job.
Nynex busied out the bad modem, so the rack should be fine for the
moment. He said there was a definite break at the Central Office somewhere.
03/09/96 Saturday 2:47pm EST
It's a bad phone line, not a bad modem rack!
03/09/96 Saturday 11:14am EST
The modem rack is divided into two sections by a bad slot at
modem 26.
Modems 1-25 start at 277-5026
Modems 28-48 start at 277-3567
Modem 27 is dedicated and not available for public use.
03/09/96 Saturday 01:15am EST
The modem rack was reset at about 12:30am bouncing everyone off.
Slot 26 is bad, its not the modem. The modem tries to answer,
but the slot doesn't convey the information. This produces a ring no
answer when you run into that modem.
This divides the rack into two sections of about 25 modems each,
one starting at 277-5026 and one at 273-3567. If you get
ring-no-answer on the first number try the second. If you still get
ring no answer, then all modems are busy.
Modems 9 and 17 were bad a few days ago, and resetting the rack
fixed that. This indicates that there was nothing wrong with the
modems which had been individually reset to no avail, and that the
rack mother board is going bad.
Today's travails is just more of the same.
03/08/96 Friday 5:43pm EST
Modem 26 was the cause of the ring no answer.
It would light up and pretend to answer, but it wouldn't.
It is presently off line.
03/08/96 Friday 4:28pm EST
We are having definite problems with ring no answer.
That means you call up the modem banks, there are modems free,
but the phone just rings. If you call up on one phone it will just
ring, but a second phone will get a modem.
This is often caused by a modem refusing to answer in the middle
of a rotary, but every time I try to narrow it down to a modem, it
picks up and someone gets on!
The ring no answer also happens when all modems are busy, because
there are 12 new lines at the end of the rotary that (sniff sniff)
don't have any modems on them yet.
03/08/96 Friday 1:14pm EST
Light crashed from news expire at 12:36pm.
03/05/96 Tuesday 5:11pm EST
Light rebooted to clear out jammed tape backup process.
03/04/96 Monday 11:08am EST
Light taken down to remove 64meg of memory to move over to Majesty
for news server.
Modems cold booted to help clear our 9 and 17. Seems to have
worked.
Harmony rebooted.
03/03/96 Sunday 1:03pm EST
We have two modems have the gone sour. 9 and 17.
They are presently off line.
Apparently modem 17 has been bad since 3/1/96
03/03/96 Sunday 11:53am EST
Apache web server shifted over to new logging.
03/03/96 Sunday 11:36am EST
We suffered a bum modem this morning that prevented people
from getting on beyond that modem. The phone would just ring and not
pickup. That modem is now off line.
02/29/96 Thursday 09:26am EST
Light crashed.
A number of monitoring procedures we had in place to warn us that
light was down failed. This probably resulted in a number of hours of
down time without awareness that we were down. Just before light
crashed, the load had gone very high for unknown reasons, possibly
massive spamming through our remailer, which is now off line until
further notice.
Unfortunately two web hit log files were lost during the crash,
and we are not presently keeping backups of them because they are so
massive. With the new web hit software in place this will change. In
the meanwhile my apologies to those who lost web hit files.
02/27/96 Tuesday 3:50pm EST
Light was rebooted to clear out misbehavior of the server
that wouldn't fix on its own.
02/27/96 Tuesday 3:08pm EST
All referer and agent logs have been temporarily turned off as
the server is crashing from them apparently.
02/27/96 Tuesday 2:01pm EST
The apache server has run out of file handles.
This will take some serious revamping to fix.
This caused the server to not respond.
02/27/96 Tuesday 1:11pm EST
apache web server reset
Added new mime.types for x-director
02/26/96 Monday 5:00pm EST
ftp server reset
02/23/96 Friday 9:22pm EST
Apache server reset.
02/23/96 Friday 12:31pm EST
Light crashed at 12:07pm during news expire.
02/22/96 Thursday 1:29pm EST
FTP server reset, everyone bounced.
02/22/96 Thursday 01:58am EST
News was killed momentarily and rebooted.
02/20/96 Tuesday 12:58pm EST
I had to kill news a few times to get the expire to start working.
Something has to change around here.
02/20/96 Tuesday 12:21pm EST
Man was that a mess. The process which crashed light last night
managed to start itself up from cron again, even though I had disabled
the cron file. I guess I failed to restart cron so it had it in
memory.
This process took the load to 6.0 at around 2pm. This slowed
everything down to a point where the various hard drive backups and
tape backups didn't complete on time. Thus the tape backup started
while the hard drive back up was still going, causing things to slow
down even more. Then the tape back up failed to complete by the time
other jobs started at 6am which failed to complete by the time news
expire started at 11am, and by 11:30 the load was 12.0 and EVERYTHING
was still running and nothing was getting done.
I rebooted light to clear it all out, after trying to tear it
apart process by process. There is only so much you can do with an
autopsy.
02/19/96 Monday 11:27pm EST
Light went out of control again for reasons that are not clear.
(This was later determined to be caused by a user program scanning
the news spool for data.)
02/19/96 Monday 10:57pm EST
Light went out of control. I tried to bring it down gracefully
and was barely able to halt it in time before it crashed for real.
Reasons unknown.
02/15/96 Thursday 12:14pm EST
Light was down for 10 minutes to oil a noisy fan in the news drive
bay. It may still need to be replaced.
02/14/96 Wednesday 9:56pm EST
Light crashed at 9:34pm probably caused by my moving a disk drive
bay around. SCSI cables suck even when they cost $99 each.
02/13/96 Tuesday 01:24am EST
Web server was down for 10 minutes for a recompile, to add
referer and agent logs.
02/10/96 Saturday 11:43pm EST
News was taken down for 5 minutes to check load averages used
by news.
02/10/96 Saturday 01:58am EST
The outbound mail queue was lost last night due to a spam.
The mail queue holds outgoing mail that can not be delivered for
various reasons at the receiving end. The sendmail program tries
every hour to send the mail for many days and then quits.
02/08/96 Thursday 11:51pm EST
Apparently news throttled itself when /var filled up. I did
not see this and so news was down all afternoon.
02/08/96 Thursday 4:03pm EST
/var ran out of spool space today for a few minutes. This
has been temporarily fixed. I also moved the www log directories
to a new partition, so the web server was down for about 5 minutes.
02/06/96 Tuesday 12:09pm EST
News was down for about an hour as we made preparations for moving
it to Majesty.
02/05/96 Monday 12:09pm EST
Light crashed at 11:45am during news expire. There was a huge
load spike just prior indicating that the apache server had gone our
of control, however all log records from 11:03am on were eradicated
when the disks were fsck'd
The apache server is supposed to kill off excess servers and not
allow them to go over 50 in any case, usually such a crash indicates
300 or more.
I am going to install a little script that will reset the apache
server any time the servers go over 50 as part of the monitor program.
Monitor data can be found in /var/log/monitor and is open to public
view. through our http://www.lightlink.com/stats.html home page.
02/04/96 Sunday 1:08pm EST
Kernel reinstalled yet again, light rebooted.
Now we are running in Asynch mode again.
OK hopefully this will handle the crashes.
02/04/96 Sunday 1:00pm EST
jes kernel reinstalled scsi set to 0x58 and light rebooted.
02/04/96 Sunday 12:19pm EST
Light crashed 11:46am.
Tried to reset scsi bus to asynch mode, but it doesn't
seem to be taking.
It got reset from asynch back to synch during the last install
of the new kernel.
02/03/96 Saturday 4:55pm EST
Light crashed 4:31pm
02/03/96 Saturday 1:41pm EST
Top 18 modems were disconnected and reseated in the correct
order.
02/01/96 Thursday 4:05pm EST
Light crashed 3:51pm
02/01/96 Thursday 11:26am EST
Harmony is failing to do name service properly for unknown reasons.
OK, this is fixed.
02/01/96 Thursday 12:34am EST
Web server was up and down a few times as we played with new
installation.
No permanent changes yet.
01/30/96 Wednesday 10:29am EST
Light was rebooted to install new kernel to fix bug in tape drive
software, and install NFS to bring Majesty on line.
01/30/96 Tuesday 12:32pm EST
ftp jammed for a while due to too many zombie processes left
over from something or other. I killed all ftp processes, perhaps
bumping a few real ones in present time.
01/26/96 Friday 11:26am EST
11:00am
Modems were rebooted and reburned with correct init strings. All
modems visually check for correct init speed and speed. Please report
garbage on the screen during login incidents immediately.
Lower 39 modems have retraining off, top 9 modems have retrain on.
Harmony rebooted to clear out defunct routes in the cache.
Light rebooted.
01/25/96 Thursday 4:17pm EST
Light rebooted. We were getting load spikes, signifying
impending crash. Have moved swap space back off of /sd3b to /sd0b.
If this clears up the spikes then that means sd3 is bad.
01/25/96 Thursday 11:38am EST
Light crashed at 11:25am from news expire during expireover.
We are getting a second Sparc 20 which will carry news, so
hopefully this will move the crashes off of light to that machine,
until we can find out what is going on.
01/24/96 Wednesday 7:04pm EST
CSU/DSU was brought off line for 2 minutes by accident via a
loose power plug while moving it.
01/21/96 Sunday 10:50pm EST
Light went unstable with spike loads of 101 or so. I was
barely able to halt it gracefully, nothing would 'vfork', and we
were getting stack errors on the swap partition.
This was AFTER the move to /sd4.
01/21/96 Sunday 7:36pm EST
After the crash this morning during news overview expire, we
started to get repeated load spikes indicating near crashes.
Since the only change that was made was to move the overview data
base to /sd3a, this indicates that /sd3a is causing troubles.
So now we are moving it to /sd4a.
At the same time the erotica groups will be moved /sd3d to /sd4h,
and /sd3 will take taken fully off line to see if the system
stabilizes.
No news will be lost, but news will be fully down while the
transfer takes place.
01/21/96 Sunday 11:56am EST
Light crashed during news expire at 11:56am.
It crashed during expiry of the overview data base. which is
presently on /sd3a. When I first moved it to /sd3a a few days ago I
noticed a number of high load spikes every few minutes which is
usually a sign of instability and 'near' crashes.
It is possible that /sd3a is causing some of our problems. I may
consider moving the over view data base once again to /sd4a, taking
/sd3a out of the loop.
01/19/96 Friday 6:19pm EST
News overview data base is being rebuilt, to allow threaded news
readers like tin to access old news.
01/19/96 Friday 3:39pm EST
Light taken down for 10 minutes to move news overview data base
to /sd3a. This caused the destruction of the old data base.
No news was lost, however old news is not accessible unitl
overview data base is rebuilt.
01/18/96 Thursday 9:50pm EST
News was down for posting for 7 hours while the history data base
was rebuilt. Obnoxious.
01/13/96 Saturday 3:45pm EST
Light went out of control at 3:33pm from SCSI Bus errors on sd5a.
I tried to take it down gracefully, but to no avail.
01/12/96 Friday 5:30pm EST
News has been unreadable from modems 37 through 48 for a while.
We have been running an open nntp server until recently, which
means everyone in the world can read news from our server.
In trying to locate where our outgoing bandwidth is going, I
turned this off temporarily. By mistake modems 37 through 48 were
been enabled to read news, and were only enabled by default of
everyone being able to read. When I closed off the open access, those
modems were locked out also. They are now properly enabled as
themselves.
I may reopen up the news port in the future, our bandwidth is NOT
being consumed by news reading by external sites.
01/12/96 Friday 4:02pm EST
Light was taken down at 2:40pm to install new Forced Perfect
Terminator on SCSI bus.
Rebooting failed a few times due to typos in rc.local.
Later permissions were incorrectly changed on root directory
locking a few people out for a few minutes.
01/11/96 Thursday 6:16pm EST
News expire is presently acting strange. Expireover went for
7 hours and did not complete. I killed it.
01/06/96 Saturday 12:29pm EST
Light crashed at 12:06pm during news expire.
Installed new kernel with vif=64.
adb -w /vmunix
scsi_options?W 5*
$q
Presumably that sets the scsi bus to asynch mode.
12/31/95 Sunday 11:12am EST
Web server was down from 11:00pm last night due to a failure
to restart it properly.
12/27/95 Wednesday 07:35am EST
Light taken down for 10 minutes to replace scsi cable.
We are now running with two Gold Cables.
Tape drive is first in the chain.
12/26/95 Tuesday 12:09pm EST
Light crashed at 11:55am during news expire.
12/24/95 Sunday 12:07pm EST
Light crashed at 11:58am during news expire presumably from scsi
bus error.
12/20/95 Wednesday 09:25am EST
Light taken down at 8:00am for system work on scsi bus.
12/19/95 Tuesday 10:17am EST
Light crashed from scsi bus errors during news expire at 10:00am.
12/14/95 Thursday 09:28am EST
Light crashed from Scsi bus errors during news expire at 9:07am.
I took the opportunity to install two new gold plated cables ($99
each), so that sun techies can stop asking us if we are using lousy
cables.
12/12/95 Tuesday 3:59pm EST
Light taken down to bring news back on line.
Before putting the disks on line, I put the main scsi cable in
light and terminated it with the active terminator. Light would not
boot, but gave nasty errors resulting in needing to manually fsck
/dev/rsd0g.
I replaced the cable with an earlier 6 foot cable, and did the
same thing and light booted fine. Then I attached the news drives,
and booted again. The tape drive is still off line.
12/12/95 Tuesday 12:17pm EST
I took light down to take off the news drives which are suspected
of causing the scsi buss errors. When I went to reboot, it wouldn't
reboot. It wouldn't load vmunix, when it did load vmunix it said it
couldn't find /sd8d which is on another bus entirely.
I finally had to take the tape machine off the external scsi bus,
and then it booted fine.
Something is very strange in Denmark.
12/12/95 Tuesday 09:17am EST
Harmony taken down to install 12 new lines.
12/10/95 Sunday 08:45am EST
Light taken down to reinstall sd0. Going to go back to original
factory configuration and take external drives off line to see if
errors still happen. This will require that news be down.
12/09/95 Saturday 12:04pm EST
After replacing the internal cable and terminating board, light
became more unstable than before, producing load peaks of 20 (near
crashes) every 5 minutes or so during news expire. I took light down
and put back in the old terminator and cable, and also took the drive
tower apart to verify that all termination settings were correct, and
reseated all scsi cables.
12/09/95 Saturday 09:38am EST
Light crashed, and then crashed again on reboot.
Anyone want an ISP cheap?
12/09/95 Saturday 09:08am EST
Light taken down to replace internal scsi terminator back plane.
Got a scsi error 10 minutes after coming up again.
Took light down again to replace internal scsi cable.
That's the last of it, there is nothing more to replace except
for the entire external disk drive set.
Homer
12/08/95 Friday 2:37pm EST
Sendmail was down for about an hour from a uucp typo.
Sorry.
12/07/95 Thursday 12:51am EST
Light crashed at 11:36pm.
12/05/95 Tuesday 08:13am EST
Light taken down to unplug and reseat internal scsi cable from
motherboard and pc hard drive termination board. System running with
cable in place but no floppy nor CDROM (and no internal hard drive).
Found permissions set wrong on web counter directory, causing web
counters to fail. Web counters should be working correctly now.
12/05/95 Tuesday 02:29am EST
Coffee break 1:53am
12/04/95 Monday 12:55am EST
Light took coffee break at 12:14am
12/03/95 Sunday 1:55pm EST
Light was rebooted with new kernel.
At the same time, just after compiling new kernel, tape backup
became jammed, couldn't kill the process at all.
Then shutdown jammed, until I turned off tape drive.
Then boot -s jammed with "can't find swap on sd8b, no such etc."
Turned everything off, and on, and boot worked fine.
I changed the MAXUPROC from 25 to 256 in sys/param.h in kernel.
Apache .8 has an apparent limit of 26 virtual hosts, but the change to
256 did not fix the problem.
I also changed the max process id from 30000 to 300000.
I wonder what happens when the machine runs into a max process id?
12/03/95 Sunday 11:44am EST
Tape backup failed again with sd3 out of the loop.
Now we take sd1 off line, which is the main news drive, so
news will be down for a while.
Homer
12/03/95 Sunday 09:27am EST
Light took a coffee break at 5:37am, rebooting itself. The tape
drive was off line, so it is not the tape drive electronics.
I took light down at 7:00am to move /usr/local/ off of /sd3a and
over to /sd8d. sd3 has been taken off line, as it is suspected as
being the culprit for the crashes.
News is a mess because light keeps crashing during expires, so we
may have to clean the slate and and start over. I may institue a
policy whereby newsgroups are kept for a shorter period of time,
except for those explicitly being read or requested. This will
greatly aid in cleaning up the mess after succh crashes.
cgi's are still not working.
12/02/95 Saturday 8:21pm EST
Light was rebooted to try and solve cgi failures by increasing
kernel allocation tables (maxusers = 200) Cisco was brought off line
to stop incoming hits on web site. Neither worked.
At this time we have no idea why cgi's are not working.
Homer
12/02/95 Saturday 7:57pm EST
Cisco was rebooted again.
12/02/95 Saturday 3:26pm EST
Cisco was rebooted and Harmony was rebooted, kicking everyone
off, in order to clear out the old arp tables and install the new
ethernet address of the motherboard.
Prior to this the virtual domains were not working at all.
12/02/95 Saturday 2:22pm EST
Light was taken down for 10 minutes to swap /sd3d and /sd7a.
sd3d was the backup partition and sd7a was the erotica partition. Now
its the other way around. The purpose of this is to decrease the
backup time as /sd3d is single wide and /sd7a is double wide.
In the process all .pictures.erotica were lost.
News is still down and will continue to be down for a while.
12/02/95 Saturday 1:36pm EST
Light was down to replace the mother board for a few
hours this morning. Unfortunately, this did not repair the problem.
11/28/95 Tuesday 07:59am EST
System down for lab time from 7:30 till now.
Main boot drive sd4 was moved from scsi id 5 to 3. The system
boots by default from 3. This increases the chances the system will
reboot itself after a severe crash.
All swap space has been removed from the single wide scsi bus, to
take the pressure off the bus and to decrease the likelihood that a
scsi error will happen on the swap partition, an event most likely to
crash the system.
Sun has offered to send us a new motherboard which will be
installed when it arrives.
11/27/95 Monday 2:07pm EST
Due to a typo, sendmail was not functioning correctly for about
3 hours until now.
11/27/95 Monday 12:22pm EST
Light crashed at 2:00am from scsi bus errors.
We were down until 7:am.
Home drive is out again, however it is unlikely it is the source
of the problem.
Sun is sending a new mother board.
11/26/95 Sunday 08:37am EST
System down for lab time from 8 until now.
We put the home drive back in to see if it would fail to
boot because it was cold. It booted fine.
We then took light down for 30 minutes and turned it off to
see if it would fail to boot because it was cold. It booted fine.
The cold idea came from the fact that light would not boot after
being down the night of the blackout during the storm after being off
for about 30 minutes.
At this time I have no idea what is going on, we are still
getting errors and are considering how to best approach the problem
with minimum down time.
11/25/95 Saturday 08:37am EST
System was down for lab time from 7:30 until now.
We successfully moved all of the home drive partitions to the
sd4, and got the machine to boot from sd4. Not obvious.
If the system proves stable in this configuration, the home drive
will be reinstalled, formated to erase all its data, and sent off for
replacement.
News is still down for another 30 minutes or so.
Harmony was stable over night.
11/24/95 Friday 08:53am EST
The system was down for lab time from 7:30am until now.
Harmony was stable over night.
We moved the /usr and /var (mail) partitions to sd4 to get them
off the failing home drive sd0. We are still booting from sd0 but all
important partitions have been moved in anticipation of its
replacement.
Harmony was rebooted to set its configuration file correctly, for
the new subnets. However apparently a bug in the harmony code
prevents it from reading in subnetted masks properly, so they all have
to be added by hand after the booting takes place. This will require
more work and a few questions to Tech Support at Xylogics.
11/23/95 Friday 08:55am EST
System was down for lab time. Harmony again lost its default
route to the internet twice over night. We think we know what is
causing it, but not why.
The route loss seems to be connected with hand entering the route
after deleting it for test purposes. When the route is added through
the normal configuration procedure during boot up it is stable, if it
is then deleted and reentered by hand it seems to have a time to live.
We will query Xylogics tech support about this on Monday.
11/22/95 Wednesday 10:05am EST
For reasons that are still unclear, Harmony lost its default
route out to the internet this morning at about 8:30am. It should be
working again at the moment. We may have to reboot harmony if things
continue to be unstable.
Thanks to all who brought this to my attention.
11/20/95 Monday 08:55am EST
System was down from 7am until now for lab time.
11/17/95 Friday 12:19pm EST
CSU/DSU was brought off line resulting in a 20 second
disconnection from the internet.
Since the storm it has been getting errors, Sprint will be
testing it this morning between 6am and 7am. During that time there
will be interruption of service to the internet.
11/16/95 Thursday 10:22am EST
System was down from 8am to 9am for system work to rebuild new
home drive.
Expect continued down times during early morning hours.
News was down until now.
11/15/95 Wednesday 04:58am EST
System was down for 2 hours due to a power outage in upper
college town. We then had a LOT of trouble bringing it back up again.
possibly due to the same problem causing intermittent scsi buss
errors, which are going to be dealt with terminatedly in short order.
11/12/95 Sunday 07:38am EST
Cisco router was rebooted intentionally to clear out a problem
with config file transfers. It didn't fix it. Connection to the internet
was down for about 20 seconds.
11/08/95 Wednesday 07:18am EST
System was down for an hour until now to add the new disk drive.
All four 4gigs are now in one tower, this got rid of a SCSI cable
bringing the total to two on scsi bus one. The system has been stable
for a while, but it is not yet clear why, I believe the scsi bus
errors have merely gone dormant. We have taken a load off the home
drive by moving the swap partition to another drive, this may have
contributed to the quieting of the errors.
Please continue to expect down times during early morning hours
between 3am and 7am.
11/06/95 Monday 09:17am EST
System was down from about 6:00am to 7:00am to begin move of
home drive over to new drive. It is suspected that the home drive
is going bad which is causeing all the SCSI buss errors.
The system will be down every morning for some time until this is
resolve, between 3am and about 8am.
11/05/95 Sunday 10:09am EST
An error has been found in the Virtual Domain code, rebooting
to repair.
11/05/95 Sunday 07:32am EST
Going down again.
Swap partition moved to /dev/sd7b in preparation for moving root
directory to /dev/sd7.
My guess presently is that sd0 main drive is bad.
11/05/95 Sunday 06:52am EST
Going down to replace swap partition.
11/04/95 Saturday 03:06am EST
System down for 10 minutes to change SCSI cables. All cables
are brand new. Next check will be to change the swap partition off the
home drive.
11/03/95 Friday 05:12am EST
System was down for about 1 hour while we worked on scsi cables.
11/03/95 Thusday 012:02pm EST
The system crashed for unknown reasons, and rebooted itself,
probably due to sporadic I/O errors we have been getting on the
primary SCSI bus. The system will be down tonight to work on the
problem.
10/30/95 Monday 04:58am EST
News was down for 2 hours due to disk full errors. Sprint has
been down again, and maybe when they came back up, we got flooded with
back log that would have other wise expired.
10/25/95 Wednesday 02:14am EST
System was down for 2 hours due to system crash. Apparent cause
was badly seated cables, complicated by resulting corrupted super
blocks on 2 drives.
10/21/95 Saturday 04:51am EST
Light rebooted to increase virtual domains to 32.
Down for 20 minutes while I forgot to bring it back up again,
having too much fun with new color scanner.
10/04/95 Wednesday 12:45am EST
I crashed light playing with security holes.
10/02/95 Monday 04:48am EST
Harmony was rebooted to increase time out to 40 minutes.
09/27/95 Wednesday 01:52am EST
Harmony was down for about 15 minutes while we tried a security
upgrade. The upgrade failed. We are running as normal.
Light was rebooted during the process.
09/26/95 Tuesday 03:35am EST
We had a run away process last night send the system load to 4.0
and then to 7.0. This made things very slow.
09/16/95 Saturday 01:38am EST
System was down for 2 hours for upgrades. See news for details.
09/13/95 Wednesday 01:04am EST
Routes out of light to the internet were down for about
30 minutes by accident.
09/08/95 Friday 9:15pm EST
web server was down for about 2 hours by accident.
09/02/95 Saturday 4:59pm EST
An error in programming caused all password files to be renamed
improperly resulting in momentary loss of an ability to signon to the
system, and/or execute certain programs.
08/31/95 Thursday 3:23pm EST
System rebooted to physically rearrange drives back into place
after yesterdays fiasco.
08/30/95 Wednesday 8:46pm EST
We suffered a total power outage this afternoon around 3pm. A
transformer caught fire down the block towards college town. The
system stayed up for 20 minutes into the power blackout, and then we
had to shut down as the backup batteries had run out.
During this time I chose to do some back up maintenance on two
micro fans cooling disk drives that were beginning to whine and make
noise, a sign that they would soon jam risking overheating and damage
to the power supplies.
When I went to hook the whole system back together about 30
minutes after the power came on, I found that it would not boot. This
was apparently caused by SCSI connections gone bad. It took me three
hours or so to get the system working again, after trying many
different cables. The final answer was to cable the various
components in a different order. A very dissatisfactory solution.
08/28/95 Monday 1:30pm EST
System rebooted at 2am this morning to check out new virtual
domain kernel. It didn't work, so system rebooted again with normal
kernel.
08/24/95 Thursday 1:56pm EST
News was down until 8am this morning, to rebuild the
history data base.
08/23/95 Wednesday 8:18pm EST
News has been down for 2 hours and will continue to be down
for another 2 hours or so.
08/23/95 Wednesday 02:34am EST
We managed to get the modems converted to 115200 without bringing
down the system, however during the transitions some of the modems
were set wrong, and some people may have gotten rings without picking
up, or gotten garbage on the screen from mis set modems. All should
be working now.
08/21/95 Monday 6:45pm EST
System was down for 2 hours for intallation of 16 more modem lines.
New modems have not yet been configured.
08/20/95 Sunday 5:43pm EST
System was down for 3 hours for upgrades.
Annex was rebooted at end of upgrade.
See 'news' for details.
08/13/95 Sunday 7:50pm EST
News was not responding to slip requests since last night
due to an error in a configuration file.
08/12/95 Saturday 2:26pm EST
System rebooted to clear out some things.
08/11/95 Friday 4:35pm EST
FTP was broken for about 2 hours today while we tried to install
the new version. We are presently running the old version.
08/10/95 Thursday 1:47pm EST
System was taken down for about 30 minutes for upgrades. The
annex was also rebooted to install new settings.
08/09/95 Wednesday 2:42pm EST
News was down from about 12;00am last night till 2:30am when I
spotted it. Reading was not affected, but incoming articles were
stopped. The cause was an incorrect permission on the news directory
junk which caused a failure to write to it.
08/06/95 Sunday 8:24pm EST
System down for reboot to bring two new fast wide scsi drives on
line.
08/05/95 Saturday 8:04pm EST
System down for 30 minutes to install fast wide scsi bus card.
08/05/95 Saturday 7:32pm EST
I created my first mail loop today. Took the system to load 31
or so, for a few minutes.
08/04/95 Friday 7:08pm EST
T1 was down for 5 minutes while we rebooted the T1 CSU/DSU modem.
We have a defective T1 modem, nothing serious, but I can't log on to
it.
We are also getting data errors higher than normal which is a
completely separate matter.
The modem will be replaced shortly, and the data errors are being
looked into by sprint.
08/04/95 Friday 3:34pm EST
System down for 30 minutes.
I had to bring the system down to clean up an error I made.
08/03/95 Thursday 7:16pm EST
System rebooted to clean things up.
08/03/95 Thursday 7:05pm EST
I crashed the system.
07/30/95 Sunday 7:07pm EST
News was down for about an hour.
07/29/95 Saturday 8:25pm EST
System rebooted.