| Fri Dec 21 04:39:58 2007: |
The LDEO main mail server is operational again. |
| Thu Dec 20 18:03:32 2007: |
The facility to read mail has been turned off temporarily to allow the mail spool to finish rebuilding. We expect to turn this back on several hours from now.
You can continue to send mail. |
| Thu Dec 20 11:30:50 2007: |
Sending and receiving email will be slower than normal today as the mail spool rebuilds after a disk failure. The rebuild will continue for 24 hours. |
| Wed Nov 21 18:46:48 2007: |
Network Update: Yesterday prior to the 10PM scheduled CU router replacement, the defective router crashed at about 4:40PM and our Verizon 100Mbit circuit to CU failed independently at about 5:35PM. We were able to access the Internet through Qwest, but there were problems getting to Internet 2 sites and to CU sites. CUIT worked on this problem until it was resolved at about 9pm.
The scheduled router replacement was successfully completed at about 11PM, but the Verizon circuit remained out until 12:05PM today. During this period we had Internet connectivity, but there were minute long outages every 10 minutes or so. CUIT worked on this problem all day and at 4PM they found the external routing problem that was causing the LDEO problems. It appears that from 4PM on today the network has been stable.
Please note that because our Mirapoint SPAM filter could not always call home during these network dropouts to update it's SPAM signatures, you will have received more SPAM than usual between 11PM yesterday and 4PM today. When in doubt the Mirapoint delivers the message. |
| Wed Nov 21 09:09:34 2007: |
The 100Mbps connection between Morningside campus and Lamont is currently down. This is due to a circuit problem with our provider, Verizon. Verizon is working on the problem but there is no estimated fix time.
Nevis is connected to Lamont. Therefore this caused an outage for both Lamont and Nevis.
CUIT was able to restore connectivity for Lamont and Nevis using Nevis's Qwest internet connection. However, the connectivity is degraded.
Lamont and Nevis can communicate with the Internet, but they have problems communicating with Morningside campus and Internet 2.
CUIT is looking into these problems as well.
|
| Tue Nov 20 17:56:16 2007: |
CUIT's Vortex gateway is down again, and as a result we have lost internet connectivity. We had an outage earlier from 4:47pm to 5:03pm, which was restored.
CUIT personnel are enroute to troubleshoot the router now.
An emergency replacement of vortex-gw has already been scheduled for 10pm-12am tonight.
|
| Tue Nov 20 17:52:21 2007: |
CUIT's Vortex-gw is back up and connectivity to remote sites has been restored. This outage started at 4:47pm and ended at 5:03pm. |
| Thu Nov 15 17:23:47 2007: |
We have been experiencing a higher than normal amount of spam all afternoon. It looks like a denial-of-service attack. This has caused intermittent sending problems. We are continuing to investigate and we are trying to mitigate the problem. |
| Mon Oct 1 09:42:40 2007: |
We are trying to resolve a potential problem with email relaying. If
you've sent a email message to an outside user and have received a bounce back
with the reason "Relaying Denied" please contact request@ldeo.columbia.edu and
let us know.
|
| Wed Sep 26 09:43:03 2007: |
We need to apply security patches to Sun workstations and
will be doing this through the day today. The patches
require a reboot and we may be rebooting overnight tonight
(between 10:00 PM and 8:00 AM tomorrow). If this is a
problem please notify tnt@ldeo.columbia.edu.
|
| Wed Sep 19 14:55:19 2007: |
We appear to have lost Internet connectivity again at 1445. There is no further information at this time. |
| Wed Sep 19 14:30:35 2007: |
Internet connectivity has been restored as of 14:25. The problem was with CU's primary ISP who had a major network failure that was affecting all of its customers |
| Wed Sep 19 13:41:54 2007: |
All CU fell off the Internet at about 1325 today. CUIT is working to resolve the problem. |
| Tue Aug 14 10:38:46 2007: |
A major network upgrade at CU last night did not go as planned and CU's main link to the Internet failed. Backup plans were put into place and connectivity was temporarily retored at 8AM over an alternate route. LDEO is not experiencing any abnormal network problems as our commodity Interent traffic takes a different route. |
| Sun Jul 29 13:44:22 2007: |
There was an accident on 9W that caused a power outage at LDEO. For unknown reasons the generator in Geoscience powered up, but the UPS failed. B&G reset the UPS and the network and most services are back to normal. |
| Wed Jun 27 14:36:52 2007: |
This evening starting at 5PM a conrtactor will be making repairs on our fiber infrastructure. No outages are planned, however, it's possible that there may be a disruption. |
| Tue May 29 10:13:59 2007: |
The LDEO web was down from midnight last night until 10AM this morning. The Apache processes did not restart after normal automatic maintenance. |
| Sun May 20 10:51:00 2007: |
Yet another power failure from 8 - 9:30 AM today. Several home directories remain unavailable and the LDEO web site was blocked for 30 minutes by mounted filesystems in buildings that did not have generator power. |
| Tue May 15 14:21:22 2007: |
From about 12:44 to 13:27 today, May15th, the main LDEO mail server refused
to send mail because it's virus definition file was corrupted. Safeguards
to prevent this are in place but apparently failed. We are investigating. |
| Mon Apr 16 15:43:16 2007: |
Due to CPU Fan failure on web2, the LDEO main web pages are now served from web1 server.
Of course, the URL stay the same:
http://www.ldeo.columbia.edu |
| Fri Apr 13 10:18:00 2007: |
Matlab license servers will be down today on Friday April 12th from 12:30-1:00pm. This is another attempt to upgrade MATLAB to the latest 64bit version of 2007a. |
| Tue Apr 3 12:08:21 2007: |
The downtime for chaos is being rescheduled for a second time -- please note the new time below:
Chaos will be down from approximately 8:00 AM to 8:30 AM Wednesday, April 4th, to install a firmware patch. Dialup service will not be available during this time. Logins to chaos will not work. The Matlab license server will be down. We expect other services to remain up. |
| Mon Apr 2 16:21:44 2007: |
The downtime scheduled for chaos will not happen tonight. It
has been rescheduled for tomorrow, Tuesday, April 3rd, from 5:00 PM to 5:30 PM.
Chaos will be down from approximately 5:00 PM to 5:30 PM Tuesday, April 2nd, to install a firmware patch. Dialup service will not be available during this time. Logins to chaos will not work. We expect other services to remain up. |
| Mon Apr 2 09:41:58 2007: |
Today, April 2, starting at 10 AM, we will be installing an additional spam filter box on the Lamont main mail system to test it out. The filtering done by the box will be in addition to that which we're already doing. No downtime is expected. |
| Thu Mar 29 16:39:43 2007: |
Chaos will be down from approximately 5:00 PM to 5:30 PM Monday, April 2nd, to install a firmware patch. Dialup service will not be available during this time. Logins to
chaos will not work. We expect other services to remain up. |
| Tue Mar 20 14:50:09 2007: |
On Wednesday, 3/28/07,CUIT will be reloading the router which handles our Morningside Campus and Internet2 connections. The reload will begin at 4AM and should take no more than 5 minutes. Network service to remote sites should be restored by 4:10 AM. The reload is necessary to diagnose the router's memory problem.
|
| Mon Feb 5 13:07:19 2007: |
The CPU Fan replacement failed on Friday Feb 2nd, 2007 at 5 pm, and the web primary server could not be restarted.
At that time, we served the LDEO web main web pages on a backup server with a few services not available during the weekend. On Monday Feb 5th all web related services were up at 10:30 am. Hardware on web1 was replaced on Monday morning, and we hope to run web1 and web2 shortly to serve our web pages http://www.ldeo.columbia.edu at all times. |
| Mon Feb 5 10:00:18 2007: |
The DHCP server (ouzel) is back up and running.
|
| Mon Feb 5 09:40:18 2007: |
Currently the DHCP server is down. This server also hosts some home directories.
You may not be able to connect to the net from a machine that uses dynamic IP
addressing. We hope to have the server back up shortly.
|
| Fri Feb 2 14:15:29 2007: |
Emergency Web Server Reboot today (2/2/2007) at
5:00 p.m or 5:30pm, due to a CPU fan failure.
The down time is expected to be less than 1/2 hour.
|
| Sat Nov 25 11:56:08 2006: |
We are adjusting spam filter scores. This will happen as a matter of course from now on, as we try to stem the constantly changing tactics of the spammers. Please be aware and let us know immediately of problems. |
| Thu Nov 9 11:23:34 2006: |
Homedir, one of the main home directory servers, failed last night. It took until 11:15AM this morning to recover. Other machines including web and email hung on the inability to access home directories. |
| Fri Oct 13 17:52:16 2006: |
Our matlab license server was upgraded to MATLABŪ 7.3
to accommodate students and faculty who own a MacBook Pro, which
is based on the Intel chip. The previous versions of matlab did
not run on these machines.
These licenses are backward compatible. You don't need
to change your license.dat file.
I have the new CDs, and the new PLP number for the users
who wish to upgrade.
As usual please report any problems to request@ldeo.columbia.edu |
| Tue Oct 3 10:45:24 2006: |
One of our home directory server is unresponsive. Many users are experiencing difficulties to login or check/send email. Our web server is slow to respond, and some personal web pages are inaccessible. We are working on the problem.
|
| Mon Oct 2 07:01:24 2006: |
At 6:55 AM today spam filtering was upgraded to a new version. Hopefully you will notice a reduction in the amount of spam you receive. |
| Thu Sep 28 14:55:27 2006: |
QWEST is performing a network upgrade starting on 3 AM GMT on 9/30/06 to
12 PM GMT on 10/01/06. Qwest is the main network uplink for Nevis Labs
and the Lamont campus. Users at Lamont shouldn't be affected
by this outage due to network redundancy, but due to the intermittent
nature of the outage they may experience slowness in connecting to the
outside world.
|
| Thu Aug 17 15:24:11 2006: |
Lotus Domino Application server runs part of our website at LDEO.
I am upgrading this server software next week on
Tuesday August 22nd.
The online phone directory web site
http://www.ldeo.columbia.edu/directory
is NOT going to be available during this upgrade time.
Our LDEO research portals "Database & Repository",
"Facilities", "Research Tools", and "Projects & Initiatives"
will NOT be accessible as well.
The rest of the lamont website, and the individual project
web sites listed in these portals are not effected by this down
time.
Please contact me directly, if you have any questions or
concerns. |
| Thu Aug 3 15:39:07 2006: |
The CUIT Host Master will be pointing www.earth.columbia.edu and www.earthinstitute.columbia.edu from an LDEO site to Mandela server at EI around 5pm today. According to them the batch will run around 5 so hopefully within an hour or so it will be running off our Mac XServe.
|
| Thu Aug 3 14:00:29 2006: |
The CUIT Host Master will be pointing www.earth.columbia.edu and www.earthinstitute.columbia.edu from an LDEO server to Mandela server at EI around 5pm today. According to them the batch will run around 5.
|
| Wed Jul 26 14:25:17 2006: |
ESRI (Environmental Systems Research Institute, Inc) software license upgrade was rescheduled for Friday July 28, 2006 at 10am. A short interruption in the use of ArcInfo/ArcGIS suite is expected. |
| Fri Jul 21 14:15:37 2006: |
Break-in & email outage incident report:
The intruders mounted their attack from within by using LDEO computer accounts with weak or guessed passwords. It is much easier to attack from within with downloaded software than it is to try through the Internet front door. Their attacks were successful on Sun machines that had not been patched within the previous month. Once they gained privileged access the attackers installed password sniffing software. We applied the latest Sun patches overnight, but had to reboot machines to have the patches take effect. The mail server was unsuccessfully rebooted at 0715 Thursday morning and it took until 1245 to get it back up.
Timely information about the email outage was available at the Computer Help Line X8181 and on the web at About LDEO -> Computing -> Latest System Status
http://www.ldeo.columbia.edu/ldeo/it/announce.htm
We will be announcing a mandatory password change campaign very shortly. |
| Thu Jul 20 12:50:55 2006: |
LDEO email is now working as of 12:45PM. As far as we know, no mail was lost it was only delayed. |
| Thu Jul 20 12:37:41 2006: |
We have made good progress returning the email server to service, but we still can not estimate an exact time email will be available. |
| Thu Jul 20 11:17:28 2006: |
LDEO email is still unavailable after an emergency reboot early this morning. We are working with Sun to try and resolve the problem and we do not have a time to resolution yet. |
| Thu Jul 20 08:57:48 2006: |
LDEO email is not available. The server did not come back online after patching and rebooting. We are working to resolve the problem. |
| Mon Jul 17 11:18:20 2006: |
ESRI (Environmental Systems Research Institute, Inc) software license upgrade is scheduled for Wednesday July 18, 2006 at 4pm. A short interruption in the use of ArcInfo/ArcGIS suite is expected.
|
| Wed Jul 12 14:53:43 2006: |
Columbia's oracle server connection problem and process control issues has affected some applications.
These affected applications include:
WIND
CubMail
Remedy
Samurai
Sakai
LUNA
ID systems functions
CUIT are working on the problem.
|
| Tue Jul 11 09:28:36 2006: |
MATLAB is not supported on MacBook Pro or any Intel-Based Mac machines. MATLAB 7.1 (R14SP3) and 7.2 (R2006a) are not supported on Intel-based Mac (Rosetta) machines. MATLAB R2006b will also not be supported on Intel-based Mac machines. The MathWorks does plan to support Intel-based Macs. However, they do not yet have an estimated release date for this support.
http://www.mathworks.com/support/solutions/data/1-25L41S.html?solution=1-25L41S
|
| Sat Jun 17 19:53:37 2006: |
Internet traffic was severely impacted between 1600 and 1930 today as the LDEO firewall fought off an internal denial-of-service attack. The external target systems were never harmed as the firewall did it's job, however, the local bandwidth from our main router to the firewall was saturated and normal traffic was unable to pass. We are still investigating the primary cause. |
| Sat Jun 3 09:32:12 2006: |
The Verizon 100Mbit circuit came back online earlier this morning. The network is operating normally. |
| Fri Jun 2 15:59:15 2006: |
We have lost our Verizon 100Mbit circuit to Columbia. The backup microwave circuit and alternate routing is working, however, some web sites and some hosts remain unreachable or slow. CUIT folks are trying to figure out and fix the problem which is external to LDEO. |
| Mon May 22 10:53:12 2006: |
The network switches in the second floor New Core Lab closet will be replaced at 7:30PM this Tuesday May 23rd. Folks in the Core Lab at that time will loose network connectivity for about 30 minutes. |
| Mon May 8 15:58:06 2006: |
There are reports that some websites don't load and certain protocols to some sites don't work. At this time it appears these problems are external to LDEO and CU, but we continue to investigate. |
| Mon May 8 14:09:00 2006: |
Leadbelly was repaired and is back in service as of noon. |
| Sun May 7 16:18:52 2006: |
The power supply on leadbelly has failed and the system is down. Users with home directories on leadbelly will have trouble logging in and reading email. Arrangements are being made to replace the failed part. |
| Tue Apr 25 16:25:37 2006: |
iCal publishing web server is being moved to a new faster harder on
Monday May 1st from 3:00-5:00pm. This service will not be available during the
down time. The move should be transparent to you once the server is back up again.
No changes in the accessing, publishing, or the host name of |
| Mon Apr 24 10:15:58 2006: |
Matlab license servers will be down on Tuesday April 25th from 12:30-1:00pm. |
| Fri Apr 14 14:28:34 2006: |
Matlab server will be down on Monday April 17th from 5:00-5:30pm. |
| Thu Apr 13 14:50:27 2006: |
The HP60 plotter is working again. |
| Wed Apr 12 14:47:45 2006: |
The hp60 plotter has a hardware problem. Please call Doug x8548 or
Mahdad x8698 or send mail to request to arrange
for a plot. We can plot your plot on the older HP54 plotter. |
| Tue Mar 21 15:14:22 2006: |
We are continuing to see commodity Internet outages. CUIT will manually re-route our traffic over the backup link at 5PM today. The delay is because CUIT does not want to disconnect Nevis. |
| Tue Mar 21 11:06:21 2006: |
MATLAB license server upgrade to version 2006a1 is scheduled for tomorrow March 22nd at 9:30am. The outage for MATLAB should not last more than 10-15 minutes. If the installation is successful, you can request the client CDs by contacting me at mahdad@ldeo.columbia.edu |
| Mon Mar 20 11:43:55 2006: |
Our route to the commodity Internet is dropping in and out. We are retransmitting about 15% of the packets. CUIT (former AcIS) has been notified. |
| Fri Feb 10 11:20:50 2006: |
Our uplink provider will be performing a 5 minute maintenance on our commodity Internet link on Sunday 2/12/06. Maintenance could happen anytime during the 4 hour window of 1 AM to 5 AM. Users attempting to connect to the commodity internet during this maintenance window may experience slow connections as routes are cutover to our backup link. Users connecting to Internet2 shouldn't be affected by this outage. |
| Mon Jan 30 11:24:44 2006: |
If you are archiving files on the LDEO ftp server you need to read this message.
In the recent past the LDEO anonymous ftp server (ftp.ldeo.columbia.edu) has been
routinely running out of space and we now need to implement new policies to ensure
there is always enough space for normal day-to-day operation of this resource. There
are two reasons /home/ftp fills up. First, when files are published in the
/home/ftp/pub directory they are often not removed by the owner once they are picked
up. Second, there are a growing number users wanting to maintain files for months or
years on the ftp server. The new policy addresses both these problems.
A new /home/ftp/archive area has been created. Files can be downloaded from this
area, but outsiders may not upload to it. LDEO users should create a directory
(folder) in the archive area and then move those files that really need to remain on
the ftp server for more than two weeks into this new folder. Users will have until
Feb 10th to move archive files. From then on we will automatically remove files from
/home/ftp/pub that are more then 14 days old.
You may move archive files using your favorite FTP GUI (WsFTPle, WinSCP, FuGu etc)
or you may login and use the command lines as follows:
1) Login into the system: ssh ftp.ldeo.columbia.edu
2) Create an archive directory: mkdir /home/ftp/archive/YOUR-FOLDER-NAME
3) Navigate to the pub ftp area: cd /home/ftp/pub/YOUR-FOLDER-NAME
4) Move your files to Archive area: mv FILENAME /home/ftp/archive/YOUR-FOLDER-NAME
|
| Wed Jan 18 14:10:25 2006: |
Some mail was delayed today (1/18/2006) when power failures prevented
the mail server from accessing home directories. The problem was resolved
at about 1:43 PM. If your home directory was unavailable, mail may have been
delivered to your mailbox instead of the place(s) designated by your
.forward or .procmailrc file. |
