SERVICE RESTORED: Geckomail, 3:35 pm, Monday, Nov. 23rd, 2009
Symptoms:
Earlier this afternoon, IT Express received reports that some users could not access their campus mail via Geckomail. A configuration setting on one of the authentication servers caused the problem.
Actions:
System administrators have solved the problem by removing the offending server from the authentication pool. Geckomail service has been fully restored.
Affected Services:
Geckomail
If you have any questions concerning this outage, please contact IT Express at (530) 754-HELP (4357).
- Notification modified: 2009-11-23 16:24:40
Listproc unavailable: Mon 11/23/09 0207 - 0230
ListProc was temporarily unavailable. System administrators investigated and restored service by 0230.
- Notification modified: 2009-11-23 02:42:27
Service Degradation: CAS 09:37- 09:42 11/16/2009
CAS authentication experienced a brief degradation of service due to a high load.
- Notification modified: 2009-11-16 10:09:15
Network Maintenance: Areas 42,51 IDF Switch 11/13/09 06:00-06:30
=================== Network Maintenance ====================
Description: Areas 42 & 51 IDF Switch Maintenance
Start of Window : 11/13/2009 06:00
End of Window : 11/13/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the code on various IDF switches to the latest
vendor recommended version.
Impact:
Multiple brief outages as the devices are rebooted.
Affected:
BUILDINGS:
DV 200 B ST
DV 400 F ST
GENOME & BIO SCI FACILTY
STORER
- Notification modified: 2009-11-13 06:55:32
Adobe Connect Service
The UCD-hosted Adobe Connect service has been restored.
We made a temporary DNS change during the service outage. If you or any of your clients are unable to
connect to the service, we recommend rebooting the computer or manually flushing the DNS cache.
- Notification modified: 2009-11-10 13:24:22
Network Maintenance: Area 6 IDF Switch 11/10/2009 06:00-07:00
=================== Network Maintenance ====================
Description: Area 6 IDF Switch Maintenance
Start of Window : 11/10/2009 06:00
End of Window : 11/10/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the code on various IDF switches to the latest
vendor recommended version.
Impact:
Multiple brief outages as the devices are rebooted.
Affected:
BUILDINGS:
ASMDSN
HOAGLD
HOUSING ADMN
HUTCH
KERR
ROBBNS
TB 109-112
WELLMAN
- Notification modified: 2009-11-10 08:13:54
Network Maintenance: Area 1 IDF Switch 11/16/2009 06:00-07:00
=================== Network Maintenance =====================
Description: RESCHEDULED: Area 1 IDF Switch Maintenance
Start of Window : 11/16/2009 06:00
End of Window : 11/16/2009 07:00
(all times are in 24 hour format)
Purpose:
Rebooting various IDF switches to load the latest vendor recommended version of code
Impact:
Multiple brief outages as the switches are rebooted
Affected:
BUILDINGS:
CONFERENCE CENTER
EARTH AND PHYSICAL SCI
FACILITIES SERVICES
GSM GALLAGHER HALL
KING HALL TRAILER
MRAK
RMI NORTH
RMI SENSORY
RMI SOUTH
- Notification modified: 2009-11-16 06:39:28
Campus Email Service Degradation:23:00 11/06/09 - 01:30 11/07/09
The campus email service experienced degraded performance from 23:00 11/06/2009 - 01:30 11/07/2009 due to a network switch failure . Email messages continued to be delivered during the service degradation. Data Center administrators have resolved the problem.
- Notification modified: 2009-11-07 01:51:19
Network Maintenance: Area 3 IDF Switch 11/06/2009 06:00-07:00
=================== Network Maintenance =====================
Description: Area 3 IDF Switch Maintenance
Start of Window : 11/06/2009 06:00
End of Window : 11/06/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the code on various IDF switches to the latest vendor recommended version.
Impact:
Multiple brief outages as the devices are rebooted.
Affected:
BUILDINGS:
BKSTR WAREHS
FIRE&POLICE
HUBERT HEITMAN STAFF CTR
PLANT REPRODUCTIVE BIOLOG
TB 201
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-11-06 06:31:43
Network Maintenance: Campus Border Router 11/05/2009 06:00-07:00
=================== Network Maintenance =====================
Description: Campus Border Router - CENIC DC Upgrade
Start of Window : 11/05/2009 06:00
End of Window : 11/05/2009 07:00
(all times are in 24 hour format)
Purpose:
Adjust network traffic to utilize a second Gigabit Ethernet connection to CENIC's Digital California (DC) router in Oakland.
Impact:
None expected. An additional Gigabit Ethernet circuit will be added to the current network connection. CENIC DC provides service to California K-20 and Internet Commodity Peering sites.
Affected:
128.120.0.0/16
169.237.0.0/16
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-11-05 06:20:17
Network Maintenance: Area 2 Switch 11/05/2009 06:00 - 07:00
=================== Network Maintenance ====================
Description: Area 2 Switch Maintenance
Start of Window : 11/05/2009 06:00
End of Window : 11/05/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading various IDF switches to the latest vendor
recommended version of code.
Impact:
Multiple brief outages as the devices are rebooted.
Affected:
BUILDINGS:
ADV MATERIALS RES LAB
BAINER
ENGINEERING 3
KEMPER HALL
WATERSHED SCIENCE RES CTR
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-11-05 06:46:03
SmartSite Maintenance: Tuesday 11/03/2009 05:00 - 06:22
SmartSite was down for a regularly scheduled maintenance.
- Notification modified: 2009-11-03 06:23:39
Network Maintenance: UCDHS Border Router 11/03/2009 05:00-05:27
=================== Network Maintenance ====================
Description: UCDHS Border Router
Start of Window : 11/03/2009 05:00
End of Window : 11/03/2009 06:00
(all times are in 24 hour format)
Purpose:
Replace a failing module in router's host subsystem.
Impact:
No loss of network connectivity is expected between UCDHS
and Campus sites. Network connectivity between UCDHS and external sites may experience slowdowns and connection resets as repair work is performed during the maintenance window.
Affected:
152.79.0.0/16
- Notification modified: 2009-11-03 05:31:43
Campus Internet Outage : 11/02/2009 11:41 -11:44
Impact:
Campus connectivity to many internet sites was disrupted by appearent issues with the campus firewalls.
Locations Affected:
Most campus internet bound traffic.
Status:
BGP issues with the campus firewalls had disrupted connectivity to off-campus sites. As this message was being composed, BGP sessions restored and off campus traffic appeared to be pass again. NOC personnel are investigating the issue.
- Notification modified: 2009-11-02 12:00:33
Network Maintenance: Area 4 IDF Switch : 11/09/2009-06:00
=================== Network Maintenance ====================
Description: Area 4 IDF Switch Maintenance
Start of Window : 11/09/2009 06:00
End of Window : 11/09/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the code on various IDF switches to the latest
vendor recommended version.
Impact:
Multiple brief outages as the devices are rebooted.
Affected:
BUILDINGS:
GENOME & BIO SCI FACILTY
THURMAN
TUPPER HALL
VET MED 3A
VM OFC ANNEX
VMIF (AKA) GLADYS VALLEY
VMTH
VMTH HOLD
VMTH ISOL
VMTH SURG
VMTH WARD
VLANs:
ANML-RES-SVC-1
CTR-CPANHLTH-1
CTR-CPANHLTH-2
CTR-EQUIN-HL-1
CTR-EQUIN-HL-2
DEAN-MED-ADM-11-NR
DEAN-MED-ADM-12-FW
DEAN-MED-ADM-13-FW
DEAN-MED-ADM-14-FW
DEAN-MED-ADM-2-FW
DEAN-MED-ADM-3-FW
DEAN-MED-ADM-4-FW
DEAN-MED-ADM-5-DMZ
DEAN-MED-ADM-5-FW
DEAN-MED-ADM-6-FW
DEAN-MED-ADM-8-FW
DEAN-MED-ADM-9-FW
DEAN-VETMED-1-DMZ
DEAN-VETMED-1-FW
DEAN-VETMED-2-DMZ
DEAN-VETMED-2-FW
DEAN-VETMED-3-FW
DEAN-VETMED-4-NR
DEAN-VETMED-5-NR
GENOME-CTR-1-FW
HVAC-1-FW
HVAC-2-FW
INTRCOL-ATHL-1
IT-COMMUNRES-13-FW
M-GEN-I&R-1
MOL&CELL-BIO-1-FW
MOOBILENET(X)
MU-GENERAL-1
NMR-FACILITY-1-FW
O&M:UTILITIES-1
PHYS-PLANT-7
PLANT-PATH-1-FW
VET-DIAG-LAB-1
VH-ADMIN-1-DMZ
VH-ADMIN-1-FW
VH-ADMIN-2-FW
VH-ADMIN-3-NR
VH-SANS-1-NR
VM-ACADPROGS-1-FW
VM-CENT-SERV-1
VM-GEN-I&R-1-FW
VM-GEN-I&R-6-FW
VM-PATHOLOGY-1-FW
VM-RESEARCH-1-FW
VM-VECTOR-DIS-RSCH-1
WLS-UNMANAGED-1
- Notification modified: 2009-11-09 06:42:18
Network Maintenance: Areas 5,8,10 & 11 IDF Switch: 11/12 - 06:00
=================== Network Maintenance ====================
Description: Areas 5,8,10 & 11 IDF Switch Maintenance
Start of Window : 11/12/2009 06:00
End of Window : 11/12/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the code on various IDF switches to the latest
vendor recommended version.
Impact:
Multiple brief outages as the devices are rebooted.
Affected:
BUILDINGS:
ATIRC
ATIRC OFFICE TRL #1
CNPRC CEF
DV 267 COUSTEAU PLACE
HEALTH & WELLNESS CENTER
NEUROSCI BLD
PRIMATE CENTRAL SUPPLY
PRIMATE MED OFF
TB 196
TSRF
VLANs:
ARCH-&-ENGR-2
CTR-MIND-BRAIN-1-DMZ
CTR-MIND-BRAIN-1-FW
CTR-MIND-BRAIN-3-DMZ
CTR-MIND-BRAIN-3-FW
CTR-NEUROSCI-3
DEAN-L&S-1-FW
DEAN-MED-ADM-6-FW
ENG-CIVL-ENV-8
ENG-MCH&AERO-3
HVAC-1-FW
HVAC-2-FW
IT-COMMUNRES-13-FW
M-NEUROLOGY-1-FW
M-NEUROSURG-1
M-NEUROSURG-2
M-OTOLYMGOL-1
MATHEMATICS-1-FW
MOOBILENET(X)
O&M:UTILITIES-1
PHYS-PLANT-2-FW
PHYS-PLANT-7
PRIM-RES-CTR-2-FW
PRIM-RES-CTR-3-FW
PRIM-RES-CTR-4
WLS-UNMANAGED-1
- Notification modified: 2009-11-12 06:56:38
Network Maintenance:Campus Border Router 10/29/2009 06:00-07:00
=================== Network Maintenance ====================
Description: Campus Border Router - CENIC ISP Upgrade
Start of Window : 10/29/2009 06:00
End of Window : 10/29/2009 07:00
(all times are in 24 hour format)
Purpose:
As a part of CENIC's Internet Service Provider (ISP) network
upgrade project, traffic to/from the CENIC ISP network will
be migrated from the current Oakland ISP (OAK-ISP) fiber
equipment to new hardware.
Impact:
Campus may experience minor delays in response from commodity
Internet sites while network traffic transitions to the new
network path.
Affected:
128.120.0.0/16
169.237.0.0/16
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-10-29 06:30:49
Network Maintenance: Campus Border Router 10/28/2009 06:00-07:00
=================== Network Maintenance =====================
Description: Campus Border Router - CENIC DC Upgrade
Start of Window : 10/28/2009 06:00
End of Window : 10/28/2009 07:00
(all times are in 24 hour format)
Purpose:
Establish a second Gigabit Ethernet connection to CENIC's Digital California (DC) router in Oakland.
Impact:
None expected. An additional network circuit will be established to the CENIC DC network, but Campus will continue to prefer the current CENIC DC network connection to reach California K-20 sites.
Affected:
128.120.0.0/16
169.237.0.0/16
- Notification modified: 2009-10-28 07:05:26
Network Maintenance: Resnet Packetshaper 10/23/2009 06:00-06:30
=================== Network Maintenance =====================
Description: Resnet Packetshaper Maintenance
Start of Window : 10/23/2009 06:00
End of Window : 10/23/2009 07:00
(all times are in 24 hour format)
Purpose:
Rebooting the Resnet Packetshaper to enable vendor recommended system settings
Impact:
None expected. The traffic will be re-routed during the reboot.
Affected:
All Resnet Internet and I2 traffic.
- Notification modified: 2009-10-23 06:35:39
Network Maintenance: Campus Border Router 10/22/2009 06:00-07:00
=================== Network Maintenance =====================
Description: Campus Border Router - CENIC DC Upgrade
Start of Window : 10/22/2009 06:00
End of Window : 10/22/2009 07:00
(all times are in 24 hour format)
Purpose:
As a part of CENIC's Digital California (DC) network upgrade
project, traffic to/from the CENIC DC network will be
migrated from the current Oakland DC (OAK-DC) fiber
equipment to new hardware.
Impact:
Campus may experience minor delays in response from CENIC DC
sites while network traffic transitions to the new network path.
Affected:
128.120.0.0/16
169.237.0.0/16
______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-10-22 06:25:01
Network Maintenance: Tipping Point 10/22/2009 05:00 - 08:00
=================== Network Maintenance =====================
Description: Tipping Point IPS Upgrade
Start of Window : 10/22/2009 05:00
End of Window : 10/22/2009 08:00
(all times are in 24 hour format)
Purpose:
Upgrade the Tippingpoint IPS sensors and management cluster
Impact:
Campus should not see connectivity issues as the service is bypassed during the upgrade
______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-10-22 06:51:53
CAS Degraded Performance 10/21/09 14:10 - 14:16
The Campus Authentication system experienced degraded performance. System Administrators successfully restarted the service and are still investigating the root cause.
- Notification modified: 2009-10-21 14:54:39
Network Maintenance: Core Router Hardware 10/21/2009 05:30-06:00
=================== Network Maintenance =====================
Description: Core Router Hardware Maintenance
Start of Window : 10/21/2009 05:30
End of Window : 10/21/2009 07:00
(all times are in 24 hour format)
Purpose:
A failing interface card in one of the two core campus routers will be replaced.
Impact:
Several brief periods of network slowness may be experienced as traffic re-routes around the affected device. No extended periods of network degradation are expected.
Affected:
Campus traffic passing through the core routers may experience brief periods of slowness or packet loss. In general, this includes most traffic that is not restricted to a single building.
- Notification modified: 2009-10-21 06:01:54
SmartSite Maintenance: Tuesday 10/20/2009 05:00 - 05:45
SmartSite-PROD experienced downtime for a regularly scheduled maintenance.
- Notification modified: 2009-10-20 05:53:23
Service Outage: AFS 10/17/2009 12:45-14:30
The following services were down for nearly two hours while System Administrators repaired an AFS problem. The services:
MySpace
Course Management
Were down during the maintenance period.
All administrative services (Banner,DaFIS,web sites) located on the admin cell were degraded for maintenance period.
- Notification modified: 2009-10-17 14:36:26
Course Management System 10/17/2009 09:00-14:30
Clients experienced problems downloading and uploading files within the Course Management system. System Administrators have resolved the underlying AFS server problem and restored Course Management functionality.
- Notification modified: 2009-10-17 14:38:24
Mothra Database Extended Downtime: 10/17/2009 04:00 - 05:00
The Mothra Database underwent an extended maintenance. Services which relied upon the Mothra database were unavailable.
- Notification modified: 2009-10-17 05:32:44
Network Maintenance: CENIC HPR Upgrade 10/15/2009 06:00 - 07:00
=================== Network Maintenance =====================
Description: Campus Border Router - CENIC HPR Upgrade
Start of Window : 10/15/2009 06:00
End of Window : 10/15/2009 07:00
(all times are in 24 hour format)
Purpose:
As a part of CENIC's High Performance Research (HPR) network upgrade project, traffic to/from the CENIC HPR network will be migrated from the current Oakland HPR (OAK-HPR) router to a new Sunnyvale HPR (SVL-HPR2) router.
Impact:
Campus may experience minor delays in response from CENIC HPR, Internet2 and National Lambda Rail (NLR) sites while network traffic transitions to the new path.
Affected:
128.120.0.0/16
169.237.0.0/16
- Notification modified: 2009-10-15 06:54:21
Network Maintenance: Campus Wireless : 10/16/2009-17:00
=================== Network Maintenance =====================
Description: Campus Wireless
Start of Window : 10/16/2009 17:00
End of Window : 10/16/2009 18:00
(all times are in 24 hour format)
Purpose:
Upgrade code on wireless routers to latest vendor recommended
version
Impact:
Loss of wireless connectivity as Controllers and Access
Points were rebooted. Users needed to re-authenticate
after upgrade.
Affected:
All Campus Wireless:
moobilenet & moobilenetx
- Notification modified: 2009-10-16 20:42:14
Moobilenet/Moobilenetx Outage - 2009-10-14 12:00- 15:02
======================== Network Outage Notice ====================
0000-00-00 00:00:00
UPDATE:
Impact:
Moobilenet and Moobilenetx are not working.
Locations Affected:
Multiple buildings on campus are experiencing a loss of
wireless networking.
Status:
A combination of issues lead to the wireless issues this
morning and afternoon. Failing hardware likely caused the
initial trouble reports of absent moobilenet(x) SSIDs at
various campus locations. When the primary controller was
failed over to the redundant backup, a configuration
mismatch on the backup caused additional reports of sluggish
and failing connections.
At present, the backup controller remains in service and all
wireless traffic should be passing normally. NOC personnel
will continue to work to isolate the issue with the primary
wireless controller. Additionally, the wireless vendor will
assist in determining the source of the config
synchronization issue between the primary and backup
controllers.
Connections to the wireless network continue to be sluggish
and intermittent. Additional information will be provided as
it becomes available.
- Notification modified: 2009-10-14 15:32:23
SmartSite Maintenance : Oct 14, 2009 : 12:00 to 17:00
We are performing scheduled maintenance on the *non-production* SmartSite servers on October 14, 2009 from 12:00 to 17:00. For more information, please see:
https://confluence.ucdavis.edu/confluence/x/Q4Kw
- Notification modified: 2009-10-14 17:49:02
Power Outage 10/13/09 15:46 - 10/13/09 19:00
======================== Network Outage Notice ====================
10/13/2009 15:46 - 10/13/09 19:00
UPDATE:
Impact:
There is a widespread power outage affecting multiple areas on campus.
Locations Affected:
Multiple buildings in areas 2-5, 8, 9 and 12.
Status:
Access to the weather station was granted, and all network devices are back online at this time.
- Notification modified: 2009-10-14 15:28:43
SmartSite Maintenance, Tuesday, October 13th, 2009 04:00 - 05:20
SmartSite was unavailable due to a scheduled maintenance.
- Notification modified: 2009-10-13 05:38:15
XEDA Mail: Sporadic Issues 10/11/09 - 10/12/09 08:00
The XEDA Mail Service experienced possible sporadic issues this weekend as one of exhub servers smtp service was malfunctioning.
- Notification modified: 2009-10-12 09:33:24
DNS Server Move, Wednesday, October 14, 2009 6:00-6:30pm
PURPOSE:
One of three off-campus DNS servers was moved to a different rack at the NOC to provide higher availability.
IMPACT:
Some off-campus users may have experienced slight lag when establishing new connections to services on campus.
If you have any questions concerning this maintenance please contact IT Express at the number listed below.
IT Express
530-754-HELP (530-754-4357)
- Notification modified: 2009-10-14 18:30:20
Network Maintenance: Packetshaper Upgrade 10/09/2009 06:00-06:30
Description: Campus Packetshaper Software Upgrade
Start of Window : 10/09/2009 06:00
End of Window : 10/09/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on the Campus Packetshaper to the latest vendor recommended version.
Impact:
None expected. The traffic will be rerouted during the reboot.
Affected:
All Campus commodity Internet traffic.
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-10-09 06:30:43
Smartsite Maintenance: Thursday, October 8, 2009 05:30 - 06:00
SmartSite was unavailable due to maintenance which will resolve several minor issues which have been reported during the past week.
- Notification modified: 2009-10-08 06:03:43
Network Maintenance: Campus Border Router 10/08/2009 06:00-07:00
Start of Window : 10/08/2009 06:00
End of Window : 10/08/2009 07:00
(all times are in 24 hour format)
Purpose:
Establish BGP peering sessions between the Campus Border router and CENIC's Oakland Digital California (OAK-DC2) and Oakland Internet Service Provider (OAK-ISP) routers.
Impact:
None expected. New BGP peering sessions will be established, but Campus will continue to prefer the current CENIC Oakland DC and ISP routers to reach California K-20 and commodity Internet sites.
Affected:
128.120.0.0/16
169.237.0.0/16
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-10-08 06:43:12
Temporary Affiliate Form, Monday 10/5/09 1500 - 1630
Subject: SERVICE OUTAGE: Temporary Affiliate Form, 3:pm, Monday, October 5th, 2009
Symptoms:
Users were reporting blank pages when updating information via the Temporary Affiliate Form (TAF) web site.
Actions:
System administrators took the service offline to isolate and fix the issue.
Time to Repair:
System administrators restored service as of 4:30 pm on October 5th, 2009.
Affected Services:
The Temporary Affiliate Form web forms accessed via
http://taf.ucdavis.edu
If you have any questions concerning this outage, please contact IT Express at the number listed below.
IT Express
530-754-HELP (530-754-4357)
-------------------------------------------------------------------------
Answers to 'ithelp' frequently asked questions are posted on the Web.
Please check: http://itexpress.ucdavis.edu/
-------------------------------------------------------------------------
- Notification modified: 2009-10-05 20:53:05
Resnet and Campus Packetshaper upgrade 10/05/09 0600 - 0700
Start of Window : 10/05/2009 06:00
End of Window : 10/05/2009 07:00
(all times are in 24 hour format)
Purpose:
Performing a software upgrade on the Campus and Resnet Packetshapers to address a recently released security vulnerability.
Impact:
None expected. Traffic will be rerouted during the reboot of the devices
Affected:
All Campus traffic to the commodity Internet. All Resnet traffic to the commodity Internet and I2.
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-10-05 06:50:28
Network Outage: Kerr(B043) Wellman(B042) 10/02/2009 23:45-23:50
Due to failed UPS, the scheduled Kerr(B043) Wellman(B042)Electric Utility Shutdown brought down the network equipment on the site, affecting the network at and around Kerr Hall (Area6). Technicians brought up generator power to the equipment.
- Notification modified: 2009-10-03 00:38:34
Campus network issues Mon 9/28 22:40 - 23:10
Data Center system administrators resolved issues with DNS server 169.237.250.250 that had degraded network responsiveness both to and from campus. Everything is back to normal.
- Notification modified: 2009-09-28 23:22:37
Campus Network Problems: Broadcast Storm 09/24/2009 04:25-06:00
There was a localized broadcast storm that was caught and fixed. All services were functioning normally but were unavailable.
- Notification modified: 2009-09-24 06:24:15
UCDHS Datacenter Downtime Network Maintenance
=================== Network Maintenance =====================
Description: UCDHS Datacenter Downtime
Start of Window : 09/26/2009 19:00
End of Window : 09/27/2009 05:00
(all times are in 24 hour format)
Purpose:
The UCDHS Datacenter fire suppression equipment was upgraded to a waterless gas system. In order to bring the new system online and test it in accordance with fire codes, electricity to all equipment in the UCDHS datacenter had to be shut down for an extended period of time.
Impact:
The UCDHS datacenter houses equipment for a number of services owned by the main UCD campus. The impact to each was as follows:
External DNS:
UCD campus houses one of its three external DNS servers at the UCDMC Data Center. That server was unavailable during this downtime. DNS lookups from off campus were slow until services were restored. Clients saw delays while initially connecting to on-campus
services from off campus.
Voice Network:
The UCDMC2 trunk group was removed from service on Friday, September 25, 2009 at 3PM. Restoration of services to the trunks is scheduled for Monday, September 28 at 8AM. The removal of these trunks did not impact the UCD campus’ Med Center or local Sacramento bound voice traffic.
Backup routes were utilized.
Data Network:
The UCD campus' border router at the Med Center was powered off for the duration of the window. All UCD campus traffic to/from UCDHS and to/from the internet were re-routed around the powered off device, but there was no redundant path to UCDHS or the Internet for the duration.
800MHz Radio:
The primary path for 800MHz traffic was not affected by this downtime. Backup paths were unavailable for the duration, but no noticeable impact to the service was expected.
Affected:
Campus services as outlined above.
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.eucd
- Notification modified: 2009-09-27 03:11:07
2112 Watershed node room downtime, 09/16/2009 06:00 - 07:15
This downtime was the final planned interruption required to bring the server room fully online. Power to all systems were interrupted as the EPO system final configuration was completed.
- Notification modified: 2009-09-16 07:24:53
Network Maintenance: Data Center 09/16/2009 06:00 - 07:00
Description:
The Campus Data Center will experience brief network interruptions with several network upgrades planned for the early morning hours during the week of 9/14/09.
Outage Summary:
All network interruptions will occur during the hours of 06:00-07:00.
Wednesday 9/16/2009 06:00-07:00:
Data Center firewall and layer 2 switch firmware upgrade. Several long outages for all network connectivity to all hosts in the Data Center.
- Notification modified: 2009-09-16 07:18:34
Network Maintenance: Router and Switch 09/16/2009 05:00 - 06:00
=================== Network Maintenance =====================
Description: Area 3 BDF 7 Router and Switch Software Maintenance
Start of Window : 09/16/2009 05:00
End of Window : 09/16/2009 06:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BigIron BDF and FES IDF switches in Area 3 BDF 7 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load new code versions.
Affected:
BUILDINGS:
TB 200
TB 201
TB 202
TB 203
- Notification modified: 2009-09-16 07:00:45
Network Maintenance: Router and Switch 09/16/2009 05:00 - 06:00
=================== Network Maintenance =====================
Description: Area 11 Router and Switch Software Maintenance
Start of Window : 09/16/2009 05:00
End of Window : 09/16/2009 06:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BigIron BDF and FES IDF switches in Area 11 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load new code versions.
Affected:
BUILDINGS:
CTR NEUROSCI
DV 1450 DREW AVE
DV 1460 DREW
DV 1480 DREW
DV 1560 DREW
DV 1616 DA VINCI CT
DV 1632 DA VINCI CT
DV 1633 DA VINCI CT
DV 1850 RESEARCH PARK DR
NEUROSCI BLD
UNIV EXT BLD
UNIV SVCS BL
- Notification modified: 2009-09-16 06:59:32
Network Maintenance: Router and Switch 09/16/2009 06:00 - 06:00
=================== Network Maintenance =====================
Description: Area 42 Router and Switch Software Maintenance
Start of Window : 09/16/2009 06:00
End of Window : 09/16/2009 06:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BigIron BDF and FES IDF switches in Area 42 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load new code versions.
Affected:
BUILDINGS:
DV 105 E ST
DV 116 B ST
DV 1554 DREW
DV 1712 PICASSO STREET
DV 200 B ST
DV 2795 2ND STREET
DV 328 D STREET
DV 501 OAK AVE
DV 607 PENA DR
DV REGNCY SQ
SAC 1 CAP ML
SOCIAL SCIENCE & HARC
TELCOM
- Notification modified: 2009-09-16 06:48:44
Network Maintenance: Electrical CNPRC CEF 09/16/2009 06:00-08:00
=================== Network Maintenance =====================
Description: Electrical Shutdown CNPRC CEF
Start of Window : 09/16/2009 06:00
End of Window : 09/16/2009 08:00
(all times are in 24 hour format)
Purpose:
Electrical shutdown to bring the new VIL building online.
Impact:
Network connectivity loss at Primate Med Office after approximately 30 minutes. The other buildings will be on backup power and generator and are not expected to have any loss of network connectivity.
Affected:
BUILDINGS:
Primate Center CEF
TSRF
Primate Med Office
McCarthy Construction Trailer
- Notification modified: 2009-09-16 08:28:57
Network Maintenance: Campus Border Router 09/15/2009 06:00-07:00
=================== Network Maintenance =====================
Description: Campus Border Router
Start of Window : 09/15/2009 06:00
End of Window : 09/15/2009 07:00
(all times are in 24 hour format)
Purpose:
Establish a BGP peering session between the Campus Border Router and CENIC Sunnyvale High Performance Research (HPR) router (SVL-HPR2).
Impact:
None expected. A new BGP peering session will be established, but Campus will continue to prefer the CENIC Oakland HPR router (OAK-HPR) to reach CENIC HPR,
Internet2 and NLR sites.
Affected:
128.120.0.0/16
169.237.0.0/16
- Notification modified: 2009-09-15 07:04:30
Network Maintenance: Campus Wireless 09/15/2009 05:00 - 06:55
=================== Network Maintenance =====================
Description: Campus Wireless
Start of Window : 09/15/2009 05:00
End of Window : 09/15/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrade firmware to latest vendor recommended version
Impact:
Brief wireless outages as controller and individual access points are rebooted to new code version. Users will need to re-authenticate after reboot.
Affected:
All campus managed wireless
- Notification modified: 2009-09-15 06:57:34
Network Maintenance: Area 13(Data Center) 09/15/2009 06:00-06:30
=================== Network Maintenance =====================
Description: Area 13 (Data Center) Router and Switch Software Maintenance
Start of Window : 09/15/2009 06:00
End of Window : 09/15/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BigIron BDF and FES IDF switches in Area 13 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load new code versions.
Affected:
BUILDINGS:
DATA CENTER
- Notification modified: 2009-09-15 06:37:12
Production Xeda environment patch: 09/11/2009 21:00 - 00:00
The Production Xeda environment was patched during the normal patch window: Friday September 11th from 9PM-Midnight. These patches required a reboot, so some Xeda users may notice a brief Exchange disconnect during this maintenance window.
- Notification modified: 2009-09-12 00:36:59
Mothra DB Upgrade 09/11/2009 21:00 - 09/12/2009 00:15
The mothra DB (on Brownshoe and Tenspeed) underwent an Oracle 10.2.0.4 upgrade.
- Notification modified: 2009-09-12 00:29:42
Moobilentx and VPN Service Authentication: 09/10/09 08:56-09:35
Impact:
The radius server used to authenticate users on moobilnetx and VPN services was not properly responding to authentication requests. The issue has been resolved.
- Notification modified: 2009-09-10 10:45:41
Network Maintenance: Resnet Router/Switch 09/10/2009 05:00-06:00
=================== Network Maintenance =====================
Description: Resnet Router and Switch Software Maintenance - PART 2
Start of Window : 09/10/2009 05:00
End of Window : 09/10/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BigIron BDF and FES IDF switches in Resnet to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load new code versions
Affected:
The following locations will be affected:
Castillian
Emerson
La Rue
Leach
Orchard Park
Oxford Commons
Primero
Regan
Segundo
Solano
Tercero
Thoreau
Webster
- Notification modified: 2009-09-10 06:07:05
Network Maintenance: Shutdown CNPRC/CEF 09/10/2009 06:00-08:00
=================== Network Maintenance =====================
Description: Electrical Shutdown CNPRC CEF
Start of Window : 09/10/2009 06:00
End of Window : 09/10/2009 08:00
(all times are in 24 hour format)
Purpose:
Electrical shutdown to bring the new VIL building online.
Impact:
Network connectivity loss at Primate Med Office after approximately 30 minutes. The other buildings will be on Power Supply and Generator and are not expected to have any loss of network connectivity.
Affected:
BUILDINGS:
Primate Center CEF
TSRF
Primate Med Office
McCarthy Construction Trailer
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-09-10 08:41:31
Network Maintenance: ADF/Core Router 09/10/2009 04:00 - 07:00
=================== Network Maintenance =====================
Description: Area 1,2,4,6,7 ADF and Core Router Software Upgrade
Start of Window : 09/10/2009 04:00
End of Window : 09/10/2009 07:00
(all times are in 24 hour format)
Purpose:
Install vendor recommended software on the Core routers and ADF routers in areas 1,2,4,6, and 7.
Impact:
Multiple brief outages throughout the course of the maintenance window as the new version of software is applied.
Affected:
VLANS Serviced by Core Routers
------------------------------
Any VLAN that is attempting to communicate across network areas or to the Internet.
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-09-10 08:41:37
Network Maintenance: Resnet Router/Switch 09/09/2009 05:00-07:00
=================== Network Maintenance =====================
Description: Resnet Router and Switch Software Maintenance - PART 1
Start of Window : 09/09/2009 05:00
End of Window : 09/09/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BigIron BDF and FES IDF switches in Resnet to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load new code versions
- Notification modified: 2009-09-09 11:32:18
Area 13 and 42 ADF Router Software Upgrade:9/14/09 06:00-07:00
=================== Network Maintenance =====================
Description: Area 13 (The Campus Data Center) and 42 ADF Router Software Upgrade
Start of Window : 09/14/2009 06:00
End of Window : 09/14/2009 07:00
(all times are in 24 hour format)
Purpose:
Install vendor recommended software on the Core routers and
ADF routers in areas 13 and 42
Impact:
Multiple brief outages throughout the course of the
maintenance window as the new version of software is
applied.
Area 13 maintenance will affect data network connections to The Campus Data Center.
- Notification modified: 2009-09-14 08:15:47
Network Maintenance: UCDMC Border Testing 09/03/2009 03:00-06:00
=================== Network Maintenance =====================
Description: UCDMC Border Network Failover Testing
Start of Window : 09/03/2009 03:00
End of Window : 09/03/2009 06:00
(all times are in 24 hour format)
Purpose:
Perform comprehensive failover testing of redundant paths between UCDMC and external sites.
Impact:
Loss of connectivity may be experienced if an issue is discovered during the testing period. In the event this should occur, connections between UCDMC and external sites will experience multiple interruptions in service while work is performed to resolve the issue.
Affected:
UC Davis Medical Center
- Notification modified: 2009-09-03 05:58:35
Campus Border Network Failover Testing: 09/02/2009 23:00 - 03:00
Start of Window : 09/02/2009 23:00
End of Window : 09/03/2009 03:00
(all times are in 24 hour format)
Purpose:
Apply configuration change to address known issues with SNMP and PIM-SM RP. Perform comprehensive failover testing of redundant paths between Campus and external sites.
Impact:
Loss of connectivity may be experienced if an issue is discovered during the testing period. In the event this should occur, connections between Campus and external sites will experience multiple interruptions in service while work is performed to resolve the issue.
Affected:
Campus Core - Areas 1,2,3,4,5,6,7,8,9,10,11,42
Residential Network (ResNet) - Area 12
Campus Data Center - Area 13
Research Network - Area 51
- Notification modified: 2009-09-03 03:22:18
DavisMail/Gmail : Tuesday, September 1, 2009 ~1250 hrs - 14:37
On Tuesday, September 1st, 2009 web access to DavisMail, powered by GMail, was unresponsive, though POP3 and IMAP
connections were working. This was related to a general
outage affecting web access for all GMail users.
- Notification modified: 2009-09-01 14:46:15
Network Maintenance: Area 6 Router/Switch 09/01/2009 05:00-07:00
=================== Network Maintenance =====================
Description: Area 6 Router and Switch Software Maintenance
Start of Window : 09/01/2009 05:00
End of Window : 09/01/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BDF and IDF switches in Area 6 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load the new version of software.
Affected:
The following locations will be affected:
ASMDSN
BRIGGS
COWELL
CREUSS
FOODSCI
HARING
HART
HOAGLAND
HOUSING
HUTCH
KERR
MANN
ROBBINS
SCILAB
SOLVENT STOR
STORER
TB114
TB188
WALKER
______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-09-01 06:24:14
Course Management Unavailable August 30, 2009 05:20 - 06:50
Sunday, August 30, 2009 05:20 - 06:50
Course Management was unavailable due to a problem with its database. The database and systems administrators restored service and are continuing to investigate the outage.
- Notification modified: 2009-08-30 07:32:25
Adobe Connect Server, 8/29/2009 15:30-16:45
The Adobe Connect server became unresponsive and was serving only blank pages without error messages. System administrators determined that a reboot of the host was necessary and the service was restored.
- Notification modified: 2009-08-29 16:45:32
Network Maintenance: Area 4 Router/Switch 08/28/2009 05:00-06:00
=================== Network Maintenance =====================
Description: Area 4 Router and Switch Software Maintenance
Start of Window : 08/28/2009 05:00
End of Window : 08/28/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BDF and IDF switches in Area 4 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load the new version of software.
Affected:
The following locations will be affected:
MEDSCI1B
MEDSCI1C
VET MED 2
CCAH
GBSF
WHNRC
VMIF
VET MED 3A
AGGIE STADIUM
VET MED MULTI PURPOSE
EQUINE PERFORMANCE LAB
SCHALM
- Notification modified: 2009-08-28 06:01:54
Network Maintenance: Area 3 Router/Switch 08/27/2009 05:00-07:00
=================== Network Maintenance =====================
Description: Area 3 Router and Switch Software Maintenance
Start of Window : 08/27/2009 05:00
End of Window : 08/27/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BDF and IDF switches in Area 3 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load the new version of software.
Affected:
Buildings affected:
TELCOM
A&E
PARKING SVCS
SILO
SURGE 2
SURGE 3
TB 201
POLICE&FIRE
TAPS
TB 191
GIEDT HALL
LIFE SCI ADN
DV 3820 CHILES
CR TRLR
PLANT REPRO
GENOME LAUNCH
BOOKSTORE WHSE
REPRO PSL
TB 172
TB 169
TB 192
TB 171
WEST ENTRY PKNG
TB 202
TB 200
TB 203
TB 24
CAMPUS CHILD CARE CENTER
REC LODGE
BOWLEY
TB 182
TB 181
PARKING TRLR
TERCERO TRLR D
BUELER ALUMNI & VISITORS CENTER
MECH 2
TERCERO TRLR A
IT NET21 C
ASTF 2
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-08-27 07:02:57
Network Maintenance: Area 2 Router/Switch 08/26/2009 05:00-07:00
=================== Network Maintenance =====================
Description: Area 2 Router and Switch Software Maintenance
Start of Window : 08/26/2009 05:00
End of Window : 08/26/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BDF and IDF switches in Area 2 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load the new version of software.
Affected:
Buildings Affected:
EU3
CHEM
EU2
ACAD SURGE
BAINER
CENTRAL GARAGE
CHEM ANNEX
EVERSON
PHYSICAL SCIENCES LIBRARY
PHYS GEO
TB30
MATH SCIENCES
- Notification modified: 2009-08-26 06:25:36
Network Maintenance: Academic Surge 08/26/2009 05:00 - 08:00
=================== Network Maintenance =====================
Description: Network Upgrade - Academic Surge
Start of Window : 08/26/2009 05:00
End of Window : 08/26/2009 08:00
(all times are in 24 hour format)
Purpose:
Upgrade the BDF equipment in Academic Surge
Impact:
There will be a loss of connectivity as the switches are migrated to the new BDF equipment. During this time, internal VLAN and external network resources may be unavailable.
Affected:
The following locations will be affected:
ACADEMIC SURGE
WATERSHED SCIENCE RES CTR
TEMP CLASSROOM 2
- Notification modified: 2009-08-26 07:48:37
Network Maintenance: ADF Router Upgrade 08/25/2009 04:00 - 06:00
=================== Network Maintenance =====================
Description: Area 5,8,9,10,11 ADF Router Software Upgrade
Start of Window : 08/25/2009 04:00
End of Window : 08/25/2009 07:00
(all times are in 24 hour format)
Purpose:
Install vendor recommended software on the ADF routers in Areas 5, 8, 9, 10 and 11.
Impact:
Multiple brief outages throughout the course of the maintenance window as the new version of software is applied.
- Notification modified: 2009-08-25 06:02:43
Network Maintenance: Area 1 Router/Switch 08/25/2009 05:00-07:00
=================== Network Maintenance =====================
Description: Area 1 Router and Switch Software Maintenance
Start of Window : 08/25/2009 05:00
End of Window : 08/25/2009 07:00
(all times are in 24 hour format)
Purpose:
Upgrading the software on all BDF and IDF switches in Area 1 to the vendor recommended versions.
Impact:
Multiple brief outages as the devices are reset to load the new version of software.
Affected:
Buildings affected:
WRIGHT HALL
KING HALL
PHYS PLANT
SHIELDS LIBRARY
SPROUL HALL
MONDAVI CTR
OLSON
ART
ENV HORTICULTURE
MUSIC
- Notification modified: 2009-08-25 06:25:39
Network Outage: UPDATE: Infoblox DNS Issues : 08/24/2009 11:38
Impact:
CNAME records in zones other than ucdavis.edu, pointing
back to hosts in ucdavis.edu did not import correctly into
the Infoblox.
Locations Affected:
CNAME records in zones other than ucdavis.edu, which point
back to ucdavis.edu hosts. For instance,records of the
form: anything.somezone.org IN CNAME anything.ucdavis.edu
This impacted approximately 80 DNS records.
Status: At this time, the missing entries have been added to the infoblox. If issues persist, please notify the IT Helpdesk at 754-HELP.
- Notification modified: 2009-08-24 12:02:14
Campus DHCP System Migration, 8/21/2009 17:00-19:00
Purpose:
The Campus DHCP system, NetID, is being replaced with the new Infoblox system.
Impact:
The DHCP helper addresses on all Campus and Resnet router interfaces will be updated to point to the new Infoblox system. Some hosts may require a DHCP release and renew (or reboot) to be performed in order to successfully gain an IP address lease from the new system.
Affected:
All Campus and Resnet hosts currently using the Campus DHCP system.
- Notification modified: 2009-08-22 02:58:29
Network Maintenance: NOC Generator Test: 08/21/2009 05:45-07:00
Description: NOC Generator Test
Start of Window : 08/21/2009 05:45
End of Window : 08/21/2009 07:00
(all times are in 24 hour format)
Purpose:
Performing monthly test of the Telecom Building backup power generator.
Impact:
None expected. All power should automatically switch over to generator power for the duration of the test.
Affected:
All Campus and Resnet networks; all connectivity to Internet and I2; all T1 connected sites would be affected should an unanticipated outage occur.
- Notification modified: 2009-08-21 08:19:57
DaFIS Prod Scheduled Maintenance on 09/13/2009 8 am - 1 pm
As part of the ongoing network restructuring at the Data Center, the FIS servers that reside there are moving to a new subnet IP range.
We are moving the systems in batches - the next batch is scheduled to move on Sunday September 13th.
The production servers for DaFIS TP, DS and GL are included in this batch and will be unavailable during the move. Sunday is our normal maintenance day
for DaFIS production systems. We expect the downtime to be 8 am - 1 pm.
- Notification modified: 2009-09-13 13:27:08
LDAP Service Degredation: 08/18/2009 10:25 - 12:25
On Tuesday, August 18th, 2009 the campus LDAP service experienced two
failures, from 10:25am to 11:05am and from 11:50am to 12:25pm PDT. The
outage resulted in the loss of several central campus services
including, but not limited to, authentication to the campus wireless
network, creating new computing accounts, and authenticated SMTP
service. Data Center Administrators are confident that they have
isolated the problem and there should be no more interruptions to the
campus LDAP service.
- Notification modified: 2009-08-18 13:50:00
LDAP Service 08/18/2009 10:24 - 11:00
The LDAP service was unreachable.
LDAP was restarted.
Affected:
Wireless Authentication
(Moobilenet and MoobilenetX)
Library VPN
Any service that uses LDAP directory services
- Notification modified: 2009-08-18 11:09:37
Network Maintenance: Area 3 ADF Router Software Upgrade
=================== Network Maintenance =====================
Description: Area 3 ADF Router Software Upgrade
Start of Window : 08/18/2009 06:00
End of Window : 08/18/2009 07:00
(all times are in 24 hour format)
Purpose:
Install vendor recommended software on the ADF routers in Area 3.
Impact:
Multiple brief outages throughout the course of the maintenance window as the new version of software is applied.
Affected:
VLANs Serviced by Area 3
------------------------
- Notification modified: 2009-08-18 08:06:07
Network Maintenance 08/19/2009; 0630 - 0700 hrs
Description: Change card with errors in CDF-1
A card in one of the core routers is generating errors and
needs to be replaced per vendor.
Impact:
The network core is redundant so observed outages should
be minimal. Areas 9, 10, 11, and 13 will be affected.
Area 13 includes
Campus Data Center
In regard to the Campus Data Center network:
The NOC has scheduled a card replacement on a router that affects the Data Center. The existing card is exhibiting errors, and the vendor recommends replacing it. All our networking at that level is redundant, and we should see nothing more than a brief interruption lasting only a few seconds, while the link fails over.
- Notification modified: 2009-08-19 06:54:29
Primate Center Network Upgrade Aug 13, 17:00 - 19:00
Description: Network Upgrade - Primate Center
Start of Window : 08/13/2009 17:00
End of Window : 08/13/2009 19:00
(all times are in 24 hour format)
Purpose:
Upgrade the BDF equipment at the Primate Center
Impact:
There was a loss of connectivity as the switches were
migrated to the new BDF equipment. During this time,
internal VLAN and external network resources may have been unavailable.
Affected:
The following buildings were affected:
PRIMATE SECURITY KIOSK 1.1
PRIMATE CENTRAL SUPPLY 1.1
PRIMATE CENTER ANIMAL WING 1.1
PRIMATE CENTER RESEARCH OFFICE 1.1
PRIMATE COLONY OFFICE 1.1
TB-184 1.2
SCEV 1.1
Users and services on the following networks were affected:
IT-CRETV-3
NEUROPHY-BEH-3
O&M:UTILITIES-1
PRIM-RES-CTR-1-FW
PRIM-RES_CTR-2-FW
PRIM-RES-CTR-3-FW
WIRELESS
- Notification modified: 2009-08-13 18:46:12
- Notification modified: 2009-08-12 06:52:07
Network Maintenance Wed, Aug 12, 0600 - 0700
=================== Network Maintenance =====================
Description: Continue Area 5 Troubleshooting
Start of Window : 08/12/2009 06:00
End of Window : 08/12/2009 07:00
(all times are in 24 hour format)
Purpose:
Continuing to troubleshoot likely packet loss in Area 5 (in and around the ARC).
Impact:
Initial troubleshooting steps will only affect resources in Area 5 for several minutes at a time.
If packet loss is happening in the core, a card replacement may need to be done that affects Areas 5, 6, 7, and 8 for approximately 5 minutes.
Affected:
The following Areas/Buildings will be affected. In some cases, buildings not listed but nearby may also be affected.
Area 5 Includes
ARC
Human Resources Admin Building
Area 6 Includes
Kerr
Hart
Hoagland
Housing
Robbins
Storer
Walker
Asmundson
Briggs
Cruess
Food Science
Hutchison
Haring
TB114
Science Lab
Briggs
Area 7 Includes
AOB4
Freeborn
Hickey Gym
Hunt
MU
North
Socials Sciences and Humanities
South
Veihmeyer
Voorhies
Wickson
Young
Dutton
PES
Area 8 Includes
Airport Area
Primate Center
CCM
HSC Receiving/Mail
- Notification modified: 2009-08-12 06:52:21
Courseware Maintenance 08/07/09 2200 - 2230
The Courseware service was unavailable on Friday August 7th from 22:00 - 22:30 due to maintenance.
- Notification modified: 2009-08-07 21:03:19
Authentication Issues 08/07/09 08:20 - 10:00
Around 8:20 we begun to receive reports from users that they were unable to connect to Banner.
At approximately 8:55am on August 7th, 2009 the Central Authentication Service and DistAuth stopped accepting new connections. CAS and DistAuth sessions opened prior to this time remained valid, however new login attempts were timing out.
Network issues related to this mornings scheduled switch maintenance were the indirect cause of the experienced authentication issues. All networking issues have been resolved. Authentication services have been restored.
- Notification modified: 2009-08-07 10:21:43
Emergency Data Center Network Downtime Friday Aug 7 06:00-07:00
The campus Data Center experienced emergency network downtime,
Friday, Aug 7, 2009 from 0600 hrs to 0700 hrs.
The downtime was necessary to troubleshoot, and possibly replace, a failing blade on the master aggregate network switch.
ALL DATA CENTER SERVICES, except collocation, were affected and unavailable from 0605-0620 (15 minutes).
- Notification modified: 2009-08-07 07:31:36
IMAP email server degradation, 8/6/2009 08:50-08:53
The ms13 mail service was restarted for emergency maintenance at 8:50 AM. Systems administrators resolved an issue with the IMAP service.
- Notification modified: 2009-08-06 09:09:24
Campus Intrusion Prevention System (IPS) 08/06/09 05:00-05:50
Description: Campus Intrusion Prevention System (IPS)
Purpose:
Upgrade system software on the IPS sensors.
Apply system software patch on the IPS management servers.
Impact:
None
- Notification modified: 2009-08-06 05:54:59
Xbase & TSP.ucdavis.edu Unavailable 7/30/09 13:00 - 13:25
A server problem prevented access to the Xbase and TSP websites. System Administrators have resolved the issue.
- Notification modified: 2009-07-30 13:38:05
Gladys Valley Hall Wireless Replacement 7/24 8am - 7/28 5pm
Purpose:
All of the wireless access points in this building will be
replaced.
Impact:
Wireless networking will be unavailable in this building.
- Notification modified: 2009-07-26 07:04:06
MyTravel, MyEffort, EDMS and Discoverer Downtime 07/26/09
Downtime: July 26th, 2009 08:00 - 13:00
The MyTravel, MyEffort, EDMS and Discoverer, was moved to new IP addresses on July 26, 2009, from 08:00 to 13:00. This change allowed the services to reside in the Data Center new IP Range.
- Notification modified: 2009-07-26 14:21:00
Retirement of old mail.ucdavis.edu address July 29, 2009
The old mail.ucdavis.edu address, 128.120.32.8, was retired as of July 29, 10 a.m. For mail account settings, please contact IT Express at (530) 754-HELP (4357).
- Notification modified: 2009-07-29 17:35:49
Mailing lists maintenance Mon 7/20/09 2130 - 2225
System administrators performed maintenance on the mailing lists. The service is now functioning normally.
- Notification modified: 2009-07-20 22:34:10
Router and Switch Configuration Update: July 22 06:00 - 07:00
=================== Network Maintenance =====================
Description: Router and Switch Configuration Update
Start of Window : 07/22/2009 06:00
End of Window : 07/22/2009 07:00
(all times are in 24 hour format)
Purpose:
Updating the configurations on the router and switches servicing Mayer Hall and the AH Horse buildings.
Impact:
Brief outages as the switches and router are rebooted to activate the new configuration variables.
Affected:
All network connectivity in Meyer Hall and AH Horse.
_______________________________________________________________________
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2009-07-22 07:30:38
Campus Wide Network Maintenance 07/17/2009 05:45 - 07:00
=================== Network Maintenance =====================
Description: Telecom Building Generator Test
Start of Window : 07/17/2009 05:45
End of Window : 07/17/2009 07:00
(all times are in 24 hour format)
Purpose:
Performed monthly test of the Telecom Building backup power generator.
Impact:
None. All power automatically switched over to generator power for the duration of the test.
Affected:
All Campus and Resnet networks; all connectivity to Internet and I2; all T1 connected sites could have been affected if an unanticipated outage occured.
If you have questions or concerns regarding this event, please contact: UC Davis Network Operations Center 1.530.752.7656 noc@ucdavis.edu
- Notification modified: 2009-07-17 07:13:38
Smartsite Maintenance July 17 10pm - July 18 8am
Smartsite experienced downtime this weekend for normal maintenance starting Friday, July 17 at 10pm until Saturday, July 18, 2009 at 8am.
- Notification modified: 2009-07-18 09:46:18
Primate Center Voice Network Maintenance July 22, 23, 27, and 28
The voice network infrastructure at the Primate Facility will be undergoing maintenance. Buildings at the following locations may experience limited periods of degraded or intermittent service.
Date and Times for Affected Locations:
-July 22, 8:00AM - 12:00PM
Quarantine
Lift Station / Landfill
Security Kiosk
-July 23, 8:00AM - 12:00PM
TB 184
Animal 1
Animal 2
Animal 3
Animal 4
Primate Store
-July 27, 8:00AM - 12:00PM
CCM Animal Wing
Animal Housing 1
Animal housing 2
-July 28, 8:00AM - 12:00PM
TB 176
TB 177
TB 178
TB 178a
TB 196
BB Building
Childhood Health
South Colony Office
J30
J32
Primate Facilities Trailers
If you experience degraded or intermittent service past the scheduled hours of maintenance, please contact Julie Willis at 752-0438.
- Notification modified: 2009-07-29 06:31:04
Kerr Hall Network Maintenance July 22 07:00 - July 22 07:20
=================== Network Maintenance =====================
Description: Kerr Hall ADF6-1 Interface Module
Start of Window : 07/22/2009 07:00
End of Window : 07/22/2009 07:30
(all times are in 24 hour format)
Purpose:
Replace interface module that is generating non-standard
error messages.
Impact:
None.
- Notification modified: 2009-07-22 07:21:42
XEDA patching, Friday, 7/17/2009, 21:00-00:00
The XEDA test environment was successfully patched on Tuesday, July 14. Since no issues were encountered administrators patched the production environment during the normal patch window (Friday 9PM-Midnight). These patches did not require a reboot, however administrators performed some regular maintenance which required a reboot.
- Notification modified: 2009-07-18 03:13:22
Campus Central Email Server, ms2., July 16, 10:54 to 11am
IT Express received a small number of reports from email users on ms2.ucdavis.edu being unable to connect to their email accounts starting about 10:54am. Campus Data Center Administrators and Operators isolated the issue and the server was responding by 11am.
- Notification modified: 2009-07-16 11:13:02
Course Management hardware issue Wed Jul 15 1400- 1420
The AFS fileserver serving Course Management suffered a hardware reboot which would have taken offline some Course Management content from 14:00 to 14:20.
- Notification modified: 2009-07-15 20:32:23
Smartsite non-production maintenance Wed Jul 15 1200 - 1700
We performed scheduled maintenance on the *non-production* SmartSite servers on July 15, 2009 from 12:00 to 17:00. For more information, please see:
https://confluence.ucdavis.edu/confluence/x/Q4Kw
- Notification modified: 2009-07-15 20:30:40
Maintenance Data Center SAN Tuesday July 14th 7:00 AM - 8:00 PM
On Tuesday July 14th at 7:00 AM - 8:00 AM we performed maintenance on the new Data Center SAN. We installed new software and licenses to enable additional provisioning functionality. Unfortunately, this require a storage processor reboot. Hosts connected via redundant paths to the SAN were not affected as the connections failed over to the second storage processor.
- Notification modified: 2009-07-14 08:49:42
Mailing lists degredation, 15:30-16:00
The server hosting mailing lists was experiencing usually high loads. System administrators restarted the server and restored the service.
- Notification modified: 2009-07-10 16:06:46
CAS disruption - Thurs., July 9 1548hrs - 1555
The Campus Authentication system was unavailable for 14 minutes. System Administrators successfully restarted the service.
- Notification modified: 2009-07-09 16:18:30
Service degradation: CAS 15:50-15:56 7/9/2009
CAS authentication was unavailable from 15:50 until 15:56. System administrators restarted the service to resolve the issue.
- Notification modified: 2009-07-09 15:58:37
Backup Disruption 7/7/09 10:00 - 14:00
Data Center administrators performed maintenance on a backup storage device. No backups or restores were available between 10:00 and 14:00. All normally scheduled backups during that time period completed later in the day.
- Notification modified: 2009-07-08 18:49:28
KMS Service Upgrade and Move to New IP Address
On Wednesday, July 1, between 06:00 and 07:00, the campus Key Management System (KMS) was upgraded and moved to a new IP address. The KMS provides activation of volume licenses of Microsoft Vista and 2008 Server.
- Notification modified: 2009-07-02 02:50:28
SERVICE DEGRADATION: SMTP and Listproc, July 1 2009
Some campus mail system users had experienced intermittent slowness and error messages when sending mail throughout the day. System administrators determined the occasional slowness issue was most likely related to heavy load on some of the servers that comprise the smtp.ucdavis.edu service, which has since subsided.
In addition, a corrupted database had caused degradation in the listproc service.
- Notification modified: 2009-07-02 12:15:44
Confluence Upgrade - June 30, 2009 9pm-11pm
Confluence was upgraded from version 2.3 to 2.10.
- Notification modified: 2009-07-01 00:26:01
Smarsite-TEST downtime 06/30/09 1700
SmartSite-TEST was down June 30, 2009, after 5pm for the following changes:
* Programmer: Michael J. Wenk
* Jira:
* sak-2880 Critical bug fixes for gb2 (week of June 23rd)
Please see the following link for further information:
https://confluence.ucdavis.edu/confluence/display/UCDSAKAI/SmartSite+Test+-+Maintenance+Details+and+Communication
- Notification modified: 2009-07-01 00:29:19
Network Outage Primate Center Area 13:01
06/28/2009 15:01
UPDATE:
Impact:
IT-ENV Cntrl building suffered an electrical outage 06/28/09
morning damaging high voltage electrical service equipment,
full loss of power detected at approximately 1300 hours.
NOC technicians have connected switch gear to functioning
electrical circuits however clients experienced a few more outagest as Facilities electricians repair the failed
equipment.
Locations Affected:
Status:
Eletricians are working to remedy the situation
- Notification modified: 2009-06-29 12:07:10
MyInfoVault Maintenance, 08:00 - 10:00, 6/28/09
MyInfoVault was unavailable on Sunday, June 28, 2009 at 8 a.m. to 10 a.m. due to scheduled maintenance.
- Notification modified: 2009-06-29 11:57:19
Campus Network Maintenance; Wed., June 17, 2009 - 7AM - 10:55AM
Wednesday, June 17, 2009 - 07:00 - 10:55
Due to unforeseen issues, this morning's border router maintenance was extended beyond the originally announced 04:00-07:00 window. While the majority of campus network services continued to function normally, however, connectivity to UCDMC and the NOC were down for most of the period.
- Notification modified: 2009-06-17 14:47:39
Banner/SIS/Dafis/SISdocs Emergency Repair 11:30-12:15 6/12/2009
One of the servers managing IET-maintained Oracle databases experienced a fatal system error yesterday at 10:50 and this morning at 04:00, and had to be manually rebooted. Users experienced connection and time-out issues when using IET-maintained databases and dependent services including Banner, SISWeb, SISDocs, and DaFIS. Administrators resolved the issue temporarily. A hardware fault had been identified. An emergency outage time starting at 11:30 to 12:15 was required to repair the fault.
- Notification modified: 2009-06-12 13:15:59
SIS/Banner/Dafis/SISdocs Connection 06/12/2009 04:00 - 04:37
One of the servers managing IET-maintained Oracle databases experienced a fatal system error, and had to be manually rebooted. Users may have experienced connections and time out issues when connecting to IET-maintained databases and dependent services including Banner, SISWeb, SISDocs, and DaFIS. Administrators have resolved the issue.
- Notification modified: 2009-06-12 04:47:16
SIS/Banner/Dafis/SISdocs Degredation 06/11/2009 10:50 - 11:45
One of the servers managing IET-maintained Oracle databases was experiencing a system error. Users may have experienced slow connections and time out issues when connecting to IET-maintained databases and dependent services including Banner, SISWeb, SISDocs, and DaFIS.
Administrators have resolved the issue.
- Notification modified: 2009-06-11 11:58:13
Banner/Dafis Connections Issues 06/10/09 09:15
One of the servers managing IET maintained Oracle databases experienced a memory issue. Users experienced slow connections and
time out issues when connecting to IET maintained databases and
dependent services including Banner, SISWeb, and DaFIS. System Administrators found the problem and restored service at 09:35.
- Notification modified: 2009-06-10 10:06:06
Med Center network connectivity. 06/04/2009 03:44 - 0705
UCDMC has lost network connectivity at 03:44 and regained connectivity at 07:05. This had no impact on UCD's services.
- Notification modified: 2009-06-04 08:15:50
Exchange Service Outage - Wed, June 3, 2009; 2055hrs - 2125hrs
The Exchange Service was interrupted for approximately 30 minutes. System administrators determined the cause and corrected it.
- Notification modified: 2009-06-03 21:56:57
Banner connection issues 06/02/2009 14:55 - 21:00
Users reported problems connecting to Banner during the day. System administrators brought Banner down for maintenance from 19:00 till 21:00. Everything is back up and operational.
- Notification modified: 2009-06-02 21:03:43
XEDA Mail degraded service. 06/02/09 06:54 - 07:18
XEDA Mail experience a small problem that affected a few clients. Problem was resolved at 07:18.
- Notification modified: 2009-06-02 07:34:08
Banner Database were Down 06/01/09 10:18 - 12:08
The Banner Databases were down. System Administrators fixed the problem and brough databases back online at 12:08.
- Notification modified: 2009-06-01 12:17:19
XEDA Mail was unavailable. 06/01/09 08:25 - 08:50
XEDA Mailbox2 service was unavailable. System Administrators fixed the problem at 08:50.
- Notification modified: 2009-06-01 08:56:40
SmartSite unavailable, Course Management degraded 15:30-17:40
SmartSite was unavailable and Course Management was degraded as both were experiencing heavy load. System Administrators have fixed the issue.
- Notification modified: 2009-05-23 18:29:13
Exchange Mail / XEDA unavailable 14:10-14:25
Several departments reported they were unable to connect the Campus Exchange Service (XEDA). System Administrators traced the issue to queued log files filling a storage array. Service was restored after the log were moved.
- Notification modified: 2009-05-21 14:56:21
MyInfoVault Maintenance, 17:00 - 17:45, 5/22/09
MyInfoVault System maintenance was scheduled for Friday, May 22, 2009 at 5pm. No service disruptions were encountered.
- Notification modified: 2009-05-23 15:48:56
SmartSite - Wednesday, May 20, 2009 18:45 – 19:30
One of the SmartSite servers experienced an application problem and was temporarily unavailable. Users connected to that system may have experienced degraded service, such as a disruption of their session. Users connected to any of the several other servers in the pool would not have been affected. System administrators resolved the situation and returned the server to the pool.
- Notification modified: 2009-05-20 20:25:14
Monitoring Service Degredation: 05/16/09 01:15 - 07:10
Data Center Administrators noticed some issues with false reports from our monitoring systems. Information on the status page was not correctly being displayed. All services except for Smartsite were currently functional. Smartsite was undergoing scheduled maintenance until 08:00.
- Notification modified: 2009-05-16 07:11:50
CAS and DistAuth - 05/14/09 07:25 - 22:00
From 7:25am until 8:42am on May 14th, 2009 the campus KDC servers were not providing tickets to the CAS and DistAuth login attempts resulting in users being unable to access secure resources on campus. Data Center Administrators corrected a number of issues and the service was restored. New login attempts are being successfully processed by the KDC and users can access secure sites.
UPDATE 09:30 - Data Center Administrators have traced the source of the failure to the campus KDC servers. After making adjustments on the KDC servers administrator felt the problem had been resolved as of 8:42am. At 9:24am the KDC servers once again started failing to provided tickets to CAS and DistAuth requests. Administrators are continuing to work to resolve this issue.
UPDATE 11:08 - Administrators continue to research possible solutions and will try to provide as much up time to the KDCs while still implementing fixes to the service.
UPDATE 16:10 - At 22:00 tonight, Data Center system administrators will reboot several of the authentication systems including the AFS server that handles the read/write volume used by DistAuth. All authentication services (CAS and DistAuth) will be unavailable. We estimate services will be restored by 22:30.
UPDATE 05/15/09 08:25 - The maintenance done at 22:00 on 5/14 has stabilized the authentication systems.
AFFECTED SERVICES: Any service that uses DistAuth or CAS to authenticate will be affected by this outage. A partial list of popular services include: moobilenet, moobilenetx, DaFIS, Banner/SISWeb, MyUCDavis, SmartSite, Geckomail, DavisMail, Library VPN
- Notification modified: 2009-05-15 08:27:09
UCDavis Software Website: 5/13/09 09:20 - 10:40
Application Administrators have resolved an issue with the Software Licensing Group's website within MyUCDavis, http://software.ucdavis.edu and https://my.ucdavis.edu/software. The websites were unavailable between 09:20 and 10:40.
- Notification modified: 2009-05-13 10:43:59
ListProc Degredation: 05/11/09 09:00 - 10:00AM
At approximately 09:00AM - 10:00AM ListProc stopped passing out messages. System Administrators restarted listproc services and everything was back to normal.
- Notification modified: 2009-05-11 11:16:00
ListProc Degredation: 05/08/09 11:00-12:15
At approximately 11:00AM ListProc stopped passing out messages. No mail was lost. The process that was problematic was restarted at 12:15PM. All mail was queued and delivery will catch up on the backlog shortly.
- Notification modified: 2009-05-08 12:32:12
Listproc is Down 5/5/09 12:06 - 12:36
Listproc was down.
- Notification modified: 2009-05-05 12:39:40
SISWEB 5/4/09 11:04 - 11:40
A high volume of requests for admissions information caused some Data Center services to reach session limits. Administrators addressed the session limit issue at approximately 11:40am, but http://sisweb.ucdavis.edu and http://myadmissions.ucdavis.edu continued to be slow due to high admissions load until 1:09.
- Notification modified: 2009-05-04 13:10:09
Authentication maintenance, Sat May 2nd, 10:00 PM
The campus Kerberos authentication servers were rebooted at 10:00pm tonight. No service disruption was experienced, as the servers are redundant in providing authentication services.
- Notification modified: 2009-05-02 22:57:03
WWW.UCDAVIS.EDU 05/01/09 11:20-11:50
www.ucdavis.edu experienced a high load on Friday May 1st. Users may have experienced intermittent slowness during the half hour the site was affected. The issue has been resolved by system administrators.
- Notification modified: 2009-05-01 12:07:37
Firmware on SAN Switches May 2 and May 3, 2009
Data Center Staff performed maintenance on the Storage Area Network (SAN) switches this weekend. As all services are dual connected to the SAN via separate fiber switches, and automatic failover has been tested successfully in the past. No outages on product services were experienced.
The firmware on the SAN switches for ms11, ms12, and ms13 were upgraded on Saturday, May 2, from 0700 till 1500 hours.
The firmware on the SAN switches for mothra, MyUCDavis, and the Course Management System were upgraded on Sunday, May 3, from 1000 till 1800 hours.
- Notification modified: 2009-05-04 09:55:10
SmartSite Maintenance 4-28-09 from 5:30 to 7:00
We performed scheduled maintenance on the SmartSite servers on April 28, 2009 from 05:30 to 07:00.
- Notification modified: 2009-04-29 14:56:46
Authentication Connections: 4/22/09 08:45 - 10:35
The campus had been experiencing slow authentication to all central campus email servers. Campus Data Center system administrators restarted the LDAP service on all ldap servers.
- Notification modified: 2009-04-22 17:36:50
Data center network issues - Wed 4/22/09 0115-0116
There was a brief network outage due to maintenance that affected access to Data Center services. Sys admins have examined systems and everything is operating normally at this time.
- Notification modified: 2009-04-22 01:26:13
SmartSite Downtime 04-21-2009 5am to 5:45am
SmartSite-PROD experienced downtime to complete an upgrade that began this past Saturday during Patch Weekend. The change included completing the creation of one materialized view (Banner Meeting) which includes new programming to include sak-2661, which provides additional section-level meeting time information to Smartsite.
- Notification modified: 2009-04-21 06:29:42
Off-campus DNS issues Mon 4/20/09 2230 - 4/21/09 9:35
Users making off-campus DNS queries to campus experienced slow response.
- Notification modified: 2009-04-21 11:26:34
CAS outage Friday, April 17, 2200 - 2230
On Friday, April 17 just after 10PM, The Campus Authentication System was briefly unavailable during a scheduled patching event. The Campus Data Center System Administrators worked quickly to resolve this problem and service was restored just after 10:30PM.
- Notification modified: 2009-04-17 23:13:01
financialaid.ucdavis.edu potential problem
http://financialaid.ucdavis.edu was reported as unavailable. The issue has been resolved.
- Notification modified: 2009-04-10 13:27:59
MyUCDavis myspace Potential Problem . 04/05/09 13:45 - 14:28
MyUCDavis and myspace was unavailable. apache restarted on all my's and everything cleared up.
- Notification modified: 2009-04-05 14:34:42
Moobilentx and VPN Service Authentication
04/02/09 12:15 - 04/03/09 10:00
Impact:
The radius server used to authenticate users on moobilnetx and VPN services was not properly responding to authentication requests.
Status:
The Data Center has reconfigured the LDAP servers load balancing pool to better handle the authentication requests. The problem has been resolved.
- Notification modified: 2009-04-03 10:28:13
CAS down from 10:28 - 10:48 03/30/09
CAS stoped responding to requests at 10:28 am we restarted services and CAS was running again at 10:48am
- Notification modified: 2009-03-30 10:57:39
LDAP Down 3/29/09 13:13 - 15:06
LDAP was Down from 13:13 til 15:06.
- Notification modified: 2009-03-29 15:54:59
Listproc Down for database rebuild 3/25/09 10:54 - 11:07
Listproc was down for revdb rebuild
- Notification modified: 2009-03-25 12:03:39
Wireless Maintenance 03/25/09 - 000:00 - 06:00
Purpose:
Upgraded Campus Wireless infrastructure to vendor recommended software version
Impact:
Access Points will reboot after wireless controllers are upgraded and reconfigured. Expect multiple brief outages.
Users will need to re-authenticate after reboots
Affected:
All Campus Wireless
- Notification modified: 2009-03-25 06:07:53
Update: Network Outage: Campus Power: 3-23-2009 09:00-13:45
Monday, March 23rd, 2009 - 10:45
Description: Due to widespread power issues, network devices in multiple locations across campus were not passing traffic.
- Notification modified: 2009-03-23 13:54:20
POWER RESTORED: Outage affecting IET Services 9:13am - 10:04
Power has been restored to most areas of campus. The Campus Data Center successfully switched from back generators to regular power without disruption to services and the IT Express phone center is also back on regular power.
- Notification modified: 2009-03-23 13:56:40
Campus Wide Power Outage 3/23/09 900-10:04
There is a campus wide power outage that began at approximately 0900 today, Monday, March 23, 2009. The problem was at the substation.
- Notification modified: 2009-03-23 10:36:46
LISTPROC INTERFACE DOWNTIME 3/20/09, 12:15 - 13:50
Listproc mailing list services were down. System administrators have resolved the issues.
- Notification modified: 2009-03-20 14:10:33
Campus authentication issues Wed 3/18 2215 - 2245
System administrators resolved issues with CAS, DistAuth, and RADIUS. Users are now able to access authenticated services.
- Notification modified: 2009-03-18 22:48:57
MyUCDavis Course Management Tools issues Wed 3/18 2100 - 2200
Clients using the MyUCDavis Course Management Tools may have experienced degraded performance due to a repeat of network problems seen earlier in the day. The Data Center performed switch maintenance and also relocated some servers to resolve the issue.
- Notification modified: 2009-03-18 22:33:36
Course Management Degraded Service, March 18, 2009
Wednesday, March 18, 2009 13:00 – 13:40
Clients using the MyUCDavis Course Management Tools may have experienced degraded performance. System administrators identified the source of a network problem and resolved the issue. Further work is planned to prevent this from recurring. In the meantime, the service has been restored.
- Notification modified: 2009-03-18 14:00:18
Network Outages- 3/16/08 - 10:00 - 10:50
Description: Due to widespread power issues, network devices in several locations are not passing traffic.
Status: Facilities is working to identify and resolve the cause of the power issues. NOC technicians are currently working to restore connectivity to the failed network devices where power is available.
Affected:
Some network devices in the following buildings:
Storer
Briggs
VMTH SURG
Estimated time to repair: completed
- Notification modified: 2009-03-16 11:13:00
SmartSite Maintenance - 3/16/09 12noon - 12:15
Maintenance to upgrade SmartSite-TEST with two new guest access enhancements / bug fixes has been completed. Test tag 042 was deployed to smartsite-test.
- Notification modified: 2009-03-16 12:31:02
Jira UPGRADE 3/15/2009 10:00pm
The problem with the previous night's Jira upgrade was resolved, and so, the upgrade was performed this night which completed at around 10:50pm.
- Notification modified: 2009-03-16 00:49:26
Network Outage- Radius 1800 - 3/9/09 - 10:59 3/10/09
Impact:
SSL VPN subscribers utilizing campus radius for authentication.
Locations Affected:
The campus Radius server suffered a hardware failure and was no longer serving authentication requests.
Status:
The cause of the problem has been identified and corrected. VPN Authentication should operate as normal.
- Notification modified: 2009-03-10 11:09:56
Moobilenetx issues 03/09/09 1800 - 2300
Faulty hardware led to moobilenetx becoming unavailable. System administrators worked around it and restored moobilenetx availability.
- Notification modified: 2009-03-09 23:05:58
Smartsite Maintenance Sunday, March 8, 2009 01:55 ďż˝ 03:05
SmartSite was unavailable due to maintenance related to the Daylight Savings time change.
- Notification modified: 2009-03-09 11:19:50
NOTICE: Campus Data Center Load Balancer reboot
On Friday 3/6/2009 between 0600 and 0630 the network traffic load balancers were rebooted. These devices are redundant, and clients did not see interruption in service.
- Notification modified: 2009-03-06 06:10:09
Website Maintenance Sunday February 22nd 1000 - 1200
The following websites were unavailable between 1000 and 1200 on Sunday February 22nd.
Clientservices.ucdavis.edu
Desdev.ucdavis.edu
Itexpresssurvey.ucdavis.edu
Itx.ucdavis.edu
Itxbusiness.ucdavis.edu
Itxnotes.ucdavis.edu
Itxsurvey.ucdavis.edu
Itxsurveydata.ucdavis.edu
Itxsurveyreporting.ucdavis.edu
Itxtest.ucdavis.edu
Kevmor.ucdavis.edu
Offline.ucdavis.edu
Online.ucdavis.edu
Remedy.ucdavis.edu
Slcdocs.ucdavis.edu
Smartsite2.ucdavis.edu
Tsp.ucdavis.edu
Uccsc.ucdavis.edu
Xbase.ucdavis.edu
- Notification modified: 2009-02-22 22:58:36
CAS problem 2/19 1900hrs - 2200hrs
CAS authentication was down for various sites except MyUCDavis.
System admins and network operations staff traced the source of the problem to the network traffic load balancers. The problem was identified and corrected at 22:00 and CAS was returned to service.
- Notification modified: 2009-02-19 22:24:55
Intermittent Access to Restricted Resources 2/19/09 1845-1900hrs
An issue with the Campus' authentication function was intermittently preventing access to Campus Services requiring authentication. Only some users were affected. During this event, access to services such as MyUCDavis, Course Management, smartsite, and any other service requiring authentication was unreliable. Affected users received an error message.
If you have any questions or concerns regarding this outage, please
contact IT Express at 754-HELP (4357).
- Notification modified: 2009-02-19 19:31:28
Course Management: Thursday, February 19, 2009
Course Management was unavailable from 13:20 to 14:15 due to a database issue.
Course Management is currently available. Users may have experienced intermittent disruptions due to ongoing hardware problems which has been resolved.
- Notification modified: 2009-02-20 10:23:35
LDAP Directory Service - 2/12/09 14:40-15:00
LDAP Directory Service was unavailable from 14:40 to 15:00. System administrators restarted the services on the systems.
- Notification modified: 2009-02-12 15:07:06
Primate Center Voice Connectivity has been lost 2/10/09- 2/13/09
Tuesday, February 10: 07:00
Description: Partial voice connectivity has been lost due to cable damage from construction activity at the corner of Dairy Rd and Hutchison Dr.
Locations Affected: Primate Center, CCM, and various buildings west of Hwy 113
Status: Repairs to the damaged cable have been completed.
Estimated Time to Repair: Completed
- Notification modified: 2009-02-13 14:20:53
System maintenance for MyInfoVault, Feb. 7th
System maintenance for MyInfoVault was completed. Saturday, Feb. 7th, from 7 - 8 a.m.
The web site is unavailable during that time.
- Notification modified: 2009-02-09 08:38:31
SmartSite Mail Server 02/03/09 7:00am - 07:40 am
SmartSite Mail Server service was restarted at 07:40.
- Notification modified: 2009-02-03 08:03:51
SmartSite Scheduled Maintenance 02/03/09 5:00am - 7:00am
SmartSite Community: We experienced downtime for a SmartSite upgrade, February 3, 2009, from 5am to 7am. Details of the upgrade can be found here:
https://confluence.ucdavis.edu/confluence/display/UCDSAKAI/Sakai+Production+Server+Maintenance+Schedule
- Notification modified: 2009-02-03 15:25:32
Campus Email, Thursday 1/22/09 13:10 - 13:25
The campus SMTP service was experiencing delays. System administrators have resolved the issue.
- Notification modified: 2009-01-22 13:33:15
XEDA Service interruption; 2100 - 2105 Friday, January, 16, 2008
The Xeda system administrators will be replacing the firewall device hardware with new hardware at 2100. Xeda systems, including Exchange and domain authentication will be temporarily unavailable during this switch-over.
Once completed, the Xeda system administrators will patch and update the Xeda systems during this regularly scheduled maintenance window from 9pm-Midnight
- Notification modified: 2009-01-16 21:53:42
UCDMC Connectivity Friday, 2008-01-16, 07:30 - 09:30
Connectivity between some areas on campus and the UC Davis Health System were down or intermittent.
It was determined that there was an issue with the Campus Data Center building router. NOC technicians have resolved the issue. Connections between the campus and the UC Davis Health System have been restored.
- Notification modified: 2009-01-16 09:45:57
AFS proxy server: Friday, 2008-01-16 02:30 - 09:30
The AFS proxy server located at UCDHS was currently unavailable. Clients were encouraged to install the AFS client per the instructions located here:
http://dafis.ucdavis.edu/install/install/andrew.cfm
The problem was resolved when connections from campus and the UCD Health Center were restored.
- Notification modified: 2009-01-16 15:25:02
Wednesday, January 14, 2009 14:55 - 14:56
ListProc was shut down momentarily for list maintenance.
- Notification modified: 2009-01-14 15:47:45
Resolved: Network Outage: AFS/Captive Portal: 1/14/2008 11:45:00
Impact:
Due to issues with the NOC firewall, AFS-3, AFS-Admin, Kraken-noc, and captive portal wireless authentication services were unavailable for a brief period.
Locations Affected:
All Campus locations attempting to access the hosts AFS-3, AFS-Admin, Kraken-noc, or clients attempting to utilize captive portal wireless authentication services.
Status:
The issues with the NOC firewall have been resolved. NOC personnel will work with the firewall vendor to determine a long term solution to the problem.
- Notification modified: 2009-01-14 12:02:29
Tuesday, January 13, 2008 09:40 - 13:17
ListProc was down during emergency maintenance to repair a corrupt database. System administrators corrected the error and restored service.
- Notification modified: 2009-01-13 16:18:11
SmartSite Scheduled Maintenance 01/13/09 5:30am - 5:47am
A scheduled maintenance was performed on SmartSite.
- Notification modified: 2009-01-13 06:16:38
LISTPROC INTERFACE DOWNTIME 1/08/09 21:00-22:30
Maintenance was performed on Listproc from 21:00 to 22:30.
- Notification modified: 2009-01-08 21:33:08
LISTPROC INTERFACE DOWNTIME 1/07/09 21:00-22:04
Maintenance was performed on Listproc from 21:00 to 22:04.
- Notification modified: 2009-01-07 22:11:13
MyUCDavis is down : 01/02/2009 0604 - 0832
MyUCDavis was temporarily unavailable due to a database connection issue. System administrators were contacted and they restarted the database. MyUCDavis access was restored as of 08:32 am.
If you have any questions about this or any other outage you can call IT Express at 530-754-HELP (754-4357).
-IT Express
- Notification modified: 2009-01-02 09:09:37
Power issues in Kerr Hall: 2008-12-29 00:00:00 - 01/29/09
2009-01-02 16:55:00
UPDATE:
Power to the network equipment in Kerr Hall has been temporarily restored, pending permanent repair.
- Notification modified: 2009-01-29 12:54:49
Intermittent Dafis Service 12/29/2008 07:45 - 09:30
System Adminitrators restarted coldfusion on several of the decision support web servers but the database had to restarted to restore connections.
- Notification modified: 2008-12-29 09:51:41
Network Device Reboot 12/23/2008 23:00
Some services may have been impacted temporarily or experienced a short delay while a network device was rebooted.
- Notification modified: 2008-12-23 23:27:10
Data Center scheduled maintenance - Sun 12/21 0600 - 1300
The Campus Data Center was shut down for electrical and network system upgrades. Services provided by and hosted in the Campus Data Center were unavailable during this interval. All work was completed by 1300 and access to affected services was restored.
If you have questions concerning this service outage, please contact the IT Express Computing Services Help Desk at 530-754-HELP (4357).
- Notification modified: 2008-12-21 13:47:06
listings.ucdavis.edu unavailable Wed 12/17 0600 - 0800
The Online Directory Application (listings.ucdavis.edu) was intermittently unavailable due to routine maintenance on Wednesday, December 17th from 6 am to 8 am. Please direct any questions about this service interruption to campusdirectory@ucdavis.edu or call the campus operator at 530-752-1011.
- Notification modified: 2008-12-17 08:00:46
MyInfoVault Unavailable 12/12/08 17:00 - 12/15/08
MyInfoVault was unavailable due to a system upgrade December 12 at 5:00 p.m., and was available again Monday, December 15, 2008.
- Notification modified: 2008-12-14 23:32:11
Degraded service MyUcdavis Courseware, Thurs. 12/11, 7:00-8:20PM
Some of the courseware servers experienced problems that caused connectivity problems for MyUCDavis. Problems were resolved by restarting services on affected servers.
- Notification modified: 2008-12-11 20:22:24
Smartsite-test maintenance, Dec 10, 2008, noon-5pm
Sak-2486 asset: update Java 1.5 to JDK / JRE 5.0 to update 1 was completed at 14:30.
- Notification modified: 2008-12-10 14:45:50
New Login Failures for MyUCDavis 12/8/08 10:00 - 10:25
The IT Express Service Desk received reports of new login failures for MyUCDavis. The Campus Data Center system administrators repaired the service at 10:25.
Updates will be available on http://status.ucdavis.edu and sent to the technotices mailing list.
If you have any questions about this or any other outage you can call IT Express at 530-754-HELP (754-4357).
- Notification modified: 2008-12-08 11:32:03
Paging Services intermittent problems. 12/8/08 9:00 -11:54
Thursday, December 8, 2007: 0900
Description: Intermittent paging services.
Locations Affected: Campus wide.
Status: Paging services have been restored. Trouble was caused by a bad UPS battery that allowed configs to be lost when power was interrupted. Repair complete. Paging services have been restored.
- Notification modified: 2008-12-08 12:05:07
Some MySpace accounts were unavailable 12/7/08 - 15:35 - 21:15
One of the servers providing MySpace service was experiencing problems. Some MySpace accounts were unavailable. System administrators resolved the issue.
- Notification modified: 2008-12-07 21:19:14
MyUCDavis & MySpace 12/5/08 14:10- 15:40
MyUCDavis was intermittently unavailable and the UC Davis MySpace tool within it was down due to AFS issues. System Administrators have resolved the problem.
MyUCDavis was available at 15:15. The MySpace tool within it was available at 15:40.
- Notification modified: 2008-12-05 15:44:15
SmartSite Scheduled Maintenance 12/02/08 5:30am - 7:00am
A regularly scheduled maintenance upgrade for SmartSite Production was performed Tuesday morning, December 2, at 5:30am – 7:00am. These include fixes for the Poll Tool and the Test and Quizzes Tool specifically for timed tests. For additional information, please follow the link below:
https://confluence.ucdavis.edu/confluence/display/UCDSAKAI/Sakai+Production+Server+Maintenance+Schedule
- Notification modified: 2008-12-02 06:07:01
MyUCDavis & MySpace intermittently unavailable 12/1/08 15:30-
15:30 to 16:40
MyUCDavis was intermittently unavailable and the UC Davis MySpace tool within it was down due to AFS issues. Access has been restored. Users of AFS clients may need to restart their programs.
- Notification modified: 2008-12-01 17:07:04
Router issues 12/1/08 8:33-8:41
A major area router serving DSL lines, T1 connections, and the modem pool was down. NOC itself is also on this router and was affected. NOC rebooted the router and everything is fine.
- Notification modified: 2008-12-01 08:42:09
Friday, November 28, 2008 21:00 -21:45
Geckomail underwent scheduled maintenance to move systems to a new subnet. Clients experienced brief downtime while the database was moved. Service was available during part of the above window.
- Notification modified: 2008-11-28 23:57:26
MyUCDavis - November 26, 2008, 08:10 - 08:35
MyUCDavis experienced degraded performance due to a software problem with one of its servers. Users who reached this server encountered an error.
Users who reached any of the other servers in the pool did not experience any trouble. System administrators isolated the problematic server and corrected the problem.
- Notification modified: 2008-11-28 09:05:36
SmartSite Wednesday, November 26, 2008 10:25 10:35
SmartSite showed an alarm on the status page. This was due to a monitoring configuration error.
There was no problem with SmartSite. The configuration issue has been resolved.
- Notification modified: 2008-11-26 11:30:04
Tuesday, November 25, 2008 16:20 - 16:30
SmartSite showed an alarm on the status page. This was due to a monitoring configuration error. There was no problem with SmartSite. The configuration issue has been resolved.
- Notification modified: 2008-11-25 17:04:00
Experienced Problems Bastion Host. 11/19/08 7:30
Logins into bastion host may have been hung as we were experiencing problems with one of our two servers. The issue has been resolved.
- Notification modified: 2008-11-20 10:20:58
Sched- Maintenance *non-production* SmartSite11/19/8 12:00-17:00
We are performing scheduled maintenance on the *non-production* SmartSite servers on November 19, 2008 from 12:00 to 17:00. For more information, please see:
https://confluence.ucdavis.edu/confluence/x/Q4Kw
- Notification modified: 2008-11-19 17:53:51
SmartSite downtime, 6:00AM, Saturday November 15, 2008
SmartSite experienced downtime from 6:00am to 6:10am, Saturday November 15, 2008. The SmartSite mail service needed to be restarted so that we do not experience delays in delivering site-related emails to users. Users online at that time lost their active sessions.
For additional information, please see the following confluence site:
https://confluence.ucdavis.edu/confluence/display/UCDSAKAI/Sakai+Production+Server+Maintenance+Schedule
- Notification modified: 2008-11-15 06:23:57
SmartSite Scheduled Maintenance 11/11/08 5:30am - 5:38am
Details for this maintenance: https://confluence.ucdavis.edu/confluence/display/UCDSAKAI/Sakai+Production+Server+Maintenance+Schedule
- Notification modified: 2008-11-11 05:49:36
LDAP service degradation, Thur. Nov. 6, 18:00-20:50
The campus LDAP servers were experiencing slowness. SSL connections were rejected and the People search tool on www.ucdavis.edu was unavailable. The System Adminstrators identified and blocked an off-campus computer that was causing an excessive load on the campus LDAP service. After some cleanup, the LDAP service was fully restored at 8:50PM.
- Notification modified: 2008-11-06 22:45:53
MyUCDavis issues - Wed 11/5/08 16:05 - 16:25
MyUCDavis experienced degraded performance during emergency hardware maintenance of one of its database servers. Clients may have experienced errors connecting to MyUCDavis during this time. System administrators resolved the issue as soon as possible.
- Notification modified: 2008-11-05 16:45:10
Slow mail delivery - Tue 11/4 14:37 - 15:24
Due to network problems mail delivery was slow between 2:37pm and 3:24 pm today.
- Notification modified: 2008-11-04 16:22:20
SMARTSITE DOWNTIME: Sunday, November 2, 2008
SmartSite required an hour and a quarter of down time to safely roll the clock back for Daylight Savings time this year. As a result *Smartsite was unavailable early Sunday November 2 from 1:50 a.m. PDT until 2:05 a.m PST*
- Notification modified: 2008-11-02 03:57:13
SmartSite Maintenance: Tuesday, October 28, 2008 04:20 - 05:55
SmartSite was unavailable due to maintenance.
- Notification modified: 2008-10-28 06:00:46
SmartSite, Saturday Oct. 25, 2008 6pm-7pm
We scheduled an emergency system restart for today, Saturday October 25, 2008 at 6:00 p.m. for repairs to SmartSite Email.
Mail delivery on SmartSite had slowed to the point that several of the servers required restarts to regain message delivery to SmartSite users. We scheduled this for 6:00 this evening to minimize the impact. Downtime was very short. If you were logged in during this period, you may have beeen forced off if you were on one of the affected servers.
- Notification modified: 2008-10-25 19:17:19
Emergency Reboot Banner Server 10/25/08 10:10-10:20
System Administrators performed an emergency reboot on one of the Banner servers to clear up some errors that were found after this mornings maintenance. The Banner service was unavailable for a brief period.
- Notification modified: 2008-10-25 10:25:55
Friday, October 24, 2008 22:00 - 22:40
SmartSite was temporarily unavailable due to a configuration error.
System administrators corrected that error and restored service.
- Notification modified: 2008-10-24 22:45:19
vmth.ucdavis.edu Unavailable 10/23/08 07:00 - 08:25
A building router in Vet Med Teaching Hospital was not passing traffic. NOC technicians were able to resolve the issue.
- Notification modified: 2008-10-23 08:28:01
SmartSite Maintenance : Oct 17 22:00 - Oct 18 09:00, 2008
SmartSite Team, Our weekend work resulted in completing scheduled patching by our system administrators. However, we were unable to complete the upgrade to SmartSite that would have created the My Campus Messages portlets. We will deploy this application next month or sooner if time allows.
You can review details of our weekend work here:
https://confluence.ucdavis.edu/confluence/display/UCDSAKAI/Sakai+Production+Server+Maintenance+Schedule
- Notification modified: 2008-10-18 08:36:03
AFS PERMISSION PROBLEMS 10/16/08 11:50-12:05
Due to an AFS server problem, a number of services became unavailable at approximately 11:50 a. m. Affected services include Geckomail and www.ucdavis.edu, the campus home website.
The AFS problem was resolved at approximately 12:05 p. m. but some of the dependent services were not restored till 13:00.
- Notification modified: 2008-10-17 13:44:22
MyInfoVault Scheduled Maintenance: Wed. 10/15/08 17:15 - 17:30
MyInfoVault was temporarily unavailable due to a system upgrade between 5:15pm-5:30pm on Wednesday 10/15/08.
The Web site was available again in fifteen minutes.
- Notification modified: 2008-10-18 06:29:44
SmartSite Scheduled Maintenance 10/15/08 12:00 - 1700
We performed scheduled maintenance on the *non-production* SmartSite servers on October 15, 2008 from 12:00 to 17:00. For more information, please see:
https://confluence.ucdavis.edu:8443/confluence/x/URM
- Notification modified: 2008-10-18 06:30:31
Campus DHCP Degraded Service 10/13/2008 13:30 - 14:00
The Campus is experienced degraded DHCP service. System Administrators restored the configuration file on the DHCP server and restarted NetID.
- Notification modified: 2008-10-13 16:26:05
Course Management Service Disruption 10/11/08 15:30 - 18:45
System administrators investigated a report of trouble accessing the Course Management system through the "My Classes" tab on MyUCDavis. The problem was found to be with content of a specific course. Administrators are working directly with the client to resolve the issue with their site. There were -no- problems with the Course Management system or MyUCDavis.
- Notification modified: 2008-10-11 18:47:36
Service Outage Notification: Data Center Services 10/09/08 18:00
A Data Center switch was rebooted at 6:00PM to resolve network connectivity problems. The reboot caused an outage lasting less than 1 minute for virtually all services hosted at the Data Center.
- Notification modified: 2008-10-09 18:56:31
Thursday, October 9, 2008 15:30 - 18:30
Email to SmartSite "Email Archive" tools was not being delivered. Smartsite was restarted at 6:30 p.m. to fix the problem and delivery of email resumed at 6:45 p.m.
- Notification modified: 2008-10-09 19:03:10
Xedamail Failed over and hung 10/03/08 15:50-15:55
Xedamail2 failed over and hung. System Administrators resolved the issues within 5 minutes.
- Notification modified: 2008-10-03 16:22:51
Authentication outage, 10/3/2008 15:50-16:00
Authentication issues affected access to secure services such as MyUCDavis, Geckomail, Smartsite, and My Admissions.
- Notification modified: 2008-10-03 16:02:28
XEDA Email Delay - 10/1/08 07:00-08:10
Due to an error with one of the Active Directory Exchange Servers (XEDA), exchange email delivery sent this morning was delayed for approximately one hour. System Administrators have resolved the issue.
- Notification modified: 2008-10-01 08:19:54
SmartSite Scheduled Maintenance: 9/30/08 05:30 - 05:54
A scheduled maintenance on SmartSite was performed.
- Notification modified: 2008-09-30 06:04:56
Banner Downtime Extension: 9/27/08 11:30
The scheduled Banner maintenance for 9/27/08 lasted till 11:30am instead of 10:00am. This affected services such as SIS and MyUCDavis Coursemanagement.
- Notification modified: 2008-09-27 12:05:30
ListProc Maintenance 9/26/08 21:45-22:20
System Administrators performed emergency maintenance on the ListProc server. The service was impacted for 35 minutes.
- Notification modified: 2008-09-26 22:22:52
ListProc Maintenance 9/26/08 11:30-11:45
System Administrators performed an emergency maintenance task on the ListProc server. Service was impacted for 15 minutes.
- Notification modified: 2008-09-26 11:47:54
SmartSite Maintenance: 9/23/08 05:30 - 05:56
A scheduled maintenance was performed on SmartSite.
- Notification modified: 2008-09-23 06:14:10
SmartSite Outage - Saturday, September 20, 2008 16:00 - 18:00
SmartSite was temporarily unavailable due to a configuration problem with its load-balancer. System administrators re-created the configuration, which resolved the issue.
- Notification modified: 2008-09-20 18:52:33
Upgrading firmware on DC switches. 2300 - 0100 ongoing 9/17/08
Switch firmware throughout the data center will be upgraded. Services connected to the switches may become unavailable for about 5 minutes during the necessary reboot of switch hardware between 2300 and 0100 hours.
- Notification modified: 2008-09-29 12:43:12
AFS Outage: Tue 9/16/08 00:00 - 05:00
There was a AFS hardware problem and the services that would've been affected were those that try to use the RW-path at the root.cell level. Those services that changed their links to the RO-path until they got to unreplicated volumes worked the whole time. Those that haven't, or couldn't because they're replicated all the way down their tree would've been out.
- Notification modified: 2008-09-16 06:12:00
Thursday, Sept 11th, 2008 13:52 -13:58
Description: Area 3 has become disconnected from the campus core network.
All traffic is passing normally at this time.
Estimated time to repair: Repairs complete.
Locations Affected:
All traffic into/out of area 3.
- Notification modified: 2008-09-11 14:35:15
Upgrading firmware on the Load Balancers 09/17/08 06:10 - 06:30
Between the hours of 0610 and 0630, Upgrading the firmware on the load balancers. This required a reboot of those systems. All load balanced services were unavailable for approximately 5 min while the hardware reboots.
- Notification modified: 2008-09-17 06:36:18
Data Center Network Maintenance 09/17/2008 06:00 - 06:10
The Data Center performed network maintenance on Wed, 9/17/2008 between the hours of 0600 and 0630. Between 0600 and 0610 hrs, the NOC rebooted the DC ADF. This interrupted all network connectivity to the Data Center, including XEDA and collocation clients. Downtime will be approximately 3 min.
- Notification modified: 2008-09-17 06:37:33
Xeda Exchange Mail Disconnect 09/10/2008 09:45
The Xeda Exchange Mail servers experienced a brief service outage affecting clients on XEDAMAIL2. Clients experience a disconnection with Outlook. Problem has been resolved.
- Notification modified: 2008-09-11 09:18:26
Xeda Exchange Mail Slow 09/10/2008 06:00 - 08:45
The Xeda Exchange mail queues were being delayed. New e-mail messages were being delivered but there were approximately 200 plus messages in the queue awaiting next opportunity to deliver. System administrators released the messages manually. Everything is back to normal.
- Notification modified: 2008-09-10 09:00:24
Listproc Mailing List Downtime: Wednesday 09/10 03:10 - 6:10
Listproc mailing list services were down. System administrators have resolved the issue. If you have any questions concerning this outage please contact IT Express at the number listed below.
IT Express
-------------------------------------------------------------------------
IT Express consultants are available via telephone Monday-Friday from 7am until 9pm at 530-754-HELP. Walk-in service hours and other information about our services may be found at: http://itexpress.ucdavis.edu/
-------------------------------------------------------------------------
- Notification modified: 2008-09-10 07:41:29
Monday, September 8, 2008 13:38 - 14:12
MyUCDavis and Course Management experienced degraded performance during
hardware maintenance. System administrators identified and resolved the
issue.
- Notification modified: 2008-09-08 15:06:04
MyInfoVault upgrade 9/3/08 17:00-17:10
MyInfoVault was temporarily unavailable due to a system upgrade after 5pm.
The Web site was available again in ten minutes.
- Notification modified: 2008-09-03 17:15:24
SmartSite Maintenance Tuesday 9/02/08 - 5:30am - 7:00am
Maintenance was performed on Smartsite on Tuesday, Sept. 2 from 5:30 AM to 7:00 AM
For more information please visit...
https://confluence.ucdavis.edu/confluence/x/URM
- Notification modified: 2008-09-03 11:47:06
Degradation of Services Friday, Aug. 29, 2340hrs - 0005hrs
Power was disrupted to a few machines at 11:45PM on Friday Aug 29, causing a brief degradation to a number of services; smartsite, distauth, exchange and a few others were affected. System administrators carefully assessed these systems and determined that all services were completely restored.
- Notification modified: 2008-08-30 04:31:08
Authentication Outage Aug, 26, 23:00 - 02:00
Authentication issues that began at 23:00 affected access to secure services such as MyUCDavis, Geckomail, Smartsite, and My Admissions. Access to services was restored by 02:00.
- Notification modified: 2008-08-27 02:11:43
Geckomail unnavailable Fri. Aug. 22, 2008 2200hrs - 2359hrs
Geckomail was unavailable from 10PM until midnight, while system administrators performed maintenance. If you have any questions, please contact IT Express at 754-4357.
- Notification modified: 2008-08-23 22:51:08
Geckomail outage Wed 08/20/08 1900 - 19:45
Data Center staff worked to resolve a hardware issue that affected access to geckomail. Everything is back to normal.
- Notification modified: 2008-08-20 19:47:48
Remedy Help Desk Application Unavailable August 18, 16:30-17:10
The Remedy Help Desk application was unavailable.
- Notification modified: 2008-08-18 17:24:17
Network Maintenance: Saturday 8/16/2008, 23:00hrs - 23:05hrs
On Saturday August 16, at 23:00, the Campus Data Center conducted maintenance on its network infrastructure. Network services were interrupted for no longer than 5 minutes. This affected, although not limited to, the following firewalls and services:
Firewalls: dafis, sevis, edrs, exchange, and era,
Services: myucdavis, geckomail, www, smartsite, distauth.
- Notification modified: 2008-08-17 01:45:59
Geckomail outage, 8/14 23:40 until 8/15 00:15
An unanticipated hardware failure resulted in a brief outage of the geckomail service.
The outage lasted from 11:40PM Thursday night until 12:15AM.
- Notification modified: 2008-08-15 01:28:29
Data Center Switch Firmware Updates 8/15/08 00:00 - 00:30
The Data Center updated the firmware on a couple switches tonight at midnight. Campus services may have experienced a 30 second disruption as the firmware was updated. No significant downtime was experienced during the firmware updates.
- Notification modified: 2008-08-15 06:18:40
MoobileNetX Licensing Outage 8/13/08 15:21-15:45
MoobileNetX was unavailable from 15:21 to 15:45. Support from IdEngines provided a base license within that 20 minutes and service was restored immediately as it was loaded.
- Notification modified: 2008-08-13 16:26:09
MyInfoVault unavailable 8/15/08 5:00pm - 8/18/08 7:00am
MyInfoVault was unavailable starting Friday, August 15 at 5PM for an upgrade to version 2.1. MyInfoVault was up and running the new version on Monday, August 18 at 7:00 a.m.
- Notification modified: 2008-08-19 15:11:16
Switch Reboot on Wednesday, 08/13/2008 @ 06:00-06:30
A network switch was rebooted on Wednesday, August 13 during 06:00 - 06:30. The expected downtime was for one minute or more. The affected firewalls will be: Utilities and Enceinte.
- Notification modified: 2008-08-13 07:02:52
SmartSite Maintenance - 8/12/08 5:30am to 7:00am
We performed scheduled maintenance on SmartSite on August 12, 2008 from 05:30 to 07:00.
- Notification modified: 2008-08-12 07:26:32
Emergency Firewall Downtime 8/10/2008 22:00 - 22:35
On Sunday, August 10, at 22:00, a network switch was replaced in an attempt to solve intermittent service disruptions. This affected, but was not limited to, the following firewalls:
Utilities, ORMP, Enceinte, EDRS, and EDMS.
Banner systems in particular were affected; however, other services may also have been affected. Downtime was 35 minutes.
- Notification modified: 2008-08-11 00:03:28
Network Outage at the Colleges at La Rue 8/10/2008 17:00
Some dorms at the Colleges at La Rue experienced a network outage.
- Notification modified: 2008-08-12 10:59:36
Course Management Service Degradation, 8/9/08 17:00 - 8/10 22:35
After the network problems from 8/8/08 at 09:55 services returned to full operational standards. However services linked to Banner, such as the MyUCDavis Course Management pages, continued to experience slowness and the inability to accept new connections.
The Banner firewall was rebooted at noon on Friday, 8/8/08, in order to reset the necessary network connections and network traffic returned to normal. However, approximately 30 minutes later, the firewall started to show more dropped packets. System administrators worked with the firewall vendor to resolve this issue. Services that depend upon Banner were in a degraded or unavailable state.
A network switch was replaced at 22:00 on Sunday, 8/10/08, which resolved connectivity issues. All affected services are operating normally.
- Notification modified: 2008-08-11 00:13:03
XEDA EXCHANGE 08/08/08
VEXBE2 is experienced issues due to network outage at the Data Center. System administrators resolved the issue with VEXBE2.
- Notification modified: 2008-08-08 10:39:09
Network Outage 08/08/08 09:50 - 12:00
A very brief network outage in the Data Center resulted in several services, including the XEDA Exchange server, to be momentarily unavailable. Users may have experienced continued difficulty reaching the XEDA servers until they closed and re-opened their email program.
Additional services were still experiencing some degradation and System Administrators inventoried affected systems and took measures to restore them to full operational standards.
- Notification modified: 2008-08-08 13:09:04
Degragated Campus List Server 08/06/2008 01:45 - 16:10
The campus list server was experiencing intermittent stoppages and slowness through out the day. Listproc has been restored to normal functionality. Due to a mail loop, the mail queues became backed-up and slowed down mail delivery. New messages are being processed normally. Any queued messages that are deliverable should be processed by early evening. All messages bound for @ucdavis.edu addresses have already been processed.
- Notification modified: 2008-08-06 17:49:43
Listproc slowness Wed 08/06/08 02:00 - 04:00
System administrators resolved issues with listproc slowness.
- Notification modified: 2008-08-06 04:16:09
Geckomail and Confluence Maintenance 08/02/08 04:00-04:30
System Administrators performed maintenance on a database server at 04:00-04:30 on August 2, 2008.
This affected Geckomail, Jira, and Confluence services.
- Notification modified: 2008-08-02 04:56:40
The SmartSite Gradebook tool is currently unable to submit final
The problem with the SmartSite Gradebook has been resolved.
SmartSite was shutdown at 3:00 a.m. and successfully restarted at 3:40 a.m.
The SmartSite team has tested a fix and will implement it beginning at 3:00 a.m. on Thursday, July 31. Estimated downtime for SmartSite will be 45—90 minutes. Final grades submitted using either the MyUCDavis Gradebook tool or the stand-alone e-grading tool (classes.ucdavis.edu/grading) are not affected.
- Notification modified: 2008-07-31 06:28:39
Kerberos Authentication 07/30/2008 11:45 - 12:00
Authentication services (CAS, Secureweb, kerberos) were unavailable from 11:45-12:00 due to a system malfunction. Systems administrators identified the problem and restored service.
- Notification modified: 2008-07-30 12:11:16
Emergency Maintenance for Ms1 - 20:00 07/25/08
Due to a failed hardware component, the email server, Ms1, was unavailable for approximately 10 minutes in order to fail it over to a secondary server. During that time, email was queued, and it was delivered once Ms1 was again available.
- Notification modified: 2008-07-25 22:20:47
Data Center Load Balancer Emergency Reboot 7/23/08 06:00-06:30
On Thursday, July 24th 2008, between the hours of 0600 and 0630, the Campus Data Center performed emergency maintenance to a network load balancer. During the maintenance window, services were unavailable for brief periods of time. Services that were affected include proxy-AFS, CAS, Computing Accounts, DaFIS, Distauth, EDRS, Geckomail, LDAP, MyUCDavis, MyTravel, Smartsite, Student Information Systems, Whitepages, & www.ucdavis.edu.
All services were fully restored by 0630 hrs.
- Notification modified: 2008-07-24 06:24:45
Cold Fusion service having issues. 7/23/08 14:38 - 15:00
The Cold Fusion service on all of the MyUCDavis and Course Management systems was having issues. The web servers were restarted and the problem was resolved as of 15:00.
- Notification modified: 2008-07-23 16:40:00
DaFIS proxy access 7/23/08 10:30 -22:16
As of approximately 10:30 this morning, some DaFIS users were experiencing problems with DaFIS proxy access. Campus Data Center staff were made aware of the issue, and worked with DaFIS personnel to resolve the problem.
- Notification modified: 2008-07-23 22:19:27
Xeda Exchange brief outage 7/23/08 09:45-09:55
On Wednesday, July 23, around 9:45am a Xeda Exchange server experienced a brief outage for approximately 10 minutes. System administrators quickly restored the services. Only a few reported mailboxes in IET and Financial Aid were affected.
- Notification modified: 2008-07-23 10:19:31
Remedy Down 7/23/08 - 5:00am - 6:38am
Remedy Service was unavailable due to System issues.
- Notification modified: 2008-07-23 06:41:37
Distauth issues Tue 7/22/08 21:55 - 01:05
Distauth issues that began at approx. 21:55 Tue 7/22 were resolved at 01:05 Wed. 7/23. These problems affected authenticated services including MyUCDavis, Geckomail, Moobilenet, SmartSite, and My Admissions. While indications are that the outage may have been caused by a denial of service attack, system administrators will be reviewing system logs to determine the root cause of the issues.
- Notification modified: 2008-07-23 01:34:43
ms11 outage 7/21/08 11:30-11:50
Users with their UC Davis email delivered to the the Cyrus mail store ms11.ucdavis.edu were unable to make a connection to their email server this morning between 11:30 and 11:50am. The cause of the issue was traced to the failover process. The process has been corrected and service restored.
- Notification modified: 2008-07-21 12:18:25
XEDAMAIL1:Outlook Email Unavailable 07/19/2008 - 7/21/2008 07:55
One of the Exchange Cluster Servers for Xeda had experienced a a problem delivering mail starting Saturday 07/19/2008. All groups located on XEDAMAIL1 including Accounting, AE, HR, ORMP, Vice Chancelor Office Mrak were not been able to access their mail. System Adminitrastors were able to fix the problem and as of 07:55 Exchange Mail delivery is working.
- Notification modified: 2008-07-21 08:30:12
ISUN Servers Unavailable. 7/11/08 09:35 - 09:45
The ISUN servers were temporarily unavailable.
- Notification modified: 2008-07-11 09:52:07
AFS services are unavailable 07/11/08 8:00am-11:30am
Due to storage array failure, some AFS services, such as SISWEB, were unavailable. System administrators replaced controller.
- Notification modified: 2008-07-11 12:25:05
NOC / DC System Relocation. Thr. 7/11/08 5AM-7AM
The Network Operations Center moved some existing equipment and placed new equipment into the Data Center network racks on Fri, 7/11/2008 between 0500 and 0700 hrs. No no interruption in services were expected or experienced.
The new equipment has prepared us for Network 22 (10GB) connectivity.
- Notification modified: 2008-07-12 09:56:48
Campus Wireless Network (Moobilenet) 7/8/08 10:41-11:53
The virtual interface on one of the primary campus wireless controllers became unreachable for a period of approximately 10 minutes. The controller was manually failed over to the secondary physical interface and all campus wireless connectivity has been restored. NOC then had to work on a certificate issue on the Aruba controller.Users were still able to log on while NOC corrected the certificate issue.
- Notification modified: 2008-07-08 12:04:43
SmartSite Upgrade 8AM Sat. 7/19/08 - 8:00 - 14:54
SmartSite was unavailable from 8:00 am Saturday July 19th until 14:54 due to an upgrade.
- Notification modified: 2008-07-21 08:36:53
DaFIS Proxy Service
Data Center and DaFIS administrators are continuing to investigate intermittent connection problems with the DaFIS proxy service. DaFIS TP users are strongly advised to use the AFS client to access the application. AFS client information and installation instructions are available at http://dafis.ucdavis.edu/install/afs/.
- Notification modified: 2008-07-01 23:52:16
Geckomail and distauth issues Tue 7/1/08 18:30 - 21:00
Geckomail was unavailable and distauth experienced slowness due to a hardware failure. System administrators resolved the issue.
- Notification modified: 2008-07-01 21:40:35
Smartsite maintenance; Sat. morning, Jun 28, 00:00 - 04:00
We have finished maintenance on SmartSite. Users will be able to access SmartSite after the maintenance window ends at 02:40.
- Notification modified: 2008-06-28 06:27:21
Software web site unavailable Thursday 06/26/08 from 12:00p-1:00
The Campus' Software License Coordination (SLC) Unit web site, https://my.ucdavis.edu/software/, was unavailable on Thursday , June 26, 2008 from 12:00 pm to 1:00 pm during a planned outage.
Please contact the SLC unit via software@ucdavis.edu if you have any questions regarding this downtime.
- Notification modified: 2008-06-26 15:55:45
Data Center Generator Test. Wed. 7/16/08 6AM- 7AM
Facilities tested the Data Center generator Wednesday July 16 between the hours of 6:00AM and 7:00AM.
- Notification modified: 2008-07-21 08:37:53
MyBill Service Unavailable 06/26/08 06:30 - 09:00
MyBill was unavailable. System administrators fixed a problem with the mybill firewall and service has been restored.
- Notification modified: 2008-06-26 09:18:49
Saturday, June 28, 2008 00:00 - 04:00
SmartSite was temporarily unavailable due to an upgrade to the Modules tool.
- Notification modified: 2008-06-30 06:17:24
Distauth issues Tue 6/24/08 08:45 - 09:30
Services reliant on central campus Kerberos authentication were unavailable from 08:45 to 09:30 this morning. The software running the Kerberos software on both the primary and redundant KDC was running, but 'frozen' and not serving Kerberos authentication. A full restart of both servers was necessary to restore service.
- Notification modified: 2008-06-25 13:54:51
MyInfoVault will be unavailable: Thu, 06/26/08 - Mon, 06/30/08
MyInfoVault was unavailable starting Thursday, June 26 at 5PM for an upgrade to version 2.0. MyInfoVault is up and running the new version on Monday, June 30 at 7:46 a.m.
- Notification modified: 2008-06-30 07:47:49
Banner Database Problems 06/21/08 - 06/23/08 08:30
The Banner Database, beginning at 8:30PM 06/20/08 maintenance window was unavailable to all clients. Several services that rely on the Student Information Database were unavailable, including sisweb, myadmissions, course management, and smartsite. Some services were made available at the end of the normal maintenance window (10 AM Saturday). However some services were down due to a connection problem to the database. System Administrators fixed the problem and all services were return at 08:30.
- Notification modified: 2008-06-24 09:09:19
Extended Banner Maintenance. Friday, June 20, 8:30PM - 10AM
An extended maintenance window was required for the Banner Database, beginning at 8:30PM instead of the usual 10PM start-time, therefore several services that rely on the Student Information Database were unnavailable, including sisweb, myadmissions, course management, and smartsite. All services were made available at the end of the normal maintenance window (10 AM Saturday).
- Notification modified: 2008-06-21 10:58:27
MyUCDavis service interruption - 6/19/08 1945hrs - 2045hrs
The MyUCDavis portal experienced a service disruption lasting approximately 1 hour, starting from 7:45 PM on Thursday, June 19. System administrators identified the cause of the outage and restored the service. If yoou have any questions concerning this outage please contact IT Express at 530-754-HELP (530-754-4357)
- Notification modified: 2008-06-19 21:15:11
Campus Wireless Network Unavailable 6/17/08 8:00 - 6/18/08 12:30
Description: A hardware failure on a core campus router caused one of the wireless controllers to become unreachable for an extended period of time. Once connectivity to the controller was restored, the access points should have re-associated with it and continued functioning normally.
Due to what is currently presumed to be a bug with the vendor code, the software image on 238 access points became corrupted, which prevented the devices from starting up correctly and connecting back to the controller.
NOC Engineers worked extensively with the vendor throughout the day Tuesday to determine a resolution to the issue. Once it was determined that a remotely implemented fix was not possible, NOC technicians and Engineers began dispatching to end user sites throughout campus to manually bring each of the access points back online.
Status: At this time, all campus and remote wireless access has been restored. Estimated time to repair: Repairs complete.
- Notification modified: 2008-06-18 12:51:59
MyTravel.ucdavis.edu Software Error 6/13/08 07:30 -09:30
Users may have experienced an unexpected software error when connecting to mytravel.ucdavis.edu. System administrators have resolved the issue.
- Notification modified: 2008-06-13 09:38:36
Data Center Network Switch Emergency Reboot 6/10/08 06:30
Due to emergency maintenance, a data center network switch was rebooted at 06:30. Services that might have been affected include EDRS, Utilities, MyTravel, MyInfo Vault, Active Directory, and eRA.
- Notification modified: 2008-06-10 08:11:47
DaFIS Service Degredation 6/6/08 10:30-11:00
Many campus clients across departments and networks were reporting Uniface errors when attempting to launch DaFIS. System administrators restarted services on two of the DaFIS license servers. The issue has been resolved.
- Notification modified: 2008-06-06 10:59:33
Blackberry messaging issues 6/6/08 10:00
We are receiving reports that some Blackberry users are not receiving email messages on their handheld devices. The server and systems appear to function normally and preliminary reports point to a Sprint/Nextel carrier related problems. System administrators are contacting the carrier as well as RIM (Blackberry) to receive further information and isolate the problem.
**Update**
The few Blackberry messaging outage problems appeared to have been transient and are resolved now. We have confirmed functionality for those who were affected, but if you continue to see any issues, please contact your department technical support.
- Notification modified: 2008-06-06 10:24:05
SmartSite Unavailable 6/04/08 - 05:30 - 7:00
SmartSite was unavailable due to maintenance.
- Notification modified: 2008-06-04 07:20:16
Data Center Emergency Network Downtime 5/31/2008 0700am-1:30pm
Lingering issues from the firewall maintenance, which affected DaFis, MyTravel, and Utilities services were resolved at 1:30pm
In troubleshooting some network latency at the Campus Data Center, we
discovered a defective blade in the DC firewall. That blade needs to be
replaced.
On Saturday, 5/31/2008, from 0600 hrs to 0700 hrs, the DC network will
be unavailable to replace the failing part in the firewall. All
services except the campus AD/Exchange service (XEDA) will be affected
and unavailable during that time.
- Notification modified: 2008-05-31 14:22:59
Distauth/authentication restart 5/31/08 1:00am -
The restart was done within the firewall maintenance.
On Saturday at 1am, there will be a 10-minute Distauth/authentication restart that may affect services that rely on it such as MyUCDavis, SmartSite, Geckomail, Distauth, SISWeb and Geckomail. The purpose of this scheduled maintenance is to attempt to increase the capacity and throughput of the Distauth back-end in order to avoid unplanned downtimes. Estimated time is approximately ten minutes.
- Notification modified: 2008-05-31 14:24:33
Distauth issues Tue 5/20/08 22:20 - 23:05
Distauth issues that began at 22:20 affected access to secure services such as MyUCDavis, Geckomail, Smartsite, and My Admissions. Distauth was functioning normally by 22:40. Lingering problems with MyUCDavis were resolved by 23:05.
- Notification modified: 2008-05-20 23:34:39
UCDavis Gmail Pilot Outage Tue 5/20 18:00 - 21:15
A CAS issue affected access to the Gmail Pilot site. Gmail was the only service affected and is functioning normally as of 21:15.
- Notification modified: 2008-05-20 22:10:05
Distauth issues Mon 5/19 22:20 - 22:45
System administrators investigated distauth issues that affected access to services dependent upon distributed authentication such as MyUCDavis, Geckomail, SmartSite, and My Admissions. As of 22:45, the issues have been resolved and all services are functioning normally.
- Notification modified: 2008-05-19 23:05:23
Exchange Services Slow Response 04/19/08 09:00 - 16:00
Earlier today we had issues with vexbe1. There was a public folder replication storm that occurred when the public folders on Vexbe1 were trying to replicate with Xedamail1. This flood of emails caused the mail queues to get backed up so that message delivery was extremely slow. Xeda admins were able to find a global command that stopped public folder replication temporarily so that mail flow could resume to normal.
The public folder replication problem started because a replication over the weekend did not finish. Exchange 2003 public folders were being replicated to Exchange 2007. That replication has been turned off and will be rescheduled for a future date.
Outlook users may experience some slowness as the Exchange queues get caught up.
Xeda administrators will monitor the queues closely and ensure that messages continue to flow properly over the next day or two.
- Notification modified: 2008-05-19 16:48:47
MyUCDavis - MySpace Unavailable 5/17/08 13:10 - 15:30
The Campus Data Center had been notified from several computer labs on campus that users are unable to connect to MySpace on MyUcdavis. Only connections from computer labs on campus had been reported as unable to connect. System Administrators rebooted the problematic servers. Service was restored at 15:30.
- Notification modified: 2008-05-17 15:42:01
SIS Banner maintenance extended 5/10/08
Weekly scheduled maintenance is done on the SIS Banner hosts Saturday morning until 10:00am. Due to a database issue, the maintenance ran over by approximately 10 minutes.
- Notification modified: 2008-05-15 15:32:55
Mytravel.ucdavis.edu Software Error 5/15/08 07:25 - 15:45
Users may have experienced an unexpected software error when connecting to mytravel.ucdavis.edu. The problem was isolated to a single server which has been removed from the load balanced pool. If you are still having connection issues, please contact the MyTravel helpdesk (mytravel-info@ucdavis.edu) or ITX (530-754-HELP).
- Notification modified: 2008-05-15 17:33:23
Distributed Authentication Issues 20:05 - 20:40
System administrators resolved issues with distributed authentication. Clients were unable to access Geckomail and MyUCDavis, MyAdmissions.
- Notification modified: 2008-05-13 20:50:31
Distributed Authentication Issues 14:35 - 14:45
05/13/08 14:35 - 14:45
Clients reported problems with distributed authentication. Clients were unable to access Geckomail and MyUCDavis, MyAdmissions. High load cleared up after a few minutes.
- Notification modified: 2008-05-13 14:47:01
Distributed Authentication Issues 12:55 - 13:25
05/13/08 12:55 - 13:25
Clients reported problems with distributed authentication. Clients were unable to access Geckomail and MyUCDavis, MyAdmissions. System admin rebooted afsadmin9.
- Notification modified: 2008-05-13 13:52:30
Distributed Authentication Issues 12:35 - 13:47
05/12/08 12:35 - Ongoing
Clients reported problems with distributed authentication. Clients were unable to access Geckomail and MyUCDavis, MyAdmissions. System admin rebooted afsadmin9.
- Notification modified: 2008-05-12 14:03:18
Smartsite interruption; Wed., May 7, 2008 16:10 - 16:30
One SmartSite application server in the pool experienced a brief network
loss due to a configuration error. Clients connected to that server in
the pool may have experienced a disconnection, but should have been able
to re-connect to another server in the pool.
- Notification modified: 2008-05-07 18:18:29
SmartSite Scheduled Maintenance 5/6/08 5:30 - 7:00
We are performed scheduled maintenance on SmartSite on May 06, 2008 from 05:30 to 07:00.
- Notification modified: 2008-05-12 14:05:06
Sisweb is slow in authentication 14:00 - 15:16
SISWeb was slow responding to authentication.
- Notification modified: 2008-05-05 15:41:55
Campus Library Proxy via Distauth 5/5/08 10:05 - 11:32 .
The campus library proxy via distauth was not working since friday 05/02/05. Logging on through the library VPN was working. "A *workaround* was to first authenticate by logging in to GeckoMail or MyUCDavis and then accessing the Library services."
Problem was fixed when a AFS volume was recovered from backups. Everything was back to normal at 11:32.
Off-campus access to the distauth-protected news.ucdavis.edu web reader is now restored. Given the similarity in services and the timeframes involved, this was almost certainly related to the library issue as well.
- Notification modified: 2008-05-05 13:42:06
Distributed authentication issues Wed 4/30 20:25 - 21:00
Clients reported problems with distributed authentication. System administrators restored distributed authentication and clients were able to access Geckomail and My Admissions by 20:50. Access to MyUCDavis was restored by 21:00.
- Notification modified: 2008-04-30 21:20:01
Campus Data Center firewall issue Tue Apr 29 22:22 - Wed 01:30
A firewall issue affected connectivity to various Campus Data Center hosted services. System administrators resolved the firewall issue by 23:34 and distributed authentication service by 01:30 Wed. The Secureweb service did not recover upon network restoration due to the bad state of the servers.
- Notification modified: 2008-04-30 01:53:12
Network Load Balancer Reboot 4/29/08 - 06:00 - 06:02
On Tuesday, 4/29/2008 at 0600 we rebooted a network load balancer to resolve a mac address discovery problem. This disrupted service to some data center services for approximately 2 min.
- Notification modified: 2008-04-29 06:16:57
Decision Support Coldfusion Brief Outage 4/28/08
Restared coldfusion on sugar.ucdavis.edu at 2:55 pm 4/28/08 - Lisa
- Notification modified: 2008-04-28 15:36:06
- Notification modified: 2008-04-26 16:18:43
Kerberos Tickets Unavailable 4/26/08 15:45 - 16:15
Kerberos Tickets were not being distributed resulting in logins to MyUCDavis.edu being rejected. System Administrators have resolved the issue.
- Notification modified: 2008-04-26 16:16:49
SmartSite Maintenance 4/22/08 5:30 - 7:00
We performed scheduled maintenance on SmartSite on April 22, 2008 from 05:30 to 07:00.
- Notification modified: 2008-04-22 07:10:26
MS13 Down for maintenance 4/21/08 11:43 - 11:54
The email server, ms13, was unavailable for 15 minutes to resolve a software problem.
- Notification modified: 2008-04-21 12:04:33
Authentication services 11:45am - 12:15pm, 4/17/08
Data Center Administrators have updated DNS entries for the secureweb and distauth authenication services and authentication services have been restored.
However users may still experience problems with logging in, in which case they should clear their web browser's cache and restart their web browsers, in order to get the new DNS entries for the authenication servers. In some cases users may need to re-start their computers.
Additionally the secure web log in page may be slow due to load as individuals try to log back in.
If you have any questions, or are still experiencing problems logging in after re-starting a computer, please call IT Express at 530-754-HELP
(754-4357)
-IT Express
- Notification modified: 2008-04-17 16:58:37
SERVICE OUTAGE: Authentication Services 11:45am April 17, 2008
System Administrators resolved the issue with the authentication services. The connections to secureweb.ucdavis.edu were failing and returning a "Wrong user name or password" error for all users. System Administrators resolved the issue at 12:15pm today.
- Notification modified: 2008-04-17 13:03:43
Exchange Services Slow Response 04/16/08 08:00 - 09:30
04/16/08 08:00 - 09:30
The Exchange servers are running normally now. The queues on VEXBE1 were backed up which points to the message transfer agent becoming sluggish. We got the messages in the queue to deliver and it is delivering normally now.
- Notification modified: 2008-04-16 10:28:38
Slow Response Exchange Service 04/15/08 08:00 - 10:00
The Exchange mailserver VEXBE1 had received bulk email messages this morning causing slow response times. The recipient list included >1000 external recipients at approx 700KBs in size.
We had failed the cluster node over to the passive node and manually removed the queued messages, with services restored at approximately 9am. We have notified the sender to contact the bulk mail folks at ITX on handling messages of this type.
VEXBE1 is operational at this time, so our plan is to failback the cluster node at 12:00pm this afternoon, when less folks are connected. The failback will cause a disconnect of email clients for approximately 90 seconds.
- Notification modified: 2008-04-15 10:04:25
Banner Forms 04/14/08 04:45 - 05:45
The Banner Forms servers was refusing connections at 2:45pm April 14th and
was unavailable from about 3pm until 3:45pm. A server reboot resolved the
problem.
- Notification modified: 2008-04-15 08:37:58
SmartSite Scheduled Maintenance 4/15/08 5:30 - 7:00
We performed scheduled maintenance on SmartSite on April 15, 2008 from 05:30 to 07:00.
- Notification modified: 2008-04-21 13:15:50
MyUCDavis Maintenance Saturday, 4/12/08, 12am-6am
The MyUCDavis portal was unavailable due to maintenance this Saturday, April 12th, 2008, from 12am to 6am. System Administrators migrated the background databases for MyUCDavis to a new configuration.
- Notification modified: 2008-04-14 14:43:22
Remedy Outage 04/08/08 08.00 - 08:35
The Remedy Service was unavailable this morning from approximately 8:00 am - 8:36 am due to problems with the Remedy database server. The server was rebooted and service was restored.
- Notification modified: 2008-04-08 10:03:47
Scheduled Maintenance of SmartSite: 04/08/2008, 05:30 - 07:00
A scheduled maintenance was performed on SmartSite.
- Notification modified: 2008-04-08 05:54:15
knockdc2 ssh problems
04/07/08 08:00 - 09:00
The ssh-server process on knockdc2 was pegged up at about 70% CPU usage and unresponsive. Users connecting to the pooled address would've gotten connection timeouts until they retried and were pointed to knockdc1; existing connections should've stayed up.
- Notification modified: 2008-04-07 09:28:40
SmartSite Scheduled Maintenance: 04/01/08 05:30 - 07:00
Scheduled maintenanced on SmartSite for April 01, 2008 from 05:30 to 07:00 was completed at 06:55.
- Notification modified: 2008-04-01 07:12:42
SmartSite degradated services 03/30/08 12:00
SmartSite experienced a brief degradation of service in which access to Site Info will generate errors. Our DBA fixed the error and the error was fixed within 5 - 10 minutes.
- Notification modified: 2008-03-31 14:06:27
ms11 hardware issue 03/26/08 18:12 - 18:42
ms11 experienced a hardware failure. Users on ms11 were unable to access their email. System administrators resolved the issue.
- Notification modified: 2008-03-26 18:57:38
Usenet News Service Down 12:00 - 14:30 3/26/08
Usenet was down for a couple hours while system administrators rebuilt its file structure.
- Notification modified: 2008-03-26 14:55:37
Campus Mail Service Planned Outage 03/26/08
03/26/08
Start Time: 18:00
End Time: 21:00
System administrators performed maintenance on the campus mail servers between the hours of 6pm and 9pm, Wednesday, March 26. During this time, ms11, ms12, and ms13 may have experienced brief downtime.
- Notification modified: 2008-03-26 22:36:56
SmartSite Scheduled Maintenance: 03/25/08 05:30 - 07:00
We are performed scheduled maintenance on SmartSite. For more information, please see:
https://confluence.ucdavis.edu:8443/confluence/x/URM
- Notification modified: 2008-03-25 11:28:47
DAFIS AFS ProxyPool Degradation 3/10/08 - 18:37 - 3/11/08 11:35
Starting on the morning of Monday, March 10, 2008, reports from several departments came in that they were unable to successfully use the DaFIS AFS-proxy pool to make DaFIS connections. The problem was isolated from another reported Oracle client installation issue and is being investigated by the appropriate network groups.
The resulting conclusion was that several networking problems were misreported, coupled with confusion on weekend DaFIS application upgrades and ongoing network issues located on certain parts of campus, caused both the DaFIS Help Desk (an existent issue) and two departments (not five as reported) to have trouble accessing the app.
- Notification modified: 2008-03-11 13:05:35
MoobilnetX Maintenance 3/12/08 17:15 - 17:45
The MoobilnetX service was down for hardware replacement from 17:15 to 17:45. Existing/currently authenticated users did not lose their connections, but no new authentications were issued while the service was down.
- Notification modified: 2008-03-12 19:47:05
SmartSite Maintenance: Sunday, March 9, 2008 01:50 - 03:10
SmartSite was unavailable due to a scheduled maintenance.
- Notification modified: 2008-03-09 03:15:27
Myweb5 and Myweb10 power cycled.
03/04/2008 09:00 - 09:12
Systems were manually power cycled to remove from a halt state.
- Notification modified: 2008-03-04 09:13:05
MoobilenetX connectivity issues 2/29/08 10:00 - 11:00
Description: Users reported issues connecting to the moobilenetX campus wireless network. Regular moobilenet wireless was functioning normally.
Status: Network Operations and Datacenter technicians resolved the issue.
- Notification modified: 2008-02-29 11:07:51
Network connectivity issues 2/26/08 1530 - 1720
Between the hours of approximately 1530 and 1720 the campus data center experienced network connectivity problems isolated between certain networks on campus and certain data center VLANs. Network operations technicians made some programming changes to campus routers to help isolate the problem. Network connectivity was restored around 1720 HRS, although technicians are still investigating the cause of the actual
outage.
- Notification modified: 2008-02-26 18:11:42
SmartSite Maintenance: 2/26/08 05:30 - 07:00
We performed scheduled maintenance.
- Notification modified: 2008-02-26 09:44:37
MyUCDavis Saturday, February 23, 2008 00:00 - 06:00
The MyUCDavis portal maintenance has been postponed.
- Notification modified: 2008-02-22 15:13:54
Power to Campus Data Center, 6pm - 8:45pm, 2/14/08
At approximately 6PM, the UC Davis Campus experienced a prolonged power interruption which also affected the Campus Data Center. PG&E initially reported an Estimated Time of Repair of 2150hrs, however, power was restored at 2045hrs. The Data Center is equipped with automatically-activated generators in addition to UPS units, and these power-back-up systems operated to prevent any Data Center Service interruptions or outages. There were no Campus Data Center service outages in connection to this event.
- Notification modified: 2008-02-14 21:34:49
Power to Campus Data Center, 4pm, 2/13/08
At approximately 4PM, the UC Davis Campus Data Center experienced a momentary power interruption. The Data Center
is equipped with automatically-activated generators in addition to UPS units, and these power-back-up systems operated briefly to prevent any Data Center Service interruptions or outages. No service outages were
reported during this event.
- Notification modified: 2008-02-13 16:39:44
Smartsite Service Degradation 2/13/08 08:30-08:40
Smartsite users may have experienced slow response times. System administrators and database administrators have resolved the issue
- Notification modified: 2008-02-13 08:41:48
SmartSite Maintenance 02/12/08 - 05:30 to 07:00
A scheduled maintenance was performed.
- Notification modified: 2008-02-12 07:03:42
Blackberry outages 2/11/08 3:14pm - 6:55pm
Users of enterprise e-mail services for Research In Motion's BlackBerry devices experienced a disruption in service. The outage affected BlackBerry users located in North America. Reports indicated that the disruption was limited to the enterprise e-mail services, while the phone and messaging services were reportedly unaffected.
- Notification modified: 2008-02-11 18:59:59
Data center network outage 02/06/08 16:55 - 17:25
A network outage at the data center affected access to a majority of data center services. System administrators and the NOC resolved a firewall hardware issue and network access has been restored.
- Notification modified: 2008-02-06 17:38:59
MyTravel service intermitenly unavailable.
02/05/2008 14:35 - 15:00
MyTravel service was intermitenly unavailable. The problem fixed itself after a few minutes.
- Notification modified: 2008-02-05 15:24:06
All area 3 DSL and T1 connections are down.
2/05/2008 14:06
UPDATE:
Impact:
All area 3 DSL and T1 connected sites have no connectivity due to a switch hardware failure.
Locations Affected:
All serial sites and area 3 DSL connected sites.
Status:
The NOC has replaced the failed hardware and connectivity has been restored.
- Notification modified: 2008-02-05 16:04:42
casweb5 down - 2/4/08 14:38 - 14:58
casweb5 was having problems with CASAUTH: Connection failed 500 Timeout. Casweb5 was removed from the pool and replaced with casweb3.
- Notification modified: 2008-02-04 16:27:30
Maintenance for Smartsite
SmartSite was unavailable Saturday morning, February 2nd from 06:00 a.m. until 09:00 a.m. for a database upgrade.
- Notification modified: 2008-02-02 09:15:30
ms2 Downtime 1/29/08 10:30 - 10:35
After patching and rebooting the idle ms2 node (cyrus2-2), it came
back up and attempted to steal the virtual IP address for ms2 (the
line was not commented out the last time the nodes were switched).
This caused the active node to disable the ms2 interface temporarily.
- Notification modified: 2008-01-29 10:51:22
SmartSite Maintenance : January 29, 2008 : 05:30 to 07:00
We performed scheduled maintenance.
- Notification modified: 2008-01-29 10:52:34
Part of LDAP services unavailable 1/27/08 21:00 - 1/28/08 12:09
01/27/08 21:00 - Ongoing
Due to a misconfiguration of the montioring for the service, an internal LDAP directory server was offering an expired certificate. This affected other services such as the on-line UC Davis People search interface. The systems administrators acquired and installed a new certificate. The service is now available.
The general-use LDAP directory is unaffected by this outage.
- Notification modified: 2008-01-28 12:23:06
Usenet News Service Interruption 1/24/08 14:30 - 14:55
The Usenet news service was briefly unavailable. System Administrators have resolved the issue.
- Notification modified: 2008-01-24 15:12:21
Usenet News unavailable 01/22/2008 09:00 - 09:25
Usenet News service was unavailable for a brief moment. The problem fixed itself when a dislike article by our server seized to be sent by our peers at newsfeed.berkeley.edu.
- Notification modified: 2008-01-22 11:11:06
System Patching - Fri Jan 18, 2008 - 10PM - 8AM
Various services were unavailable during this period. See the Recurring Maintenance link to learn more.
- Notification modified: 2008-01-19 08:13:32
Campus Authentication - Fri. Jan 18, 2008 7PM- 7:10PM
A couple of the DistAuth servers were being patched simultaneously; this potentially caused a slight delay to some users who may have logged onto restricted campus services (Ie. Geckomail, MyUcdavis, etc.) which rely on our Distributed Authentication service. System Administrators quickly resolved the issue.
- Notification modified: 2008-01-18 19:18:42
Datacenter firewall reboot Friday Jan. 18, 2008 : 06:00 - 06:30
The Data Center firewall was rebooted in order to reinstall the latest firmware. Most Data Center services was unavailable for approximately 5 minutes during the reboot.
- Notification modified: 2008-01-18 10:53:43
MS13 unavailable: 01/16/08 2200 - 2239
MS13 was unavailable. Systems administrators resolved the system.
- Notification modified: 2008-01-16 23:46:49
SmartSite was unavailable
Tuesday, January 15, 2008 07:00 - 07:28
SmartSite was unavailable due to database connectivity problems after schedule downtime.
- Notification modified: 2008-01-15 07:36:09
SmartSite Maintenance : January 15, 2008 : 05:30 to 07:27
We performed scheduled maintenance on SmartSite on January 15, 2008 from 05:30 to 08:15. SmartSite encountered problems with database connectivity during this morning's maintenance. SmartSite was back up by 7:27, all maintenance was compeleted by 8:15.
- Notification modified: 2008-01-15 09:31:45
Campus Authentication is down 01/14/2008 11:11- 11:22
01/14/2008 11:11- 11:22
Campus Authentication was unavailable for a brief moment. System Administrators fixed the problem by restarting the KDC deamons. All back to normal.
- Notification modified: 2008-01-14 11:30:05
MS1 System Maintenance: 01/13/2008 22:00-22:10
System Administrators performed maintenance on ms1.ucdavis.edu. During that time, email was unavailable.
- Notification modified: 2008-01-14 15:54:48
MS1 System Maintenance 01/10/2008: 22:30 - 22:40
System Administrators performed maintenance on ms1.ucdavis.edu. During that time, email was unavailable.
- Notification modified: 2008-01-10 23:02:12
Network Outage Update 1/05/08 11:36
Below is list of sites that was without network connectivity due to lack of power. The buildings were restored to power on 01/07/2008 at 11:11.
ACF Hatchery
Castillian 1440
Castillian 1460
County Road 95
255 Cousteau Pl
202 Cousteau Pl
267 Cousteau Pl
279 Cousteau Pl
2795 2nd Street
607 Pena Drive
Soc Sci and HARC
UPDATED: 01/07/2008 11:11
UPDATED: 01/09/2008 09:13
TB 127 Power was restored 1/10/08
- Notification modified: 2008-01-10 08:54:48
Veni ssh Down 01/08/2008 12:30 - 12:57
Veni was not responding to ssh. Over 500 old ssh processes were running with only 5 logged in users. Veni had to be rebooted to clear ssh sessions. It is back up and functioning fine after 10 minutes.
- Notification modified: 2008-01-08 13:40:49
Data Center Firewall Reboot, Tue, Jan 8, 2008 0600 - 0630
The Data Center rebooted its firewall Tue, Jan 8, 2008, between 0600 and 0630 HRS. All network connectivity to all Data Center services was interrupted for approximately five minutes during the reboot. This was an effort to resolve the ongoing email disconnects.
- Notification modified: 2008-01-08 06:51:34
Data Center Connectivity Issues / LDAP Authentication
The intermittent Data Center connectivity issues affected access to services. Systems administrators focused on LDAP authentication or the DC firewall as potential sources of the trouble. Some changes to LDAP were made.
UPDATE: 1/10/08 - 10:30
The changes made to try to determine whether the load balancer is the problem suggested that it is not. The error rates have dropped off tremendously, though not entirely disappeared. There were very few reports of authentication problems yesterday. At the worst, only about 2% of the POP authentication attempts failed for any reason, and since the change made on Monday at 12:30, it's been under 0.5%. Work is still being done to resolve the issue completely.
- Notification modified: 2008-01-10 13:59:00
Network Outage Update 1/07/08 14:49 - 15:51
======================== Network Outage Notice ====================
01/07/2008 14:49
UPDATE:
Impact:
A failed UPS during a utility outage has caused the primary router in Area 10 to go down. All devices with redundant links have failed over to the backup path. The following locations in this area are not redundantly connected, and are without network services at this time.
Locations Affected:
All traffic outside the area for the following buildings:
SEROLOGY LAB
IT-Env Cntrl
WASTE WATER TR
VET HDQRS
PP LAB
CRC TR 1222-RA
IEHR CLN MED
CELL BIO LAB
IEHR OFC&LAB
LEUKEMIA LAB
ARS R1
TB 127
CTR EQ HLTH
IEHR RECVING
ARS TR 1537
VET VES EX B
ARS M3
IEHR CLN MED
VET META RES
Status:
The failed UPS has been replaced, and commercial power to the building has been restored. All connectivity should be back to normal as this time.
- Notification modified: 2008-01-07 15:56:26
SmartSite Maintenance : January 05, 2008 : 00:00 to 03:00
A scheduled maintenance was performed on SmartSite.
- Notification modified: 2008-01-05 03:05:33
SERVICE OUTAGE resolved: listproc.ucdavis.edu 1:10pm - 1:55pm, F
The UC Davis list server, listproc.ucdavis.edu, was non-responsive to network queries. System Administrators rebooted the system and the list server has begun delivering queued messages.
- Notification modified: 2008-01-04 14:17:53
Network Outage Update 1/05/08 11:36
Below is an updated list of sites without network connectivity due to lack of power.
Ongoing power outages in the following buildings/areas are currently preventing restoration of network services:
UPDATED: 01/07/2008 11:11
UPDATED: 01/09/2008 09:13
TB 127
- Notification modified: 2008-01-10 08:53:29
Multiple Campus Network Outages 1/4/08
The campus Network Operations Center (NOC) received multiple automated warnings of network connectivity loss throughout the campus. Area 1 (Mrak, Shields Library, King, etc.), Area 7 (MU, Young, Dutton, etc.), and Employee Health paged the NOC for connectivity loss. While not all buildings in these areas were affected, some buildings in these general areas may not have had network connectivity. The NOC corrected the problems.
- Notification modified: 2008-01-04 11:35:02
Campus Authentication Disruption 1/3/08 16:00-16:15
During an administrative change to a Campus Authentication system, authentication to campus resources became unavailable. The issue has been resolved, and all campus authentication should be functioning again.
- Notification modified: 2008-01-03 16:34:23
All Data Center Services - Jan 3, 2008 06:00 - 06:30
The Data Center performed routine maintenance activites on the Campus Data Center firewall. During the window, there may have been intermittent loss of connectivity to all Data Center services.
- Notification modified: 2008-01-03 06:42:15
Email service disruption MS11 - 5:40PM - 6:50PM, 29-Dec-07
A malfunction on campus email server MS11 was preventing the users assigned to this host from accessing their e-mail. System administrators remedied the cause of the problem.
- Notification modified: 2007-12-29 19:30:48
Internal DNS Server Disruption 12/28/07 12:00 - 12:15
An internal Data Center DNS server experienced a disruption in service. Certain systems that depend on it might have experienced degraded performance. System administrators rebooted the system and service has been restored.
- Notification modified: 2007-12-28 12:16:44
Outlook Address Book 12/26/07 13:40 - 21:00
Systems administrators have resolved an issue regarding the address book in Outlook.
- Notification modified: 2007-12-26 21:30:40
Banner Down - Tuesday 12/26 03:30 - 06:55
One of the Banner servers was non-responsive. System administrators have resolved the issue.
- Notification modified: 2007-12-26 06:59:12
Data Center Power Outage 12/21/07-09:40-10:50
Due to a power outage, several services at the Data Center were unavailable or experienced degraded performance. Affected services included listproc, DHCP, DNS, and MyUCDavis.
We resolved the issue and services were restored.
- Notification modified: 2007-12-21 11:00:53
Saturday, December 22, 2007 06:00 - 08:30
SmartSite was unavailable during a database upgrade.
- Notification modified: 2007-12-22 08:44:22
Data Center Generator Test 0630 - 0730 12/18/2007
Facilities conducted an end-to-end generator test for the Data Center from 6:30 to 7:30 AM on Tues 12/18/2007. This test of the Data Center generator included shutting power off to the building, having systems fail to UPS, and testing the startup and load on the generator. This was a test that had been completed in the past without a problem. No service interruptions were experienced during the generator test this year as well. If you have questions regarding the test, call IT Express at 752-HELP (4357).
- Notification modified: 2007-12-20 10:35:51
Banner down Friday 12/14 01:54 - 02:18
A Banner server was at a hung state and a System Administrator fixed a file system problem and rebooted the server.
- Notification modified: 2007-12-14 02:34:35
SmarSite outage: Thurs Dec 13, 2007 1835hrs - 2000hrs
Symptoms: Extremely slow response from all SmartSite sites resulting in web browser timeouts for many of the sites' users.
Cause & Resolution:
Preliminary data indicates that an automated maintenance process which runs at 6PM daily failed. A DBA checked the tables and then manually ran the process to restore functionality.
- Notification modified: 2007-12-13 20:21:22
Banner down Thursday 12/13 02:00 - 03:02
A Banner server was hung and the System Administrators rebooted it.
- Notification modified: 2007-12-13 03:19:29
Data Center Network Downtime: Friday, Dec 14, 0630-0700
The Data Center had a scheduled downtime for a network device reboot. The reboot took place during the 0630-0700 window and resulted in a lack of connectivity for about five minutes. All services that were affected are redundant and nothing more than slow response should have been noticeable.
- Notification modified: 2007-12-14 07:01:47
Listings is unavailable 09:15 - 10:15
Listings.ucdavis.edu was not working.
- Notification modified: 2007-12-11 14:56:18
Email Server ms2 unavailable 12/6/07 12:45 PM - 12:53 PM
Email Server ms2 was presently unavailable due to networking problems. Consequently, clients assigned to this email server ms2 were unable to access their email. System administrators address the networking problem and restored mail service.
- Notification modified: 2007-12-06 13:00:55
Xeda: Outlook email disruption: 12/5/07 07:45 - 08:15
One of the Exchange Cluster Servers for Xeda had experienced a resource issue briefly this morning. System Administrators manually moved the resources to the standby cluster node and restored services. The disruption of email services affected those whose mailboxes resided on this server.
- Notification modified: 2007-12-05 08:39:59
Blackberry Problems 12/04/07 08:15 - 13:00
The Blackberry server wasn't sending messages from handheld units. This problem was narrowed down to a permission issue in Active Directory. To resolve it, the router on the Blackberry Enterprise Server was taken offline and full service restored by 13:00.
- Notification modified: 2007-12-05 03:21:13
Mail server issues - Tue 12/4/07 16:02 - 17:44
No new email messages were being received by any of the central email delivery servers. All new messages were being queued on the SMTP servers and were delivered as soon as the issue with the delivery servers was resolved. The actual sending of email messages was not affected by this outage.
- Notification modified: 2007-12-05 02:48:20
Campus Network Slow Response. 12/04/07 15:05 - 15:15
Some network traffic to locations on and off campus
experienced brief period of packet loss and latency due to
a bridge loop.
Locations Affected:
Some data traffic to sites on and off campus
Status:
The NOC has identified the source of the loop and has
removed it from the network. At this time all traffic
should be passing normally.
- Notification modified: 2007-12-04 15:32:36
NetID failed on willits. 12/4/07 14.55
NetID was restarted by operations.
- Notification modified: 2007-12-04 14:59:37
Service maintenance - Mail System 2 - 12/03/07 22:00 - 22:10
Ms2 was taken down for maintenance at 10PM. System administrators tuned the Cyrus databases in order to improve performance on the system. During this time, email was unavailable.
- Notification modified: 2007-12-04 00:09:26
Service Degradation - Mail System 2 - 12/03/07 10:00 - 16:30
Clients who receive their email on ms2.ucdavis.edu experienced sluggish response and generally degraded email performance. System administrators resolved the issue.
- Notification modified: 2007-12-03 16:52:03
Service Degradation - Mail System 2 - 11/30/07 16:00
Clients who receive their email on ms2.ucdavis.edu experienced sluggish response and generally degraded email performance. System administrators moved additional clients to other ms servers to help alleviate the problem.
- Notification modified: 2007-12-01 12:13:28
Service Degradation - Mail System 2 - 11/29/07 13:00 - 1800
Clients who receive their email on ms2.ucdavis.edu endured sluggish responses and generally degraded email performance as a result of heavy traffic loads. System administrators moved 1000 users to other mail servers to alleviate the load and resolve the issue.
- Notification modified: 2007-11-29 18:41:05
ListProc was unavailable. Wed.11/28/07 11:45-12:20
Listproc, system crashed due to a problem with one of the disk. System administrators recovered the disk and restore service.
- Notification modified: 2007-11-28 12:59:08
Listproc Software Problems 11/27/07 15:50 - 16:02
Listproc Sofware experienced problems at 15:40. Administrators reconstructed the reverse database to fix the problem. List manipulation functions were offline for 14 minutes.
- Notification modified: 2007-11-27 16:26:37
Campus Authentication unavailable for 2 minutes. 11/20/07 14:50
11/20/2007 14:50
An administrative change to authentication severs blocked authentications from getting to them for about 30 seconds.
- Notification modified: 2007-11-20 15:04:31
MS2 Unavailable 11/18/2007 17:50 - 18:40
MS2 was unavailable, system administrators have been able to get it operational once again. The exact cause of the downtime is still under investigation.
- Notification modified: 2007-11-18 18:58:59
Intermittent Time-outs: www.ucdavis.edu 11/15/07 14:00 -16:00
We received reports that users periodically were getting time outs to the www.ucdavis.edu website. System administrator resolved the issue.
- Notification modified: 2007-11-15 16:44:45
DAFIS inaccessible 11/12/07 0000-0150
Data Center administrators found and fixed an internal firewall issue that affected DAFIS network access.
- Notification modified: 2007-11-12 02:09:50
Network switch reboot 11/11/07 23:30-23:35
A data center switch was rebooted at 23:30. Affected servers that were temporarily unavailable included various DAFIS and mail systems.
- Notification modified: 2007-11-11 23:36:28
email server ms1 unavailable- Sat. Nov. 10, 2007, 1815-1835
Users assigned to email server ms1 were unable to access their email for about 20 minutes. This was attributed to misbehavior by email server ms1, when it became unresponsive to network traffic from 6:15PM to 6:35PM on Saturday November 10, 2007 . The specific cause for the disruption is still under investigation. The same behavior was exhibited minutes before by another server, ms1, from 5:30PM to 5:50PM.
- Notification modified: 2007-11-10 19:25:57
email server ms2 unavailable- Sat. Nov. 10, 2007, 1730-1750
Users assigned to email server ms2 were unable to access their email for about 20 minutes. This was attributed to misbehavior by email server ms2, when it became unresponsive to network traffic from 5:30PM to 5:50PM on Saturday November 10, 2007 . The specific cause for the disruption is still under investigation. The same behavior was exhibited by ms1 shortly after; from 6:15PM to 6:35PM.
- Notification modified: 2007-11-10 19:24:00
Scheduled downtime for mail server ms1 11/05/07 22:00 - 22:15
Data Center system administrators rebooted mail server ms1 tonight at 10pm. Users with ms1 accounts was unable to check their email during the reboot.
Rebooted ms1, downtime 22:03 to 22:10.
- Notification modified: 2007-11-05 23:33:56
Slow Connectivity Issues with mail server 11/05/07 9:45
UPDATE: Data Center system administrators were monitoring ms1 while they prepared to migrate some ms1 accounts to other servers. ms1 was processing email; however, the load was higher than normal. The load did cause slow connections.
IT Express received 6 calls due to slow server response this morning.
We will send an update as more information is available.
If you have any questions or are experiencing any email problems, please call the ITX Express computing services help desk at 530-754- HELP (4357).
- Notification modified: 2007-11-05 23:32:36
Slow Connectivity Issues with mail server 11/05/07 9:45 -
At approximately 9:45 AM on 11/05/2007, the IT Express service desk received reports of slow connectivity issues to the campus mail server
- Notification modified: 2007-11-05 11:09:56
MS1 unavailable - Fri, Nov. 2, 22:00 - 22:10
Mailserver MS1 was unavailable while it underwent a hardware maintenance procedure to diagnose a suspect network card.
- Notification modified: 2007-11-02 22:43:45
Mail server ms1 11/02/07 12:00-12:36
Mail server ms1 was under heavy load. Mail service on this server was slow and user's were experiencing intermittent time-outs. System administrators resolved the issue.
- Notification modified: 2007-11-02 12:36:54
Web authentication was degraded 10/30/07 14:33- 14:38
CAS Web authentication was degraded from 14:33 to 14:38
- Notification modified: 2007-10-30 14:45:20
Course Management and MyUCDavis 0239-0338
Course Management and MyUCDavis were experiencing problems. Administrators identified the problem and restored system performance.
- Notification modified: 2007-10-30 03:44:58
DHCP & DNS; Thurs, Oct 25, 2007 19:00 - 20:26
Campus DNS and DHCP experienced degraded service due to excessive traffic from a few computers in ResNet. NOC personnel identified the IP addresses of the systems causing trouble and System administrators blocked access for those IPs and relieved the strain on the system.
- Notification modified: 2007-10-25 20:32:45
Course Management degradation ; Thur, Oct 25, 2007 19:30 - 2030
Course Management is experienced degraded performance due to the heavy load on the database server. Database administrators identified the hosts causing the heavy load and restored system performance.
- Notification modified: 2007-10-25 20:52:57
Thursday, October 25, 2007 15:00 - 15:40
Course Management experienced degraded performance due to heavy load on the database server. Database administrators identified the source of the load and relieved the situation.
- Notification modified: 2007-10-25 20:03:55
Slow Response Sending Mail- 10/24/07 11:13-- 11:28
The Incoming SMTP pool was experiencing heavy activity. Users experienced slow response when sending mail.
- Notification modified: 2007-10-24 11:29:48
DNS Slow Response 10/22/2007 15:00 - 15:30
Slow DNS resolution responses were causing moderate slowdowns on some Data Center services. Systems administrators had the problem resolved by 15:30.
- Notification modified: 2007-10-23 13:40:14
http://gradstudies.ucdavis.edu - Unavailable 7:30-8:30 10/20/07
The website http://gradstudies.ucdavis.edu/ was unavailable due to scheduled maintenance on the web server.
- Notification modified: 2007-10-20 08:29:51
Course management outage 20:55 - 21:30
Lingering AFS issues are affected access to the course management database. Everything is back to normal.
- Notification modified: 2007-10-17 21:36:51
Course Management and Sakai outage 10/17 1845 - 20:50
The AFS Fileserver holding Course Management and Sakai volumes crashed at approximately 18:45. Service was restored by 20:50.
- Notification modified: 2007-10-17 20:54:46
Authentication issues 10-16 16:40-17:30
System administrators investigated and found a KDC issue that was affecting new authentication to secure services. Everything is back to normal.
- Notification modified: 2007-10-16 17:32:43
Listproc Software Problems 10/15/07 12:00 pm - Ongoing
10/15/07 12:00 pm - 10/16/07 3:55
Due to software problems we were unable to approve new ListProc list creation requests. All requests were queued and were processed once that part of the service was restored. All lists that had not been automatically created were manually created by help desk staff. The problem was due to two corrupted application configuration files.
- Notification modified: 2007-10-16 15:58:23
New Listproc mailing lists problem: Oct. 9 - Oct. 12, 1800hrs
Service: Listproc, the central campus mailing list server.
Symptom: Newly created mailing lists returned an error message which stated "User Unknown".
Problem: The list server software failed to properly create mailing lists requested since October 9th, 2007. Lists created prior to 5pm on October 9th, 2007 functioned correctly.
Resolution: Programmers reviewed the logs and manually updated the affected lists. They also identified and corrected a corrupted data file, thus restoring proper functionality to the software.
- Notification modified: 2007-10-12 18:36:15
XEDA outage 10/10/07 15:46 - 17:10
Active directory and Exchange Services were impacted by a denial of service attack that lasted from 1524 HRS till 1710 HRS. During that time, clients only had intermittent connectivity to XEDA services. NOC and Data Center staff traced the problem to misbehaving software installed on some campus client computers. Those systems were blocked and XEDA services restored.
- Notification modified: 2007-10-10 21:43:56
Infrastructure Services Inpacted 10/09/07 13:31 - 13:46
"Infrastructure services that rely on AFS was impacted due to an AFS fileserver failure. These included Distauth authentication, Course Management/Sakai, MySpace, etc.
- Notification modified: 2007-10-09 13:51:02
Resolved: Network Outage: Campus: 2007-10-8-10:51
Monday, Oct 8, 2007 - 10:51
Description: A denial of service attack from off-campus hosts has caused elevated CPU levels on the border firewalls, causing intermittent and slow connectivity to some off campus sites.
Locations Affected: Network connections to off campus sites.
Status: At this time, all of the offending traffic has ceased and off site connections should be functioning normally.
Estimated time to repair: Repairs complete.
- Notification modified: 2007-10-08 11:29:11
Emergency Downtime: Mail Server Reboot 10/08/2007 09:15
10/08/2007 09:15 - 09:36
One of the mail servers was rebooted to remove a couple of hung processes. This impacted 2,000-2,500 users for a few minutes.
- Notification modified: 2007-10-08 10:03:24
Mail Server 1 - Thermal Outage 10/06/07 9:30-10:30am
The mail server storage array for MS1 overheated during a planned AC maintenance at the Data Center. System administrators have brought MS1 back online.
- Notification modified: 2007-10-06 10:31:43
Email delivery is being delayed-10/05/07 11:20- 16:00
MS1 experienced high load. Email was still accessible, but email delivery was being delayed.
- Notification modified: 2007-10-06 00:18:26
Mail Server 3 - Brief service degradation - 10/04/07 10:40-11:00
MS3 started experiencing connectivity issues at approximately 10:40am. Users may have had issues retrieving new messages, or logging in. The Cyrus service was restarted at 11:03 and is operating normally.
- Notification modified: 2007-10-05 08:31:20
SMARTSITE: Database Loaded Wed., Oct 3 1400-1410
SmartSite was down for approximately 10 minutes during the 2 o'clock hour when the SmartSite application and database servers were restarted. System administrators reported that SmartSite began performing satisfactorily after the restarts.
- Notification modified: 2007-10-04 06:24:33
SmartSite Service Notice 10/03/07 14:02 14:32
Due to extremely high system loads, the SmartSite application and database servers was restarted.
SmartSite was unavailable for approximately 30 minutes.
If you have any questions, please call the IT Express Computing Services Help Desk at 530-754-HELP (4357).
- Notification modified: 2007-10-03 14:55:48
SmartSite Available 10/02/07 14:59
SmartSite database administrators have implemented a number of changes to the database structure and feel that the load issue experienced by SmartSite this afternoon should be resolved.
If you continue to experience slow response from SmartSite please call IT Express 530-754-4357.
- Notification modified: 2007-10-02 16:44:11
SmartSite is Unavailable-High Load 10/02/07 12:55 -14:59
SmartSite was unavailable due to database load issues.
- Notification modified: 2007-10-02 15:01:52
Network Degredation: Campus: 2007-09-28 13:05-13:30
High CPU levels on the border routers were causing slowness and intermittent connectivity loss to some sites outside of campus.
The reason for high CPU level on the ISP firewall was found and Network Operation Administrators have resolved the issue.
- Notification modified: 2007-09-28 13:51:15
Campus Central Mail Server MS1 09/28/07 12:05-12:15
We suffered a brief interruption where any user on ms1 could not login.
This has been corrected and users should no longer have any issues.
- Notification modified: 2007-09-28 12:22:26
Distauth Problems. Thurs. Sept. 27, 19:00 - 20:15
Data Center network and system administrators isolated the distauth problem to the KDC servers. The predominant symptoms were the extensive wait required for distauth to respond to client requests for authentication, and finally no responses during the time which both servers were restarted (which restored service at approximately 8:15 PM). The root cause of the problem is still unknown. System administrators are following-up with the software vendor's technical support.
- Notification modified: 2007-09-27 21:11:25
Email Slowness Thur. Sept. 27, 17:30 - 20:00
Data Center system administrators observed high loads on email
server ms2. IT Express did not receive any calls from users about
slowness or problems. Otherwise, mail was functioning normally.
- Notification modified: 2007-09-27 20:43:19
Smartsite Service Degredation 9/27 - 14:00 - 18:00
SmartSite experienced brief moments of excessive load. Some users may have experienced slow response times. The system administrators worked to resolve the issue.
- Notification modified: 2007-09-27 18:48:14
MS6 Mail Down
MS6 hung about 12:10, Vincent Fox noticed that one of the mail servers stopped responding. He consoled in rebooted the server and everything was back to normal as of 12:19.
- Notification modified: 2007-09-26 08:31:43
SERVICE OUTAGE: SMARTSITE: Login Problem 09/26/07
09/26/07 07:00 - 08:16
The SmartSite problem that prevented logins has been resolved. The problem was due to a database problem that affected a table where user account information is stored.
- Notification modified: 2007-09-26 09:09:22
MS6 Unavailable 09/26/2007 12:10am - 12:19am
MS6 was unavailable temporarily.
- Notification modified: 2007-09-26 00:31:06
SERVICE OUTAGE: Banner SISWeb 4:15pm - 6:15pm , 9/27/07
After restoring the root AFS cell, Data Center Administrators discovered additional issues with the Banner/SISWeb service. Banner/SISWeb was restored at about 6:15pm.
- Notification modified: 2007-09-24 19:19:12
Distributed authentication outage 09/24/07 1630 - 1730
Data Center System Administrators investigated an outage to the distributed authentication service. The AFS root cell was unavailable from 4:30pm until 5:00pm. Most services dependent on AFS, eg. Secureweb, DaFIS, MyUCDavis, & Banner, were restored by 5:30pm.
- Notification modified: 2007-09-24 19:24:18
SmartSite Application Problem 09/22/07- 10:00AM to 6:00PM
SMARTSITE DATABASE PROBLEM
Due to a problem with a SmartSite database that contains materialized
views of the Banner database, some users of SmartSite were unable to access
course sites for classes in which they were enrolled. We do not know how
many students and faculty were affected by the problem.
SMARTSITE URGENT MAINTENANCE FROM 5:00 PM to 6:00 PM, Saturday, September
22, 2007
SmartSite programmers and database administrators shut down SmartSite
at 5:00 PM to fix the database problem. When SmartSite was shut down,
all users who had been logged into SmartSite lost their connections. A
message was posted on the login page to warn users of the urgent
maintenance. An email message was sent to all of those people who had
logged in before the message on the login page was posted. The maintenance
was expected to (and did) take 1 hour.
Updates were to be sent to technotices@ucdavis.edu and posted to
http://status.ucdavis.edu/ as they became available. If you have any
questions, please call IT Express at 530-754-HELP (4357). IT Express phone
support was available from 10:00 AM to 6:00 PM on this particular Saturday and Sunday.
IT Express SmartSite Support Team
530-754-HELP (754-4357)
- Notification modified: 2007-09-22 18:10:47
Saturday, September 22, 2007 00:00 - 05:00
SmartSite was not available due to a scheduled maintenance.
- Notification modified: 2007-09-22 06:03:40
Campus Central Mail Servers Message removed on 9/24/07 13:32
The email servers ms1.ucdavis.edu and ms2.ucdavis.edu experienced heavy load since Friday morning.
Data Center system administrators migrated user accounts off of ms1 and ms2 in order to reduce the load.
These systems were monitored very closely by Data Center system administrators.
- Notification modified: 2007-09-24 13:33:41
Central Email Server Maintenance - Thurs. Sept. 20, 22:00-22:55
At 10:00PM ~ 10:55PM, maintenance were performed on the campus' central email servers. Each server was unavailable for approximately 10 minutes for a software update. During that time, there was no Geckomail access nor downloads with an email client. Emails weren't lost during this maintenance.
- Notification modified: 2007-09-20 23:24:39
Load Balancer 09/19/2007 6:30am
"All load balanced data center services were unavailable for a 5 minute period around 0630 9/19/2007 for a software upgrade." All went well.
- Notification modified: 2007-09-19 11:14:21
Campus Central Mail Servers- Sat. Sept. 15, 2007
All of the campus' central email servers are functioning normally and have been continuously available to retrieve messages since Friday evening. No email was lost at any time during the problems this past week.
Some people using POP to retrieve their email have reported duplicate downloads and other similar problems. If you are experiencing this kind of problem, please call the IT Express Computing Services Help Desk at 530-754-HELP (4357) between 7:00 AM and 9:00 PM, Monday through Friday.
If you have any questions about the status of the email system or you experience any problems with it, please call the IT Express Computing Services Help Desk at 530-754-HELP (4357) between 7:00 AM and 9:00 PM, Monday through Friday.
- Notification modified: 2007-09-20 16:32:54
Campus Central Mail Servers 9/17/07 13:45 - 23:15
Email sent to accounts on ms1 was deferred for about an hour while one of the Cyrus databases check-pointed. By 1 PM, email was again being accepted by ms1. During the time delivery was deferred, clients were able to connect to retrieve messages that had already been delivered. IT Express reported that a few clients called about slowness on ms1 but there are no indications of client connectivity problems.
With the exception of the period noted above, all of the campus'
central email servers have been functioning normally since Friday evening. All email that had been deferred and queued for delivery on Friday morning and afternoon was delivered Friday evening. No email was lost at any time during the problems this past week. Data Center system administrators are continuing to monitor the service very carefully.
Due to some system configuration changes, some people using POP to download their messages will experience a "double download" the first time they connect to the server. This should be a one-time event. We apologize for the inconvenience caused by the "double download" and for all the difficulties caused by last week's email service outages.
If you have any questions about the status of the email system or you experience any problems with it, please call the IT Express Computing Services Help Desk at 530-754-HELP (4357) between 7:00 AM and 9:00 PM, Monday through Friday.
- Notification modified: 2007-09-17 23:17:57
Campus Central Mail Servers - 9/17/07 10:33 - 13:45
All of the campus' central email servers have been functioning normally since Friday evening. All email that had been deferred and queued for delivery on Friday morning and afternoon was delivered Friday evening. No email was lost at any time during the problems this past week. Data Center system administrators are continuing to monitor the service very carefully.
Due to some system configuration changes, some people using POP to download their messages will experience a "double download" the first time they connect to the server. This should be a one-time event. We apologize for the inconvenience caused by the "double download" and for all the difficulties caused by last week's email service outages.
If you have any questions about the status of the email system or you experience any problems with it, please call the IT Express Computing Services Help Desk at 530-754-HELP (4357) between 7:00 AM and 9:00 PM, Monday through Friday.
- Notification modified: 2007-09-17 14:11:49
Campus Central Email Service 9/16/07 22:21 - 9/17/07 10:33
The email server ms1.ucdavis.edu has been in normal operations mode since Friday evening. It has been processing email with no indications of any problems. All email queued for delivery on Friday was delivered Friday night. No email was lost at any time during the problems this past week.
In order to reduce the load on the two main email servers (ms1 and ms2), Data Center system administrators have been installing and configuring new email servers and moving email accounts from ms1 and ms2 to the new servers. This activity has been going on around the clock throughout the weekend. No changes are required from clients whose accounts have been moved.
Data Center system administrators are continuing to consult with Sun Microsystems and the email application's primary developer (from Carnegie Mellon University) to pinpoint the root cause of ms1's sudden catastrophic failure. At this time, we believe the problem may be due to a limitation caused by how the operating system's kernel manages multi-processor global memory consistency. We anticipate having more information from Sun on Monday.
If you have any questions about the status of the email system or you experience any problems with it, please call IT Express Computing Services Help Desk at 530-754-HELP (4357) between 7:00 AM and 9:00 PM, Monday through Friday.
We sincerely apologize for the inconvenience and difficulties caused by this week's email service outages.
- Notification modified: 2007-09-17 10:54:25
Campus Central Email Service
The campus central email server ms1.ucdavis.edu is continuing to experience problems. The system is running and accepting IMAP and POP connections; however, it is not accepting messages from the SMTP servers. Email messages sent to accounts on ms1.ucdavis.edu have been queuing on the SMTP servers since approximately 8:30 AM Friday. Anybody who retrieved email during the afternoon and early evening on Friday would have received only those messages that had been sent prior to 8:30 AM (with very few exeptions). Later in the evening on Friday, mail will again be accepted for delivery. No mail has been lost.
System administrators stopped accepting mail on this server in order to move accounts from it to other servers. As of 5 PM, two additional email servers have been put into service (ms3 and ms4). Throughout the weekend system administrators will continue moving accounts off of ms1 and ms2 to the two new servers with the goal of reducing the number of accounts on ms1 and ms2. Other system adminsitrators are building and preparing two more systems that will be ms5 and ms6. The goal is to distribute accounts across the six servers to reduce the load on all the servers.
Further updates will be posted as we have more information.
- Notification modified: 2007-09-16 22:30:22
Degraded Mail Server Performance 9/13/07 8AM - 1845 PM
One of the campus central email servers experienced periods of severe performance degradation during which times it refused connections. These disruptions were intermittent but the cumulative result caused mail queues to become backlogged.
UCD Data Center administrators consulted with Cyrus administrators from the University of Michigan and UC Berkeley and conferred with the primary Cyrus developer and the primary email administrator from Carnegie Mellon University.
It was determined that the database thresholds had been reached with our current user population. On their recommendations we recompiled the Cyrus Software and the Berkely DB software with settings that will address these limitations.
The issue was resolved and ms1 was returned to service by 6:45pm.
- Notification modified: 2007-09-13 19:15:45
Degraded Mail Server Performance 9/12/07 8:03 - 15:17
The campus central email server ms1.ucdavis.edu was continuing to experience brief periods of severe performance degradation during which times it refused IMAP and POP connections. This performance degradation was intermittent but the cumulative result of the problems was causing mail queues to back up.
[TSP-INFO:5819]
- Notification modified: 2007-09-12 19:07:56
Degraded Mail Server Performance 9/11/07 9:48 - 7:54
The campus central email server ms1.ucdavis.edu was continuing to experience brief periods of severe performance degradation during which times it refused IMAP and POP connections. This performance degradation was intermittent but the cumulative result of the problems was causing mail queues to back up.
- Notification modified: 2007-09-12 08:04:14
Degraded Mail Server Performance 9/10/07 13:15 - 22:15
Clients experienced degraded performance from the campus mail servers. Data Center Administrators noticed a significant decrease in the load on the ms1.ucdavis.edu mail server since 7pm but the load was still above average. Administrators upgraded hardware on ms1.ucdavis.edu at 10pm in hopes of further accelerating the processing of backlogged messages. The hardware upgrade resulted in a 20 minute window in which all users on the ms1.ucdavis.edu server were unable to access their emails.
- Notification modified: 2007-09-10 23:56:01
BANNER DOWN 09/01/07 07:00 am-07:45
Banner was down to install memory.
- Notification modified: 2007-09-01 08:00:03
SmartSite Experienced Issues 8/28/07 7:00-8:42
SmartSite experienced issues following maintenance.
2 of the 3 application servers did not return from maintenance properly.
SmartSite was available with possibly degraded performance.
- Notification modified: 2007-08-28 08:56:16
SmartSite Production Maintenance : August 21, 07 : 06:00 to 0700
We performed scheduled maintenance on SmartSite on August 21, 2007 from 06:00 to 07:00. For more information, please see:
https://confluence.ucdavis.edu:8443/confluence/x/URM
--
Thomas Amsler
Programmer
IET Mediaworks, UC Davis
- Notification modified: 2007-08-21 07:10:59
MS1 Experienced a Momentary Outage - 08/20/2007 16:05 - 16:30
MS1 experienced a momentary outage while the service was restarted. During this time, email was unavailable.
- Notification modified: 2007-08-20 16:30:03
Campus email server ms1.ucdavis.edu 8/20/07 13:28 -15:16
Symptoms: Some email messages were being queued instead of delivered immediately. Queues were being processed, but email might have been taking longer than usual to deliver.
- Notification modified: 2007-08-20 15:24:56
Campus email server ms1.ucdavis.edu 8/20/07 8:20 -8:20 - 13:28
Symptoms: On an intermittent basis new connections to the email server ms1.ucdavis.edu were unable to connect and previously connected sessions were dropped. Previous connections were not be able to re-connect.
- Notification modified: 2007-08-20 14:27:05
SmartSite Unavailable 10:00 pm Fri, 8/17/07 - noon Sat, 8/18/07
In preparation for fall 2007, SmartSite will be upgraded to fix several outstanding issues and add functionality to existing tools, particularly the Assignment and Gradebook tools. The window of server downtime also includes recurring monthly maintenance to campus systems on which SmartSite depends. During this entire window, the campus community will not be able to access sites, data, tools or features within the SmartSite system. The upgrade window will begin at 10:00 p.m. Friday, August 17 and finish no later than noon on Saturday, August 18.
For additional information about the downtime, please visit TechNews or contact us at smartsite-help@ucdavis.edu or (530)754-HELP.
- Notification modified: 2007-08-18 11:57:34
MS2 Mail Server Restart, 08/16/07, 12:05 -12:10
The mail service on ms2.ucdavis.edu was restarted at 12:05. Clients experienced a short disconnect while the mail service restarted.
- Notification modified: 2007-08-16 12:26:15
HAUS, Momentary Disruption - 08/15/07 12:25-12:35
There will be a momentary disruption to the HAUS home directories in order to restore home directory mappings. Home directories will be unavailable for approximately 5-10 minutes. This will also affect access to mailboxes on mailbox.ucdavis.edu.
- Notification modified: 2007-08-15 12:30:36
- Notification modified: 2007-08-15 11:33:09
ISUN Services, Momentary Disruption - 08/15/07 11:15-11:25
There will be a momentary disruption to the ISUN service in order to restore home directory mappings. Home directories will be unavailable for approximately 5-10 minutes. This will also affect access to mailboxes on mailbox.ucdavis.edu.
- Notification modified: 2007-08-15 11:32:44
Authentication Service Outage 14:55-17:15 Tue 8/14
The MyUCDavis, SmartSite and Web email services were inaccessible via the network. System and network administrators resolved this issue.
- Notification modified: 2007-08-14 17:32:23
SmartSite is available - Tuesday 8/14/07 7:00 - 9:25
SmartSite was available, but one of its application servers was experiencing trouble. The firewall would not unload modules and restart. System was rebooted. Ok now.
- Notification modified: 2007-08-14 09:29:17
Computing Accounts Service Degradation - Aug 9, 14:20 - 17:30
The computing accounts website experienced service degradation. Symptoms include users having the site freeze when trying to make account modifications. System administrators found the cause of the service disruption to be a DOS attack on the Data Center firewall.
- Notification modified: 2007-08-09 17:30:03
SmartSite Performance Problems, Aug 9, 09:05 - 15:10
SmartSite experienced performance problems. The problem was caused by a database issue. Users may have experiencing long login times after providing Kerberos Name and password. Slow access times to the site were also a symptom.
The issue was resolved at 15:10
- Notification modified: 2007-08-09 15:10:14
Mail Proxy Servers Unresponsive 8/8/07, 16:15 - 16:25
The mail proxy servers were unresponsive for 10 minutes. System administrators resolved the issue caused by the load balancer.
- Notification modified: 2007-08-08 16:32:29
Listproc Service Down - 8/8/07 10:50 am
The listproc service had stopped sending out mail. The service was down from 10:50-11:25. The cause was due to a misconfiguration with the system file. Any mail sent to the list at that time should have been bounced back to the sender and was not queued up to be sent to the recipients.
- Notification modified: 2007-08-08 14:02:21
Degraded operation of Secureweb 10:31- 16:17 Aug 4, 2007
We experienced sporadic slowness for applications that authenticated through SecureWeb. This included Wireless login page and My and some other services. IET services were working fine, but there was an offsite dependency for SiteSeal that was not.
This was not under direct control of campus personnel but was reported to the vendor and resolved shortly.
- Notification modified: 2007-08-07 17:31:54
Degraded operation of Secureweb 14:00-15:30 Aug 3, 2007
We experiencing sporadic slowness for applications that authenticate through SecureWeb. This includes Wireless login page and My and some other services. IET services were working fine, but there is an offsite dependency for SiteSeal that was not.
This was not under direct control of campus personnel but was reported to the vendor resolved shortly.
- Notification modified: 2007-08-03 15:49:12
Intermittent slowness to secure-web 8/1/07 14:25 - 16:07
There was an intermittent slowness to secure-web services such as wireless. System Administrators resolved the issue.
- Notification modified: 2007-08-02 00:32:16
AFS Service Degraded 8/1/07 10:23 -11:04
The AFS service was experiencing a degradation of service. Several of the read/write volumes was unavailable.
Also: The XEDA and MYINFOVAULT websites were unavailable due to issue with the AFS service.
- Notification modified: 2007-08-01 11:06:00
Distauth, CASWEB, and other services down - Aug 1, 01:00 - 02:45
Distauth, CASWEB, the campus website and other potential services were experiencing difficulties which left them slow or unreachable. This effected access to services such as geckomail and myucdavis.
- Notification modified: 2007-08-01 02:49:18
Listproc maintenance; Fri. Jul 27, 2245hrs - 0000hrs
The Electronic Listprocessor (listproc) underwent maintenance. No messages were lost from the electronic mailing lists and all queued messages were delivered once the maintenance completed.
- Notification modified: 2007-07-28 00:03:57
CAS Authentication outage Thu. 7/26/2007 16:40PM - 1710
CAS authentication was down from 1640-1710 due to a hardware configuration issue; This affected services such as: smartsite, jira, remedy-web-access; and potentially other services.
- Notification modified: 2007-07-26 17:11:15
Computingaccounts issues Wed 7/25 02:40-current
intermittent problems with computing accounts.
- Notification modified: 2007-08-01 08:02:17
Investigated account and password issues Wed 7/25 00:30-02:00
We investigated possible issues with new accounts and password changes. System administrators were unable to find any problems.
- Notification modified: 2007-07-25 02:00:41
MS1 authentication issues. 7/21/07 - 10:45 - 7/22/07 - 10:56 am
From 11:45PM, July 21 - 10:45AM, July 22 MS1 was experiencing authentication issues. During this time, some clients were not able to access their email.
- Notification modified: 2007-07-22 12:26:54
Listproc command line interface unavailable 07/21/07 08:00-09:45
Listproc command line interface was unavailable from 08:00 to 09:45. System administrators resolved the issues.
- Notification modified: 2007-07-21 09:52:35
SmartSite:Login not allow-07-20-07 23:57-07/21/07 07:53
After system patching 07/20/07 23:57, SmartSite experienced difficulty processing logins via distauth.
The problem was resolved after restarting Sakai on each application node.
- Notification modified: 2007-07-21 11:32:57
Campus authentication system became unavailable.
Friday July 20, 11:22p.m. - Saturday July 21, 2:30a.m. The campus authentication system became unavailable. System administrators resolved the issue.
- Notification modified: 2007-07-21 05:11:49
Firewall settings on the Cyrus servers prevented email delivery.
Friday, July 20, 4:30 pm - The firewall settings on the Cyrus servers were preventing some messages originating from the Cyrus servers to be delivered. System Administrators have resolved this issue. However, approximately 57,000 old queued messages (less than 5% of the average daily email volume) were delivered.
- Notification modified: 2007-07-21 02:29:09
Friday, July 20,2007 02:03 - 11:12
Mail server ms1.ucdavis.edu experienced trouble with mail delivery. The mail server was deferring some connections, causing mail destined for accounts on the host to be delayed. System administrators resolved the issue. Clients may have experienced slow server response while the large number of deferred messages were delivered.
- Notification modified: 2007-07-20 15:27:42
Oracle Names Service 7/20/07 - 11:15am
The Oracle Names Service was experiencing problems, preventing people from logging into DaFIS TP. Some other database connections were affected. The service was down for about 15 minutes (11:30am).
- Notification modified: 2007-07-20 12:53:10
computingaccounts services - Thur July 19, 1610-1625
The computingaccounts website produced an error message whenever user data was submitted. System administrators resolved the problem.
- Notification modified: 2007-07-19 16:34:57
Emergency Downtime for Cyrus Mail Server
07/16/2007
11:00 PM - 11:02 PM
Emergency system maintenance was performed on ms2.ucdavis.edu (Cyrus mail server) at 11:00PM. The system was unavailable for approximately 2 minutes during maintenance. If you use ms2.ucdavis.edu as your mail server, you were not able to access your email during that time.
- Notification modified: 2007-07-17 01:23:44
Wednesday, July 11, 2007 15:37 - Thursday, July 12, 2007 12:12
A server configuration issue prevented IPOP accounts from being created and password changes from taking effect. IKRB accounts and password changes were not affected. System administrators have resolved the issue.
Transactions which were delayed during this outage should now have taken effect.
- Notification modified: 2007-07-12 16:44:09
SISDOCS SYSTEMS ARE UNAVIALABLE
SISDOCS systems were unavailable between 22:00 7/12/07 through 08:30 7/13/07. This was due to a networking switch issue. The issue has been resolved, and SISDOCs are available again.
- Notification modified: 2007-07-12 08:56:15
AD domain services unavailable Wed 7/11 23:00 - Thu 7/12 02:45
AD domain services were temporarily unavailable due to a network equipment failure.
- Notification modified: 2007-07-12 02:50:48
MS1 Deferring Delivery of E-mail 7/9/07 13:59 - 14:05
MS1 was deferring delivery of email.
- Notification modified: 2007-07-09 14:10:29
Webcast and Podcast are Down 7/2/07 7:30 -7:40
Webcast and podcast were down, cleared on its own.
- Notification modified: 2007-07-02 07:43:43
UC Davis Website - Degraded Service - 06/28/07 11:50- 12:40
The UC Davis website experienced service disruptions. System Administrators resolved the issue.
- Notification modified: 2007-06-28 13:34:40
Exchange Servers Temporary Downtime - 06/26/07 20:00 - 20:10
Exchange servers experienced momentary downtime for 10 minutes due to maintenance.
- Notification modified: 2007-06-26 20:08:53
SmartSite Experiencing Problems 6/26/07 14:23
SmartSite was experiencing difficulty processing logins due to trouble with the database server. System administrators have resolved the issue.
- Notification modified: 2007-06-26 19:23:05
SmartSite Down 6/26/07 10:00-01:03
SmartSite was down due to database problems.
- Notification modified: 2007-06-26 13:13:01
Downtime email server, AZURE-Sunday, June 24 10:00pm to 1:00am
AZURE DOWNTIME
The email server, azure.ucdavis.edu, was unavailable from 10 p.m.
to 1:00am on Sunday, June 24. The downtime of azure was required to make a Data Center infrastructure change.
IMPACT
Account holders whose mail is delivered to azure were not able to
receive or read mail from 10 p.m. to 1:00am on Sunday, June 24.
All incoming messages sent to azure accounts during the downtime were delivered after azure was back online. Geckomail users were not able to send, receive, or read mail during the downtime window.
- Notification modified: 2007-06-25 01:06:05
Banner Unaccessible - 06/23 22:46 - 6/24/07 9:55am
Access to banner was unavailable. There was an issue with hard token authorization.
- Notification modified: 2007-06-24 10:03:29
Kerberos Upgrade 06/23/07 18:00-21:00
All web authentication and modem authenication were down while kerberos systems are upgraded. MyUCDavis, Computing Accounts, and Smartsite were also unavailable during the upgrade.
- Notification modified: 2007-06-23 21:29:06
System Upgrade: SmartSite Unavailable June 23-24
In preparation for the fall campus launch of SmartSite, the developers must upgrade the entire SmartSite environment to the latest version. A significant window of server downtime is required, during which the campus community will not be able to access sites, data, tools or features within the SmartSite system. The upgrade window will begin at midnight of Friday, June 22 and end no later than noon on Sunday, June 24. For more information about the upgrade, please read the complete announcement at TechNews.
Smartsite upgrades were finsihed at ~07:30am
- Notification modified: 2007-06-23 18:36:34
XEDA Exchange System Problems 08:55-10:15
The XEDA Exchange Service experienced performance degradation for some users. System administrators restarted specific services. Therefore some clients may have experienced connectivity issues between 10:05 - 10:15 AM.
System administrators were able to resolve the problem. No loss of data occurred.
- Notification modified: 2007-06-21 10:32:54
XEDA Exchange System Problem 13:57 - 15:15
One of the two exchange servers became unavailable at approximately 1:57 this afternoon. System administrators were able to resolve the problem. No loss of data occurred.
- Notification modified: 2007-06-20 15:21:55
CAMPUS POWER OUTAGE June 14 10:50 - 12:35
Description: An unscheduled power outage occurred and affected multiple buildings on campus.
Data Center went to generator power.
Locations Affected: Kemper, Storer Hall, Leach complex, Briggs, ARC, ARC Pavilion, A&E, Meyer, VMTH, Bainer, Surege2, Surge3, Engineering3, Data Center.
Power was restored. Data Center back on house power.
- Notification modified: 2007-06-14 13:13:33
AFS proxy servers - high load - June 6, 11:20am - June 11, 14:18
The AFS proxy servers were experiencing high load. System Administrators worked on balancing the load and increasing capacity for this service.
- Notification modified: 2007-06-11 14:22:21
AFS proxy server maintenance Tue 6/5 17:15-17:45
AFS proxy server sidney was undergoing hardware maintenance for approximately 30 minutes. It was temporarily removed from the pool.
- Notification modified: 2007-06-05 18:01:18
Voice Network Connectivity -Wednesday, May 30, 2007 11:00-14:10
Description: Voice Network connectivity has been interrupted. Calls to and from campus are limited. Data services for off campus T-1s are down.
Locations Affected: All voice services on campus. Data network connections are down for all off campus T-1 services.
Status: The problem has been isolated to the AT&T network. AT&T has corrected the problem and services have been restored.
Estimated Time to Repair: Repair complete – all services restored.
Updated information pertaining to this network outage will be posted at the IET Systems/Services Outage web page at http://status.ucdavis.edu/ and at the Network Operations Center homepage at http://www-noc.ucdavis.edu/.
For questions or reports of similar connectivity issues please contact the NOC at 752-7656 or via e-mail at noc@ucdavis.edu.
- Notification modified: 2007-05-30 14:17:09
SmartSite Unavailable-Friday , May 25, - Saturday,May 26
Friday, May 25, 2007 23:00 - Saturday, May 26, 2007 06:00,
SmartSite was unavailable for a planned maintenance.
- Notification modified: 2007-05-26 06:11:13
Downtime for Salmon, 10:00 p.m. Friday, 5/25, to 1:00 a.m. Sat.
Continuing issues with Salmon's disks required system downtime to resolve. The server was actually offline between 10:00 p.m. and 12:20 a.m.
- Notification modified: 2007-05-26 01:57:53
Downtime for Salmon, 10:00 p.m. to 11:00 p.m. Tue 5/22
Issues with Salmon's disks required system downtime to resolve. The server was off line between 10:00 p.m. and 11:00 p.m.
- Notification modified: 2007-05-22 23:28:02
Salmon (pop30) and Ivory (pop29) outage
At 4am, the Salmon (pop30) and Ivory (pop29) were unavailable due problems that developed from patching.
System Administrators have been able to restore Ivory (pop29) mail services as of 08:45. Salmon (pop30) mail services were restored as of 09:37.
- Notification modified: 2007-05-20 10:10:27
Temporary disruptions to E-mail and other services this evening
This is normal for this monthly maintenance activity.
For more information, see the "Recurring Maintenance" tab at right.
- Notification modified: 2007-05-20 01:31:31
DaFIS Proxy Services
The DaFIS AFS-proxy service has been restored to normal service.
- Notification modified: 2007-05-21 14:58:01
Isun Degraded Performance 05/16/07 12:25pm - 12:40pm
Isun experienced degraded performance for ~15 minutes. System administrators resolved the issue.
- Notification modified: 2007-05-16 12:43:07
Confluence was Unavailable 05/16/2007 07:00am - 01:10pm
Confluence website was unavailable. System Administrators have resolved the issue.
- Notification modified: 2007-05-16 13:14:51
Exchange service interruption - Sat.May 12, 9PM - 12AM
The exchange servers are being patched this evening and the disks hosting the IET mailboxes is going to be defragmented.
===========================================================
The IET mailboxes are finished being defragmented and are now available. 5/13, 2:40AM
- Notification modified: 2007-05-13 02:42:41
Decision Support Down from 4/30/07 9:21 - 4/30/07 10:00
Decision Support was down due to problems with Cold Fusion.
- Notification modified: 2007-04-30 10:57:05
ISUN Servers Unavailable 4/27/07 -6:00pm-6:25pm
The ISUN servers were unavailable. System administrators were able to bring the ISUN servers back online.
- Notification modified: 2007-04-27 18:26:03
IET EXchange Conversion 04/20/07 19:00 - 04/23/07 8:00
The IET Exchange Conversion completed this weekend.
- Notification modified: 2007-04-23 14:32:42
ISUN Down 4/23/07 14:06 - 14:18
The ISUN login servers were unavailable due to a network issue.
System administrators corrected this problem and the servers are now available.
- Notification modified: 2007-04-23 14:46:04
Scheduled Monthly Maintenance 4/21/07
Certain servers are undergoing their scheduled monthly maintenance tonight. Due to this, some services may experience brief interruptions. Please check the reccuring maintenace page for more specific information.
- Notification modified: 2007-04-22 05:22:41
ONGOING: Campus Wireless - Thursdays 06:30-07:00
Start of Window : 06:30
End of Window : 07:00
(all times are in 24 hour format)
Purpose:
Rebooting wireless gateways to avoid issues with users not being redirected
Impact:
No UCDNet/Internet access while gateways reboot. Currently attached users will need to re-authenticate.
Affected:
All Campus Wireless Networks
*** This will be an ongoing weekly reboot pending trouble isolation and resolution ***
- Notification modified: 2007-05-17 07:24:08
- Notification modified: 2007-04-19 07:13:36
Wireless Scheduled Maintenance. 06:30-07:10
Wireless network was down due to scheduled maintenance.
- Notification modified: 2007-04-19 07:14:30
Blackberry services down April 20th 5pm - 23th 7am
Blackberry services for IET will be disabled from Friday, 20th at 5 PM until Monday, 23rd, at 7 AM
- Notification modified: 2007-04-23 14:23:51
Degraged BlackBerry Services 4/18, ~6:00am-8:30am
Blackberry users are now able to send and receive messages as RIM has resolved their service issue. Messages may have beeen delayed due to the message back log. All messages sent during the outage were back logged and are starting to be delivered. The impacted Blackberry services were Provisioning, messages sent and received via Web, XML, and Blackberry Devices, Internet Service connections and Pin to Pin messaging.
- Notification modified: 2007-04-18 17:24:53
Degragaded BlackBerry Services 4/17/07 20:00 - 4/18/07 8:30
Tuesday 04/17/2007 20:00 - Ongoing
A service interruption occurred Tuesday night that affected BlackBerry services in North America. Email delivery was delayed or intermittent during this service interruption. However, the phone service on BlackBerry handsets were unaffected.
Problem has been resolved. E-mail is backlogged but passing through.
- Notification modified: 2007-04-18 08:39:14
BlackBerry service was spotty Tuesday 04/17
Due to RIM network issues: 22:30 - 00:30
- Notification modified: 2007-04-18 05:30:04
CAS auth problems Tues April 17, 20007 1820-1830
CAS was temporarily down. Sys admins resolved the issue.
- Notification modified: 2007-04-17 22:54:15
Slow response for mail pop servers.
Tuesday April 17, 2007 1025-2100
Due to a high load on all pop servers, mail experienced a slow response.
- Notification modified: 2007-04-17 22:55:29
Decision Support Down 9:05 - 9:49 4/17/07
Apache was down on Decision Support with made Decision Support unavailable. Once apache was brought up Decison Support was back up.
- Notification modified: 2007-04-17 09:56:05
Scheduled Exchange Server Maintenance - April 13, 8pm-11pm
The Xeda servers (Active Directory and Exchange) were updated to Microsoft Service Pack 2 and the monthly MS patches for April were also being applied from 8pm-11pm. Mailbox disconnects to the campus Exchange servers at various times during this time period were expected.
- Notification modified: 2007-04-13 23:14:50
Exchange service interruption - Wed. Apr 4, 9PM - 10PM
Between 9pm and 10pm XEDA Exchange users may have experienced a service outage of a few minutes as system administrators applied a critical security patch Microsoft sent yesterday. Please see http://www.microsoft.com/technet/security/bulletin/ms07-apr.mspx for details.
- Notification modified: 2007-04-05 02:14:29
ISUN Connection Difficulties 04/04/07 14:10 - 14:38
ISUN systems were intermittently available due to networking difficulties. Administrators fixed the problem.
- Notification modified: 2007-04-04 14:39:04
Saturday, March 31, 2007 15:45 - April 1, 2007 9:46am
The security certificate for smtp.ucdavis.edu expired. Users sending mail might have received a warning that the certificate has expired. Sysadmin put the new SSL Certificate in place in the smtp pool and restarted sendmail on the servers, and the warning is gone.
- Notification modified: 2007-04-01 10:22:14
Some of MySpace folders for MyUCDavis are unavailble.
03/27/2007 11:15 - 12:16
MySpace volumes served by home cell file server afshome5 was unavailable as it recovered from an AFS software malfunction and restart.
- Notification modified: 2007-03-27 13:26:06
Network Maintenance: Datacenter Switch Reboot : 03/20/2007-06:15
Description: Datacenter Switch Reboot
Start of Window : 03/20/2007 06:15
End of Window : 03/20/2007 06:30
(all times are in 24 hour format)
Purpose:
Reload a switch in the Datacenter to enable configuration changes.
Impact:
One brief outage to all devices that connect through this switch.
Affected:
128.120.32.0/20
128.120.192.0/24
169.237.58.0/24
169.237.59.0/24
169.237.104.0/24
169.237.105.0/24
169.237.121.0/24
169.237.224.64/26
169.237.224.128/26
169.237.224.192/26
169.237.229.64/26
169.237.232.96/27
169.237.234.0/27
169.237.246.0/28
169.237.91.0/24
169.237.144.0/26
If you have questions or concerns regarding this event, please contact:
- Notification modified: 2007-03-27 02:55:48
System Performance Notice
March 15th was Notification Day for the Fall 2007 undergraduate applicants. Consequently, the Computing Accounts interface and any service that contacts the Computing Accounts Database ("mothra") was under a heavy load of traffic. This annual event continued throughout (March 16th).
- Notification modified: 2007-03-16 10:18:37
ITX 754-HELP 03/15/07 07:10 - 07:50
The campus help desk was available however, the 754-HELP number was not connecting to their phone pool. Network operations staff have fixed the issue and the phones are working properly.
- Notification modified: 2007-03-15 07:56:13
Slow response for "www.ucdavis.edu"
03/13/2007
11:25 - 11:50
A high load in http connections caused a slow response for "www.ucdavis.edu". System administrators fixed dwight and added it to the load balancer to help harry with high loads.
- Notification modified: 2007-03-13 12:01:37
Database upgrade of PPS 3/9/07-3/11/07
There was an Oracle upgrade on the PPS production data warehouse Friday 03/09/07. The database upgrade lasted until Monday 03/11/07.
- Notification modified: 2007-03-14 06:59:11
Computing Accounts and DHCP
Computing Accounts
DHCP
Directory Services
was due to a monitoring configuration change and was false.
System administrators put the new rules in place. The alarms are cleared.
The services are actually OK.
- Notification modified: 2007-03-09 11:43:33
ISUN outage 3/7/2007 4:50pm - 5:20pm
Due to firewall isssues with the POP servers, users experienced timeout issues when attempting to directly access mail stored on mailbox.ucdavis.edu.
- Notification modified: 2007-03-07 18:02:53
SmartSite Down 10minutes 3/7/07 12:00 - 12:10
SmartSite was down. The application servers encountered Java OutOfMemory errors again. The application was restarted on each host and service restored.
- Notification modified: 2007-03-07 13:42:17
System and Network Issues 03/02/07 09:30-10"40
System administrators investigated multiple system timeout issues. The problem was a web application problem.
System administrators restarted the application.
The data center at UC Davis was experiencing internal network issues.
One of the servers was having trouble and one was ok so depending on where you got routed you could get in or not. Both servers are now back online and we think all is well.
- Notification modified: 2007-03-02 10:49:01
Remedy Service Unavailable March 2nd - March 4th
Subject: Remedy Service Unavailable Weekend of March 2nd - March 4th due
to Hardware Upgrade.
Remedy was unavailable from
9PM Friday, March 2nd, through
6AM Monday March 5th,
in order to move the system to new hardware.
Access via the Remedy MidTier client was unavailable as well.
Should you have any questions or concerns about this scheduled downtime,
please contact the Remedy Admin group at ars-admin@ucdavis.edu
- Notification modified: 2007-03-04 05:48:08
Saturday, February 24, 2007 10:00 - 23:10
SISWeb was unavailable due to trouble following scheduled maintenance.
System administrators restored the service.
- Notification modified: 2007-02-24 23:28:42
SmartSite Upgrade Sat. 2/24/07
SmartSite was unavailable because of a planned upgrade Sat. 2/24/07 from midnight until 8AM.
- Notification modified: 2007-02-24 08:03:11
Blue.ucdavis.edu looping mail. 2/23/07 07:50-08:10
Email server blue.ucdavis.edu experienced a problem due to looping mail. System administrators stopped mail delivery on that host in order to resolved the issue.
Blue service was restored at 08:10.
- Notification modified: 2007-02-23 08:26:50
SMTP degraded service 02/23/07 07:50-09:20
The incoming mail servers (smtp/mx) experienced difficulty due to large amounts of spam we were receiving. System administrators resolved the issues.
Clients experienced degraded service.
Service restored at 09:20.
- Notification modified: 2007-02-23 09:34:56
Cas.ucdavis.edu Unavailable 2/22/07 11:35-11:45
cas.ucdavis.edu was unavailable. System Administrators resolved the issue.
- Notification modified: 2007-02-22 11:57:30
Network intermittent connectivity 02/20/07 11:28 - 12:44
The source of the bridge loop has been removed from the network and the link has been re-enabled.
- Notification modified: 2007-02-20 13:14:54
MySpace folders for MyUCDavis are unavailble for maintanance.
02/18/2007
12:30 - 14:30
Emergency Maintanance for home cells. MySpace folders for MyUCDavis users were unavailable.
- Notification modified: 2007-02-18 16:57:55
Pop server green is unavailable
02/18/2007
10:37 - 11:35
Pop server green was unavailable from 10:37 - 11:35. System administrators cycled green to bring it back online.
- Notification modified: 2007-02-18 12:17:47
AFS PROBLEMS 2/17/07 07:40 - 13:30
We are experienced problems with our AFS servers, which were affecting MYUCDAVIS, and SMARTSITE. Systems administrators resolved the issues.
- Notification modified: 2007-02-17 14:19:03
Meeting Maker
02/13/07 08:30 - 09:15
Meeting Maker is experienced degragated services do to a high load.
- Notification modified: 2007-02-13 14:42:20
AFS Connectivity Slowness 2/12/07 19:30 - 20:15
For a brief time period, AFS connectivity experienced slowness.
- Notification modified: 2007-02-12 20:23:51
Data Center Services Network Outage Feb 2, 5:12-5:20PM
The Data Center experienced a network outage on one VLAN that is affected some services.
- Notification modified: 2007-02-01 17:26:07
CAS Web 02/01/07 11:30 - 14:00
Due to a load balancer malfunction, the CAS service was intermittently unavailable to certain web sites from 11:30 until 14:00.
- Notification modified: 2007-02-01 14:51:18
Heavy load on MyUCDavis 1/31 2250 - 2315
MyUCDavis was experiencing extremely high load due to the unavailability of the housing channel. Sys admins resolved the issue.
- Notification modified: 2007-01-31 23:19:47
Email server yellow unavailable Tue 1/30 19:35 - 2000
Email server yellow was unavailable due to a hardware failure. Sys admins were able to resolve the problem.
- Notification modified: 2007-01-30 20:02:47
Account Creation Problems for Student Applicants - 1/27/07 17:00
Account creation problems were reported for student applicants. System administrators have fixed the issue.
- Notification modified: 2007-01-28 17:35:43
Grad Studies Web services disruption
Graduate Studies Web services are off-line until further notice. Contact the Campus Data Center (754-1566) if you have questions.
- Notification modified: 2007-01-29 22:42:41
Computing Accounts Connection Problem 01/27/07 10:20 - 11:20
Campus computing accounts experienced database timeouts and delays. Administrators resolved the issue.
- Notification modified: 2007-01-27 11:18:05
Remedy Maintenance 9pm - 1am January 26, 2007
Remedy production server was down from 9:00pm Friday night - 1:00am Saturday morning for maintenance/patching. If you have any questions or comments about this scheduled maintenance, please email or call Danh 752-3979 dtdoung@ucdavis.edu
- Notification modified: 2007-01-27 01:16:04
CASWEB 1/25/07 09:25
We are experienced problems with casweb authentication.
System administrators resolved problem.
- Notification modified: 2007-01-25 11:12:43
Geckomail accessing tan 1/22/07 7:00am -9:30am
There were reports of trouble accessing mail server tan via geckomail. To work around this problem they used: "LoginID@pop32.ucdavis.edu".
- Notification modified: 2007-01-22 09:37:53
Mail Server Salmon Unavailable via Geckomail 1/21 16:00-21:00
Mail server salmon.ucdavis.edu was having issues with mail being checked via Geckomail.
System administrators fixed authentication required for users of mail server salmon.ucdavis.edu to access their mail via Geckomail.
- Notification modified: 2007-01-21 21:21:28
Mail servers Blue &Tan are unavailable. 1/21/07 - 3:00 - 14:10
Blue.ucdavis.edu and tan.ucdavis.edu were temporary unavailable via geckomail. The local ip addresses on geckomail were changed to reflect the changes made last night to the tan and blue servers. Everything is back to normal.
- Notification modified: 2007-01-21 17:38:23
news website- Saturday, January 20, 2007 13:20
The news.ucdavis.edu web site was inaccessible from off-campus IP addresses. System administrators restored access.
- Notification modified: 2007-01-20 16:56:43
Saturday, January 20, 2007 11:45 - 12:25
Two of the servers in the SMTP pool were experiencing trouble, resulting in degraded performance. System administrators investigated and restored full functionality of the SMTP pool.
- Notification modified: 2007-01-20 12:26:39
Friday, January 19, 2007 14:50 - 15:20
The Data Center experienced temporary network slowness or congestion that has since been resolved.
- Notification modified: 2007-01-19 15:36:06
Heavy Load on Purple 1/10/07 10:45-15:00
Purple experienced heavy load, system administrators resolved the issues.
- Notification modified: 2007-01-10 15:23:35
Account Creation Problems 1730-2130 1/9/07
Computingaccounts.ucdavis.edu and MyAdmissions websites experienced problems with creation of new accounts and setting passwords. Staff restored full functionality.
- Notification modified: 2007-01-09 22:36:37
Yellow was hung from 23:45 on 1/08/07 to 7:30 on 1/09/07
Yellow/pop17 was reported to be unresponsive about 07:30. Operations power-cycled it and it's back up now.
Checking the console, the last timestamp there was approx 23:45 yesterday so it may have been down as long as that. This is the 2nd hang, 1st was on Saturday so time to file a report with Sun for hardware problem.
- Notification modified: 2007-01-09 08:06:03
pop17/yellow.ucdavis.edu, down - Sat. 01/07/07 1:14a - 212a
Yellow/pop17 was affected by a brief power glitch. System was powered back up and service has been restored.
- Notification modified: 2007-01-07 02:46:23
- Notification modified: 2007-01-04 21:12:39
NetID & DHCP Registration - Tues. 12/26/06 to Wed. 01/03/07
NetID Administration and DHCP Registration/De-Registration were not communicating correctly and newly added MAC address were not being recgonized by the campus DHCP servers. Programmers have resolved the issue and new registration should be effective. Programmers have also repushed the registration information from 12/26/06 to 1/3/07 so MAC addresses registered during that time period should now have DHCP access. After reviewing the logs for both NetID and DHCP Registration, programmers feel that all missed registrations have been successfully updated.
- Notification modified: 2007-01-05 07:08:12
Secalert.ucdavis.edu web site down 1/02/07 10:05 -14:34
The secalert.ucdavis.edu web site was down.
- Notification modified: 2007-01-02 14:42:34
Data Center Services Network Outage Dec 29, 8 AM
On Friday, December 29, 2006 8AM-8PM, the network in the Campus Data Center underwent a major reconfiguration that affected the availability of all systems and services in the Data Center. Most services were available by 6PM. A few services continued to experience intermittent problems through Tues. January 2. By Wed. January 3, network & system administrators believed that all major problems were resolved and all services were available to clients.
- Notification modified: 2007-01-09 21:13:32
Exchange outage 17:40 - 17:54 12/21/2006
There was a brief exchange outage, system administrators expediently resolved the issue.
- Notification modified: 2006-12-21 17:57:53
ISUN DOWNTIME 5-7PM Dec 21. 2006
The ISUN servers, veni.ucdavis.edu, vici.ucdavis.edu and vidi.ucdavis.edu, were moved to new IP addresses on December 21, 2006, from 5:00 PM to 7:00 PM. This change allowed the service to reside behind the Data Center firewall. The ISUN servers were unavailable during this time.
- Notification modified: 2006-12-28 18:03:04
Tan was unavailable due to maintennce 12/27 1700-2200
Hardware for the mail server tan.ucdavis.edu was replaced on December 27 from 5 to 8 P.M.
- Notification modified: 2006-12-28 18:04:07
Network Outage: Campus: 2006-12-14 08:50 - 09:10
A bridgeloop on campus was causing high CPU levels on core network devices. Connectivity to sites on and off campus may have been intermittent.
The loop was removed from the network, and all traffic should be passing normally at this time.
- Notification modified: 2006-12-14 09:18:59
Wednesday, December 13, 2006 14:30 - 15:30
Oracle on ISUN server veni.ucdavis.edu was unavailable. The ISUN Unix login servers remained available for general use.
- Notification modified: 2006-12-13 15:54:48
Email Intermittent Connectivity 12/07/06 17:39 - 18:00
Email clients experienced intermittent connectivity issues. System administrators corrected the problem.
- Notification modified: 2006-12-07 18:15:36
Dafis TP experiencing problems 09:55 - 11:00
Clients reported issues logging into Dafis Transaction Processing. System administrator found the uniface license servers in a broken state. System administrator restarted the servers to resolve the problem and redistribute the load on those servers.
- Notification modified: 2006-12-07 11:28:01
Wireless Authentication Unavailable 12/6/06 1:00pm
One of the wireless gateways stopped redirecting users to the authentication page for a short time.
Locations Affected:
Wireless users throughout campus.
Status:
Service has been restored. 1:40pm
An emergency reboot of one of the wireless gateways was performed to resolve redirection issues.
- Notification modified: 2006-12-06 13:45:45
Exchange Services Brief Outage
On Monday, December 4th at around 12:57pm we received notification from sys administrators that exchange services were unavailable due to a brief network outage.
- Notification modified: 2006-12-04 13:36:25
MyUCDavis - Connection issue - 12/2/06 - 15:00 -12/3/06 6:30
Myucdavis was experiencing connectivity issues due to a database connectivity problem. System administrators cleared up the oracle listener files.
- Notification modified: 2006-12-03 07:44:56
MyUCDavis Experiencing High Load Volume12/02/06 08:30
MyUCDavis website experienced a high load volume from 08:30-11:30. No services were impacted.
- Notification modified: 2006-12-02 12:20:43
Exchange Services Unavailable
On Thursday, November 30 at around 12:30pm we received notification from sys administrators that exchange services were unavailable due to a brief network outage.
Services were restored at 12:55pm when system administrators were able to failover the services to backup servers.
- Notification modified: 2006-11-30 13:04:27
Tan's disk crashed 11/26/06 8:13 - 8:39
Tan's disk crashed. Sys Admin had it back up shortly.
- Notification modified: 2006-11-26 09:23:57
Downtime for email server, mailbox, 11/20/2006 5p.m. - 8 p.m.
Scheduled downtime for email server, mailbox, Nov. 20, 5 p.m. - 8 p.m.
The email server, mailbox, was unavailable for up to three hours on Monday, November 20, 2006, beginning at 5 p.m.
Mailbox users were able to send email during the scheduled downtime, however, they could not retrieve their email. All email that was received during the downtime was queued and delivered after the system was returned to normal operations.
REASON FOR DOWNTIME
During the downtime, approximately 1,000 accounts from mailbox.ucdavis.edu will be moved to a new email server, charcoal.ucdavis.edu. This move will improve email performance for both the users being moved and those remaining on the mailbox server.
QUESTIONS
If you have any questions, please contact IT Express at 530-754-4357.
- Notification modified: 2006-11-20 19:54:40
E-mail server TAN is down 11/19/2006 12:02 - 15:19
The e-mail server TAN was down. Problems occured after patching. After fscking /var/mail everything is ok.
- Notification modified: 2006-11-19 15:29:15
UCDavis Website Down 11/17/06 - 11:00pm
The www.ucdavis.edu website was down for ~10 minutes Fri. Nov 17 at ~11:00pm. A web server was breifly overloaded, but system administrators have solved the issue.
- Notification modified: 2006-11-17 11:22:37
UCDMC Dafis Proxy -11/17/06 - 9:30am
A DaFIS proxy server located at the UCD Med Center was down from 9:30am-3:00pm. System Administrators have resolved the issue.
- Notification modified: 2006-11-17 15:20:06
MyUCDavis Down
MyUCDavis was unavailable Thursday, Nov 16 from 12:30-13:30.
- Notification modified: 2006-11-16 14:07:28
AFS Outage
An admin AFS server had a brief outage due to some network maintenance. The server was down from 7:10-7:25am.
- Notification modified: 2006-11-15 13:19:58
Banner unavailable from 5 p.m., Thu, 11/09 - 8 a.m., Mon, 11/13
The software program that manages data in the Student Information System Banner was upgraded this weekend. The Banner system was unavailable from 5 p.m., Thursday, Nov. 9 through 8 a.m. Monday Nov. 13. SISWEB was down during the upgrade; however, faculty was be able to use the course management tools in SmartSite and MyUCDavis. Other UC Davis online applications were affected during the upgrade weekend.
- Notification modified: 2006-11-13 13:04:59
SERVICE DEGRADATION: Email 11/9/06 8:30am - 11:15am
Data Center Systems Administrators isolated an issue with the Data Center firewall which resulted in intermittent email connections for UC Davis clients. The Systems Administrators corrected the issue and full email access was restored.
- Notification modified: 2006-11-09 11:57:00
Brief Email Interruption Tues 11/07 19:55-20:05
A few email systems including pop server azure were affected by a brief UPS glitch. Systems were powered back up and service has been restored.
- Notification modified: 2006-11-07 20:17:44
Exchange Email Servers
An Issue with one of the exchange email servers was causing emails not to be delivered for exchange clients. This issue has been resolved. Any emails sent to or from the client should now have reached their destionation.
Down time: ~9:00am - 3:00pm
- Notification modified: 2006-11-04 15:32:38
Replace UPS Mrak Hall
Network Maintenance:
Start of Window : 11/02/2006 17:30
End of Window : 11/02/2006 18:00
(all times are in 24 hour format)
Purpose:
Need to move three switches on the 4th floor from house power to a new UPS.
Impact:
Estimated loss of data network connectivity of approximately 5 minutes per switch.
Affected:
One or more NAMs of the following VLANs was affected:
VC-RESEARCH-1 IDAV-1 CHANCELLOR-1-FW VC-STU-AFFRS-1
- Notification modified: 2006-11-03 00:21:26
XEDA Mail Delivery : Thu. Nov 2, 0700hrs - 1300hrs
The UC Davis XEDA Mail service experienced a mail queuing issue. From 7am-1pm mail from outside of the Exchange organization was delayed. The issue has been resolved and the delayed messages are now being delivered.
There was no mail lost.
- Notification modified: 2006-11-03 17:27:48
IET Secure Web Sites Outage Tues. Oct. 31 2100-2130
The secure IET web sites, http://security.ucdavis.edu/ and http://tif.ucdavis.edu/, were unavailable for approximately 30 minutes to allow updates to the server software.
- Notification modified: 2006-10-31 22:29:36
"Tan" email server outage; Mon.Oct.30.2235hrs - 2255hrs
Tan was down due to a hardware failure. System Administrators replaced the faulty components and service was restored.
- Notification modified: 2006-11-03 19:15:52
Network Outage - ISUN & Mailbox: Sat.Oct.28. 8:15am-8:45am
Due to a network outage, the mailbox email server and ISUN were unavailable. System administrators were able to resolve the issue.
Time of repair: 8:45am
- Notification modified: 2006-11-03 19:16:51
Email Server Outage: Sat.Oct.28. 04:50 - 23:13
"The email server, Tan, was currently down.
System administrators resolved this issue.
Emails sent to Tan were queued up and delivered once the system came back online."
- Notification modified: 2006-10-28 23:15:32
Network Problems 10/27/06 11:45-
IT Express was reporting calls of network problems. NOC staff investigated the problems. NOC found the rash of network reports turned out to be several customer issues which, though similar in symptoms, were unrelated.
- Notification modified: 2006-10-27 12:09:27
IET Web Site Update Thurs.Oct.26.1700hrs
Starting on Thurs. 10/26, IET began updating most IET web sites to a new look and feel. The update process has been completed for all sites.
Completed Thrus. 11/9
- Notification modified: 2006-11-09 14:18:32
Network Outage: Power Ouage: 2006-10-25 13:00
Description: A brief power outage disrupted network connectivity to some areas of campus.
Locations Affected: One or more users in the following buildings:
Housing Adm.
Mail Cust.
TB 166
Solano 2200
Repairs were completed at ~14:00.
- Notification modified: 2006-10-25 14:42:12
Area 10 DSLAM Outages October 24, 2006 08:10 - 8:49
Tuesday, October 24, 2006 - 0810 - 8:49
Description: The Area10 DSLAM stopped passing traffic.
Locations Affected: All DSL sites fed by the Area 10 DSLAM. This mostly impacted locations on campus south of I-80.
Status: The Area 10 DSLAM lost it’s configuration again as a result of power issues last night. The configuration on the Area 10 DSLAM has been restored
- Notification modified: 2006-10-24 09:06:48
Monday 11:33pm - Tuesday 12:28am Power Outage on Campus
On Monday 11:33pm - Tuesday 12:28am, power outages on campus caused a network outage in several parts of campus. Facilities was able to restore power.
- Notification modified: 2006-10-24 00:28:21
Mailbox unavailable Sun 10/22/06 12:49am -Mon 1-/23/06 8:11am
Clients using mailbox.ucdavis.edu experienced problems accessing their mail. System Administrators resolved a problem with the security certificate for mailbox.ucdavis.edu
- Notification modified: 2006-10-24 06:41:43
9:10 - 9:40 10/23/06 Area10 DSLAM stopped passing traffic.
Description: The Area10 DSLAM stopped passing traffic. The Area 10 DSLAM lost it’s configuration as a result of power issues last night.
Locations Affected: All DSL sites fed by the Area 10 DSLAM. This mostly impacted locations on campus, south of I-80.
Status: The configuration on the Area 10 DSLAM has been restored.
- Notification modified: 2006-10-23 15:07:57
Power outage Sunday Oct 22 2110-2150
A power outage affected network access to various parts of the main campus. Power and network connectivity have been restored.
- Notification modified: 2006-10-22 22:05:09
Distauth-Protected Services Experienced Outages
Saturday 10/21/2006 00:00 - 00:43
All distauth related services (MyUCDavis, Course Management, Geckomail, Distauth-protected websites, etc.) experienced outages. This was due to a longer than normal patching window for the AFS service and the primary campus DNS servers being down for hardware maintainance.
- Notification modified: 2006-10-21 01:03:26
Listproc and System Maintenance 22:00 10/20 to 00:30 10/21
Listproc was unavailable and email sent to the lists during that time was scheduled for delivery after the maintenance completed.
- Notification modified: 2006-10-21 00:58:25
Network Outage: Campus and Resnet: 2006-10-20-0930
Description: The Campus firewalls were experiencing periods of high cpu due to DoS attacks from the Internet. This is causing intermittent loss of connectivity to the Internet and Internet2.
Locations Affected: All Campus and Resnet traffic to the Internet and I2
Status: Adjustments to the session limits on the Campus and Resnet packet-shapers has alleviated the high CPU levels on the campus firewalls.
Estimated time to repair: Repairs completed
- Notification modified: 2006-10-20 15:51:10
ResNet Outage - 2006-10-18 16:15
One of the two central campus firewalls has failed. Resnet traffic did not re-route correctly around the failed device.
All Resnet traffic to off-campus sites. Resnet traffic to on-campus sites should be unaffected.
The NOC is working to re-route the Resnet traffic around the ISP firewall. All other campus traffic to the internet did re-route correctly. If you are experiencing connectivity problems to off-campus sites from outside of Resnet, please notify the IT Help Desk at 754-HELP.
- Notification modified: 2006-10-18 16:43:33
Slow email delivery - Wednesday 10/18/06 8am - 6pm
Due to a heavy spam run on the campus servers email delivery for some clients took longer than expected. Systems Administrators deleted the spam messages in the email queues and blocked the offending IP Addresses. Queued messages have been delivered to individuals but there where delays due to the backlogged queues.
- Notification modified: 2006-10-18 21:09:11
IET Electronic Mailing Lists (ListProc) 10-11-06 1300 - 2030hrs
IET Electronic Mailing Lists (ListProc) 10-04-06 Emails sent to IET maintained mailing lists took an exceptionally long time to be delivered.
The campus list server experienced an extremely heavy load which resulted in delays in the processing of list emails. System Administrators devised a short term solution to ease the load for the time being, and are looking at long term architecture changes to maintain effective communications through the list server. In particular, the mailing list hardware will be upgraded on 10-13-06.
If you have any questions concerning this issue or any other concerns with campus computing services please contact IT Express at 530-754-4357 or ithelp@ucdavis.edu.
- Notification modified: 2006-10-11 21:07:46
Outbound Email Problems 15:00 - 15:25 10/10/2006
Campus email system experienced network problems. Consequently, some clients are unable to send email. System administrators remedied the problem.
- Notification modified: 2006-10-11 15:40:53
Primate Center Network Problem 08:46 - 09:15 10/10/2006
A router in area 8 (Primate Center) stopped passing traffic to several of its interfaces. The issue cleared itself during troubleshooting. Captures from the device have been sent to the vendor for analysis, and the NOC will continue to work to determine the cause of the failure. If problems persist, please call the IT Helpdesk at 754-HELP. Thank you.
- Notification modified: 2006-10-10 10:29:57
IET Electronic Mailing Lists (ListProc) 10-04-06 Emails
IET Electronic Mailing Lists (ListProc) 10-04-06 Emails sent to IET maintained mailing lists take an exceptionally long time to be delivered.
The campus list server is experiencing an extremely heavy load which is resulting in delays in the processing of list emails. System Administrators worked on a short term solution to ease the load for the immediate time frame and are looking at long term architecture changes to maintain effective communications through the list server. In particular, the mailing list hardware will be upgraded on 10-13-06.
If you have any questions concerning this issue or any other concerns with campus computing services please contact IT Express at 530-754-4357 or ithelp@ucdavis.edu.
- Notification modified: 2006-10-06 14:34:26
Internal Network Latency
Some network latency issues had been noticed and were resolved.
- Notification modified: 2006-10-04 11:59:19
Listproc down from 11:39 till 11:52 10/03/06
Listproc was down while the database was being rebuilt.
- Notification modified: 2006-10-03 12:52:28
IET Electronic Mailing Lists (ListProc) 9-28-06 to 9-29-06
Symptoms: Emails sent to IET maintained mailing lists took an exceptionally long time to be delivered.
Issue: The campus list server was experiencing an extemely heavy load which was resulting in delays in the processing of list emails. System Administrators worked on a short term solution to ease the load for the immediate time frame and are looking at long term architecture changes to maintain effective communications through the list server.
If you have any questions concerning this issue or any other concerns with campus computing services please contact IT Express at 530-754-4357 or ithelp@ucdavis.edu.
- Notification modified: 2006-09-29 21:01:23
Wireless Network OutageThurs. Sept 28, 2006 10:10 - 18:46
Description: Problems with one of the wireless gateways prevented some users from authenticating on the Moobilenet wireless network.
Locations Affected: Wireless users connecting to through the problematic gateway. These networks are located primarily in the Vet Med and King Hall areas.
Status: The NOC worked with the wireless gateway vendor to resolve the problem.
- Notification modified: 2006-09-28 22:32:47
SMTP Service Degradation
The smtp server pool was experiencing short but frequent interruptions to its ability to accept mail from campus users. Systems administrators have resolved this issue.
- Notification modified: 2006-09-22 15:47:15
Power Outage for West Campus Friday Sep. 22, 2006 07:46 - 08:43
Description: A power outage affecting the entire west campus is disrupting
connectivity to campus buildings in the west campus area.
Locations Affected: All west campus buildings.
Status: The power has been restored.
Estimated Time to Repair: Complete.
- Notification modified: 2006-09-22 08:44:03
NetID & DHCP Registration - Wed. 9/20/06 12:24 - 21:30hrs
NetID Administration and DHCP Registration/De-Registration was unavailable while sysadmin resolved a hardware problem. Functionality remained for the current DHCP registered users and people who needed to obtain DHCP addresses. Services were restored by 9:30PM.
- Notification modified: 2006-09-20 21:50:04
Delayed outbound email. - Wed., Sept. 20, 1900hrs -22:00hrs
Outbound email was experiencing long delays and occasional delivery failures. System administrators resolved this degradation of system performace.
- Notification modified: 2006-09-20 22:17:02
Network Outage Saturday 9/17/06 11:00 - 1:09
The GBIC went bad, in the switch, in NOC network cabinet 2, here at the Data Center. Operator was unable to see any screens. Everything frozen. This effected the entire campus. NOC techs have replaced the GBIC and all is well.
- Notification modified: 2006-09-17 14:39:44
Systems Scarlet and Yellow -- 9/16/06 23:00 to 9/17/06 01:30
The mail server Yellow is currently unavailable for service.
System administrators are currently working on the issue.
- Notification modified: 2006-09-17 01:31:43
Intermittent SMTP Issues
SMTP servers were having intermittent issues. One of the SMTP server was failing System Administrators resloved the issues.
- Notification modified: 2006-09-16 14:26:35
Thurs., Sept. 14, 17:30 - Sat. Sept. 16, 12:30
why.ucdavis.edu, admissions.ucdavis.edu, virtualtours.ucdavis.edu, and studentaffairs.ucdavis.edu were down. Systems Administrators had to rebuild the database. problem.
- Notification modified: 2006-09-16 14:07:09
Network Degredation: 09/14/2006 13:00 - 14:00
Affected Areas: Campus managed wireless access
Campus modem pools
Campus DSL
Remote T-1 connections.
Any individual machine which only had 169.237.1.250 listed
for their DNS server.
Status: The new code revision was removed from the router, and the old code
was reloaded. Complications during the software reload caused a longer
outage than expected to the locations listed above. At this time, DNS
traffic is passing normally through the Area 42 router and no further
trouble is expected.
Please contact IT Express at 754-HELP if problems persist.
Estimated Time to Repair: Repair complete.
- Notification modified: 2006-09-15 08:22:37
Mailing lists system maintenance
Friday, 10pm, 08 Sep. - Saturday, 12:15pm, 09 Sept.
The mailing lists system, Listproc, was unavailable during the maintenance window. Email was queued up during the maintenance window, and it was delivered after the system was back online.
- Notification modified: 2006-09-09 12:35:08
Network Outage: Campus: 2006-09-1 15:36-16:26
Friday Sep. 1, 2006 16:26
Description: A denial of service attack is disrupting connectivity to hosts throughout campus.
Locations Affected: All campus traffic to off campus sites.
Status: Multiple off-campus hosts were sourcing a denial of service attack against hosts on campus. The attack caused elevated CPU levels on the campus firewalls, resulting in a disruption of internet traffic throughout campus. The NOC placed filters at the campus border to block the offending traffic. At this time all connectivity should be back to normal. Please contact the IT Express Helpdesk at 754-HELP if you are still experiencing problems.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2006-09-01 16:36:31
Network Traffic Disruption 8/22/06 13:55 -14:25
Locations Affected: All campus hosts.
Status: A network loop was detected on campus. The loop caused elevated CPU levels on network devices throughout campus. Users noticed heavy network latency as a result. NOC personnel tracked down and removed the source of the loop.
- Notification modified: 2006-08-22 14:44:12
Listproc temporarily unavailable 8/21/06 21:30 - 21:55
ListProc was unavailable while system administrators resolved an issue with a full disk. Mail delivery was temporarily stopped, but messages were queued and delivered once service was restored.
- Notification modified: 2006-08-21 21:56:02
DCHP Problems 8/21/06 8:26 -8:58
Problems with one of the netid servers, consquently some clients experienced issues with obtaining an ip address via dchp.
- Notification modified: 2006-08-21 08:58:52
SAT AUG 19 Data Center Transformer Upgraded 06:00-17:30
On Sat Aug 19 from 06:00AM to 17:30PM, the planned electrical equipment upgrade interrupted some UC Davis online services. As a precautionary effort to prevent hardware failures during the transition, several central campus Data Center services were shut down.
- Notification modified: 2006-08-19 19:05:21
Computing Accounts & Mothra - NIS issues 8/15/2006 7:26 - 7:50
NIS failed on sanford, which caused Computing Accounts Service and Mothra to be unavailable.
- Notification modified: 2006-08-15 08:11:43
TRANSACTION PROCESS ON DaFIS DOWN 08/10/06 06:30 AM -1:00 PM
DaFIS TP and the following applications were unavailable at 6:30 AM:
UCD Buy
MyTravel pre-trip payment
Vendor request form
Cost Share
Effort Commitment
Garnishments
Gift Processing
Direct Deposit
MyTravel – some parts of this application was affected.
System administrators resolved the problem and TP and the above applications were available as of 1:00 PM.
- Notification modified: 2006-08-10 13:37:09
Power Surge 7/26/06 7:02 - 8:45
Power Surge this morning effected Exchange and Active Directory. Both were up by 8:45.
- Notification modified: 2006-07-26 14:44:32
Network Outage Wednesday July 26, 2006 10:00 - 10:21
Wednesday July 26, 2006 10:00
Description: Campus connectivity to HPR internet was lost. Denial Service Attack at the Campus Border.
Locations Affected: All campus hosts.
Status: An issue with the HPR firewall disrupted campus connectivity to some internet sites.
- Notification modified: 2006-07-26 14:56:14
Mothra Intermittent Availability 7/26/06 7:02 - 10:10
The Mothra database experienced intermittent availability. System administrators fixed the problem.
- Notification modified: 2006-07-26 18:08:53
Data Center Network Outage
The Data Center is experienced internal network difficulties due to a failed UPS hardware. The DAFIS, MyTravel, Facilities, and Active Directory services were unavailable during this outage. Active Directory was back online at 7:45, Dafis at 9:15, MyTravel 11:30, and Facilities 9:55.
- Notification modified: 2006-07-25 11:31:06
Area 7 Router is Down 7/23/06 08:38 - 11:15
Area 7 Router was down due to electrical problems in AOB 4 which took down the network cabinet.
- Notification modified: 2006-07-23 11:28:36
Slow Network Connectivity 7/21/06 10:17-11:08
The cause of the multicast loop was located and removed from the network. All campus traffic should be passing normally at this time.
Estimated Time to Repair: Repairs complete.
A multicast loop has been detected on campus. The loop has caused elevated CPU levels on network devices throughout campus. Users may notice heavy network latency as a result. NOC personnel are attempting to locate the source of the loop, and hope to have the issue resolved shortly.
- Notification modified: 2006-07-21 11:44:53
Sakai System Maintenance 7/15/2006 00:00 - 01:00
System maintenance on sakai2.ucdavis.edu was performed and during that time period, sakai2.ucdavis.edu was not running.
- Notification modified: 2006-07-15 02:01:00
AFS Slow Response 20:00 - 06:00 7/13 - 7/14
One of the three AFS database servers is located in a building that had its cooling turned off for a significant period of time on 7/13/06. This system was shut down to prevent overheating. AFS response times might have been delayed as client systems contacted to the remaining two database servers to obtain information.
- Notification modified: 2006-07-14 06:32:12
DNS Servers down 7/11/06 10:20 - 10:26
The DNS servers, trapper, houlihan, klinger and hawkeye were down for about six minutes. Problem was due to a configuration error.
- Notification modified: 2006-07-11 10:46:50
Power Outage Monday July 10, 2006 16:11-18:25
Facilities reported a power outage to several areas in the northern part of campus. Some network services residing in this area of campus were unavailable until power was restored. Facilities does not yet know the cause of the outage. Power has been restored.
- Notification modified: 2006-07-10 20:02:55
AFS UPGRADE
IET is upgraded the AFS hardware and software. These changes improved the AFS service performance and increased the available storage space.
Friday June 30, until Tuesday July 4, 10pm, the remaining AFS admin cell data were be transitioned to the new hardware and software.
There was no downtime for any applications or services.
- Notification modified: 2006-07-05 10:34:02
Mailbox.ucdavis.edu unnavailable, Friday, June 30, 2200- 2230
The email server, Mailbox.ucdavis.edu, had an inconsistent file system. Mailbox was rebooted to resolve this problem. No e-mails were lost during the maintenance window. They were queued up and delivered after the system was restored to service.
- Notification modified: 2006-06-30 23:38:53
SISWEB outage. Friday, June 30, 16:30 - 17:00
Sis-Web was unnavailable. System Administrators identified the cause and remedied the problem.
- Notification modified: 2006-06-30 17:23:10
Authoritative DNS Server Outage Notice
On Monday, July 3rd, 2006, at 9:00am, the two authoritative DNS servers that service external traffic to campus was moved from internal network infrastructure to the campus network border to provide better resiliency and availability during internal campus network outages. There was a one-minute downtime window during the logical move of the servers from the internal network to the border, during which time non-cached initial queries to these two DNS servers from outside of campus (ISPs, Internet backbone infrastructure, etc.) will be unable to resolve DNS queries specifically pertaining the .ucdavis.edu DNS address space.
There was NO impact to on-campus users who use the internal DNS servers residing at 169.237.1.250 and 169.237.250.250 except for Med Center.
The Med Center was not able to connect to/use the two external DNS servers that were moved. All non-ICMP traffic to the boxes to/from the Med Center is cut-off at some firewall somewhere. Statis routing at the NOC border(s)
seems to have exluded the 152.79 nets and they fixed after an hour of downtime
- Notification modified: 2006-07-03 14:48:42
AFS Service Impacted
Access to DaFIS from clients using the AFS proxy servers, instead of direct connections from the AFS client, experienced slow response times due to the end-of-the-fiscal-year crunch. Please be patient with access, or install the AFS client for a direct connection on your workstation. Directions for doing so can be found at http://dafis.ucdavis.edu/install/afs/
- Notification modified: 2006-07-05 10:36:02
Problems with sending email to off-campus addresses.
Saturday, June 24, 2006 at 10:15-10:48
We experenced problems sending email to off-campus addresses.
System Administrators resovled the problems.
Everything is back to normal.
- Notification modified: 2006-06-24 11:09:36
Data Center Router Maintenance
================= Network Maintenance =====================
Start of Window : 06/27/2006 06:00
End of Window : 06/27/2006 07:00
(all times are in 24 hour format)
Purpose:
Rebooted router to activate routing feature.
Impact:
No network connectivity to the campus Data Center for approximately 5 minutes while system reboots. Also, there was no period of slow connectivity while routes reconverge.
Affected:
CAAN Building
4471 Data Center
- Notification modified: 2006-06-27 09:20:59
Distauth maintenance - Saturday, June 24, 11:00 PM -- 1:00 AM
Beginning Saturday June 24 at 11:00 PM, the distauth volumes will be moved
to new software and hardware to enhance performance. Distauth services may
be down for up to two hours. Services that use the distauth service include
all campus web-based applications that use the campus web-based
authentication system.
- Notification modified: 2006-06-25 06:04:58
myInfoVault service disruption June 18, 2006: 12:00 -- 14:00
myInfoVault will be unavailable Sunday, June 18 from Noon to 2pm due to a hardware upgrade.
- Notification modified: 2006-06-18 23:40:42
A Possible Campus Wide Network Outage 6/13/06 12:25 - 1:01
The NOC experienced a building power failure. The campus border router and wireless networks were down. Campus connectivity to internet was being routed via MedCenter.
- Notification modified: 2006-06-13 13:12:47
Computing accounts services were disabled 21:30 - 24:00 6/12
On Monday, June 12th from 9:30pm to midnight, all computing accounts services were disabled while the accounts database was migrated to a new clustered configuration. During this time:
* The Computing Account Services web page (computingaccounts.ucdavis.edu) was unavailable, so
No new accounts could be created
No password changes could be processed
No services could be added to accounts
No mailIDs could be redirected
No NICs could be registered
No directory updates could be entered
* Applicants were not able to create accounts
* The Wireless Guest Access process was not available
- Notification modified: 2006-06-13 00:38:53
DNS security change on June 22, 2006
The DNS change affected a few off-campus people who have configured their computers to use campus DNS settings. Off-campus users do not have access to any online services until their DNS settings are changed.
See: [http://security.ucdavis.edu/dns.cfm].
- Notification modified: 2006-06-27 09:22:40
Home Cell & MySpace disruption - 6/5/06 16:40-00:00
The Home Cell experienced some fileserver corruption; UCD MySpace file service and the home cell were unavailable from 16:40 on Monday June 5th till midnight. Everything is back to normal.
- Notification modified: 2006-06-06 00:02:02
AFS Home Cell Upgrade Continued June 2
From 11:00 PM Friday, June 2 - 1:00 AM on Saturday, June 3, the AFS home cell database servers was migrated to new hardware and software. These changes will improve performance. The migration had been tested and there was no downtime.
- Notification modified: 2006-06-03 00:22:27
myInfoVault Outage 5/31/06 10:15am to 12:10pm
myInfoVault was unavailable due to hardware issues. System administrators have removed a server from the hardware cluser and this resolved the issue.
- Notification modified: 2006-05-31 16:01:54
Listproc maintenance 5/26/06 2200 - 5/27/06 0125
Listproc was unavailable and email sent to the lists during this time was queued for later distribution.
- Notification modified: 2006-05-27 01:26:42
AFS Home Cell Upgrade 05/26/06 2000 - 05/29/06 1200
AFS home cell file servers will be migrated to new hardware and software. These changes will improve performance and increase storage space. No downtime is expected.
- Notification modified: 2006-05-29 22:23:16
Blue email server maintenance 5-21 1915-1945
Email server blue was temporarily unavailable due to hardware maintenance.
- Notification modified: 2006-05-21 19:46:08
Wireless and Geckomail Authentication 05/17/06 11:05 - 11:45
There was a sporadic authentication problem with geckomail and wireless due to high loads. System loads returned to normal.
- Notification modified: 2006-05-17 11:52:48
8:17 - 8:27 05/15/06 Moobile Wireless Service Unavailable.
An issue with the wireless gateway was preventing moobilenet traffic from passing normally.
The wireless gateway was rebooted, and the backup was brought online to help alleviate recent high CPU issues. At this time all wireless traffic should be passing normally.
- Notification modified: 2006-05-15 08:38:15
Distauth Outage 12:10 - 12:20 05/12/2006
Distauth servers went down at 12:10 causing an outage for ten minutes while the servers restarted.
- Notification modified: 2006-05-12 12:44:40
MS Exchange Server patching 11:59PM Thur. 5/11
In response to a recent Microsoft Security Bulletin (MS06-019), the Campus Exchange Servers were patched and updated from Midnight to 12:30am (night of 05/11/2006). Momentary disconnects to mailboxes occurred as the cluster resource were moved, required by the security update.
- Notification modified: 2006-05-12 00:35:47
Campus IDS - Wednesday, May 10, 2006 20:00 - 2145
The Campus IDS experienced a hardware failure. The data collected prior to the downtime remains available via the Secalert web site.
- Notification modified: 2006-05-10 21:57:00
Network Outage 05/10/06 8:00 AM 8:45 AM
Effected Network:
IT Express was down for 45 min while a bad UPS was removed.
- Notification modified: 2006-05-10 08:58:12
SISWEB Registration Outage 5/08/2006, 06:00 -- 5/9/2006, 06:00
Summer 2006 Registration Alert! (updated: 9:15am - May 8, 2006)
The technical problem has been identified and corrected. To provide all
students an equal and fair opportunity to prepare for registration,
Summer Session One, Session Two, and Special Session registration
re-starts on Tuesday, May 9, 2006 at 6:00am. We sincerely apologize for
the inconvenience.
- Notification modified: 2006-05-09 05:54:41
Code Upgrade of Campus Firewall 5/2/06 6:30 am - 6:40am
NOC rebooted the firewalls to upgrade campus firewall. This affected off campus networks.
- Notification modified: 2006-05-02 06:45:45
Power Issues at UCDHS 05/01/06 11:05
Description: UCDHS is experiencing power issues. Connectivity to UCDHS was briefly interrupted.
Locations Affected: Campus connection to UCDHS.
Status: The campus connection to UCDHS was briefly interrupted due to power outages on the UCDHS side. All connectivity has been restored at this time.
Estimated Time to Repair: Connectivity restored.
- Notification modified: 2006-05-01 11:38:45
Scheduled Network Maintenance 04-28-2006 6:00pm - 7:00pm
The Network Operations Center performed scheduled network maintenance from 6:00pm - 7:00pm. Clients may have experienced momentary connectivity outages during that time.
- Notification modified: 2006-04-28 19:15:43
Brief Network Interruption 6:30 AM-6:40 AM 4/26/06
Description: The Code Upgrade for Campus Firewall was successful.
Impact: There was no interruption of traffic between campus and off-campus sites while the firewall was reloaded.
- Notification modified: 2006-04-26 21:06:06
DNS Server Maintenance 4/26/06 19:30-21:00
The 169.237.1.250 DNS server was down for maintenance.
- Notification modified: 2006-04-26 21:05:28
Network Maintenance Notice, 4/20/06 0700-0730
Description: Increased the capacity in Data Center Equipment
Start of Window: Thursday, 20 April 2006 7:00 AM
End of Window: Thursday, 20 April 2006 7:30 AM
Impact: There was no interruption in service. Had there been an interruption for any unforeseen reason the equipment would have required a reboot, an outage lasting approximately 5 minutes would have interrupted traffic to and from the Data Center.
- Notification modified: 2006-04-20 07:11:00
MeetingMaker Downtime Tuesday, 4/18/06, 8-10pm
Access to MeetingMaker accounts was not be available Tuesday evening, from 8:00pm to 10:00pm, due to maintenance.
- Notification modified: 2006-04-18 22:14:31
Campus IDS service was down.
From 1330hrs 04/12/2006 until 0800hrs 04/13/2006
System administrators resolved the problem.
- Notification modified: 2006-04-15 16:30:39
Mothra service unavailable: Tuesday, 4/11/06, 6:30 - 8:00 AM
The Mothra service will be down on Tuesday, April 11, from 6:30 AM until approximately 8:00 AM.
- Notification modified: 2006-04-11 07:15:26
Intermittent Network Outage Monday April 10, 2006 14:51- 15:23
Monday April 10, 2006 14:51 - 15:23
Description: Network Connectivity to campus hosts were being heavily impacted by a denial of service type attack.
Locations Affected: Campus wide.
Status: A network loop was discovered in Area 3. This caused a multicast storm which heavily disrupted traffic throughout campus. NOC staff disabled the offending link, and all traffic should be back to normal at this time.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2006-04-10 15:38:42
Planned Emergency outage of mail server, "magenta" 04/11/2006
Date: Tuesday, April 11, 2006
Time: midnight - 01:00
Description:
Mail server magenta.ucdavis.edu will be temporarily unavailable during
replacement of a faulty power supply.
Those that get their email from magenta.ucdavis.edu will not be able to
access it during this time period. No email will be lost because
incoming
email will queue up and be delivered once magenta.ucdavis.edu is back
online.
- Notification modified: 2006-04-11 01:31:58
Tan email server maintenance 0300 to 0500, Tuesday April 11
System Administrators will conduct testing and maintenance on the mail server Tan. During this time, e-mail will be unavailable to Tan's email users.
- Notification modified: 2006-04-11 04:27:03
Network Connectivity 04/06/06 08:00 AM - 08:35 AM
Multiple locations were effected. Network Operations fixed the problem.
Estimated Time to Repair: Done
- Notification modified: 2006-04-06 09:18:17
Network Connectivity
Locations Affected: The University Extension Building and surrounding areas, including 1850 Research Park Drive, and 1480 Drew.
Status: A hardware problem with the head end switch was identified. The device was replaced, and network connectivity has been restored.
Estimated Time to Repair: Repairs Complete
Outage time: 2006-04-05-12:53 to 2006-04-05-14:10
- Notification modified: 2006-04-05 14:34:49
Database Maintenance 04/06/06 06:30-07:00
The accounts database was taken down for a software upgrade on Thursday 04/06/06 from 06:30 – 07:00 AM.
During that time, the following services were unavailable:
* Computing Accounts web services
* Whois
* DNS changes.
- Notification modified: 2006-04-06 07:00:19
Sisweb down due to proble with file system 0641-0803 04/03/06
Sisweb was down due to a problem with a file system. System administrators were notified and they remedied the problem.
- Notification modified: 2006-04-03 08:09:14
tan unavailable due to hardware failure 08:00-22:40 04/01/2006
Mail server tan was unavailable due to a hardware failure. System administrators resolved the issue and tan is operating normally.
- Notification modified: 2006-04-02 23:15:16
MyUCDavis Outage 04/01/2006 07:00 - 16:55
Access to MyUCDavis was temporarily Disabled. Access was re-enabled once load had subsided.
While access to the MyUCDavis portal was disabled, the functionality of Classes and Geckomail continued through links at the MyUCDavis portal access page.
- Notification modified: 2006-04-01 16:59:46
SISWeb, MyUCDavis: Slow Response 03/29/2006 09:00 - 17:00
Services: SISWeb, MyUCDavis: Course Management, MySpace, MyAdmissions
Issue: Due to the heavy traffic from the beginning of quarter activity, sites such as SISWeb and MyUCDavis: Course Management, suffered slow response at times and new connections were not being opened properly.
- Notification modified: 2006-03-31 21:22:52
AFS UPGRADE - Tuesday 3/28, 2000-2300
A new system was added to the AFS pool of servers to accommodate new Courseware material for spring quarter. There was no service disruption.
- Notification modified: 2006-03-28 23:27:11
Accounts Database Downtime Notice: Tues 03/28/06 0700-0800
The accounts database was taken down for approximately one hour for a software upgrade. During this time, the following services were unavailable: * Computing Accounts web services * Whois * DNS changes.
- Notification modified: 2006-03-28 23:23:23
Type 1 Network Outage 03/27/2006 11:24 - 15:30
A denial of service attack stopped all connectivity to off campus sites.
NOC personnel blocked a number of hosts at the border which were initiating a large number of sessions to on campus machines. Despite the blocks, BGP sessions to our ISP routers did not come back up. Static routes were put in place. The BGP instability caused routes into campus to be dropped from the global internet. The NOC worked with CENIC to have these routes reinserted. Connectivity was restored soon after.
- Notification modified: 2006-03-27 15:44:23
Network Outage 03/22/06 17:45 - 19:15
The campus experienced a Type 1 Outage affecting the entire campus. Data connections to the Internet were unavailable. NOC staff isolated the failure to DWDM equipment at the UCD NOC.
- Notification modified: 2006-03-22 19:22:30
Slow acces to MyUCDavis resources 03/20/06 0700-2230
MyUCDavis was experiencing heavy load due to end-of-quarter finals and Undergraduate Admission Notifications.
- Notification modified: 2006-03-21 02:00:30
Dafis.ucdavis.edu website down. Decision Support accessible
The dafis.ucdavis.edu website was unavailable 07:00 10:00. However, clients were able to access Decision Support via dafisds.ucdavis.edu until access is restored. System administrators are working on the problem.
- Notification modified: 2006-03-20 10:13:08
Mail Server Purple Slow 03/16 14:30 - 03/16/2006 19:00
The mail server Purple was slow due to heavy use (mass mailing sent from studenthousing).
- Notification modified: 2006-03-16 20:10:06
Distauth and status inconsistency 03/15 14:30 - 03/16/2006 08:00
Distauth and computingaccounts.ucdavis.edu experienced slow response due to high load this translates into inconsistencies on the campus status page. This high load is caused by student applicants checking their admissions status information.
- Notification modified: 2006-03-16 20:04:09
Courseware and MyUCDavis 03/15 14:30 - 03/16/2006 03:00
Users experienced heavy load on the Courseware and MyUCDavis servers.
- Notification modified: 2006-03-16 20:05:29
Distauth and computing accounts 03/15 14:30 - 03/16/2006 08:00
Distauth and computingaccounts.ucdavis.edu experienced slow response due to high load. This high load was caused by student applicants checking their admissions status information.
- Notification modified: 2006-03-16 20:06:56
Intermittent off-campus network access 03/06/06 - 03/07/06
The campus experienced intermittent network connectivity to some off-campus sites. That problem has been resolved. If you should experience any additional network problems, please call the IET helpdesk at 754-4357.
- Notification modified: 2006-03-07 13:26:35
Dafis Decision Support down. 03/06/2006 08:00 - 08:20
Dafis Decision Support was unavailable due to server problems. System administrators were contacted and they resolved the issue.
- Notification modified: 2006-03-06 10:43:42
ITHELP WEBPAGE WAS UNAVAILABLE: March 4, 2006 13:40 - 21:38
ITExpress web page itexpress.ucdavis.edu is unavailable at this time due to technical problems. System Administrators are working on the problems. The web page will be available Monday, March 6, 2006.
- Notification modified: 2006-03-05 06:23:04
Emergency Outage of ISUN and mailbox 03/03/06, 22:00 - 22:45
The ISUN systems and mailbox.ucdavis.edu were shut down to replace a failing power supply in the ISUN Home server. Mailbox was shut down during the maintenance due to its dependence upon ISUN homes.
Those that receive their email from mailbox.ucdavis.edu, didn't loose them because incoming mail were queued up and delivered once the server came back online.
- Notification modified: 2006-03-03 23:36:31
Tuesday February 28, 2006 09:49
Description: A denial of service attack from off campus hosts has briefly disrupted traffic to some internet sites.
Locations Affected: Campus traffic to various internet sites.
Status: A denial of service attack from off campus hosts briefly disrupted traffic from campus to various internet sites. NOC personnel have blocked the offending hosts at the border. All traffic should be flowing normally at this time.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2006-02-28 10:06:03
Mail Server Blue Rebooted - Monday, February 27, 2006 15:10
Blue rebooted. System administrators investigated and diagnosed a hardware problem.
- Notification modified: 2006-02-27 18:25:00
Planned Emergency outage of Mail Server Blue 2/27/0 1700-1730
Blue.ucdavis.edu was shut down to replace a failing power supply from 1700-1730, on 2/17/06.
- Notification modified: 2006-02-27 18:26:16
Denial of service attack. Thursday, February 23, 2006 10:26
Description: A denial of service attack from campus hosts has briefly
disrupted traffic to some internet sites.
Locations Affected: Campus traffic to various internet sites.
Status: A denial of service attack from campus hosts briefly disrupted traffic from campus to various internet sites. NOC personnel have tracked down and disabled the offending hosts. All traffic should be flowing normally at this time.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2006-02-23 10:46:44
Connectivity Problems with www.ucdavis.edu 8:00 - 9:00 2/23/06
Access to www.ucdavis.edu was unavailable due to problems with a network device. System administrators were notified and resolved the problem.
- Notification modified: 2006-02-23 09:08:38
Listproc Down 02/17/2006 14:10 - 14:50
Listproc services kept failing due to a corrupted list configuration file. System administrators isolated the list in question and returned listproc services back to normal.
- Notification modified: 2006-02-17 15:20:39
Problems logging into Sisweb 02/17/2006 08:00 - 08:35
Clients reported problems logging on to sisweb.ucdavis.edu. System administrators were notified and they resolved the problem.
- Notification modified: 2006-02-17 09:04:15
Network Outage: Campus Traffic 2-16-2006-08:32 - 09:19
Thursday February 16, 2006 09:19
Description: Recurring problems from the previous few days have caused elevated CPU levels on core network hardware.
Locations Affected: Various campus locations.
Status: NOC personnel have performed a code rollback on the core layer 2 switches, per the findings of the hardware vendor. At this time, CPU levels have returned to normal. The NOC will continue to closely monitor the equipment, but no additional service interruptions are expected.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2006-02-16 10:03:00
Emergency Network Maintenance Notice
Description: Resolve network issues
Start of Window: Thursday, 16 February 2006 06:00 AM
End of Window: Thursday, 16 February 2006 07:00 AM
Purpose: Additional window to collect more information.
Working with network equipment vendor to determine
cause and resolution for recent campus outages
related to multicast/broadcast storms.
Impact: Potential for multiple brief outages during window
Affected Building: All Campus
- Notification modified: 2006-02-16 07:16:59
Several Data Center-hosted services experienced network outage
Thursday 02/16/2006 01:11 - 01:30
Services such as Geckomail, MyUCDavis & SIS/Banner, experienced a network related outage. This resolved itself and is being investigated as to the cause.
- Notification modified: 2006-02-16 12:26:24
Planned Distauth Outage: 2/18/06 23:00 -- 2/19/06 02:00
System Administrators will be moving and testing Distauth on new hardware. During this time, new logins to services that depend on Distauth such as MyUCDavis, Geckomail, and Courseware will not be available for up to three 15-minute windows.
- Notification modified: 2006-02-19 22:53:15
Network Outage: Campus Traffic 2-15-2006-11:50
A multicast storm was disrupting traffic to and from campus. Connectivity issues ranged from intermittent slowdowns to complete loss of connectivity. NOC personnel restored the core campus network.
- Notification modified: 2006-02-16 07:16:42
Reported network outages in several campus buildings
We received several reports of network outages in different buildings across campus. The Network Operations Center was apprised of the situation and personnel worked on the problem.
UPDATED: 08:40 02/14/2006
The Network Operations Center confirmed that several buildings on campus was experiencing connectivity problems. They stabilzed the network, but some clients still experienced intermittent connectivity throughout the day. NOC programmers worked with the vendor on a more permanent solution.
UPDATED: 07:00 02/15/06
NOC programmers performed tests this morning and resolve the connectivity issue.
- Notification modified: 2006-02-15 07:42:30
Mail Server Tan
Monday, 2/13/06, 02:30 to 03:10
The mail server "tan" is currently experiencing disk problems and is not in service.
System administrators are working on the issue.
- Notification modified: 2006-02-13 04:53:16
Network Service Interruption 2/10/2006 08:30 - 10:30
A multicast storm disrupted traffic to and from campus. Connectivity issues range from intermittent slowdowns to complete loss of connectivity. NOC personnel disabled the offending VLAN and CPU levels are down again. At this time all connectivity should be restored, with the exception of the WIRELESS-CAMPUS-2 general use wireless VLAN. NOC staff will continue to troubleshoot the problem on the isolated VLAN, and
will bring it back up when it no longer poses a threat to the core campus network.
- Notification modified: 2006-02-10 12:37:22
Campus Wireless Down 2/8/2006 10:40 - 11:40
The authentication server for Moobilenet was rebooted to reset socket. Connectivity for wireless users was down for few minutes.
- Notification modified: 2006-02-08 11:49:40
AFS issues 2006/02/06 19:00 - 2006/02/07 08:00
Due to AFS issues, services including but not limited to Geckomail and Course Management were temporarily unavailable. System administrators resolved the problem at around 07:20 02/07/2006.
- Notification modified: 2006-02-07 08:24:08
MyUCDavis and Geckomail unavailable 02/06/2006 13:10-15:20
Due to AFS server issues, MyUCDavis and Geckomail were unavailable for approximately 2 hours.
- Notification modified: 2006-02-06 15:54:45
Course Management unavailable 02/06/2006 13:10-15:20
The Course Management AFS volumes were unavailable for approximately 2 hours.
- Notification modified: 2006-02-06 15:54:30
Dafis Services Down 1/30/06 11:30 - 1735
All Dafis services were impacted by firewall configuration upgrades. All services were back to normal operation at 17:35.
- Notification modified: 2006-01-30 17:37:33
pop19/scarlet e-mail servers down on 1/28/06.
On Saturday January 28, 2006, at approximately 17:40, the scarlet e-mail server experienced a hardware failure and had to be powered down. System administrators were able to resolve the problem and service is restored as of 18:40.
- Notification modified: 2006-01-28 18:43:00
Alder Hall connectivity problems
There was a problem with the campus DHCP server. Users in Alder Hall were not able to get a valid UCD network address.
Operation restarted netid.
- Notification modified: 2006-01-27 11:03:16
Blackberry Service unavailable - 01/28/06 08:00-09:30
Description: Campus Blackberry Server Upgrade.
Start of Window: Saturday, 28 January 2006 08:00 (8:00 AM)
End of Window: Saturday, 28 January 2006 10:00 (10:00 AM)
Purpose: Upgrade and migrate services to new Blackberry Enterprise
Server.
Impact: Brief outage of Campus Blackberry service as the database and
system is reloaded.
Affected Users: All central campus Blackberry users only.
If you have questions or concerns regarding this event, please contact me or Campus Data Center Operations at 530-752-1566 email:
operators@ucdavis.edu
Please note: The maintenance and migration is NOT related to the current legal litigation involving RIM (Blackberry service provider).
- Notification modified: 2006-01-29 18:42:59
Banner System is experiecing intermittent hardware problems
The Banner system experienced intermittent hardware problems. Other services affected by this issue were Courseware, MyUCDavis, and Sisweb. System administrators resloved the problem with vendors.
- Notification modified: 2006-01-27 09:37:24
01/24/06 Zeus Down
Banner hardware rebooted itself, due to failing backup power supply.
- Notification modified: 2006-01-24 14:33:55
01/23/06 20:25-21:31 MySpace
MySpace was down because fileserver on home3 restarted. Other myucdavis services were affected by banner hardware rebooting.
- Notification modified: 2006-01-24 14:32:34
01/23/06 21:24-21:41 Zeus Down
Banner hardware rebooted itself, due to backup up power supply failing.
- Notification modified: 2006-01-24 14:33:18
Blue (Pop24) Downtime Notice 1/23/06 9:30PM - 10:30PM
Pop server blue was offline from 9:30PM (21:30) until 10:30PM (22:30) to replace failing hardware. Mail was queued for delivery until after the repair.
- Notification modified: 2006-01-23 22:37:37
DS Security HoneyPot Service unavailable 1/23/2006
The IDS Security HoneyPot Service was unavailable till 2006-01-23 2100. System administrators resolved the problem.
- Notification modified: 2006-01-23 22:11:05
Distauth slowness - January 22, 2006 2140-2200
Distauth experienced slow service.
- Notification modified: 2006-01-22 22:03:05
January 22, 2006 9:52 - 10:41 Degraded Network Connectivity
Network Connectivity was degraded. NOC said we were under an attack of denial of service. Everything is cleared up now.
- Notification modified: 2006-01-22 10:47:50
Course Management AFS volumes, and Distauth unavailable
Outage time for the Course Management volumes: 10:00am-12:30 am; Distauth: 10:00am-11:30 am.
Description: AFS class volumes and distributed Authentication services (distauth) were unavailable due to problems with the AFS servers.
Locations Affected: Campus-wide, any service that is dependent upon distauth services for AFS class volumes
Status: Data Center system adminstrators worked to resolved the problem with AFS. Authentication services and AFS class volumes were unavailable.
- Notification modified: 2006-01-20 20:26:50
All ISUN home directories will be transitioned to a new server.
01/20/06 22:00 to 01/21/06 08:00
All ISUN home directories were transitioned to a new server. During this time, the ISUN login servers and the email server mailbox were not available. Mail remained queued for delivery until after the migration was complete.
- Notification modified: 2006-01-21 17:44:09
Software.ucdavis.edu problems. January 10, 2006 09:57-1156
The Software Licensing site, software.ucdavis.edu, experienced high load and was not responding. System administrators resolved this issue.
- Notification modified: 2006-01-10 12:06:43
POP Server Azure unnavailable tonight (01/02 - 11:59PM to 1AM)
Downtime: Tuesday, 1/3/06, 00:00-00:09
The pop server Azure (pop18) received emergency maintenance by system administrators. The temporary outage was expected to last about 1 hour.
- Notification modified: 2006-01-20 20:29:46
Network Problem 01/01/05 08:15 - 08:25
The campus is experienced a very slow response to outside networks.
- Notification modified: 2006-01-02 08:33:52
ISUN & Mailbox Downtime
Wednesday, December 28, 2005 22:00 To Thursday, December 29, 2005 06:25
System administrators attempted to move all ISUN user home directories to a new server, but weren't successful. Another attempt will be scheduled later. Mail were queued for delivery after they were brought back online.
- Notification modified: 2005-12-29 06:39:03
E-mail List Processor Downtime
Tuesday, December 27, 2005 21:00 To 23:40
E-mail list processing was taken down for maintenance and was unavailable during this maintenance window.
- Notification modified: 2005-12-27 23:45:48
MyInfoVault experienced intermittent service.
On 12/24/2005, myInfoVault experienced intermittent service. As of 12/28/2005 16:15, services are stable again.
- Notification modified: 2005-12-28 16:18:35
UPGRADE OF MAIL SERVERS 12/23/05-22:00 TO 12/24/05 02:32
Yellow, azure, scarlet and tan were each down for some part of the maintenance window for disk upgrades to increase their storage capacity.
Mail were queued for delivery after the upgrades.
- Notification modified: 2005-12-24 03:13:33
AFS Outage
Friday, December 16, 2005 13:10-17:00
On Friday, December 16th, logins to campus services such as MyUCDavis and
Geckomail were unavailable for about two hours due to AFS server problems.
The AFS server problems were resolved and logins began working normally.
Subsequently, there were high loads on some web services, such as MyUCDavis,
which had to be mitigated by administrators.
Additionally, the DBA restarted the database to increase the connection limit
and removed lingering connections which were occupying slots unnecessarily.
- Notification modified: 2005-12-17 16:04:00
IET and Security websites were down
IET and Security websites were down
Thursday December 15, 2005 10:30 - 10:40
The following websites were affected by apache on pans.ucdavis.edu being down.
ac4.ucdavis.edu
adc3.ucdavis.edu
ccfit.ucdavis.edu
emailforwarding.ucdavis.edu
ftg.ucdavis.edu
ietsymposia.ucdavis.edu
itlc.ucdavis.edu
itpb.ucdavis.edu
itsecuritysymposium.ucdavis.edu
itstrategicplan.ucdavis.edu
itsymposia.ucdavis.edu
ittimes.ucdavis.edu
itweb.ucdavis.edu
lead.ucdavis.edu
middleware.ucdavis.edu
nti.ucdavis.edu
projectmanagement.ucdavis.edu
scg.ucdavis.edu
security.ucdavis.edu
securitytest.ucdavis.edu
snap.ucdavis.edu
techguide.ucdavis.edu
technews.ucdavis.edu
tif.ucdavis.edu
vpiet.ucdavis.edu
iplanet on pans was restarted.
- Notification modified: 2005-12-15 10:48:44
SMTP pool experienced severely heavy load
Tuesday 12/13/05 12:30 - Wednesday 12/14/05 09:00
The SMTP pool experienced severely heavy loads, causing intermittent delivery failure of messages.
- Notification modified: 2005-12-14 09:30:59
IET and Security websites are down
Monday December 12, 2005 09:05 - 10:30
The following websites were affected by apache on pans.ucdavis.edu being down.
ac4.ucdavis.edu
adc3.ucdavis.edu
ccfit.ucdavis.edu
emailforwarding.ucdavis.edu
ftg.ucdavis.edu
ietsymposia.ucdavis.edu
itlc.ucdavis.edu
itpb.ucdavis.edu
itsecuritysymposium.ucdavis.edu
itstrategicplan.ucdavis.edu
itsymposia.ucdavis.edu
ittimes.ucdavis.edu
itweb.ucdavis.edu
lead.ucdavis.edu
middleware.ucdavis.edu
nti.ucdavis.edu
projectmanagement.ucdavis.edu
scg.ucdavis.edu
security.ucdavis.edu
securitytest.ucdavis.edu
snap.ucdavis.edu
techguide.ucdavis.edu
technews.ucdavis.edu
tif.ucdavis.edu
vpiet.ucdavis.edu
- Notification modified: 2005-12-12 09:35:16
Type II Voice System Outage
Date: 12/06/05
Time: 09:05 - 9:15
Type II Voice System Outage
The campus experienced problems with incoming calls from off campus numbers.
SBC fixed a translation error occuring on their end. All has been resolved.
- Notification modified: 2005-12-06 09:29:33
SMTP SERVERS SLOW RESPONSE
Wednesday, November 23, 2005 11:25-14:29
The SMTP servers experienced high load due to incoming mail with viruses.
As a result, clients may have experienced delays sending mail. System
administrators modified system configurations to tune performance and
resolve the issue.
- Notification modified: 2005-11-23 14:40:26
Learning Website Down
Wednesday, November 23, 2005 08:09-10:30
The Learning website was down due to disk problems.
- Notification modified: 2005-11-23 14:40:01
ListProc outage
ListProc is currently unavailable. System administrators are working to restore service ASAP.
- Notification modified: 2005-11-22 05:51:35
Listproc Services are temporarily unavailable
ListProc was temporarily unavailable due to lack of disk space. System administrators restored service.
- Notification modified: 2005-11-21 19:28:34
Green Mail Down at 12:00
Due to high system load on green, System Administrators took green down for maintanance. After a modification in the quota size in the kernel, green's load has dramatically improved. System was down between 12:00 - 13:00.
- Notification modified: 2005-11-17 13:18:57
Mail server green.ucdavis.edu unavailable
Wednesday November 16, 2005 2300 - Thursday November 17, 2005 0236
Due to a hardware failure, green.ucdavis.edu underwent emergency maintenance. E-mail was unavailable to clients who were hosted on green during that time.
- Notification modified: 2005-11-17 02:44:17
Listproc e-mail list processing service
November 16, 2005 2230 - November 16, 2005 2350
Listproc e-mail list processing service was unavailable during a rebuild of its database.
- Notification modified: 2005-11-17 00:47:00
Tuesday November 15, 2005 11:51 - 12:57
Azure (mail server) was down due to hardware problems.
- Notification modified: 2005-11-15 13:05:18
Problem with homes server
November 8, 2005 11:30am - 11:59
A brief power spike at the data center caused a hardware failure for the ups powering home directories disk arrays. System administrators resynced the drives on pueblo to resolve the problem.
- Notification modified: 2005-11-08 12:49:02
Type I Network Outage
Wednesday, October 26, 2005 17:20
Description: Campus traffic to off external sites is experiencing heavy latency.
Locations Affected: All campus locations and off campus sites fed through campus connections.
Status: NOC personnel have located the cause of the high CPU issues on the centralized firewall. Campus hosts were sourcing a denial of service type attack which heavily impacted the firewall. The offending machines were disabled, and connectivity should be restored. Please contact IT Technical Support at 752-7656 if you continue to experience problems.
Estimated Time to Repair: Repairs Complete.
- Notification modified: 2005-10-26 18:01:15
Listproc is down.
October 25, 2005 11:10 - 11:49
Hardware failure is the cause. Nixon's drives were placed in another system and brought back up. All is well.
- Notification modified: 2005-10-25 12:26:23
Intermittent internet connectivity
10/25/05 1:15 - 7:25
We experienced intermittent internet connectivity. NOC was called in to solve this problem. All fine at this time.
- Notification modified: 2005-10-25 07:26:33
Scheduled Course Management maintenance
This Saturday, October 22, 2005, the servers that house the Course Management tools (i.e., GradeBook, Quiz Builder, Website Builder) within MyUCDavis will be upgraded and unavailable from 12 midnight until 10:00 AM.
Between 10:00 AM and approximately 8:00 PM that evening (i.e., until the upgrade is completed), you may experience an overall sluggish performance when using any of the Course Management tools. Other MyUCDavis applications and content will not be affected during this upgrade.
Once this upgrade is complete, a message will be posted in the MyUCDavis message center and you'll notice faster page loading and overall quicker functioning when using these tools.
If you have questions regarding this announcement or require additional assistance, please contact the IT Express Computing Help Desk at 530-754-HELP (4357).
- Notification modified: 2005-10-26 16:29:20
Email Server Downtime
Monday, October 17, 2005 22:00-23:00
Email server blue.ucdavis.edu is temporarily unavailable due to the replacement of a failing power supply.
- Notification modified: 2005-10-17 22:48:34
Honeypot Services Unavailable
10/17/05 12:01 - 10/18/05 10:48
Taffy (Honeypot services) was unavailable.
- Notification modified: 2005-10-18 11:15:38
Slow Response Time on UCDavis Homepage
10/17/05 17:07 - 10/18/05 13:43
The UCDavis homepage was experiencing a slow response time.
- Notification modified: 2005-10-18 13:52:28
Network Outage:Area 10 DSL
October 17, 2005 9:44 - 12:09
DSL connectivity to sites in Area 10 was interrupted.
Locations Affected: All DSL sites in Area 10 (South of 80 and various
other sites).
Status: The Area 10 DSLAM has stopped passing traffic. NOC technicians
have replaced the management card. NOC personnel rebuilt
the configuration, and connectivity has been restored.
- Notification modified: 2005-10-17 12:43:14
EMERGENCY DOWNTIME NOTICE 10/17/05 22:00-23:00
EMERGENCY DOWNTIME NOTICE
Monday, October 17, 2005 22:00-23:00
Description:
Email server blue.ucdavis.edu was temporarily unavailable during
replacement of a failing power supply.
Thanks!
- Notification modified: 2005-10-17 21:11:39
Azure mail server outage
Azure experienced Hardware failures between 4:45PM and 5:15PM on October 13th, 2005. System administrators were able to diagnose and resolve the problem.
- Notification modified: 2005-10-13 17:25:12
On Thursday, October 13th, at 10:00am to 11:00am, Azure.ucdavis.edu was unavailable due to server problems. Consequently, clients who utilize azure as their pop server, was unable to access their email inbox.
System administrators resolved the problem.
- Notification modified: 2005-10-13 11:08:04
Configuration of a web server updated.
Downtime Notice
Date: Saturday, October 8, 2005
Time: 20:00 - 21:00
Description: System administrators will update the configuration of a web server.
The following sites hosted by this server will briefly be unavailable:
computerownership.ucdavis.edu
dc.ucdavis.edu
distauth.ucdavis.edu
email-devel.ucdavis.edu
email.ucdavis.edu
iet-business.ucdavis.edu
ism.ucdavis.edu
khamel-devel.ucdavis.edu
w2k.ucdavis.edu
win2k.ucdavis.edu
windows2000.ucdavis.edu
windows.ucdavis.edu
------------------------------------
The actual dowtime will hopefully be very brief - only the time it takes for apache to restart.
- Notification modified: 2005-10-17 10:07:21
Mail server "tan" was DOWN.from 4:00am to 12:15pm
The mail server tan.ucdavis.edu was down due to hardware problems. The Field Engineer had to come swap out the "I/O Module" part of the primary RAID controller on one of the disk arrays.
- Notification modified: 2005-10-05 13:11:28
Type I Outage
Friday, September 30, 2005, 14:40-15:15
Description: ISP traffic to and from campus was experiencing heavy latency and packet loss
Locations Affected: All campus ISP bound traffic.
Status: A denial of service attack sourced from within the campus was disrupting ISP traffic throughout the network. Network Operations personnel identified and eliminated the source of the attack.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2005-09-30 15:24:47
Slow response for Authentication
The extreme load on DistAuth was due to bombardment from the Bluesocket wireless gateway.
In the short term, the NOC is pointing the Bluesocket to the NOC RADIUS server to handle authentication to the wireless network; however, users of the wireless network will now have to do a separate DistAuth authentication to access DA-protected resources, as there is (for now) no longer a unified login.
Although xyzzyd is still noticably busier than xyzzye, both systems are now functioning within fairly normal parameters
- Notification modified: 2005-09-29 11:41:54
Network Outage: Campus Wireless 9-29-2005 07:40 - 09:36
Thursday, September 29, 2005
Description: Wireless network connectivity throughout campus has been impacted.
Locations Affected: All campus moobilenet 802.11 wireless network sites.
Status: The certificate which allows users to authenticate with the campus wireless network had expired. The ISM group installed the new ssl certificate for secureweb.ucdavis.edu resolved the issue
- Notification modified: 2005-09-29 10:35:01
MyInfoVault
MyInfoVault was down for mainitenance from 6am to 7am.
- Notification modified: 2005-09-28 12:37:30
Isun oracle database down.
There is a power supply issue with veni.ucdavis.edu. This server is part of isun, and hosts the Oracle databases for students. System Administrators are working on the issue as quickly as possible.
- Notification modified: 2005-09-27 01:07:25
Active Directory Firewall Maintenance
Friday, September 23, 2005 8PM - 10PM
System administrators applied firmware updates to the Active Directory firewalls. The network may have been unavailable for short periods during the upgrade.
- Notification modified: 2005-09-23 23:02:46
Restarted Apache 20:00-20:15, Wednesday September 21, 2005
Due to updates, apache was restarted Wednesday, September 21, 2005, between 20:00-20:15. There was only a brief outage affecting the following sites:
assess.ucdavis.edu
computerownership.ucdavis.edu
dc.ucdavis.edu
distauth.ucdavis.edu
email-devel.ucdavis.edu
email.ucdavis.edu
iet-business.ucdavis.edu
ism.ucdavis.edu
khamel-devel.ucdavis.edu
scs.ucdavis.edu
w2k.ucdavis.edu
win2k.ucdavis.edu
windows2000.ucdavis.edu
windows.ucdavis.edu
- Notification modified: 2005-09-21 22:40:18
DSLam Down in Area 10
DSL was down in area 10 south of the freeway. The management card has been replaced and rebuilt the configuration. Everything is back to normal.
- Notification modified: 2005-09-21 12:48:37
IDS Service down.
The IDS part of the Vulnerability Scanning and Remediation Service is down. There is no ETR.
- Notification modified: 2005-09-19 22:45:35
Geckomail
The Geckomail Service is currently unavailable. Data Center staff is working on this issue.
- Notification modified: 2005-09-18 07:24:36
Power outage in area around Storer Hall
The Network Operations Center reported that the power was out in and around Storer hall. They have personnel working on the problem, but have no estimated time of repair/return to service.
- Notification modified: 2005-09-16 15:55:44
Update - Network Outage: Campus: Sep 16 2005 12:52-13:30
Description: Intermittent and slow network connectivity was been reported throughout campus.
Locations Affected: ISP traffic from all on and off campus data network connections.
Status: The NOC has isolated the cause of the continued latency problems. Corrective action was taken, and traffic should be normalized.
Estimated Time to Repair: Completed.
- Notification modified: 2005-09-16 13:36:39
Friday, September 16, 2005, 10:30
MyInfoVault experienced difficulties after system maintenance activities and was unavailable until 10:30 a.m.
- Notification modified: 2005-09-16 12:17:52
Intermittent and slow network connectivity
Friday, September 16, 2005, 1030 - 1200
Description: Intermittent and slow network connectivity has been reported throughout the campus.
Locations Affected: Data network connections to various buildings on and off campus.
Status: The new ISP firewall was maxing out its sessions. An additional firewall was put in place and traffic is being routed over it. Connections should begin to stabilize shortly.
Estimated Time to Repair: Repairs complete.
- Notification modified: 2005-09-16 12:16:50
MyInfoVault unavailable - Sep 16, 2005 0917-0937
MyInfoVault was unavailable. IET-ADDA programmers resolved the problem.
- Notification modified: 2005-09-16 10:17:00
ResNet unavailable - Sep 16, 2005 0800-0940
The Network Operations Center reported a Type II outage for ResNet. Consequently, ResNet access was unavaliable. Network Operations Center Personnel localized the problem to a firewall. As of 09:40 ResNet connectivity has been restored.
- Notification modified: 2005-09-16 10:17:20
Network Maintenance: Data Center: 2005-09-07-0700
====================== Courtesy Maintenance Notice ======================
Description: Courtesy Notice - Network Maintenance.
Start of Window: Wednesday, 07 September 2005 07:00 AM
End of Window: Wednesday, 07 September 2005 07:10 AM
Purpose: Router expansion. Adding router card to provide
additional Gigabit ports.
Impact: None. The cards are hot swappable.
Affected Building: CAAN BUILDING
4471 Data Center
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
+1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2005-09-09 06:43:05
Degraded Sevice: MyInfoVault - September 6 - 9, 2005
MyInfoVault was experiencing slow performance. The issue has been resolved. Thank you for your patience.
- Notification modified: 2005-09-15 08:05:36
Purple Emergency Downtime Notice
Scheduled downtime for purple.ucdavis.edu
this Saturday night, September 3, 2005, from 22:00-24:00.
This is for the urgent replacement of a failing power supply.
The actual downtime will hopefully be much
less than 2 hours.
- Notification modified: 2005-09-03 23:09:50
Type II Network Outage
Tuesday, August 30: 16:11
Description: A bridge loop interrupted network connectivity to multiple buildings across campus.
Locations Affected: Multiple buildings across campus.
Status: NOC personnel have located the source of the bridge loop and taken corrective action.
Estimated Time to Repair: Issue Resolved
- Notification modified: 2005-08-30 16:31:43
Type II Network Outage
The campus experienced a Type II network outage for area one. A bridge loop was repaired by the NOC after a brief outage between 15:00 and 15:20.
- Notification modified: 2005-08-30 15:29:44
Network Maintenance
======================= Network Maintenance Notice =======================
Description: Upgrade system software on the UCD Border router.
Remove decommissioned hardware from the router
chassis.
Start of Window: Wednesday, 24 August 2005 1900 ( 7:00 PM)
End of Window: Wednesday, 24 August 2005 2030 ( 8:30 PM)
Purpose: There is a known security vulnerability with the
current version of software running on the UCD
Border router. The software on the router will
be upgraded to address this issue.
Decommissioned OC-12 ATM interfaces will be
removed from service.
Existing Gigabit Ethernet interfaces will be
reconfigured to prepare for the installation of
a second campus border router.
Impact: No loss of connectivity is expected between the
UCD Campus and UCDMC. Connections between UCD
and the Internet will experience two 5-10 minute
interruptions in service when the router is
reloaded with the new software and ATM interfaces
are removed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you have questions or concerns regarding this event, please contact:
UC Davis Network Operations Center
+1.530.752.7656
noc@ucdavis.edu
- Notification modified: 2005-08-24 22:55:19
IT Express Knowledgebase Unavailable
The IT Express Knowledgebase web site http://xbase.ucdavis.edu was unavilable, due to a hardware failure. The service is once again available.
If you have any questions about this outage, please contact IT Express at 530-754-4357 or ithelp@ucdavis.edu during normal business hours.
- Notification modified: 2005-09-09 15:04:46
Past Outages
- Notification modified: 2005-08-18 02:52:48