Network Outage: The UC Davis Campus Data Center is experiencing a network outage.
Incident
Report for UC Davis
Resolved
A server caused high cpu on the Area 3 BDF that affected all colocation clients housed at the Data Center. The offending server was removed, and the BDF rebooted. Technicians monitored the situation throughout the day to ensure the BDF was operating normally.
No IET central services were affected by this outage.
No IET central services were affected by this outage.
Monitoring
======================== Network Outage Notice ====================
09/08/2018 08:08
Impact:
The co-location stack will lose connectivity for over 10 minutes as it reloads.
Locations Affected:
VLANS:
ACAD-PREP-PROG-1-FW
ACTG&FINAC-1-FW
ACTG&FINAC-2-FW
ACTG&FINAC-3-FW
ACTG&FINAC-5-FW
ACU-1-DMZ
ACU-1-FW
ADMIN-&-RESRC-MGMT-1-FW
ADMIS&OUTRCH-2-FW
ADMISSIONS-1-FW
ADVISING-SVS-1-DMZ
ALUMNI-AFFRS-1-FW
ALUMNI-AFFRS-2
ALUMNI-AFFRS-3
ANIMAL-SCI-1-FW
ANIMAL-SCI-2
ANIMAL-SCI-2-DMZ
ANR-1-FW
ANTHROPOLOGY-1-FW
ARCH-&-ENGR-1-FW
ARCH-&-ENGR-2
ARM-ALARMS-1-FW
ARM-ALARMS-1-NR
ARM-SAN-1-NR
ART-1-FW
ART-2-FW
ART-3-FW
AUSSC-1-FW
AVC-ENRLMT-SVCS-1-DMZ
AVC-ENRLMT-SVCS-1-FW
BANNER-1-FW
BIO-SCI-1-DMZ
CALSPACE-CNTR-1
CALSPACE-CNTR-2
CAMPUS-EV&IN-1-FW
CASHIER-1-FW
CBST-1
CEPRAP-1-FW
CGM-3-NR
CHANCELLOR-11-12-DMZ
CHANCELLOR-11-FW
CHANCELLOR-12-FW
CHANCELLOR-1-FW
CHANCELLOR-2
CHANCELLOR-3
CHANCELLOR-4
CHANCELLOR-5-FW
CHANCELLOR-6-FW
CHANCELLOR-9-FW
CHANCELLOR-DMZ
CLTC-1-DMZ
COE-ITSS-1-FW
CROC-NUC-LAB-1
CTR-AG-ISSUE-1-FW
CTR-COMP-SCI/EN-1
CTR-CPANHLTH-2-FW
CTR-MIND-BRAIN-1-FW
CTR-MIND-BRAIN-3-FW
CTR-MIND-BRAIN-5-FW
CTR-NEUROSCI-1-FW
CULTURE-STDS-1-FW
DANR-1-FW
DANR-2-FW
DANR-3-FW
DANR-4-FW
DANR-5-FW
DANR-6-FW
DANR-7-FW
DANR-COMMUNI-1-FW
DANR-COMMUNI-2-FW
DANR-DMZ
DC-CLIENT-SVC-12-DMZ
DC&CLIENT-SVC-12-FW
DC&CLIENT-SVC-36-FW
DC&CLIENT-SVC-43
DC&CLIENT-SVC-46
DC&CLIENT-SVC-51-DMZ
DC&CLIENT-SVC-51-FW
DC&CLIENT-SVC-54-FW
DC&CLIENT-SVC-94
DC-COLO-1-DMZ
DC-COLO-1-FW
DEAN-AG-1-DMZ
DEAN-AG-1-FW
DEAN-AG-2-DMZ
DEAN-AG-2-FW
DEAN-AG-3
DEAN-AG-4-FW
DEAN-AG-5-FW
DEAN-AG-6-FW
DEAN-AG-7-FW
DEAN-BIO-SCI-1-FW
DEAN-ENGIN-1
DEAN-ENGIN-5
DEAN-L&S-1-FW
DEAN-L&S-2-FW
DEAN-L&S-4-FW
DEAN-L&S-6
DEAN-L&S-7-DMZ
DEAN-L&S-7-FW
DEAN-L&S-DMZ
DEAN-MED-ADM-11-NR
DEAN-MED-ADM-12-FW
DEAN-MED-ADM-13-FW
DEAN-MED-ADM-14-FW
DEAN-MED-ADM-15-FW
DEAN-MED-ADM-1-FW
DEAN-MED-ADM-2-FW
DEAN-MED-ADM-3-FW
DEAN-MED-ADM-4-FW
DEAN-MED-ADM-5-DMZ
DEAN-MED-ADM-5-FW
DEAN-MED-ADM-6-FW
DEAN-MED-ADM-7-FW
DEAN-MED-ADM-8-FW
DEAN-MED-ADM-9-FW
DEAN-VETMED-1-DMZ
DEAN-VETMED-1-FW
DESIGN-1-FW
DESIGN-2-FW
DESIGN-LDA-DMZ
DOE-1-FW
DRAMATIC-ART-1-FW
ECO-1-NR
ECO-2-NR
ECO-3-NR
ECO-4-NR
ECO-5-NR
ECONOMICS-1-FW
ECONOMICS-2-FW
EDUCATION-1-DMZ
EDUCATION-1-FW
EDUCATION-2-FW
EMPHLTHOAK-1
ENG-CHEM&MAT-1
ENG-CMPR-SCI-7
ENV-HLTH&SAF-1-FW
ENV-STEWARD-1-NR
ENV-STUDIES-3-FW
ENV-STUDIES-4
ENV-TOX-1-FW
EOC-1-FW
EVOL&ECOLOGY-1-FW
FAC-ADV-INST-1-DMZ
FAC-ADV-INST-1-FW
FIN-AID&EMPL-1-FW
FIN-AID&EMPL-3-FW
FIRE-DEPT-2-FW
FLEET-1-FW
FPMS-1-DMZ
FPMS-1-FW
GEN-ACAD-1
GEN-ACAD-1-DMZ
GEN-ACAD-2-FW
GENOME-CTR-4-FW
GEOLOGY-1-FW
GRAD-STUDIES-1-2-DMZ
GRAD-STUDIES-1-FW
GRAD-STUDIES-2-FW
HISTORY-2-FW
HPC-DC-1-DMZ
HR&RISK-MGMT-1-DMZ
HR&RISK-MGMT-1-FW
HR&RISK-MGMT-6-FW
HVAC-1-FW
HVAC-2-FW
IDAV-CSE-1-FW
IET-EAIS-SANDBOX-1-FW
INFORMATICS-1-DMZ
INFO-TECH-SVCS-1-NR
INFO-TECH-SVCS-2-NR
INFO-TECH-SVCS-3-NR
IN-GOVT-AFF-1-FW
IN-GOVT-AFF-2-FW
IN-GOVT-AFF-3-FW
IN-GOVT-AFF-4-FW
INST-TRNS-ST-1
INST-TRNS-ST-2-FW
INTRCOL-ATHL-1
INTRCOL-ATHL-3-FW
INTRN&CR-CTR-1-FW
IPM-PROJECT-1-FW
ISO-1-DMZ
ISO-FORENSICS-1-FW
ISO-LAB-1-FW
ISO-SOC-1-FW
IT-COMMUNRES-13-FW
IT-COMMUNRES-1-FW
IT-COMMUNRES-2-FW
IT-COMMUNRES-2-FW
IT-CRETV-2-FW
IT-CRETV-3-FW
IT-CRETV-COM-1-DMZ
IT-CRETV-COM-1-FW
IT-INFO-RES-17-FW
IT-INFO-RES-19-20-DMZ
IT-INFO-RES-19-FW
IT-INFO-RES-20-FW
IT-INFO-RES-26-FW
IT-SEC-1-FW
JMIE-1
JMIE-2-FW
JMIE-3-FW
LANG-LIT-1-FW
LANG-LIT-2-FW
LANGUAGE-LAB-1-FW
LARRY-1-DMZ
LAW-1-FW
LAW-2-FW
LAW-3-DMZ
LAW-3-FW
LAW-CLSSRM-NR
LAWR-1-FW
LAWR-3
LAWR-4-NR
LAW-SECURITY-NR
LDA-3-FW
LIGHTING-1-NR
LIGHTING-2-NR
LS-1-DMZ
MCCLELLAN-NRC-1-DMZ
MCCLELLAN-NRC-1-FW
M-COM-HEALTH-1-2-DMZ
M-COM-HEALTH-1-FW
M-COM-HEALTH-2-FW
M-CURIC-SUPT-1-DMZ
M-CURIC-SUPT-1-FW
MEYER-COMM-1
MGMT-SCH-OF-1-DMZ
MGMT-SCH-OF-1-FW
MGMT-SCH-OF-2-FW
MICROBIOLOGY-1-FW
M-NEUROLOGY-1-FW
M-NEUROSURG-1
M-NEUROSURG-2
MOL&CELL-BI0-1-FW
MONDAVI-1-FW
MONDAVI-2-FW
MONDAVI-4-NR
MONDAVI-5
MU-GENERAL-1
MU-GENERAL-2
MU-GENERAL-3
MUSIC-1-FW
NEAT-ORU-1-FW
NEUROPHY-BEH-1-FW
NMR-1-DMZ
NMR-FACILITY-1-FW
NUTRITION-1-FW
OCM-1-FW
O&M:UTILITIES-1
O&M:UTILITIES-2
O-M:UTILITIES-DMZ
PARKING-SVCS-1-FW
PARKING-SVCS-2-FW
PD-VIDEO-1-FW
PD-VIDEO-1-NR
PHYSICS-1-FW
PHYSICS-2-FW
PHYSICS-4-FW
PHYS-PLANT-1-FW
PHYS-PLANT-2-FW
PHYS-PLANT-3-FW
PHYS-PLANT-4-FW
PHYS-PLANT-5-FW
PHYS-PLANT-6-FW
PLANT-BIOLOGY-1-FW
PLANT-PATH-4
PLANT-SCIENCES-11-NR
PLANT-SCIENCES-8-FW
POLICE-3-FW
POLICE-4-FW
POLIT-SCI-1-FW
POWER-1-NR
PRIM-RES-CTR-1-FW
PRIM-RES-CTR-4-FW
PSYCHOLOGY-1-FW
PSYCHOLOGY-2-FW
PSYCHOLOGY-3-FW
PUB-COMMUNIC-1-FW
REC-HALL-2-FW
REC-HALL-3
REGISTRAR-1-FW
REGISTRAR-2-FW
SAFETY-SVCS-1-DMZ
SAFETY-SVCS-1-FW
SAFETY-SVCS-2-FW
SIP-2
SOCIOLOGY-1-FW
SOC-SCI-ADM-1-FW
SOC-SCI-DSS-1-FW
SOC-SCI-DSS-2-FW
SOC-SCI-DSS-3-FW
SOC-SCI-DSS-4-FW
STDT-JUD-AFF-1-FW
STUDENT-AFFRS-1-FW
STUDENT-AFFRS-2-NR
STUDENT-AFFRS-3-FW
STUDENT-AFFRS-4-DMZ
STUDENT-AFFRS-4-FW
STUDENT-AFFRS-5-FW
STUDENT-AFFRS-6-FW
STUDENT-AFFRS-7-FW
STUDENT-AFFRS-8-NR
STUDENT-AFFRS-9-FW
STU-DIS-CTR-1-FW
UCD-MNRC-1
UCD-VPN-1-DMZ
UC-LANG-CONS-1-FW
U-EXTENSION-3-FW
U-EXTENSION-5-FW
U-RELATIONS-1-4-DMZ
U-RELATIONS-1-FW
U-RELATIONS-4-FW
USDA-2-FW
UTILITIES-1-VPN
VC-RESEARCH-1-DMZ
VC-RESEARCH-1-FW
VC-RESEARCH-3-DMZ
VC-RESEARCH-7-FW
VC-STU-AFFRS-2
VC-STU-AFFRS-3-FW
VM-GEN-I&R-5
VM-RESEARCH-1-FW
VM-RESEARCH-2-FW
WASTEWATER-1-FW
WATER-1-NR
W-&-F-BIO-1-FW
Status:
After a late-night multicast packet-storm, some client networks did not return to normal after the storm was mitigated.
Hosts and firewalls have been rebooted, so rebooting the co-location stack is the next logical progression.
09/08/2018 08:08
Impact:
The co-location stack will lose connectivity for over 10 minutes as it reloads.
Locations Affected:
VLANS:
ACAD-PREP-PROG-1-FW
ACTG&FINAC-1-FW
ACTG&FINAC-2-FW
ACTG&FINAC-3-FW
ACTG&FINAC-5-FW
ACU-1-DMZ
ACU-1-FW
ADMIN-&-RESRC-MGMT-1-FW
ADMIS&OUTRCH-2-FW
ADMISSIONS-1-FW
ADVISING-SVS-1-DMZ
ALUMNI-AFFRS-1-FW
ALUMNI-AFFRS-2
ALUMNI-AFFRS-3
ANIMAL-SCI-1-FW
ANIMAL-SCI-2
ANIMAL-SCI-2-DMZ
ANR-1-FW
ANTHROPOLOGY-1-FW
ARCH-&-ENGR-1-FW
ARCH-&-ENGR-2
ARM-ALARMS-1-FW
ARM-ALARMS-1-NR
ARM-SAN-1-NR
ART-1-FW
ART-2-FW
ART-3-FW
AUSSC-1-FW
AVC-ENRLMT-SVCS-1-DMZ
AVC-ENRLMT-SVCS-1-FW
BANNER-1-FW
BIO-SCI-1-DMZ
CALSPACE-CNTR-1
CALSPACE-CNTR-2
CAMPUS-EV&IN-1-FW
CASHIER-1-FW
CBST-1
CEPRAP-1-FW
CGM-3-NR
CHANCELLOR-11-12-DMZ
CHANCELLOR-11-FW
CHANCELLOR-12-FW
CHANCELLOR-1-FW
CHANCELLOR-2
CHANCELLOR-3
CHANCELLOR-4
CHANCELLOR-5-FW
CHANCELLOR-6-FW
CHANCELLOR-9-FW
CHANCELLOR-DMZ
CLTC-1-DMZ
COE-ITSS-1-FW
CROC-NUC-LAB-1
CTR-AG-ISSUE-1-FW
CTR-COMP-SCI/EN-1
CTR-CPANHLTH-2-FW
CTR-MIND-BRAIN-1-FW
CTR-MIND-BRAIN-3-FW
CTR-MIND-BRAIN-5-FW
CTR-NEUROSCI-1-FW
CULTURE-STDS-1-FW
DANR-1-FW
DANR-2-FW
DANR-3-FW
DANR-4-FW
DANR-5-FW
DANR-6-FW
DANR-7-FW
DANR-COMMUNI-1-FW
DANR-COMMUNI-2-FW
DANR-DMZ
DC-CLIENT-SVC-12-DMZ
DC&CLIENT-SVC-12-FW
DC&CLIENT-SVC-36-FW
DC&CLIENT-SVC-43
DC&CLIENT-SVC-46
DC&CLIENT-SVC-51-DMZ
DC&CLIENT-SVC-51-FW
DC&CLIENT-SVC-54-FW
DC&CLIENT-SVC-94
DC-COLO-1-DMZ
DC-COLO-1-FW
DEAN-AG-1-DMZ
DEAN-AG-1-FW
DEAN-AG-2-DMZ
DEAN-AG-2-FW
DEAN-AG-3
DEAN-AG-4-FW
DEAN-AG-5-FW
DEAN-AG-6-FW
DEAN-AG-7-FW
DEAN-BIO-SCI-1-FW
DEAN-ENGIN-1
DEAN-ENGIN-5
DEAN-L&S-1-FW
DEAN-L&S-2-FW
DEAN-L&S-4-FW
DEAN-L&S-6
DEAN-L&S-7-DMZ
DEAN-L&S-7-FW
DEAN-L&S-DMZ
DEAN-MED-ADM-11-NR
DEAN-MED-ADM-12-FW
DEAN-MED-ADM-13-FW
DEAN-MED-ADM-14-FW
DEAN-MED-ADM-15-FW
DEAN-MED-ADM-1-FW
DEAN-MED-ADM-2-FW
DEAN-MED-ADM-3-FW
DEAN-MED-ADM-4-FW
DEAN-MED-ADM-5-DMZ
DEAN-MED-ADM-5-FW
DEAN-MED-ADM-6-FW
DEAN-MED-ADM-7-FW
DEAN-MED-ADM-8-FW
DEAN-MED-ADM-9-FW
DEAN-VETMED-1-DMZ
DEAN-VETMED-1-FW
DESIGN-1-FW
DESIGN-2-FW
DESIGN-LDA-DMZ
DOE-1-FW
DRAMATIC-ART-1-FW
ECO-1-NR
ECO-2-NR
ECO-3-NR
ECO-4-NR
ECO-5-NR
ECONOMICS-1-FW
ECONOMICS-2-FW
EDUCATION-1-DMZ
EDUCATION-1-FW
EDUCATION-2-FW
EMPHLTHOAK-1
ENG-CHEM&MAT-1
ENG-CMPR-SCI-7
ENV-HLTH&SAF-1-FW
ENV-STEWARD-1-NR
ENV-STUDIES-3-FW
ENV-STUDIES-4
ENV-TOX-1-FW
EOC-1-FW
EVOL&ECOLOGY-1-FW
FAC-ADV-INST-1-DMZ
FAC-ADV-INST-1-FW
FIN-AID&EMPL-1-FW
FIN-AID&EMPL-3-FW
FIRE-DEPT-2-FW
FLEET-1-FW
FPMS-1-DMZ
FPMS-1-FW
GEN-ACAD-1
GEN-ACAD-1-DMZ
GEN-ACAD-2-FW
GENOME-CTR-4-FW
GEOLOGY-1-FW
GRAD-STUDIES-1-2-DMZ
GRAD-STUDIES-1-FW
GRAD-STUDIES-2-FW
HISTORY-2-FW
HPC-DC-1-DMZ
HR&RISK-MGMT-1-DMZ
HR&RISK-MGMT-1-FW
HR&RISK-MGMT-6-FW
HVAC-1-FW
HVAC-2-FW
IDAV-CSE-1-FW
IET-EAIS-SANDBOX-1-FW
INFORMATICS-1-DMZ
INFO-TECH-SVCS-1-NR
INFO-TECH-SVCS-2-NR
INFO-TECH-SVCS-3-NR
IN-GOVT-AFF-1-FW
IN-GOVT-AFF-2-FW
IN-GOVT-AFF-3-FW
IN-GOVT-AFF-4-FW
INST-TRNS-ST-1
INST-TRNS-ST-2-FW
INTRCOL-ATHL-1
INTRCOL-ATHL-3-FW
INTRN&CR-CTR-1-FW
IPM-PROJECT-1-FW
ISO-1-DMZ
ISO-FORENSICS-1-FW
ISO-LAB-1-FW
ISO-SOC-1-FW
IT-COMMUNRES-13-FW
IT-COMMUNRES-1-FW
IT-COMMUNRES-2-FW
IT-COMMUNRES-2-FW
IT-CRETV-2-FW
IT-CRETV-3-FW
IT-CRETV-COM-1-DMZ
IT-CRETV-COM-1-FW
IT-INFO-RES-17-FW
IT-INFO-RES-19-20-DMZ
IT-INFO-RES-19-FW
IT-INFO-RES-20-FW
IT-INFO-RES-26-FW
IT-SEC-1-FW
JMIE-1
JMIE-2-FW
JMIE-3-FW
LANG-LIT-1-FW
LANG-LIT-2-FW
LANGUAGE-LAB-1-FW
LARRY-1-DMZ
LAW-1-FW
LAW-2-FW
LAW-3-DMZ
LAW-3-FW
LAW-CLSSRM-NR
LAWR-1-FW
LAWR-3
LAWR-4-NR
LAW-SECURITY-NR
LDA-3-FW
LIGHTING-1-NR
LIGHTING-2-NR
LS-1-DMZ
MCCLELLAN-NRC-1-DMZ
MCCLELLAN-NRC-1-FW
M-COM-HEALTH-1-2-DMZ
M-COM-HEALTH-1-FW
M-COM-HEALTH-2-FW
M-CURIC-SUPT-1-DMZ
M-CURIC-SUPT-1-FW
MEYER-COMM-1
MGMT-SCH-OF-1-DMZ
MGMT-SCH-OF-1-FW
MGMT-SCH-OF-2-FW
MICROBIOLOGY-1-FW
M-NEUROLOGY-1-FW
M-NEUROSURG-1
M-NEUROSURG-2
MOL&CELL-BI0-1-FW
MONDAVI-1-FW
MONDAVI-2-FW
MONDAVI-4-NR
MONDAVI-5
MU-GENERAL-1
MU-GENERAL-2
MU-GENERAL-3
MUSIC-1-FW
NEAT-ORU-1-FW
NEUROPHY-BEH-1-FW
NMR-1-DMZ
NMR-FACILITY-1-FW
NUTRITION-1-FW
OCM-1-FW
O&M:UTILITIES-1
O&M:UTILITIES-2
O-M:UTILITIES-DMZ
PARKING-SVCS-1-FW
PARKING-SVCS-2-FW
PD-VIDEO-1-FW
PD-VIDEO-1-NR
PHYSICS-1-FW
PHYSICS-2-FW
PHYSICS-4-FW
PHYS-PLANT-1-FW
PHYS-PLANT-2-FW
PHYS-PLANT-3-FW
PHYS-PLANT-4-FW
PHYS-PLANT-5-FW
PHYS-PLANT-6-FW
PLANT-BIOLOGY-1-FW
PLANT-PATH-4
PLANT-SCIENCES-11-NR
PLANT-SCIENCES-8-FW
POLICE-3-FW
POLICE-4-FW
POLIT-SCI-1-FW
POWER-1-NR
PRIM-RES-CTR-1-FW
PRIM-RES-CTR-4-FW
PSYCHOLOGY-1-FW
PSYCHOLOGY-2-FW
PSYCHOLOGY-3-FW
PUB-COMMUNIC-1-FW
REC-HALL-2-FW
REC-HALL-3
REGISTRAR-1-FW
REGISTRAR-2-FW
SAFETY-SVCS-1-DMZ
SAFETY-SVCS-1-FW
SAFETY-SVCS-2-FW
SIP-2
SOCIOLOGY-1-FW
SOC-SCI-ADM-1-FW
SOC-SCI-DSS-1-FW
SOC-SCI-DSS-2-FW
SOC-SCI-DSS-3-FW
SOC-SCI-DSS-4-FW
STDT-JUD-AFF-1-FW
STUDENT-AFFRS-1-FW
STUDENT-AFFRS-2-NR
STUDENT-AFFRS-3-FW
STUDENT-AFFRS-4-DMZ
STUDENT-AFFRS-4-FW
STUDENT-AFFRS-5-FW
STUDENT-AFFRS-6-FW
STUDENT-AFFRS-7-FW
STUDENT-AFFRS-8-NR
STUDENT-AFFRS-9-FW
STU-DIS-CTR-1-FW
UCD-MNRC-1
UCD-VPN-1-DMZ
UC-LANG-CONS-1-FW
U-EXTENSION-3-FW
U-EXTENSION-5-FW
U-RELATIONS-1-4-DMZ
U-RELATIONS-1-FW
U-RELATIONS-4-FW
USDA-2-FW
UTILITIES-1-VPN
VC-RESEARCH-1-DMZ
VC-RESEARCH-1-FW
VC-RESEARCH-3-DMZ
VC-RESEARCH-7-FW
VC-STU-AFFRS-2
VC-STU-AFFRS-3-FW
VM-GEN-I&R-5
VM-RESEARCH-1-FW
VM-RESEARCH-2-FW
WASTEWATER-1-FW
WATER-1-NR
W-&-F-BIO-1-FW
Status:
After a late-night multicast packet-storm, some client networks did not return to normal after the storm was mitigated.
Hosts and firewalls have been rebooted, so rebooting the co-location stack is the next logical progression.
Update
Network engineers and system administrators are continuing to investigate the network issue affecting Co-location systems.
Update
We are continuing to investigate this issue.
Update
The outage is affecting Data Center Colocation clients. The scope of the outage has been minimized, but the investigation is still ongoing.
Update
We are continuing to investigate this issue.
Investigating
Impact: UC Davis Data Center Network
Locations Affected: UC Davis Data Center
Status: Investigating
----------------------------
IT Express Service Desk
ithelp@ucdavis.edu
530.754.HELP (4357)
Locations Affected: UC Davis Data Center
Status: Investigating
----------------------------
IT Express Service Desk
ithelp@ucdavis.edu
530.754.HELP (4357)
This incident affected: Data Center Colocation.