[EMERGENCY OUTAGE] CHC-IX | CCE | pe1.chc1 Switch Lock Up
Incident Report for New Zealand Internet Exchange
Resolved
This incident has been resolved, newer firmware has been loaded that resolves this bug. Maintenance notices will be booked in soon to apply this code across all NZIX.
Posted Mar 27, 2023 - 08:23 NZDT
Identified
NZIX Engineers have been alerted to a management plane lock up on our switch at Christchurch Central Exchange (CCE) in CHC-IX. An emergency reboot is underway to resolve this. This bug is also impacting statistics on https://metrics.ix.nz/ for any service on this switch.

Expected Downtime: ~5-10 minutes

Summary of impact during this window:
* Members connected to pe1.chc1 should expect their port to go offline.
* ALL Members at CHC-IX should expect connectivity to rs1.chc.ix.nz to go down

Due to the nature of the software fault, we cannot appropriately install BCP214 compliant filters to ports on the switch to reduce BGP disruption. This means ALL members that have routes via peers on pe1.chc1 may see a period of packet loss whilst BGP re-converges to alternate paths.
Posted Mar 27, 2023 - 08:12 NZDT
This incident affected: NZIX (CHC-IX).