[EMERGENCY OUTAGE] AKL-IX | Vocus DC Albany | RS2.AKL | Switch Reboot
Scheduled Maintenance Report for New Zealand Internet Exchange
Completed
The scheduled maintenance has been completed.
Posted Dec 09, 2021 - 01:39 NZDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Dec 09, 2021 - 01:02 NZDT
Scheduled
NZIX Engineers will be rebooting a switch to resolve a management plane lock up that triggered tonight 9:26pm (NZDT). This bug is also impacting statistics on https://metrics.ix.nz/ for any service on this switch.


Summary of impact during this window:
* Members connected to pe2.akl2 should expect their port to go offline.

* Members connected to pe1.akl2 should expect BGP sessions to drop to both rs1.akl.ix.nz and rs2.akl.ix.nz. BCP214 compliant filters will be applied on member ports here.

* ALL Members should expect connectivity to rs2.akl.ix.asn.au to go down.

Expected Downtime: ~15 minutes
If Hardware Fault: up to 4 hours


Due to the nature of the software fault, we cannot appropriately install BCP214 compliant filters to ports on pe2.akl2 to reduce BGP disruption. This means ALL members that have routes via peers on pe2.akl2 may see a period of packet loss whilst BGP re-converges to alternate paths.

NZIX has spare hardware on standby in the unlikely event this is a hardware issue, thus a 4 hour window has been set.


Please contact noc@ix.nz or +64 9888 9221 if you have any queries regarding this notice.


Déjà vu is appropriate here, previously this issue triggered at DataCentre220 and at that time we attempted a process reset with the vendor but that failed and we had to reboot. We upgraded and discovered another nasty bug that we've reported to the vendor with a fix coming this month - not soon enough it seems...
Posted Dec 08, 2021 - 22:45 NZDT
This scheduled maintenance affected: NZIX (AKL-IX).