Update @ 00:15 UTC: Network connectivity is back online. Total downtime was 9 minutes.
RFO from upstream:
One of our routers in our virtual-chassis reloaded unexpected and booted up with a different firmware version (the firmware at the second boot partition). Because we upgraded the firmware from 11.4 to 12.3 last week (during the emergency maintenance) the old 11.4 firmware was moved to the second boot partition. After the reloaded router came back online, we had one router running at 11.4 and one at 12.3. This caused a version mismatch and the routing features got disabled by this.
We upgraded to a new released firmware from 1 week ago to fix the unexpected reload issue and updated the second boot partition to a 12.3 firmware to prevent a version mismatch in the future.
We are sorry for the inconvenience.
Saturday, March 8, 2014