Packet Loss (Resolved)
  • Priority - Critical
  • Affecting System - SSDKVM03NL
  • We're investigating packet loss on SSDKVM03NL

    Update @ 11:07 UTC: Things should be back to normal again. We'll monitor the server for the next few hours.

  • Date - 02/05/2014 10:43 - 02/05/2014 12:37
  • Last Updated - 02/05/2014 11:09
Server Down - cPanel web1 (Resolved)
  • Priority - Critical
  • Affecting Server - web1.fusioned.net
  • cPanel web1 experienced an extended downtime (4:30hrs) due to a kernel panic. As the server was replying to pings, Pingdom didn't fire off an alert so it wasn't until a customer opened a ticket that we realized the server was not responsive.
    We're deeply sorry for the inconvenience, and are already working on setting up additional monitoring probes which will monitor individual services (http, mysql, mail etc) for this not to happen again in the future.

  • Date - 26/02/2014 03:15 - 26/02/2014 07:45
  • Last Updated - 26/02/2014 10:47
Kernel Panic - cPanel web1 (Resolved)
  • Priority - Critical
  • Affecting Server - web1.fusioned.net
  • Rebooting "web1" after a kernel panic. We are working to stabilize the machine.

    Update @ 18:31 UTC: Server is back up and running.
    Update @ 01:09 UTC: Forced reboot due to kernel panic. Services were restored after 4 minutes. We are still working to stabilize the server.
    Update @ 02:14 UTC: All clear

  • Date - 27/01/2014 18:26 - 28/01/2014 02:14
  • Last Updated - 05/02/2014 12:21
Network Interuption - cPanel web1 (Resolved)
  • Priority - Critical
  • Affecting Server - web1.fusioned.net
  • The NIC for cPanel server "web1" is currently being replaced for better performance and stability.
    Services will resume within the next 10 minutes.

    Update 1 @ 10:23 UTC: Server is back online.
    Update 2 @ 10:48 UTC: The NIC was replaced, but the issue hasn't been resolved. Server has been shut down and is getting it's motherboard replaced.
    Update 3 @ 11:22 UTC: After replacing the motherboard, the server seems to be running stable now.

    We are sorry for the inconvenience this may have caused.

  • Date - 27/01/2014 10:12 - 27/01/2014 11:36
  • Last Updated - 27/01/2014 11:36
Network hiccup for cPanel web1 (Resolved)
  • Priority - Critical
  • Affecting Server - web1.fusioned.net
  • We experienced a network hiccup on Enterprise Hosting server "web1". Everything is back to normal again.

  • Date - 25/01/2014 23:02 - 25/01/2014 23:04
  • Last Updated - 25/01/2014 23:04
Enterprise Hosting DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - web1.fusioned.net
  • We have mitigated a DDoS attack against web1.fusioned.net

  • Date - 29/10/2013 23:14 - 29/10/2013 23:20
  • Last Updated - 29/10/2013 23:23
SATAKVM01DE Reboot (Resolved)
  • Priority - Critical
  • Affecting System - SATAKVM01DE
  • SATAKVM01DE locked up a few minutes ago. We have rebooted it and are investigating the cause.

  • Date - 25/10/2013 14:25 - 25/10/2013 14:34
  • Last Updated - 25/10/2013 14:27
SolusVM Temporarily Down (Resolved)
  • Priority - Critical
  • Affecting System - SolusVM Control Panel
  • We're currently troubleshooting a network connectivity issue with our SolusVM control panel.

    Update @ 15:38 UTC: SolusVM is back up again.

  • Date - 17/10/2013 15:30 - 17/10/2013 15:43
  • Last Updated - 17/10/2013 15:43
SSDKVM02NL Down (Resolved)
  • Priority - Critical
  • Affecting System - SSDKVM02NL
  • SSDKVM02NL suffered a kernel panic this morning and had to be manually restarted. We're investigating the root cause of this issue.

    Update: We checked the IPMI log and found Uncorrectable Memory errors for one of the RAM modules. An emergency hardware replacement has been scheduled.

  • Date - 09/10/2013 06:12 - 09/10/2013 07:32
  • Last Updated - 09/10/2013 09:25
SolusVM Master Down (Resolved)
  • Priority - Critical
  • Affecting System - SolusVM Master
  • We are seeing network connectivity issues with our SolusVM Master (core.fusioned.net). We're investigating the issue and will update you again.

    Update [13:22 UTC]: core.fusioned.net is back online.

  • Date - 28/09/2013 13:15 - 28/09/2013 13:22
  • Last Updated - 28/09/2013 13:23
DDoS - SSDKVM03NL (Resolved)
  • Priority - Critical
  • Affecting System - SSDKVM03NL
  • We have mitigated a DDoS attack against an IP on SSDKVM03NL.

  • Date - 20/09/2013 01:11 - 20/09/2013 01:17
  • Last Updated - 20/09/2013 01:37
Emergency Network Maintenance (Resolved)
  • Priority - Critical
  • Affecting Other - NL Network
  • We have received notification of an upcoming emergency maintenance in which a necessary firmware and software update will be performed at one of our carriers.

    More information can be found in our Announcements page: https://my.fusioned.net/announcements/19/Emergency-Network-Maintenance.html

    Update: The maintenance has completed without any downtime.

  • Date - 18/09/2013 00:00 - 18/09/2013 04:00
  • Last Updated - 18/09/2013 13:27
R1Soft CDP Server Maintenance (Resolved)
  • Priority - Critical
  • Affecting System - backup01.fusioned.net
  • Dear all,

    We're currently migrating backup01.fusioned.net to new hardware.

    Backup policies are disabled temporary, and will be automatically re-enabled after the migration has completed.
    Data transfer is expected to take around 18 hours.

    We will update this post once maintenance has been completed.

    Update Aug 22 09:39
    Migration has been completed. Thank you for your patience.

  • Date - 20/08/2013 12:45 - 22/08/2013 09:39
  • Last Updated - 23/08/2013 00:19
SATAKVM01DE Down (Resolved)
  • Priority - Critical
  • Affecting System - SATAKVM01DE
  • Server SATAKVM01DE is currently not responding. We're working with the datacenter to get the server up and running as soon as possible.

    Update @ 16:35: Server is up again.

  • Date - 07/06/2013 15:41 - 07/06/2013 16:35
  • Last Updated - 07/06/2013 16:51
Level3 peering (Resolved)
  • Priority - Critical
  • Some customers experienced connectivity issues during the last hour. This was caused because of an issue with the announcement of our IP range to Level3.

    Update @ 29/05/13 - 10:51: A temp fix has been applied and a ticket with Level3 was opened.

    Update @ 30/05/13 - 14:11: Level3 has fixed the announcement issue, network latency and redundancy have been restored.

  • Date - 29/05/2013 10:29 - 30/05/2013 14:11
  • Last Updated - 30/05/2013 14:14
Emergency Reboot - helium, lithium (Resolved)
  • Priority - Critical
  • Affecting System - helium & lithium cPanel servers
  • There is a new root exploit making rounds that affects CloudLinux 6/CentOS 6/RHEL 6 systems. CloudLinux released a new kernel to fix the exploit so a quick server reboot on the lithium and helium cPanel servers was necessary.

    Total downtime <1m.

  • Date - 15/05/2013 06:09 - 15/05/2013 06:10
  • Last Updated - 15/05/2013 06:15
DDoS - KVM02NL (Resolved)
  • Priority - Critical
  • Affecting System - KVM02NL
  • We have mitigated a DDoS attack against an IP on KVM02NL.

  • Date - 09/05/2013 20:05 - 10/05/2013 20:12
  • Last Updated - 09/05/2013 22:17
SolusVM Core Node - Network Interuption (Resolved)
  • Priority - Critical
  • Affecting System - SolusVM Core
  • We are seeing network connectivity issues with our SolusVM Core server. We're investigating the issue and will update you again.

    Update 1 [07:22 UTC]: The node on which our SolusVM core server is running, is undergoing urgent maintenance and will be back in a few minutes again.

    Update 2 [07:40 UTC]: Our SolusVM Core server is back online.

  • Date - 06/05/2013 07:06 - 06/05/2013 07:40
  • Last Updated - 06/05/2013 07:41
R1Soft CDP Server Maintenance (Resolved)
  • Priority - Critical
  • Affecting System - R1Soft CDP Server
  • During the weekend we will be moving our R1Soft CDP Server to new hardware for additional disk space and redundancy.


    We have therefore opened a 24 hour maintenance window between 13 Apr 2013 08:00 - 14 Apr 08:00.
    While we move files over to the new server, your Backup policies will be temporarily disabled. Upon completion, all Backup policies will be re-enabled again and Backups will resume to original state.
    Data transfer is expected to take around 12 hours.

    We will update this post once maintenance has been completed.

    Update 14/04/2013 23:00 GMT: Server migration has been completed and all Backup Policies have been verified to work.

  • Date - 13/04/2013 08:00 - 14/04/2013 23:00
  • Last Updated - 14/04/2013 23:11
Maintenance Window - KVM03NL (Resolved)
  • Priority - Critical
  • Affecting System - KVM03NL
  • This message is to all customers on node KVM03NL. 

    After weeks of emails with LSI's support, we decided that we can't use the Samsung 840 Pro SSD drives for our CachedSSD plans anymore, due to their very slow performance and various compatibility issues with our LSI RAID controllers. There has been extensive talk about this matter in numerous forums, and a fix is apparently not coming after all.
    We have therefore decided to replace those SSDs with drives from Intel. We're opening a maintenance window for this work between 11:00 and 16:00 GMT on April 11th.

    No downtime is expected during the drive replacement, although you may notice degraded I/O until the new SSDs are online and fully synced again.
    Also, while no data loss is expected, we have taken full backups of your VPSes as a precaution.

    Update: The drive replacement was completed successfully.


  • Date - 11/04/2013 11:00 - 11/04/2013 13:37
  • Last Updated - 11/04/2013 13:37
Network Outage (Resolved)
  • Priority - Critical
  • Affecting System - Netherlands Datacenter
  • We are currently experiencing an issue with our main upstream provider. Traffic is currently migrated to alternative paths.


    Update 07:34 Main upstream back online. No downtime occurred during this outage.

  • Date - 09/04/2013 07:32 - 09/04/2013 07:34
  • Last Updated - 09/04/2013 08:12
cPanel DNS Cluster #1 - Network Interuption (Resolved)
  • Priority - Critical
  • Affecting System - dns1.fusioned.net
  • Investigating a network connectivity issue with dns1.fusioned.net. Updates will follow.


    Update: dns1.fusioned.net is up again after 8m of downtime.

  • Date - 03/04/2013 13:20 - 03/04/2013 13:28
  • Last Updated - 03/04/2013 13:56
Maintenance Reboot - backup01 (Resolved)
  • Priority - Critical
  • Affecting System - backup01 (R1Soft Backup Server)
  • We are rebooting our R1Soft Backup Server in order to boot into the new CentOS 6.4 kernel.

    Any backup policies scheduled to run during the reboot, will start immediately after.

  • Date - 09/03/2013 23:52 - 10/03/2013 00:01
  • Last Updated - 10/03/2013 00:17
Server issues on Hydrogen (Resolved)
  • Priority - Critical
  • Affecting Server - web1.fusioned.net
  • We’re aware of some issues with Hydrogen server. Web services should be up in a couple of minutes.

    Update: Due to kernel panic we had to restart the server. All web services should now be up and running. If you're experiencing any problem, please open a ticket with Support.

  • Date - 08/03/2013 10:43 - 08/03/2013 10:50
  • Last Updated - 08/03/2013 10:53
Maintenance Reboot - Hydrogen (Resolved)
  • Priority - Critical
  • Affecting Server - web1.fusioned.net
  • This message is to all Enterprise Hosting clients on server Hydrogen. CloudLinux has been upgraded to v6.4 following the recent RedHat Enterprise 6.4 release. A reboot will be necessary in order to load into the new kernel. We have scheduled the server reboot for March 6th 01:00 AM GMT.

    Downtime should be less than 5 minutes.

    Update: Unfortunately due to unforeseen reasons, cPanel would not start after rebooting the server.
    We repaired the Perl installation, and had to force a cPanel update in order to get everything fixed, which took some time.
    Total downtime was 1h51m

    Please submit a ticket to Sales within the next 5 days to claim for SLA credit.

  • Date - 06/03/2013 01:00 - 06/03/2013 02:51
  • Last Updated - 07/03/2013 02:43
SolusVM Core Node - Network Interuption (Resolved)
  • Priority - Critical
  • Affecting System - SolusVM Core
  • We are seeing network connectivity issues with our SolusVM Core server. Connection should stabilize soon.

  • Date - 05/03/2013 04:00 - 05/03/2013 04:51
  • Last Updated - 06/03/2013 00:16
Emergency Reboot (Resolved)
  • Priority - Critical
  • Affecting System - KVM01NL
  • Hello,

    This message is to all clients on node KVM01NL. Unfortunately there seems to be a bug with qemu-kvm and the kernel we're running on this node, causing some KVM VPS to terminate randomly & unexpectedly. In order to fix this we will be performing a kernel upgrade and will have to reboot the server. We will be doing so shortly. We apologize for the short notice, but we have received several reports on this matter today, and need to get it fixed asap.

    Total downtime for your VPSs should be no more than 10 minutes.

    Thanks for your continued business and support.

  • Date - 05/03/2013 00:27 - 05/03/2013 01:03
  • Last Updated - 05/03/2013 01:03
Incoming DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - KVM02NL
  • A DDoS targeting our KVM02NL node's gateway resulted in a network outage between 21:28 and 21:44 UTC.
    Engineers at the datacenter quickly mitigated the attack, and network connectivity went back to normal after 16 minutes of downtime.

    Please accept our apologies for this issue.

    - The Fusioned Team

  • Date - 30/12/2012 21:28 - 30/12/2012 21:44
  • Last Updated - 30/12/2012 21:44
Brief Network Outage (Resolved)
  • Priority - Critical
  • Affecting System - KVM02NL
  • Brief network outage (~3 minutes) on KVM02NL caused by an incoming 2.4G DDoS. Attack was mitigated and network is stable again.

  • Date - 16/12/2012 20:25 - 16/12/2012 20:30
  • Last Updated - 16/12/2012 20:30
Network Issue with DE-CIX (Resolved)
  • Priority - Critical
  • Affecting System - cPanel Node 01, Xen VPS Hypervisors & R1Soft CDP Servers
  • We are currently experiencing a fault with the uplink (DE-CIX). Please bear with us. As soon as more information is available, we shall inform you via this website.

    We apologize for any inconvenience caused.

    Thank you for your understanding.

    Update:
    The fault at DE-CIX is fixed.

    We apologize for any inconvenience caused.

    Thank you for your understanding.

  • Date - 14/11/2012 08:50 - 14/11/2012 10:09
  • Last Updated - 14/11/2012 10:09
Maintenance Reboot - Hydrogen (Resolved)
  • Priority - Critical
  • Affecting System - Hydrogen
  • We will be restarting hydrogen at 00:30 AM UTC to load the new kernel from CloudLinux. No more than 2 minutes of downtime are expected.

  • Date - 03/10/2012 00:30 - 03/10/2012 00:34
  • Last Updated - 03/10/2012 00:34
Xen Security Update (Resolved)
  • Priority - Critical
  • Affecting System - All Xen Hypervisors
  • SolusVM published an RPM Update for Xen v4.1.2 yesterday which includes security patches for CVE-2012-3494 & CVE-2012-3496.

    We will update our Xen Hypervisors on September 29th, at 12:30 AM UTC. After the software update, each Hypervisor will have to be restarted and we estimate approximately 10 minutes downtime, as a result of this.

    Updates:
    Xen VPS Hypervisor 01 :: Done (00:31 AM UTC)
    Xen VPS Hypervisor 02 :: Done (00:44 AM UTC)
    Xen VPS Hypervisor 03 :: Done (01:07 AM UTC)

  • Date - 29/09/2012 00:30 - 29/09/2012 01:09
  • Last Updated - 29/09/2012 01:09