28-Mar-2024 Connectivity Issues

Dashboard Announcements Concerning Site Connection Issues Seen on 28-March-2024

Timeline:

• (all times listed in PDT)
  16:37:11 4/3/2024
  In case you weren't aware, we are working on a large project to migrate your website to the cloud. The benefits to you and why we took on this important product include additional security, better redundancy, more scalability, and flexibility for your site. During this transition, you may experience some downtime. I want to assure you we are doing everything possible to minimize the downtime and impact on your business. We believe this move will provide long-term benefits to your organization.
  
  Additionally, either a member of our leadership team or your dedicated Client Success Manager will be reaching out to you via phone in the upcoming days to provide further assistance and address any questions or concerns you may have.
  
  This ticket you opened regarding the recent service interruption has been resolved. If you have any further questions or concerns, please do not hesitate to reach out to our support team. Thank you for your understanding and patience throughout this process.
  
  15:38:22 4/2/2024
  We want to update you on the ongoing issue related to our cloud environment. Earlier today, we worked closely with our cloud provider to address the challenges. As a proactive measure, we have added additional capacity to enhance system stability.
  
  However, we want to emphasize that while we've taken steps to improve the situation and expect to be stable through the evening, we are not yet in the all-clear status.
  
  There are additional actions that need to be taken before we can provide a definitive resolution. Our team is actively working on these next steps, and we will provide another update tomorrow late in the day.
  
  Your patience and understanding during this time are greatly appreciated. If you have any questions or concerns in the meantime, please don't hesitate to reach out to us. We are committed to keeping you informed and ensuring minimal disruption to your operations. Thank you for your continued trust and support.
  
• 14:44:16 4/2/2024
  Leadership & Development Teams are currently on a call with our Cloud Provider working on the cloud environment to address the connectivity and outage issues.
  
  As this work is performed we expect there will be additional events. Our team is actively working to address these issues and restore stability.
  
• 11:42:11 4/2/2024
  We want to provide you with an update regarding the recent service interruption caused by an issue with our cloud environment. Due to an unexpected incident, our cloud infrastructure experienced a disruption, resulting in temporary unavailability of services.
  
  We are pleased to inform you that our technical team promptly responded to the issue, and within just 20 minutes, we were able to mitigate the impact and restore normal operations. Additionally, we are continuing to monitoring to ensure the stability and reliability of our systems moving forward.
  
  Your trust is of utmost importance to us, and we understand the critical nature of our services to your business operations. We apologize for any inconvenience this incident may have caused and assure you that we are continuously working to strengthen our infrastructure to prevent similar occurrences in the future.
  
  If you have any questions or require further assistance, please don't hesitate to contact our support team. We are here to help.
  
  Thank you for your patience and understanding.
  
• 14:23:53 4/1/2024
  
  

  We experienced another outage today, which was traced to a different issue within our firewall system. We have addressed this specific problem and are currently implementing measures to monitor the situation over the next 24 hours before issuing an all-clear.

  We understand the importance of consistent and reliable service and are taking all necessary steps to ensure stability. Our team is closely monitoring our systems to promptly address any potential issues that may arise.

  We appreciate your patience and understanding as we work to maintain and improve our service quality. We are committed to providing you with the most reliable and effective solutions.

  Thank you for your continued trust in our services.

• 15:01:17 3/29/2024
  
  

  We are pleased to inform you that we have identified and resolved the issue that led to the slow site performance and outage experienced today. While we believe the situation has been stabilized, we will observe a 24-hour monitoring period before officially declaring an all-clear status.

  Heading into the weekend, we are confident in the stability of our services and anticipate that the scheduled maintenance on Sunday will further enhance our system's reliability and provide additional insights.

  Our team will be vigilantly monitoring the network from now through the weekend to ensure ongoing stability and to swiftly address any unexpected issues.

  A comprehensive update will be shared next week, following the completion of our maintenance activities and the resolution process with our firewall provider. This will include detailed findings and any subsequent measures taken to prevent future occurrences.

  Thank you for your understanding and patience during this time. We remain committed to providing you with the reliable service you depend on.

• 13:34:53 3/29/2024
  
  We want to provide an update on the stability of our network and the recent service interruptions. Our services are currently stable in regards to the Firewall; however, the issue with our Firewall provider remains unresolved. We will be gathering the necessary additional information during our maintenance window on Sunday. Details for the scheduled maintenance will be posted shortly.
  
  We also wish to address the outages that occurred yesterday afternoon and evening as well as the slow sites and brief outages today. These were not related to the firewall issues. We are working closely with our cloud provider and a contracted agency to determine the cause of these disruptions.
  
  It's important to acknowledge that since we have not yet fully identified the cause of these non-firewall-related outages, there is a possibility they may recur. Rest assured, our team is on high alert and ready to address any issues as quickly as possible to minimize any impact.
  
  A comprehensive report on the cause and resolution of these issues will be posted once our investigations are complete. We are committed to maintaining transparency and providing you with the latest information.
  
  We appreciate your patience and understanding as we work to resolve these challenges. Your trust in our service is vital to us, and we are dedicated to restoring and maintaining the reliability you expect.
  
  Thank you for your continued support.
  

• 15:21:43 3/28/2024

  We are pleased to report that our services are currently stable, and we are maintaining close monitoring to ensure continued operation.

  However, we are still awaiting a comprehensive resolution from our Firewall provider, who is actively troubleshooting the issue to identify a long-term fix.

  We want to assure you that every effort is being made to address this situation as swiftly and effectively as possible.

  Our team will continue to monitor the system closely throughout the night. Please note that we will not have further updates until after our scheduled meeting with all parties involved tomorrow at 9 AM PST.

  We are committed to keeping you informed and will share any significant developments as soon as they become available.

  Thank you for your continued patience and understanding.  Your trust in our services is greatly appreciated, and we are dedicated to resolving this matter promptly.

  Should you have any immediate concerns, please feel free to reach out. We are here to support you.

• 13:50:15 3/28/2024

  Following our previous communication, we experienced another unexpected outage.

  This was during an attempt by our third-party support team to implement a fix aimed at resolving the ongoing issues. Unfortunately, the attempted solution did not produce the desired outcome, and as a result, their team is continuing their investigative efforts to identify a more effective resolution.

  Please be assured that we have multiple agencies, including our cloud provider and specialized support teams, working diligently to address and rectify the situation.

  Our focus remains on achieving a long-term solution that will prevent future occurrences and ensure the stability and reliability of our services.

  Thank you for your patience and understanding during this time.

  Should you have any immediate concerns or require assistance, please do not hesitate to reach out to our support team.

• 12:53:53 3/28/2024

  Update on the recent performance issues you may have experienced with our services.

  We have actively engaged with our cloud provider and an additional third-party agency to address the situation.
  
  In a recent collaborative effort, a temporary solution has been implemented to alleviate some of the load on our firewall, which has resulted in an immediate improvement in load time. Our next steps involve working closely with our firewall provider to diagnose and rectify the underlying issue, ensuring a sustainable and long-term resolution.
  
  Rest assured, we now have four dedicated teams working diligently to fully resolve this matter. While we have yet to declare a complete resolution, please know that keeping you informed is our top priority.
  
  We will continue to share updates as more information becomes available. We appreciate your patience and understanding as we work to restore our service to the high standard you expect from us.

  If you have any concerns or need further assistance, please do not hesitate to reach out.

• 09:16:30 3/28/2024
  We are currently experiencing an issue that is causing slower than usual loading times for websites.
  
  Please be assured that our team is aware of the problem and is actively working to resolve it as quickly as possible.
  
  We apologize for any inconvenience this may cause and appreciate your patience and understanding as we work to restore normal service levels.
  
  Should you have any concerns or need further assistance, please do not hesitate to contact our support team.