15-May-2024 Server Replication Issues

Timeline of events concerning issues on backend updates not being reflected on the frontend of sites.

Timeline:

(all times listed in PDT)

• 11:08:54 05/17/2024
  
  We're excited to inform you that the admin issue has been successfully resolved! Our team has been hard at work to ensure that all systems are functioning smoothly, and we're happy to report that everything is now back on track.
  
  
  We are beginning the process of rebuilding each of the affected replicas. Although this issue is rare, we’re addressing the symptoms thoroughly.
  Action Plan:
  1. Rebuild Replicas: Step-by-step rebuilding to ensure stability.
  2. Debugging: We will conduct a detailed debugging session to identify root causes.
  The replicas are now operational, and to prevent this issue in the future we are fully rebuilding them, the process of which is already in progress.
• 11:08:54 05/16/2024
  

  We wanted to provide you with an update on the issue with the site admin. Our development team is actively engaged in investigating the matter to determine the root cause and find a solution.

  Currently we believe there to be a replication issue from the main database. This is causing the delays in processing of changes.
   

  We understand the impact this issue may have on your experience, and we want to assure you that resolving it is our top priority. We are committed to ensuring the functionality of our admin feature and appreciate your patience and understanding as we work through this process.

• 9:00:00 05/15/2024
  We wanted to inform you that we are currently investigating an issue with site admin. Currently back end changes are not reflecting to the front end. Our team is actively working to resolve this issue as quickly as possible to ensure minimal disruption to your experience.