We are marking this incident as resolved. Please contact help@finalsite.com if you have any questions or concerns.
Thank you for your continued partnership with Finalsite.
Posted Feb 07, 2022 - 11:49 EST
Update
While our team has restored functionality for the majority of clients, we're leaving this incident open until full functionality is restored for all. We're happy to announce that we've resolved a number of issues with the following fixes:
-Resolved errors preventing saves for new recurring payment plans -Implemented a fix that was preventing some forms from loading properly -Fixed an issue that prevented integrations from running on their automated schedule
The following issues remain for some clients: -Group space resources not displaying properly for a subset of clients -Forms Manager attachments may not be accessible for a subset of clients
Please contact our Support Team if we can assist with anything else. Thank you for your continued patience.
Posted Jan 20, 2022 - 19:09 EST
Update
While our team has restored functionality for the majority of clients, we're leaving this incident open until full functionality is restored for all. We're happy to announce that we've resolved a number of issues with the following fixes:
-Resolved errors preventing saves for new recurring payment plans -Implemented a fix that was preventing some forms from loading properly -Fixed an issue that prevented integrations from running on their automated schedule
The following issues remain for some clients: -Group space resources not displaying properly for a subset of clients -Forms Manager attachments may not be accessible for a subset of clients
Please contact our Support Team if we can assist with anything else. Thank you for your continued patience.
Posted Jan 20, 2022 - 18:21 EST
Update
Thank you for your continued patience on restoration efforts. While our team has restored functionality for the majority of clients, we're leaving this incident open until full functionality is restored for all. Some clients may see the following:
-Group space resources not displaying properly for a subset of clients -Forms Manager attachments may not be accessible for a subset of clients -Integrations aren't syncing for a subset of clients. We are rolling out a fix that should resolve this issue going forward. -Some clients are reporting errors when attempting to save new recurring payment plans
We're investigating reports of forms not loading properly. If you are seeing this occur, please contact our Support Team.
Posted Jan 18, 2022 - 18:33 EST
Update
iCal and Alerts have been restored across all regions and are now fully functional.
Some clients have reported forms intermittently not loading. If you're experiencing this, please contact Support for assistance.
You may still see the following, which we are working to resolve for all clients and does not require reaching out to us.
-Group space resources not displaying properly -Forms Manager attachments may not be accessible -Integrations may not work for a subset of clients
Thank you for your continued patience during these final stages of restoration efforts.
Posted Jan 13, 2022 - 10:14 EST
Update
We have now restored front-facing assets from File Manager and Media Manager to all websites.
The majority of integrations, iCal feeds, and Alerts have also been restored. Our team will continue working diligently until everything has been brought back online. During this time you may continue to see the following:
-Group space resources not displaying properly -Alerts and iCal syncs may not function properly for a subset of clients -Constituent Manager imports are unavailable for a subset of clients
If you are experiencing a time-sensitive or high priority issue, or if you believe that you are still missing files from File Manager or Media Manager, please contact our Support Team. Thank you for your patience during restoration efforts.
Posted Jan 12, 2022 - 21:48 EST
Update
Restoration of integrations and file and media manager are underway and will continue until we have a full resolution. Until everything is restored, you may see the following:
-Directory/Constituent images may not load correctly -Slideshows relying on Media Manager may not display correctly -Virtual tour/Campus tour files may not load -Online Shop that stores images in File Manager may not display correctly -Broken image icons in place of File Manager images, PDFs -Files and folders physically missing from File Manager -Some data integrations won't run properly -Group space resources not displaying properly -Alerts and iCal syncs may not function properly -Constituent Manager imports are unavailable
If you spot any of the above, please know that we are actively working to resolve these and you do not need to reach out to our support team at this time. If you are experiencing a time-sensitive or high priority issue, please contact us at help@finalsite.com.
We will continue to post here with information on the remaining product restoration efforts.
Posted Jan 11, 2022 - 11:47 EST
Update
While our team works to restore all remaining File Manager, Media Manager, and Integrations; we wanted to provide more details surrounding how these ongoing efforts may be exhibited on your website.
During this time, you may see the following: - Directory/Constituent images may not load correctly - Slideshows relying on Media Manager may not display correctly - Virtual tour/Campus tour files may not load - Online Shop that stores images in File Manager may not display correctly - Broken image icons in place of File Manager images, PDFs - Files and folders physically missing from File Manager - Some data integrations won't run properly
If you do see these, please know that we are actively working to restore them and you do not need to reach out to our Support team at this time. Thank you for your continued patience and understanding.
Posted Jan 10, 2022 - 11:19 EST
Update
All websites have now been restored. Messages and eNotify are both fully operational.
File Manager and Media Manager assets will continue to be brought back online today. This is a slower process due to the volume of files involved. Our team is diligently working on to restore data integrations as well as some auxiliary services today. These are the final remaining elements leading to full restoration of our systems and we will send a status update upon resolution.
Please contact our Support Team if you experience any issues outside of the items mentioned above.
We will send another status update in the evening (EST) today, or upon full resolution of the final elements listed above. Please expect additional communication later today with a forensic investigation status update.
Thank you for your immense patience and understanding.
Posted Jan 10, 2022 - 07:29 EST
Update
Our team has restored admin access to all sites. We are still working to restore some assets from File Manager and Media Manager, which may affect the display of pages that rely on items like photos or videos loaded from these locations.
We are still working to restore eNotify and will provide an update as soon as we are finished. Our Messages module, the replacement for eNotify, is active and available to anyone who currently uses eNotify. Please see the following article for more information on enabling Messages: https://fnlst.com/bdvsWD
We have identified a segment of data integrations that require the restoration of mapping files, which we will restore and provide an update upon completion.
Please contact our Support Team if you experience any issues outside of the items mentioned above. Thank you for your continued patience.
Posted Jan 09, 2022 - 19:09 EST
Update
At this time, 99.9 percent of websites have been restored including front end and admin access.
Our team is still working to restore eNotify, File Manager and Media Manager assets, and integrations. We’ll provide you with another update this evening or as soon as new information is available.
Posted Jan 09, 2022 - 09:28 EST
Update
We have restored front end and admin access to nearly all websites, including Messages and Page Pops. We have also restored calendar events, athletics, news/posts, and directory data in Composer. Our team is still working to restore File Manager, eNotify, Integrations, and Media Manager assets, and we will monitor performance through the night.
If you notice any content missing or broken functionality, please contact our Support team with details. We will provide an update in the morning (EST) on the status of our full restoration effort. Thank you for your immense patience and understanding during this time.
Posted Jan 08, 2022 - 23:03 EST
Update
At this time, more than 90 percent of websites now have restored admin access. Progress continues bringing files back online. Our engineers will continue working around the clock until everything has been restored.
As we work to restore all modules, we still recommend limiting software use to important page updates and avoid reliance on Alerts, Messages and eNotify. We will provide an update on usage of those modules as soon as possible.
We will continue to work through the weekend and will provide you with another update in the morning.
Posted Jan 08, 2022 - 18:26 EST
Update
Our team has worked through the night and has now restored admin access to over 70% of sites. We are still working to restore styling, calendar events, athletics, news/posts, and directory elements and expect to make more progress throughout the day.
We still recommend that you limit software use to important page updates or edits and avoid relying on Alerts, Messages, and eNotify until our monitoring metrics demonstrate greater stability.
We continue to restore more websites and core functionality every hour. We remain committed to restoration, as quickly and safely as possible. Our team is actively monitoring performance and security across our infrastructure, and we will continue to work through the night and weekend until every site is back online.
Thank you for your patience and understanding.
Posted Jan 07, 2022 - 20:02 EST
Update
Our team continues to make progress restoring functionality to websites. We will continue working through the night and will provide updates as they become available.
Additionally, we wanted to provide some updated communication resources. A parent template letter and media talking points are available to view here: https://fnlst.com/NkyE9x
Thank you for your continued patience.
Posted Jan 07, 2022 - 16:25 EST
Update
We’re continuing to work to restore admin access as quickly as possible, and many more sites can now log in. We are continuing to restore styling, calendar events and constituents for directories and will update you on our progress along the way.
Posted Jan 07, 2022 - 14:00 EST
Update
The vast majority of sites are up and running on the front end, but we still have work to do to bring everything back to normal.
-You may notice missing styling which we are still actively restoring. -You may not be able to access the /admin side of your site, even though your website is up. We are aware of this and are actively working to correct this for the affected sites. -You may continue to see 503 errors, which should subside as we restore more and more content.
We highly recommend that you limit software usage to critical information updates for your front-end, until we've confirmed that all functionality is working as expected. Examples of usage to avoid include sending email/notifications, workflows, relying on calendar and athletic alerts, uploading data etc.
Engineers will continue working around the clock and through the weekend until we reach a full resolution and we are dedicated to continuous communication with you throughout the process.
Thank you for your continued patience and understanding.
Posted Jan 07, 2022 - 10:25 EST
Update
Our team continues to make progress bringing more websites back up. While we still have work to do, the vast majority of front-facing websites are online. Some sites may still lack proper styling, admin log-in functionality, calendar events, or constituent directories, but the team is currently working to restore these elements.
Earlier this evening (EST), we pushed an update which has greatly reduced the incidence of 502 "Oops" errors. Additionally, we have updated that "Oops" screen to display a more informational interface.
We will continue working through the night (EST), and will post another update when we have new information to share.
Posted Jan 06, 2022 - 23:18 EST
Update
We continue to make progress restoring websites connectivity. As we work to bring more sites up, we're also monitoring performance for active websites and implementing the following:
-Replaced the previous error message to properly identify as a Finalsite error -Working to resolve some general slowness -Addressing the display of 502 errors -Resolving an issue with some sites unable to access the /admin side
We will provide additional information as it becomes available. Thank you for your continued patience and understanding. We will continue working until all sites are fully functional.
Posted Jan 06, 2022 - 19:51 EST
Update
Our team remains hard at work restoring connectivity to websites, and we expect the number of restorations to increase throughout the day and evening. We will provide more updates and information as they become available.
Also, please visit the following link for language you can use when communicating with parents. See https://fnlst.com/jFzmIr
Posted Jan 06, 2022 - 15:40 EST
Update
We are incredibly sorry for this prolonged outage and fully realize the stress it is causing your organizations. While we have made progress overnight to get all websites up and running, full restoration has taken us longer than anticipated.
The Finalsite security team monitors our network systems 24 hours a day, seven days a week. On Tuesday, January 4, our team identified the presence of ransomware on certain systems in our environment. We immediately took steps to secure our systems and to contain the activity. We quickly launched an investigation into the event with the assistance of third-party forensic specialists, and began proactively taking certain systems offline.
In the ensuing time since the incident, our security, infrastructure, and engineering teams have been working around the clock to restore backup systems and bring our network back to full performance, in a safe and secure manner. Third-party forensic specialists are assisting us in bringing things back slowly and carefully to ensure the environment is safe and stable.
We have full access to our files and data. The forensic investigation is ongoing and at this time, we have no evidence that our data or client data has been taken. If we determine otherwise through the course of the investigation, we’ll act swiftly to notify you and will take all appropriate actions.
I want to assure you that the integrity, safety, and security of our network and the information held in our care are amongst our top priorities. Due to the nature of the investigation, we were unable to share this information with you until now. We are taking steps to secure the environment and ensure this type of incident does not occur again. We will share more details with you as we learn more.
At Finalsite, we value the relationships we’ve built with our clients and sincerely apologize for the longevity of this outage and what it has meant for your school communities. We’d like to invite you to a webinar at 2 p.m. EST to have your questions answered and hear from us face-to-face. Please register for the webinar online at https://finalsite.zoom.us/webinar/register/WN_ztQX4WC4SQ-ZKEFzuiefLQ
Thank you again for your patience with us through this challenging situation and we hope to see many of you this afternoon.
Posted Jan 06, 2022 - 12:04 EST
Update
Please note that as websites come back online, they may lack styling and access to admin logins. No action is necessary on your part, we are aware of this and are working to restore their full appearance and functionality. Thank you for your understanding.
Posted Jan 06, 2022 - 07:50 EST
Update
We know your website is a critical communication hub for your schools and sincerely apologize that we are not fully online this morning as promised. Websites are beginning to go back online, however full restoration has taken us longer than anticipated.
We fully understand this is not the message you hoped to hear this morning and are sorry for the delay. You have our word that more information will be forthcoming soon on what happened and where we are with restoring service to your website.
Sincerely, Tim McDonough Chief Client Officer
Posted Jan 06, 2022 - 07:23 EST
Update
We continue to make progress bringing websites back online and are still on track to restore connectivity to all Composer regions by the morning (Eastern Time). We will post an update here as soon as we have more to share.
Posted Jan 05, 2022 - 21:56 EST
Identified
We have made significant progress in restoring our network. The engineering team is beginning to bring websites back online and will continue to do so throughout the evening.
We are confident that all sites will be restored by tomorrow morning (Eastern Time) and will update you once again when we have a full resolution.
The process of bringing sites back online is happening as follows: - Restoring connectivity to the front end of your websites; - Monitoring performance to ensure sites are stable; and - Restoring core admin functionality.
Please understand you may see intermittent error messages on your website until we reach full functionality. We recommend that you refrain from relying on admin functionality (instructing users to log in to portals, complete forms, sending bulk emails etc.) until we’ve notified you that we’ve completed this full process.
We realize how difficult this outage has been for your school communities and appreciate your patience as we work through this process. Please stay tuned for additional updates from us as the evening progresses.
Sincerely, Tim McDonough Chief Client Officer
Posted Jan 05, 2022 - 18:10 EST
Update
We are making progress on restoring connectivity to sites. We’ll provide an additional update within the hour. Thank you for your continued patience and understanding.
Posted Jan 05, 2022 - 15:42 EST
Update
Resolution efforts continue and we will provide an update in an hour, or as soon as we have additional information to share.
Posted Jan 05, 2022 - 14:42 EST
Update
We are making progress on a resolution to our system disruption and will continue to provide updates on the hour.
Posted Jan 05, 2022 - 13:40 EST
Update
Thank you for your patience as we work to resolve the disruption to our computer systems. We understand the frustration you are experiencing and are dedicating all of our resources to restore full functionality to our systems.
We are pleased to let you know that we are making progress toward resolution and will get your websites back online as quickly as possible.
Thank you for your continued patience and support. Please continue to check back for updates on the hour.
Posted Jan 05, 2022 - 12:37 EST
Update
Our team has been working through the night to restore full performance to all sites. We are still experiencing a disruption to certain computer systems on our network. Upon learning of this activity, we immediately took steps to secure our systems and to contain the activity, including proactively taking certain systems offline. We are working diligently with third-party specialists to investigate the source of this disruption, confirm its impact on our systems, and to restore full functionality to our systems as quickly and safely as possible.
We know the importance of your website and have all resources focused on this effort. We appreciate your patience and understanding and sincerely apologize for any inconvenience. We will provide further updates as we bring our systems back online.
Sincerely, Tim McDonough Chief Client Officer
Posted Jan 05, 2022 - 09:05 EST
Update
We apologize for the continued outage. Investigation efforts continue and we will provide an update as soon as we have additional information to share.
Posted Jan 05, 2022 - 06:05 EST
Update
Thank you for your ongoing patience and understanding.
Our team continues the effort to restore full access and connectivity to all websites.
We understand that this has been a significant disruption for your teams and users. We have been working through the night, and we will continue to work until this is resolved.
We will continue to post updates at status.finalsite.com, but we will only post and send notifications when we have new information to share that may impact decision-making and operations.
Posted Jan 05, 2022 - 03:01 EST
Update
Work is ongoing to restore full access and connectivity to all websites, however, we do not yet have additional updates to provide at this time. We'll continue to post here as we have more to pass along. Thank you for your continued patience and understanding.
Posted Jan 04, 2022 - 23:53 EST
Update
Our team continues the effort to restore full access and connectivity to all websites.
We want to acknowledge the similar updates since the beginning of this incident. Our intention was to post frequent updates; so it is clear that the team is actively working to reach a resolution. Even if we don’t have new information to communicate, we want our clients across the globe to know that we will not rest until service is restored.
We understand that this has been a significant disruption for your teams and users. We have been working through the afternoon and evening, and we will continue to work through the night until this is resolved.
We will continue to post updates at status.finalsite.com, but we will only post and send notifications when we have new information to share that may impact decision-making and operations.
Thank you for your patience and understanding.
-The Finalsite Team
Posted Jan 04, 2022 - 21:17 EST
Update
Our team is continuing work to restore full site performance at this time. We apologize for the extended outage and will provide an update within the hour or as soon as we have additional information to share.
Posted Jan 04, 2022 - 19:36 EST
Update
We apologize for the continued outage. Investigation efforts continue and we will provide an update within the hour, or as soon as we have additional information to share.
Posted Jan 04, 2022 - 18:33 EST
Update
Efforts are ongoing to restore connectivity as quickly as possible. We apologize for the extended outage and will provide an update within the hour, or as soon as we have additional information to share.
Posted Jan 04, 2022 - 17:17 EST
Update
Our team is continuing work to restore connectivity as quickly as possible. We apologize for the extended outage and will provide an update within the hour or as soon as we have additional information to share.
Posted Jan 04, 2022 - 16:17 EST
Update
We apologize for the continued inconvenience. We're continuing to investigate this issue and will provide an additional update in approximately one hour.
Posted Jan 04, 2022 - 15:25 EST
Update
We're aware of an increase in Cloudflare error messages ("Your secure connection to this website via Cloudflare...") and the intermittent appearance of sites being down. Our investigation remains ongoing, and at this time indications are that these Cloudflare errors are tied to the same root cause. We apologize for the inconvenience and will continue to provide updates as soon as we have them.
Posted Jan 04, 2022 - 14:50 EST
Update
Our team is continuing to investigate this issue and we’ll provide further updates as soon as we have them. Please note that this issue is impacting our legacy modules, and the elements in the Composer CMS they interact with. This impact may include, but is not limited to, Groups Manager, Constituent Manager, Login, Forms Manager (old), Registration Manager, Directory Elements, Athletics Manager, Calendar Manager.
Potential errors may display as: “There was an error loading the settings. Please try again” or “Could not find the ColdFusion component or interface D:\ColdFusion2016\cfusion\CustomTags\com\adobe\coldfusion\query.cfc. Ensure that the name is correct and that the component or interface exists. The error occurred on line 66.”
Posted Jan 04, 2022 - 14:07 EST
Investigating
We are currently investigating an issue leading to increased error rates and performance issues across our legacy modules, and our team is investigating a fix as a critical priority. We'll provide additional updates as soon as we have them.
Posted Jan 04, 2022 - 13:16 EST
This incident affected: Composer CMS (Content Management System) (Composer US East, Composer US Central, Composer US West, Composer UK, Composer Asia).