Hello Addigy Customers,
I'm going to get to the incident details in a moment. It's important to me to first apologize for the disruption and frustration you felt today. I know this was a major issue for you and your users. The last thing we want to do is cause a fire for you, and I'm deeply sorry that happened.
Today, we had a scheduled release that impacted a specific patch reboot flag within Policies. The result caused spurious reboots on a subset of devices. Specifically, the timeline of events was:
3:01pm EST: Status Page maintenance notification of a software release
3:26pm EST: Release went out to all infrastructure
3:38pm EST: First report of issue
4:35pm EST: Identified issue
4:55pm EST: Packaged up release patch
5:24pm EST: Release of patch completed
5:39pm EST: Resolution verified across all Addigy portfolio
Root cause.** Release AM-5496 was generated to address how user notifications are sent for System Updates. As part of our processes when rolling out releases, we completed full automated testing and targeted regression testing. Yet, this bug was released without being caught. A subset of Addigy customers had devices in their environment with the available update, "macOS Mojave 10.14.6 Supplemental Update (0-2);" those devices were impacted and experienced spurious reboots. We are continuing our investigation to confirm that no other Supplemental Update strings were impacted.
While this was a difficult day for us at Addigy, I know this was a far more difficult situation for you. Please expect a check-in from our team; how you and your environment were impacted today is important to us. Or, please reach out to me directly.
Founder & CEO