Certain Policies are causing Reboot Loops on Devices

Incident Report for Addigy

Postmortem

Hello Addigy Customers,

I'm going to get to the incident details in a moment. It's important to me to first apologize for the disruption and frustration you felt today. I know this was a major issue for you and your users. The last thing we want to do is cause a fire for you, and I'm deeply sorry that happened.

Today, we had a scheduled release that impacted a specific patch reboot flag within Policies. The result caused spurious reboots on a subset of devices. Specifically, the timeline of events was:

3:01pm EST: Status Page maintenance notification of a software release
3:26pm EST: Release went out to all infrastructure
3:38pm EST: First report of issue
4:35pm EST: Identified issue
4:55pm EST: Packaged up release patch
5:24pm EST: Release of patch completed
5:39pm EST: Resolution verified across all Addigy portfolio

**
Root cause.** Release AM-5496 was generated to address how user notifications are sent for System Updates. As part of our processes when rolling out releases, we completed full automated testing and targeted regression testing. Yet, this bug was released without being caught. A subset of Addigy customers had devices in their environment with the available update, "macOS Mojave 10.14.6 Supplemental Update (0-2);" those devices were impacted and experienced spurious reboots. We are continuing our investigation to confirm that no other Supplemental Update strings were impacted.

**
Resolution.** We reverted the change to our Release AM-5496. We pushed a patch and our Status Page is updated and will continue to be updated as we find any additional details.

While this was a difficult day for us at Addigy, I know this was a far more difficult situation for you. Please expect a check-in from our team; how you and your environment were impacted today is important to us. Or, please reach out to me directly.

Sincerely,

Jason Dettbarn
Founder & CEO

Posted Oct 10, 2019 - 00:03 UTC

Resolved

This issue has been resolved. You can view the resolution on our releases page: https://releases.addigy.com/AM-5544.

If you continue to experience issues with this issue, macOS devices restarting periodically, or any other issue, please reach out to the Addigy Support team immediately: support@addigy.com, (305)330-6677. Our knowledgeable team will be happy to assist you.

Posted Oct 09, 2019 - 22:35 UTC

Update

We have published the solution to this issue.
Devices which needed the update "macOS Mojave 10.14.6 Supplemental Update 2- " (and similar versions) were not properly handling reboots. We are conducting an investigation into the root cause of this incident. You can also view these notes on our release page: https://releases.addigy.com/AM-5544.

Posted Oct 09, 2019 - 21:57 UTC

Monitoring

We have released a fix to this which issue caused certain macOS devices to restart periodically and are closely monitoring the situation.

If you applied a remediation to prevent the Addigy policier from running, restarting your device will resume normal policier activity. If you like assistance resuming the policier, please reach out to Addigy Support: support@addigy.com, (305)330-6677.

Posted Oct 09, 2019 - 21:42 UTC

Update

We have identified an issue with today's release that has caused some macOS devices to restart periodically.

If you are experiencing this issue, please reach out to Addigy Support: support@addigy.com, (305)330-6677. Our skilled support team can help you apply a remediation while we implement a solution to the problem.

Posted Oct 09, 2019 - 21:27 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Oct 09, 2019 - 21:26 UTC

This incident affected: Addigy Cloud Interface.