Buggy Windows 10 update patches upset some admins, but Microsoft stands its ground.
After the quality of Windows 10 feature updates was called into question in an open letter from Susan Bradley, Microsoft posted a blog post clarifying their intention with the constant barrage of updates.
The open letter addresses several update issues and showcases polls that were taken by those in the IT community to ask their satisfaction level on items such as the quality of the patches, their satisfaction with patching, patch features, cadence and meeting the needs of their business. The results were not surprising and it’s no surprise that Microsoft went on the defensive.
While Microsoft didn’t directly respond to the open letter, they did stand firm in their defense of how, when, and why they are patching in this manner, and outlined the specifics of their cadence.
Susan Bradley’s open letter as it was published is below. You can also read the letter directly from here. https://www.computerworld.com/article/3293440/microsoft-windows/an-open-letter-to-microsoft-management-re-windows-updating.html
From: Susan Bradley
To: Mr. Satya Nadella, Mr. Carlos Picoto and Mr. Scott Guthrie
Dear Sirs:
Today, as Windows 10 turns three years old, I am writing to you to ensure that you are aware of the dissatisfaction your customers have with the updates released for Windows desktops and servers in recent months. The quality of updates released in the month of July, in particular, has placed customers in a quandary: install updates and face issues with applications, or don’t install updates and leave machines subject to attack.
In the month of July 2018 alone there are 47 knowledge base bulletins with known issues. Some of these were stop issues, but most concerning were the .Net side effects with your own software: SharePoint, BizTalk and even Exchange servers were impacted by these July 10 updates.
I am a moderator on a community listserve that focuses on the topic of patch management, patchmanagement.org. Recently many of the participants on the listserve have expressed their concerns and dissatisfaction with the quality of updates as well as the timing of updates.
I recently asked the list members to answer several questions about patching on Windows 7 to Windows 10. The full results of this unscientific survey can be read here. I urge you to take the time to read the responses. It showcases that your customers who are in charge of patching and maintaining systems are not happy with the quality of updates and the cadence of feature releases, and feel that it cannot go on as is.
Question 1 I asked on a scale of 1 to 5, 5 being the highest, how satisfied respondents are with the quality of Windows updates in general.
Many respondents were not satisfied with Windows updating in general.
Question 2 I asked about satisfaction with patching of Windows 10 specifically:
Many respondents were not happy with the quality of Windows 10 updates.
Question 3 I asked if Windows 10 feature updates were useful to the respondents’ business needs.
Many respondents indicated that the feature updates were either not useful at all or rarely useful to their business needs.
In Question 4, I asked about the cadence of feature releases.
Most of the survey respondents did not want feature releases as often as they are being released now.
In Question 5 I asked if Windows 10 is meeting respondents’ business needs.
Most of the survey respondents answered that it was meeting their needs.
Finally, I asked an open-ended question as to what could be changed in Windows 10 to make it better for respondents’ business. You can read the response to Question 6 here.
I also did a similar survey for consumers. The results of the survey targeted to consumers were similar to the results from the consultants and patching administrators. The majority thought that the feature updates occurred too many times during the year, and the said that they were overall not happy with the quality of updates from Microsoft. The full survey results from Microsoft consumer customers can be found here.
I urge you to take the time to look at both the results from patching administrators, and also consumers and home users in detail. You will see similar trends in both surveys.
Insider process is not identifying issues
It appears that there is a breakdown in the testing process. The Windows 10 insider process is not able to identify issues on released products. When your own products break with these releases, it is clear that current testing processes are not good enough.
It is concerning when issues with Microsoft’s own software releases have detrimental side effects with other Microsoft software. Case in point: the recent .Net 4.7.2 and Azure AD connect that causes side effects and issues with high CPU.
At one time you had a program called the Security Update Validation Program that allowed firms with special nondisclosure agreements to test security updates ahead of their release. I urge you to increase this program and include a broader testing process. While your MSRC communication says that for best practice one needs to install updates immediately, the reality is that the prudent patcher is waiting at least a week, if not more, before installing updates. I hope you find this trend as concerning and disturbing as I do.
Feature releases are causing patch fatigue
I am disturbed when I see users and consultants talk about taking drastic measures to take back control of updating and rebooting. Some are disabling Windows Update as a drastic measure to ensure that updates do not reboot systems when they are not wanted. It’s clear that your team also acknowledge that unexpected updates are problematic. But your customers deserve better than “promising” results. They deserve a stable platform that reboots only when they want it to. The operating system needs to do a better job of communicating to the end user and especially to the patching administrator when a machine will receive an update. The addition of the Windows Update for Business settings that often conflict with other group policy settings cause confusion, not clarity.
While it’s commendable that you’ve listened to feedback and made changes to Windows update during these three years, the fact is that these changes in each version release have caused confusion, and in some cases behavior that was not expected at all. Dual scan is one such change that caused confusion, and as a side effect caused administrators to have updates installed when they did not want them. The lack of clear communication regarding update changes leads to this confusion. Administrators are having to follow various blogs and sites and even Twitter channels to be able to understand the changes. The lack of basic documentation of Windows update error codes, the fact that it took several feature releases to make changes to the unreadable Windows update log, the fact that it took several feature releases before acknowledging the problem of symbol publishing showcases that the changes in Windows updating have had a major impact in the servicing and handling of Windows 10. I personally know of several large enterprises that are not on the current Semi Annual channel release of 1803 and are in fact several feature releases behind. The constant change and churn is not helping firms in their deployment strategies.
Patch communication needs work
Starting in January of this year with the release of Spectre/Meltdown patches, there have been numerous instances where patching communication has been wrong, registry entries detailed in Knowledge Base articles regarding registry key application was initially incorrect and later updated, or vendor updates had to be stopped and in general patching communication has been lacking. We in the patching community understand that the coordination with other vendors means that this communication process was not easy, but needless to say, communication and follow-up in regards to side effects and known issues need to be faster and more communicative. On a regular basis, it is difficult to identify if there are known issues with an update and if our firms will be directly impacted. Often the patching known issues refer to undefined “third-party software” and we often must ask each other in the patching community If we were impacted and what vendors we were using. Clarity in documenting known issues would be greatly appreciated.
Impact on Azure
When one downloads a Windows 10 virtual machine in Azure and deploys it, is often built from a release from several months ago. These patching side effects we see in the traditional operating system channels, impact patching on Azure as well. Recently a RDP patch that was released in March and ultimately implemented fully in June impacted Azure virtual machines. The fact that you had to release a Knowledge Base article to instruct customers to go around this issue showcases that delays in patching Azure, and the lack of clear patching communication causes ripple effects to your cloud platforms.
An ask
I ask you to take time out of your very busy schedule to review these survey results and see the customer dissatisfaction. Many of your customers are not happy. We need action to fix these issues with patch quality.
As both a user of Microsoft software and a shareholder of Microsoft, I ask that you please take this feedback as it’s intended: We want Microsoft software to be such that we can indeed install all updates and patches immediately without reservation. As it stands right now, we do not trust the software and the patching quality enough to do so.
I thank you in advance for the opportunity to share with you your customers’ views.
Susan Bradley
Moderator at Patchmanagement.org
Writer on the topic of patches for Askwoody.com
July 29, 2018
Shortly thereafter, John Wilcox of Microsoft wrote a blog addressing some of the open letter, while not responding to it directly.
John Wilcox of Microsoft writes:
Windows 10 update servicing cadence
I’ve heard from many of you that you’d like a primer on our monthly Windows 10 quality update servicing cadence and terminology. In response, I’d like to share our guiding principles, then dive into them further to provide context for the quality updates themselves.
Guiding principles
We use the following principles for the monthly Windows servicing process:
- Be simple and predictable. IT managers should be able to plan for a simple, regular and consistent patching cadence. You shouldn’t need to stop what you’re doing to test and deploy an update. You should be able to plan a time, well in advance, to work on new updates. You also shouldn’t have to memorize multiple release schedules; the Windows release cadence should align with that of other Microsoft products.
- Be agile. In today’s security landscape, we must be able to respond to threats quickly when required. We should also provide you with updates quickly without compromising quality or compatibility.
- Be transparent. To simplify the deployment of Windows 10 in large enterprises or small businesses, you should have access to as much information as you need, and you should be able to understand and prepare for updates in advance. This includes guides for common servicing tools, simple release notes, and access to assistance or a feedback system to provide input.
Monthly quality updates
Next, I’d like to offer a quick summary of our monthly quality update types:
- At times referred to as our “B” release, Update Tuesday (most often referred to as Patch Tuesday) updates are published the second Tuesday of each month. These updates are the primary and most important of all the monthly update events and are the only regular releases that include new security fixes.
- An out-of-band release is any update that does not follow the standard release schedule. These are reserved for situations where devices must be updated immediately either to fix security vulnerabilities or to solve a quality issues impacting many devices.
- The “C” and “D” releases occur the third and fourth weeks of the month, respectively. These preview releases contain only non-security updates, and are intended to provide visibility and testing of the planned non-security fixes targeted for the next month’s Update Tuesday release. These updates are then shipped as part of the following month’s “B” or Update Tuesday release.
Now let’s align our principles to our monthly quality update releases.
Be simple and predictable
Across Microsoft, we have aligned on releasing updates on the second Tuesday of every month. It is the common, shared release date for Windows updates and for other products like Office. This consistent approach gives you the ability to simplify planning, testing, and deploying in advance.
For Windows, Update Tuesday is the most important monthly service event. This quality update does not include new features; instead, it serves to enhance system stability and security. We develop and test these updates quickly to minimize the impact of a vulnerability should one be made public, and they should be installed as soon as possible once released.
As an IT professional, you should have an established process and plan to ingest Update Tuesday releases each month.
Be agile
As much as we try to simplify and standardize our release cadence, there will always be situations that require agility, and an out-of-band update is necessary. As mentioned earlier, out-of-band updates are reserved for security vulnerabilities in active exploit or significant quality issues that must be fixed before the next B, C or D release.
Out-of-band updates may similarly require an out-of-band effort from IT pros to test and deploy them. While you should keep an eye out for out-of-band updates, they are rare and we have set a high threshold for releasing them.
Be transparent
Due to the sensitive nature of security fixes, Update Tuesday releases must be coordinated internally between our product teams and tested externally with our partners. Non-security releases do not have this limitation so, for the latest version of Windows 10, we typically release the majority of non-security updates the fourth week of every month, two weeks after the last Update Tuesday and two weeks before the next, in a “D” release.
During the two-week period between the initial release of a D release and our active push to install them on devices, you can test the updates included in the release and provide feedback, reducing the amount of testing necessary following Update Tuesday and, thereby, improving our ability to solve issues before they even happen.
For older versions of Windows 10 (as well as supported versions of Windows 7 and Windows 8.1), we sometimes release updates during the third week with a “C” release to provide you with extra time to test your legacy systems. In addition, as a new feature release draws near, we shift the current release to the “C” week, since there are fewer fixes and improvements necessary on the current version. Having just a few updates to test on the “C” week and none on the “D” week gives you the chance to concentrate on other responsibilities and frees up time for when the next semi-annual update arrives.
In most cases, “C” and “D” releases do not need be deployed to your broader device ecosystem. Instead, you can use these releases to identify any issues that could impact your next “B” deployment and provide feedback. This helps you get a head start on testing and understanding the potential impact of updates and gives you a chance to provide suggestions before those updates are officially released, providing a smoother and more tailored experience when the “B” release comes around.
The history of Update (aka Patch) Tuesday
Before I conclude this post, I wanted to provide a brief look back at the origins of our second Tuesday release schedule. “Patch Tuesday” was formalized in October 2003 after years of updates shipping whenever they were ready, a method called “ship-when-ready.” While this allowed fixes to go out almost immediately, it was a burden on IT pros, who were forced to start their workdays not knowing whether they would have to test and deploy an update. It was also a challenge for users, who sometimes had to reboot their computers multiple times a month to apply new updates, rather than just one reboot to apply a cumulative update, the process we use today.
We chose the second Tuesday at 10:00 a.m. Pacific time for two reasons:
- To provide you with a day (Monday) to deal with any other issues you need to work through from the previous week.
- To give you plenty of time to test the updates and deploy them to devices, then respond to any issues that may arise during the rest of the week.
Microsoft also spends the rest of the week watching for feedback and issues identified by businesses and consumers so we can begin preparing fixes immediately if necessary.
In addition to giving us time to respond to user feedback, the Update Tuesday schedule has enabled us to employ artificial intelligence in our deployment process. As John Cable noted back in June, “We continuously collect update experience data and retrain our models to learn which devices will have a positive update experience, and where we may need to wait until we have higher confidence in a great experience. Our overall rollout objective is for a safe and reliable update, which means we only go as fast as is safe.” This careful, strategic approach ensures that devices will be updated quickly and without any problems, even if we don’t have those specific devices available to test on, so that users can enjoy a seamless update experience.
While it would have been great for Microsoft to address Susan Bradley’s open letter directly, this may be the closest thing to an answer that can be expected for the time being. Hopefully, Microsoft will use the constructive user feedback to implement changes or increase assistance to those that need it.
As always, if we can be of help with your network or computer, give us a call here at RHYNO Networks. (855) 749-6648
