The importance of cleaning up WSUS – even if your are only using it for SCCM

In the distant past, we would generally say to clients “Leave WSUS admin alone – the SCCM settings will take precedence, so there is no point in using it”

As the years have passed, and the number of updates available has grown considerably, this is no longer the case. The SCCM settings still take precedence, however the pure number of updates has gotten so large that it can cause performance issues for the SCCM server – and even the IIS timeout to expire when SCCM is syncing updates. This generally results in and endless loop and 100% CPU for w3wp.exe.

Unfortunately, trying to list the updates in the WSUS console will often lead to the console crashing and the dreaded prompt to “reset server node”.

The best way to address this, isn’t really one of the many articles you will find by googling “sccm high CPU w3wp.exe” (or similar). Generally these will suggest modifying a number of entries in your IIS config to increase time outs, etc – these can assist, but they don’t really address the root cause of the issue – which is simply the huge number of updates.

The best way to resolve this is to simply reduce the number of updates shown in WSUS. This will reduce your memory usage, reduce the number of updates that SCCM has to scan each time, and generally put less load on your server.

There are two ways you can go about this:

 

The manual method

If you have the resources available, I’ve found increasing the RAM and CPU count on the SCCM server temporarily can help allevate the issue of the “reset node” issue.

Once you get in (it may take a few attempts), go to ‘updates’ > ‘all updates’, set the search criteria to “Any except declined” and “any” and hit refresh. Once loaded, add the “supersedence” column to the view and sort by that.

Decline all updates that are superseded. If you don’t clean up regularly, this number could be very high.

After this, you can create views to decline updates for products you no longer use (e.g. Windows 7 and Office 2010) or search for things including “beta”, “preview” and “itanium” and decline those updates as well.

After all that is done, run the server cleanup wizard. You will likely need to do this a number of times, as if your server is struggling already, this also will struggle to complete (and it seems to be quite poorly coded to handle large numbers of updates on low end servers)

 

The scripted method

A guy called “AdamJ” has written a very useful script which you can get at https://community.spiceworks.com/scripts/show/2998-wsus-automated-maintenance-formerly-adamj-clean-wsus . I know, I can see some of you recoiling at the suggestion of using a user submitted spiceworks script… they do have a whole bunch of people (just like the MS forums) suggesting to use “sfc /scannow” for anything and everything – which is a sign of a non-enterprise tech that has NFI… however, this script is really very good – and something I’ve been using for approx 2 years, with nothing but good things to say about it.

You can run it with the “-firstrun” parameter and it will, by default, clean out superseded updates – which is the main cause of huge update numbers, but it will also grab the ever annoying itanium, preview and expired updates. At approx line 629 of the script, you can also configure it to remove IE7,8,9,10 updates, Beta updates etc (or if you are one of the few people in the world with itanium, keep itanium updates!).

This script, unlike the console, will keep plugging away… and if it should happen to get stopped for whatever reason, will resume where it left off.

When removing obsolete updates, I have seen some clients (with lower spec servers) where this process can take a long time, so long that you may have to leave it overnight (or over the weekend), and sometimes restart the process.

This process will get you a fair chunk of the way, and allow you to then open the WSUS console and decline further updates, such as products you no longer use (Windows 7 and office 2010 are reasonably common ones these days), x86 updates if you have an all x64 environment, and in the case of Win 10, updates that no longer apply to you (e.g. if your entire fleet is on Win 10 1609 and 1703, you don’t need the 1511 updates)

After all this is complete, you do need to run the server cleanup wizard again – which does frequently crash and end up with the “reset server node” error. So you can re-run the WSUS cleanup script, or simply run the server cleanup wizard multiple times.

 

My experiences using these methods

I’ve found that in environments that were previously at 100% CPU, they start working again, and environments that were massively over-specc’d that didn’t have the high CPU issue went from using 6gb of RAM for the w3wp.exe process down to 500mb. This will obviously vary from environment to environment.

After this process is completed, you should be able to get into the WSUS console and run the server cleanup wiazrd, without crashes.

If you’re interested, you can also sync SCCM software updates and look at the wsyncmgr.log and see the far smaller list of updates it will sync against now.

Longer term, the AdamJ script does have monthly options that you can schedule in, or for our clients that are uncomfortable with that, simply get in and clean up once every 3 months or so, so you list of updates doesn’t get out hand.

The first cleanup is the biggest, after that, performing the same operations once every 3 months is plenty, and if you forget about and it happens to be once every 6 months instead, you’ll still be fine.

 

Taking it a step further – shrinking the SUSDB

One of the things which the AdamJ cleanup script does is truncate the SQL table “tbEventInstance” which uses up the majority of space in most WSUS databases that have been in use for a while.

If you are not comfortable with a script doing this, you can connect to the database and execute the following query against the “SUSDB” database – “truncate table tbEventInstance”.

If the DB is on a full version of SQL (which, if your running SCCM, i would argue the SUSDB should be on the same SQL instance, rather than installing an additional windows internal database), you can then create a maintenance plan to reindex, shrink etc the database.

If you are using Windows internal database, you can still install SQL management studio, then connect to “\\.\pipe\MICROSOFT##WID\tsql\query”, from there you can execute the truncate, shrink the database etc. Keep in mind that you cannot use maintenance plans with Windows internal databases.

 

What about large enviornments where you do require a wide range of updates ?

In large environments, you may not be able to decline entire product sets for extended periods (e.g. its relatively easy to move everyone onto Windows 10 (and get rid of all Win 7) for 2,000 PC’s, but not so easy for 50,000 PC’s), however, many of the points in this article still hold true.

  • The largest reduction in updates will still come from superceded updates
  • Language packs are another area where there’s lot’s of opportunity for reduction (e.g. if you require english and french – there are many other languages that can declined)
  • Ensure your SUSDB is on your full SQL instance…. that way you are running one less database instance (and therefore utilising less resources) and also have maintenance plans at your disposal
  • Use a maintenance plan to keep your SUSDB database optimal

 

SCCM Update Cleanup

It’s also worth noting that once the SUSDB has been cleaned up, SCCM will execute its own cleanup after the next sync. This cleanup removes obsolete Update CIs (Configuration Items) that corresponded to the items removed from the SUSDB. In most environments, this isn’t usually something noticeable, however in severely under resourced SCCM servers it can cause its own set of problems (though there’s not a huge amount you can do about it other than wait). This will generally present as the SCCM console locking up while it’s doing back-end SQL processes – and if you look at the SQL threads, you’ll see a WSUS related one blocking all other threads. Realistically your best option to resolve this is to increase the resources available to the server – and if that isn’t a possibility, settle in for a long wait!

Microsoft Surface Laptops and SCCM OS Deployment

So you’ve just picked up your shiny new Microsoft Surface Laptop and want to put your current Windows 10 Enterprise SOE on it. Of course you have it all set up in SCCM (right?), so you figure you’ll kick off a build and be up and running in a half hour or so.

Granted, the Surface Laptop isn’t marketed as an Enterprise device and comes with Windows 10 S pre-installed, but you can do a simple in-place upgrade to Windows 10 Pro. So obviously it should be capable of running any of the Windows 10 OS’s (which it is).

Unfortunately it’s not quite so simple to get them to build through SCCM. Below are the three issues I experienced when trying to build one via SCCM (and was silly enough to think it’d be straight plug and play!):

  • PXE doesn’t work. I’ve got a couple of USB to Ethernet adapter that works fine with PXE on other devices – but they simply don’t work with the Surface Laptop. Updated the firmware on the laptop to the latest (as Surface Pro’s had similar issues with old firmware), and still had the same issue. Whether this is unique to the Surface Laptop I have, or all of them – I’m not sure (as I’ve only got the one so far).
  • The keyboard doesn’t work in WinPE. Touchpad mouse works fine – so you can kick off the task sequence – you just can’t use the keyboard at all. Fine if your task sequence is fully automated, but if you need to enter a computer name or troubleshoot issues, you’re going to need to import some drivers.
  • SecureBoot. Once I’d decided to use a bootable USB to kick off the task sequence, it started running through fine. Rebooted into Windows Setup, then proceeded to complain about the “Microsoft-Windows-Deployment” component of the Specialize pass in the Unattend.xml. Long story short, it’s caused by SecureBoot (more on this further down).

So lets break this down into how to fix each.

PXE: Simply put, I haven’t been able to fix this. Updated firmware, used different PXE capable USB to Ethernet devices. As above, I’m not sure if this is just the one I have, or all of them – or even if it’ll be fixed in newer firmwares. At this stage, it looks like the only option is SCCM boot media. Since the Surface Laptop only has a single USB port, you’ll either need to use a USB/Ethernet combo (like one of these – not sure if they’re PXE capable or not, haven’t tested), or you’ll need to use an external USB hub. Note: you can initiate PXE from full power off by pressing Volume Down + Power. If you want to USB boot, you need to go into the UEFI setup by pressing Volume Up + Power. On the boot settings page, you can swipe left on any of the boot options to initiate a ‘boot now to selected device’.

SecureBoot/Unattend.xml: This one is a little more tricky. Essentially if you look at the setupact.log file, you’ll see something along the lines of “[SETUPGC.exe] Hit an error (hr = 0x800711c7) while running [cmd / c net user Administrator /active:yes]“. 0x800711c7 translates to “Your organization used Device Guard to block this app”. According to this Microsoft article, it’s due to the Code Integrity policy in UEFI being cleared as part of the OS upgrade (from Windows 10 S to something else). Since Windows 10 S won’t let you launch certain executables, it blocks them during the OS Deployment as well. Supposedly fixed by a reboot – but by then you’ve broken your OS deployment. The obvious fix is to disable SecureBoot, then re-enable it after the task sequence completes. I’m not really a fan of this approach, and I’m not sure how you’d automate it (even if you could).

During my research, I found a reference to someone suggesting that applying the Surface Laptop drivers during the OS Deployment actually fixes the issue. I’m not 100% sure if this is the case – as I disabled SecureBoot, rebooted and re-enabled SecureBoot before testing out this process – but doing so afterwards did actually work (with SecureBoot enabled). Since I’ve only got the one device so far, I’ll update this blog once I’ve tested on others. I’ve put some instructions further below on how to import the drivers into SCCM (as they’re provided in MSI format now, and you’ll need them to apply before Windows Setup).

Keyboard during WinPE: Essentially, you need 5 drivers imported into the boot image for the keyboard to work (as detailed by James in this technet blog) Below is an image of the drivers required in the boot image:

Adding Surface Laptop Drivers to SCCM

Surface drivers all now come in MSI format – which is good for normal deployments, but doesn’t really help you for OS Deployments (assuming you want to apply the drivers prior to Windows Setup). After downloading the latest Surface Laptop drivers from Microsoft, you can use the following example command to extract them (which goes from around 300MB to a whopping 1.3GB):

msiexec.exe /a C:\Temp\SurfaceLaptop_Win10_15063_1802008_1.msi targetdir=C:\Temp\Extract /qn

From there you can import the “Drivers” sub-folder into SCCM as per usual. If you plan on applying them as a proper ‘Driver Package’ during the OS Deployment, you’ll need to import them into their own driver package, distribute it to the relevant Distribution Points, then add it to the task sequence. You can use the following WMI query to filter it to just Surface Laptops:

SELECT * FROM Win32_ComputerSystem WHERE Model LIKE “%Surface Laptop%”

Obviously you can now also add the keyboard drivers to the required boot image!

Meltdown and Spectre patches available

Hi all,

For many of you that switched off over the xmas break (like me), you may have missed that there are now patches (released Jan 3rd 2018) for the creatviely (almost bond movie like) named vulnerabilities of “meltdown” and “spectre”.

You can find more detail on these Vulnerabilities  here – https://meltdownattack.com/

Advice for Microsoft client OS’s is here – https://support.microsoft.com/en-us/help/4073119/protect-against-speculative-execution-side-channel-vulnerabilities-in . The page still indicates to “contact your vendor” for microcode updates – which isnt going to overly helpful for standard end-users.

Advice for Microsoft server OS’s is here – https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution. There is additional work required over and above the patch for Remote Desktop and Hyper-V servers. Additionally, Windows server 2008 and 2012 are not yet patched, only Server 2008 R2/2012 R2/2016/1709 – read into that what you will.

The register has a good article (as they do most of time) cutting through the intel PR bullshit. Importantly, there has been various reports of performance impacts after installing these patches – but it is still too early to tell exactly how large/important those perfomrance impacts are.

There are links to many vendor advisories (which in turn have links to updates) @ https://meltdownattack.com/ – which is quite useful.

The patches and additional mitigations are fairly easy to implement if you have patching/management infrastructure in place – but if your company needs any assistance, we’re happy to help too.

Preparing to update for Intel® Management Engine Critical Firmware Update (Intel SA-00086)

Intel released a security advisory yesterday (22/11/2017) advising of vulnerabilities with their management engine firmware – which can he found here

The reason why this is concerning for corporate customers is that basically every PC, Server and laptop that you have, most likely will be exposed, as the vast majority of corporate level hardware contains this hardware.

Intel has provided a detection tool available at https://downloadcenter.intel.com/download/27150. Contained within is a couple of applications, the GUI version will probably be handy for those orgs with only a handful of makes and models – where-as the command line tool will be more useful for larger organisations to run and centralise results via tools such as SCCM.

 

The original page contains links for various vendors update, the downside is that there doesnt seem to be many patches as yet, as per this page (at the time of writing, all Dell entries are marked as “TBD”, where-as Lenovo lists a target date of 24/11/2017)

 

So, what can you do while waiting for the patches to be released?

  • Test your systems using the provided tool to see if they are vulnerable. Testing at least one of each make/model will give you a good idea what you will need to target
  • Setup SCCM collections ready to go, which would entail
    • A collection for each make and model (I imagine many places would have this already)
    • A series of collections which each include one/make model and the criteria “AMT Agent – Flash is NOT equal to <insert version number of patched FW – once released>” – This will enable you quickly identify machines which need to be targeted with the update
    • (Please note the above is an assumption, some vendors may patch this via a BIOS update, in which case, the BIOS version may be the identifier instead)

 

Please post a comment if there are any questions – and ill update thios post once the patches are released – if there are any gotcha’s we run into.

Importing AD powershell module into Windows PE and then using encrypted creds

Powershell makes life much easier than vbscript…. however it does have its downsides…  signing policy can sometimes be a bit of pain and the modules you need have to be available…. which is an issue in particular for Windows PE.

Mick (good aussie name there) was nice enough to write a blog on how to import powershell into PE – without having to add it statically to the boot wim – http://mickitblog.blogspot.com.au/2016/04/import-active-directory-module-into.html

As a little shortcut from the blog, you can copy both the x86 and x64 required directories via robocopy rather than determining the version via powershell.

The next step however is the more important one…. a task sequence doesn’t allow us to run a powershell command in PE with credentials, we need a secure way of running the command. In this case, we want to delete a computer object….

Step 1 – Generate a key file (perform on any full OS)

$KeyFile = “\\sccm\PSource$\OSD.DeleteComputer\DeleteComputer.key“

$Key = New-Object Byte[] 16

[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($Key)

$Key | out-file $KeyFile

 

Step 2 – Encrypt a password using the key

$PasswordFile = “\\sccm\PSource$\OSD.DeleteComputer\DeleteComputer.txt“

$KeyFile = “\\sccm\PSource$\OSD.DeleteComputer\\DeleteComputer.key“

$Key = Get-Content $KeyFile

$Password = “Your password here” | ConvertTo-SecureString -AsPlainText -Force

$Password | ConvertFrom-SecureString -key $Key | Out-File $PasswordFile

 

Step 3 – Create your script utilising the creds – (Below is the one I use to delete a computer object)

Import-module ActiveDirectory

#SCCM TS Object
$tsenv = New-Object -COMObject Microsoft.SMS.TSEnvironment

#SCCM Variables
$CompName = $tsenv.Value(“_SMSTSMachineName”)

# Get current path in order to get encrypted password
$MyDir = [System.IO.Path]::GetDirectoryName($myInvocation.MyCommand.Definition)
$User = “Domain\Account”
$PasswordFile = “$MyDir\DeleteComputer.txt”
$KeyFile = “$MyDir\DeleteComputer.key”
$key = Get-Content $KeyFile
$MyCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $User, (Get-Content $PasswordFile | ConvertTo-SecureString -Key $key)

# Remove the computer from AD
Remove-ADComputer -Identity $CompName -server <DC name required> -Credential $MyCredential -confirm:$false

 

One obvious risk is, this is not very secure. It will stop a random snooper type person from seeing a plain text password…. but it will not stop someone who is capable of pressing F8 to get into the running TS (if you have it enabled) and then grabbing the key and txt and being able to use them…. So, take this into consideration when choosing to use (or not use) for your environment.

UEV now included in Windows 10 1607 (and above)

User Experience Virtualization (UEV) use to be part of the MDOP packs…. however MDOP’s last update was in 2015…. leaving some of us wondering what was happening to awesome tools contained within.

Given Microsoft’s strong movement towards cloud platforms, it seemed likely that these tools were dead.

Fortunately for UEV, its now included in Windows 10 Enterprise as a default service, for versions 1607 and 1703 (and we may be able to assume future releases as well). Some details on the release are here – https://docs.microsoft.com/en-us/windows/configuration/ue-v/uev-whats-new-in-uev-for-windows

Unfortunately, the documentation is somewhat unhelpful.

The UEV documentation is located here – https://docs.microsoft.com/en-us/microsoft-desktop-optimization-pack/uev-v2/deploy-required-features-for-ue-v-2x-new-uevv2

However, there are a couple of quite important things that anyone deploying this should be aware of

  • Even though it isn’t stated anywhere in the doco, and seems quite counter-intuitive based on what’s presented in the GPO settings, the default Microsoft included templates do not automatically register on clients. These can be copied to your custom templates path, or you can register them with powershell on each machine as per http://ccmexec.com/2017/02/synchronizing-ie-favorites-with-ue-v-in-windows-1607/
  • The UEV template generator is part of the ADK (1607 or 1703) – however, it does not show up if you try and run the ADK installer on Windows 8.1 or server 2012 R2. I haven’t tried on Windows 10 versions below 1607 or 1703 – but it will show/be installable on those versions.

ConfigMgr CB 1610 released

In case you missed it with all the excitement that the end of year brings, Microsoft have released the latest build of ConfigMgr 1610 to all ConfigMgr customers.

Microsoft have released information on a number of features in this release that are going to improve projects I’m currently involved with, as well as some enhancements for ConfigMgr Admins day-to-day usage;

  • Windows 10 Upgrade Analytics integration allows you to assess and analyse device readiness and compatibility with Windows 10 to allow smoother upgrades.
  • Office 365 Servicing Dashboard and app deployment to clients features help you to deploy Office 365 apps to clients as well as track Office 365 usage and update deployments.
  • Software Updates Compliance Dashboard allows you to view the current compliance status of devices in your organization and quickly analyse the data to see which devices are at risk. Monitoring > Security > Software Updates Dashboard.
  • Client Peer Cache is a new built-in solution in Configuration Manager that allows clients to share content with other clients directly from their local cache with monitoring and troubleshooting capabilities.
  • Enhancements in Software Center including customizable branding in more dialogs, notifications of new software, improvements to the notification experience for high-impact task sequence deployments, and ability for users to request applications and view request history directly in Software Center.
  • New remote control features including performance optimization for remote control sessions and keyboard translation.

This release also includes new features for customers using Configuration Manager connected with Microsoft Intune. Some of the new feature include:

  • New configuration item settings and improvements now only show settings that apply to the selected platform. We also added lots of new settings for Android (23), iOS (4), Mac (4), Windows 10 desktop and mobile (37), Windows 10 Team (7), Windows 8.1 (11), and Windows Phone 8.1 (3).
  • Lookout integration allows to check device’s compliance status based on its compliance with Lookout rules.
  • Request a sync from the admin console improvement allows you to request a policy sync on an enrolled mobile device from the Configuration Manager console.
  • Support for paid apps in Windows Store for Business allows you to add and deploy online-licensed paid apps in addition to the free apps in Windows Store for Business.

I’ve deployed the 1610 update at a couple of clients now in production use, and the upgrade has gone smoothly. Be prepared; the update can take a couple of hours to complete. I have hit one hurdle with implementing the new Office 365 Servicing Dashboard, the dashboard is stuck at loading data, this issues has been reported to Microsoft by several people and the product group is aware of it. This is unfortunate as this was one of the features that I believe will be beneficial for rolling out and managing Office 365.