Windows and NTP

It’s important that Windows time is set correctly – but how Windows time works seems to be a poorly understood area.

In this article, I’ll try to clear up the concepts and explain what is, in my opinion, the best way to implement time services throughout your domain(s).

Background

  • Windows, by default, will automatically set its time from the domain controller which holds the FSMO role “PDC emulator”
  • In a multi-domain environment, the PDCe in the forest root domain is the overall master
  • Port 123 (NTP) is used for all communications
  • All other DC’s will, by default, look for the PDCe as their time source. There is no need to set anything here unless something has gone wrong.
  • All workstations will, by default, look for the PDCe as their time source. There is no need to set anything here unless something has gone wrong.
  • Windows 2016 time service offers (optionally) more accurate time services than previous versions – https://docs.microsoft.com/en-us/windows-server/networking/windows-time-service/accurate-time

Setting up NTP on the PDCe

I strongly recommend utilising group policy to set up NTP on your PDC emulator, not the command line. Using a group policy makes the settings a) obvious and b) easily transportable to new DC’s as your migrate upgrade in the future

  • Create a new GPO, I name mine “Domain Controller – Set NTP on PDCe”
    • Narrow it down to your PDCe by either
      • Removing “authenticated users” and adding your current PDCe (This will need to be manually updated if/when the PDCe role moves)
      • Utilising the WMI query “Select * from Win32_ComputerSystem where DomainRole = 5” (This will auto-update when the PDCe moves)
    • Set the following within the group policy
      • Computer Configuration > Administrative Templates > System > Windows Time Service > Time Providers
      • Enable Windows NTP Client: Enabled
      • Enable Windows NTP Server: Enabled
      • Configure Windows NTP Client: Enabled
        • NtpServer: <YourExternalNTPServer1>,0x1 <YourExternalNTPServer2>,0x1 (for Adelaide based clients, i used ntp.internode.on.net and ntp.adelaide.edu.au – a local ISP and a local University – but these could be any publicly available NTP server)
        • Type: NTP
        • CrossSiteSyncFlags: 2
        • ResolvePeerBackoffMinutes: 15
        • Resolve Peer BAckoffMaxTimes: 7
        • SpecilalPoolInterval: 3600
        • EventLogFlags: 0

Commands to check status and troubleshoot

  • w32tm /monitor – this exceedingly useful command will show you the status of all DC’s in the domain, where they are configured to get their time source from and their offset from the authoritative time source
  • if a domain controller is having issues
    • w32tm /config /syncfromflags:domhier /update
    • net stop w32time
    • net start w32time
  • w32tm /query /status

Using policy to set clients to look at AD for time

This is the default behaviour of windows – and you should not need to set this, however, for some places I’ve found we have had to

  • Computer Configuration -> Administrative Templates -> System -> Windows Time Service -> Time Providers
    • Configure Windows NTP Client: Enabled
      • NtpServer: <YourDC1>,0x1 <YourDC2>,0x1
      • Type: NTDS5
      • CrossSiteSyncFlags: 2
      • ResolvePeerBackoffMinutes: 15
      • ResolvePeerBackoffMaxTimes: 7
      • SpecilalPoolInterval: 3600
      • EventLogFlags: 0

References

https://docs.microsoft.com/en-us/windows-server/networking/windows-time-service/how-the-windows-time-service-works

https://docs.microsoft.com/en-us/windows-server/networking/windows-time-service/accurate-time

https://blogs.technet.microsoft.com/nepapfe/2013/03/01/its-simple-time-configuration-in-active-directory/

Configure NTP Time Sync using Group Policy

 

Active Directory 2019 and Exchange 2019 – what’s new

Cross-post with http://www.hayesjupe.com/active-directory-2019-and-exchange-2019-whats-new/

 

The short answer is – not much.

Exchange 2019 was released a few weeks back, but was effectively un-usable, as Exchange 2019 requires Windows Server 2019…. and Windows server 2019 got pulled from release (like Windows 10 1809) due to some issues.

Windows Server 2019 was re-released a few days ago, which allowed nerds everywhere (including me) to put Server 2019 and Exchange 2019 into a test environment.

The most striking thing that is immediately noticeable is that everything looks the same…. The install process, the GUI, the management, all looks the same as it did in 2016. To me, this is a good thing – while Microsoft of the past seemed to believe that moving functions between areas was good – some consistency is nice to have too.

 

Active Directory

First appearances indicate there is nothing new in AD 2019, the installation process and management is exactly the same as 2016.

While installing, there is not even an option to set the forest and domain functional level to “2019” – only 2016.

A quick look at the schema version indicates it has increased and quick google finds this article

https://blogs.technet.microsoft.com/389thoughts/2018/08/21/whats-new-in-active-directory-2019-nothing/

So, while there is something new in the schema, its an incredibly small update….. and there are no new features or functionality of any type to focus on.

 

Exchange 2019

Exchange 2019 is a bit the same as AD, everything appears to be the same as Exchange 2016, from the install process to the management interface.

A google comes up with this

Should you upgrade to Exchange Server 2019?

So there are some changes and feature updates – but these updates may not have an impact/matter to your organization.

 

I found these two releases interesting overall as

  • AD is the core of many enterprise networks
  • Exchange is a core business application

To see a new release of both of these products with very minimal improvements I think demonstrates where all Microsoft’s development effort is going (which, to be fair, we already knew)

Deploy Win32 applications with Intune

from http://www.scconfigmgr.com/2018/09/24/deploy-win32-applications-with-microsoft-intune/

WIN32 APPLICATION DEPLOYMENTS

The ability to “package” applications for deployment in Microsoft Intune is something that has been highly requested by many organisations making the move to management of devices through Intune. Although there is a fundamental difference in deploying applications through Configuration Manager and Intune, Microsoft is developing tools to provide similar functionality across the management stack. Up until now it it has been possible to deploy applications through Intune, this relied on a single MSI installation file with no external dependencies. In some cases this meant that repackaging of applications was the only method of deploying those business applications, thus being a time consuming process.

Today it is now possible to deploy applications through Intune without those restrictions, this process creates a packaged container of the setup files along with command line installation and uninstall commands.

 

This is a significant feature towards bringing Intune from the realms of “good for mobile device management only” to “also good for desktop management”.

SCCM currently does (and probably will for quite a while) have additional functionality which larger enterprises require – however, this is a good step in allowing smaller organisations more flexibility in their deployment options.

 

Note: as of 25/9, this feature is available with an Intune tenant running a preview of the GA release.

Creating large stand-alone media from SCCM – issues when the HP SPP 2018.03 is a package in the TS

We often create one task sequence for all workstation builds and another for all server builds, utilising task sequence variables to perform the decision making within the task sequence.

One of the downsides of this is that the task sequence binaries can get quite large, especially for server builds where we have current and legacy versions of the HP SPP, the Dell SUU and (at a minimum) server 2012 R2 and server 2016.

This isn’t an issue for network based builds, as non-required content is simply skipped, however, for media builds, it can lead to 40GB+ requirements, which the SCCM console doesnt handle well.

This is where Rufus comes in.

Rufus can help out by allowing use of larger hard drives (just tick the “list USB hard drives” option) and can apply bootable iso’s generated by SCCM (utilising the “unlimited size” option) to a USB hard drive.

This has been incredibly useful for us in the past, utilising large hard drives as USMT stores at slow link sites for client deployment.

In this instance, ive been using Rufus to apply a 50GB server build iso to a hard drive, but keep getting presented with a warning

“This ISO image seems to use an obsolete version of ‘vesamenu.c32’. Boot menus may not display properly because of this.”

Irrevelant of how you proceed (allow Rufus to download an update or not), the drive is not bootable.

Upon investigation of the rusfus logs and the resultant media, i found that syslinux.cfg was actually pointing to my HP SPP package.

This forum post then confirmed that Rufus is finding a syslinux.cfg, assuming that it is “real” bootable media and hence the ‘vesamenu.c32’ prompt.

After a few hours of troubleshooting and trying to get around it, i simply removed the “usb” and “system” folders from my HP SPP packages (as we wont be booting to it ever, its only for use in SCCM), re-created my standalone media iso – then used Rufus to write the bootable iso to the USB HDD, this time with no issues.

I realise this is a fairly obscure issue , but hopefully it helps someone.

 

 

Speed up offline servicing

Currently i am creating some server builds for a place which will be deploying large numbers of servers over the coming months.

One of things that is/was taking up a great deal of time was offline servicing for the base OS, primarily because the SCCM server is currently sitting on a virtual environment with disk that is struggling. With 2016, this isn’t so bad, as due to cumulative updates, there are only a few updates to be installed. With 2012 R2 however, there is a large number of updates – and the process continually fails due to the poor performance of the server.

One of things you can do to speed this process up is to remove unused images from your wim.

Both Server 2012 R2 and 2016 come with 4 images (with an index of 1 to 4) within the install.wim. These generally correlate with:

  • Index1 – Server 2012R2/2016 standard core
  • Index2 – Server 2012R2/2016 standard desktop experience
  • Index3 – Server 2012R2/2016 datacentre core
  • Index4 – Server 2012R2/2016 datacentre desktop experience

If you view Logs\OfflineServicingMgr.log during an offline servicing operation, you will notice lines that state things such as:

Applying update with ID xxxxxx on image at index 1

Then the same update will apply to image 2,3 and 4. In this enviornment, we are not deploying server core, so we only need indexes 2 and 4 (standard and datacentre with desktop).

We can view the indexes available within the wim by typing:

dism /get-imageinfo /imagefile:E:\<path to wim>\Install.wim

Then, if you dont need indexes 1 and 3 (as we dont in this scenario)

dism /delete-image /imagefile:E:\<path to wim>\Install.wim /index:1
dism /delete-image /imagefile:E:\<path to wim>\Install.wim /index:3

Now when you use offline servicing, each update will only be compared against 2 images, instead of 4, significantly reducing the processing time/disk usage, especially for 2012 R2 (where there are a large number of updates to apply)

This can also be used for client OS’s, such as Windows 10.

One important note – this will not reduce the size of the WIM. It will simply remove the index and save you time for offline servicing.

If your image is already in SCCM, then you must

  1. Go to Software Library | Operating systems | Operating system images
  2. Right click on the appropriate image | properties | Images tab
  3. Click on “reload”, then notice the dropdown has been reduce from 4 index’s, then hit “ok” to exit.
  4. Go into your task sequence
  5. Update the image index as required.

Importing updates into WSUS on Server 2016 fails

I ran into a situation recentlly where i needed to import a specific update from the Windows update catalog into WSUS (and in turn into SCCM)

I opened WSUS, clicked on “import updates”, seletced my update and was presented with

“This update cannot be imported into Windows Server Update Services, because it is not compatible with your version of WSUS”

Strange…. WSUS on 2016 is extremely similar to WSUS on 2012 R2… so whats going on here ?

Long story short… there seems to be issue with the url passed by the WSUS console when you click “import updates” to the browser.

When you first click on “Import updates”, IE will open (or you will use IE because it makes importing updates into WSUS easier) to

http://catalog.update.microsoft.com/v7/site/Home.aspx?SKU=WSUS&Version=10.0.14393.2248&ServerName=<servername>&PortNumber=8530&Ssl=False&Protocol=1.20

Simply change the last part “1.20” to “1.80” – and importing updates will now work

i.e

http://catalog.update.microsoft.com/v7/site/Home.aspx?SKU=WSUS&Version=10.0.14393.2248&ServerName=<servername>&PortNumber=8530&Ssl=False&Protocol=1.80

The importance of cleaning up WSUS – even if your are only using it for SCCM

In the distant past, we would generally say to clients “Leave WSUS admin alone – the SCCM settings will take precedence, so there is no point in using it”

As the years have passed, and the number of updates available has grown considerably, this is no longer the case. The SCCM settings still take precedence, however the pure number of updates has gotten so large that it can cause performance issues for the SCCM server – and even the IIS timeout to expire when SCCM is syncing updates. This generally results in and endless loop and 100% CPU for w3wp.exe.

Unfortunately, trying to list the updates in the WSUS console will often lead to the console crashing and the dreaded prompt to “reset server node”.

The best way to address this, isn’t really one of the many articles you will find by googling “sccm high CPU w3wp.exe” (or similar). Generally these will suggest modifying a number of entries in your IIS config to increase time outs, etc – these can assist, but they don’t really address the root cause of the issue – which is simply the huge number of updates.

The best way to resolve this is to simply reduce the number of updates shown in WSUS. This will reduce your memory usage, reduce the number of updates that SCCM has to scan each time, and generally put less load on your server.

There are two ways you can go about this:

 

The manual method

If you have the resources available, I’ve found increasing the RAM and CPU count on the SCCM server temporarily can help allevate the issue of the “reset node” issue.

Once you get in (it may take a few attempts), go to ‘updates’ > ‘all updates’, set the search criteria to “Any except declined” and “any” and hit refresh. Once loaded, add the “supersedence” column to the view and sort by that.

Decline all updates that are superseded. If you don’t clean up regularly, this number could be very high.

After this, you can create views to decline updates for products you no longer use (e.g. Windows 7 and Office 2010) or search for things including “beta”, “preview” and “itanium” and decline those updates as well.

After all that is done, run the server cleanup wizard. You will likely need to do this a number of times, as if your server is struggling already, this also will struggle to complete (and it seems to be quite poorly coded to handle large numbers of updates on low end servers)

 

The scripted method

A guy called “AdamJ” has written a very useful script which you can get at https://community.spiceworks.com/scripts/show/2998-wsus-automated-maintenance-formerly-adamj-clean-wsus . I know, I can see some of you recoiling at the suggestion of using a user submitted spiceworks script… they do have a whole bunch of people (just like the MS forums) suggesting to use “sfc /scannow” for anything and everything – which is a sign of a non-enterprise tech that has NFI… however, this script is really very good – and something I’ve been using for approx 2 years, with nothing but good things to say about it.

You can run it with the “-firstrun” parameter and it will, by default, clean out superseded updates – which is the main cause of huge update numbers, but it will also grab the ever annoying itanium, preview and expired updates. At approx line 629 of the script, you can also configure it to remove IE7,8,9,10 updates, Beta updates etc (or if you are one of the few people in the world with itanium, keep itanium updates!).

This script, unlike the console, will keep plugging away… and if it should happen to get stopped for whatever reason, will resume where it left off.

When removing obsolete updates, I have seen some clients (with lower spec servers) where this process can take a long time, so long that you may have to leave it overnight (or over the weekend), and sometimes restart the process.

This process will get you a fair chunk of the way, and allow you to then open the WSUS console and decline further updates, such as products you no longer use (Windows 7 and office 2010 are reasonably common ones these days), x86 updates if you have an all x64 environment, and in the case of Win 10, updates that no longer apply to you (e.g. if your entire fleet is on Win 10 1609 and 1703, you don’t need the 1511 updates)

After all this is complete, you do need to run the server cleanup wizard again – which does frequently crash and end up with the “reset server node” error. So you can re-run the WSUS cleanup script, or simply run the server cleanup wizard multiple times.

 

My experiences using these methods

I’ve found that in environments that were previously at 100% CPU, they start working again, and environments that were massively over-specc’d that didn’t have the high CPU issue went from using 6gb of RAM for the w3wp.exe process down to 500mb. This will obviously vary from environment to environment.

After this process is completed, you should be able to get into the WSUS console and run the server cleanup wiazrd, without crashes.

If you’re interested, you can also sync SCCM software updates and look at the wsyncmgr.log and see the far smaller list of updates it will sync against now.

Longer term, the AdamJ script does have monthly options that you can schedule in, or for our clients that are uncomfortable with that, simply get in and clean up once every 3 months or so, so you list of updates doesn’t get out hand.

The first cleanup is the biggest, after that, performing the same operations once every 3 months is plenty, and if you forget about and it happens to be once every 6 months instead, you’ll still be fine.

 

Taking it a step further – shrinking the SUSDB

One of the things which the AdamJ cleanup script does is truncate the SQL table “tbEventInstance” which uses up the majority of space in most WSUS databases that have been in use for a while.

If you are not comfortable with a script doing this, you can connect to the database and execute the following query against the “SUSDB” database – “truncate table tbEventInstance”.

If the DB is on a full version of SQL (which, if your running SCCM, i would argue the SUSDB should be on the same SQL instance, rather than installing an additional windows internal database), you can then create a maintenance plan to reindex, shrink etc the database.

If you are using Windows internal database, you can still install SQL management studio, then connect to “\\.\pipe\MICROSOFT##WID\tsql\query”, from there you can execute the truncate, shrink the database etc. Keep in mind that you cannot use maintenance plans with Windows internal databases.

 

What about large enviornments where you do require a wide range of updates ?

In large environments, you may not be able to decline entire product sets for extended periods (e.g. its relatively easy to move everyone onto Windows 10 (and get rid of all Win 7) for 2,000 PC’s, but not so easy for 50,000 PC’s), however, many of the points in this article still hold true.

  • The largest reduction in updates will still come from superceded updates
  • Language packs are another area where there’s lot’s of opportunity for reduction (e.g. if you require english and french – there are many other languages that can declined)
  • Ensure your SUSDB is on your full SQL instance…. that way you are running one less database instance (and therefore utilising less resources) and also have maintenance plans at your disposal
  • Use a maintenance plan to keep your SUSDB database optimal

 

SCCM Update Cleanup

It’s also worth noting that once the SUSDB has been cleaned up, SCCM will execute its own cleanup after the next sync. This cleanup removes obsolete Update CIs (Configuration Items) that corresponded to the items removed from the SUSDB. In most environments, this isn’t usually something noticeable, however in severely under resourced SCCM servers it can cause its own set of problems (though there’s not a huge amount you can do about it other than wait). This will generally present as the SCCM console locking up while it’s doing back-end SQL processes – and if you look at the SQL threads, you’ll see a WSUS related one blocking all other threads. Realistically your best option to resolve this is to increase the resources available to the server – and if that isn’t a possibility, settle in for a long wait!

Microsoft products – consolidated table of end of life dates

Microsoft product end of support dates are sometimes not easy to find and its not getting any better with the “current branch” releases and cloud solutions being governed by the Modern lifecycle policy.

The Modern lifecycle policy page further links to 3 product catagories, O365, Cloud platform and Dynamics. Unfortunately, its not clear (at least to me) how this helps with products such as SCCM current branch (be it 1606, 1702, 1706, 1710 or 1802) – however this information is available at another location

Likewise with the “traditional” products, most end of life information is available here – but to say that the information is difficult to search through is an understatement.

It also sometimes lacks detail, for example, there is no metion of the differing support for Windows 8.1 without update 1 and with update 1.

We have a number of clients that take the approach that while a server is running, to leave it there – and while I may personally not like this approach (i prefer to roll through the OS upgrades as they come out) – they have a valid approach and end of life information is important for them.

Keep in mind that everything listed below is end of extended support, not mainstream support – and i have taken some liberties (e.g. assumed that windows 8.1 is 8.1 with update 1)

Windows 10 dates have been sourced from the product lifecycle page, however this blog entry states than an additional 6 months has been granted to displayed Windows 10 versions.

If you find the below useful – cool. If i’ve got something wrong, or missed something that is key (in your opinion), please leave a comment.

 

ProductEnd of life date (end of extended support)
Windows 2003 SP2July 14, 2015
Windows 2008July 12, 2011
Windows 2008 R2 SP1Jan 14, 2020
Windows 2012Oct 10, 2023
Windows 2012 R2Oct 10, 2023
Windows 2016Jan 11, 2027
Windows XP SP3Jan 11, 2011
Windows Vista SP2April 4, 2017
Windows 7 SP1Jan 14, 2020
Windows 8Jan 12, 2016
Windows 8.1 with update 1Jan 10, 2023
Windows 10 RTM (1507)May 9, 2017
Windows 10 1511Oct 10, 2017 (End of support)
April 10, 2018 (additional servicing for enterprise and education)
Windows 10 1607April 10, 2018 (End of support)
Oct 9, 2018 (additional servicing for enterprise and education)
Windows 10 1703Oct 9, 2018 (end of support)
April 9, 2018 (additional servicing for enterprise and education)
Windows 10 1709April 9, 2019 (End of support)
Oct 8, 2019 (additional servicing for enterprise and education)
Office 2007Oct 10, 2017
Office 2010 SP2Oct 13, 2020
Office 2013April 11, 2023
Office 2016Oct 14, 2025
Lync 2010April 13, 2021
Lync 2013April 11, 2023
Skype for Business 2015April 11, 2023
Exchange 2010 SP3Jan 14, 2020
Exchange 2013 SP1April 11, 2023
Exchange 2016Oct 14, 2025
Forefront TMGJuly 12, 2011
Sharepoint 2010July 10, 2012
Sharepoint 2013 SP1April 11, 2023
Sharepoint 2016July 14, 2026
SCCM 2012 SP2July 12, 2022
SCCM 2012 R2 SP1July 12, 2022
SCCM 1606July 22, 2017
SCCM 1610Nov 18, 2017
SCCM 1702March 27, 2018
SCCM 1706July 31, 2018
SCCM 1710May 20, 2019
SCCM 1802Sept 22, 2019
SCVMM 2012 SP1July 12, 2022
SCVMM 2016Jan 11, 2027
SCOM 2012 SP1July 12, 2022
SCOM 2012 R2July 12, 2022
SCOM 2016Jan 11, 2027
SCORCH 2012 SP1July 12, 2022
SCORCH 2012 R2July 12, 2022
SCORCH 2016Jan 11, 2027
SCSM 2016Jan 11, 2027

Meltdown and Spectre patches available

Hi all,

For many of you that switched off over the xmas break (like me), you may have missed that there are now patches (released Jan 3rd 2018) for the creatviely (almost bond movie like) named vulnerabilities of “meltdown” and “spectre”.

You can find more detail on these Vulnerabilities  here – https://meltdownattack.com/

Advice for Microsoft client OS’s is here – https://support.microsoft.com/en-us/help/4073119/protect-against-speculative-execution-side-channel-vulnerabilities-in . The page still indicates to “contact your vendor” for microcode updates – which isnt going to overly helpful for standard end-users.

Advice for Microsoft server OS’s is here – https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution. There is additional work required over and above the patch for Remote Desktop and Hyper-V servers. Additionally, Windows server 2008 and 2012 are not yet patched, only Server 2008 R2/2012 R2/2016/1709 – read into that what you will.

The register has a good article (as they do most of time) cutting through the intel PR bullshit. Importantly, there has been various reports of performance impacts after installing these patches – but it is still too early to tell exactly how large/important those perfomrance impacts are.

There are links to many vendor advisories (which in turn have links to updates) @ https://meltdownattack.com/ – which is quite useful.

The patches and additional mitigations are fairly easy to implement if you have patching/management infrastructure in place – but if your company needs any assistance, we’re happy to help too.