Server Down
“Context -
An interesting journey that I wanted to share with my colleagues in an effort to display the level of diligence required when tracking down issues. ”
posted Jan 30, 2018, 9:59 AM
Captains log star date 1-29-2018......
The day started bright and early with the lovely sound of a Hangout reminding me that this was Monday. In true Monday fashion there was a crisis behind this particular message. No it was not a greeting by any means. This was an all out assault on our beloved SCCM server. The app catalog was down (no application installs) and all critical services were down.
After a quick attempt of various solutions yielded no results, I decided this might be a case for Microsoft due to the somewhat critical nature (depends on who u talk to) of this issue. So I proceed to open a case with the mighty MS and quickly determined that support ended this morning at 6AM (yes I asked the support guru exactly when on 1-29-2018 this death sentence commenced while hoping for a window of opportunity).
I gathered my thoughts and enlisted the assistance of our support and licensing pit bull; Diamond Dave Palmer. He could work the background while I began swabbing for DNA in order to help solve this mystery. As I sat back down to my desk with a freshly filled cup of Starbucks finest, I noticed the following:
Yikes...what in the ~!@#$%^&* is going on today. This is the perfect Monday morning storm. No joke...it really happened and as usual in my neck of the woods there are some bozo's down the street who decided to cut a cable. I even had to go confirm (yes its real).
Moving on I decided to do what any seasoned veteran would do. Hmmm... what has changed??? I reviewed some logs, conducted some interviews with a few "persons of interest" and decided to focus my attention on the Ivanti agent that was recently installed. Looking back I knew that this had caused some issues before with SCCM servers in other domains. I had previously determined that a reinstall of the MP would typically fix this issue. No such luck here. Like I stated before all services were stopped. It was at this point, I decided to uninstall the Ivanti agent. DUH, why did I wait so long? After the uninstall and restart completed, a quick examination reminded me of the Hall and Oates song:
She's Gone
She's gone, she's gone
Oh I, oh I
I better learn how to face it
She's gone, she's gone
Oh I, oh I
I'd pay the devil to replace her
She's gone, and she's gone
Oh why, what went wrong?
At this point its late in the game....people are hitting me up left and right...still no word from our MS support dilemma...I decided to review some logs that are specifically created during a "crash". Upon review I noticed several failures that were occurring with DLL / assembly....kind of library related.
So I started thinking what is it that Ivanti might change, leave behind, residual u know. This prompted me to look at the installed applications. I sorted these by date and noticed several Visual C++ redistributables were recently installed.
This got me thinking what the requirements were from an SCCM standpoint. After checking I saw the following illustration below. If u do a quick comparison u will notice that Visual C++ 2013 Redistributable (x64) is required and is missing from above.
Digging a bit deeper into the event logs allowed me to begin to confirm my suspicions. Below is the Ivanti agent installing.
Next... the actual removal of the library.
Checking the Ivanti server agent INI reveals that the x86 version did install, thereby uninstalling the x64 version.
And verification of the version of the redistributable from the Ivanti server.
Good news there is a FIX for this particular issue - https://support.microsoft.com/en-us/kb/3138367
After installing the library, I restarted the services and BOOM, we have lift off.
There was a sizable backlog to process but soon thereafter we were back to a normal operating state.
All is good in the world again...........For now!