iThinkVirtual™

Fixing A Corrupt Domain Controller – Stop Code 0x00002e2

Yesterday morning I discovered that my Synology NAS had an unexpected shutdown in the middle of the night while my homelab VMs/workloads were still running.  This caused both of my Domain Controllers databases to become corrupt resulting in being unable to boot those machines.  When attempting to boot them, they would get stuck in a BSOD boot-loop and would display a Stop Error Code of 0x00002e2.

.

After some research I was able to figure out how to recover my VMs and get them to boot up again.  This had happened to me once before sometime earlier this year and luckily I remembered that I had taken some notes on how to fix it so I figured this time I would put together a formal “How To:” guide to have it documented for myself and hopefully to help others facing this issue as well.  So without further adieu…let’s get to it!

.

To start, you will need to power-on the machine and then keep pressing the F8 key to bring up the “Advanced Boot Options” boot menu.  Navigate down to Directory Services Repair Mode enter press Enter to boot you into Safe Mode.

When you reach the login screen, log in with the Local Administrator account since Active Directory Domain Services are obviously unavailable.

Once at the Desktop, open an elevated Command Prompt window.

As a best practice, I feel it is always wise and important to make a backup of the files before doing any modifications.  Since we will be backing up the NTDS directory, create a directory at your preferred location to store the backup files.  I chose to make a backup folder on the root of “C:\” and a sub-directory with the name/date of the directory I am backing up.

md C:\Backup\NTDS_11122017

Then copy everything from the “C:\Windows\NTDS” directory to this new location using xcopy.

xcopy C:\Windows\NTDS\* C:\Backup\NTDS_11122017 /E /Y /V /C /I

Once done, let’s rename any .log file extensions in the NTDS directory to .log.old

cd C:\Windows\NTDS

ren *.log *.log.old

Now, this is where we get to the good stuff!

First, run the following command to repair the database.

esentutl /p "C:\Windows\NTDS\ntds.dit"

This will display the following warning message, click “OK

Let it do its thing and you will see the following once complete.

Now we need to use the NTDS Utility (ntdsutil.exe) to activate the NTDS instance and compact the DB to a new file which will then be used to overwrite the original.  I will be compacting it to a new TEMP directory within the NTDS directory.  The command will automatically create the new directory if it’s not already present.  Enter the following commands.

ntdsutil

activate instance ntds

files

compact to C:\Windows\NTDS\TEMP

If successful, you will be presented with instructions to copy the newly compacted file to the NTDS directory, overwriting the original, and also to delete any .log files in the NTDS directory.  Before we can do that we need to exit out of the ntdsutil.  Type quit twice to exit.

quit

quit

Let’s follow those instructions and also delete the *.log.old files we renamed earlier.

copy "C:\Windows\NTDS\TEMP\ntds.dit" "C:\Windows\NTDS\ntds.dit"

Yes

Ensure you are still in the NTDS directory before entering the following delete commands.

del *.log

del *.log.old

The hard part is now over!  Let’s go ahead and reboot the machine normally.

If all goes well and as expected, your machine will boot successfully and you can log in again with an Active Directory Domain Admin account.

Awesome!  Well, I hope you’ve found this guide useful.  Please feel free to share this and provide me some feedback/comments below.  Thanks for reading!

 

-virtualex-

How To: Create A VMFS5 Datastore On A USB Drive

Create A VMFS5 Datastore On A USB Drive

Ever wondered if it was possible to use a USB Drive as a VMFS5 datastore in VMware vSphere 6.0?  I sure know that I have!  Not that I would like to run any VM’s on said datastore, as I’m sure performance would not be optimal, but instead to test its functionality and use it for storing ESXi host logs for example.  Well, I ran into an issue today where I needed to unmount all of my NFS mounts on ESXi 6.0 U2 in order to recreate some of the volumes before remounting them.  The problem was that I was unable to unmount one of my volumes because it was bound to the ESXi host for scratch logs.  As I didn’t have a spare drive of any sort to attach to my host so that I could reconfigure the location for scratch logs, I began tinkering with the idea of using a small USB drive as a temporary datastore for these logs.

After doing a little research, I came across a post from Florian Grehl aka @virten  showing exactly how to accomplish this so I figured I’d share the process of doing so.  Keep in mind that this should only be used for testing purposes and should not be used in production environments.  This is unsupported by VMware.  Here we go!

Ensure that the USB device is unplugged from the ESXi host then begin by connecting to your ESXi host and stopping the USB arbitrator service.  This service is responsible for allowing USB device passthrough from an ESXi host to a virtual machine, so keep in mind that you will no longer be able to pass through USB devices to VM’s until this is restarted.  (Note: restarting service after creating and mounting USB datastore will break connectivity and recognition of the USB datastore).   To stop the service, run the following command:

/etc/init.d/usbarbitrator stop

2016-08-11_17-12-192016-08-11_17-12-35

Optionally, if you’d like to permanently disable the service so it persists thru reboots, run the following command:

chkconfig usbarbitrator off

Plug the USB drive into your ESXi host.  For the purposes of this tutorial, I am using a small Lexar 8GB USB device.  If you navigate to the storage devices section on your host, you should now see the connected USB device is recognized by the hypervisor.  Make note of the device identifier number (mpx.vmhbaXX) for this device.

vSphere Client:

 2016-08-14_16-53-34

vSphere Web Client:

2016-08-14_17-02-22

vSphere HTML5 Web Client:

2016-08-14_17-02-44

You can also list the device information to determine the identifier by running the following command:

ls /dev/disks/

2016-08-11_17-13-09

As we can see, my identifier is mpx.vmhba40:C0:T0:L0 for this device which also matches the identifier from the GUI pics above.  Note: The other USB (mpx.vmhba32:C0:T0:L0) is a separate USB where ESXi is installed on.

2016-08-14_16-48-33

Next, we need to create a GPT (GUID Partition Table) label on the device.  To do so, run the following command using the correct identifier for the drive.  In my case, I will run with mpx.vmhba40 for all of the following commands.  Be sure to change this to your correct ID.

partedUtil mklabel /dev/disks/mpx.vmhba40\:C0\:T0\:L0 gpt

2016-08-14_17-04-20

Now run the following command to get the partition table information.

partedUtil getptbl /dev/disks/mpx.vmhba40\:C0\:T0\:L0

2016-08-14_17-05-36

This returned the following output for me…

gpt

973 255 63 15634432

2016-08-14_17-06-01

Next, we need to create a partition in which you will need to know the start sector and end sector which all depend on the size of the device drive and GUID.  As an FYI…

  • The start sector is always 2048
  • The GUID for VMFS is always AA31E02A400F11DB9590000C2911D1B8
  • The end sector is calculated using the values obtained by running the previous command.
    • Formula: 973 x 255 x 63 – 1 = 15631244

2016-08-11_17-33-01

We can also run the following command to calculate the end sector value.  This should return an identical value that matches the previous calculation. 

eval expr $(partedUtil getptbl /dev/disks/mpx.vmhba40\:C0\:T0\:L0 | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1

2016-08-14_17-10-052016-08-14_17-10-45

If everything has gone smoothly so far, you should be ready to create the VMFS partition.  Run the following command, ensuring to replace the identifier and end sector values with your own.

partedUtil setptbl /dev/disks/mpx.vmhba40\:C0\:T0\:L0 gpt "1 2048 15631244 AA31E02A400F11DB9590000C2911D1B8 0"

2016-08-14_17-12-022016-08-14_17-12-42

Lastly, we need to format the partition with VMFS using vmkfstools.  Do so by running the following (Note: “” in the command below can be any name you like so feel free to use a different name for your datastore):

vmkfstools -C vmfs5 -S USB_Datastore /dev/disks/mpx.vmhba40\:C0\:T0\:L0:1

2016-08-14_17-13-58

Sit tight…wait about one minute…and…voila!  

2016-08-14_17-16-35

After a quick rescan/refresh you should now have and see your mounted VMFS5 USB Datastore!

vSphere Client:

2016-08-14_17-17-40

vSphere Web Client:

2016-08-14_17-18-21

vSphere HTML5 Web Client:

2016-08-14_17-18-52

After I changed the “Syslog” configuration for my scratch logs to use this new datastore, I was finally able to unmount my NFS datastores.  I hope this helps so please feel free to comment below.

Shoutout to Florian Grehl for his wonderful post!

Cheers!

-virtualex-

Pingbacks:

USB Devices as VMFS Datastore in vSphere ESXi 6.0

PernixData FVP Freedom Woes With Missing Supermicro System UUID

PernixData FVP Freedom Woes With Missing System UUID

Recently, I’ve been wanting to give PernixData FVP Freedom a run in my HomeLab Datacenter to better familiarize myself with the product and see how much of a performance improvement I’d get if any at all.  I’ve heard from so many people how much they love the product so I figured “why not”?

For those who are not familiar with PernixData FVP, it accelerates Storage and Virtual Machines by moving read and write operations to the server tier, instead of the storage tier, using Flash or RAM to ensure the fastest VM performance.  This, in turn, reduces VM latency by a claimed 10x and overall SAN utilization by over 80%.

 To start off, I visited the PernixData website and went ahead to register for the free FVP Freedom product.  A short time later  I received an email and obtained my download and license key information, along with all the documentation needed to get it up and running.  I installed the ESXi host VIBs and opted to deploy the .ova appliance version so that the deployment would be a piece of cake.  Once I got the product up and running, I logged into the Management appliance and attempted to configure my cluster and add resources, but for some reason, none of my hosts’ were showing up.  I kept getting the “No PernixData compatible hosts have been detected in the cluster“, and only (1) of my (5) hosts was detected but it was not part of the cluster that I was configuring yet.

This is where I ran into a snag that took quite a bit of time to research and find a fix.  Luckily, another blogger by the moniker “vWilmo” who’d experienced this same issue and described how to fix it, so I figured I’d write a similar entry for my own reference, and to help others who may frequent my blog.  I will also be sharing his link at the bottom of this page.

Ultimately, the issue stemmed from the fact that Supermicro did not generate any system UUIDs for my boards and FVP needs them to detect the hosts to use as resources.  KB 1006250 references the situation of an ESXi host not having a unique UUID but did not offer a solution other than to contact the manufacturer (which I did via email and am still awaiting a reply).  To confirm this, I ran a script I found online called Get-VMHostUUID to pull the UUID’s from all of my hosts connected to my vCenter server.  Upon review, it only returned a value for my “white box” host, and returned all “zero” values for my remaining (4) Supermicro systems.  I also ran prnxcli via SSH connection to my host which returned an error as well.

 2016-08-05_15-10-50 2016-08-05_19-20-02

As my Supermicro systems run an AMI (American Megatrends) BIOS, there is a BIOS utility that can be used to generate a new UUID for the system which can be found here.  Download this file and extract the contents.  The file we need to use is named AMIDEDOS.exe, so I took this file and placed it on a DOS formatted USB drive that I had created with Rufus back when I needed to flash my BIOS and upgrade my Intel NIC firmware.

 2016-08-05_19-39-09

Insert and boot into the USB, then navigate to the directory that houses the file mentioned above. 

 2016-08-05_20-18-14

Enter the following command:

AMIDEDOS.exe /su auto

 2016-08-05_20-18-25

If successful, this will generate a new system UUID for you and you should then receive an output like this:

 2016-08-05_20-19-25

Reboot your host, SSH into it and run the prnxcli command.  If it runs successfully you should see an output like this:

 2016-08-05_20-30-04

After I completed this process on all of my (4) impacted hosts, I ran the Get-VMHostUUID script again and was happy to see that I now had a valid UUID for each host which match each prnxcli output.

 2016-08-05_20-52-42

Upon logging into the vCenter Web Client, I noticed that there is now a PernixData plugin icon in the vCenter Web Client interface which can be selected to launch the PernixData Management Console or access the FVP dashboard from within the Web Client.

 2016-08-06_11-02-462016-08-06_11-03-07

Lastly, I logged into the PernixData FVP Management console again, and I was now able to create my cluster and assist hosts as resources.  The only caveat is that there is a single-cluster limitation with the FVP Freedom version license, so if have all of your hosts in a single cluster then you are good.  Unfortunately for me, I have (3) clusters so I need to pick which one I want to use FVP with.  I decided to use my Management Cluster since that houses the majority of my VMs at the moment.

 2016-08-05_20-54-12

After letting it work its magic for a few hours, I noticed that the VM latency had reduced drastically to an average ~ +/- 2.0 ms and overall performance was great!  I must say that I am really impressed, satisfied, and glad that I gave this program a shot!

2016-08-06_16-18-52 2016-08-06_16-19-00 2016-08-06_16-19-25

Well,  I hope that you have found this useful, thanks for stopping by!

Special thanks to Geoff Wilmington aka @vWilmo for helping me to solve this as I am lucky I found your post.  Another shout out to Andy Daniel aka @vNephologist from PernixData for his willingness to communicate with and try to assist me with this problem.

Pingbacks:

Intel NIC not detected by ESXi

| 21/03/2016 | Tags: ,

Intel NIC Not Detected by ESXi

 

In this post I am going to cover a random issue I encountered after installing ESXi 6.0 Update 2 on one of my new Home Lab 2016 hosts.  The actual installation of ESXi was extremely easy and painless (I may cover that in another post).  After I had completed the installation, I was attempting to configure my Management network interfaces and suddenly noticed that only 4 network interfaces were being detected!  

2016-03-18_19-07-54

As I then thought to myself, “I wonder what is going on here since I didn’t get any POST issues?”, I noticed that I was getting a message during POST regarding the initialization of the Intel Boot Agent for PXE booting.  The message stated:

“…PXE-E05: The LAN adapter’s NVM configuration is corrupted or has not been initialized.  The Boot Agent cannot continue.” 

 

2016-03-18_18-02-50

Immediately, I began to consult “Mr. Google” and see if there was anything I could find related to this particular problem.  After reading a few threads, many users had mentioned and/or suggested that the NIC’s firmware was corrupted and needed to be “re-flashed”.  I quickly got to work and researched a bit further to understand the process of flashing the NIC firmware.  I downloaded the latest version of PREBOOT (at the time of this writing, it was version 20.7)which contains the “BootUtility” needed to perform the flash.

Next, I prepared myself a DOS-bootable USB using Rufus.  I then extracted the PREBOOT.exe file using 7zip and placed the contents on the newly created USB.  This would allow me to either boot into the USB and access DOS or boot into the UEFI: Built-in Shell on Supermicro motherboards, and access the necessary files.  Once I had my drive ready, I went ahead and plugged it into my server and initiated a reboot.  During POST, I invoked the boot menu so and chose the option to boot into the Built-in Shell.

 2016-03-21_21-19-54

Once in the shell, I determined that my USB was mounted at fs4:

 2016-03-21_21-21-36

I navigated through the directories so I can see the contents of each folder until I saw the BootIMG.FLB file which is the new flash image I want to apply.  I then navigated to the location of the BootUtility.  Since I am using the built-in shell, I needed to ensure that I used the BootUtil for x64-bit EFI so I navigated to the following location:

...\PREBOOT\APPS\BootUtil\EFIx64

 2016-03-21_21-24-17

Running the BOOTUTIL64E.EFI file will simply list your network interfaces and I could then see the current firmware version for all of my interfaces, although for some reason the ones in question are displaying “Not Present”.  Adding a “-?” suffix will bring up the help and list all the parameters to execute the commands properly.  I found a great reference article here which made it easier for me to see what parameters I needed in my command.  

2016-03-21_21-25-48

To begin, I entered the following command since I wanted to enable flash on all of my NICs.  

BOOTUTIL64E.EFI -ALL -FLASHENABLE

2016-03-21_21-29-17

Or, if you wanted to do each one individually, you could specify the NIC  number (referenced as X below) manually.

BOOTUTIL64E.EFI -NIC=X -FLASHENABLE

A reboot is required after successful completion of this command before proceeding, so I went ahead and rebooted my system and then booted back into my USB via the built-in shell.  

2016-03-21_21-29-51

Afterwards, simply running the utility again showed that NIC ports 1-4 were all PXE ready.  

2016-03-21_21-34-35

Now it was time to run the following flash command.  Note: Specifying the file parameter is optional.  Without it, it will assume that the BOOTIMG.FLB file is in the same location you are executing the command from.  Since I left the file in its originating location, I had to specify it manually.

BOOTUTIL64E.EFI -UP=PXE -ALL -FILE=\SuperMicro\PREBOOT\APPS\BootUtil\BOOTIMG.FLB

You will then be prompted to create a restore image, in the event something goes awry but I chose not to create a restore image.

 2016-03-21_21-38-31

Upon successful completion, I can now see that my firmware version has been upgraded from version 1.3.98 and is now running version 1.5.78

 2016-03-21_21-40-55

Now that my firmware has been upgraded, I rebooted the host and accessed the Management Network settings screen, and to my delight, ESXi was now detecting all of my network interfaces!  Woohoo!!  

2016-03-21_22-17-38

I am still trying to figure out why flash is not present on the NICs that previously were not detected, and my assumption is that it’s due to these being relatively cheap $80 network cards instead of full priced (~$300) network cards.  In any case, they still work and I am quite happy with them.  I hope you have found this information useful.  Thanks for reading!