Exchange 2013 Recovery Procedures

Proper documentation is vital for recovery. Experiences taught me how important it is to test and document it. Also I’ve come across Exchange or MS Windows admins who have taken a snapshot of Exchange vms as a backup despite all the writing on the wall it is a disaster. Microsoft through all the versions of Exchange have said that they do not support Exchange snapshot since it is not application aware. Microsoft is very clear on this topic in Exchange 2016 requirements for hardware virtualization, “Some hypervisors include features for taking snapshots of virtual machines. Virtual machine snapshots capture the state of a virtual machine while it’s running. This feature enables you to take multiple snapshots of a virtual machine and then revert the virtual machine to any of the previous states by applying a snapshot to the virtual machine. However, virtual machine snapshots aren’t application aware, and using them can have unintended and unexpected consequences for a server application that maintains state data, such as Exchange. As a result, making virtual machine snapshots of an Exchange guest virtual machine isn’t supported.”

Recovery procedures are carried out on the following environment:

VMware VM
Exchange 2013 CU10
HPE Data Protector ver A 10.0.0. for backup
RTO is between 3-5 hours depending on the MB data restore time.

Recovery of mail server is the key to business continuity. Most of the exchange configuration settings are stored in Active Directory. As long as Active Directory is healthy you can recover an Exchange Server using its former configuration.
Some items are to be restored from backups such as databases, certificate, and custom settings made outside of the Exchange management tools.
There are certain requirements:

  1. The server on which recovery is performed must be running the same operating system as the lost server. You can’t recover a server that was running Exchange 2013 and Windows Server 2008 R2 on server running Windows Server 2012 R2, or vice versa.
  2. The same disk drive letters on the failed server for mounted databases must exist on the server on which you’re running recover.
  3. The server on which recovery is being performed should have the same performance characteristics and hardware configuration as the lost server.
    VM Name : contoso-exch
    IP      : 10.0.0.25
  4. Use the same version of Exchange server 2013 setup. In our case we are running Exchange Server 2013 CU10.
  5. If Exchange install location/folder. If it was installed on different location, it must be specified during the installed.
    To gather the above information, do the following:

    1. To find the Operating System version:
      Go to ADUC
    2. To find the install directory for Exchange, use ADSI edit.
      1. Open ADSIEDIT.MSC or LDP.EXE.
      2. Navigate to the following location:
        CN=ExServerName,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=ExOrg Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=DomainName,CN=Com
      3. Right-click the Exchange server object, and then click Properties.
      4. Locate the msExchInstallPath This attribute stores the current installation path.
      5. To find the cumulative Update (or build) that failed sever was running we can also use ADSI edit.

        Exchange 2013 is version 15.0.Remove first 1 digit (3) from the build number we have the build number of 1130.7. Lookup this build number Exchange server updates: build numbers and release dates and we see we are on Cumulative Update 10.

Recovering a Lost Exchange Server:

  1. Reset the computer account for the lost server.
    Go to ADUC and locate the computer account for the failed Exchange Server. Right-click on the account and select Reset Account from the context menu. Click yes to confirm.
  2. Create a VM with same hardware configuration – memory, network, hard drives.
  3. Install the proper operating system and name the new server with the same name as the lost server. Recovery won’t succeed if the server on which recovery is being performed doesn’t have the same name as the lost server.
  4. Give VM a static IP same as before.
  5. Join the server to the same domain as the lost server.
  6. Install the necessary prerequisites and operating system components.
    1. Use PowerShell to install the required server roles and features in Windows Server 2012 R2
      Install-WindowsFeature RSAT-ADDS
      Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation, RSAT-Clustering-CmdInterface

      A restart is required after the roles and features have finished installing.

      Next, install the following software in this order.

    2. Unified Communications Managed API 4.0 Runtime
    3. Microsoft Office 2010 Filter Pack 64 bit
    4. Microsoft Office 2010 Filter Pack CU10 64 bit
    5. Verify necessary version of the .NET Framework has been installed.
    6. Reboot the server.
  7. Log on to the server being recovered and open a command prompt.The recovery process cannot be completed via the GUI setup.
  8. Navigate to the Exchange 2013 CU10 installation files, and run the following command.

    Setup /m:RecoverServer /IAcceptExchangeServerLicenseTerms

    When setup completes, reboot the server.

  9. After Setup has completed, but before the recovered server is put into production, reconfigure any custom settings that were previously present on the server, and then restart the server.After Setup has completed, but before the recovered server is put into production, reconfigure any custom settings that were previously present on the server, and then restart the server.
    Configure virtual directories servicesConfigure External and Internal URLS in the Exchange Servers.

    1. ecp
    2. ews
    3. active-sync
    4. OAB
    5. owa
    6. Outlook Anywhere
      Make sure Outlook Anywhere is configure properly.

      **Change from Negotiate to NTLM**
    7. Autodiscover
      AutoDiscover feature in Exchange 2013 let’s client application such as Office Outlook 2007, 2010 and 2013 to connect to Exchange server automatically. AutoDiscover feature automatically discovers the mailbox settings for user profile in Office Outlook application. AutoDiscover also works for supported mobile applications. In Exchange 2013, you can configure SCP for AutoDiscover service via Exchange Management Shell. The command below will update SCP (Service Connection Point) object. SCP is active directory object and is used by internal domain-joined clients to retrieve autodiscover URL.

       [PS] c:\windows\system32>Set-ClientAccessServer -Identity MBG-MAIL -AutoDiscoverServiceInternalUri https://autodiscover.mustbegeek.com/Autodiscover/Autodiscover.xml

       

      To view the changes type following command in Exchange Management Shell.

      [PS] c:\windows\system32>Get-ClientAccessServer | FL AutoDiscoverServiceInternalUri

       

    8. Configure Certificate
      Request a certificate, install and assign the certificate to services.Create a Cert folder on C:\ and share.

       

      1. Open the EAC by browsing to the URL of your Client Access server. For example, https://Ex2013CAS/ECP.
      2. Open the EAC by browsing to the URL of your Client Access server. For example, https://Ex2013/ECP.
      3. Enter your user name and password in Domain\user name and Password, and then click Sign in.
      4. Go to Servers > Certificates. On the Certificates page, make sure your Client Access server is selected in the Select server field, and then click New .
      5. In the New Exchange certificate wizard, select Create a request for a certificate from a certification authority and then click Next.
      6. Specify a name for this certificate and then click Next.
      7. If you want to request a wildcard certificate, select Request a wild-card certificate and then specify the root domain of all subdomains in the Root domain field. If you don’t want to request a wildcard certificate and instead want to specify each domain you want to add to the certificate, leave this page blank. Click Next.
      8. Click Browse and specify an Exchange server to store the certificate on. The server you select should be the Internet-facing Client Access server. Click Next.
      9. For each service in the list shown, verify that the external or internal server names that users will use to connect to the Exchange server are correct. For example:
        1. If you configured your internal and external URLs to be the same, Outlook Web App (when accessed from the Internet) and Outlook Web App (when accessed from the Intranet) should show contoso.com. OAB (when accessed from the Internet) and OAB (when accessed from the Intranet) should show mail.contoso.com.
        2. If you configured the internal URLs to be contoso.com, Outlook Web App (when accessed from the Internet) should show owa.contoso.com and Outlook Web App (when accessed from the Intranet) should show internal.contoso.com.
        3. These domains will be used to create the SSL certificate request. Click Next.Add any additional domains you want included on the SSL certificate.Click Next.
      10. Add any additional domains you want included on the SSL certificate.
      11.  Select the domain that you want to be the common name for the certificate and click Set as common name. For example, contoso.com. Click Next.
      12. Provide information about your organization. This information will be included with the SSL certificate. Click Next.
      13. Specify the network location where you want this certificate request to be saved. Click Finish.
      14. After you’ve saved the certificate request, submit the request to your certificate authority (CA). This can be an internal CA or a third-party CA, depending on your organization. Clients that connect to the Client Access server must trust the CA that you use. After you receive the certificate from the CA, complete the following steps:
      1. On the Server > Certificates page in the EAC, select the certificate request you created in the previous steps.
      2. In the certificate request details pane, click Complete under Status.
      3. On the Complete pending request page, specify the path to the SSL certificate file and then click OK.
      4. Select the new certificate you just added, and then click Edit .
      5. On the certificate page, click Services.
      6. Select the services you want to assign to this certificate. At minimum, you should select IIS but you can also select IMAP, POP, and UM call router if you use these services. If you want to use secure transport, you can also select SMTP to make this certificate available to Exchange 2013 transport. Click Save.
      7. If you receive the warning Overwrite the existing default SMTP certificate?, click Yes.

      The typical services to assign to an SSL certificate are IIS and SMTP

    2. Check the settings for send and receive connectors.
  10. Log back into Exchange Admin Center and confirm all settings are restored from Active Directory.

Restoring all Mailbox Databases from Backup

Following behavior will be observed until MB databases are being restored.

  1. Outlook will not be able to connect to Exchange Server.
  2. As soon as the server restarts from the recovery process, we will notice that all mailbox databases are dismounted, and at this point we should have all drive letters but without any mailbox database files on them. We can always check the properties of the mailbox databases to make sure that the path defined on the Mailbox Database object exists on the server.
  3. Install HP Data Protector Client.
  4. Restore Mailbox database using Data Protector Manager GUI.
    1. In the Scoping Pane, expand MS Exchange 2010+ Server , expand the DAG virtual system or standalone Microsoft Exchange Server system and click MS Exchange 2010+ Server
    2. In the Source page, Data Protector displays all Microsoft Exchange Server databases backed up from the selected DAG or standalone environment.Select the Microsoft Exchange Server databases to restore.When you select a database, the Properties for Database dialog box is displayed automatically.

      Restore to a temporary location with Perform database recovery option set: Data Protector restores the database from the backup created in the last backup session (and the corresponding restore chain – Full, incremental, differential backups).
    3. Once back to the Restore DP screen. Click Restore.
    4. Select Next on Start Restore Session dialog box.
    5. On the Start Restore Session choose Finish

      Monitor currently running session. It should display restore successful when it is done.

      After the database restore is completed. Go to EAC and mount the Mailbox Database. Outlook will be able to connect and access the information. Outlook might need to restart.
Tags: , , , ,