Run Automated Tests When Your D365 CE Org Gets Updated

With online D365 CE instances getting updates pushed regularly it makes it almost impossible to ensure that you’ve tested against the latest bits. That’s assuming you even know about each update since the majority go in silently in the background. Realizing that organizations need to test before updates roll out to production or at least be aware of issues caused by updates before users start reporting them, Microsoft has taken the first steps to allow you to get ahead of potential issues.
The Microsoft.Xrm.Tooling.Testing PowerShell library (in preview at time of writing) has been released which provides hooks into a few key backend operations that you hadn’t had access to previously. When one of these events occurs you have the opportunity to run automated tests.

Current backend triggers:

• OrganizationDatabaseUpdate
• OrganizationDatabaseUpgrade
• OrganizationBitsUpdate
• OrganizationPackageInstall
• OrganizationPackageUninstall

How does it work?

To start, a developer will need to create a .NET unit test project that will contain any tests that need to be run. The tests really can’t be ‘unit tests’ because by definition unit tests are meant to be totally decoupled from everything. So running tests that don’t act on a live environment isn’t going to tell you anything you didn’t already know. Really what you are after would be better described as an ‘integration tests'. Tests that connect to the live environment that just registered the change are going to uncover things that might not be working. Depending on the complexity of your business and customizations it might be a little tricky to create meaningful tests. Once the tests are written you can use the provided PowerShell commands to upload the tests to the Microsoft environment and register when they should be run. Then when a trigger you’ve hooked into is tripped, your tests run.

Test suite setup

I’ll point out right now that the documentation doesn’t appear to be complete yet so there’s probably a better way to do some of these things but without knowing I just improvised to get things working.

For test purposes I created a unit test project that is only going to perform one operation which is to create a contact record. My ‘test’ consists of a CrmServiceClient connection that creates the record and a basic assert statement that passes the test if there aren’t any exceptions. I’ve also got an app.config file that contains the connection string populated with valid credentials. Also note that a better solution would be to use Azure Key Vault and have it read any credentials from there. It’s a bit more secure and allows for updating the values without having to re-deploy things.

Once you’ve got the code working, zip up the contents of the debug/release folder. I mentioned before that the tests get uploaded to Microsoft’s environment. Best I can tell is that the zip file containing the tests needs to be available via a URL on the public internet as opposed to uploading a local file (again maybe I’m doing it wrong). This appears to only be required initially as the documentation states, “Testing infrastructure will download and store the test suite ZIP file for future runs. Changes to this URL will not affect the test run. If you need to update the test suite, please run this cmdlet again pointing to the new test suite's URL”. I dropped my zip file on a blob store and generated a short-lived SAS token I could add on to the URL.

If you haven’t already done so, install Microsoft.Xrm.Tooling.Testing in PowerShell

install-module Microsoft.Xrm.Tooling.Testing

Then register a test suite to contain the test triggers. Based on how things get configured, a test suite will center around the tests being executed. If you’re going to want different sets of test to run for different triggers it looks like you’ll need separate suites.

• $apiUrl = Online Management API Service URL
• $orgId = D365 CE organization ID found on the Developer Resources page
• $orgUniqueName = D365 CE organization Unique Name found on the Developer Resources page
• $testSuiteUrl = URL of the unit test project zip file
• $completionCallbackUrl = URL that will have basic test data sent to it when complete via webhook

$user = "admin@test.onmicrosoft.com"
$password = "password"
$cred = New-Object System.Management.Automation.PSCredential `
     ($user, (ConvertTo-SecureString $password –ASPlainText –Force))
$apiUrl = "https://admin.services.crm.dynamics.com"
$orgId = '38600bb5-ed48-47e4-8844-e05c5516b446'
$orgUniqueName = 'org1a1aa611'
# Test Suite
$testSuiteName = 'IntegrationTests'
$testSuiteUrl = 'https://mystorage.blob.core.windows.net/test/AutoTest1.zip?sv=SASKEY'
$testUserCredential = $cred
$completionCallbackUrl = 'https://prod-03.westus.logic.azure.com:443/workflows/...'
# Test trigger
$testTriggerName = 'IntegrationTests-OrganizationPackageInstall'
$testTriggerType = 'OrganizationPackageInstall'

Register-TestSuite -ApiUrl $apiUrl -OrganizationId $orgId -OrganizationUniqueName $orgUniqueName `
     -TestSuiteName $testSuiteName -TestSuiteUrl $testSuiteUrl -TestUserCredential $testUserCredential `
     -ApiCredential $cred -CompletionCallbackUrl $completionCallbackUrl

You should see a result of true be returned if successful.

Test trigger setup

Setup one or more of the backend triggers in the test suite that will execute your tests. Here the trigger is going to be a 'OrganizationPackageInstall' which I’ve assumed to mean when one of the system level solutions or a solution found in the D365 CE Administration Center is installed (and hopefully updated as well).

Register-TestTrigger -ApiUrl $apiUrl -OrganizationId $orgId -TestSuiteName $testSuiteName `
     -TestTriggerName $testTriggerName -TestTriggerType $testTriggerType -ApiCredential $cred

Again you should see a result of true returned if successful.

To confirm everything is set up correctly, you can run this command to see what is registered for a specific organization.

Get-TestSuites -ApiUrl $apiUrl -OrganizationId $orgId -ApiCredential $cred

It will show the suites and triggers that are currently configured.

Testing this out

You can manually trigger all the tests in a suite to ensure that everything is working.

Start-TestRun -ApiUrl $apiUrl -OrganizationId $orgId -TestSuiteName $testSuiteName `
     -ApiCredential $cred

This will return the ID of the test run.

You can check the status of a run at any point.

$testRunId = '2c5d1e62-ff35-43c3-9681-83de5fbb2235'
Get-TestRun -ApiUrl $apiUrl -OrganizationId $orgId -TestRunId $testRunId `
     -TestSuiteName $testSuiteName -ApiCredential $cred

This will output information about the run including status (running, succeeded, etc.). If complete it returns an event log but its purpose must be something other than reporting on the actual test results as it contains no references to what was executed.

Installing a solution from the Administration Center will trigger this particular event as well.

More about test results

From what I can tell so far, there isn’t a way to see if individual tests have passed or failed. That said, it might be best to add your own way of logging the test results so you can see exactly what passed and what didn’t. When the run completes it triggers a webhook with some basic information about the run. Lots of options open up to get notified so you can go and look at the results. I used a Flow which runs based on an HTTP request being received and I used the URL it generated as the CompletionCallbackUrl when registering the suite. Then it just sends an email notification with all the data.

EasyRepro tests

At the moment it doesn’t appear that whatever is running the tests is set up to handle EasyRepro / UI tests. All my attempts failed which again might mean I’m just doing it wrong. For v1 of this solution I can see where Microsoft wouldn’t have yet built out or allowed for the various supported browsers being installed on the infrastructure to allow for this sort of testing.

If you’re interested in running these types of tests you’re going to have to rely on your own test platform for the execution. In my case I’ve got EasyRepro tests set up to run in Azure DevOps (ADO). In order to get them to run based on one of these provided triggers you can either create a dummy unit test which uses the ADO APIs to execute a build or release directly or just pass the dummy test and have the webhook call a Flow where you use the built in ADO connector to kick something off.

Should the built in ADO connector not work for you, a HTTP action can also be used. Here's an example of queuing a build using the REST API.

POST  https://{your instance}.visualstudio.com/{project name}/_apis/build/builds?api-version=5.0

Headers
Content-Type: application/json

Authorization: Basic
Username: {anything}
Password: {Azure DevOps Personal Access Token}

Body
{
        "definition": {
            "id": {build definition id}
        }
}

Once that kicks off and runs its up to you to handle any further notifications and reporting on test results.