Microsoft Dynamics 365 Regression Suite Automation Tool (RSAT)

Welcome Guest Blogger Latha Kalaga
Hello World!...for the first time ever, I have a guest blogger. Please help me welcome Latha Kalaga. I have been working with Latha for nearly 13 years and she brings a myriad of expertise on all things technical related to Dynamics 365. Because Microsoft has introduced her in more than one session as the "number one user of RSAT", I thought, hey why not get her to write this blog about RSAT. I can honestly say, Latha taught me everything I know about RSAT, and I hope you enjoy this post.

Introduction to the RSAT
Microsoft introduced the concept of continuous update in July of 2018 to ensure that all Dynamics 365 for finance and operations will be on One Version of the software. The customer will have 7 calendar days to validate an update that has been applied to a sandbox environment prior to rolling out the update to production. To support this activity, Microsoft also rolled out Regression suite automation tool (RSAT) to help the customers validate the update.

It is very important to test every update with your mission critical business processes in mind. Typical manual testing of major critical processes could take 40 to 60 hrs or more and depending on the number of test cases, a company may have to allocate one or more resources to this testing effort prior to every update. The cost of this manual testing prior to taking the update is too high and prone to errors. The regression suite automation tool significantly reduces time and cost of user acceptance testing by allowing the acceptance testing to be mostly automated.

The regression suite automation tool allows functional power users to record core business processes using the Dynamics for Finance and Operations task recorder and convert these task recorders into test suites without having to write a single line of code.

Technical Details
Regression testing is an art as much as it is a science. Laying out what you want to test, the combinations of a process that you want to test needs to happen before any testing or testing suite finalization comes into play. You will want to start with a release validation plan layout. This is a paper and pencil exercise in which the functional testing subject matter expert lay outs the process and the intended validations of the expected values. It is also important to think about data dependent testing suites versus a data independent testing suite. Once the release validation plan is in place, the testing suites can be assembled from the processes that have been laid out in the business process modeler.

The business process modeler that resides in the Life Cycle Services (LCS) is one of the, if not the most important component of a regression testing life cycle. It is important to lay out the business processes that are needed to successfully run a business and then record these business processes using the task recorder provided. For a task recording to become a regression testable recording there are two components that are required. 
  • Validations
  • Reference variables
The first component is adding validations and the second, optional but a valuable component is the reference variable. Validations in a task recorder allows RSAT to compare the actual value to the expected value. The reference variable allows the tasks in a test suite to be chained.

The business process modeler is synced over to Azure DevOps in two passes. The first pass syncs the business process modeler nodes as epics and the second sync, synchronizes the task recordings as test cases over to Azure DevOps. Using the release validation plan the user can create a test plan in Azure DevOps and add the test cases to the testing suite. A Microsoft Visual Studio Enterprise level license is required to create a test plan in Azure DevOps or per user per month license can be purchased to allow the user to create these test plans. 

Once the test suite is created the test suite can be loaded into the RSAT tool that is typically installed on an Azure Server or a laptop of a quality assurance tester. The RSAT tool runs the business processes through the user interface of Dynamics 365 for Finance and Operations as it runs through the steps of the task recorder while validating the values called for in the task recorder. 

The results of the run are stored in Azure DevOps allowing the user to evaluate the results of each run and compare different runs of the same test suite. Errors are logged in Azure DevOps in easily decipherable comments allowing the functional experts and the user acceptance testing teams to figure out the issues if any caused by the uptake of an update be it a customization or a Microsoft update. This triage will enable businesses to identify issues prior to taking the update in production. The values used by the RSAT tool are decoupled from the original recording and can be changed in the excel spread sheet that is generated by the RSAT tool using the task recording. The power of using reference variables to chain tests allows for running of test suites that are complete end to end processes without having any manual intervention.

The following is a screen capture of the type of test suites that can be created. This is a data independent suite for order to cash that can be run against any dynamics for finance and operations environment. The inputs and expected values are the only pieces that needs to be checked in the excel spread sheet that is attached to the test case prior to running this against any environment.

Below is a screen capture of the suite from Azure DevOps loaded into the RSAT tool:

RSM value
At RSM we pride ourselves on being the First Choice Advisor to our customers. We participated in the beta release of the Regression Suite Automation Tool and are heavily engaged in helping improve the tool and the components that go into making RSAT truly a good testing tool. We are working with several customers to put together their release validation plan and help them build out their suites using the tasks recorded and attached to the business process modeler. We also train our customers in using RSAT if they choose to build out their own suites. We advocate best practices for all the components starting from what it takes to create a good business process modeler, a good task recorder that will then lend itself to creating a powerful RSAT suite. We are participating in the release validation program that Microsoft announced by enrolling our RSAT suites that we have built out for our customizations sold through the Market place. We are one of the leading RSAT users if not the leading user with a staff that is trained in everything RSAT starting from installation, to test creation to execution.

Here are some statistics from our efforts at some of our customers.
  • Recorded 140 tests (4 resources, 1 month), 50 processes covered enabled the client to cut down manual testing by 52% (Once we record the mobile processes this percentage will reduce further)
  • Recorded 208 tests (4 resources, 1 month), this suite runs in about 2.5 hours. Manual testing takes about 40 hours today for this same set of tests

Conclusion
RSAT has a lot to offer and it is very hard to capture all the information in a blog post. Please feel free to reach out to RSM if you would like for us to demo the capabilities of RSAT or in general the planning and execution of a test suite.

Quote

“Why do we never have time to do it right, but always have time to do it over?” - Anonymous

References

Again, a HUGE THANK YOU to Latha Kalaga for writing this post. You can connect with her on LinkedIn here.

If you would like to see me live presenting on the RSAT be sure to sign up for the Dynamics 365 User Group Conference here. My session (along with many others) will be recorded and available to attendees who have registered to receive the recordings. Don't miss out! This is a special add-on to your conference pass for just $100 that adds a lot of value that keeps on giving even long after the conference is over.

Comments