Agile Test Tools

How often can you update the tools?

In our scenario today, the disease is Versioning and the frozen solution the cure. To clarify, rooted in the software for medical devices, the so-called frozen solution schema allows you to work with a defined version of critical components. And at the same time, having the benefits of newer versions of non-critical components. This is a life saver in an agile world.

Tool versioning is a problem in testing today

Modern test tools have dependencies. For instance, Selenium, as example, is a framework (Selenium), requires a runtime (junit, …) and talks to a browser via the WebDriver protocol. Versioned components are therefore all three. Which means, all three of those components are dependent. In other words, all three components can break your test case.

As for Selenium, you need to answer the following questions (for every browser that you want to test):

• Firstly, which version of the WebDriver is required for this browser version?
• Secondly, which version of Selenium is required for this WebDriver?
• Then, which version of Java (or another runtime) is required to run this version of Selenium?
• And finally, which version of your test runner is supported on this Java platform?

It gets even worse with frameworks like Appium. If there are different WebDriver implementation for the same target (like Android or iOS).

Bear in mind, on top of Selenium, you use your own tool. This means, you have the following challenging questions that rise. Does your own tool and CI Pipeline support that versioning? And what if you update your own tool? Do you need to update all the possible dependencies? Furthermore, can your CI pipeline still execute those tests?

The Protocol that Web Driver has been running for a long time is called JSON Wire Protocol. However, starting with Selenium 4 this protocol will no longer be supported. Although, there will be some scenarios where Selenium still provides mechanisms for people to use the former protocol. This may be done through the Java Bindings and Selenium Server. This legacy support provides backward compatibility to organizations that cannot run their cases using the new protocol. These issues could mainly happen either in old infrastructure or when using old driver versions.

The question is: How can you make sure that you can test on all platforms without the hassle of updating all the time?

Old test tool version

Essential you are stuck in a specific version of a tool chain. Which is ok. If you have a short running project, no one will ever notice this problem. In bigger projects that take longer than a year to deliver this problem will manifest itself.

Agile development makes it even harder as you will usually see a push to adapt to the newest framework versions based on an automatic packaging solution. Once again, testing cannot keep pace with development.

The Frozen Solution approach helps you: New features, reliable execution

The idea of the frozen solution is simple: Components. You split all required functionality in independent components. All these components follow a common version schema and provide a full interface (interface not as in UI but as in a programming language) abstraction of all components against each other. You have two-sided interfaces with versioning interfaces in between. That is not easy.

Define a version schema

Our version schema for example is the following: A.B.C.D:

• A: Major version number
• B: Minor version number
• C: Patch version number
• D: Bug fix version number

Let us begin, with a change in A or B, which means that core functionality can break, and that interface abstraction is required. Furthermore, a change in C means that new features are available, but the previous interface is still working. On the other hand, a change in D means that bugs were fixed, and the interface is equal.

Apply Version Pinning

For a frozen solution all you need to do is: Version pinning. That means you tell the system which differences in the version you do allow. At TestResults.io we provide a default pinning of 0.0.0.1. We only allow bug fixes to be automatically be applied.

Our proposal for regulated environments is a pinning of 0.0.0.0 which blocks all updates.

Frozen Solutions in 1-minute for developers If you have a central build server with .net 4.6 it does not matter in which Visual Studio version your code is written. You have the latest usability features local and the stability of the build system centrally. If you update locally you do not need to retest your build server.

Conclude with classification

Now, that you understood component splitting, dual-side interface abstraction and version pinning it is important to understand that this system makes only sense if you classify your components in critical and non-critical. Start by applying the versioning pining for critical components. Carry on by not applying versioning pining for non-critical components.

Critical components

At TestResults.io we classified the following components as critical:

• Test Engine: This is the part that is executing your automated test case step by step and oversees the flow control of the automation.
• Base & Application Models: These models provide the generic and specific abstractions that allow you to talk to an application from your test case.

And non-critical components

Non-critical, and therefore always updateable are:

• Execution Host: This is where the test engine version is. It translates the request from a client to the test engine. Furthermore, it translates the results returned by the test engine into the client’s format.
• Designer: This refers to the component you are mostly working with, the environment that you use to design the test case

Unblock your testing but stay safe!

How does it all come together? Components that are non-critical need to be able to support all previous and future versions of the critical components:

Updating the designer

The designer supports any previous and future version of the test engine, base model, and application models. Therefore, you can always have the latest design experience with the latest AI based features while still relying on a year-old test engine. The designer will automatically create a test case in the format defined by the test engine that you have chosen on startup.

You conclude your test case writing and in the meantime the designer will register the version number of the critical components and includes it in your test package.

Updating the host

The same setup you have defined in the designer, will execute your test case with a host. Although, it might be a test engine version that is already 3 years old, running on the brand-new execution host. Frozen Solutions are the only solution for regulated environments, today. Nevertheless, they save everyone a lot of time on updates as the system just works.