My test automation is not being used because it is too slow. Today we will speed it up.
Do you recognize this? Your test automation is running, everyone is excited. But a few days later nobody is using it anymore and increasing numbers of the tests fail, in the case that these tests even actually take place, and nobody seems to care?
Why is that? You should only continue reading this blog entry if you have already implemented a working test automation, this is because today`s issue doesn`t concern technical problems such as “how do I find an element?” or “why has my test case been stopped?” or even “why is the correct software version not being used?”, but about what occurs afterwards.
The first time you start the automation
After putting days of effort into the automation, the first time you are able to carry out a test you are excited and you assume it will be flawless, and it is. All of the test cases have been executed as planned and have produced the expected results.
The question at this point is, what now? Due to your efficiency all of the test cases have been automated. Normally, it would take a test team around 14 days to execute these test cases manually, but thanks to your automation, it takes only 6½ hours.
As however it is necessary to start the tests on your computer each time, even 6½ hours cannot be considered as particularly convenient.
This should be the last time
You make the next logical step, the test automation becomes embedded in the DevOps pipeline. Continuous Integration (CI) is our keyword here.
The development team has up to this point only had unit tests, they are almost always green, but your testers as well as the end users still find errors in the software. A classic scenario. What`s new is the unit tests will be executed, followed by your automated end-to-end tests, integration tests, etc.
Everyone is excited because the errors have been found before the release. Quality has improved.
Week four however suddenly arrives. Once again your tests have disappeared from the CI, or they are only being run once a week, and when they are being run, by this point no one is interested in the results, too many tests have errors or no longer run.
What happened? Your results are available too late! Even for a nightly build, at 6½ hours the runtime is too long. In order to be accepted we need to bring the runtime of all the tests to a total of below 10 minutes.
A difficult task, but not completely impossible. All we need to do is run all the tests in parallel.
To start, there will be a few tests that are automated and that have a longer runtime. Respectively. we outsource those and run them only once per night, all the others can be run for each developer check-in.
To run all tests simultaneously, we need disproportionately more test systems, which are currently not available to us. We have 2 options: Extremely overdraw our project budget to purchase hardware that is no longer needed after project completion, or we switch our tests to dynamic execution.
Dynamic execution means that for the time of execution, we automatically start all the required systems with a cloud provider, execute the tests in a distributed manner, and then shut the systems down again.
Cloud provider
As cloud providers fundamentally assume that the systems which they activate should perform a permanent service, we fall through the price cracks with our model of dynamic execution and receive nearly unlimited resources for an extremely small amount of money.
Let’s take Microsoft’s Azure Cloud, for example. Here we receive a so-called B4MS instance for $0.166/hour. This means that we have four virtual cores. Even more, the advantage of the MS variant is that we get 150% of the paid computing power for free for the first 30 minutes, i.e. we pay for the performance of 4 cores but receive 6. Therefore, if we assume that our tests run for about 10 minutes, then we pay $0.027 per test, so for 100 tests we pay $2.76 per test run. Instead of 6½h, however we have thus completed the entire test run in 10 minutes!
This is an investment of resources that is worth carrying out for every project, there is no requirement to have our own test infrastructure. i.e. this eliminates all of the internal IT costs. We pay about $55/month if we run all the tests every day.
This means that for $55/month you can run 100 tests a day simultaneously, with results after 10 minutes. Compare this to your current runtime and your current costs (including internal IT costs). Working past the actual cloud pricing model makes it possible.
The same is true of Amazon’s AWS or Google’s cloud.
Central control
This approach works only with some sort of repository. The repository is for your test cases. At any time, this is a central repository that allows the latest automations to be referred to during a test run.
It would be optimal if this repository also provided a direct overview of the available tests, the tests already automated, the successful tests, etc.
With this in mind, we would then have a central management dashboard for quality that reflects the current state of software quality at any point in time, and this not based on static analyses or unit tests, but on complete end-to-end tests which you have developed yourself.
But decentralized automation
Although the central control and administration is an important cornerstone. It is out of question, that the repository must still guarantee that all testers can work with it simultaneously. Preferably without additional tools and without directly requiring the application.
In the ideal case, each tester can create new test cases graphically via a web page based on the application. Directly stored in the repository are the test cases. And executed the next time the developer checks in. Every time.
Modern testing without damaging your budget
In relation to test automation, it’s important to understand that we don’t automate for the sake of automating, we automate to increase quality. However, if no one is running the automation or paying attention to the results. Moreover, we have not done our job and quality levels will not be increased by our efforts.
It is therefore important to pay attention to the following during test automation :
Budget : What does the licensing model look like – is it even possible that I can start 100 tests at the same time.
Accessibility: How are results returned, how are test cases managed, how open is the system?
Acceptance : How user-friendly is my test environment, also for the developers?
I’ve given you some ideas to help you take your automation to the next level. I would like to share my ideas with you and learn from your experience.
Contact me directly today by writing me an email
Leave a Reply