Artificial intelligence is everywhere, it solves problems but at the same time jobs are put at stake due to the advanced technology manpower is no longer needed. While the last part may become true in the distant future it is unable to solve the present testing problems.
On the ground of this statement, we will look at what artificial intelligence means to you in your daily business as a tester, test automation engineer or as a developer who understand the importance of testing.
Artificial intelligence (AI) is just a name for algorithms
Without all the bells and whistles, artificial intelligence is just a name for a group of algorithms in computer science. Those algorithms are not new, and they did not change drastically. So let’s look at this one example, it is the classical k-means algorithm which is used as the k-nearest neighbor (kNN) classifier to identify data clusters or assign elements to existing clusters based on similarity.
Let’s make it more visual for you: so take a look at the data points below. Imagine that you would be asked to generate three clusters, how would they look like? As it is simple for you, it is difficult for a computer.
visual recognition vs. approximations
If the algorithms are not new, what changed? Firstly, extremely magnified computer power and secondly, storage space is the answer. In addition, think about a business rules engine with an unlimited amount of business rules processed in no time. Not to mention, it can look intelligent but eventually it only follows said rules.
Which cannot compare to your brain
The human brain is a massive storage space with enormous processing power. While not faster as today’s supercomputer in general, it is far more specialized to daily tasks like image and speech recognition. Correspondingly, its specialization is the reason that the human brain is more versatile and faster in all its tasks than any supercomputer currently available. On the other hand, for non-specialized tasks, e.g. mathematical calculations, the brain is already far behind what even the smallest computer is capable of today.
AI solves many problems
Hence artificial intelligence finds solutions faster than traditional approaches, but they might not be the better solutions. Generally, AI is used to solve optimization problems. For instance, take for example the question: What is the best position for my car to be in the current situation on this highway? Certainly, the answer is only useful if you have it within a reasonable amount of time (for example 100 ms). Therefore, you are not interested in the best answer, you are interested in the best answer that can be provided within those 100ms.
Lastly that brings us to the problems that AI can solve better than traditional approaches: Path finding, Pattern Matching, Recognition, … as you can see, all of them are traditional optimization problems, solved by approximations.
It is neither meant for testing nor for adaption
As AI is about approximations, you should not be hyped about it in testing. Specifically, not in the part that is sometimes referred to as checking, e.g. if I press button “Close” the dialog should disappear. At a minimum, you want your software to behave as specified. Consequently, we check the functionality against the specification.
Therefore, think about the following scenario: You want to enter a username and password and press the login button on a web page. Pretty straight forward example. In addition, for simplicity reasons we use a test engine that is based on object identification through attributes. This gives us the following (simplified view) on our web page:
| Attribute | USERNAME | PASSWORD | TOKEN | LOGIN |
| Type | Textbox | Textbox | Textbox | Button |
| Text Mask | (none) | * | * | (none) |
| Parent | LoginDiv | LoginDiv | LoginDiv | LoginDiv |
| Style | none | none | display:hidden | None |
Which results in following property selector (the web page did not provide unique IDs)
USERNAME: (Type:TextBox;Parent:LoginDiv:TextMask:none:Style:none)
PASSWORD: (Type:TextBox;Parent:LoginDiv:TextMask:*;Style:none)
TOKEN: (Type:TextBox;Parent:LoginDiv:TextMask:*;Style:’display:hidden’)
LOGIN: (Type:Button;Parent:LoginDiv:TextMask:none;Style:none)
The test case is defined as:
Step 1: Enter “My name” in USERNAME
Step 2: Enter “My password” in PASSWORD
Step 3: Click LOGIN
It works, we are happy. As you can see in the table above, the developer has one additional field in the web page: He reused the login dialog as the password reset form as well. If the user is requesting to reset his password, he gets a token which allows him or her to change the password. The TOKEN field is usually hidden and only visible if you get to the page via a special URI.
In the latest release our developer has a technical bug, the password box is not part of the parent div anymore. Nevertheless, the designer fixed the styling so that everything looks nice again.
In the latest release our developer has a technical bug, the password box is not part of the parent div anymore. Nevertheless, the designer fixed the styling so that everything looks nice again.
| Attribute | USERNAME | PASSWORD | TOKEN | LOGIN |
| Type | Textbox | Textbox | Textbox | Button |
| Text Mask | (none) | * | * | n/a |
| Parent | LoginDiv | body | LoginDiv | LoginDiv |
| Style | none | none | display:hidden | None |
And here comes the promise of AI “No more broken test cases”. As with everything in AI it depends on how the AI is trained, how the feature vectors look like etc. but for this simple example we assume a change for an attribute is a 1, no change is a 0. Our well-trained AI is a bit biased for layout, which means the distance in the DOM is higher weighted than other attributes.
Which gives us the following weight distribution
| Attribute | Weight | USERNAME | PASSWORD | TOKEN | LOGIN |
| Type | 2 | Textbox | Textbox | Textbox | Button |
| Text Mask | 2 | (none) | * | * | n/a |
| Parent | 3 | LoginDiv | body | LoginDiv | LoginDiv |
| Style | 1 | none | none | display:hidden | none |
Resulting in the following weighted distances from the original PASSWORD field
| Attribute | USERNAME | PASSWORD | TOKEN | LOGIN |
| Type | 0 | 0 | 0 | 0 |
| Text Mask | 0 | 0 | 0 | 0 |
| Parent | 0 | 3 | 0 | 0 |
| Style | 0 | 0 | 1 | 0 |
| Total Distance | 0 | 3 | 1 | 0 |
Consequently, the AI automatically replaces the PASSWORD selector with the one that matches TOKEN just because the new PASSWORD property set has a higher distance from the original set than the TOKEN property set.
This will end up in a test execution as follows:
Step 1: Enter “My name” in USERNAME
Step 2: Enter “My password” in TOKEN
Step 3: Click LoginBtn
And the test case will fail, show a software error that no user will ever see. This error is as artificial, as the source of this confusion.
In conclusion, let me make one point clear. This is not a problem of artificial intelligence. It is the problem of applying the wrong tool to a well understood problem. You would never shovel snow with a screwdriver. You can, but it does not make much sense. Artificial intelligence is good if you can train it on a problem.
Use generic artificial intelligence (GAI)
Bear in mind, there is nothing like a GAI out there right now, that can understand your application and knows how it should work. So, if you need to train an AI on your application, it takes far more effort than just writing maintainable test cases. And you would have to retrain for every software change.
AI can assist you in testing
By now, you should understand what artificial intelligence stands for, what it can solve and why it is no good for manipulating test cases at all. Furthermore, the last case being typically sold as “no more broken test cases”, in general AI should never make the final decision.
Promise of AI “No more broken test cases”
On the other side, artificial intelligence can assist you as a tester. Hence, it is perfect for optimization problems. For example, if you need to find all text fields and buttons in this screenshot, a well-trained AI will give you the results in milliseconds.
Salesforce Printscreen as an example for the description
While for you, it will take a few minutes to select all of them. This is an actual example where AI can help you in testing. Naturally only if your testing tool supports visual testing.
Another example is in calculating the minimal amount of test cases required to exercise all features, by finding the perfect combination of required application interactions. Again, this is another optimization problem, a perfect fit for algorithms which fall into the category “artificial intelligence”.
Therefore, the AI automatically replaces the PASSWORD selector with the one that matches TOKEN just because the new PASSWORD property set has a higher distance from the original set than the TOKEN property set.
Conclusion
As demonstrated, artificial intelligence is a helpful tool for testers if used correctly. Consequently, using it as a problem solver for wrong selenium selectors or to work around limitations in the test tool is the wrong approach.
For the most part, artificial intelligence in testing is a hype. So if a vendor talks about providing “AI augmented” or “AI assisted” testing, it is worth a second look to see what the real value of this assistance is.
As two rules of thumb:
FOR DECISIONS: do not rely on artificial intelligence
- If AI is changing your checks/verification/test logic, do not rely on it.
- If you have a huge task which an AI can simplify for you by pre-calculating some candidates, go for it!
AI tests your system automatically. This statement must be crystal clear to you, thus as the vendor just the follwoing two questions:
- How does the artificial intelligence know the intended use of my application?
- How did you test your AI implementation?
In conclusion, you will most likely identify that the promised AI is no better than a random clicking machine, in testing your application.
- Is that how you would test?
- Is that good enough for you?
Leave a Reply