• June 13, 2017

Challenges of Automated Testing

Challenges of Automated Testing

Challenges of Automated Testing 1024 717 Webteam

Automation Testing has become an essential part of software quality assurance and it will not change in the upcoming years. We must face it. The demand on delivering high quality software in the world where fast market response is essential for success is increasing each day. Automation Tests come in handy as they provide fast feedback to all the stakeholders and help to decide if we can release a SW product to the market or not. What is important to understand though is that as each approach to testing it has its challenges which you need to understand before jumping on the automation testing train.

Market Tool or My Own Framework

Whenever the management decides that automation is essential for the company the usual approach is to first research the already available tools. Why try to reinvent the wheel when there are multiple tools which can help you in testing? This seems like a natural approach but there are multiple downsides of this approach.

  1. You evaluate the test automation tool based on your current product portfolio. You cannot predict if the commercial tool will work with your next project.
  2. You need to learn how to use the tool. Often, it’s a completely new coding language you need to explore and you or your test group will need to learn how to use it. That often doesn’t scale as the whole team cannot start automating at the same time (someone still needs to test).
  3. If you would like to incorporate your tool into a continues integration process you often need to write additional  scripts/tools which will only deploy your nightly build and present you with the results of the tests.
  4. You need to be aware of fact that a commercial tool will not handle all the cases as it is designed to meet an average users’ needs. A great example would be WPF and Virtual Lists. The commercial test tool will not be able to find elements on the list as they are not yet available in the memory.
  5. You cannot adjust the tool you have bought. That sounds likes Captain Obvious but often there are functionalities which would make your life easier that could be easily implemented if it was your own tool.

It sounds like there are only downsides to this approach but building your own test framework has its challenges as well.

  1. First, you need a test automation specialist or someone from your development team to help you set everything up.
  2. Same as when buying an off-the-shelve tool you will need to learn how to code.
  3. You will need to understand how your software is built. Which limitation does it have? Is the software you are supposed to test Automation Friendly (e.g. AutomationIDs are given) or is it a 20-year old legacy code you need to deal with?
  4. How close will you need to work with the developers. Will you be able to join their team and work in close collaboration?

You need to keep in mind and make everyone aware that building a test framework is not a one day job and requires time and dedicated resources. If your management believes that this a simple task you need to make them aware that test automation must be treated as separate project. Your framework will need to grow together with your software and will have to be correctly maintained exactly as your software product.

Regardless of the approach you will chose both options require an upfront investment from your company. The uncertainty of the outcome and the money investment are usually what stops the companies from automating their tests. When your project has a tight budget and schedule, it is even more difficult to convince your management that it is the correct way to go. The managers are usually interested in fast results which will bring benefits to their project. In the end, they want to know what the quality of your software is based, on the test automation results. This is the key point. The bottom line is that test automation must provide test results regardless of the approach which is used to implement it. The test results and not the test framework itself, are the added value here. Nobody will be interested in the state of the art framework which uses the newest technologies as long it won’t provide reliable results.

I’ve seen multiple companies which tried to build their automation test environment and failed as the test results were not present in the desired time and budget. So, is there a different approach you can take? Well, you can choose not to automate at all and employee more people. We all know how it usually ends. It is never a linear benefit. You wouldn’t expect that 9 women deliver a baby in 1 month, would you?

Second option is that you outsource your manual tests and leave the automation to the experts. The benefit to that approach is that your current company organization remains as it is. There is no investment you need to do. You only pay for what is delivered to you. You get the test results on time, whenever you need them without having the burden of maintaining the whole infrastructure on your side. You pay only for what is done and not for something that might be done. Simple and measurable. We, at progile, offer such a service for you with our in-house solution TestResults.io. For more details go to http://www.progile.ch/testresults-io/ if you wish to bring quality to your software solution.

Leave a Reply