Why it pays to have your own Automation Framework

Steve Beck, Test Automation Engineering Lead, Inspired Testing.

---

One of the things that sets Inspired Testing apart is having access to our own, self-developed, self-maintained Automation Framework. I’m often asked what this is, why it’s important, and what companies should look for when reviewing an Automation Framework for their own purposes, so I’ve taken the opportunity below to address these specific questions.

What is an Automation Framework and why is it important?

An Automation Framework is, in simple terms, a set of tools, procedures and outcomes designed to enable and guide companies through the potentially complex and sometimes limiting process of Test Automation.

Enterprise or commercial tools are marketed in a specific way to solve all of your company’s issues with one tool. However, they tend to require specific expertise in the tool and are often limited to solving the problems of a mythical ‘generic type application’, and if there is anything I have learned from working in software development it’s that there is no such thing as a standard application.

The most valuable aspect of having a custom-built Automation Framework is that it can be tailored to solve the problems that you are facing and then continually enhanced to meet those ever-changing requirements.

Why did we develop our own Automation Framework?

The idea behind developing our own Framework was not to do something unique or that someone else can’t do, but rather to ensure we can customise our solution to meet our clients' needs without being limited to one or more vendors or tools.

It has our collective experience built in. Our framework has the basic toolset for solving automation problems but is extensible, both from the tester’s side and from the framework development side. Our framework also contains a library of pre-built functions which enable our teams to quickly start automating without re-inventing the wheel when it comes to mobile, Web and desktop applications. If one of our teams encounters and solves an issue which we believe could be a more global problem, we will then have them submit the solution for inclusion in the framework. This way our framework is constantly evolving (in a controlled manner) to solve clients’ future problems too.

It’s customisable and extensible. If, for example, a client has requirements like database reporting in specific formats, or they don’t like the output of a certain tool, we can easily change it. Some clients want an HTML report with interactive fields, some want a PDF, while others want an Excel sheet with all the data and Pivot Tables. Some even have their own tools and want our solution to use their reporting system.

We can customise and adapt our solutions specifically to clients’ needs.

Making it affordable. Test automation is complex and the skills needed to execute it properly can be expensive and difficult to come by. Commercial tools often come with expensive annual seat licences and runtime licences which limit your ability to scale software testing, then on top of that cost the client still needs to pay for the skills to use the tool. Clients just pay for the skills and get access to the framework as part of their contract with us. No double payments, and simple language accessibility when it comes to their own people using it.

Our value goes beyond the software we use, to include the skills and expertise we have in using the right tools to solve your unique software problems.

What makes a good Automation Framework?

Any good Automation Framework must be scalable, maintainable and readable.

Scalable. We want to be able to add as many tests as we need without it becoming a nightmare to manage. We also want to be able to do parallel execution. This means we can run a mobile test against as many mobile devices you’ve plugged in to your server; or a Web test can open as many page instances as your browser or Selenium Grid allows. We can run tests and report on them without getting lost in translation. In most cases it means we can run tests in the 100s or 1 000s.

For example, at one of our major insurance clients, we run 200 scenarios in a UI test format which, if run sequentially, would take eight hours to complete, but run in parallel via our Framework using eight separate browser instances, takes less than an hour.

Maintainable. We follow the traditional Page Object Model which we use to track where our automation scripts and locators are without getting overwhelmed by poorly named test classes. This layout is designed to imitate how an application presents to a user. There is a Test Class per Page of your application, which means that if we know that we’re doing tests on a given page, we can just go to the folder linked to the page (eg, a Login page folder) and we can see all the objects and actions that we could be testing. That way we can map out an application and reduce duplication of work. If a test fails because of an Automation issue we don’t have to search through our code for all the instances of the issue, rather we go to the page or component where the failure occurred and update the script in a single location.

Readable. We can do mobile testing on Android and iOS; interact with XLS, databases (all major formats) and CSV files. We can do desktop automation through older coded UI or new WinApp drivers; execute e-mail automation using SMTP, PoP or iMAP; use our Report Portal or integrate into the client’s CI system and report through there. We can also create pivot tables out of database reports and do Web testing via Selenium.

Despite all of these different tools and systems our tests remain readable as our framework prescribes a format which, if adhered to, means that a test designed for mobile has the same layout as a test designed for a Web application. The Page Object Model also helps to write readable tests, as each action is called from the page it is on, which means that anyone can read the test and see that a given action was performed on a given page.

The bottom line is this: we have taken all our experience working with multiple clients in diverse industries on hundreds of different projects to solve the problems that testers typically face when they automate an application under test. Rather than clients having to create their own Automation Frameworks at great cost over a long period, we’ve done that already to a very high standard, and it’s included as part of our Managed Delivery service.