Saturday, June 20, 2020

Selenium As A Modern Automation Tool

2020 Akien Maciain, Paul Castillo, Alejandro Garcia

Introduction 

This document covers a deployment of Selenium using Python for a large retail firm. It was started in 2015, and as of this writing is still in use today. Here we provide an overview of key features. A companion file includes a functional mockup in Python 3, which is not taken from the firm's code.

The systems we were testing were customer and retail associate facing pages responsible for connecting consumers with home service professionals who would perform services such as house painting, plumbing installation and repair, and so on.

Terms

  • Control - A proxy object that represents a control on the screen.
  • Element - A DOM object on the screen that a "control" interacted with.
  • Golden Path - A flow that a user might perform that is without errors, backtracking, or other tangents.
  • SUT - System Under Test

Primary features

  • Fully Automatic Self Synchronization for all control interaction, with interfaces loosely based on tools like UFT and SilkTest. Zero sync failures.
  • Storyboarding is implemented as instances of a"flow class", an object layer between the tests and page objects. These abstract a flow a user might do. Adding this layer made the tests far more resilient.
  • Tagged Interfaces that prevented the need for maintenance in most cases. This required buy in from development, but became the channel that developers used to communicate UI changes to QA. Almost as good as AI-based self healing.
  • Engine based testing that validates all controls whether the automation "engages with them" or not.
  • Fully automated self setup for all environments, including a Docker container to actually run the tests to reduce the differences between individual computers.

Fully Automatic Self Synchronization

The framework as implemented did not have issues with synchronizing with the SUT. This was accomplished by implementing the controls as objects, with instances on the page objects for each control. When the controls are instantiated, the selector is one of the passed arguments.

The system performs automatic retries until a desired condition is met or timeouts are reached on all interactions with the page elements. This prevents stale element exception failures and waiting even one millisecond longer than is needed.

Asserting states before actions (eg asserting that a page is loaded before trying to interact with it) means the test only stops when it can't continue. Other failures that don't block test completion are logged as the test continues.

Storyboarding via Flow Layer

A flow is any activity a user might do. It often spans more than one page.

The test knows which flows to call but does not know anything about pages or controls. This decoupling of tests and pages means even in the face of a broad revision to all the page code the tests are unchanged. Pages are rebuilt and flows updated to point to the new pages. As long as the underlying application features still do basically the same things, the tests are unaffected.

In our model, tests are responsible for data, which is passed into flows as a hash table. Flows knew which page object and methods to use and passed the dictionary along. Pages implement page-specific subflows as methods and which controls handle what actions.

Strict encapsulation meant very low maintenance even in the face of big change.

Tagged Interfaces

In this framework, we could specify control selectors in css at instantiation time. Usually those css tags were not native products of the front end framework, but rather "selenium tags", like so:

<h3 selenium="tile-name" class="TILE-NAME">Bath</h3>

It took some convincing some developers to add these tags. We'd create tasks for them to implement these tags, and then we'd instantiate a control like this:

tile_header = Control(selector='[selenium="tile-name"]')

When the dev team rearranged a page, they would move the tags to the new controls. If they failed to do so, we'd write a blocking bug, and they'd get it fixed straight away.

Eventually, Dev trained the Quality Assurance teams to make these changes to the front end code ourselves.

Tagged interfaces are theoretically not as robust as Artificial Intelligence "self healing" features of many of the tools now on the market. However, on the ground, this proved entirely adequate.

Engine Based Testing

In the normal course of automation, we would touch only specific controls. For instance, in a login window, we would touch the username, password, and submit button. This limits the coverage of the automation to what we called "key controls".

However, a page is made up of much more than the key controls. Many have common objects such as headers, footers, hero images, title text, and so on. While not critical to the golden path, these objects can all be extremely important to validate.

To deal with this, we implemented an engine. When we instantiated the control objects, we would also add attributes that would tell the engine how to test the control. So if a control should exist, it might have a VERIFY_EXIST=True as one of it's instantiation arguments. We could verify existence or not, visible or not, enabled or not, whether a control was unique, what it's expected text value was, and so on.

We also implemented KEY as one of our instantiation keywords, and the KEY pointed to a keyed value in the test data dictionary. So we could deal with dynamic values as well.
A flow would simply call the engine before doing anything else on the page. For some golden path tests we could even populate fields like username and password using the KEY value, and not even have to write a subflow for that page.

Self Setup

One of the features that was mentioned repeatedly with appreciation by the team was the fully automated setup. A person could download a script from the knowledge base, run it locally, and it would set up everything about the automation. All system packages, download all the needed repos from github, set up their docker container… Everything. This was implemented in Bash, but today we'd use something like Ansible.

We used Docker to provide a single, consistent environment for the testing tools and execution environment, which meant that the specifics of how individual computers didn't impact the testing.

Conclusion

After having conducted a thorough high level overview of many of the tools on the market, we were amused to see how many claimed these features could not be implemented in Selenium and expensive tools were instead required in order to make maintainable tests. This framework proves that assertion wrong. However, it does require building out the control object, getting dev buy-in for tags, and implementation of an Ansible (or similar) setup script. None of this is beyond the capabilities of a competent engineer.

No comments:

Post a Comment