Wednesday, October 6, 2021

Engine Driven GUI Testing Design Pattern

Problem Statement:

Most GUI automation tests redo the same pattern repeatedly:

  • Controls are represented by a selector

  • Page objects encapsulate selectors and often methods which make use of the controls

  • Either tests or page methods manipulate these controls

  • Either tests or page methods validate the result

  • And we write these little pieces of code over and over for each action or validation we want to do

There is a lot of duplication in this model, even when it's done really well. 

Justification Intent:

By building an engine to validate all the controls on a page, and validating them whenever the page is loaded or changed, automation can catch things otherwise missed.


Reduce automated testing to writing descriptions of pages and data, which are then processed by an engine. No additional coding is required for most tests.

A broadly applicable special class of data driven test, where the data provided describes not only the test conditions, but also how the page is manipulated given said data.


In engine driven testing, we provide lists of controls to check, with not only the attributes for the control, but also attributes that described how the control was to be tested. The engine would then process the control lists.


Controls become collections of data that include not only identification data (eg a selector string) but also includes the information required in order to determine how to test the control. Tests provide test data (eg username, password, first name, and so on). An engine steps through the set of controls, applying or capturing data as directed.

Test data includes information like:

{'first_name': 'FQAUser', 'last_name': 'LQAUser'}

Control data, on the other hand, includes data similar to that shown in the example below:

my_control = SWADLControl(    selector="blahblah", is_text="Please enter your first name",    test_data_key="first_name",    actual_value_key="actual_first_name", validations={ VERIFY_EXIST: True, VERIFY_VISIBLE: True, VERIFY_ENABLED: True, VERIFY_UNIQUE: True, VERIFY_VALUE: "Please enter your first name", VERIFY_PROPERTY: ("selenium", "first_name"), VERIFY_INPUT: True, }, )

Using the test_data_key as the pointer into the test data, this then is enough to scan the input control and take a value from the test data to apply to the control.

Based on the control type, additional information is included in the control such as the expected value for labels, link targets for links and so on... And this then tells the engine how to manipulate and validate the control and it's contents. 

Values to capture for later evaluation are captured if a actual_value_key is specified in the control's data. 

The engine then uses data control's data to determine which tests to perform:

  • Verify exist

  • Verify visible

  • Verify enabled

  • Verify unique

  • Verify text value

  • Verify attribute value

  • Verify clickable = verify that it exists, is visible, is enabled, and is unique.

  • Verify Input = Set a value (from the test_data_key, for relevant types)

  • Verify Click = Click on the thing

Or other tests as needed. Some instructions are implicit... Eg you don't need to test the default text on a label if it's expected value isn't in the verify data.

Control information can also contain flags such as "any error with this control is fatal", or "this control is optional/required, and so on.

By adding another test to the engine, all the existing controls that are eligible for the new test also get tested, provided they have all the data needed. 

Of course not all tests will fit into this pattern, but in our experience, most will.

Collections of controls then can simply iterate through all the controls, calling the engine for each one.


my_control = SWADLControl(   
is_text="Please enter your first name",   
VERIFY_VALUE: "Please enter your first name",
VERIFY_PROPERTY: ("selenium", "first_name"),

This defines a control (in python). No flows or page methods will typically access this control other than via an inherited process_control(passed_dict) method.

In the example above, self refers to the page.

While this example is in python, it would even be possible to implement these descriptions as xml instead of code.


The engine is simply a set of conditionals, like if control.validations["verify_visible"]==True then assert control.get_visible().

Some control classes will take data and some won't. Those if statements will prevent the wrong thing from happening with the wrong control class. For example, if it's a label type control, we won't go looking to see if there's a value to put in the control. 

For any special case control, we just replace the process_control() method with a custom one. 

Personal Experience:

At Home Depot, we implemented this using wrapper objects for the controls, where the wrapper carried the selector, but also flags to indicate what was to be tested and what the expected result was. Then we just ran a loop to validate all the controls, calling the engine. The engine then read the flags so it knew what to test.

When confronted with significant changes to the AUT, we nonetheless were able to turn around our fixes in a matter of moments.

Friday, July 2, 2021

SQA: Our Tasking

  • We have to produce and maintain more tests than is physically possible. Obviously this is not an attainable goal, so our attainable goal is simply to automate as much as possible with the time we have, with sensitivity to both the business and dev priorities, and to do that consistently (which means not killing ourselves in the process).
  • What is our remit if our role is Quality Assurance Engineer?

    • Define tests which will prove the positive cases defined in the requirements/user stories/etc. (Getting the correct error message on invalid input is still a positive case for the purposes of this discussion.)

      • Absent that, we make wild guesses at what the System Under Test is supposed to do, and hold it to that until told better.

    • Document those tests (and advocate for suitable tools if they are not available, and no, Excel doesn’t count)

    • Execute those tests and deliver bugs for requirements not met

    • Think of the things the developer didn’t

    • User advocate

  • What is our remit if our role is Automation Engineer?

    • Deliver tests which will prove the positives

    • Implement those tests in a way where maintenance can be minimized and stability maximized

    • Document the code units that implement those tests such that the whole team can use them

    • We report on defects found in automation AND FLAG THEM AS FOUND IN AUTOMATION.

  • What is our remit if our role is Automation Architect?

    • Consistently deliver build go/no go decisions

    • Define the standards by which the tests and their execution environment(s) will be built and maintained such that:

      • We maximize reuse

      • We maximize stability and uptime

      • We maximize the access to information

      • We minimize the time to bring new people up to speed

      • We minimize the time required to identify and isolate issues

      • We minimize the time required to look stuff up during test development and debugging

    • We document, design, implement, delegate, train, validate, source expertise and otherwise do what it takes to make sure that our team can do all of the above

      • In particular, we want to be training our replacement(s) from day one. We are responsible for assuring success even if we’re no longer available.

    • Communicate with other teams about our needs, accomplishments, test results, RPA, automation training, and anything else we can provide.

Monday, December 21, 2020

SDET vs Architect?

 I had somebody asked me the other day what the difference was between a test automation architect and an SDET.

An SDET is a software developer in test. Software developer skillset applied to testing. I'm glad this now has a name and certification. When I started, there wasn't such.

An Architect is any developer who's got enough miles under their belt that they look at everything from a the perspective of how it all fits together and interacts with everything outside of itself.

Can an SDET be an architect? Absolutely. Are all SDETs experientially equipped to approach their work as Architects? No more so than all developers are architects.

I started working with computers in 1979, I got my first job in computers in 1982, got my first support gig in 1987. my first QA gig in 1990. First automation assignment in dec 1990. 

A year later, and I was overwhelmed by the maintenance. It was a month after I realized that fact that I started thinking about test automation from the perspective of architecture.

My own interest came first from trying to solve my own problems, and then from those my management asked of me. How can I reuse most of what I produce? How can I reduce maintenance? Can you build out a lab so we can run these over and over? I want you to make it so everybody in QA can create automated tests, even the non programmers. We have too much data left over from all those old builds and test runs. 

I've solved every problem thrown at me. I'm a good tester, but I'm a much better architect.

Saturday, June 20, 2020

Selenium As A Modern Automation Tool

2020 Akien Maciain, Paul Castillo, Alejandro Garcia


This document covers a deployment of Selenium using Python for a large retail firm. It was started in 2015, and as of this writing is still in use today. Here we provide an overview of key features. A companion file includes a functional mockup in Python 3, which is not taken from the firm's code.

The systems we were testing were customer and retail associate facing pages responsible for connecting consumers with home service professionals who would perform services such as house painting, plumbing installation and repair, and so on.


  • Control - A proxy object that represents a control on the screen.
  • Element - A DOM object on the screen that a "control" interacted with.
  • Golden Path - A flow that a user might perform that is without errors, backtracking, or other tangents.
  • SUT - System Under Test

Primary features

  • Fully Automatic Self Synchronization for all control interaction, with interfaces loosely based on tools like UFT and SilkTest. Zero sync failures.
  • Storyboarding is implemented as instances of a"flow class", an object layer between the tests and page objects. These abstract a flow a user might do. Adding this layer made the tests far more resilient.
  • Tagged Interfaces that prevented the need for maintenance in most cases. This required buy in from development, but became the channel that developers used to communicate UI changes to QA. Almost as good as AI-based self healing.
  • Engine based testing that validates all controls whether the automation "engages with them" or not.
  • Fully automated self setup for all environments, including a Docker container to actually run the tests to reduce the differences between individual computers.

Fully Automatic Self Synchronization

The framework as implemented did not have issues with synchronizing with the SUT. This was accomplished by implementing the controls as objects, with instances on the page objects for each control. When the controls are instantiated, the selector is one of the passed arguments.

The system performs automatic retries until a desired condition is met or timeouts are reached on all interactions with the page elements. This prevents stale element exception failures and waiting even one millisecond longer than is needed.

Asserting states before actions (eg asserting that a page is loaded before trying to interact with it) means the test only stops when it can't continue. Other failures that don't block test completion are logged as the test continues.

Storyboarding via Flow Layer

A flow is any activity a user might do. It often spans more than one page.

The test knows which flows to call but does not know anything about pages or controls. This decoupling of tests and pages means even in the face of a broad revision to all the page code the tests are unchanged. Pages are rebuilt and flows updated to point to the new pages. As long as the underlying application features still do basically the same things, the tests are unaffected.

In our model, tests are responsible for data, which is passed into flows as a hash table. Flows knew which page object and methods to use and passed the dictionary along. Pages implement page-specific subflows as methods and which controls handle what actions.

Strict encapsulation meant very low maintenance even in the face of big change.

Tagged Interfaces

In this framework, we could specify control selectors in css at instantiation time. Usually those css tags were not native products of the front end framework, but rather "selenium tags", like so:

<h3 selenium="tile-name" class="TILE-NAME">Bath</h3>

It took some convincing some developers to add these tags. We'd create tasks for them to implement these tags, and then we'd instantiate a control like this:

tile_header = Control(selector='[selenium="tile-name"]')

When the dev team rearranged a page, they would move the tags to the new controls. If they failed to do so, we'd write a blocking bug, and they'd get it fixed straight away.

Eventually, Dev trained the Quality Assurance teams to make these changes to the front end code ourselves.

Tagged interfaces are theoretically not as robust as Artificial Intelligence "self healing" features of many of the tools now on the market. However, on the ground, this proved entirely adequate.

Engine Based Testing

In the normal course of automation, we would touch only specific controls. For instance, in a login window, we would touch the username, password, and submit button. This limits the coverage of the automation to what we called "key controls".

However, a page is made up of much more than the key controls. Many have common objects such as headers, footers, hero images, title text, and so on. While not critical to the golden path, these objects can all be extremely important to validate.

To deal with this, we implemented an engine. When we instantiated the control objects, we would also add attributes that would tell the engine how to test the control. So if a control should exist, it might have a VERIFY_EXIST=True as one of it's instantiation arguments. We could verify existence or not, visible or not, enabled or not, whether a control was unique, what it's expected text value was, and so on.

We also implemented KEY as one of our instantiation keywords, and the KEY pointed to a keyed value in the test data dictionary. So we could deal with dynamic values as well.
A flow would simply call the engine before doing anything else on the page. For some golden path tests we could even populate fields like username and password using the KEY value, and not even have to write a subflow for that page.

Self Setup

One of the features that was mentioned repeatedly with appreciation by the team was the fully automated setup. A person could download a script from the knowledge base, run it locally, and it would set up everything about the automation. All system packages, download all the needed repos from github, set up their docker container… Everything. This was implemented in Bash, but today we'd use something like Ansible.

We used Docker to provide a single, consistent environment for the testing tools and execution environment, which meant that the specifics of how individual computers didn't impact the testing.


After having conducted a thorough high level overview of many of the tools on the market, we were amused to see how many claimed these features could not be implemented in Selenium and expensive tools were instead required in order to make maintainable tests. This framework proves that assertion wrong. However, it does require building out the control object, getting dev buy-in for tags, and implementation of an Ansible (or similar) setup script. None of this is beyond the capabilities of a competent engineer.

Tuesday, February 5, 2019

Automation Pain Points II: Resilience Part 3, Selectors

In the last two sections, we looked at strategies that orchestrate the behavior of pages and flows. These have made a huge difference in our day to day work. The third piece requires buy in from dev.

We kept rebuilding page objects, but sorting the new selectors was kind of a nightmare. Just when we thought it was all done, changed again!

We realized that the developer still knows where the first name field is... Or the submit button... Or whatever.

So one of our engineers made this case to dev: We will add selectors to our pages as we normally do. We will also open a ticket to have new properties added to each control. The property name will be 'selenium', so we can have selectors like '[selenium="first-name"]'.

The engineer is responsible for adding those tags. And when the page is changed, they're the ones changing it, and they are required to keep the selenium properties intact as they redesign the page. If a selenium tag disappears, it's a P0 bug, because it blocks testing.

They are not required to put selenium tags on anything we didn't identify as needing them.

This took us from a couple of days before everything was smooth again to a few minutes. Huge benefit!

Thursday, November 29, 2018

Automation Pain Points II: Resilience Part 2, Flows

In the last piece, we looked at the benefits I've seen at other companies using strict encapsulation at the page level. But in my current gig, we take it a step farther. We have another layer between tests and pages called flows. A flow interacts with one or more page objects.

This means that pages are the workers, they know how to do specific pieces of work.

Flows are the managers, they line up which workers and in what order. Flows are also a series of interactions the user will take to perform some action. They model the end user's interaction with the system.

The tests are the executives that decide which flows to call to test a specific underlying feature or group of features.

In practice, this looks like a lot of unnecessary complexity, especially for test cases who's flows are one line long. But it also further isolates the test from the changes in the UI.

We've had large scale redesigns that mostly take page object updates. Sometimes flows need to be updated. But the underlying functionality hasn't really changed.

This means tests are updated more quickly, and so the automation can do more for less effort.

It also means we can write automated tests and flows before we have page mockups, as long as we have detailed enough specs. We can add the page objects as they become available.

Woo hoo!

Saturday, November 3, 2018

Automation Pain Points II: Resilience Part 1, The Page

I am a fundamentally lazy guy. That's why I write test automation, because I don't want to keep doing the same crap over and over.

But the problem with encoding knowledge into source code is that the representation of knowing thus created is brittle. Every time the AUT changes, the tests have to change.

I saw this first at Symantec in 1991, working on OnTarget, a project management package. There were 3 of us automators, and we built up a rather small lot of tests, before we became unable to create any more. We were too busy keeping up with AUT changes.

By this time, we were using SilkTest, and their implementation of page objects. Their page objects were really just intended to be a better tool for managing selectors than the huge lists of constants that was the way QAWorkbench (which became SilkTest) had done in the alpha stage of it's development.

But we took it a step further, and built what folks today would recognize as page objects. The point behind the way we use page objects at Home Depot today is to encapsulate all the details of interacting with a page into one place.

When I started there, we had tests directly calling methods in page objects. So when they completely redesigned the app, all the tests had to be thrown out. This is not good resilience.

The problem was that the test code was tightly coupled to the page code. The rule when I arrived was that no test used the selectors on a page. Instead, methods were created. I understand why they created that rule, but the problem was they had routines like click_submit_button(), which meant that how to operate the page was encoded in the test. The AUT underwent a complete redesign, and all the tests were then trash, and had to be rebuilt almost completely from scratch.

Today, each page object implements subflows. A subflow takes a dictionary with all the data needed for that page, and implements all the steps for a single action on that page. Everything about how the page does it's work is encapsulated and isolated from the test. Even default values we expect controls to have, such as the text of error messages that should appear, are stored in the page object itself.