Wednesday, October 6, 2021

Engine Driven GUI Testing Design Pattern

Problem Statement:

Most GUI automation tests redo the same pattern repeatedly:

  • Controls are represented by a selector

  • Page objects encapsulate selectors and often methods which make use of the controls

  • Either tests or page methods manipulate these controls

  • Either tests or page methods validate the result

  • And we write these little pieces of code over and over for each action or validation we want to do

There is a lot of duplication in this model, even when it's done really well. 

Justification Intent:

By building an engine to validate all the controls on a page, and validating them whenever the page is loaded or changed, automation can catch things otherwise missed.


Reduce automated testing to writing descriptions of pages and data, which are then processed by an engine. No additional coding is required for most tests.

A broadly applicable special class of data driven test, where the data provided describes not only the test conditions, but also how the page is manipulated given said data.


In engine driven testing, we provide lists of controls to check, with not only the attributes for the control, but also attributes that described how the control was to be tested. The engine would then process the control lists.


Controls become collections of data that include not only identification data (eg a selector string) but also includes the information required in order to determine how to test the control. Tests provide test data (eg username, password, first name, and so on). An engine steps through the set of controls, applying or capturing data as directed.

Test data includes information like:

{'first_name': 'FQAUser', 'last_name': 'LQAUser'}

Control data, on the other hand, includes data similar to that shown in the example below:

my_control = SWADLControl(    selector="blahblah", is_text="Please enter your first name",    test_data_key="first_name",    actual_value_key="actual_first_name", validations={ VERIFY_EXIST: True, VERIFY_VISIBLE: True, VERIFY_ENABLED: True, VERIFY_UNIQUE: True, VERIFY_VALUE: "Please enter your first name", VERIFY_PROPERTY: ("selenium", "first_name"), VERIFY_INPUT: True, }, )

Using the test_data_key as the pointer into the test data, this then is enough to scan the input control and take a value from the test data to apply to the control.

Based on the control type, additional information is included in the control such as the expected value for labels, link targets for links and so on... And this then tells the engine how to manipulate and validate the control and it's contents. 

Values to capture for later evaluation are captured if a actual_value_key is specified in the control's data. 

The engine then uses data control's data to determine which tests to perform:

  • Verify exist

  • Verify visible

  • Verify enabled

  • Verify unique

  • Verify text value

  • Verify attribute value

  • Verify clickable = verify that it exists, is visible, is enabled, and is unique.

  • Verify Input = Set a value (from the test_data_key, for relevant types)

  • Verify Click = Click on the thing

Or other tests as needed. Some instructions are implicit... Eg you don't need to test the default text on a label if it's expected value isn't in the verify data.

Control information can also contain flags such as "any error with this control is fatal", or "this control is optional/required, and so on.

By adding another test to the engine, all the existing controls that are eligible for the new test also get tested, provided they have all the data needed. 

Of course not all tests will fit into this pattern, but in our experience, most will.

Collections of controls then can simply iterate through all the controls, calling the engine for each one.


my_control = SWADLControl(   
is_text="Please enter your first name",   
VERIFY_VALUE: "Please enter your first name",
VERIFY_PROPERTY: ("selenium", "first_name"),

This defines a control (in python). No flows or page methods will typically access this control other than via an inherited process_control(passed_dict) method.

In the example above, self refers to the page.

While this example is in python, it would even be possible to implement these descriptions as xml instead of code.


The engine is simply a set of conditionals, like if control.validations["verify_visible"]==True then assert control.get_visible().

Some control classes will take data and some won't. Those if statements will prevent the wrong thing from happening with the wrong control class. For example, if it's a label type control, we won't go looking to see if there's a value to put in the control. 

For any special case control, we just replace the process_control() method with a custom one. 

Personal Experience:

At Home Depot, we implemented this using wrapper objects for the controls, where the wrapper carried the selector, but also flags to indicate what was to be tested and what the expected result was. Then we just ran a loop to validate all the controls, calling the engine. The engine then read the flags so it knew what to test.

When confronted with significant changes to the AUT, we nonetheless were able to turn around our fixes in a matter of moments.

Monday, July 12, 2021

Layers 101: Test Automation and the The Flow Design Pattern


Of all the design patterns that contribute to a good test automation environment, the layers design pattern is the single most important.

The layered architecture pattern divides the components into a number of horizontal layers. These layers each know how to make requests of the lower layers, but not how the lower layer does it’s work (encapsulation). This is the traditional method for designing most software.

The central idea is to use structured divisions to break the code into pieces who each know how to ask other pieces to do things, but have no idea how it’s done. The internet is built this way. Your home router doesn’t know how the web servers do what they do, it just hands you the packets addressed to you. Layers above that in your computer don’t know how the packets get here, but does know how to reassemble them. This isolation of functionality is why you don’t have to replace your home router when a new kind of service, such as email or chat, is added.

What follows is a discussion with many examples of pseudocode. The same layers discussed below are applicable whether you’re using a code free or low code tool as they are for any of the coded tools.

Layers Tutorial and Example Use Case

Imagine we had thousands of tests. And that they look sorta like this:

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 Driver.OpenPage(globals.env_url) 6 Driver.WaitForControlVisible("user-name-selector", 20) 7 Driver.SendKeys("user-name-selector", EmailAddr) 8 Driver.SendKeys("password-selector", PassWord) 9 Driver.Click("login-button-selector") 10 assert Driver.GetVisible("hero-image-selector") 11 Driver.Click("my-account-selector") 12 assert Driver.GetVisible("my-account-change-email-button-selector") 13 Driver.Click("my-account-change-email-button-selector") 14 Driver.TypeKeys("my-account-email-box-selector", NewEmailAddr) 15 Driver.Click("my-account-save-button-selector")

We have a test which logs in, and changes the user’s email. We eventually hope to have thousands of tests like this.

Imagine if every test case has to log in, so they all start with that part of this code. Now imagine that the selector (strings which define how to find that control on the page) user-name-selector changes. That means we could have thousands of places to change it.

To solve this, we have to move the selectors out of the tests. To do this, we use the layers pattern to create a new place to put the selectors.

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 Driver.OpenPage(globals.env_url) 6 Driver.WaitForControlVisible(LoginPage.EmailAddrSelector, 20) 7 Driver.SendKeys(LoginPage.EmailAddrSelector, EmailAddr) 8 Driver.SendKeys(LoginPage.PassWordSelector, PassWord) 9 Driver.Click(LoginPage.LoginButton.Selector) 10 assert Driver.GetVisible(LoggedInPage.HeroImage.Selector) 11 Driver.Click(LoggedInPage.MyAccountSelector) 12 assert Driver.GetVisible(MyAccountPage.ChangeEmailButtonSelector) 13 Driver.Click(MyAccountPage.ChangeEmailButtonSelector) 14 Driver.TypeKeys(MyAccountPage.EmailBoxSelector, NewEmailAddr) 15 Driver.Click(MyAccountPage.SaveButtonSelector")

In this example, there are two layers: the test and the page. The Page has the selectors for the controls we’re going to manipulate. The test sets up the data, loads the page, uses the selectors to populate the fields, and click Login. Having a separate page object is called The Page Object Pattern.

In this case, if the selector changes, when we next run the tests, they will fail. But we only have to update the page, and all the tests will work once again.

In this example, the two layers are tightly coupled. So the test knows how the page is set up, and manipulates it to make things happen.

In general, having the tests tightly coupled means that any time I make a change to the page that’s greater than just changing a selector, I have to also change all the tests that do the same thing. We want Loosely Coupled components to make the code more maintainable.

This is however a login, and many tests will have to do that. So we turn it into functions, and now we can do this:

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 LoginPage.OpenLoginPage() 6 LoginPage.PopulateUser(EmailAddr) 7 LoginPage.PopulatePassword(PassWord) 8 LoginPage.Submit() 9 LoggedInPage.VerifyLoaded() 10 LoggedInPage.OpenMyAccount() 11 MyAccountPage.VerifyLoaded() 12 MyAccountPage.EnterEmail(NewEmailAddr) 13 MyAccountPage.Save()

No more pesky selectors in the test! We have created multiple methods (functions belonging to an object class) on the login page, which do part of the work for us. This is even less tightly coupled.

We can simplify this further, by moving the individual actions into a wrapper function which does all the login for us.

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 LoginPage.Login(EmailAddr, PassWord) 6 LoggedInPage.VerifyLoaded() 7 LoggedInPage.OpenMyAccount() 8 MyAccountPage.VerifyLoaded() 9 MyAccountPage.EditEmail(NewEmailAddr)

EditEmail now also performs the Save operation. Woo hoo! Much simpler! And if the login function changes radically, but is still on one page, we just update the one page, and all the tests now work again.

But life is seldom so simple, so now let’s make our login more complicated. Let’s say it has a second page, with a Two Factor Authentication (2FA).

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse" 6 LoginPage.Login(EmailAddr, PassWord) 7 TwoFAPage.Enter2FA(TwoFactorAuth) 8 LoggedInPage.VerifyLoaded() 9 LoggedInPage.OpenMyAccount() 10 MyAccountPage.VerifyLoaded() 11 MyAccountPage.EditEmail(NewEmailAddr)

Now every time we log in, and do 2FA, we have to call those in every test. And if that changes, we’re still back to updating all the tests.

The Flows Design Pattern

So to really make this robust, we need a way to get that UI specific functionality out of the test. So we make a new layer. This one will be called flows. (Some tools and practitioners use the term storyboarding.)

A flow is any action a user might do. So logging in is a user action, rather than an action specific to individual pages.

Let’s look at that:

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse" 6 LoginFlow.Login(EmailAddr, PassWord, TwoFactorAuth) 7 MyAccountFlow.UpdateEmail(NewEmailAddr)

This example is about as close as we can get to the test thinking about what’s to be done the way a human would. This example is clearly incomplete, so let’s show a more complete version of this example.

Development Example

We’ll tackle this in the order I usually do when developing a test. Your mileage may vary…

  • We know we want to change the email address of somebody, so we make sure we have a somebody in the system.

  • We also need an address to change to

  • And the login password and 2FA

1function Test1() 2 EmailAddr = "" 3 NewEmailAddr = "" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse"
  • And we’re going to need two flows, one which can log in, and one which can change the email.

1 LoginFlow.Login(EmailAddr, PassWord, TwoFactorAuth) 2 MyAccountFlow.UpdateEmail(NewEmailAddr)
  • Yes, this does look just like the example above, but this is the order I do it in. The important point here is that right now, at this point in the process, I have a test. But it won’t run. LoginFlow and MyAccountFlow don’t yet exist. So that’s what’s next. I know I’m gonna need a login page object, a logged in page object, and a my account page... But I can specify them without them existing yet. (Some tools, such as Worksoft Certify, may require stubbing such routines until the new ones are ready.)

1Class LoginFlow 2 Function Login(EmailAddr, PassWord, TwoFactorAuth) 3 LoginPage.Login(EmailAddr, PassWord, TwoFactorAuth) 4 LoggedInPage.VerifyLoaded() 5 6Class MyAccountFlow 7 Function UpdateEmail(NewEmailAddr) 8 LoggedInPage.OpenMyAccount() 9 MyAccountPage.VerifyLoaded() 10 MyAccountPage.EnterEmail(NewEmailAddr) 11 MyAccountPage.Save()

Now we can build out this version of the page object.

1Class LoginPage 2 UserNameSelector = "user-name-selector" 3 PassWordSelector = "password-selector" 4 LoginButtonSelector = "login-button-selector" 5 6 function Login(self, EmailAddr, PassWord, TwoFactorAuth) 7 LoginPage.OpenLoginPage() 8 LoginPage.PopulateUser(EmailAddr) 9 LoginPage.PopulatePassword(PassWord) 10 LoginPage.Submit() 11 12 function OpenLoginPage() 13 Driver.OpenPage(globals.env_url) 14 Driver.WaitForControlVisible(UserNameSelector, 20) 15 16 Function PopulateUser(EmailAddr) 17 Driver.SendKeys(UserNameSelector, EmailAddr) 18 19 Function PopulatePassword(PassWord) 20 Driver.SendKeys(PassWordSelector, PassWord) 21 22 Function Submit() 23 Driver.Click(LoginButtonSelector)

Notice that this is all the same actual lines of code doing the work… But it’s broken up into many functions. We don’t actually NEED to break it up this much, login is a fairly simple process… But imagine that we may also need to test login failures in the future. So we can’t only have the one wrapper function. We might need to Submit(), and then verify an error message, for instance.

Now we can start debugging it. Because of course it won’t work first try.  

The Layers We Just Built

We’ve just built a stack with 3 layers:

  • Tests – Tests know about data, and about which flow objects to ask to do it’s bidding. Tests know nothing at all about pages ever. Tests can talk to flows, and that’s it. Tests can also validate data that comes back to them from the flows.

  • Flows – Flows know which pages do what, and what to ask them to do. But they don’t validate data. Flows can call pages and other flows as well. Flows can also manipulate data, but this should be kept to a minimum. A flow may need to reformat data from one page object call before passing it to a different page in a later step.

  • Pages – Pages know about the controls on their page, and how to get it all to do what it does. Pages NEVER call other pages, flows or tests. Pages can pass data back to the flows.

(There’s a lot more than that which we’re leaving out, like the layer that is the browser, and the layer that is the testing tool, and so on.)