Uniform Output

In my current gig, we have more than one testing framework. For predominantly historical reasons, we have 3 different test automation frameworks at work. One in python, one in java, and one in javascript. 

When I started here, I was given ownership of the team producing test automation in python. I set one of them up as the Framework Owner, responsible for all the framework changes, and supporting the rest of the team in the use of it. 

And then my framework owner had an emergency and had to take a month's hiatus. I was left to fill his shoes. But I hadn't actually used the framework. I'd built some helpers, and built unit tests for them, but I hadn't even myself had to debug any of the cases. Much less rummage though the output to understand what was happening.

I wound up spending time to solve the short term problem. I added code to output the test status and debugging info I needed to be able to report results. When he got back, I removed that code. 

And thought about that experience. And the fact that, where we've been, if we had a java opening, and a surplus python programmer, we'd probably fire the one and go looking for the other. Thing is, when that happens, we lose a lot of institutional knowledge. 

And the language itself is the easy part. I learned python in a week by asking google how to do things I already knew how to do in other programming languages. I even won a "fastest code wins" competition with a developer at a company I was trying to get on with. In a week.

The harder part is finding the pieces in the code and the output. 

As I ruminated on that, I was reminded that while I was at Home Depot, I was given the directive that every person with the company shall be able to read and understand the summary portions of the test result output. Which we did.

And I realized that if I had that same model, with all the pieces talked about below, not only would everybody be able to understand it, it would also speed up any debugging we needed to do. The reasons will become clear in the examples below.

And, if we implemented this same kind of output in all three of our frameworks, engineers who knew to look for these landmarks in the output of one would be able to switch to other projects, even ones where they had to learn a new programming language, and be able to add value within a week. So maybe we can keep folks we might otherwise have to lay off. 

Work has given me leave to roll out Uniform Results. This doesn't replace any existing thing. In fact, we're also looking at moving all the frameworks to Allure for at least some of the reporting. But this will add to the console output, and may also eventually serve as a springboard for deploying a logging server (since now all the traffic will come though one set of pipes).

BANNERS

A banner is a chunk of output which looks like this:

(using images because Blogger didn't like the emoji)

This particular one is performing an assertion that a substring will be in a string. If instead it had been a failure from a selector not finding the element, then the fields would instead include things like what the selector was, how long the timeout was, and similar information.  

My own experience has been that in the past I've often had to run a failed test again to get some missing piece of information. The point behind the banner is to never have to run the test again for more information. If you need to do that, you add that to the template for whatever kind of banner it is.

Banners get emitted for every assertion failure (including soft ones), or error.

TERMINAL REPORT

The Terminal Report comes at the end of the test run, and summarizes a lot of information. It comes in three parts:

TERMINAL REPORT: TEST ENVIRONMENT REPORT

This basically just lists anything about the test run which might be of value later in investigating any issues. This is a completely faked up example: 

TERMINAL REPORT: TEST CASE SUMMARY

For each test case, it could end with any of the following end states:

We can attach marks to the test cases indicating if there's a bug against that case. This lets us report Unchanged and Fixed as a test status.

This shows status information for 3 cases, at least, most of 3. :) The middle one is a data set test, and row 6 suffered an assertion failure, while row 11 suffered an error.

TERMINAL REPORT: TEST RUN SUMMARY

This comes at the very end, and summarizes the whole run:

On the history of test automation...

I was watching a fabulous video on test automation and "flows" here: https://www.youtube.com/watch?v=706G6v2kMzQ

But his notion of history is a bit out of sync. This is my reply:

An interesting point, you say the page objects, UI objects and intermediate layers were build from early 2000s to 2015... 

SilkTest is a Windows automated testing product. It was shipped in beta form in 1991 by Segue software (Marsten Parker and Dave Laroche). It had page objects, page object methods, and control objects. These features, to one extent or another, were common in commercial tools for the Windows platform by 2000. 

I myself was adding an abstraction layer between pages and tests to isolate the tests from the UI in 1991. At that time I was abstracting features, but in 2015, I switched to user role centric flows. Much easier to teach my team to think in terms of designing tests for flows rather than features. :)

I had a conversation with whoever was leading Mozilla's efforts in 2010 about bringing all this to Selenium back then. I build it in 2015, and there's a newer version here: https://github.com/akienm/swadl which also supports engine driven testing, where the page objects describe what needs to be tested about each control, and a single call does it all.  https://testautomationarchitecture.blogspot.com/2021/10/engine-driven-gui-testing-design-pattern.html

Engine Driven GUI Testing Design Pattern

Problem Statement:

Most GUI automation tests redo the same pattern repeatedly:

• Controls are represented by a selector

• Page objects encapsulate selectors and often methods which make use of the controls

• Either tests or page methods manipulate these controls

• Either tests or page methods validate the result

• And we write these little pieces of code over and over for each action or validation we want to do

There is a lot of duplication in this model, even when it's done really well. 

Justification Intent:

By building an engine to validate all the controls on a page, and validating them whenever the page is loaded or changed, automation can catch things otherwise missed.

Description:

Reduce automated testing to writing descriptions of pages and data, which are then processed by an engine. No additional coding is required for most tests.

A broadly applicable special class of data driven test, where the data provided describes not only the test conditions, but also how the page is manipulated given said data.

Detail:

In engine driven testing, we provide lists of controls to check, with not only the attributes for the control, but also attributes that described how the control was to be tested. The engine would then process the control lists.

Example:

Controls become collections of data that include not only identification data (eg a selector string) but also includes the information required in order to determine how to test the control. Tests provide test data (eg username, password, first name, and so on). An engine steps through the set of controls, applying or capturing data as directed.

Test data includes information like:

{'first_name': 'FQAUser', 'last_name': 'LQAUser'}

Control data, on the other hand, includes data similar to that shown in the example below:

my_control = SWADLControl(    selector="blahblah", is_text="Please enter your first name",    test_data_key="first_name",    actual_value_key="actual_first_name", validations={ VERIFY_EXIST: True, VERIFY_VISIBLE: True, VERIFY_ENABLED: True, VERIFY_UNIQUE: True, VERIFY_VALUE: "Please enter your first name", VERIFY_PROPERTY: ("selenium", "first_name"), VERIFY_INPUT: True, }, )

Using the test_data_key as the pointer into the test data, this then is enough to scan the input control and take a value from the test data to apply to the control.

Based on the control type, additional information is included in the control such as the expected value for labels, link targets for links and so on... And this then tells the engine how to manipulate and validate the control and it's contents. 

Values to capture for later evaluation are captured if a actual_value_key is specified in the control's data. 

The engine then uses data control's data to determine which tests to perform:

• Verify exist

• Verify visible

• Verify enabled

• Verify unique

• Verify text value

• Verify attribute value

• Verify clickable = verify that it exists, is visible, is enabled, and is unique.

• Verify Input = Set a value (from the test_data_key, for relevant types)

• Verify Click = Click on the thing

Or other tests as needed. Some instructions are implicit... Eg you don't need to test the default text on a label if it's expected value isn't in the verify data.

Control information can also contain flags such as "any error with this control is fatal", or "this control is optional/required, and so on.

By adding another test to the engine, all the existing controls that are eligible for the new test also get tested, provided they have all the data needed. 

Of course not all tests will fit into this pattern, but in our experience, most will.

Collections of controls then can simply iterate through all the controls, calling the engine for each one.

Example:

my_control = SWADLControl(   
selector="blahblah",
is_text="Please enter your first name",   
test_data_key="first_name",   
actual_value_key="actual_first_name",
validations={
VERIFY_EXIST: True,
VERIFY_VISIBLE: True,
VERIFY_ENABLED: True,
VERIFY_UNIQUE: True,
VERIFY_VALUE: "Please enter your first name",
VERIFY_PROPERTY: ("selenium", "first_name"),
VERIFY_INPUT: True,
},
)

This defines a control (in python). No flows or page methods will typically access this control other than via an inherited process_control(passed_dict) method.

In the example above, self refers to the page.

While this example is in python, it would even be possible to implement these descriptions as xml instead of code.

Engine:

The engine is simply a set of conditionals, like if control.validations["verify_visible"]==True then assert control.get_visible().

Some control classes will take data and some won't. Those if statements will prevent the wrong thing from happening with the wrong control class. For example, if it's a label type control, we won't go looking to see if there's a value to put in the control. 

For any special case control, we just replace the process_control() method with a custom one. 

Personal Experience:

At Home Depot, we implemented this using wrapper objects for the controls, where the wrapper carried the selector, but also flags to indicate what was to be tested and what the expected result was. Then we just ran a loop to validate all the controls, calling the engine. The engine then read the flags so it knew what to test.

When confronted with significant changes to the AUT, we nonetheless were able to turn around our fixes in a matter of moments.

Layers 101: Test Automation and the The Flow Design Pattern

Introduction

Of all the design patterns that contribute to a good test automation environment, the layers design pattern is the single most important.

The layered architecture pattern divides the components into a number of horizontal layers. These layers each know how to make requests of the lower layers, but not how the lower layer does it’s work (encapsulation). This is the traditional method for designing most software.

The central idea is to use structured divisions to break the code into pieces who each know how to ask other pieces to do things, but have no idea how it’s done. The internet is built this way. Your home router doesn’t know how the web servers do what they do, it just hands you the packets addressed to you. Layers above that in your computer don’t know how the packets get here, but does know how to reassemble them. This isolation of functionality is why you don’t have to replace your home router when a new kind of service, such as email or chat, is added.

What follows is a discussion with many examples of pseudocode. The same layers discussed below are applicable whether you’re using a code free or low code tool as they are for any of the coded tools.

Layers Tutorial and Example Use Case

Imagine we had thousands of tests. And that they look sorta like this:

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 Driver.OpenPage(globals.env_url) 6 Driver.WaitForControlVisible("user-name-selector", 20) 7 Driver.SendKeys("user-name-selector", EmailAddr) 8 Driver.SendKeys("password-selector", PassWord) 9 Driver.Click("login-button-selector") 10 assert Driver.GetVisible("hero-image-selector") 11 Driver.Click("my-account-selector") 12 assert Driver.GetVisible("my-account-change-email-button-selector") 13 Driver.Click("my-account-change-email-button-selector") 14 Driver.TypeKeys("my-account-email-box-selector", NewEmailAddr) 15 Driver.Click("my-account-save-button-selector")

We have a test which logs in, and changes the user’s email. We eventually hope to have thousands of tests like this.

Imagine if every test case has to log in, so they all start with that part of this code. Now imagine that the selector (strings which define how to find that control on the page) user-name-selector changes. That means we could have thousands of places to change it.

To solve this, we have to move the selectors out of the tests. To do this, we use the layers pattern to create a new place to put the selectors.

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 Driver.OpenPage(globals.env_url) 6 Driver.WaitForControlVisible(LoginPage.EmailAddrSelector, 20) 7 Driver.SendKeys(LoginPage.EmailAddrSelector, EmailAddr) 8 Driver.SendKeys(LoginPage.PassWordSelector, PassWord) 9 Driver.Click(LoginPage.LoginButton.Selector) 10 assert Driver.GetVisible(LoggedInPage.HeroImage.Selector) 11 Driver.Click(LoggedInPage.MyAccountSelector) 12 assert Driver.GetVisible(MyAccountPage.ChangeEmailButtonSelector) 13 Driver.Click(MyAccountPage.ChangeEmailButtonSelector) 14 Driver.TypeKeys(MyAccountPage.EmailBoxSelector, NewEmailAddr) 15 Driver.Click(MyAccountPage.SaveButtonSelector")

In this example, there are two layers: the test and the page. The Page has the selectors for the controls we’re going to manipulate. The test sets up the data, loads the page, uses the selectors to populate the fields, and click Login. Having a separate page object is called The Page Object Pattern.

In this case, if the selector changes, when we next run the tests, they will fail. But we only have to update the page, and all the tests will work once again.

In this example, the two layers are tightly coupled. So the test knows how the page is set up, and manipulates it to make things happen.

In general, having the tests tightly coupled means that any time I make a change to the page that’s greater than just changing a selector, I have to also change all the tests that do the same thing. We want Loosely Coupled components to make the code more maintainable.

This is however a login, and many tests will have to do that. So we turn it into functions, and now we can do this:

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 LoginPage.OpenLoginPage() 6 LoginPage.PopulateUser(EmailAddr) 7 LoginPage.PopulatePassword(PassWord) 8 LoginPage.Submit() 9 LoggedInPage.VerifyLoaded() 10 LoggedInPage.OpenMyAccount() 11 MyAccountPage.VerifyLoaded() 12 MyAccountPage.EnterEmail(NewEmailAddr) 13 MyAccountPage.Save()

No more pesky selectors in the test! We have created multiple methods (functions belonging to an object class) on the login page, which do part of the work for us. This is even less tightly coupled.

We can simplify this further, by moving the individual actions into a wrapper function which does all the login for us.

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 LoginPage.Login(EmailAddr, PassWord) 6 LoggedInPage.VerifyLoaded() 7 LoggedInPage.OpenMyAccount() 8 MyAccountPage.VerifyLoaded() 9 MyAccountPage.EditEmail(NewEmailAddr)

EditEmail now also performs the Save operation. Woo hoo! Much simpler! And if the login function changes radically, but is still on one page, we just update the one page, and all the tests now work again.

But life is seldom so simple, so now let’s make our login more complicated. Let’s say it has a second page, with a Two Factor Authentication (2FA).

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse" 6 LoginPage.Login(EmailAddr, PassWord) 7 TwoFAPage.Enter2FA(TwoFactorAuth) 8 LoggedInPage.VerifyLoaded() 9 LoggedInPage.OpenMyAccount() 10 MyAccountPage.VerifyLoaded() 11 MyAccountPage.EditEmail(NewEmailAddr)

Now every time we log in, and do 2FA, we have to call those in every test. And if that changes, we’re still back to updating all the tests.

The Flows Design Pattern

So to really make this robust, we need a way to get that UI specific functionality out of the test. So we make a new layer. This one will be called flows. (Some tools and practitioners use the term storyboarding.)

A flow is any action a user might do. So logging in is a user action, rather than an action specific to individual pages.

Let’s look at that:

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse" 6 LoginFlow.Login(EmailAddr, PassWord, TwoFactorAuth) 7 MyAccountFlow.UpdateEmail(NewEmailAddr)

This example is about as close as we can get to the test thinking about what’s to be done the way a human would. This example is clearly incomplete, so let’s show a more complete version of this example.

Development Example

We’ll tackle this in the order I usually do when developing a test. Your mileage may vary…

• We know we want to change the email address of somebody, so we make sure we have a somebody in the system.

• We also need an address to change to

• And the login password and 2FA

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse"

• And we’re going to need two flows, one which can log in, and one which can change the email.

1 LoginFlow.Login(EmailAddr, PassWord, TwoFactorAuth) 2 MyAccountFlow.UpdateEmail(NewEmailAddr)

• Yes, this does look just like the example above, but this is the order I do it in. The important point here is that right now, at this point in the process, I have a test. But it won’t run. LoginFlow and MyAccountFlow don’t yet exist. So that’s what’s next. I know I’m gonna need a login page object, a logged in page object, and a my account page... But I can specify them without them existing yet. (Some tools, such as Worksoft Certify, may require stubbing such routines until the new ones are ready.)

1Class LoginFlow 2 Function Login(EmailAddr, PassWord, TwoFactorAuth) 3 LoginPage.Login(EmailAddr, PassWord, TwoFactorAuth) 4 LoggedInPage.VerifyLoaded() 5 6Class MyAccountFlow 7 Function UpdateEmail(NewEmailAddr) 8 LoggedInPage.OpenMyAccount() 9 MyAccountPage.VerifyLoaded() 10 MyAccountPage.EnterEmail(NewEmailAddr) 11 MyAccountPage.Save()

Now we can build out this version of the page object.

1Class LoginPage 2 UserNameSelector = "user-name-selector" 3 PassWordSelector = "password-selector" 4 LoginButtonSelector = "login-button-selector" 5 6 function Login(self, EmailAddr, PassWord, TwoFactorAuth) 7 LoginPage.OpenLoginPage() 8 LoginPage.PopulateUser(EmailAddr) 9 LoginPage.PopulatePassword(PassWord) 10 LoginPage.Submit() 11 12 function OpenLoginPage() 13 Driver.OpenPage(globals.env_url) 14 Driver.WaitForControlVisible(UserNameSelector, 20) 15 16 Function PopulateUser(EmailAddr) 17 Driver.SendKeys(UserNameSelector, EmailAddr) 18 19 Function PopulatePassword(PassWord) 20 Driver.SendKeys(PassWordSelector, PassWord) 21 22 Function Submit() 23 Driver.Click(LoginButtonSelector)

Notice that this is all the same actual lines of code doing the work… But it’s broken up into many functions. We don’t actually NEED to break it up this much, login is a fairly simple process… But imagine that we may also need to test login failures in the future. So we can’t only have the one wrapper function. We might need to Submit(), and then verify an error message, for instance.

Now we can start debugging it. Because of course it won’t work first try.  

The Layers We Just Built

We’ve just built a stack with 3 layers:

• Tests – Tests know about data, and about which flow objects to ask to do it’s bidding. Tests know nothing at all about pages ever. Tests can talk to flows, and that’s it. Tests can also validate data that comes back to them from the flows.

• Flows – Flows know which pages do what, and what to ask them to do. But they don’t validate data. Flows can call pages and other flows as well. Flows can also manipulate data, but this should be kept to a minimum. A flow may need to reformat data from one page object call before passing it to a different page in a later step.

• Pages – Pages know about the controls on their page, and how to get it all to do what it does. Pages NEVER call other pages, flows or tests. Pages can pass data back to the flows.

(There’s a lot more than that which we’re leaving out, like the layer that is the browser, and the layer that is the testing tool, and so on.)