Monday, July 12, 2021

Layers 101: Test Automation and the The Flow Design Pattern

Introduction

Of all the design patterns that contribute to a good test automation environment, the layers design pattern is the single most important.

The layered architecture pattern divides the components into a number of horizontal layers. These layers each know how to make requests of the lower layers, but not how the lower layer does it’s work (encapsulation). This is the traditional method for designing most software.

The central idea is to use structured divisions to break the code into pieces who each know how to ask other pieces to do things, but have no idea how it’s done. The internet is built this way. Your home router doesn’t know how the web servers do what they do, it just hands you the packets addressed to you. Layers above that in your computer don’t know how the packets get here, but does know how to reassemble them. This isolation of functionality is why you don’t have to replace your home router when a new kind of service, such as email or chat, is added.

What follows is a discussion with many examples of pseudocode. The same layers discussed below are applicable whether you’re using a code free or low code tool as they are for any of the coded tools.

Layers Tutorial and Example Use Case

Imagine we had thousands of tests. And that they look sorta like this:

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 Driver.OpenPage(globals.env_url) 6 Driver.WaitForControlVisible("user-name-selector", 20) 7 Driver.SendKeys("user-name-selector", EmailAddr) 8 Driver.SendKeys("password-selector", PassWord) 9 Driver.Click("login-button-selector") 10 assert Driver.GetVisible("hero-image-selector") 11 Driver.Click("my-account-selector") 12 assert Driver.GetVisible("my-account-change-email-button-selector") 13 Driver.Click("my-account-change-email-button-selector") 14 Driver.TypeKeys("my-account-email-box-selector", NewEmailAddr) 15 Driver.Click("my-account-save-button-selector")

We have a test which logs in, and changes the user’s email. We eventually hope to have thousands of tests like this.

Imagine if every test case has to log in, so they all start with that part of this code. Now imagine that the selector (strings which define how to find that control on the page) user-name-selector changes. That means we could have thousands of places to change it.

To solve this, we have to move the selectors out of the tests. To do this, we use the layers pattern to create a new place to put the selectors.

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 Driver.OpenPage(globals.env_url) 6 Driver.WaitForControlVisible(LoginPage.EmailAddrSelector, 20) 7 Driver.SendKeys(LoginPage.EmailAddrSelector, EmailAddr) 8 Driver.SendKeys(LoginPage.PassWordSelector, PassWord) 9 Driver.Click(LoginPage.LoginButton.Selector) 10 assert Driver.GetVisible(LoggedInPage.HeroImage.Selector) 11 Driver.Click(LoggedInPage.MyAccountSelector) 12 assert Driver.GetVisible(MyAccountPage.ChangeEmailButtonSelector) 13 Driver.Click(MyAccountPage.ChangeEmailButtonSelector) 14 Driver.TypeKeys(MyAccountPage.EmailBoxSelector, NewEmailAddr) 15 Driver.Click(MyAccountPage.SaveButtonSelector")

In this example, there are two layers: the test and the page. The Page has the selectors for the controls we’re going to manipulate. The test sets up the data, loads the page, uses the selectors to populate the fields, and click Login. Having a separate page object is called The Page Object Pattern.

In this case, if the selector changes, when we next run the tests, they will fail. But we only have to update the page, and all the tests will work once again.

In this example, the two layers are tightly coupled. So the test knows how the page is set up, and manipulates it to make things happen.

In general, having the tests tightly coupled means that any time I make a change to the page that’s greater than just changing a selector, I have to also change all the tests that do the same thing. We want Loosely Coupled components to make the code more maintainable.

This is however a login, and many tests will have to do that. So we turn it into functions, and now we can do this:

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 LoginPage.OpenLoginPage() 6 LoginPage.PopulateUser(EmailAddr) 7 LoginPage.PopulatePassword(PassWord) 8 LoginPage.Submit() 9 LoggedInPage.VerifyLoaded() 10 LoggedInPage.OpenMyAccount() 11 MyAccountPage.VerifyLoaded() 12 MyAccountPage.EnterEmail(NewEmailAddr) 13 MyAccountPage.Save()

No more pesky selectors in the test! We have created multiple methods (functions belonging to an object class) on the login page, which do part of the work for us. This is even less tightly coupled.

We can simplify this further, by moving the individual actions into a wrapper function which does all the login for us.

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 LoginPage.Login(EmailAddr, PassWord) 6 LoggedInPage.VerifyLoaded() 7 LoggedInPage.OpenMyAccount() 8 MyAccountPage.VerifyLoaded() 9 MyAccountPage.EditEmail(NewEmailAddr)

EditEmail now also performs the Save operation. Woo hoo! Much simpler! And if the login function changes radically, but is still on one page, we just update the one page, and all the tests now work again.

But life is seldom so simple, so now let’s make our login more complicated. Let’s say it has a second page, with a Two Factor Authentication (2FA).

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse" 6 LoginPage.Login(EmailAddr, PassWord) 7 TwoFAPage.Enter2FA(TwoFactorAuth) 8 LoggedInPage.VerifyLoaded() 9 LoggedInPage.OpenMyAccount() 10 MyAccountPage.VerifyLoaded() 11 MyAccountPage.EditEmail(NewEmailAddr)

Now every time we log in, and do 2FA, we have to call those in every test. And if that changes, we’re still back to updating all the tests.

The Flows Design Pattern

So to really make this robust, we need a way to get that UI specific functionality out of the test. So we make a new layer. This one will be called flows. (Some tools and practitioners use the term storyboarding.)

A flow is any action a user might do. So logging in is a user action, rather than an action specific to individual pages.

Let’s look at that:

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse" 6 LoginFlow.Login(EmailAddr, PassWord, TwoFactorAuth) 7 MyAccountFlow.UpdateEmail(NewEmailAddr)

This example is about as close as we can get to the test thinking about what’s to be done the way a human would. This example is clearly incomplete, so let’s show a more complete version of this example.

Development Example

We’ll tackle this in the order I usually do when developing a test. Your mileage may vary…

  • We know we want to change the email address of somebody, so we make sure we have a somebody in the system.

  • We also need an address to change to

  • And the login password and 2FA

1function Test1() 2 EmailAddr = "BobSmith@foo.com" 3 NewEmailAddr = "BobSmith@bar.com" 4 PassWord = "secret" 5 TwoFactorAuth = "somethingelse"
  • And we’re going to need two flows, one which can log in, and one which can change the email.

1 LoginFlow.Login(EmailAddr, PassWord, TwoFactorAuth) 2 MyAccountFlow.UpdateEmail(NewEmailAddr)
  • Yes, this does look just like the example above, but this is the order I do it in. The important point here is that right now, at this point in the process, I have a test. But it won’t run. LoginFlow and MyAccountFlow don’t yet exist. So that’s what’s next. I know I’m gonna need a login page object, a logged in page object, and a my account page... But I can specify them without them existing yet. (Some tools, such as Worksoft Certify, may require stubbing such routines until the new ones are ready.)

1Class LoginFlow 2 Function Login(EmailAddr, PassWord, TwoFactorAuth) 3 LoginPage.Login(EmailAddr, PassWord, TwoFactorAuth) 4 LoggedInPage.VerifyLoaded() 5 6Class MyAccountFlow 7 Function UpdateEmail(NewEmailAddr) 8 LoggedInPage.OpenMyAccount() 9 MyAccountPage.VerifyLoaded() 10 MyAccountPage.EnterEmail(NewEmailAddr) 11 MyAccountPage.Save()

Now we can build out this version of the page object.

1Class LoginPage 2 UserNameSelector = "user-name-selector" 3 PassWordSelector = "password-selector" 4 LoginButtonSelector = "login-button-selector" 5 6 function Login(self, EmailAddr, PassWord, TwoFactorAuth) 7 LoginPage.OpenLoginPage() 8 LoginPage.PopulateUser(EmailAddr) 9 LoginPage.PopulatePassword(PassWord) 10 LoginPage.Submit() 11 12 function OpenLoginPage() 13 Driver.OpenPage(globals.env_url) 14 Driver.WaitForControlVisible(UserNameSelector, 20) 15 16 Function PopulateUser(EmailAddr) 17 Driver.SendKeys(UserNameSelector, EmailAddr) 18 19 Function PopulatePassword(PassWord) 20 Driver.SendKeys(PassWordSelector, PassWord) 21 22 Function Submit() 23 Driver.Click(LoginButtonSelector)

Notice that this is all the same actual lines of code doing the work… But it’s broken up into many functions. We don’t actually NEED to break it up this much, login is a fairly simple process… But imagine that we may also need to test login failures in the future. So we can’t only have the one wrapper function. We might need to Submit(), and then verify an error message, for instance.

Now we can start debugging it. Because of course it won’t work first try.  

The Layers We Just Built

We’ve just built a stack with 3 layers:

  • Tests – Tests know about data, and about which flow objects to ask to do it’s bidding. Tests know nothing at all about pages ever. Tests can talk to flows, and that’s it. Tests can also validate data that comes back to them from the flows.

  • Flows – Flows know which pages do what, and what to ask them to do. But they don’t validate data. Flows can call pages and other flows as well. Flows can also manipulate data, but this should be kept to a minimum. A flow may need to reformat data from one page object call before passing it to a different page in a later step.

  • Pages – Pages know about the controls on their page, and how to get it all to do what it does. Pages NEVER call other pages, flows or tests. Pages can pass data back to the flows.



(There’s a lot more than that which we’re leaving out, like the layer that is the browser, and the layer that is the testing tool, and so on.)

Friday, July 2, 2021

SQA: Our Tasking

  • We have to produce and maintain more tests than is physically possible. Obviously this is not an attainable goal, so our attainable goal is simply to automate as much as possible with the time we have, with sensitivity to both the business and dev priorities, and to do that consistently (which means not killing ourselves in the process).
  • What is our remit if our role is Quality Assurance Engineer?

    • Define tests which will prove the positive cases defined in the requirements/user stories/etc. (Getting the correct error message on invalid input is still a positive case for the purposes of this discussion.)

      • Absent that, we make wild guesses at what the System Under Test is supposed to do, and hold it to that until told better.

    • Document those tests (and advocate for suitable tools if they are not available, and no, Excel doesn’t count)

    • Execute those tests and deliver bugs for requirements not met

    • Think of the things the developer didn’t

    • User advocate

  • What is our remit if our role is Automation Engineer?

    • Deliver tests which will prove the positives

    • Implement those tests in a way where maintenance can be minimized and stability maximized

    • Document the code units that implement those tests such that the whole team can use them

    • We report on defects found in automation AND FLAG THEM AS FOUND IN AUTOMATION.

  • What is our remit if our role is Automation Architect?

    • Consistently deliver build go/no go decisions

    • Define the standards by which the tests and their execution environment(s) will be built and maintained such that:

      • We maximize reuse

      • We maximize stability and uptime

      • We maximize the access to information

      • We minimize the time to bring new people up to speed

      • We minimize the time required to identify and isolate issues

      • We minimize the time required to look stuff up during test development and debugging

    • We document, design, implement, delegate, train, validate, source expertise and otherwise do what it takes to make sure that our team can do all of the above

      • In particular, we want to be training our replacement(s) from day one. We are responsible for assuring success even if we’re no longer available.

    • Communicate with other teams about our needs, accomplishments, test results, RPA, automation training, and anything else we can provide.

Monday, December 21, 2020

SDET vs Architect?

 I had somebody asked me the other day what the difference was between a test automation architect and an SDET.

An SDET is a software developer in test. Software developer skillset applied to testing. I'm glad this now has a name and certification. When I started, there wasn't such.

An Architect is any developer who's got enough miles under their belt that they look at everything from a the perspective of how it all fits together and interacts with everything outside of itself.

Can an SDET be an architect? Absolutely. Are all SDETs experientially equipped to approach their work as Architects? No more so than all developers are architects.

I started working with computers in 1979, I got my first job in computers in 1982, got my first support gig in 1987. my first QA gig in 1990. First automation assignment in dec 1990. 

A year later, and I was overwhelmed by the maintenance. It was a month after I realized that fact that I started thinking about test automation from the perspective of architecture.

My own interest came first from trying to solve my own problems, and then from those my management asked of me. How can I reuse most of what I produce? How can I reduce maintenance? Can you build out a lab so we can run these over and over? I want you to make it so everybody in QA can create automated tests, even the non programmers. We have too much data left over from all those old builds and test runs. 

I've solved every problem thrown at me. I'm a good tester, but I'm a much better architect.

Saturday, June 20, 2020

Selenium As A Modern Automation Tool

2020 Akien Maciain, Paul Castillo, Alejandro Garcia

Introduction 

This document covers a deployment of Selenium using Python for a large retail firm. It was started in 2015, and as of this writing is still in use today. Here we provide an overview of key features. A companion file includes a functional mockup in Python 3, which is not taken from the firm's code.

The systems we were testing were customer and retail associate facing pages responsible for connecting consumers with home service professionals who would perform services such as house painting, plumbing installation and repair, and so on.

Terms

  • Control - A proxy object that represents a control on the screen.
  • Element - A DOM object on the screen that a "control" interacted with.
  • Golden Path - A flow that a user might perform that is without errors, backtracking, or other tangents.
  • SUT - System Under Test

Primary features

  • Fully Automatic Self Synchronization for all control interaction, with interfaces loosely based on tools like UFT and SilkTest. Zero sync failures.
  • Storyboarding is implemented as instances of a"flow class", an object layer between the tests and page objects. These abstract a flow a user might do. Adding this layer made the tests far more resilient.
  • Tagged Interfaces that prevented the need for maintenance in most cases. This required buy in from development, but became the channel that developers used to communicate UI changes to QA. Almost as good as AI-based self healing.
  • Engine based testing that validates all controls whether the automation "engages with them" or not.
  • Fully automated self setup for all environments, including a Docker container to actually run the tests to reduce the differences between individual computers.

Fully Automatic Self Synchronization

The framework as implemented did not have issues with synchronizing with the SUT. This was accomplished by implementing the controls as objects, with instances on the page objects for each control. When the controls are instantiated, the selector is one of the passed arguments.

The system performs automatic retries until a desired condition is met or timeouts are reached on all interactions with the page elements. This prevents stale element exception failures and waiting even one millisecond longer than is needed.

Asserting states before actions (eg asserting that a page is loaded before trying to interact with it) means the test only stops when it can't continue. Other failures that don't block test completion are logged as the test continues.

Storyboarding via Flow Layer

A flow is any activity a user might do. It often spans more than one page.

The test knows which flows to call but does not know anything about pages or controls. This decoupling of tests and pages means even in the face of a broad revision to all the page code the tests are unchanged. Pages are rebuilt and flows updated to point to the new pages. As long as the underlying application features still do basically the same things, the tests are unaffected.

In our model, tests are responsible for data, which is passed into flows as a hash table. Flows knew which page object and methods to use and passed the dictionary along. Pages implement page-specific subflows as methods and which controls handle what actions.

Strict encapsulation meant very low maintenance even in the face of big change.

Tagged Interfaces

In this framework, we could specify control selectors in css at instantiation time. Usually those css tags were not native products of the front end framework, but rather "selenium tags", like so:

<h3 selenium="tile-name" class="TILE-NAME">Bath</h3>

It took some convincing some developers to add these tags. We'd create tasks for them to implement these tags, and then we'd instantiate a control like this:

tile_header = Control(selector='[selenium="tile-name"]')

When the dev team rearranged a page, they would move the tags to the new controls. If they failed to do so, we'd write a blocking bug, and they'd get it fixed straight away.

Eventually, Dev trained the Quality Assurance teams to make these changes to the front end code ourselves.

Tagged interfaces are theoretically not as robust as Artificial Intelligence "self healing" features of many of the tools now on the market. However, on the ground, this proved entirely adequate.

Engine Based Testing

In the normal course of automation, we would touch only specific controls. For instance, in a login window, we would touch the username, password, and submit button. This limits the coverage of the automation to what we called "key controls".

However, a page is made up of much more than the key controls. Many have common objects such as headers, footers, hero images, title text, and so on. While not critical to the golden path, these objects can all be extremely important to validate.

To deal with this, we implemented an engine. When we instantiated the control objects, we would also add attributes that would tell the engine how to test the control. So if a control should exist, it might have a VERIFY_EXIST=True as one of it's instantiation arguments. We could verify existence or not, visible or not, enabled or not, whether a control was unique, what it's expected text value was, and so on.

We also implemented KEY as one of our instantiation keywords, and the KEY pointed to a keyed value in the test data dictionary. So we could deal with dynamic values as well.
A flow would simply call the engine before doing anything else on the page. For some golden path tests we could even populate fields like username and password using the KEY value, and not even have to write a subflow for that page.

Self Setup

One of the features that was mentioned repeatedly with appreciation by the team was the fully automated setup. A person could download a script from the knowledge base, run it locally, and it would set up everything about the automation. All system packages, download all the needed repos from github, set up their docker container… Everything. This was implemented in Bash, but today we'd use something like Ansible.

We used Docker to provide a single, consistent environment for the testing tools and execution environment, which meant that the specifics of how individual computers didn't impact the testing.

Conclusion

After having conducted a thorough high level overview of many of the tools on the market, we were amused to see how many claimed these features could not be implemented in Selenium and expensive tools were instead required in order to make maintainable tests. This framework proves that assertion wrong. However, it does require building out the control object, getting dev buy-in for tags, and implementation of an Ansible (or similar) setup script. None of this is beyond the capabilities of a competent engineer.

Tuesday, February 5, 2019

Automation Pain Points II: Resilience Part 3, Selectors

In the last two sections, we looked at strategies that orchestrate the behavior of pages and flows. These have made a huge difference in our day to day work. The third piece requires buy in from dev.

We kept rebuilding page objects, but sorting the new selectors was kind of a nightmare. Just when we thought it was all done, changed again!

We realized that the developer still knows where the first name field is... Or the submit button... Or whatever.

So one of our engineers made this case to dev: We will add selectors to our pages as we normally do. We will also open a ticket to have new properties added to each control. The property name will be 'selenium', so we can have selectors like '[selenium="first-name"]'.

The engineer is responsible for adding those tags. And when the page is changed, they're the ones changing it, and they are required to keep the selenium properties intact as they redesign the page. If a selenium tag disappears, it's a P0 bug, because it blocks testing.

They are not required to put selenium tags on anything we didn't identify as needing them.

This took us from a couple of days before everything was smooth again to a few minutes. Huge benefit!

Thursday, November 29, 2018

Automation Pain Points II: Resilience Part 2, Flows

In the last piece, we looked at the benefits I've seen at other companies using strict encapsulation at the page level. But in my current gig, we take it a step farther. We have another layer between tests and pages called flows. A flow interacts with one or more page objects.

This means that pages are the workers, they know how to do specific pieces of work.

Flows are the managers, they line up which workers and in what order. Flows are also a series of interactions the user will take to perform some action. They model the end user's interaction with the system.

The tests are the executives that decide which flows to call to test a specific underlying feature or group of features.

In practice, this looks like a lot of unnecessary complexity, especially for test cases who's flows are one line long. But it also further isolates the test from the changes in the UI.

We've had large scale redesigns that mostly take page object updates. Sometimes flows need to be updated. But the underlying functionality hasn't really changed.

This means tests are updated more quickly, and so the automation can do more for less effort.

It also means we can write automated tests and flows before we have page mockups, as long as we have detailed enough specs. We can add the page objects as they become available.

Woo hoo!

Saturday, November 3, 2018

Automation Pain Points II: Resilience Part 1, The Page

I am a fundamentally lazy guy. That's why I write test automation, because I don't want to keep doing the same crap over and over.

But the problem with encoding knowledge into source code is that the representation of knowing thus created is brittle. Every time the AUT changes, the tests have to change.

I saw this first at Symantec in 1991, working on OnTarget, a project management package. There were 3 of us automators, and we built up a rather small lot of tests, before we became unable to create any more. We were too busy keeping up with AUT changes.

By this time, we were using SilkTest, and their implementation of page objects. Their page objects were really just intended to be a better tool for managing selectors than the huge lists of constants that was the way QAWorkbench (which became SilkTest) had done in the alpha stage of it's development.

But we took it a step further, and built what folks today would recognize as page objects. The point behind the way we use page objects at Home Depot today is to encapsulate all the details of interacting with a page into one place.

When I started there, we had tests directly calling methods in page objects. So when they completely redesigned the app, all the tests had to be thrown out. This is not good resilience.

The problem was that the test code was tightly coupled to the page code. The rule when I arrived was that no test used the selectors on a page. Instead, methods were created. I understand why they created that rule, but the problem was they had routines like click_submit_button(), which meant that how to operate the page was encoded in the test. The AUT underwent a complete redesign, and all the tests were then trash, and had to be rebuilt almost completely from scratch.

Today, each page object implements subflows. A subflow takes a dictionary with all the data needed for that page, and implements all the steps for a single action on that page. Everything about how the page does it's work is encapsulated and isolated from the test. Even default values we expect controls to have, such as the text of error messages that should appear, are stored in the page object itself.