i was watching crash course computer science on youtube. it's been kind of amusing, i have learned a few things, but there just isn't much i didn't already know... with the episodes shown so far, anyway.
but i ran across the example below, which was illustrating the 'else' clause:
if x then
print a
else
print b
and i had this weird 'wait a minute, i often don't do that in that way!'... at first, i briefly had a judgement that my way was better. when i asked myself why i thought this way was better, my cpu came back with a shrug.
this is the kind of structure i have historically often used:
r = a
if x then
r = b
print r
and as i sat there scratching my head, i realized that long ago, msdos 2.1 batch files didn't support else. in fact, the modern cmd.exe command set does support else, though i wasn't able to track down when that was added (or if it had been there all along and just poorly documented).
my approach requires extra memory. when you put all the commands on the same line, mine is slightly shorter. i don't think either of these differences qualifies as better. and i guess one of the marks of a good engineer is one that doesn't hold onto the old way of doing it just because that's what they learned. yay for having dispelled an obsolete assumption!
Tuesday, June 6, 2017
Tuesday, January 3, 2017
Libraries and API design
Some time back, I ran across this: http://www.softwaretestinghelp.com/test-automation-frameworks-selenium-tutorial-20/
And I was taken aback at the description of the framework, as he was ignoring the parts of framework development that seemed to me to be the most important parts.
Consider, if we use just Selenium and write our tests to it, then everybody we hire who knows selenium (and our target language) will be able to write automated tests. We will probably set up page objects, and that's about the minimal set of things to know. Right?
Well, yes, but...
From my experience, many QA organizations would prefer to be able to graduate successful testers into at least some automation. This often means training less experienced testers to write automation.
And if your framework is more complex than the above (and most of them are), then there's going to be some kind of a learning curve, even for more experienced automators.
My job as an architect is to 'design and maintain the framework'. And how well I do that design part will directly influence how productive the people who use that framework will be.
At my first automation gig, there were 3 of us, and we all were pretty good programmers, so we just started banging out tests. We had a set of shared page objects, and wrote any other functions that helped us out as we needed them.
This meant that, at different times, we each wrote something to remove all non numeric characters from a string. Because we happened to need that, and because there was no orchestration of our work. No good standards for code sharing, no limitations on what we could create. Doh!
At one point, I even ran into a test I'd written a month earlier that implemented some function or other (I no longer remember what) that I'd just implemented over again because I'd forgotten that it existed. My own code!
This was when I learned about interface proliferation. The tendency of good programmers to try and reuse code had led to more interface endpoints than we could keep track of.
From then on, I started looking at test automation interface design as it's own discipline. And not only the design of the interfaces, but how I communicate about them as well. What documentation I create and where.
These two pages have strongly influenced my API designs:
http://martinfowler.com/bliki/HumaneInterface.html
http://martinfowler.com/bliki/MinimalInterface.html
Now, I try to create minimal interfaces, and the interfaces I do create require the bare minimum of data to do the most common action. So for instance, when I started in my current gig, they had created a function called 'does_control_exist(selector)' to wrap the fact that selenium has no exist function. They also had a 'wait_for_control_to_exist(selector, time)'. Two interfaces to determine existence. What if it was just .exist([time])? The [time] is in brackets because it's optional. Two calls compressed into one.
It's also the case that when I used our framework's click(), I'd often get 'control not found' if the code tried to click on something that existed more than once. And there was nothing to tell me if a control was unique or not.
Now I have .click([time]), so a call might look like thingie.click(), which does the following:
since no timeout was provided, use 20 seconds
get the element for the selector
assert that the item exists
assert that the item is unique
assert that the item is visible
assert that the item is enabled
element.click()
And that happens each time we click on something. This is how most test automation tools worked before selenium.
So my page object may look something like this:
class PageHome(MyPageClass):
complete_page_button = ButtonClass("css=[id='Next-Button']")
def complete_page(self):
if self. complete_page_button.exist() is False:
self.invoke_page()
self.complete_page_button.click()
...
So for exist, we now have one function instead of 2. For click, we've collapsed at least 6 actions into one function. This means my test automation engineers have fewer end points to remember, and for the most common use of an endpoint, no additional parameters are required. I can get new people up to speed faster, and we can debug failed tests more quickly because we get more useful feedback in the logs.
For me, this is what framework design is all about.
And I was taken aback at the description of the framework, as he was ignoring the parts of framework development that seemed to me to be the most important parts.
Consider, if we use just Selenium and write our tests to it, then everybody we hire who knows selenium (and our target language) will be able to write automated tests. We will probably set up page objects, and that's about the minimal set of things to know. Right?
Well, yes, but...
From my experience, many QA organizations would prefer to be able to graduate successful testers into at least some automation. This often means training less experienced testers to write automation.
And if your framework is more complex than the above (and most of them are), then there's going to be some kind of a learning curve, even for more experienced automators.
My job as an architect is to 'design and maintain the framework'. And how well I do that design part will directly influence how productive the people who use that framework will be.
At my first automation gig, there were 3 of us, and we all were pretty good programmers, so we just started banging out tests. We had a set of shared page objects, and wrote any other functions that helped us out as we needed them.
This meant that, at different times, we each wrote something to remove all non numeric characters from a string. Because we happened to need that, and because there was no orchestration of our work. No good standards for code sharing, no limitations on what we could create. Doh!
At one point, I even ran into a test I'd written a month earlier that implemented some function or other (I no longer remember what) that I'd just implemented over again because I'd forgotten that it existed. My own code!
This was when I learned about interface proliferation. The tendency of good programmers to try and reuse code had led to more interface endpoints than we could keep track of.
From then on, I started looking at test automation interface design as it's own discipline. And not only the design of the interfaces, but how I communicate about them as well. What documentation I create and where.
These two pages have strongly influenced my API designs:
http://martinfowler.com/bliki/HumaneInterface.html
http://martinfowler.com/bliki/MinimalInterface.html
Now, I try to create minimal interfaces, and the interfaces I do create require the bare minimum of data to do the most common action. So for instance, when I started in my current gig, they had created a function called 'does_control_exist(selector)' to wrap the fact that selenium has no exist function. They also had a 'wait_for_control_to_exist(selector, time)'. Two interfaces to determine existence. What if it was just .exist([time])? The [time] is in brackets because it's optional. Two calls compressed into one.
It's also the case that when I used our framework's click(), I'd often get 'control not found' if the code tried to click on something that existed more than once. And there was nothing to tell me if a control was unique or not.
Now I have .click([time]), so a call might look like thingie.click(), which does the following:
since no timeout was provided, use 20 seconds
get the element for the selector
assert that the item exists
assert that the item is unique
assert that the item is visible
assert that the item is enabled
element.click()
And that happens each time we click on something. This is how most test automation tools worked before selenium.
So my page object may look something like this:
class PageHome(MyPageClass):
complete_page_button = ButtonClass("css=[id='Next-Button']")
def complete_page(self):
if self. complete_page_button.exist() is False:
self.invoke_page()
self.complete_page_button.click()
...
So for exist, we now have one function instead of 2. For click, we've collapsed at least 6 actions into one function. This means my test automation engineers have fewer end points to remember, and for the most common use of an endpoint, no additional parameters are required. I can get new people up to speed faster, and we can debug failed tests more quickly because we get more useful feedback in the logs.
For me, this is what framework design is all about.
Wednesday, October 12, 2016
Python and how to get the instance name of a variable
I needed the name of an instance in order to do some error reporting. I'm very lucky that each of these things I need to report on will only have one instance.
So of course, first thing I did was ask Google. And after a couple of weeks of checking now and again, I have been unable to find any solutions already published. I found a lot of "instances don't have names!", which clearly isn't true, but makes sense from the perspective that the instance name is a pointer to the thing, and there's no backwards pointing property in the thing being pointed at.
But as with most things, there's always a way. And it turns out that in python, it's actually pretty straightforward:
Returns a list of all matching instance names. Now it's possible to create an instance without a name, such as via a generator or a lambda, and I haven't tested what it will return in those cases, as that wasn't what I needed this for. It also fails for getting the name of a function, but that can gotten in other ways.
I hope this helps someone! :)
So of course, first thing I did was ask Google. And after a couple of weeks of checking now and again, I have been unable to find any solutions already published. I found a lot of "instances don't have names!", which clearly isn't true, but makes sense from the perspective that the instance name is a pointer to the thing, and there's no backwards pointing property in the thing being pointed at.
But as with most things, there's always a way. And it turns out that in python, it's actually pretty straightforward:
import gcdef instance_names(self):referrers = gc.get_referrers(self) result = [] dict_of_things = {} for item in referrers: if isinstance(item, dict): dict_of_things = item for k, v in dict_of_things.items(): if v == self: result.append(k) if not result: result = ['unnamed instance'] return result
Returns a list of all matching instance names. Now it's possible to create an instance without a name, such as via a generator or a lambda, and I haven't tested what it will return in those cases, as that wasn't what I needed this for. It also fails for getting the name of a function, but that can gotten in other ways.
I hope this helps someone! :)
Python, Selenium and the dreaded "Timed out receiving message from renderer"
We had a problem, around 15% of the time, we were getting tests that failed, and when we went to SauceLabs to look at the problem, we just had a blank browser page with "data;" in the address field.
Folks here had been ignoring this and had just pronounced the automated tests to be flaky and unreliable. Having the experience I have, this was galling.
I looked in the selenium output, and saw "Timed out receiving message from renderer".
So I looked on Google, and found lots of folks reported having this problem. Several bugs have been written, all closed with "can't duplicate". This issue is at least 4 years old as of this writing. I tried changing timeouts, using try/except, and every other thing listed, but improvements were either nonexistent or very modest.
I have solved it, but the solution is an *awful* hack. The one thing it has going for it, our failures have dropped from 15% to 0. (Testing done using a suite with 10,000 cases in it.)
Here's the code at the center of the solution:
In the above, page is my page object. The method loaded() checks controls to see if the page has finished loading. And click_element_by_text fetches matching elements and iterates through them to determine whether they have the text specified. If an element does, it clicks it. (Sorry I can't include that code, but it belongs to work, and would make this sample way too long.)
In my experiments, it came to look like the integration between the driver and the browser (at least on Chrome) creates this state where Chrome has failed to load, but never tells the driver about it. So the driver just eventually times out.
about://blank - I used this because it should render an internally generated page every time. On Chrome, it's a "This site can't be reached" error, which works just fine. Firefox and IE also generate errors or blank pages. But the assumption was that internally stored pages should load every time. And so far, they do.
So by adding the target link and clicking it, I'm bypassing that tight integration.
I've watched the code run, and I've seen it had to retry once in a while, but so far, never more than once.
Please note this has only been tested on Chrome.
I hope somebody finds this helpful! :)
Folks here had been ignoring this and had just pronounced the automated tests to be flaky and unreliable. Having the experience I have, this was galling.
I looked in the selenium output, and saw "Timed out receiving message from renderer".
So I looked on Google, and found lots of folks reported having this problem. Several bugs have been written, all closed with "can't duplicate". This issue is at least 4 years old as of this writing. I tried changing timeouts, using try/except, and every other thing listed, but improvements were either nonexistent or very modest.
I have solved it, but the solution is an *awful* hack. The one thing it has going for it, our failures have dropped from 15% to 0. (Testing done using a suite with 10,000 cases in it.)
Here's the code at the center of the solution:
webdriver.get('about://blank') my_script = 'var a = document.createElement("a");' \ 'var linkText = document.createTextNode("%s");' \ 'a.appendChild(linkText);' \ 'a.title = "%s";' \ 'a.href = "%s";' \ 'document.body.appendChild(a);' % \ (url_to_use, url_to_use, url_to_use) webdriver.execute_script(my_script) webdriver.set_page_load_timeout(20) webdriver.click_element_by_text('css=a', url_to_use) if page.loaded() is False: webdriver.click_element_by_text('css=a', url_to_use) if page.loaded() is False: webdriver.click_element_by_text('css=a', url_to_use) if page.loaded() is False: webdriver.click_element_by_text('css=a', url_to_use)
In the above, page is my page object. The method loaded() checks controls to see if the page has finished loading. And click_element_by_text fetches matching elements and iterates through them to determine whether they have the text specified. If an element does, it clicks it. (Sorry I can't include that code, but it belongs to work, and would make this sample way too long.)
In my experiments, it came to look like the integration between the driver and the browser (at least on Chrome) creates this state where Chrome has failed to load, but never tells the driver about it. So the driver just eventually times out.
about://blank - I used this because it should render an internally generated page every time. On Chrome, it's a "This site can't be reached" error, which works just fine. Firefox and IE also generate errors or blank pages. But the assumption was that internally stored pages should load every time. And so far, they do.
So by adding the target link and clicking it, I'm bypassing that tight integration.
I've watched the code run, and I've seen it had to retry once in a while, but so far, never more than once.
Please note this has only been tested on Chrome.
I hope somebody finds this helpful! :)
Tuesday, May 31, 2016
The Nature of Frameworks
In my experience, when folks talk about test automation frameworks today, they're talking only about the libraries that you use to isolate things like selectors from the tests. And that is surely an important topic, one I'll address at some future time.
But the 'automation framework' means a lot more than that. Back in the day, we had to think about:
But the 'automation framework' means a lot more than that. Back in the day, we had to think about:
- Where the source was
- How we got it built, including all variants
- Branching including which tests against which branches
- Strategic prioritization of test implementation (eg, which tests are most important to implement first; what combinatorics; what mix of api, ui, performance and stress testing; etc)
- Automated test management
- Automated test data management (eg, preset database images to test against, data for data set tests)
- Coding standards
- Training of new staff
- Design reviews
- Code reviews
- Portability, internationalization and localization
- Automated test execution
- Automated bug reporting
- Artifact aging and control
- Hardware resource allocation and maintenance
- Crisis management
- And QA Evangelizing
These days, it's a lot simpler, but it's still more than just libraries. In my current gig:
- Where the source was -- Github
- How we got it built, including all variants -- Jenkins
- Branching including which tests against which branches -- Still managing this by hand
- Strategic prioritization of test implementation -- Still managing this by hand
- Automated test management -- Nose and Python's unittest library
- Automated test data management -- fixtures help, but mostly Still managing this by hand
- Coding standards -- PEP8 + Our own standards
- Training for new staff -- Still managing this by hand
- Design reviews -- Still managing this by hand
- Code reviews -- Github
- Portability, internationalization and localization -- Honestly, I haven't run into that in my current gig
- Automated test execution -- Nose and Jenkins
- Automated bug reporting -- Jenkins
- Artifact aging and control -- Still managing this by hand
- Hardware resource allocation and maintenance -- SauceLabs and various cloud services like Amazon or Google.
- Crisis management -- Still managing this by hand
- And QA Evangelizing -- I have to admit, at my current gig, this hasn't been an issue.
While there are now systems to handle these things, they still need to be orchestrated. And as a Test Automation Architect, seeing to those systems and how they serve the delivery of useful results is still in my purview. But it sure is nice not to have to build all of it by hand.
This note has been percolating in my head for some time. It's in response to this: http://www.softwaretestinghelp.com/test-automation-frameworks-selenium-tutorial-20/
Wednesday, May 25, 2016
Rules and Whys
When I started, I had to invent everything. It was 1990, and I didn't even have Google to ask.
We discovered early on that some things needed to be *rules*. Like only page objects can talk to their controls. A hard and fast rule. And that one, with good reason. By enforcing encapsulation, maintenance is made far easier.
When I started in my current gig, I was the most junior guy on the team, and the most junior with Selenium and Git... But the most experienced automation engineer overall by a factor of 5.
Shortly after I started here, I had a code review with Clay Gould, our automation lead. In this code review, he got quite concerned about my use of a delay of a tenth of a second in the code. The rule he quoted amounted to 'never EVER use delays to synchronize, ALWAYS look for changes in the application or environment'. A good rule. One I've pounded into many new automators.
When I teach and use these kinds of rules, I find it important to keep in mind why a rule exists. With time delays in code, the problem is use of the delay *for synchronization*. Because systems respond at different speeds at different times. Synchronization means attempting to line up processes. Like waiting for a page to be done loading before trying to take actions on it.
I was not using the delay for synchronization, I was using it to assist in synchronization. I checked the state of something, paused for a 1/10th of a second, and checked the state again. I kept at this until a timeout was met.
We argued about this for some time, and finally went looking at the Selenium code directly. Lo and behold, their code has a 3/10ths of a second delay in it in exactly the same fashion. It's to give over processor cycles to the other processes so they can complete more quickly... Rather than tying up cpu to test the condition over and over until it's met.
More recently, I was implementing a selenium wrapper, and I wanted to spread out a delay across multiple functions. I wanted to validate that the control was present, visible, and enabled... And I wanted only a 20 second timeout overall. Which means exist can take a long time, and complete the other two quickly. Selenium has no native support for anything like this. I either specify different hard timeouts for each one, or I allow them to all add together, and come out with more than 20 seconds.
The answer was to note my target end time (now + timeout) and pass the time remaining to each of the functions. And with a 1/10th of a second in between again.
Rules are important. Rules are present for a reason. That reason may not be applicable in all circumstances. Experience often gives us the whys behind the rules, so we can assess when it's safe to ignore them. And in my experience, there are very few rules in test automation that don't have exceptions.
We discovered early on that some things needed to be *rules*. Like only page objects can talk to their controls. A hard and fast rule. And that one, with good reason. By enforcing encapsulation, maintenance is made far easier.
When I started in my current gig, I was the most junior guy on the team, and the most junior with Selenium and Git... But the most experienced automation engineer overall by a factor of 5.
Shortly after I started here, I had a code review with Clay Gould, our automation lead. In this code review, he got quite concerned about my use of a delay of a tenth of a second in the code. The rule he quoted amounted to 'never EVER use delays to synchronize, ALWAYS look for changes in the application or environment'. A good rule. One I've pounded into many new automators.
When I teach and use these kinds of rules, I find it important to keep in mind why a rule exists. With time delays in code, the problem is use of the delay *for synchronization*. Because systems respond at different speeds at different times. Synchronization means attempting to line up processes. Like waiting for a page to be done loading before trying to take actions on it.
I was not using the delay for synchronization, I was using it to assist in synchronization. I checked the state of something, paused for a 1/10th of a second, and checked the state again. I kept at this until a timeout was met.
We argued about this for some time, and finally went looking at the Selenium code directly. Lo and behold, their code has a 3/10ths of a second delay in it in exactly the same fashion. It's to give over processor cycles to the other processes so they can complete more quickly... Rather than tying up cpu to test the condition over and over until it's met.
More recently, I was implementing a selenium wrapper, and I wanted to spread out a delay across multiple functions. I wanted to validate that the control was present, visible, and enabled... And I wanted only a 20 second timeout overall. Which means exist can take a long time, and complete the other two quickly. Selenium has no native support for anything like this. I either specify different hard timeouts for each one, or I allow them to all add together, and come out with more than 20 seconds.
The answer was to note my target end time (now + timeout) and pass the time remaining to each of the functions. And with a 1/10th of a second in between again.
Rules are important. Rules are present for a reason. That reason may not be applicable in all circumstances. Experience often gives us the whys behind the rules, so we can assess when it's safe to ignore them. And in my experience, there are very few rules in test automation that don't have exceptions.
Friday, May 20, 2016
Getting Abstract...
After I had my first glimmerings of OOP, my next learning was an extrapolation of having code in the window classes.
I didn't know about abstractions or encapsulations yet, but I knew that having all these functions was producing too many things to take care of.
I also knew that, if I put code into the window definitions, then the tests wouldn't have to know how to do a thing, only which window to ask to do it. It wasn't until quite some time later that I was told that this was both abstraction and encapsulation. I had learned to decouple the tests from the interface being tested. Yay!
In my current position, our tests make use of page objects to accomplish this same thing. And it's a decent start. One of our rules is tests never touch controls directly, they call functions in the page object. A good idea.
One of the things I observe in the code I've inherited joining my current gig was that often, instead of clicking a link, the tests here would call a function like page.click_link_to_next_page()
I'm looking to change this, because this means the test is still coupled to the interface, but now we wrap the selector in a function. Which actually doesn't accomplish the goal.
Coming next.. about rules and whys.
I didn't know about abstractions or encapsulations yet, but I knew that having all these functions was producing too many things to take care of.
I also knew that, if I put code into the window definitions, then the tests wouldn't have to know how to do a thing, only which window to ask to do it. It wasn't until quite some time later that I was told that this was both abstraction and encapsulation. I had learned to decouple the tests from the interface being tested. Yay!
In my current position, our tests make use of page objects to accomplish this same thing. And it's a decent start. One of our rules is tests never touch controls directly, they call functions in the page object. A good idea.
One of the things I observe in the code I've inherited joining my current gig was that often, instead of clicking a link, the tests here would call a function like page.click_link_to_next_page()
I'm looking to change this, because this means the test is still coupled to the interface, but now we wrap the selector in a function. Which actually doesn't accomplish the goal.
Coming next.. about rules and whys.
Subscribe to:
Posts (Atom)