The Nature of Frameworks

In my experience, when folks talk about test automation frameworks today, they're talking only about the libraries that you use to isolate things like selectors from the tests. And that is surely an important topic, one I'll address at some future time.

But the 'automation framework' means a lot more than that. Back in the day, we had to think about:

• Where the source was
• How we got it built, including all variants
• Branching including which tests against which branches
• Strategic prioritization of test implementation (eg, which tests are most important to implement first; what combinatorics; what mix of api, ui, performance and stress testing; etc)
• Automated test management
• Automated test data management (eg, preset database images to test against, data for data set tests)
• Coding standards
• Training of new staff
• Design reviews
• Code reviews
• Portability, internationalization and localization
• Automated test execution
• Automated bug reporting
• Artifact aging and control
• Hardware resource allocation and maintenance
• Crisis management
• And QA Evangelizing

These days, it's a lot simpler, but it's still more than just libraries. In my current gig:

• Where the source was -- Github
• How we got it built, including all variants -- Jenkins
• Branching including which tests against which branches -- Still managing this by hand
• Strategic prioritization of test implementation -- Still managing this by hand
• Automated test management -- Nose and Python's unittest library
• Automated test data management -- fixtures help, but mostly Still managing this by hand
• Coding standards -- PEP8 + Our own standards
• Training for new staff -- Still managing this by hand
• Design reviews -- Still managing this by hand
• Code reviews -- Github
• Portability, internationalization and localization -- Honestly, I haven't run into that in my current gig
• Automated test execution -- Nose and Jenkins
• Automated bug reporting -- Jenkins
• Artifact aging and control -- Still managing this by hand
• Hardware resource allocation and maintenance -- SauceLabs and various cloud services like Amazon or Google.
• Crisis management -- Still managing this by hand
• And QA Evangelizing -- I have to admit, at my current gig, this hasn't been an issue.

While there are now systems to handle these things, they still need to be orchestrated. And as a Test Automation Architect, seeing to those systems and how they serve the delivery of useful results is still in my purview. But it sure is nice not to have to build all of it by hand. 

This note has been percolating in my head for some time. It's in response to this: http://www.softwaretestinghelp.com/test-automation-frameworks-selenium-tutorial-20/