Software Test Automation Myths & Facts

Introduction
Today software test automation is becoming more and more popular in both C/S and web environment. As the requirements keep changing (mostly new requirements are getting introduced on daily basis) constantly and the testing window is getting smaller and smaller everyday, the managers are realizing a greater need for test automation. This is good news for us (people who do test automation). But, I am afraid this is the only good news.

Myths & Facts
A number of articles and books are written on different aspects of Software Test Automation. “Test Automation Snake Oil” by, James Bach is an excellent article on some of the myths of automation. I like to discuss some of these myths and will try to point out the facts about these myths. I also like to discuss some of my observations and hopefully point out possible solutions. These are based on my experience with a number of automation projects I was involved with.

Myth 1 - Find more bugs: Some QA managers think that by doing automation they should be able to find more bugs. It’s a myth. Let’s think about it for a minute. The process of automation involves a set of written test cases. In most places the test cases are written by test engineers who are familiar with the application they are testing. The test cases are then given to the automation engineers. In most cases the automation engineers are not very familiar with the test cases they are automating. From test cases to test scripts, automation does not add anything in the process to find more bugs. The test scripts will work only as good as the test cases when comes to finding bugs. So, it’s the test cases that find bugs (or don’t find bugs), not the test scripts.

Myth 2 - Eliminate or reduce manual testers: In order to justify automation, some point out that they should be able to eliminate or reduce the number of manual testers in the long run and thus save money in the process. Absolutely not true. Elimination or reduction of manual testers is not any of the objectives of test automation. Here is why – as I have pointed out earlier that the test scripts are only as good as the test cases and the test cases are written primarily by manual testers. They are the ones who know the application inside out. If the word gets out (it usually does) that the number of manual testers will be reduced by introducing automation then, most if not all manual testers will walk out the door and quality will go with them as well.

Observations

I have met a number of QA managers who are frustrated with their automation. According to them the tool is not doing what it is supposed to do. Here is a true story, the client (I had the opportunity to work with them for some time) found out that the tool they have just bought does not support the application they are testing (I am not making it up). How can this happen! – It does happen more often than one would think. I will get back on this when I discuss possible solutions. A manager of one of the major telecom companies that I had a recent interview with told me that after three years and more than a million dollar he is still struggling with automation. This is pretty sad and I get the feeling that he is not alone.

Solutions / Suggestions

Let’s discuss some of the reasons for this frustration and some of the solutions to this problem.

Fact 1 - Unrealistic expectations: Most managers have their first encounter with any automation tool when they look at the demo and everything looks nice and simple. But everything is not so nice and simple when you try to use the tool with your application. The vendors will only tell you the things you want to hear (how easy to use, how simple to set up, how it will save time and money, how it will help you find more bugs etc.). This builds a false set of hopes and expectations.

Fact 2 - Lack of planning: A great deal of planning is required from selection to implementation of the tool. “Evaluating Tools” by Elisabeth Hendrickson is a very good article on step by step process of selecting a tool. She talks about “Tool Audience” as one of the steps. This would be an ideal way to select a tool. It may not happen in every place because of the everyday workload of the people involved. But the participation of the users in the process is very important, because they are the ones who will use the tool day in and day out. I am almost certain that what happened to one of my clients (the tool they have bought did not support the application they were testing) would not have happened if the users were involved in the selection process.

Fact 3 - Lack of a process: Lack of a process may also contribute to failure of automation. Most places do have some kind of process in place. In most cases (although it differs from place to place) developers write code against a set of requirements. If the requirement does not call for a change in GUI then, there should not be any change in GUI. But if the GUI keep changing constantly from one release to another without any requirement for that change then, there is a problem in the process. You may have the best tool and the best (for your environment) architecture is in place and you will still have problems with your automation because of a faulty process.

Conclusion

I think there is a need to educate QA managers about the benefits and limitations of automation. There is a need to separate the facts from the fictions. But here is the problem, in most cases consultants are brought in to fix problems of prior attempts instead of initial setup. At this point the managers have already learned (painfully) the pitfalls of automation. In order to avoid this painful experience I would recommend (most automation engineers will agree with me) to spend more time up front doing research about the styles and techniques of automation and find an architecture that fits the environment.

There is no doubt that automation adds a great value to overall QA process but, short of knowledge and understanding about automation and lack of planning can also cause a nightmare.

Testing Tool - Rational ClearCase

The Rational ClearCase tool empowers to manage all of the changes the projects encounter. This comprehensive software configuration management (SCM) solution provides version control, workspace management, process configurability, and builds management. With Rational ClearCase, the development team gets a scalable, best practices-based development process that simplifies change - shortening the development cycles, ensuring the accuracy of releases, and delivering reliable builds and patches for previously shipped products.

1. Simplifying the Process of Change

Software development and change: The two go hand in hand. To succeed at the first, need tools and processes for managing the second. Rational ClearCase, when combined with Rational ClearQuest, a flexible defect and change tracking tool, is the market leading software configuration management solution that manages change and complexity. To software teams of all sizes, Rational ClearCase offers tools and processes can implement today and tailor as grow. Rational ClearCase provides a family of products that scale from small project workgroups to the distributed global enterprise, enabling:

• Accelerate release cycles by supporting unlimited parallel development
• Unify the change process across the software development lifecycle
• Scale from small teams to the enterprise without changing tools or processes

2. Software Configuration Management

Software development is an inherently complex process. There are so many protocols, mixed platforms, distributed teams, and various roles to contend with, that team invariably gets tangled in administrative minutia. Rational ClearCase, a robust software artifact management tool, combined with Rational ClearQuest, the most flexible defect and change tracking tool on the market, creates a software configuration management (SCM) solution that helps team to handle the rigors of software development. Rational's SCM solution helps to manage complex change throughout the development lifecycle.

• Share code effortlessly and automate error-prone processes

• Rational's SCM solution offers the essential functions of transparent code sharing, version control, and advanced workspace and builds management. By automating many of the necessary, yet error-prone tasks associated with software development, Rational's SCM solution frees teams of all sizes to build better software faster.

• Unite team with a process that optimizes efficiency

• Process is critical to streamlining software development. A sound process will improve quality, increase development speed and ultimately enhance overall team collaboration and productivity. Rational's SCM solution offers Unified Change Management (UCM), a best practices process for managing change at the activity level and controlling workflow.

• Choose a solution that scales and make it last SCM decision

• Rational has an SCM solution that meets the needs of all size development teams. From small project teams to the global enterprise, Rational has the right size solution for the team. Using the same proven technology, processes and protocols, will be able to select the right product today and seamlessly grow with the product tomorrow – no conversion headaches, data disasters, or process changes. Just smooth scalability.

3. Unified Change Management
Managing the ongoing process of change is important for any development team. However, the issue is further complicated as specialized, distributed teams strive to build high-quality software in less time. Rational has responded with a solution that simplifies the process of change. Unified Change Management (UCM) integrates artifact and activity management. It is among Rational's "best practices" for software development.

UCM is delivered through an integration of Rational ClearCase, for software asset management, and Rational ClearQuest for defect and change tracking – both part of Rational Suite. UCM is a powerful, out-of-the-box workflow for automating change across the software lifecycle and across distributed multi-functional development teams.

UCM helps managers reduce risk by coordinating and prioritizing the activities of developers and by ensuring that they work with the right sets of artifacts. Extending across the lifecycle to accommodate all project domain information – requirements, visual models, code, and test artifacts, it helps teams effectively "baseline" requirements together with code and test assets. The result: accelerated team development in which quality standards are met or exceeded on time and on budget.

4. Rational ClearCase Highlights

• Offers version control, workspace management, build management and process configurability
• Versions all development artifacts
• Enables nonstop parallel development — even across geographically distributed sites
• Provides transparent workspaces for global data access
• Integrates with Rational ClearQuest to provide a seamless approach to defect and change tracking
• Enables Unified Change Management — Rational's activity-based process for managing change
• Scales from small project teams to the global enterprise
• Ships with Rational Suite for a complete change process across the lifecycle
• Offers process configurability without expensive customization
• Provides advanced build auditing
• Provides Web interface for universal data access
• Features graphical interface for easier focus on priority tasks
• Integrates with leading IDEs and development tools, and Web development and authoring tools

Capability Maturity Model (CMM) Interview Questions

1. Give examples for which causal analysis can be done
Ans:

• High / low schedule variance,
• High / low effort variance,
• Poor rating,
• Too many re-opened calls,
• High defect density,
• Very Low resource utilization etc

2. How are the goals categorized in the each process area?
Ans: Goals are categorized as generic and specific goals in each process area.

3. Which Process Area deals with team structure in project and how team goals are aligned to project goals?
Ans: Integrated Teaming

4. Which Process Area deals with doing causal analysis and take actions?
Ans: Causal Analysis and resolution

5. What is the appraisal methodology of CMMI called?
Ans: SCAMPI (Standard CMMI Appraisal Method for Process Improvement)

6. Which is the process area that deals with objective evaluation of processes and associated work products?
Ans: Process and Product QA

7. What are the common features for the generic goals of any Process Area?
Ans:

• Commitment to Perform,
• Ability to Perform,
• Directing Implementation, and
• Verifying Implementation & Generic Practices

8. There are ____________ Process Areas in at Maturity Level 2
Ans: 7

9. List down the acquisition types where in you have contract as on today.
Ans: Buy, Sell, Rent, Outsourcing

10. To provide measurement results; first _______ measurement data ; ______ measurement data.
Ans: COLLECT, ANALYZE

12. There are ____________ Process Areas in at Maturity Level 3.
Ans: 11

12. List down the process areas from Level 3.
Ans:

• Requirement Development,
• Technical Solution,
• Product Integration,
• Verification,
• Validation,
• OPF,
• OPD,
• Organizational Training,
• Risk Management,
• Decision Analysis and Resolution,
• Integrated Supplier Management

13. There are ___ Process Areas in at Maturity Level 4; List down the names of the Process Areas.
Ans: 2

14. List down the process areas from Level 4.
Ans:

• Quantitative Project Management,
• Organizational process performance

15. Which is the process area that deals with objective evaluation of processes and associated work products?
Ans: Process and Product QA

Capability Maturity Model (CMM) Interview Questions

1. How many representations does the CMMI Model have? Name the representations.
Ans: 2 - Staged and continuous

2. Which are the 5 levels in CMMi Framework in Staged Representation?
Ans: 5 Levels in Staged are:

• Performed,
• Managed,
• Defined,
• Quantitatively Managed, and
• Optimizing

3. Which are the disciplines in the base model of CMMI?
Ans:

• Software Engineering,
• System Engineering,
• Integrated product development,
• Software acquisition

4. Which Process Area deals with doing causal analysis and take actions?
Ans: CAR

5. State True or False - In Staged representations all Process Areas are at Level 5.
Ans: TRUE

6. Go through the process areas listed below and identify the measures that can be collected to implement the concepts of quantitative management in each of the practices. The process areas are:

• Project Planning
• Requirement Management
• Verification
• Configuration Management

Ans:

• Project Planning – Effort Variance, Schedule Variance, Effort distribution & productivity
• Requirement Management – Requirement Stability Index
• Verification – Number of defects, Defect density, % defect distribution by phase, %effort spent on review and testing, review and testing effectiveness
• Configuration Management – Effort spent on configuration audits

7. Write down the 7 process areas.
Ans:

• Requirement Management,
• Project Planning,
• Project Monitoring and Control,
• Supplier Agreement and Management,
• Measurement and Analysis,
• Process and Product Quality Assurance,
• Configuration Management

8. Estimates to be done during the ______ phase.
Ans: PLANNING

Software Testing - Frequently Asked Questions (FAQ) Part 3

1. What is SEI- CMM? How many levels are there? What are they?

SEI = 'Software Engineering Institute' at Carnegie-Mellon University; initiated by the U.S. Defense Department to help improve software development processes.
CMM = 'Capability Maturity Model', developed by the SEI. It's a model of 5 levels of Organizational 'maturity' that determine effectiveness in delivering quality software.

The Five levels are

• Initial
• Repeatable
• Defined
• Managed
• Optimizing.

2. What is ISO standard for Software? Explain.

ISO = 'International Organization for Standards' :- The ISO 9001 is a Standard often used by software development organizations. It covers documentation, design, development, production, testing, installation, servicing, and other processes.

ISO 9000-3 (not the same as 9003) is a guideline for applying ISO 9001 to software development organizations. To be ISO 9001 certified, a third-party auditor assesses an organization, and certification is typically good for about 3 years, after which a complete reassessment is required. Note that ISO 9000 certification does not necessarily indicate quality products - it indicates only that documented processes are followed.

3. What is load testing?

Load testing is a performance test which subjects the target-of-test to varying workloads to measure and evaluate the performance behaviors and ability of the target-of-test to continue to function properly under these different workloads. The goal of load testing is to determine and ensure that the system functions properly beyond the expected maximum workload. Additionally, load testing evaluates the performance characteristics, such as response times, transaction rates, and other time sensitive issues. The main purpose of this testing is to test whether the system is capable to handle a large number of users at a
particular time.

4. When to stop testing?

• Deadlines (release deadlines, testing deadlines, etc.)
• Testing with Test cases completed with certain percentage passed
• Test budget depleted / Test group decides.
• Coverage of code/functionality/requirements reaches a specified point
• Bug rate falls below a certain acceptance level ( Metrics use).
• Beta or alpha testing period ends
• Sometimes User decides.