Software Quality Analysis, Software Lifecycle, Software Testing

Testing Glossary - Volume Testing

2007-09-04T01:41:00.000+05:30

The purpose of Volume Testing is to find weaknesses in the system with respect to its handling of large amounts of data during short time periods. For example, this kind of testing ensures that the system will process data across physical and logical boundaries such as across servers and across disk partitions on one server.

Testing Glossary - Benchmarking

2007-09-01T21:39:00.000+05:30

To a certain degree load testing, stress testing & benchmarking, all describe the same kind of process. They all use a community of virtual users to submit a workload to real hardware and software. Benchmarking has been used for decades to describe

real application stressing
sizing of undeveloped applications
competitive positioning by vendors of servers and networks

Sizing is an estimate of the vendor's proposed configuration usually in response to an RFP or Request for Proposal.

Testing Glossary - Stress Testing

2007-08-29T09:37:00.000+05:30

The purpose of Stress Testing is to show that the system has the capacity to handle large numbers of processing transactions during peak periods. The stress testing tests the system beyond the maximum design load of the system. It tests the defect or failure behavior of the system. Defects may come to light, which are unlikely to cause system failures in normal usage. Stress testing checks that the overloading causes the system to “fail-soft” rather than collapse under its load.

How many simultaneous users can my site take without slowing down significantly or crashing? Stress testing is nothing more than applying a steadily increasing load to your site until it reaches the breaking point (when site performance degrades to unacceptable levels). What this test tells you is how well your site will handle unexpected loads that may come as a result of unplanned events like, changing market forces, the failure of your competitors to fill the needs of their customers or serendipitous national publicity.

Testing Glossary - Load Testing

2007-08-24T21:34:00.000+05:30

The creation of a simulated "load" on a real computer system by using virtual users who submit work as real users would do at real client workstations and thus "testing" the system's ability to support such a workload. The virtual users are in software running on a "driver machine" or "injector". Testing of critical software during its development and before its deployment has three components:

Functional testing: Does it conform to the design specifications?
Performance testing: Does each unit offer acceptable response time?
Load testing: What hardware and software system configuration will be required to provide acceptable response times and handle the "load" that will be created by the entire community of users when deployed?

Load testing cannot be accomplished before the application software is available. Unlike other modeling methods for estimating future performance, load testing uses real hardware and software. Load testing was historically done with real people at real terminals following a script and recording response times with stop watches.

For systems with hundreds of users, today one can use a load testing tool rather than real people; can capture the activities of real users and then create scripts that "virtual" users submit to the SUT (system under test) and measure the resulting response times.

Software Test Automation Myths & Facts

2007-06-29T10:42:00.000+05:30

Introduction
Today software test automation is becoming more and more popular in both C/S and web environment. As the requirements keep changing (mostly new requirements are getting introduced on daily basis) constantly and the testing window is getting smaller and smaller everyday, the managers are realizing a greater need for test automation. This is good news for us (people who do test automation). But, I am afraid this is the only good news.

Myths & Facts
A number of articles and books are written on different aspects of Software Test Automation. “Test Automation Snake Oil” by, James Bach is an excellent article on some of the myths of automation. I like to discuss some of these myths and will try to point out the facts about these myths. I also like to discuss some of my observations and hopefully point out possible solutions. These are based on my experience with a number of automation projects I was involved with.

Myth 1 - Find more bugs: Some QA managers think that by doing automation they should be able to find more bugs. It’s a myth. Let’s think about it for a minute. The process of automation involves a set of written test cases. In most places the test cases are written by test engineers who are familiar with the application they are testing. The test cases are then given to the automation engineers. In most cases the automation engineers are not very familiar with the test cases they are automating. From test cases to test scripts, automation does not add anything in the process to find more bugs. The test scripts will work only as good as the test cases when comes to finding bugs. So, it’s the test cases that find bugs (or don’t find bugs), not the test scripts.

Myth 2 - Eliminate or reduce manual testers: In order to justify automation, some point out that they should be able to eliminate or reduce the number of manual testers in the long run and thus save money in the process. Absolutely not true. Elimination or reduction of manual testers is not any of the objectives of test automation. Here is why – as I have pointed out earlier that the test scripts are only as good as the test cases and the test cases are written primarily by manual testers. They are the ones who know the application inside out. If the word gets out (it usually does) that the number of manual testers will be reduced by introducing automation then, most if not all manual testers will walk out the door and quality will go with them as well.

Observations

I have met a number of QA managers who are frustrated with their automation. According to them the tool is not doing what it is supposed to do. Here is a true story, the client (I had the opportunity to work with them for some time) found out that the tool they have just bought does not support the application they are testing (I am not making it up). How can this happen! – It does happen more often than one would think. I will get back on this when I discuss possible solutions. A manager of one of the major telecom companies that I had a recent interview with told me that after three years and more than a million dollar he is still struggling with automation. This is pretty sad and I get the feeling that he is not alone.

Solutions / Suggestions

Let’s discuss some of the reasons for this frustration and some of the solutions to this problem.

Fact 1 - Unrealistic expectations: Most managers have their first encounter with any automation tool when they look at the demo and everything looks nice and simple. But everything is not so nice and simple when you try to use the tool with your application. The vendors will only tell you the things you want to hear (how easy to use, how simple to set up, how it will save time and money, how it will help you find more bugs etc.). This builds a false set of hopes and expectations.

Fact 2 - Lack of planning: A great deal of planning is required from selection to implementation of the tool. “Evaluating Tools” by Elisabeth Hendrickson is a very good article on step by step process of selecting a tool. She talks about “Tool Audience” as one of the steps. This would be an ideal way to select a tool. It may not happen in every place because of the everyday workload of the people involved. But the participation of the users in the process is very important, because they are the ones who will use the tool day in and day out. I am almost certain that what happened to one of my clients (the tool they have bought did not support the application they were testing) would not have happened if the users were involved in the selection process.

Fact 3 - Lack of a process: Lack of a process may also contribute to failure of automation. Most places do have some kind of process in place. In most cases (although it differs from place to place) developers write code against a set of requirements. If the requirement does not call for a change in GUI then, there should not be any change in GUI. But if the GUI keep changing constantly from one release to another without any requirement for that change then, there is a problem in the process. You may have the best tool and the best (for your environment) architecture is in place and you will still have problems with your automation because of a faulty process.

Conclusion

I think there is a need to educate QA managers about the benefits and limitations of automation. There is a need to separate the facts from the fictions. But here is the problem, in most cases consultants are brought in to fix problems of prior attempts instead of initial setup. At this point the managers have already learned (painfully) the pitfalls of automation. In order to avoid this painful experience I would recommend (most automation engineers will agree with me) to spend more time up front doing research about the styles and techniques of automation and find an architecture that fits the environment.

There is no doubt that automation adds a great value to overall QA process but, short of knowledge and understanding about automation and lack of planning can also cause a nightmare.

Testing Tool - Rational ClearCase

2007-06-11T12:09:00.000+05:30

The Rational ClearCase tool empowers to manage all of the changes the projects encounter. This comprehensive software configuration management (SCM) solution provides version control, workspace management, process configurability, and builds management. With Rational ClearCase, the development team gets a scalable, best practices-based development process that simplifies change - shortening the development cycles, ensuring the accuracy of releases, and delivering reliable builds and patches for previously shipped products.

1. Simplifying the Process of Change

Software development and change: The two go hand in hand. To succeed at the first, need tools and processes for managing the second. Rational ClearCase, when combined with Rational ClearQuest, a flexible defect and change tracking tool, is the market leading software configuration management solution that manages change and complexity. To software teams of all sizes, Rational ClearCase offers tools and processes can implement today and tailor as grow. Rational ClearCase provides a family of products that scale from small project workgroups to the distributed global enterprise, enabling:

Accelerate release cycles by supporting unlimited parallel development
Unify the change process across the software development lifecycle
Scale from small teams to the enterprise without changing tools or processes

2. Software Configuration Management

Software development is an inherently complex process. There are so many protocols, mixed platforms, distributed teams, and various roles to contend with, that team invariably gets tangled in administrative minutia. Rational ClearCase, a robust software artifact management tool, combined with Rational ClearQuest, the most flexible defect and change tracking tool on the market, creates a software configuration management (SCM) solution that helps team to handle the rigors of software development. Rational's SCM solution helps to manage complex change throughout the development lifecycle.

Share code effortlessly and automate error-prone processes
Rational's SCM solution offers the essential functions of transparent code sharing, version control, and advanced workspace and builds management. By automating many of the necessary, yet error-prone tasks associated with software development, Rational's SCM solution frees teams of all sizes to build better software faster.
Unite team with a process that optimizes efficiency
Process is critical to streamlining software development. A sound process will improve quality, increase development speed and ultimately enhance overall team collaboration and productivity. Rational's SCM solution offers Unified Change Management (UCM), a best practices process for managing change at the activity level and controlling workflow.
Choose a solution that scales and make it last SCM decision
Rational has an SCM solution that meets the needs of all size development teams. From small project teams to the global enterprise, Rational has the right size solution for the team. Using the same proven technology, processes and protocols, will be able to select the right product today and seamlessly grow with the product tomorrow – no conversion headaches, data disasters, or process changes. Just smooth scalability.

3. Unified Change Management
Managing the ongoing process of change is important for any development team. However, the issue is further complicated as specialized, distributed teams strive to build high-quality software in less time. Rational has responded with a solution that simplifies the process of change. Unified Change Management (UCM) integrates artifact and activity management. It is among Rational's "best practices" for software development.

UCM is delivered through an integration of Rational ClearCase, for software asset management, and Rational ClearQuest for defect and change tracking – both part of Rational Suite. UCM is a powerful, out-of-the-box workflow for automating change across the software lifecycle and across distributed multi-functional development teams.

UCM helps managers reduce risk by coordinating and prioritizing the activities of developers and by ensuring that they work with the right sets of artifacts. Extending across the lifecycle to accommodate all project domain information – requirements, visual models, code, and test artifacts, it helps teams effectively "baseline" requirements together with code and test assets. The result: accelerated team development in which quality standards are met or exceeded on time and on budget.

4. Rational ClearCase Highlights

Offers version control, workspace management, build management and process configurability
Versions all development artifacts
Enables nonstop parallel development — even across geographically distributed sites
Provides transparent workspaces for global data access
Integrates with Rational ClearQuest to provide a seamless approach to defect and change tracking
Enables Unified Change Management — Rational's activity-based process for managing change
Scales from small project teams to the global enterprise
Ships with Rational Suite for a complete change process across the lifecycle
Offers process configurability without expensive customization
Provides advanced build auditing
Provides Web interface for universal data access
Features graphical interface for easier focus on priority tasks
Integrates with leading IDEs and development tools, and Web development and authoring tools

Capability Maturity Model (CMM) Interview Questions

2007-05-19T09:24:00.000+05:30

1. Give examples for which causal analysis can be done
Ans:

High / low schedule variance,
High / low effort variance,
Poor rating,
Too many re-opened calls,
High defect density,
Very Low resource utilization etc

2. How are the goals categorized in the each process area?
Ans: Goals are categorized as generic and specific goals in each process area.

3. Which Process Area deals with team structure in project and how team goals are aligned to project goals?
Ans: Integrated Teaming

4. Which Process Area deals with doing causal analysis and take actions?
Ans: Causal Analysis and resolution

5. What is the appraisal methodology of CMMI called?
Ans: SCAMPI (Standard CMMI Appraisal Method for Process Improvement)

6. Which is the process area that deals with objective evaluation of processes and associated work products?
Ans: Process and Product QA

7. What are the common features for the generic goals of any Process Area?
Ans:

Commitment to Perform,
Ability to Perform,
Directing Implementation, and
Verifying Implementation & Generic Practices

8. There are ____________ Process Areas in at Maturity Level 2
Ans: 7

9. List down the acquisition types where in you have contract as on today.
Ans: Buy, Sell, Rent, Outsourcing

10. To provide measurement results; first _______ measurement data ; ______ measurement data.
Ans: COLLECT, ANALYZE

12. There are ____________ Process Areas in at Maturity Level 3.
Ans: 11

12. List down the process areas from Level 3.
Ans:

Requirement Development,
Technical Solution,
Product Integration,
Verification,
Validation,
OPF,
OPD,
Organizational Training,
Risk Management,
Decision Analysis and Resolution,
Integrated Supplier Management

13. There are ___ Process Areas in at Maturity Level 4; List down the names of the Process Areas.
Ans: 2

14. List down the process areas from Level 4.
Ans:

Quantitative Project Management,
Organizational process performance

15. Which is the process area that deals with objective evaluation of processes and associated work products?
Ans: Process and Product QA

Capability Maturity Model (CMM) Interview Questions

2007-05-18T09:16:00.000+05:30

1. How many representations does the CMMI Model have? Name the representations.
Ans: 2 - Staged and continuous

2. Which are the 5 levels in CMMi Framework in Staged Representation?
Ans: 5 Levels in Staged are:

Performed,
Managed,
Defined,
Quantitatively Managed, and
Optimizing

3. Which are the disciplines in the base model of CMMI?
Ans:

Software Engineering,
System Engineering,
Integrated product development,
Software acquisition

4. Which Process Area deals with doing causal analysis and take actions?
Ans: CAR

5. State True or False - In Staged representations all Process Areas are at Level 5.
Ans: TRUE

6. Go through the process areas listed below and identify the measures that can be collected to implement the concepts of quantitative management in each of the practices. The process areas are:

Project Planning
Requirement Management
Verification
Configuration Management

Ans:

Project Planning – Effort Variance, Schedule Variance, Effort distribution & productivity
Requirement Management – Requirement Stability Index
Verification – Number of defects, Defect density, % defect distribution by phase, %effort spent on review and testing, review and testing effectiveness
Configuration Management – Effort spent on configuration audits

7. Write down the 7 process areas.
Ans:

Requirement Management,
Project Planning,
Project Monitoring and Control,
Supplier Agreement and Management,
Measurement and Analysis,
Process and Product Quality Assurance,
Configuration Management

8. Estimates to be done during the ______ phase.
Ans: PLANNING

Software Testing - Frequently Asked Questions (FAQ) Part 3

2007-05-11T11:33:00.000+05:30

1. What is SEI- CMM? How many levels are there? What are they?

SEI = 'Software Engineering Institute' at Carnegie-Mellon University; initiated by the U.S. Defense Department to help improve software development processes.
CMM = 'Capability Maturity Model', developed by the SEI. It's a model of 5 levels of Organizational 'maturity' that determine effectiveness in delivering quality software.
The Five levels are
Initial
Repeatable
Defined
Managed
Optimizing.

2. What is ISO standard for Software? Explain.

ISO = 'International Organization for Standards' :- The ISO 9001 is a Standard often used by software development organizations. It covers documentation, design, development, production, testing, installation, servicing, and other processes.

ISO 9000-3 (not the same as 9003) is a guideline for applying ISO 9001 to software development organizations. To be ISO 9001 certified, a third-party auditor assesses an organization, and certification is typically good for about 3 years, after which a complete reassessment is required. Note that ISO 9000 certification does not necessarily indicate quality products - it indicates only that documented processes are followed.

3. What is load testing?

Load testing is a performance test which subjects the target-of-test to varying workloads to measure and evaluate the performance behaviors and ability of the target-of-test to continue to function properly under these different workloads. The goal of load testing is to determine and ensure that the system functions properly beyond the expected maximum workload. Additionally, load testing evaluates the performance characteristics, such as response times, transaction rates, and other time sensitive issues. The main purpose of this testing is to test whether the system is capable to handle a large number of users at a
particular time.

4. When to stop testing?

Deadlines (release deadlines, testing deadlines, etc.)
Testing with Test cases completed with certain percentage passed
Test budget depleted / Test group decides.
Coverage of code/functionality/requirements reaches a specified point
Bug rate falls below a certain acceptance level ( Metrics use).
Beta or alpha testing period ends
Sometimes User decides.

Software Testing - Frequently Asked Questions (FAQ)

2007-05-01T10:17:00.000+05:30

1. What is static testing and dynamic testing?
Static testing is verifying that all documents in the project is as per the organizational standards. This type of testing can be done by reviews, walkthrough, and inspection. Dynamic testing is the process of executing a program or system with the intent of finding error or making sure that the system meets its intended requirements.

2. What are the metrics you collect during testing life cycle?
Defect resident time, Defect removal efficiency, Test case effectiveness, Testing efficiency and defect rate, defect origin, Defects Severity type.

3. What are the different approaches of integration testing available?
Top-down approach, Bottom-up approach, Incremental Integration.

4. How defect severities are normally defined?
System failure, Major, Minor/ Normal/ Moderate, Suggestion, OR Critical, High priority, Medium priority, Low priority.

5. What is defect density?
Total no. of Defects found during Testing / Size of the Software product.

6. How a software size is normally defined?
Function points (FP), KLOC (Kilo Lines of Code), SLOC ( source lines of code), Man-Hrs.

7. Write the sequential order in which the following Tests are conducted for a project: Integration, acceptance, Unit, System Testing.
Ans: Unit, Integration System and Acceptance Testing.

8. Name some automated testing tools available in the market?
Rational Test Suite, Mercury Win Runner ,Load Runner, Emprix E-Test suite,
Segue silk test, QA Load, Astra load test / Quick test.

9. What is software 'quality'?
Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable. However, quality is obviously a subjective term. It will depend on who the 'customer' is and their overall influence in the scheme of things. A wide-angle view of the 'customers' of a software development project might include end-users, customer acceptance testers, customer contract officers, customer management, the development organization's management/accountants/testers/salespeople, future software maintenance engineers, stockholders, magazine columnists, etc.

Each type of 'customer' will have their own slant on 'quality' - the accounting department might define quality in terms of profits while an end-user might define quality as user-friendly and bug-free.

10. What is 'Software Quality Assurance'?
Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented for defects 'prevention'.

Six Rules of Unit Testing

2007-04-30T10:05:00.000+05:30

1. Write the test first
2. Never write a test that succeeds the first time
3. Start with the null case, or something that doesn't work
4. Don't be afraid of doing something trivial to make the test work
5. Loose coupling and testability go hand in hand
6. Use mock objects

1. Write the test first

This is the Extreme Programming maxim, and my experience is that it works. First you write the test, and enough application code that the test will compile (but no more!). Then you run the test to prove it fails (see point two, below). Then you write just enough code that the test is successful (see point four, below). Then you write another test.

The benefits of this approach come from the way it makes you approach the code you are writing. Every bit of your code becomes goal-oriented. Why am I writing this line of code? I'm writing it so that this test runs. What do I have to do to make the test run? I have to write this line of code. You are always writing something that pushes your program towards being fully functional.

In addition, writing the test first means that you have to decide how to make your code testable before you start coding it. Because you can't write anything before you've got a test to cover it, you don't write any code that isn't testable.

2. Never write a test that succeeds the first time

After you've written your test, run it immediately. It should fail. The essence of science is falsifiability. Writing a test that works first time proves nothing. It is not the green bar of success that proves your test, it is the process of the red bar turning green. Whenever I write a test that runs correctly the first time, I am suspicious of it. No code works right the first time.

3. Start with the null case, or something that doesn't work

Where to start is often a stumbling point. When you're thinking of the first test to run on a method, pick something simple and trivial. Is there a circumstance in which the method should return null, or an empty collection, or an empty array? Test that case first. Is your method looking up something in a database? Then test what happens if you look for something that isn't there.

Often, these are the simplest tests to write, and they give you a good starting-point from which to launch into more complex interactions. They get you off the mark.

4. Don't be afraid of doing something trivial to make the test work

So you've followed the advice in point 3, and written the following test:

public void testFindUsersByEmailNoMatch()
{
assertEquals("nothing returned", 0, new
UserRegistry().findUsersByEmail("not@in.database").length);
}

The obvious, smallest amount of code required to make this test run is:

public User[] findUsersByEmail(String address) {
return new User[0];
}

The natural reaction to writing code like that just to get the test to run is "But that's cheating!". It's not cheating, because almost always, writing code that looks for a user and sees he isn't there will be a waste of time - it'll be a natural extension of the code you write when you actively start looking for users.

What you're really doing is proving that the test works, by adding the simple code and changing the test from failure to success. Later, when you write testFindUsersByEmailOneMatch and testFindUsersByEmailMultipleMatches, the test will keep an eye on you and make sure that you don't change your behaviour in the trivial cases - make sure you don't suddenly start throwing an exception instead, or return null.

Together, points 3 and 4 combine to provide you with a bedrock of tests that make sure you don't forget the trivial cases when you start dealing with the non-trivial ones.

5. Loose coupling and testability go hand in hand

When you're testing a method, you want the test to only be testing that method. You don't want things to build up, or you'll be left with a maintenance nightmare. For example, if you have a database-backed application then you have a set of unit tests that make sure your database-access layer works. So you move up a layer and start testing the code that talks to the access layer. You want to be able to control what the database layer is producing. You may want to simulate a database failure.

So it's best to write your application in self-contained, loosely coupled components, and have your tests be able to generate dummy components (see mock objects below) in order to tests the way each component talks to each other. This also allows you to write one part of the application and test it thoroughly, even when other parts that the component you are writing will depend on don't exist.

Divide your application into components. Represent each component to the rest of the application as an interface, and limit the extent of that interface as much as possible. When one component needs to send information to another, consider implementing it as an EventListener-like publish/subscribe relationship. You'll find all these things make testing easier and not-so-coincidentally lead to more maintainable code.

6. Use mock objects

A mock object is an object that pretends to be a particular type, but is really just a sink, recording the methods that have been called on it. One implementation of mock objects I wrote in Java using the java.lang.reflect.

A mock object gives you more power when testing isolated components, because it gives you a clear view of what one component does to another when they interact. You can clearly see that yes, the component you're testing called "removeUser" on the user registry component, and passed in an argument of "cmiller", without ever having to use a real user registry component.

One useful application of mock objects comes when testing session EJBs, without the hassle of going through the EJB container to do it. Here's a test class that checks a session EJB correctly rolls back the containing transaction when something goes wrong. Notice also how I'm passing a factory into the EJB - this is something that happens quite often when you want to be able to alternate implementations between test time and deployment.

QTP Interview Questions continued...

2007-03-20T17:46:00.000+05:30

Explain about the Test Fusion Report of QTP?

Once a tester has run a test, a Test Fusion report displays all aspects of the test run: a high-level results overview, an expandable Tree View of the test specifying exactly where application failures occurred, the test data used, application screen shots for every step that highlight any discrepancies, and detailed explanations of each checkpoint pass and failure. By combining Test Fusion reports with Quick Test Professional, you can share reports across an entire QA and development team.

Which environments does QTP support?

Quick Test Professional supports functional testing of all enterprise environments, including Windows, Web,..NET, Java/J2EE, SAP, Siebel, Oracle, PeopleSoft, Visual Basic, ActiveX, mainframe terminal emulators, and Web services.

What is QTP?

Quick Test is a graphical interface record-playback automation tool. It is able to work with any web, java or windows client application. Quick Test enables you to test standard web objects and ActiveX controls. In addition to these environments, Quick Test Professional also enables you to test Java applets and applications and multimedia objects on Applications as well as standard Windows applications, Visual Basic 6 applications and.NET framework applications

Explain QTP Testing process?

Quick Test testing process consists of 6 main phases:
Create your test plan - Prior to automating there should be a detailed description of the test including the exact steps to follow, data to be input, and all items to be verified by the test. The verification information should include both data validations and existence or state verifications of objects in the application.
Recording a session on your application - As you navigate through your application, Quick Test graphically displays each step you perform in the form of a collapsible icon-based test tree. A step is any user action that causes or makes a change in your site, such as clicking a link or image, or entering data in a form.
Enhancing your test - Inserting checkpoints into your test lets you search for a specific value of a page, object or text string, which helps you identify whether or not your application is functioning correctly. NOTE: Checkpoints can be added to a test as you record it or after the fact via the Active Screen. It is much easier and faster to add the checkpoints during the recording process. Broadening the scope of your test by replacing fixed values with parameters lets you check how your application performs the same operations with multiple sets of data. Adding logic and conditional statements to your test enables you to add sophisticated checks to your test.
Debugging your test - If changes were made to the script, you need to debug it to check that it operates smoothly and without interruption.
Running your test on a new version of your application - You run a test to check the behavior of your application. While running, Quick Test connects to your application and performs each step in your test.
Analyzing the test results - You examine the test results to pinpoint defects in your application.
Reporting defects - As you encounter failures in the application when analyzing test results, you will create defect reports in Defect Reporting Tool.

Explain the QTP Tool interface.

It contains the following key elements: Title bar, displaying the name of the currently open test, Menu bar, displaying menus of Quick Test commands, File toolbar, containing buttons to assist you in managing tests, Test toolbar, containing buttons used while creating and maintaining tests, Debug toolbar, containing buttons used while debugging tests. Note: The Debug toolbar is not displayed when you open Quick Test for the first time. You can display the Debug toolbar by choosing View — Toolbars — Debug. Action toolbar, containing buttons and a list of actions, enabling you to view the details of an individual action or the entire test flow. Note: The Action toolbar is not displayed when you open Quick Test for the first time. You can display the Action toolbar by choosing View — Toolbars — Action. If you insert a reusable or external action in a test, the Action toolbar is displayed automatically. Test pane, containing two tabs to view your test-the Tree View and the Expert View ,Test Details pane, containing the Active Screen. Data Table, containing two tabs, Global and Action, to assist you in parameterizing your test. Debug Viewer pane, containing three tabs to assist you in debugging your test-Watch Expressions, Variables, and Command. (The Debug Viewer pane can be opened only when a test run pauses at a breakpoint.) Status bar, displaying the status of the test.

How does QTP recognize Objects in AUT?

Quick Test stores the definitions for application objects in a file called the Object Repository. As you record your test, Quick Test will add an entry for each item you interact with. Each Object Repository entry will be identified by a logical name (determined automatically by Quick Test), and will contain a set of properties (type, name, etc) that uniquely identify each object. Each line in the Quick Test script will contain a reference to the object that you interacted with, a call to the appropriate method (set, click, check) and any parameters for that method (such as the value for a call to the set method). The references to objects in the script will all be identified by the logical name, rather than any physical, descriptive properties.

General Interview Questions on Software Testing (Part 3)

2007-03-20T10:41:00.000+05:30

What is alpha and beta testing?

Alpha Test: Testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by the end-users or others, not by the programmers or testers.

Beta Test: Testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by the end-users or others, not by the programmers or testers.

Write the testing life cycle / test work flow for a project?

Risk Analysis,
Test Planning,
Test case Design,
Test Execution,
Test Logs,
Defect tracking and management.
Test report,
Test metrics collection.

What is Test Plan and what are the contents of a test plan?

What are the techniques used for writing test cases?

What's a 'test case'?

A test case is a document that describes an input, action, or event and an expected response, to determine if a feature of an application is working correctly. A test case should contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results.

General Interview Questions on Software Testing (Part 2)

2007-02-28T05:43:00.000+05:30

What is Unit Testing?
Unit testing is the testing of individual components(units) of the Software. Unit testing is the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design, logic and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses. The test is based on coverage of code statements, paths, branches and conditions.

What is Integration Testing?
It is the testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.

Describe System Testing.
The System testing is to test the System on its overall performance with the Requirements. It is a block-box type testing , which involves in checking the System for Functional requirements, Response time/Performance , Security, Load/Stress/Volume, Usability, Interoperability, etc., (Depends on Technology like eCom, Web, Client / Server, Embedded). Software once validated for meeting the functional requirements must be verified for proper interface with other system elements like hardware, database and people. It verifies that all these system elements mesh properly and the software achieves overall function/performance.
What is acceptance testing?
Formal testing conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system. The goals of these tests are to verify actual data acceptance, processing, and retrieval, and the appropriate implementation of the business rules.

What is regression testing?
Re-testing the application after bug fixes or modifications of the software or its environment.

How testing is different from debugging?
Testing is more to do with establishing that defects exist in a software component or a system, Where as debugging is the action of isolating where the defect is. Thus testing and debugging are complementary activities.

General Interview Question on Software Testing

2007-02-26T14:36:00.000+05:30

What is the 'software life cycle'?
The life cycle begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.
What is software testing? What are the stages of testing available in SDLC? (Software Development Life Cycle).
Testing involves operation of a system or application under controlled conditions and evaluating the results. Testing is to find out the Defects. The stages of testing are unit, integration, system and User acceptance testing.

What are the different types of testing you know?
Black box testing / functional testing, White box testing, load testing, stress testing, volume testing, Sanity testing, performance testing, compatibility testing, security testing, database testing, installation testing, recovery testing, and component testing.

What is the difference between black box testing and white box testing?
In black box testing, the test is based on requirements and functionality. The test is based on inputs, the required action and the output response without considering how the application works. In white box testing, the test is based on internal logic /design of the application code. The test is based on coverage of code statements, paths, branches and conditions.

What is verification and validation?, and IV & V?
Verification refers to the set of activities that ensure that software correctly implements a specific function. It typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications. This can be done with checklists, issues lists, walkthroughs, and inspection meetings.IV & V is independent validation and verification. Validation is achieved through a series of black-box tests that demonstrate conformity with requirements.

QTP Interview Questions...

2007-01-23T17:07:00.000+05:30

What are the Features & Benefits of Quick Test Pro (QTP 8.0)?
Operates stand-alone, or integrated into Mercury Business Process Testing and Mercury Quality Center. Introduces next-generation zero-configuration Keyword Driven testing technology in Quick Test Professional 8.0 allowing for fast test creation, easier maintenance, and more powerful data-driving capability. Identifies objects with Unique Smart Object Recognition, even if they change from build to build, enabling reliable unattended script execution. Collapses test documentation and test creation to a single step with Auto-documentation technology. Enables thorough validation of applications through a full complement of checkpoints.

How to handle the exceptions using recovery scenario manager in QTP?
There are 4 trigger events during which a recovery scenario should be activated. A pop up window appears in an opened application during the test run: A property of an object changes its state or value, A step in the test does not run successfully, An open application fails during the test run, These triggers are considered as exceptions.You can instruct QTP to recover unexpected events or errors that occurred in your testing environment during test run. Recovery scenario manager provides a wizard that guides you through the defining recovery scenario. Recovery scenario has three steps: 1. Triggered Events 2. Recovery steps 3. Post Recovery Test-Run

What is the use of Text output value in QTP?
Output values enable to view the values that the application talks during run time. When parameterized, the values change for each iteration. Thus by creating output values, we can capture the values that the application takes for each run and output them to the data table.

How to use the Object spy in QTP 8.0 version?
There are two ways to Spy the objects in QTP: 1) Thru file toolbar, In the File Toolbar click on the last toolbar button (an icon showing a person with hat). 2) True Object repository Dialog, In Object repository dialog click on the button object spy. In the Object spy Dialog click on the button showing hand symbol. The pointer now changes in to a hand symbol and we have to point out the object to spy the state of the object if at all the object is not visible. or window is minimized then, hold the Ctrl button and activate the required window to and release the Ctrl button.

How Does Run time data (Parameterization) is handled in QTP?
You can then enter test data into the Data Table, an integrated spreadsheet with the full functionality of Excel, to manipulate data sets and create multiple test iterations, without programming, to expand test case coverage. Data can be typed in or imported from databases, spreadsheets, or text files.

What is keyword view and Expert view in QTP?
Quick Test’s Keyword Driven approach, test automation experts have full access to the underlying test and object properties, via an integrated scripting and debugging environment that is round-trip synchronized with the Keyword View. Advanced testers can view and edit their tests in the Expert View, which reveals the underlying industry-standard VBScript that Quick Test Professional automatically generates. Any changes made in the Expert View are automatically synchronized with the Keyword View.

Abstract Test Suites Development - A standard approach continued...

2007-01-17T10:26:00.000+05:30

Abstract Test Suite - Contents

The TTCN is part of the ISO/IEC 9646 conformance-testing framework and is specially designed for the specification of tests of communication systems. The standard introduces the concept of abstract test suites (consisting of abstract test cases), a description of a set of test cases that should be executed for a system. The abstract tests are to be described using a formal language rather than informal language. TTCN is defined in order to describe the abstract test cases.

The reference of Abstract Syntax Notation (ASN.1) definitions from protocol specifications into TTCN ensures consistency between the information transferred in the system specification and test specification.

Abstract test suite consists of four parts,
Ø Suite Overview part
Ø Declarations part
Ø Constraints part
Ø Dynamic part

(a) Suites Overview part
The suite overview part is documentary feature comprised of indexes and page references. It contains table of contents and a description of the test suite, and its purpose is mainly to document the test suite to increase clarity and readability. The quick overview of the entire test suite is possible. The test suite overview consists of four tables:
· Test Suite Structure
· Test Case Index
· Test Step Index
· Default Index

(b) Declarations part
The declaration part is used for declaring types, variables, timers, PCO’s and test components. All the data types used in the test suite are declared in this section. The types can be TTCN or ASN.1 types. Declaring types in TTCN or ASN.1 is similar to type declarations in other programming language except that in TTCN or ASN.1 tables are used instead of files. The declarations part is concerned both with the definition of new (i.e. not predefined) data types and operations and the declaration of all the test suite components.

(c) Constraints Part
The constraints part is used for describing the values sent or received. The structured types PDUs and ASPs defined in the declarations part are used as models to describe the messages sent on PCO. The instances used for sending must be complete, but for receiving there is the possibility to define incomplete values using wildcards, ranges and lists. The constraint part contains the tables for all the ASP, PDU, structure and CM constraints both in the tabular form and the ASN.1.

(d) Dynamic Part
The actual tests are described in dynamic part. It contains all test cases, test steps, and default tables with test events and verdicts. The building blocks are test groups, test cases, test steps and test events. The dynamic part contains all the test cases, all the test steps in the test step library and the all the defaults in the default library.

Test Component

The test component comprises of the test group, test case, test events and test step.

Test event – is the smallest unit of test suite. It corresponds to sending and receiving of the messages and operations for manipulating timers.

Test step – grouping of test events, which is similar to subroutines and procedures in other programming languages.

Test case - fundamental building block in test suite. A test case tests particular feature or function in the implementation under test (IUT). Test case assigns a verdict depends on outcome of the test case.

Test group – grouping of test cases. The grouping can be done based on functionality or features. Test suite – highest level encompassing all the test components and serving as root of the tree.

Abstract Test Suites Development - A standard approach continued...

2007-01-16T22:19:00.000+05:30

ISO 9646 and TTCN
ISO 9646 is a framework for conformance testing. ISO 9646 incorporates the TTCN language as ISO 9646-3. The ISO 9646 is a seven part standard as shown below

Testing Configuration or Architecture
One part of the standard is Test specification/Configuration. How is the test is set up, the responsibilities of different entities involved and the handling of the connections between these are all regulated in the standard.

PICS and PIXIT
The important item in the configuration of a test is to extract the abstract information from the test and this information can be provided during start up of the execution. The Protocol Implementation Conformance Statement (PICS) and Protocol Implementation Extra information (PIXIT) is structures as informal questionnaire. The answers can be mapped to parameters in TTCN and imported. PICS and PIXT contains different information.

PICS – It contains information regarding protocol. This could be optional parts, specific restrictions or adds-on. This information will serve as basis for determining which test cases are applicable.PIXIT – It contains information regarding physical set up and connection of the test that is not part of the protocol. This could be information regarding system under test hardware, socket.

Abstract Test Suites Development - A standard approach

2007-01-16T10:04:00.000+05:30

INTRODUCTION

Over the years, the size and complexity of software in multi-process, distributed architecture systems running in different environment have grown dramatically. Verification & Validation has become an all-important activity in this context. The terms conformance and interoperability are both important characteristics for implementations. An implementation conforms if it meets all of the mandatory requirements of the written specification fully, and also meets those conditional requirements that it claims to meet. Conforming implementations have high probabilities of interoperating with other implementations conforming to the same specification.

Test suites formalize the means with which to establish conformance or interoperability. Test suites come in two forms: abstract and executable. A test suite comprises multiple test cases each designed to test a single requirement or option. An abstract test suite is like un-compiled source code. ISO (International Standards Organization) and ITU-T, while addressing this issue, suggest the usage of TTCN (Tree and tabular combined notation), which is test-equipment-independent. The TTCN language is part of the ISO/IEC 9646 (International Electro-technical Committee) conformance-testing framework and is specially designed for the specification tests of communication systems.

We will discuss standard approach for abstract test suite development. Definition of key concepts associated with the Abstract Test Suites. It also describes about form of abstract test suites and maintenance of ATS (Abstract Test Suite).

TTCN is the defacto standard test environment/Language for communication systems. This is used world wide to test telecommunication and data-communication equipment ranging from built in communication chips to huge switched and intelligent network services. TTCN is widely used for conformance testing. Conformance testing is the process of verifying that an implementation performs in accordance with a particular standard/specification. It is concerned with external behavior (black box) but not with performance/reliability/fault tolerance.

ISO/IEC 9646 (ITU X.290 series) is a seven part standard which defines a framework and methodology for conformance testing of implementations of OSI and ITU protocols. The TTCN is the third part of this standard, i.e. ISO/IEC 9646-3. The Versions 1 and 2 are developed by ISO as part of the widely used ISO/IEC-9646 conformance-testing standard. The ISO/IEC 9646-3 and ITU-T X.292 Updates/maintenance are done by ETSI. The Version 3 is developed by ETSI.

Conformance testing
Conformance testing is verification that an implementation meets the formal requirements of the referenced standards and, more precisely, that it meets the conformance clauses contained in the standards. During the test phase, the implementation is referred to as the Implementation Under Test (IUT). The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate. The testing of performance and robustness are not part of the conformance testing process. No amount of testing can give a full guarantee of successful inter-working. The exhaustive testing of every possible aspect of protocol behavior is unrealistic and impractical for technical and economical reasons.

Conformance testing can however give a reasonable degree of confidence that an implementation, which passes the tests, will comply with the requirements in its communication with other systems. As such, conformance testing can be regarded as a prerequisite for inter-working. Any test can be easily contentious. When comparing a product with its specification, using testing tools, we could ask, in case of discrepancies: Is the product wrong? Is the specification ambiguous? Is the test biased? Is the testing method suitable? Is the testing process agreed and understood?

One-way to solve some of these questions in advance is to standardize: Use of standard protocol specifications, use of standard tests, use of standard methods, use of standard testing process. This is what Conformance Testing is about, based upon a standard testing methodology, as defined by ISO and CCITT. The benefits of conformance testing can be increased further. The use of standard methods, based on approved test suites developed for each OSI standard protocol, and on testing procedures, lead to the comparability of results produced by different testers, and thereby to the mutual recognition of test reports. This will minimize the need for repeated conformance testing, and minimize the associated costs.

Sample Questions for IBM Rational Unified Process (RUP) Exam continued...

2006-12-08T08:08:00.000+05:30

21. Which of the following is not a phase in the Rational Unified Process?

Elaboration
Inception
Implementation
Transition

Correct Answer: C

22. What statement is true of iterations?

You establish plans for each phase, but not for iterations.
A lifecycle phase may contain many iterations.
A construction iteration cannot include any activities from the Requirements workflow.
Iterations may contain many phases.
A major milestone marks the end of every iteration.

Correct Answer: B

23. What does the term "artifact" indicate in the Rational Unified Process?

A unit of work to be performed.
A piece of information that the process produces, modifies, or uses.
That a software engineer must perform the activity.
The specific job position that must perform the activity.

Correct Answer: B

24. Which of the following statements is not true regarding risk management in the Rational Unified Process?

You must update the risk list in each iteration.
The risk list should include technical, business, and management risks.
It is best to develop low-risk portions of the system in early iterations to ensure the schedule is met.
Risk reduction drives iterations.

Correct Answer: C

25. Which of the following is an activity of the Tester?

Test integrated components
Test developed components as units
Define the organization of the code in terms of implementation subsystems
Implement classes and objects in terms of components
Integrate the results produced by individual implementers (or teams) into an executable system.

Correct Answer: A

Sample Questions for IBM Rational Unified Process (RUP) Exam continued...

2006-12-01T08:05:00.000+05:30

16. Which of the following statements is true?

Most systems do not need any supplementary specifications.
Supplementary specifications capture requirements that cannot easily be expressed as use cases.
Legal and regulatory requirements are usually best captured in use cases.
Most systems have no performance requirements.

Correct Answer: B

17. Which of the following Concepts do not belong to the Configuration & Change Management discipline?

Mapping from Design to Code
Product Directory Structure
Change Request Management
Baselining

Correct Answer: A

18. Which of the following is not software engineering best practices recommended by Rational?

Manage change.
Develop iteratively.
Use component architectures.
Maximize reuse.
Freeze requirements at project start.

Correct Answer: E

19. What is a development case?

Another name for a key mechanism
A development cycle specifically devoted to maintenance
A sample architectural design used to guide architectural decisions
The development process that you have chosen to follow in your project

Correct Answer: D

20. In which lifecycle phase is software architecture the primary focus?

Design
Inception
Elaboration
Transition

Correct Answer: C

Sample Questions for IBM Rational Unified Process (RUP) Exam continued...

2006-11-26T20:00:00.000+05:30

11. Which of the following statements does not describe iterative assessment?

Provides insight into the technical progress of the project.
Is optional and often eliminated during early iterations.
Can result in updates to the plan for the subsequent iteration.
Includes a comparison of the iteration plan with the actual cost, schedule, and content accomplished.

Correct Answer: B

12. Which of the following statements is false concerning the development case?

It tells you which artifacts to produce.
You can use it to specify the tools used to produce an artifact.
You can use it to specify the degree of formality associated with an artifact.
It is fixed during Inception and does not generally change over the course of a project.

Correct Answer: D

13. Which of the following does not relate to project planning for iterative development?

Produces a coarse-grained plan identifying the number of iterations per phase, their objectives, and duration.
At the most detailed level, consists of an iteration plan establishing the fine-grained activities and milestones for each iteration.
Incorporates risk management.
Drives to details for complete plan by end of Inception

Correct Answer: D

14. Which of the following statements does not characterize a typical iteration in the Elaboration phase?

Involves more analysis and design than testing effort.
Updates the risk list.
Contributes significantly to the Software Architecture Document.
Spends a significant amount of effort in developing the vision and business case.
Requires the leadership of the software architect.

Correct Answer: D

15. Which of the following statements regarding requirements is true?

Only external requests for change need to be approved by the change review team.
To avoid confusion, all requirements should be standalone with no links to other requirements.
Requirements attributes eliminate the need for configuration and change management of requirements.
Traceability links and requirements attributes are useful in impact assessment of proposed changes.

Correct Answer: D

Sample Questions for IBM Rational Unified Process (RUP) Exam continued...

2006-11-17T19:57:00.000+05:30

6. Which of the following is not a purpose of the Requirements discipline?

To agree with the customer and users on what the system should do.
To give system developers a better understanding of the requirements of the system.
To establish a complete requirements baseline before starting the Design phase.
To delimit the system.

Correct Answer: C

7. Which of the following statements are false concerning the Implementation discipline?

Implementation occurs only in the Construction phase.
Implementers unit test the developed components.
You implement classes and objects in terms of components.
The software architect structures the implementation model.

Correct Answer: A

8. Which of the following has the least impact on the scope of an iteration?

Required system functionality
The project's specific objectives
The phase the project is in
The project's top risks

Correct Answer: B

9. Which of the following is true with respect to an iterative project?

Planning is done only at the start of each phase.
Planning is rarely done.
All planning, including iteration plans, must be completed during the Inception Phase.
Planning is done incrementally with detailed iteration planning done first.
Planning is done incrementally with coarse-grained planning done first.

Correct Answer: F

10. Which of the following statements does not characterize a typical iteration in the Inception phase?

Includes development of the Development Case for the project.
Spends a significant amount of effort in developing the vision and business case.
Includes a significant amount of testing.
Includes creation of a risk list for the project.

Correct Answer: C

Sample Questions for IBM Rational Unified Process (RUP) Exam

2006-11-14T19:48:00.000+05:30

Hi all,
I have started this new series in which I would be presenting to you some of the certification exam questions for IBM Rational Unified Process (RUP). Please note that I have taken maximum care to accurately provide the answers. However, I am not responsible for any unforeseen error here.

1. What statement is correct concerning change management?

You need it only on the later iterations, after the end of Elaboration.
You must change manage all project artifacts, not just source code.
All the changes to be managed originate with the customer and other external entities.
It is the practice of preventing changes to the requirements after the end of Elaboration.

Correct Answer: B

2. Which of the following statements does not characterize a use case? (Select all that apply.)

Is a complete and meaningful flow of events.
Is initiated by an actor to invoke functionality.
A description of the user interface
Models a dialogue between an actor and the system.

Correct Answer: C

3. Which of the following help to define an iteration plan?

A list of risks you must address by the end of the iteration
Development-organization assessment
A list of scenarios or use cases you must complete by the end of the iteration
The current status of the project

Correct Answer: B

4. Which of the following statements characterize project metrics?

Should not be collected on early iterations.
Are useful in tracking trends in important variables such as rework.
Are the basis for iteration assessment.
Provide insight into progress and quality.

Correct Answer: A

5. What does continuous integration mean in the context of the iterative lifecycle?

Integration only for every external release
Integration only at the end of Elaboration, Construction, and Transition
Integration only at the end of Elaboration and Construction
Integration at the end of every iteration

Correct Answer: D

Where Do You See Yourself AFTER 5 YEARS

2006-10-30T15:12:00.000+05:30

How many times have you cringed on being asked this question? Kunal Guha explains the logic behind it

If a poll was conducted on ‘worst interview questions’, this one would probably top the list. And as pointless as it may seem, recruiters don’t seem to tire of asking the question-‘where do you see yourself five years from now?’ Let’s find out why your answer to the above is so important.

TESTING SINCERITY
Contrary to technical questions that could be answered with mechanical ease, this one requires one to think, unless you’ve subscribed to ‘Accepted Interview Answers Digest’. Recruiters, however, can easily see through rehearsed answers and pick out the genuine ones. “A lot of interview questions are based on the person’s accomplishments and choices. Answers to these questions demonstrate a pattern, on the basis of which one can evaluate the applicant’s future aspirations. The trick is to observe how well the applicant’s responses tie in with what one has done, and how well one is able to logically break down one’s career steps. I have seen candidates who broke down their long-term goals into small logical steps,” explains Manoj Varghese, HR Head, Google India.

Recruiters also pay a lot of attention to a candidate’s career record. “Responses backed by a documented track record are valued most. The response may be rehearsed, but the same can be easily validated against the candidate’s track record. Also, an employer’s own experience helps assess whether the candidate’s aspirations are realistic,” explains M V Subramanian, Director-Staffing, HP India. Satish Venkatachaliah, HR Head, SAP Labs, seconds the thought, “We believe that past behaviour is the most reasonable predictor of future behaviour. In case of freshers, we try and look back at the nature of the candidates’ college behaviour - whether he/she was interactive and enterprising, has participated in extra-curricular activities and worked in teams for campus projects.”

JUDGING CHARACTER
Since where you see yourself in the future is based on your imagination, your answer could divulge vital learnings about your thinking pattern and how you would respond to any given situation. These factors could be crucial in evaluating a candidates’ potential. “In our case, if we ask this question, the reason would be to understand the individual’s thinking and quality of reasoning, self-awareness, etc. For example, the person may not know the mechanism of the organisation. Therefore, if he/ she responds to the question in the context of the organisation without asking exploratory questions regarding career growth in that organisation, then the person is obviously shooting from the hip. On the other hand, if the person responds generically, then it is important to understand why the person considers that path and not other alternatives. This will give an idea of the person’s self-evaluation,” explains Pankaj Bhargava-HR Head Marico Limited.

MATCHING EXPECTATIONS
It is crucial to understand how realistic a candidate’s expectations are and whether there is a match between the candidate’s goals and the organisations’. “Investigating a prospective candidate’s future aspirations tells us in advance about the candidates personal goals. This helps us match the candidates’ personal goals with those that we have set out for the role and the organisation. If we find a strong dissonance with the two, it would definitely affect the selection process,” explains Mona Cheriyan, General Manager, Employee Engagement & Europe Liaison, i-flex solutions limited.

Subramanian elaborates, “This question, though perhaps not the most significant, actually attempts to co-relate the candidate’s aspirational level with that of his/her potential. The belief here is that an employee is motivated when he/she sees the role in question as a path toward a larger career goal.”

PRESENTING ASPIRATIONS
You may say that you see yourself as a delivery head or a project director in 3 years, but unless you are able to logically break down your aspirations, it will be a lost cause. So, where you see yourself in the future must have a strong link with the current capacity being offered to you. “Aspirations should be always linked with the role and what the applicant would like to do within a period of time. I have seen candidates create a very good impression with the interviewers by spending time in understanding what their current role is going to be and thinking about ways to add value to that. Such candidates plan their long-term aspirations around mastering their current role, contributing in that role and then checking about the next level of contributions. Even titles can be misleading from organisation to organisation; hence an applicant expressing a desire to be a senior manager in five years may sometimes not make sense in the context of the organisation one is interviewing with,” elucidates Varghese.

It is certain that there is no right or wrong answer, when asked how you envision yourself in the future. But answers could be weak or strong and if your answers are backed by stong conviction, a weak response could be transformed into a strong one.So, now the next time you appear for an interview, make sure you’re honest, reasonably ambitious but not overtly aggressive and it should work out just fine!

Software Test Automation - Myths and Facts continued...

2006-10-15T11:05:00.000+05:30

Observations
I have met a number of QA managers who are frustrated with their automation. According to them the tool is not doing what it is supposed to do. Here is a true story, the client (I had the opportunity to work with them for some time) found out that the tool they have just bought does not support the application they are testing (I am not making it up). How can this happen! – It does happen more often than one would think. I will get back on this when I discuss possible solutions. A manager of one of the major telecom companies that I had a recent interview with told me that after three years and more than a million dollar he is still struggling with automation. This is pretty sad and I get the feeling that he is not alone.

Solutions/Suggestions

Let’s discuss some of the reasons for this frustration and some of the solutions to this problem.

Unrealistic expectations: Most managers have their first encounter with any automation tool when they look at the demo and everything looks nice and simple. But everything is not so nice and simple when you try to use the tool with your application. The vendors will only tell you the things you want to hear (how easy to use, how simple to set up, how it will save time and money, how it will help you find more bugs etc.). This builds a false set of hopes and expectations.
Lack of planning: A great deal of planning is required from selection to implementation of the tool. “Evaluating Tools” by Elisabeth Hendrickson is a very good article on step by step process of selecting a tool. She talks about “Tool Audience” as one of the steps. This would be an ideal way to select a tool. It may not happen in every place because of the everyday workload of the people involved. But the participation of the users in the process is very important, because they are the ones who will use the tool day in and day out. I am almost certain that what happened to one of my clients (the tool they have bought did not support the application they were testing) would not have happened if the users were involved in the selection process.
Lack of a process: Lack of a process may also contribute to failure of automation. Most places do have some kind of process in place. In most cases (although it differs from place to place) developers write code against a set of requirements. If the requirement does not call for a change in GUI then, there should not be any change in GUI. But if the GUI keep changing constantly from one release to another without any requirement for that change then, there is a problem in the process. You may have the best tool and the best (for your environment) architecture is in place and you will still have problems with your automation because of a faulty process.

Software Test Automation - Myths and Facts

2006-10-12T23:03:00.000+05:30

Some Classic Testing Mistakes

2006-10-11T10:52:00.000+05:30

The role of testing

Thinking the testing team is responsible for assuring quality.
Thinking that the purpose of testing is to find bugs.
Not finding the important bugs.
Not reporting usability problems.
No focus on an estimate of quality (and on the quality of that estimate).
Reporting bug data without putting it into context.
Starting testing too late (bug detection, not bug reduction)

Planning the complete testing effort

A testing effort biased toward functional testing.
Underemphasizing configuration testing.
Putting stress and load testing off to the last minute.
Not testing the documentation
Not testing installation procedures.
An overreliance on beta testing.
Finishing one testing task before moving on to the next.
Failing to correctly identify risky areas.
Sticking stubbornly to the test plan.

Personnel issues

Using testing as a transitional job for new programmers.
Recruiting testers from the ranks of failed programmers.
Testers are not domain experts.
Not seeking candidates from the customer service staff or technical writing staff.
Insisting that testers be able to program.
A testing team that lacks diversity.
A physical separation between developers and testers.
Believing that programmers can’t test their own code.
Programmers are neither trained nor motivated to test.

The tester at work

Paying more attention to running tests than to designing them.
Unreviewed test designs.
Being too specific about test inputs and procedures.
Not noticing and exploring “irrelevant” oddities.
Checking that the product does what it’s supposed to do, but not that it doesn’t do what it isn’t supposed to do.
Test suites that are understandable only by their owners.
Testing only through the user-visible interface.
Poor bug reporting.
Adding only regression tests when bugs are found.
Failing to take notes for the next testing effort.

Test automation

Attempting to automate all tests.
Expecting to rerun manual tests.
Using GUI capture/replay tools to reduce test creation cost.
Expecting regression tests to find a high proportion of new bugs.

Code coverage

Embracing code coverage with the devotion that only simple numbers can inspire.
Removing tests from a regression test suite just because they don’t add coverage.
Using coverage as a performance goal for testers.
Abandoning coverage entirely.

Testing Mistakes continued...

2006-10-08T17:26:00.000+05:30

From the “find important bugs” standpoint, the first testing effort was superior. It found 100 bugs before release, whereas the second found only 74. But I think you can make a strong case that the second effort is more useful in practical terms. Let me restate the twosituations in terms of what a test manager might say before release:
1. “We have tested subsystem 1 very thoroughly, and we believe we’ve found almost allof the priority 1 bugs. Unfortunately, we don’t know anything about the bugginess ofthe remaining five subsystems.”
2. “We’ve tested all subsystems moderately thoroughly. Subsystem 1 is still very buggy.The other subsystems are about 1/10th as buggy, though we’re sure bugs remain.

”This is, admittedly, an extreme example, but it demonstrates an important point. Theproject manager has a tough decision: would it be better to hold on to the product formore work, or should it be shipped now? Many factors - all rough estimates of possiblefutures - have to be weighed: Will a competitor beat us to release and tie up the market? Will dropping an unfinished feature to make it into a particular magazine’s special “JavaDevelopment Environments” issue cause us to suffer in the review? Will critical customerX be more annoyed by a schedule slip or by a shaky product? Will the product be buggyenough that profits will be eaten up by support costs or, worse, a recall?

The testing team will serve the project manager better if it concentrates first on providingestimates of product bugginess (reducing uncertainty), then on finding more of the bugsthat are estimated to be there. That affects test planning, the topic of the next theme.

It also affects status reporting. Test managers often err by reporting bug data without putting it into context. Without context, project management tends to focus on one graph:

The flattening in the curve of bugs found will be interpreted in the most optimistic possible
way unless you as test manager explain the limitations of the data:
· “Only half the planned testing tasks have been finished, so little is known about half
the areas in the project. There could soon be a big spike in the number of bugs
found.”
· “That’s especially likely because the last two weekly builds have been lightly tested.

“Furthermore, based on previous projects with similar amounts and kinds of testing effort, it’s reasonable to expect at least 45 priority-1 bugs remain undiscovered. Historically, that’s pretty high for a successful product.”

Testing Mistakes continued...

2006-09-07T19:33:00.000+05:30

What’s an important bug? Important to whom? To a first approximation, the answer must be “to customers”. Almost everyone will nod their head upon hearing this definition, but do they mean it? Here’s a test of your organization’s maturity. Suppose your product is a system that accepts email requests for service. As soon as a request is received, it sends a reply that says “your request of 5/12/97 was accepted and its reference ID is NIC-051297-
3”.

A tester who sends in many requests per day finds she has difficulty keeping track of which request goes with which ID. She wishes that the original request were appended to the acknowledgement. Furthermore, she realizes that some customers will also generate many requests per day, so would also appreciate this feature. Would she:

file a bug report documenting a usability problem, with the expectation that it will be assigned a reasonably high priority (because the fix is clearly useful to everyone, important to some users, and easy to do)?
file a bug report with the expectation that it will be assigned “enhancement request” priority and disappear forever into the bug database?
file a bug report that yields a “works as designed” resolution code, perhaps with an email “nastygram” from a programmer or the development manager?
not bother with a bug report because it would end up in cases (2) or (3)?
If usability problems are not considered valid bugs, your project defines the
testing task too narrowly. Testers are restricted to checking whether the product does what was intended, not whether what was intended is useful. Customers do not care about the distinction, and testers shouldn’t either.

Testers are often the only people in the organization who use the system as heavily as an expert. They notice usability problems that experts will see. (Formal usability testing almost invariably concentrates on novice users.) Expert customers often don’t report usability problems, because they’ve been trained to know it’s not worth their time.

Instead, they wait (in vain, perhaps) for a more usable product and switch to it. Testers can prevent that lost revenue. While defining the purpose of testing as “finding bugs important to customers” is a step forward, it’s more restrictive than I like. It means that there is no focus on an estimate of quality (and on the quality of that estimate). Consider these two situations for a product with five subsystems.

100 bugs are found in subsystem 1 before release. (For simplicity, assume that all bugs are of the highest priority.) No bugs are found in the other subsystems. After release, no bugs are reported in subsystem 1, but 12 bugs are found in each of the other subsystems.
Before release, 50 bugs are found in subsystem 1. 6 bugs are found in each of the other subsystems. After release, 50 bugs are found in subsystem 1 and 6 bugs in each of the other subsystems.

Theme One: The Role of Testing

2006-08-31T19:32:00.000+05:30

A first major mistake people make is thinking that the testing team is responsible for assuring quality. This role, often assigned to the first testing team in an organization, makes it the last defense, the barrier between the development team (accused of producing bad quality) and the customer (who must be protected from them).

It’s characterized by a testing team (often called the “Quality Assurance Group”) that has formal authority to prevent shipment of the product. That in itself is a disheartening task: the testing team can’t improve quality, only enforce a minimal level. Worse, that authority is usually more apparent than real. Discovering that, together with the perverse incentives of telling developers that quality is someone else’s job, leads to testing teams and testers who are disillusioned, cynical, and view themselves as victims. We’ve learned from
Deming and others that products are better and cheaper to produce when everyone, at every stage in development, is responsible for the quality of their work.

In practice, whatever the formal role, most organizations believe that the purpose of testing is to find bugs. This is a less pernicious definition than the previous one, but it’s missing a key word. When I talk to programmers and development managers about testers, one key sentence keeps coming up: “Testers aren’t finding the important bugs.” Sometimes that’s just griping, sometimes it’s because the programmers have a skewed sense of what’s important, but I regret to say that all too often it’s valid criticism.

Too many bug reports from testers are minor or irrelevant, and too many important bugs are missed.

Classic Testing Mistakes

2006-08-31T19:25:00.000+05:30

It's easy to make mistakes when testing software or planning a testing effort. Some mistakes are made so often, so repeatedly, by so many different people, that they deserve the label Classic Mistake.

Classic mistakes cluster usefully into five groups, which I’ve called “themes”:
· The Role of Testing: who does the testing team serve, and how does it do that?
· Planning the Testing Effort: how should the whole team’s work be organized?
· Personnel Issues: who should test?
· The Tester at Work: designing, writing, and maintaining individual tests.
· Technology Rampant: quick technological fixes for hard problems.

I have two goals for this paper. First, it should identify the mistakes, put them in context, describe why they’re mistakes, and suggest alternatives. Because the context of one mistake is usually prior mistakes, the paper is written in a narrative style rather than as a list that can be read in any order. Second, the paper should be a handy checklist of mistakes. For that reason, the classic mistakes are printed in a larger bold font when they appear in the text, and they’re also summarized at the end.

Although many of these mistakes apply to all types of software projects, my specific focus is the testing of commercial software products, not custom software or software that is safety critical or mission critical.

Software Development Lifecycles - Iterative Lifecycle Models

2006-08-12T21:00:00.000+05:30

An iterative lifecycle model does not attempt to start with a full specification of
requirements. Instead, development begins by specifying and implementing just part of the software, which can then be reviewed in order to identify further requirements. This process is then repeated, producing a new version of the software for each cycle of the model. Consider an iterative lifecycle model which consists of repeating the four phases in sequence, as illustrated by following figure.

· A Requirements phase, in which the requirements for the software are gathered and analyzed. Iteration should eventually result in a requirements phase which produces a complete and final specification of requirements.
· Design phase, in which a software solution to meet the requirements is designed. This may be a new design, or an extension of an earlier design.
· An Implementation and Test phase, when the software is coded, integrated and
tested.
· A Review phase, in which the software is evaluated, the current requirements are reviewed, and changes and additions to requirements proposed.

For each cycle of the model, a decision has to be made as to whether the software produced by the cycle will be discarded, or kept as a starting point for the next cycle (sometimes referred to as incremental prototyping). Eventually a point will be reached where the requirements are complete and the software can be delivered, or it becomes impossible to enhance the software as required, and a freash start has to be made.

The iterative lifecycle model can be likened to producing software by successive
approximation. Drawing an analogy with mathematical methods which use successive approximation to arrive at a final solution, the benefit of such methods depends on how rapidly they converge on a solution.

Continuing the analogy, successive approximation may never find a solution. The iterations may oscillate around a feasible solution or even diverge. The number of iterations required may become so large as to be unrealistic. We have all seen software developments which have made this mistake!

The key to successful use of an iterative software development lifecycle is rigorous validation of requirements, and verification (including testing) of each version of the software against those requirements within each cycle of the model. The first three phases of the example iterative model are in fact an abbreviated form a sequential V or waterfall lifecycle model. Each cycle of the model produces software which requires testing at the unit level, for software integration, for system integration and for acceptance. As the software evolves through successive cycles, tests have to be repeated and extended to verify each version of the software.

Software Development Lifecycles - Progressive Development Lifecycle Models

2006-08-12T20:55:00.000+05:30

The sequential V and waterfall lifecycle models represent an idealised model of software development. Other lifecycle models may be used for a number of reasons, such as volatility of requirements, or a need for an interim system with reduced functionality when long timescales are involved. As an example of other lifecycle models, let us look at progressive development and iterative lifecycle models.

A common problem with software development is that software is needed quickly, but itwill take a long time to fully develop. The solution is to form a compromise between timescales and functionality, providing "interim" deliveries of software, with reduced functionality, but serving as a stepping stones towards the fully functional software. It is also possible to use such a stepping stone approach as a means of reducing risk.

The usual names given to this approach to software development are progressive
development or phased implementation. The corresponding lifecycle model is referred to as a progressive development lifecycle. Within a progressive development lifecycle, each individual phase of development will follow its own software development lifecycle, typically using a V or waterfall model. The actual number of phases will depend upon the development.

Software Development Lifecycles - Sequential Lifecycle Models

2006-07-28T07:40:00.000+05:30

The software development lifecycle begins with the identification of a requirement for software and ends with the formal verification of the developed software against that requirement. Traditionally, the models used for the software development lifecycle have been sequential, with the development progressing through a number of well defined phases. The sequential phases are usually represented by a V or waterfall diagram. These models are respectively called a V lifecycle model and a waterfall lifecycle model.

There are in fact many variations of V and waterfall lifecycle models, introducing different phases to the lifecycle and creating different boundaries between phases. The following set of lifecycle phases fits in with the practices of most professional software developers.

· The Requirements phase, in which the requirements for the software are gathered and analyzed, to produce a complete and unambiguous specification of what the software is required to do.

· The Architectural Design phase, where a software architecture for the implementation of the requirements is designed and specified, identifying the components within the software and the relationships between the components.

· The Detailed Design phase, where the detailed implementation of each component is specified.
· The Code and Unit Test phase, in which each component of the software is coded and tested to verify that it faithfully implements the detailed design.
· The Software Integration phase, in which progressively larger groups of tested software components are integrated and tested until the software works as a whole.
· The System Integration phase, in which the software is integrated to the overall product and tested.
· The Acceptance Testing phase, where tests are applied and witnessed to validate that the software faithfully implements the specified requirements.

Software specifications will be products of the first three phases of this lifecycle model. The remaining four phases all involve testing the software at various levels, requiring test specifications against which the testing will be conducted as an input to each of these phases.

Software Testing & Software Development Lifecycles

2006-07-28T07:39:00.000+05:30

The various activities which are undertaken when developing software are commonly modelled as a software development lifecycle. The software development lifecycle begins with the identification of a requirement for software and ends with the formal verification of the developed software against that requirement.

The software development lifecycle does not exist by itself, it is in fact part of an overall product lifecycle. Within the product lifecycle, software will undergo maintenance to correct errors and to comply with changes to requirements. The simplest overall form is where the product is just software, but it can become much more complicated, with multiple software developments each forming part of an overall system to comprise a product.

There are a number of different models for software development lifecycles. One thing which all models have in common, is that at some point in the lifecycle, software has to be tested. This paper outlines some of the more commonly used software development lifecycles, with particular emphasis on the testing activities in each model.

The models under consideration here are:

Sequential Lifecycle Model - V Lifecycle Model & Waterfall Lifecycle Model
Progressive Development Lifecycle Model
Iterative Lifecycle Model

Let's have a brief look at all of these models...

QA Process Management - The Challenges Encountered (Summary)

2006-06-23T16:51:00.000+05:30

Summarily, the challenges faced during these stages fell into three categories:

• Category I- Team Collaboration: Conducting effective Testing Cycle, generation of good quality Test Cases, sharing and creation of the numerous indispensable information assets of the QA process required a high degree of collaboration among Quality Managers, Developers, Test Case Reviewers and Testers.

• Category II- Information Assets Management: Host of documents formed an integral part of the QA Cycle

Test Plan
Detailed Feature Specifications
Test Case
Test Case Review Template

Documents generated as an output of the QA Cycle included

Test Logs
Defects Tracking Sheet
Session Reports
QA Reports

For process consistency and uniformity, well-defined standards in the form of published templates or documentation guidelines were needed for each of these assets. Also mandatory for each, was a create-approve-publish document lifecycle. Post QA Cycle, archiving of assets in a secure, well-organized repository was needed. Search, security and tracking of assets were basic requirements on such a repository.

• Category III- Scheduling, Execution and Tracking of Test Plans: The actual Test Cycle needed to be planned, approved, efficiently executed, and tracked.
In the existing work scenario collaboration and planning happened through formal or informal meetings, asset sharing and management through mails or shared file servers all of which were cumbersome to track, slow and time inefficient.

QA Process Management - The Challenges Encountered

2006-06-23T16:45:00.000+05:30

Case Study Example:

An Offshore Service Center in Wipro was providing Testing Services to a desktop software development corporation named ABC* Corp. Three modules of their desktop suite: DeskPro, DeskNet and DeskSecure had to be tested at the Offshore Center. Most of the time the testing of the modules were performed independently but a release of the desktop suite mandated that they work in sync, thus multiplying the problems of the managers.

The problems being faced by this team were analyzed and a QA Process Management Solution was modeled. The tribulations faced are elucidated below.

The Problem Statement

Phase 1: Test Case Generation
ABC* Corp. used to send detailed feature and policy documentation for features to be tested at the Off Shore Center. It would also send guidelines for writing Test Cases (TC) for the feature. Two or three engineers had to collaborate on Test Case generation for each feature as per the guidelines sent by ABC* Corp. The first draft of the Test Cases had to undergo review and approval by peer-teams within or across modules. Review comments had to be incorporated by the authors and the modifications had to be reviewed again.

After the first draft and peer review, Test Cases had to be reviewed by the QA leads of each module and a consolidated TC Review Report for the module was generated. QM had to inspect the modular TC Review Reports, consolidate them and generate a single complete Test Case Review Report. After this the Project Manager (PM) had to approve the test report and dispatch the TCs as well as TC Review Report to ABC* Corp for approval. The client then approved or suggested modifications to Test Cases. Modifications, if needed, were made, reviewed and finally with ABC* Corp’s approval the Test Cases were published.

Phase 2: Test Cycle Planning
QA leads of all the three modules generated respective test plans and communicated it to the Quality Manager (QM). QM used to check for feasibility of plans and plausibility of estimates. Inputs, if any, on plan modification had to be communicated to the QA leads and modifications made had to be reviewed again. Issues, if any, relating to resource availability, reconsideration of deadlines etc, had to be escalated to the Project Manager by the QM. Inputs, if any, from the PM had to be incorporated into the execution plan by the QM and reviewed by the PM again. With the Project Managers’ approval the Test Plans were published.

Phase 3: Test Cycle Execution
The Test Plans, Test Case documents, Log Sheet Templates and guidelines to fill the Log Sheets had to be communicated to test engineers. Testing was carried out and defects were logged into a shared Defect Database. Any ad hoc problem faced by testers during QA process had to be addressed quickly. Progress and pace of testing were tracked. At the end of the test cycle, Test Logs were inspected. Any missing pertinent data in the logs were pointed out to the Testers and complete Log Sheets solicited. Test Logs had to be archived in a shared repository. Module leads were to generate report based on test results logged.

The Quality Manager reviewed each module’s report and consolidated it into a final QA Report. The final Report was submitted to the Project Manager. Tick off Discussions ensued between the QM, PM and the client. Based on severity of bugs reported, product modules - DeskNet, DeskPro and DeskSecure were declared ready for release. The release information and QA Reports were published.

(Continued...)

QA Process Management - Significance of QA Process

2006-06-15T11:30:00.000+05:30

Procedurally, the entire QA cycle consists of three stages:

Test Case Generation: This stage starts quite early in the SDLC, just after the Design Phase. After the feature requirements are frozen and the design agreed upon, Test Cases are generated. These Test Cases are used for feature verification at the end of Development Phase. However this initial draft may undergo several modifications based on tester and end user feedback. Succinctly put, the test cases must test for expected functionality while also ensuring functioning of software as per expectations. The former
depends on the captured requirement specifications while the latter is the end users’ expectations from the software. Thus maturity of Test Cases is a long drawn process with need for continuous review, updating, and tracking of updating.
Test Cycle Planning: This stage involves drawing of schedules and resource allocation charts for testing and their approval by supervisors before execution. It gains significance because effectiveness of the Test Cycle depends on balanced deployment of manpower and testing resources like servers, peripherals etc based on criticality of tested features. A timely release requires planning an efficient test schedule.
Test Cycle Execution: The actual testing takes place in this stage. Activities may include logging of defect(s)/suggestions encountered in testing, test case modifications, analysis of test logs by leads and generation of QA reports based on findings. The test reports are crucial to the actual release of the software.

Significance of QA Process
QA Phase vindicates the engineering effort and ensures completeness of the generated software before it is released for general availability. It forms a very critical node of the Software Development Lifecycle as can be deduced from the activities enumerated above.

As its chief objective, the QA Phase brings out the lacunae in requirement gathering, design and development of the software through reporting of defects and suggestion of improvements.
Beyond its main purpose of catching and reporting bugs in the software, QA ensures compliance of functionality with pre-set expectations.
It also ensures reliability of software over and above the expected functionality.
It guarantees scalability, concurrent accessibility and satisfactory performance of software under different usage conditions.
Specialized testing such as Beta Testing validates that the software not only functions as per the defined specifications but also meets customer expectations and market needs. Receiving and incorporating feedback from end users belonging to different market segments brings in competitiveness in the features also.

The actual testing of an application may be of different types viz Black Box Testing, White Box Testing, Graphics Interface Testing etc depending on the type of application. It can also be carried out either manually by a team of Test Engineers or through the use of Test Automation Tools with minimal manual effort.

Irrespective of these specifics, the QA Phase poses several management challenges as is illustrated through an example (check out in the next posts).

QA Process Management - QA Process Activities

2006-06-15T11:23:00.000+05:30

The activities and implications of the QA Phase can vary based on the various software development paradigms in the industry. These are depicted in the table below:

Software Product Development - A software product release is preceded by an internal QA cycle wherein Test Engineers from the product development group verify the product functionality against the feature specifications gathered from market sources/sales partners. This is followed by Beta Testing of the product by external users - customers, partners, patrons or volunteering testers.
Software Application Development or Software Servicing - In this case three levels of testing may be carried out. The party involved in software servicing or application development carries out the first level of testing at the end of the development phase. At this level the developed application’s functionality is verified strictly against client’s specific requirements. This is typically followed by System Integration and testing of the service or application at the client site, either by test engineers from the servicing project or by test engineers hired by the client. If the client is servicing a customer in turn, testing may be done at the end-customer site also by test engineers hired at the customer end.
Software Testing Services - This is a category of software services in which the development partner outsources the complete testing of its product, application or service to a third party. Dedicated teams of Test Engineers at the third party site solely perform testing of the developed software.

The example elucidated in this paper falls in the third category but the challenges that are faced and the solution that has been presented are applicable to the other two categories also.

A checklist of model requirements

2006-06-05T17:09:00.000+05:30

These requirements are grouped according to testing activity.

A test model should:

force a testing reaction to every code handoff in the project.
require the test planner to take explicit, accountable action in response to dropped handoffs, new handoffs, and changes to the contents of handoffs.
explicitly encourage the use of sources of information other than project
documentation during test design.
allow the test effort to be degraded by poor or late project documentation, but prevent it from being blocked entirely.
allow individual tests to be designed using information combined from various sources.
allow tests to be redesigned as new sources of information appear.
include feedback loops so that test design takes into account what’s learned by running tests.
allow testers to consider the possible savings of deferring test execution.
allow tests of a component to be executed before the component is fully assembled.

Summary

2006-06-05T17:06:00.000+05:30

The V model is fatally flawed, as is any model that:

Ignores the fact that software is developed in a series of handoffs, where each handoff changes the behavior of the previous handoff,
Relies on the existence, accuracy, completeness, and timeliness of development documentation,
Asserts a test is designed from a single document, without being modified by later or earlier documents, or
Asserts that tests derived from a single document are all executed together.

I have sketched – but not elaborated – a replacement model. It organizes the testing effort around code handoffs or milestones. It takes explicit account of the economics of testing: that the goal of test design is to discover inputs that will find bugs, and that the goal of test implementation is to deliver those inputs in any way that minimizes lifecycle costs.

The model assumes imperfect and changing information about the product. Testing a product is a learning process. In the past, I haven’t thought much about models. I ostensibly used the V model. I built my plans according to it, but seemed to spend a lot of my time wrestling with issues that the model didn’t address. For other issues, the model got in my way, so I worked around it.

I hope that thinking explicitly about requirements will be as useful for developing a testing model as it is when developing a product. I hope that I can elaborate on the model presented in this paper to the point that it provides as much explicit guidance as the V model seems to.

A different model

2006-06-05T16:59:00.000+05:30

Let’s step back for a second. What is our job?

There are times when some person or group of people hands some code to other peopleand says, “Hope you like it.” That happens when the whole project puts bits on a CD andgives them to customers. It also happens within a project:·

One development team says to other teams, “We’ve finished the XML enhancementsto the COMM library. The source is now in the master repository; the executablelibrary is now in the build environment. The XARG team should now be unblocked –go for it!”·
One programmer checks in a bug fix and sends out email saying, “I fixed the bug inallocAttList. Sorry about that.” The other programmers who earlier stumbled overthat code can now proceed.

In all cases, we have people handing code to other people, possibly causing themdamage. Testers intervene in this process. Before the handoff, testers execute the code,find bugs (the damage), and ask the question, “Do you really want to hand this off?” Inresponse, the handoff may be deferred until bugs are fixed.

This act is fundamental to testing, regardless of the other things you may do. If you don’texecute the code to uncover possible damage, you’re not a tester.Our test models should be built around the essential fact of our lives: code handoffs.Therefore, a test model should force a testing reaction to every code handoff in theproject. I’ll use the XML-enhanced COMM library as an example. That’s a handoff from oneteam to the rest of the project. Who could be damaged?

It might immediately damage the XARG team, who will be using those XMLenhancements in their code.
It might later damage the marketing people, who will be giving a demonstration ofthe “partner release” version of the product at a trade show. XML support is animportant part of their sales pitch.
Still later, it might damage a partner who adopts our product.

We immediately have some interesting test planning questions. The simple thing to do would be to test the XML enhancements completely at the time of handoff. (By “completely,” I mean “design as many tests for them as you ever will.”) But maybe some XML features aren’t required by the XARG team, so it makes sense to test them through
the integrated partner release system. That means moving some of the XML-inspired testing to a later handoff. Or we might move it later for less satisfying reasons, such as that other testing tasks must take precedence in the near term. The XARG team will have to resign itself to stumbling over XML bugs for a while.

Our testing plan might be represented by a testing schedule that annotates the
development timeline.

What’s wrong with the V model? (Concluded...)

2006-05-30T18:02:00.000+05:30

In the previous diagram, we had arrows pointing upward (effectively, later in time). It can also make sense to have arrows pointing downward (earlier in time):

In that case, the boxes on the left might be better labeled “Whatever test design can be done with the information available at this point”.

Therefore, when test design is derived from a description of a component of the system, the model must allow such tests to be executed before the component is fully assembled.

I have to admit my picture is awfully ugly – all those arrows going every which way. I have two comments about that:

We are not in the business of producing beauty. We’re in the business of finding as many serious bugs as possible as cheaply as possible.
The ugliness is, in part, a consequence of assuming that the order in which developers produce system description documents, and the relationships among those documents, is the mighty oak tree about which the slender vine of testing twines. If we adopt a different organizing principle, things get a bit more pleasing. But they’re still complicated, because we’re working in a complicated field.

The V model fails because it divides system development into phases with firm
boundaries between them. It discourages people from carrying testing information across those boundaries. Some tests are executed earlier than makes economic sense. Others are executed later than makes sense.

Moreover, it discourages you from combining information from different levels of system description. For example, organizations sometimes develop a fixation on “signing off” on test designs. The specification leads to the system test design. That’s reviewed and signed off. From that point on, it’s done. It’s not revised unless the specification is. If information relevant to those tests is uncovered later – if, for example, the architectural design reveals that some tests are redundant – well, that’s too bad. Or, if the detailed design reveals an internal boundary that could easily be incorporated into existing system tests, that’s tough: separate unit tests need to be written.

Therefore, the model must allow individual tests to be designed using information combined from various sources.

And further, the model must allow tests to be redesigned as new sources of
information appear.

What’s wrong with the V model? (Continued...)

2006-05-30T17:49:00.000+05:30

There’s always some dispute over how big a unit should be (a function? a class? a collection of related classes?) but that doesn’t affect my argument. I believe, a unit is the smallest chunk of code that the developers can stand to talk about as an independent entity.

The V model says that someone should first test each unit. When all the subsystem’s units are tested, they should be aggregated and the subsystem tested to see if it works as a whole.

So how do we test the unit? We look at its interface as specified in the detailed design, or at the code, or at both, pick inputs that satisfy some test design criteria, feed those inputs to the interface, then check the results for correctness. Because the unit usually can’t be executed in isolation, we have to surround it with stubs and drivers, as shown at the right. The arrow represents the execution trace of a test.

That’s what most people mean when they say “unit testing”.

I think that approach is sometimes a bad idea. The same inputs can often be delivered to the unit through the subsystem, which thus acts as both stub and driver. That looks like the picture to the right.

The decision about which approach to take is a matter of weighing tradeoffs. How much would the stubs cost? How likely are they to be maintained? How likely are failures to be masked by the subsystem? How difficult would debugging through the subsystem be? If tests aren’t run until integration, some bugs will be found later. How does the estimated cost of that compare to the cost of stubs and drivers? And so forth.

The V model precludes these questions. They don’t make sense. Unit tests get executed when units are done. Integration tests get executed when subsystems are integrated. End of story. It used to be surprising and disheartening to me how often people simply wouldn’t think about the tradeoffs – they were trapped by their model.

Therefore, a useful model must allow testers to consider the possible savings of deferring test execution.

A test designed to find bugs in a particular unit might be best run with the unit in isolation, surrounded by unit-specific stubs and drivers. Or it might be tested as part as the subsystem – along with tests designed to find integration problems. Or, since a subsystem will itself need stubs and drivers to emulate connections to other subsystems, it might sometimes make sense to defer both unit and integration tests until the whole system is at least partly integrated. At that point, the tester is executing unit, integration, and system tests through the product’s external interface. Again, the purpose is to minimize total lifecycle cost, balancing cost of testing against cost of delayed bug discovery. The distinction between “unit”, “integration”, and “system” tests begins to break down. In effect, you have the above picture.

It would be better to label each box on the right “execution of appropriate tests” and be done with it. What of the left side? Consider system test design, an activity driven by the specification. Suppose that you knew that two units in a particular subsystem, working in conjunction, implemented a particular statement in the specification. Why not test that specification statement just after the subsystem was integrated, at the same time as tests derived from the design?

If the statement’s implementation depends on nothing outside the subsystem, why wait until the whole system is available? Wouldn’t finding the bugs earlier be cheaper?

What’s wrong with the V model?

2006-05-30T17:41:00.000+05:30

I will use the V model as an example of a bad model. I use it because it’s the most familiar.

A typical version of the V model begins by describing software development as
following the stages shown here:

That’s the age-old waterfall model. As a development model, it has a lot of problems. Those don’t concern us here – although it is indicative of the state of testing models that a development model so widely disparaged is the basis for our most common testing model. My criticisms also apply to testing models that are embellishments on better development models, such as the spiral model.

Testing activities are added to the model as follows:

Unit testing checks whether code meets the detailed design. Integration testing checks whether previously tested components fit together. System testing checks if the fully integrated product meets the specification. And acceptance testing checks whether the product meets the final user requirements.

To be fair, users of the V model will often separate test design from test implementation. The test design is done when the appropriate development document is ready.

This model, with its appealing symmetry, has led many people astray.

Software Test Life Cycle (Part 3 of 3)

2006-05-28T13:47:00.000+05:30

4. Test Execution (Unit / Functional Testing Phase):
By this time. the development team would have been completed creation of the work products. Of Course, the work product would still contain bugs. So, in the execution phase - developers would carry out unit testing with testers help, if required. Testers would execute the test plans. Automatic testing Scripts would be completed. Stress and performance Testing would be executed. White box testing, code reviews, etc. would be conducted. As and when bugs are found - reporting would be done.

5. Test Cycle (Re-Testing Phase):
By this time, minimum one test cycle (one round of test execution) would have been completed and bugs would have been reported. Once the development team fixes the bugs, then a second round of testing begins. This testing could be mere correction verification testing, that is checking only that part of the code that has been corrected. It could also be Regression Testing - where the entire work product is tested to verify that correction to the code has not affected other parts of the code.

Hence this process of :
Testing --> Bug reporting --> Bug fixing (and enhancements) --> Retesting
is carried out as planned. Here is where automation tests are extremely useful to repeat the same test cases again and again.During this phase - review of test cases and test plan could also be carried out.

6. Final Testing and Implementation (Code Freeze Phase):
When the exit criteria is achieved or planned test cycles are completed, then final testing is done. Ideally, this is System or Integration testing. Also any remaining Stress and Performance testing is carried out. Inputs for process improvements in terms of software metrics is given. Test reports are prepared. if required, a test release note, releasing the product for roll out could be prepared. Other remaining documentation is completed.

7. Post Implementation (Process Improvement Phase):
This phase, that looks good on paper, is seldom carried out. In this phase, the testing is evaluated and lessons learnt are documented. Software Metrics (Bug Analysis Metrics) are analyzed statistically and conclusions are drawn. Strategies to prevent similar problems in future projects is identified. Process Improvement Suggestions are implemented. Cleaning up of testing environment and Archival of test cases, records and reports are done.

Software Test Life Cycle (Part 2 of 3)

2006-05-28T13:45:00.000+05:30

2.Test Analysis (Documentation Phase)
The Analysis Phase is more an extension of the planning phase. Whereas the planning phase pertains to high level plans - the Analysis phase is where detailed plans are documented. This is when actual test cases and scripts are planned and documented.

This phase can be further broken down into the following steps :

Review Inputs : The requirement specification document, feature specification document and other project planning documents are considered as inputs and the test plan is further disintegrated into smaller level test cases.
Formats : Generally at this phase a functional validation matrix based on Business Requirements is created. Then the test case format is finalized. Also Software Metrics are designed in this stage. Using some kind of software like Microsoft project, the testing timeline along with milestones are created.
Test Cases : Based on the functional validation matrix and other input documents, test cases are written. Also some mapping is done between the features and test cases.
Plan Automation : While creating test cases, those cases that should be automated are identified. Ideally those test cases that are relevant for Regression Testing are identified for automation. Also areas for performance, load and stress testing are identified.
Plan Regression and Correction Verification Testing : The testing cycles, i.e. number of times that testing will be redone to verify that bugs fixed have not introduced newer errors is planned.

3. Test Design (Architecture Document and Review Phase):

One has to realize that the testing life cycle runs parallel to the software development life cycle. So by the time, one reaches this phase - the development team would have created some code or at least some prototype or minimum a design document would be have been created.

Hence in the Test Design (Architecture Document Phase) - all the plans, test cases, etc. from the Analysis phase are revised and finalized. In other words, looking at the work product or design - the test cases, test cycles and other plans are finalized. Newer test cases are added. Also some kind of Risk Assessment Criteria is developed. Also writing of automated testing scripts begin. Finally - the testing reports (especially unit testing reports) are finalized. Quality checkpoints, if any, are included in the test cases based on the SQA Plan.

(Continued...)

Software Test Life Cycle (Part 1 of 3)

2006-05-28T13:41:00.000+05:30

Usually, testing is considered as a part of the System Development Life Cycle, but it can be also termed as Software Testing Life Cycle or Test Development Life Cycle.

Software Testing Life Cycle consists of the following phases:
1. Planning
2. Analysis
3. Design
4. Execution
5. Cycles
6. Final Testing and Implementation
7. Post Implementation

1. Test Planning (Product Definition Phase):
The test plan phase mainly signifies preparation of a test plan. A test plan is a high level planning document derived from the project plan (if one exists) and details the future course of testing. Sometimes, a quality assurance plan - which is more broader in scope than a test plan is also made.

Contents of a Test Plan are as follows :

Scope of testing
Entry Criteria (When testing will begin?)
Exit Criteria (When testing will stop?)
Testing Strategies (Black Box, White Box, etc.)
Testing Levels (Integration testing, Regression testing, etc.)
Limitation (if any)
Planned Reviews and Code Walkthroughs
Testing Techniques (Boundary Value Analysis, Equivalence Partitioning, etc.)
Testing Tools and Databases (Automatic Testing Tools, Performance testing tools)
Reporting (How would bugs be reported)
Milestones
Resources and Training

Contents of a SQA Plan, more broader than a test plan, are as follows :

The IEEE standard for SQA Plan Preparation contains the following outline :

Purpose
Reference Documents
Management
Documentation
Standards, Practices and Conventions
Reviews and Audits
Software Configuration Management
Problem Reporting and Corrective Action (Software Metrics to be used can be identified at this stage)
Tools, Techniques and Methodologies
Code Control
Media Control
Supplier Control
Records, Collection, maintenance and Retention

(Continued...)

SDLC Model: Progressive Development Lifecycle Model

2006-05-22T10:41:00.000+05:30

One of the common problems with software development is that software is needed quickly, but it will take a long time to fully develop.

The solution is to form a compromise between time scales and functionality, providing "interim" deliveries of software, with reduced functionality, but serving as a stepping stones towards the fully functional software. It is also possible to use such a stepping stone approach as a means of reducing risk.

The usual names given to this approach to software development are progressive development or phased implementation. The corresponding lifecycle model is referred to as a progressive development lifecycle. Within a progressive development lifecycle, each individual phase of development will follow its own software development lifecycle, typically using a V or waterfall model.

The actual number of phases will depend upon the development.Each delivery of software will have to pass acceptance testing to verify the software fulfils the relevant parts of the overall requirements. The testing and integration of each phase will require time and effort. Therefore, there is a point at which an increase in the number of development phases will actually become counter productive, giving an increased cost and time scale, which will have to be weighed carefully against the need for an early solution. The software produced by an early phase of the model may never actually be used; it may just serve as a prototype.

A prototype will take short cuts in order to provide a quick means of validating key requirements and verifying critical areas of design. These short cuts may be in areas such as reduced documentation and testing. When such short cuts are taken, it is essential to plan to discard the prototype and implement the next phase from scratch, because the reduced quality of the prototype will not provide a good foundation for continued development.

SDLC Model: V Model

2006-05-22T10:37:00.000+05:30

A variant of the waterfall model — the V-model — associates each development activity with a test or validation at the same level of abstraction.

Each development activity builds a more detailed model of the system than the one before it, and each validation tests a higher abstraction than its predecessor.

SDLC Model: Typical Spiral Model

2006-05-20T21:28:00.000+05:30

Developed by Barry Boehm in 1988. it provides the potential for rapid development of incremental versions of the software. In the spiral model, software is developed in a series of incremental releases. During early iterations , the incremental release might be a paper model or prototype.Each iteration consists ofPlanning, Risk Analysis, Engineering, Construction & Release & Customer Evaluation.

Customer Communication: Tasks required to establish effective communication between developer and customer.Planning: Tasks required to define resources, timelines, and other project related information.Risk Analysis: Tasks required to assess both technical and management risks.Engineering: Tasks required to build one or more representatives of the application.Construction & Release: Tasks required to construct, test, install and provide user support (e.g., documentation and training)Customer evaluation: Tasks required to obtain customer feedback based on evaluation of the software representations created during the engineering stage and implemented during the installation state.

SDLC Model: Classic Waterfall Model

2006-05-20T21:27:00.000+05:30

In a typical model, a project begins with feasibility analysis. On successfully demonstrating the feasibility of a project, the requirements analysis and project planning begins.

The design starts after the requirements analysis is complete, and coding begins after the design is complete. Once the programming is completed, the code is integrated and testing is done. On successful completion of testing, the system is installed.

After this, the regular operation and maintenance of the system takes place.

Software Development Life Cycle (SDLC) Models

2006-05-20T21:21:00.000+05:30

What is Software Development Life Cycle (SDLC)?

The various activities that are undertaken when developing software are commonly modelled as a software development lifecycle. The software development lifecycle begins with the identification of a requirement for software and ends with the formal verification of the developed software against that requirement.

The software development lifecycle does not exist by itself; it is in fact part of an overall product lifecycle. Within the product lifecycle, software will undergo maintenance to correct errors and to comply with changes to requirements. The simplest overall form is where the product is just software, but it can become much more complicated with multiple software developments, each forming part of an overall system to comprise a product.

There are a number of different models for software development lifecycles. Some of the more commonly used models are:
-> Waterfall Lifecycle Model
-> Modified Waterfall Lifecycle Model
-> V Lifecycle Model
-> Progressive Development Lifecycle Model
-> Iterative / Incremental Lifecycle Model
-> Spiral Lifecycle Model
-> Prototyping Model
-> RAD Lifecycle Model