QA Process Management - The Challenges Encountered (Summary)

Summarily, the challenges faced during these stages fell into three categories:

• Category I- Team Collaboration: Conducting effective Testing Cycle, generation of good quality Test Cases, sharing and creation of the numerous indispensable information assets of the QA process required a high degree of collaboration among Quality Managers, Developers, Test Case Reviewers and Testers.

• Category II- Information Assets Management: Host of documents formed an integral part of the QA Cycle

• Test Plan
• Detailed Feature Specifications
• Test Case
• Test Case Review Template

Documents generated as an output of the QA Cycle included

• Test Logs
• Defects Tracking Sheet
• Session Reports
• QA Reports

For process consistency and uniformity, well-defined standards in the form of published templates or documentation guidelines were needed for each of these assets. Also mandatory for each, was a create-approve-publish document lifecycle. Post QA Cycle, archiving of assets in a secure, well-organized repository was needed. Search, security and tracking of assets were basic requirements on such a repository.

• Category III- Scheduling, Execution and Tracking of Test Plans: The actual Test Cycle needed to be planned, approved, efficiently executed, and tracked.
In the existing work scenario collaboration and planning happened through formal or informal meetings, asset sharing and management through mails or shared file servers all of which were cumbersome to track, slow and time inefficient.

QA Process Management - The Challenges Encountered

Case Study Example:

An Offshore Service Center in Wipro was providing Testing Services to a desktop software development corporation named ABC* Corp. Three modules of their desktop suite: DeskPro, DeskNet and DeskSecure had to be tested at the Offshore Center. Most of the time the testing of the modules were performed independently but a release of the desktop suite mandated that they work in sync, thus multiplying the problems of the managers.

The problems being faced by this team were analyzed and a QA Process Management Solution was modeled. The tribulations faced are elucidated below.

The Problem Statement

Phase 1: Test Case Generation
ABC* Corp. used to send detailed feature and policy documentation for features to be tested at the Off Shore Center. It would also send guidelines for writing Test Cases (TC) for the feature. Two or three engineers had to collaborate on Test Case generation for each feature as per the guidelines sent by ABC* Corp. The first draft of the Test Cases had to undergo review and approval by peer-teams within or across modules. Review comments had to be incorporated by the authors and the modifications had to be reviewed again.

After the first draft and peer review, Test Cases had to be reviewed by the QA leads of each module and a consolidated TC Review Report for the module was generated. QM had to inspect the modular TC Review Reports, consolidate them and generate a single complete Test Case Review Report. After this the Project Manager (PM) had to approve the test report and dispatch the TCs as well as TC Review Report to ABC* Corp for approval. The client then approved or suggested modifications to Test Cases. Modifications, if needed, were made, reviewed and finally with ABC* Corp’s approval the Test Cases were published.

Phase 2: Test Cycle Planning
QA leads of all the three modules generated respective test plans and communicated it to the Quality Manager (QM). QM used to check for feasibility of plans and plausibility of estimates. Inputs, if any, on plan modification had to be communicated to the QA leads and modifications made had to be reviewed again. Issues, if any, relating to resource availability, reconsideration of deadlines etc, had to be escalated to the Project Manager by the QM. Inputs, if any, from the PM had to be incorporated into the execution plan by the QM and reviewed by the PM again. With the Project Managers’ approval the Test Plans were published.

Phase 3: Test Cycle Execution
The Test Plans, Test Case documents, Log Sheet Templates and guidelines to fill the Log Sheets had to be communicated to test engineers. Testing was carried out and defects were logged into a shared Defect Database. Any ad hoc problem faced by testers during QA process had to be addressed quickly. Progress and pace of testing were tracked. At the end of the test cycle, Test Logs were inspected. Any missing pertinent data in the logs were pointed out to the Testers and complete Log Sheets solicited. Test Logs had to be archived in a shared repository. Module leads were to generate report based on test results logged.

The Quality Manager reviewed each module’s report and consolidated it into a final QA Report. The final Report was submitted to the Project Manager. Tick off Discussions ensued between the QM, PM and the client. Based on severity of bugs reported, product modules - DeskNet, DeskPro and DeskSecure were declared ready for release. The release information and QA Reports were published.

(Continued...)

QA Process Management - Significance of QA Process

Procedurally, the entire QA cycle consists of three stages:

• Test Case Generation: This stage starts quite early in the SDLC, just after the Design Phase. After the feature requirements are frozen and the design agreed upon, Test Cases are generated. These Test Cases are used for feature verification at the end of Development Phase. However this initial draft may undergo several modifications based on tester and end user feedback. Succinctly put, the test cases must test for expected functionality while also ensuring functioning of software as per expectations. The former
  depends on the captured requirement specifications while the latter is the end users’ expectations from the software. Thus maturity of Test Cases is a long drawn process with need for continuous review, updating, and tracking of updating.
• Test Cycle Planning: This stage involves drawing of schedules and resource allocation charts for testing and their approval by supervisors before execution. It gains significance because effectiveness of the Test Cycle depends on balanced deployment of manpower and testing resources like servers, peripherals etc based on criticality of tested features. A timely release requires planning an efficient test schedule.
• Test Cycle Execution: The actual testing takes place in this stage. Activities may include logging of defect(s)/suggestions encountered in testing, test case modifications, analysis of test logs by leads and generation of QA reports based on findings. The test reports are crucial to the actual release of the software.

Significance of QA Process
QA Phase vindicates the engineering effort and ensures completeness of the generated software before it is released for general availability. It forms a very critical node of the Software Development Lifecycle as can be deduced from the activities enumerated above.

• As its chief objective, the QA Phase brings out the lacunae in requirement gathering, design and development of the software through reporting of defects and suggestion of improvements.
• Beyond its main purpose of catching and reporting bugs in the software, QA ensures compliance of functionality with pre-set expectations.
• It also ensures reliability of software over and above the expected functionality.
• It guarantees scalability, concurrent accessibility and satisfactory performance of software under different usage conditions.
• Specialized testing such as Beta Testing validates that the software not only functions as per the defined specifications but also meets customer expectations and market needs. Receiving and incorporating feedback from end users belonging to different market segments brings in competitiveness in the features also.

The actual testing of an application may be of different types viz Black Box Testing, White Box Testing, Graphics Interface Testing etc depending on the type of application. It can also be carried out either manually by a team of Test Engineers or through the use of Test Automation Tools with minimal manual effort.

Irrespective of these specifics, the QA Phase poses several management challenges as is illustrated through an example (check out in the next posts).

QA Process Management - QA Process Activities

The activities and implications of the QA Phase can vary based on the various software development paradigms in the industry. These are depicted in the table below:

1. Software Product Development - A software product release is preceded by an internal QA cycle wherein Test Engineers from the product development group verify the product functionality against the feature specifications gathered from market sources/sales partners. This is followed by Beta Testing of the product by external users - customers, partners, patrons or volunteering testers.
2. Software Application Development or Software Servicing - In this case three levels of testing may be carried out. The party involved in software servicing or application development carries out the first level of testing at the end of the development phase. At this level the developed application’s functionality is verified strictly against client’s specific requirements. This is typically followed by System Integration and testing of the service or application at the client site, either by test engineers from the servicing project or by test engineers hired by the client. If the client is servicing a customer in turn, testing may be done at the end-customer site also by test engineers hired at the customer end.
3. Software Testing Services - This is a category of software services in which the development partner outsources the complete testing of its product, application or service to a third party. Dedicated teams of Test Engineers at the third party site solely perform testing of the developed software.

The example elucidated in this paper falls in the third category but the challenges that are faced and the solution that has been presented are applicable to the other two categories also.

A checklist of model requirements

These requirements are grouped according to testing activity.

A test model should:

1. force a testing reaction to every code handoff in the project.
2. require the test planner to take explicit, accountable action in response to dropped handoffs, new handoffs, and changes to the contents of handoffs.
3. explicitly encourage the use of sources of information other than project
   documentation during test design.
4. allow the test effort to be degraded by poor or late project documentation, but prevent it from being blocked entirely.
5. allow individual tests to be designed using information combined from various sources.
6. allow tests to be redesigned as new sources of information appear.
7. include feedback loops so that test design takes into account what’s learned by running tests.
8. allow testers to consider the possible savings of deferring test execution.
9. allow tests of a component to be executed before the component is fully assembled.