Hello Folks!!!

            In this blog, we would like to talk about the new standard of ISO which is ISO-29119 and why this standard is not suitable for Software Testing or Software Test Professionals.

Firstly it is important to know or understand “What a Standard is?”

“A standard is a document that provides requirements, specifications, guidelines or characteristics that can be used consistently to ensure that materials, products, processes and services are fit for their purpose.”

Standards are basically a good thing to have as they define

a)   A shared understanding / definitions

b)   Helps in connecting to the world

c)   Acts as a repository of collective experience

d)   Protects suppliers from unscrupulous competition

e)   Protects customers from unscrupulous suppliers

ISO has published over 19500 International Standards covering almost all aspects of technology and business and now everyone might be already aware that there is a new software testing standard for the world i.e. “ISO29119”.

Before we go further into the details of this standard, it is important to gain an understanding on the definition of the standard ISO29119.

“The Standard ISO29119 was jointly produced by the two organizations – ISO and IEEE and is a result of years of work by members of various countries & inputs by experts within the IEEE. ISO states the standard as "The ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed set of standards for software testing that can be used by any organization when performing any form of software testing"

The ISO29119 standard is “open ended” for all type of users and recognizes many types of software organizations producing software and development methodologies for e.g., information technology, embedded systems, mobile, scientific, agile, waterfall, large and small production organizations and these are some of the important factors that mainly influences Software Testing.

The ISO29119 standard has 5 parts of which three parts were released in the year 2013 and the fourth part is scheduled to be released in 2014. All the parts can be used standalone without depending on the other parts, although in some cases there are some dependencies between the parts for e.g. Documents produced as part of “Processes” in Part2 are defined in Part3. Every part contains – “definitions”, “concepts”, “approaches”, “supplemental information and examples”.

A brief overview on each part:–

Part1 (Concept and Definition):– describes the basic test definitions and concepts which is a base for the other parts of the series. Part1 also describes the various test methods, test approaches and other test concepts to be used and “risk-based” testing is the primary basis for ISO29119. Some other test bases have also been recognized such as math-based, model-based and experience-based.

In addition to all these, general software testing concepts are also presented which includes the role of software testing in an organization, project context considerations, testing within different lifecycles and test planning.

Part2 (Test Processes):- is the heart of the series where as it covers the software testing processes at the organizations, test management and test execution levels. Risk-based testing is an approach which is useful to prepare strategy and to manage testing that allows testing with priority and focus. It includes planning, risk analysis, test design strategy, scheduling (or) staffing, and document production.

Part3 (Test Documentation):- consists of an extensive set of documents and is based on IEEE 829 Software Test Documentation. The standard consists sixteen documents with outlines and content descriptions. But most of the projects would not produce all of these documents, tailoring to only the documents of use to them. The basic test documentation names are as below:   

• Organization test documentation : Test Policy, Test Strategy
• Project test documentation : Project test plan, Test project completion report
• Test level documentation : Test plan, Test specification, Test results, Anomaly reports, Level test status report, test environment report and test level completion report.

Part4 (Test Techniques):- Test Techniques is to define one international standard covering software test design techniques that can be used during the test design and implementation process within any organization or software development life cycle model. The various test techniques are: Specification-Based Testing Techniques, Structure-Based Testing Techniques and Experience-Based Testing Techniques

Part5 (Keyword-driven testing):– The aim is to define an international standard for supporting Keyword-Driven Testing which is a way of describing test cases by using a predefined set of Keywords and these are names which are associated with a set of actions that are required to perform a specific step in a test case. By using keywords to describe test steps instead of natural language, test cases can be easier to understand, to maintain and to automate.

To summarize the above, based on the definitions of Part 1-5, we can say that the ISO 29119 Standard requires advanced preparation and documentation for software testing. As detailed in the Part3, the standard involves creation of documentation at different levels of the project life cycle. The Standard ISO-29119 is mainly defined to “Standardize Software Testing”.

          One of the description of the standard states “By implementing these standards, you will be adopting the only internationally-recognized and agreed standards for software testing, which will provide your organization with a high-quality approach to testing that can be communicated throughout the world”

“Why is the ISO-29119 standard not suitable for Software Testing?”

          In this section of the blog, we would bring our insights on the negatives of this standard or “Why is the ISO-29119 standard not suitable for Software Testing” based on the understanding we gained by reading several blogs which are against this standard and to which we agree as well.

We, as software test professionals believe that software testing is an “Art”, a “skilled activity” and the only true measurement of testing is the skill exhibited in live practice. Testing Standards are a good thing to have in any organization, but the ISO-29119 is set of standards for “documentation of testing or testing processes”. The stated aim of the ISO-29119 standard is to “define a set of standards that can be used by any organization when performing any form of software testing” – here any form of testing, in any organization, in any context relates directly to “US” – Software test professionals.

This standardized approach to software testing puts too much emphasis on process and documentation rather than the real testing. "Of course, that might not be its purpose or the intention of the people who developed it”. However, it is very common to see how people would react when they are dealing with a messy, complex problem and there are detailed, prescriptive standards and processes in hand. In such situations we tend to focus on complying with the standard to solve the problem and we lose sight of the real goal.

Standards in the field of manufacturing makes sense, however it does not apply to services, where demand is highly variable, or indeed in software, where every instance of demand is unique. “ISO Standards works well in manufacturing world but not in Software world”. Applying the processes and techniques of ISO 29119 standards will result in effort being expanded on activities that basically do nothing to perform testing and this can become a major problem in field of testing. Generally when we are testing, we do the testing with a purpose: for e.g. it could be to discover and share information related to the quality. Any activity, any effort that doesn’t contribute to doing so is considered a “waste”. For a project which is constrained by quality, this translates into increased time and cost. For a project which is constrained by time or money, this translates into a reduction in the information and a corresponding increase in risks to quality.

Training testers to use a standard has a tendency of framing their thinking or bounding their thinking within the standard. When some problems are encountered during the testing process rather than trying to solve testing problems, they easily tend to resolve the problem by choosing the solution from a set of ready-made solutions that may or may not fit their purpose. This directly leads to risk in the quality. Testing standards should focus on skills assessed by qualified practitioners. The standard might define boundaries for the levels of testing skill required to work. To meet such a standard requires classroom trainings along with on-the-job practical training, and judging live testing. At successful conclusion, a tester could be certified as a professional. Very skilled testers could become master testers, in demand for very high-risk software.

We would like to pen an interesting example we found in one of the blog posts… through which we can say that the standard ISO-29119 or in general standardizing software testing is not a good thought.

             "A highly informal experiment was conducted with two groups of students. The first group was trained on a set of formal test design techniques. The second group received a short briefing on some general testing principles and the use of the heuristic test strategy model. Both the groups were then tasked with creating a set of test ideas for the same product. The difference between the ideas generated by the two groups was a stark. The first group came up with a predictable set of equivalence classes etc., while the second group came up with a rich and varied set of ideas. When released, and if widely adopted, part 4 of ISO-29119(on test techniques) will give rise to a generation of testers locked firmly inside the 29119 box, without the ability or freedom to solve the problems they need to solve.”

Some of the other places where the standard ISO-29119 cannot be applied are: - “when we are testing an application software” or “if we were doing some short-term consulting (in a day or two) where we test or evaluate a product using our experience base” and “where we could probably carry out an exploratory testing approach which does not involve documentation”.

The ISO-29119 standard seems to view software testing as a manufacturing discipline and does not focus on skills and judgments to identify what is important, observing carefully in diverse ways, and reporting results appropriately. It mainly focuses on planning, monitoring and control; and not about what should be tested, and how the provided information will bring value. It gives an impression that testing follows a straight line, but in reality it is much more complicated. The standard includes many documentations and rules that are reasonable in some situations, but often are a waste of time. Good, useful documentation is good and useful, but following the standard will lead to documentation for its own sake. Purpose of a standard should make “Testing better” which is possible by having a more flexible standard.

The biggest risk with the standard is that it will lead to less testing as testing will be forced down to a common, low standard service. It will be low quality, low status work.

“We windup this blog post with an opinion that any standardization of testing will harm the quality of software produced by organizations that adopt the standards. Reliance on a standard leads organizations to expect heavy documentation, which in turn drives testers to focus on the production of heavy documentation. This focus on documentation will pull the testers away from the real job of software testing. Following ISO 29119 is far from making testing more efficient or effective; conformance with the standard actually will have the opposite effect”.