TEAM Zarate Tarrani Notes From the Field

Picking Up. My last entry opened the door to test process improvement, which is summed up in a 2800 year old quote by Hesiod:

It is best to do things systematically, since we are only human, and disorder is our worst enemy.

Isn't it amazing how something uttered so long ago by a Greek poet is relevant to software testing?

It's beyond question that the Greeks made many lasting contributions to culture and civilization. In the world of test process improvement the lasting contributions may well be coming out of the Netherlands. As an aside, our Dutch brothers and sisters are also making significant contributions to service level management (see my 5 April 2002 entry in our sister weblog, Postcards from the Revolution). The reason I believe that the Dutch are leading the way in test process improvement is because the Test Process Improvement (TPI) and Test Management Approach (TMAP). Each of these approaches are documented in the following books:

Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing. This book provides a coherent process improvement approach for software testing. It provides a model that supports the assessment of strengths and weaknesses of an existing software testing process and an approach for developing and implementing remedial action to rectify the weaknesses. As such this book is not useful to organizations that have not achieved a mature and stable testing process because the model will not apply. If you are seeking a book that will get your processes stable you will find Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel a better place to start.

However, if your processes are stable this book is among the best because it stays focused on improving the testing process and does so in the same manner that SEI's CMM does for software development. In fact, the TPI approach in this book is cross-referenced to the CMM, which gives you an approach that can be viewed as a testing maturity model that aligns nicely with the CMM (including the newer CMMI). This is one of the strong points of the book and TPI.

Another thing to know about this book is that it's written more like a specification than a narrative. Some readers may find this difficult, but if you are involved in mapping the TPI key process areas to the CMM (or SPICE, Bootstrap or PSM), you'll appreciate the format. Also, the book views TPI as a subset of software process improvement, and software process improvement as a subset of TQM. While the authors focus on the software testing process, they do not isolate it from the bigger picture. This allows you to view then entire quality process as a coherent whole when you're assessing the software testing process and developing improvement strategies.

I personally think this book adds considerably to the software testing body of knowledge, and that the approach the authors give is both practical and sensible. If you work in an organization that has a stable testing process or is at CMM level 2 or above this book is essential reading.

Software Testing: A Guide to the TMAP Approach My first introduction to TMAP was in the above book, which the author co-authored. It piqued my interest, but unfortunately all of the literature on TMAP was written in Dutch. This book makes this powerful test management approach available to English speaking readers, making it invaluable.

First, a little about TMAP to explain why I think the approach is important and useful: It views testing as a process instead of a collection of procedures. The advantage is that once a process is in place it can be stabilized and improved upon. The key to testing is repeatability, and without a process there can be no repeatability. TMAP consists of four elements that combine to form a cohesive test management model:

1. Testing life cycle that is aligned to the development life cycle. This life cycle is encapsulated within a planning and control framework that easily fits into the project management activities of the development life cycle.
2. Testing techniques - not the techniques used in the execution of test cases, but the techniques employed for defining a test strategy, developing test specifications, and the associated artifacts. This book does cover some basic test execution techniques, but they are not the focus of the book and are not covered in great detail.
3. Infrastructure and tools - addresses what are the minimums for an effective test process in the form of environments and tools. If you're establishing a test organization this aspect will be invaluable.
4. Organization - how the test organization is structured and how it relates to external functions, such as development, configuration and release management, project management and other major stakeholders.

Each of the above elements and their parts are covered in great detail, resulting in a sound framework for test management. That alone makes this book invaluable, but there are some additional gems that I especially liked:

• Test point analysis and estimation, which is an estimating method for test effort that is based on function point analysis. This is incredibly valuable because accurate estimation is one of the shortfalls in testing. This alone is reason to buy the book. For more information about Test Point Analysis you can download Test point analysis: a method for test estimation or look through the presentation slides from Conquest 2000, which also includes presentations on TPI and other items of interest. Although off topic, Test Effort Estimation Using Use Case Points is a related approach that fits nicely within the unified process.
• The wealth of checklists - I especially liked the comprehensive list of quality characteristics.
• Testing in maintenance situations - probably the most common situation for software testing and this book covers it well.

This book and the first one I discussed above combine to give a complete picture of test management and test process improvement.

There are a few other books about test process improvement that are worth reading:

• Software Testing in the Real World (see Linda's 21 March 2001 Amazon review for details).
• Lessons Learned in Software Testing
• Surviving the Top Ten Challenges of Software Testing: A People-Oriented Approach

While the last two are more slanted towards advice and examples, they do promote process improvement by showing what does and does not work.

Testing, Quality and Process. In our 13, 14 and 16 May entries Linda and I have taken turns discussing quality- and testing-related books.

The software testing profession came into its own in 1979 when Glenford Myers published The Art Software Testing. Although this book is still in print (a remarkable feat in itself), it's quaint when compared to what we now have in published works and the body of knowledge. What this book did for the profession is legitimize it as a valid career path and to portray software testing as a profession instead of an activity to which mediocre programmers were exiled. Myers deserves the credit bestowed, but there is an unsung hero in the software testing and quality movement whose prolific writing has had considerable influence: William E. Perry.

Perry was writing about maintenance, testing and quality before Myers' book arrived on the scene, and his 1991 book, Quality Assurance for Information Systems: Methods Tools, and Techniques, is an interesting blend of holistic IT quality and software testing. I still refer to my copy for ideas when I am researching metrics. This book is about mid-point in Perry's publishing career. While his subsequent books focused more on software testing, this one is among the first to cover both software quality assurance and software testing in a coherent manner.

William Lewis' Software Testing and Continuous Quality Improvement that both Linda and I have recently discussed here (and reviewed on Amazon) extends Perry's work with respect to a holistic view of software quality.

Testing vs. SQA. I make the distinction between testing and SQA as follows:

Testing is an activity to find or prevent defects in software using older inspection techniques or more modern preventive techniques. Note that I am not including value judgments in my definition, else I would have ignored the inspection approach. What I want to do is highlight differences between testing and SQA.

SQA is an oversight function that collects and analyzes quality data to be used in pursuit of process improvement.

Based on my definitions testing belongs in the application delivery domain and serves as the boundary between application delivery and service delivery (i.e., production). This is shown in the organizational diagram that Linda and I developed. SQA, in my opinion, should be a function of a program management office (an ideal spot for oversight), or an entirely separate function that reports directly to the CIO.

However, software testing is evolving to the point where testing and SQA are becoming blurred. In fact, to put it crudely, finding the boundary between testing and SQA is akin to picking fly shit of pepper. I apologize for that analogy, but it best describes the situation. The two books I've recently discussed, Systematic Software Testing and Introducing Software Testing each integrate testing and SQA, and it looks like the direction that software testing is going to take. There are some strengths and weaknesses to this:

• Strengths: end-to-end quality infrastructure, with a viewpoint that encompasses the entire systems life cycle (not just the development life cycle).
• Weaknesses: misses the big picture because testing is a narrow viewpoint of software quality. Other stakeholders in the service level management and project management domains have different viewpoints. See our Life Cycle Quality Gates document for an overview of metrics we deem important and you'll see why many will not be on the RADAR of a test organization.

I fall on the side of centralized SQA as an oversight function. I believe that Edward Deming was correct when he stated, [I]f the measurements you’re using are unfair, inconsistent and not within the control of the person being evaluated then you will demoralize and de-motivate your employees. Testers should be concerned with testing, not the politics of metrics. In fact, Craig and Jaskiel raise this as an issue (in different words) in Systematic Software Testing.

Clouds in My Coffee. The way I see it the maturity of the software testing profession, as evidenced by the two books I discussed yesterday, and the affinity of testing and SQA, are on a course that needs to be carefully considered. For small organizations this isn't such an important issue, but for large enterprises the strengths and weaknesses need to be more carefully examined and weighed than I've done in this entry. The good news is we have reached a point where quality is considered to be important and proactive approaches to achieving it are becoming more prevalent. Better yet, thses approaches are wrappd in process.

Where the issues become even more cloudy is in the growing (and excellent) body of knowledge and practices supporting test process improvement. My next entry will focus on that aspect of testing and quality before moving on to software reliability in a future entry.

Have a wonderful weekend!

Proverbs. There is a proverb about a jackass that starved to death while standing between two bales of hay because it couldn't decide which one to eat first. The moral is to make a decision and move forward. My dilemma is that I have two new books on software testing and I am having difficulty in deciding which is better. At the risk of starving with such delectable food for the mind within reach I am going to give my opinions about the merits and best audience for each. The books are: Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel (see my 14 May entry) and Introducing Software Testing by Louise Tamres. One thing is clear: May 2002 will go down as the month and year that the software testing body of knowledge dramatically improved. That said, here are my thoughts about each of the books:

Systematic Software Testing. Synopsis: Process-oriented and applicable to test professionals at all levels; test managers will benefit the most.

This book provides a detailed roadmap for establishing and managing a comprehensive test process that is closely aligned to the IEEE standards for software testing. The process, called Systematic Test and Evaluation Process (STEP) is designed to improve quality by early involvement in the development life cycle instead of having testing as an activity on the critical path at the end of the build phase. This approach ensures early detection of defects, including those introduced in the requirements, specifications and design milestones. Clearly, the STEP approach supports testing and SQA (where SQA is an oversight function outside of the testing domain).

The STEP process has three main steps:

1. Plan the test strategy (develop a master test plan and associated detailed test plans).
2. Acquire testware (define test objectives, design and create test plans).
3. Measure (execute the tests, ensure that tests are adequate and monitor the process itself).

This framework is supported in Chapters 2-8, each of which addresses supporting activities and artifacts in detail. Chapter 2 covers risk analysis since testing is by its nature done to reduce the risk of defects escaping into production systems. I like the way the authors separate technical and schedule risks in this chapter because each are integral to the realities of testing.

Chapters 3 and 4 show how to perform master and detailed test planning, and provide example plan templates and how to develop them, and requirements and factors for each test phase for the detailed planning (unit, integration, system and acceptance testing).

The analysis and design activities covered in chapter 5 are focused on test design. The systematic and structured way the authors approach these activities walks you through developing test cases. You're shown how to ensure that they account for requirements and features, and are given high level advice about how to types of tests to employ. Test implementation covered in Chapter 6 introduces organization and process issues from a team perspective. One of the strongest chapters, 7, does deeply into the issues and factors surrounding test execution, and gives metrics to consider and internal processes for managing defects. I felt that this chapter should have paid more attention to issue and defect management from an enterprise problem management perspective, but despite this the information is solid.

The chapters that will most benefit test managers, especially new ones, are 8 through 10 that address the test organization, people and management issues. These sections would warm the heart of HR professionals and is unique in that leadership is given the same weight as management techniques. The detailed comparison of certifications from ASQ (CSQE), IEEE (CSDP), QAI (CSTE) and IIST (CSTP) includes everything you need to know to select the best certification to pursue, including salary increase data for each of these certifications. I also liked the chapter on improving the test process and thought the discussions of the CMM and the TPI model that is the subject of Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing discussed. The appendices are also valuable in that they provide a glossary and templates that are consistent with IEEE specifications for software testing, and other valuable aids, such as checklists, an example master test plan and process diagrams.

Overall, the 15 years of field experience in teaching testing that is embodied in this book shows. It's practical, captures best practices and provides a solid model for a process-oriented test organization that employs preventive techniques.

Introducing Software Testing. Synopsis: Teaches good habits to new testers, and offers much to experienced test professionals.

I cannot imagine a better introductory book for software testers because this much needed text bypasses the theory that similar books inundate you with and goes straight to the essence of what testers spend most of their time doing: writing test plans and developing test cases. In fact, the first chapter (Tackling the Testing Maze) is the roadmap for the rest of the book, as well as the test process itself. The approach is modern in that it's aligned to iterative development life cycles, which is based on eight stages:

1. Exploration
2. Baseline test
3. Trend analysis
4. Inventory
5. Inventory combinations
6. Boundaries
7. Data
8. Stress the environment

What I like about this book is the no-nonsense approach to developing a test outline from which the test plan(s) and test cases will be derived, and the way that this documentation is aligned to the real world. For example, due diligence in the form of meticulous attention to sign-offs and authorities to proceed is emphasized. This alone is a common failure point in many test organizations. I also like the way that the realities of the project are highlighted, especially the interactions with the development team and the integration of project considerations into the process - in particular, the schedule constraints that all testers must juggle while meeting quality goals.

Other areas that make this a realistic look at testing include the chapters on object-oriented and web testing, and the inclusion of security testing - especially the latter which has been neglected in many advanced books and is an important, but overlooked, aspect of the full test suite.

Because this is an introductory text the author uses case studies and copious examples to illustrate and reinforce concepts and activities. But most important, the focus is on activities that reflect what testers do and theory only when required. This makes the book interesting and will give to anyone who follows the approach solid skills that will increase their worth to their team as well as dramatically increase their professional knowledge and skills.

For new testers this is probably the most important book you can buy. If you're a test manager you'll find this book to be an ideal training tool, and if combined with Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel will give you a complete reference library. The approach in the Craig and Jaskiel book is completely consistent with the approach in this one, making both books all the more valuable.

Which to get? Why not both?

In my next entry I am going to continue this theme and extend it with my thoughts on SQA, software process improvement and software reliability.

Foul Play. Linda's last entry highlighted some disturbing behind-the-scenes maneuvering that, frankly, are a threat to open computing and interoperability. A 14 May article By Wylie Wong titled Microsoft ploy to block Sun exposed uncovers more foul play.

This goes much deeper than Microsoft's shenanigans. IBM shares the guilt, and based on past history Sun isn't exactly clean either. In this case they are the victim, but do you doubt that they would have been the perpetrator given the opportunity? No, this isn't a Microsoft or IBM sin, it's an indictment of the lack of ethics in our industry and it underscores the reason why we have anti-trust legislation to begin with. However, the courts should not bear the burden of sorting this mess out. We have a responsibility to just say no to technology based on proprietary standards. Until that happens we're going to get what we deserve, and it will be a regression to closed-systems and lack of interoperability.

On a Positive Note. If you develop in the J2EE environment you should be frequently visiting The ServerSide, which contains news, articles and other resources. Registration is free, and two great reasons to register are free PDF copies of:

1. Mastering Enterprise JavaBeans (the same material that is contained in the paper book with the same title by Ed Roman, Scott W. Ambler, Tyler Jewell and Floyd Marinescu). The source code that goes with the book is also available for free download.
2. EJB Design Patterns in PDF format, which is identical to the paper book with the same title by Floyd Marinescu and Ed Roman. Note: the PDF version of the book has not been put in the download section yet, but you can still get the source code.

Remember, you need to register for a free account to download these valuable books and source code. You'll also get access to discussion boards and articles, such as Critical measures when beginning a J2EE project, that are sure to increase your knowledge (not to mention saving money on books).

The ServerSide has a sister site called The Middleware Company, which also requires registration and also has invaluable resources. Their article library is filled with whitepapers and articles that you'll find useful.

Linda's thoughts about Software Testing and Continuous Quality Improvement in the previous entry are on the mark. The book takes a wide look at software quality improvement across the life cycle and wraps it into a continuous improvement process. I just received a review copy of Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel and have to exclaim, what a difference a day makes. I've only had this book for 18 hours as I write this, and in my opinion it's destined to become the standard reference for software testing. It won't completely supplant Software Testing and Continuous Quality Improvement because, as Linda pointed out, that book's encyclopediac format for testing techniques, and the complete picture of pre- and post-production metrics make it useful in its own right.

The reason I believe that the Craig and Jaskiel book will become the standard reference is based on:

• It not only proposes a preventive testing process (called STEP; Systematic Test and Evaluation Process), but is also aligned to IEEE standards for test documentation and uses IEEE terminology throughout. These accomplish three things: (1) it gives a test process that takes the entire life cycle into account and employs an approach that begins before a single line of code is written, (2) leverages established standards and shows how they can be successfully used in practice, and (3) uses established and standardized terminology.
• The STEP approach is based on risk management, which is missing other books on testing. The up front risk analysis in the test planning phase makes sense when you consider that testing is all about finding and removing defects, which represent risks to the software to be delivered. Finally, someone gets it right!
• The chapters on master and detailed test planning add clarity and consistency to these processes. If you've worked in more than one organization you'll understand the significance of this because it seems that no two organizations approach it the same way, and I have never seen an organization approach it in the logical manner in which it's outlined in this book.
• The same structure and consistency is applied to test implementation and execution - and the combined benefits will promote repeatability, which is a fundamental goal of testing.
• Forms, checklists and templates (unfortunately only in hard copy) that are provided are invaluable. If you tailor them to your own organization you'll have a ready-made set of testware that covers every facet of the QA process.

A few other highlights that I picked up in the 18 hours since I've had this book include: a side-by-side comparison of software quality and test certifications. There were some interesting surprises here. For example, there are a little over 1900 ASQ CSQEs, compared to over 2200 QAI CSTEs, which means that the I lost the debate with Manisha Saboo of eRunway over which was the most prevalent certification. Another fact that emerged from the comparison makes Manisha's arguments in favor of CSTE even more compelling is the average salary increase for those who attain one or the other certification, which is a mere 3% for CSQE and 19% for CSTE. The two other certification programs compared in the chart, IEEE CSDP and International Institute for Software Testing CSTP, are relatively new certifications with less than 200 certified professionals each.

I also liked the chapters on test management (from a test manager's perspective) and improving the test process. If you are with an organization that is assessed against the CMM or are considering going in that direction, the brief piece on how to align the test process to the CMM is invaluable. If you are familiar with Test Process Improvement approach proposed by Koomen and Pol in Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing, you'll especially like the way that this book cross references STEP to TPI.

Obviously I will have much more to say about this book as I read through it in detail, and after I have I'll post a comprehensive review here. However, I found the book to be so impressive and compelling on the first scan through that I wanted to get the word out that this is, indeed, a book that is essential if you're involved in software testing.

Monday Morning Postscript. I forgot to include an interesting assessment of Extreme Programming within the context of CMM level 2. If this topic interests you, you'll also want the associated PowerPoint presentation and Excel worksheet that were used in connection with the assessment. An interesting viewpoint is contained in an article by Mark C. Paulk (Software Engineering Institute's CMM guru) titled Extreme Programming from a CMM Perspective.

Software Process Improvement. There are three outstanding sources of information for SPI:

1. David F. Rico's home page.
2. Tantra Management Services.
3. European Software Process Improvement.

The last one contains documents by categories ranging from assessments to strategies and is one of the best collections of information I've seen in a long time.

Software Configuration Management. The clearest overview of SCM that I've found is on a University of Calgary page for a software engineering class. This page also has a PowerPoint presentation on software configuration management that is excellent.

Configuration Management for Software by Stephen B. Compton and Guy Conner is the best book I've read on SCM. This wonderful book was out of print, but a quick check on Amazon shows that it is once again available. If you get one book on SCM this is the one I recommend.

Another source of SCM information, along with software engineering processes by CMM key process area is the Systems Engineering Process Office maintained by the Space and Naval Warfare Systems Center, San Diego. Their document collection is mostly in MS Word and PowerPoint formats.

An Old Friend. Linda and I have a document that we frequently cite. It's titled Life Cycle Quality Gates and provides key metrics for every phase of the development life cycle, with attention paid to production (a phase that is too often ignored). Another of our old standbys, titled Configuration Management - The Big Picture, is a quick reference guide for configuration, change and release management. We've included both the technical and business value of each facet of configuration, change and release management, which is a starting point analyzing the ROI that will result from establishing and managing these critical processes.

Project Management. I've written three fairly comprehensive articles in the 9, 10 and 12 May entries in Postcards from the Revolution. If PM is a topic of interest you should read the articles.

Small World. This entry is dedicated to a good friend, Julia Jamal, who is a consultant at iPerintis in Malaysia. What makes this world so small is the fact that I met Julia via e-mail when she commented on one of my Amazon book reviews. That in itself is not uncommon, even though she lives half way around the world from me. What we did discover through a chance encounter is that while we are separated by distance and culture, we have many things in common: we're in the same business (IT consulting), we're both Muslim and I speak a language (Tagalog, the national language of the Philippines) that is derived from Malay.

Julia's current research interests center around M-Commerce, so this collection of links and documents is dedicated to her:

• From IBM, a large collection of articles and papers about wireless technologies, and a related collection of papers about security. While many of the security papers are generic, some are specific to wireless and M-Commerce.
• Internet World's M-Commerce Portal.
• VoiceXML Review.
• A collection of papers (mostly in PDF format): On Designing M-Commerce Applications, Why M-Commerce Requires Robust Middleware, Next Generation of Communications Technology (M-Commerce architecture overview), Consistent M-Commerce Security on Top of GSM-based Data Protocols.
• Proceedings from Mobile Summit 2001 (sponsored by IST)
• Building Trust in Secure M-Commerce (MS Word format.
• V-Commerce Application Design (PDF format)
• Singapore Testing Mobile Location Services.
• Mobile Commerce World.
• M-Commerce Revolution (PDF format).
• Direct Payment Systems for M-Commerce.
• M-Commerce Readiness Checklist (online version), also in MS Word format.
• Wireless Web Magazine.
• Miscellaneous M-Commerce Articles and papers.
• Sun's end-to-end M-Commerce solutions based on Sun ONE.

There are also books on the subject that I either have read and reviewed or have been highly recommended by colleagues:

• The Mobile Internet: How Japan Dialled up and the West Disconnected (see Linda's 17 December 2001 and my 4 April 2002 reviews).
• Mobile Commerce: Opportunities, Applications, and Technologies of Wireless Business
• M Commerce: Technologies, Services, and Business Models
• Managing Internet-Driven Change in International Telecommunications (also see the author's personal page, which contains numerous articles).
• The Wireless Application Protocol: Writing Applications for the Mobile Internet
• GPRS and 3G Wireless Applications: Professional Developer's Guide (see my 16 September 2001 review on Amazon)

In addition to the above, Sridhar Iyer's personal page contains an impressive collection of resources, including two excellent PowerPoint presentations:

1. Wireless Application Protocol. The 100 slides in this presentation cover WAP in great detail. Regardless of whether you're designing, developing or managing WAP applications you should grab this presentation and study it.
2. M-Commerce: Mobile Applications. All you need to know that's important is contained in this 36-slide presentation.

I'll be posting more focused entries about M-Commerce in future entries. This collection will provide ample background material across a wide spectrum of technical and business factors.

Architects and developers will gain a wealth of knowledge from a new book titled The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. The short description is: adds a structure, formal approach to system and business events. This book applies structured engineering methods to systems and software engineering, making it a unique and much needed addition to the body of knowledge. Prior to this book event processing was in the domain of embedded and realtime systems developers and hardware designers. This book shows how to effectively use these techniques in IT.

The first two chapters give reasons why complex event processing (CEP)is essential to the distributed systems that characterize supply chain, e-commerce and internet-enabled applications. They also sort out the key issues and present a paradigm for a global event cloud that is decomposed in subsequent chapters. Instead of providing an in-depth analysis of each chapter, which would make for a lengthly and boring review I'll give the highlights of what I liked:

• Architecture is an important theme throughout the book. In particular the Rapide architecture description language adds formality and structure. The key elements of Rapide are causal event modeling, event patterns/pattern matching and event pattern maps and constraints.
• Events, timing and causality, and their interrelationships, are thoroughly explained. These are the key to understanding the treatment of patterns, rules and constraints that follow, and for tackling the subsequent discussion of complex events and event hierarchies. This is slow reading, but the essence of the book.
• Event processing networks, which are a practical use of the knowledge imparted by this book. Moreover, the two case studies showed real world application of the concepts instead of abstract theory. They reinforced all of the key points made earlier in the book.

CEP is particularly applicable to enterprise application integration projects that depend on business events and network and systems management instrumentation (especially developers who write Tivoli software adapters, develop network monitoring solutions or similar endeavors).