Saturday, May 18, 2002
It is best to do things systematically, since we are only human, and disorder is our worst enemy.Isn't it amazing how something uttered so long ago by a Greek poet is relevant to software testing?
It's beyond question that the Greeks made many lasting contributions to culture and civilization. In the world of test process improvement the lasting contributions may well be coming out of the Netherlands. As an aside, our Dutch brothers and sisters are also making significant contributions to service level management (see my 5 April 2002 entry in our sister weblog, Postcards from the Revolution). The reason I believe that the Dutch are leading the way in test process improvement is because the Test Process Improvement (TPI) and Test Management Approach (TMAP). Each of these approaches are documented in the following books:
Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing. This book provides a coherent process improvement approach for software testing. It provides a model that supports the assessment of strengths and weaknesses of an existing software testing process and an approach for developing and implementing remedial action to rectify the weaknesses. As such this book is not useful to organizations that have not achieved a mature and stable testing process because the model will not apply. If you are seeking a book that will get your processes stable you will find Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel a better place to start.There are a few other books about test process improvement that are worth reading:
However, if your processes are stable this book is among the best because it stays focused on improving the testing process and does so in the same manner that SEI's CMM does for software development. In fact, the TPI approach in this book is cross-referenced to the CMM, which gives you an approach that can be viewed as a testing maturity model that aligns nicely with the CMM (including the newer CMMI). This is one of the strong points of the book and TPI.
Another thing to know about this book is that it's written more like a specification than a narrative. Some readers may find this difficult, but if you are involved in mapping the TPI key process areas to the CMM (or SPICE, Bootstrap or PSM), you'll appreciate the format. Also, the book views TPI as a subset of software process improvement, and software process improvement as a subset of TQM. While the authors focus on the software testing process, they do not isolate it from the bigger picture. This allows you to view then entire quality process as a coherent whole when you're assessing the software testing process and developing improvement strategies.
I personally think this book adds considerably to the software testing body of knowledge, and that the approach the authors give is both practical and sensible. If you work in an organization that has a stable testing process or is at CMM level 2 or above this book is essential reading.
Software Testing: A Guide to the TMAP Approach My first introduction to TMAP was in the above book, which the author co-authored. It piqued my interest, but unfortunately all of the literature on TMAP was written in Dutch. This book makes this powerful test management approach available to English speaking readers, making it invaluable.
First, a little about TMAP to explain why I think the approach is important and useful: It views testing as a process instead of a collection of procedures. The advantage is that once a process is in place it can be stabilized and improved upon. The key to testing is repeatability, and without a process there can be no repeatability. TMAP consists of four elements that combine to form a cohesive test management model:
Each of the above elements and their parts are covered in great detail, resulting in a sound framework for test management. That alone makes this book invaluable, but there are some additional gems that I especially liked:
- Testing life cycle that is aligned to the development life cycle. This life cycle is encapsulated within a planning and control framework that easily fits into the project management activities of the development life cycle.
- Testing techniques - not the techniques used in the execution of test cases, but the techniques employed for defining a test strategy, developing test specifications, and the associated artifacts. This book does cover some basic test execution techniques, but they are not the focus of the book and are not covered in great detail.
- Infrastructure and tools - addresses what are the minimums for an effective test process in the form of environments and tools. If you're establishing a test organization this aspect will be invaluable.
- Organization - how the test organization is structured and how it relates to external functions, such as development, configuration and release management, project management and other major stakeholders.
This book and the first one I discussed above combine to give a complete picture of test management and test process improvement.
- Test point analysis and estimation, which is an estimating method for test effort that is based on function point analysis. This is incredibly valuable because accurate estimation is one of the shortfalls in testing. This alone is reason to buy the book. For more information about Test Point Analysis you can download Test point analysis: a method for test estimation or look through the presentation slides from Conquest 2000, which also includes presentations on TPI and other items of interest. Although off topic, Test Effort Estimation Using Use Case Points is a related approach that fits nicely within the unified process.
- The wealth of checklists - I especially liked the comprehensive list of quality characteristics.
- Testing in maintenance situations - probably the most common situation for software testing and this book covers it well.
- Software Testing in the Real World (see Linda's 21 March 2001 Amazon review for details).
- Lessons Learned in Software Testing
- Surviving the Top Ten Challenges of Software Testing: A People-Oriented Approach
Friday, May 17, 2002
The software testing profession came into its own in 1979 when Glenford Myers published The Art Software Testing. Although this book is still in print (a remarkable feat in itself), it's quaint when compared to what we now have in published works and the body of knowledge. What this book did for the profession is legitimize it as a valid career path and to portray software testing as a profession instead of an activity to which mediocre programmers were exiled. Myers deserves the credit bestowed, but there is an unsung hero in the software testing and quality movement whose prolific writing has had considerable influence: William E. Perry.
Perry was writing about maintenance, testing and quality before Myers' book arrived on the scene, and his 1991 book, Quality Assurance for Information Systems: Methods Tools, and Techniques, is an interesting blend of holistic IT quality and software testing. I still refer to my copy for ideas when I am researching metrics. This book is about mid-point in Perry's publishing career. While his subsequent books focused more on software testing, this one is among the first to cover both software quality assurance and software testing in a coherent manner.
William Lewis' Software Testing and Continuous Quality Improvement that both Linda and I have recently discussed here (and reviewed on Amazon) extends Perry's work with respect to a holistic view of software quality.
Testing vs. SQA. I make the distinction between testing and SQA as follows:
Testing is an activity to find or prevent defects in software using older inspection techniques or more modern preventive techniques. Note that I am not including value judgments in my definition, else I would have ignored the inspection approach. What I want to do is highlight differences between testing and SQA.Based on my definitions testing belongs in the application delivery domain and serves as the boundary between application delivery and service delivery (i.e., production). This is shown in the organizational diagram that Linda and I developed. SQA, in my opinion, should be a function of a program management office (an ideal spot for oversight), or an entirely separate function that reports directly to the CIO.
SQA is an oversight function that collects and analyzes quality data to be used in pursuit of process improvement.
However, software testing is evolving to the point where testing and SQA are becoming blurred. In fact, to put it crudely, finding the boundary between testing and SQA is akin to picking fly shit of pepper. I apologize for that analogy, but it best describes the situation. The two books I've recently discussed, Systematic Software Testing and Introducing Software Testing each integrate testing and SQA, and it looks like the direction that software testing is going to take. There are some strengths and weaknesses to this:
- Strengths: end-to-end quality infrastructure, with a viewpoint that encompasses the entire systems life cycle (not just the development life cycle).
- Weaknesses: misses the big picture because testing is a narrow viewpoint of software quality. Other stakeholders in the service level management and project management domains have different viewpoints. See our Life Cycle Quality Gates document for an overview of metrics we deem important and you'll see why many will not be on the RADAR of a test organization.
Clouds in My Coffee. The way I see it the maturity of the software testing profession, as evidenced by the two books I discussed yesterday, and the affinity of testing and SQA, are on a course that needs to be carefully considered. For small organizations this isn't such an important issue, but for large enterprises the strengths and weaknesses need to be more carefully examined and weighed than I've done in this entry. The good news is we have reached a point where quality is considered to be important and proactive approaches to achieving it are becoming more prevalent. Better yet, thses approaches are wrappd in process.
Where the issues become even more cloudy is in the growing (and excellent) body of knowledge and practices supporting test process improvement. My next entry will focus on that aspect of testing and quality before moving on to software reliability in a future entry.
Have a wonderful weekend!
Thursday, May 16, 2002
Systematic Software Testing. Synopsis: Process-oriented and applicable to test professionals at all levels; test managers will benefit the most.Which to get? Why not both?
This book provides a detailed roadmap for establishing and managing a comprehensive test process that is closely aligned to the IEEE standards for software testing. The process, called Systematic Test and Evaluation Process (STEP) is designed to improve quality by early involvement in the development life cycle instead of having testing as an activity on the critical path at the end of the build phase. This approach ensures early detection of defects, including those introduced in the requirements, specifications and design milestones. Clearly, the STEP approach supports testing and SQA (where SQA is an oversight function outside of the testing domain).
The STEP process has three main steps:
This framework is supported in Chapters 2-8, each of which addresses supporting activities and artifacts in detail. Chapter 2 covers risk analysis since testing is by its nature done to reduce the risk of defects escaping into production systems. I like the way the authors separate technical and schedule risks in this chapter because each are integral to the realities of testing.
- Plan the test strategy (develop a master test plan and associated detailed test plans).
- Acquire testware (define test objectives, design and create test plans).
- Measure (execute the tests, ensure that tests are adequate and monitor the process itself).
Chapters 3 and 4 show how to perform master and detailed test planning, and provide example plan templates and how to develop them, and requirements and factors for each test phase for the detailed planning (unit, integration, system and acceptance testing).
The analysis and design activities covered in chapter 5 are focused on test design. The systematic and structured way the authors approach these activities walks you through developing test cases. You're shown how to ensure that they account for requirements and features, and are given high level advice about how to types of tests to employ. Test implementation covered in Chapter 6 introduces organization and process issues from a team perspective. One of the strongest chapters, 7, does deeply into the issues and factors surrounding test execution, and gives metrics to consider and internal processes for managing defects. I felt that this chapter should have paid more attention to issue and defect management from an enterprise problem management perspective, but despite this the information is solid.
The chapters that will most benefit test managers, especially new ones, are 8 through 10 that address the test organization, people and management issues. These sections would warm the heart of HR professionals and is unique in that leadership is given the same weight as management techniques. The detailed comparison of certifications from ASQ (CSQE), IEEE (CSDP), QAI (CSTE) and IIST (CSTP) includes everything you need to know to select the best certification to pursue, including salary increase data for each of these certifications. I also liked the chapter on improving the test process and thought the discussions of the CMM and the TPI model that is the subject of Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing discussed. The appendices are also valuable in that they provide a glossary and templates that are consistent with IEEE specifications for software testing, and other valuable aids, such as checklists, an example master test plan and process diagrams.
Overall, the 15 years of field experience in teaching testing that is embodied in this book shows. It's practical, captures best practices and provides a solid model for a process-oriented test organization that employs preventive techniques.
Introducing Software Testing. Synopsis: Teaches good habits to new testers, and offers much to experienced test professionals.
I cannot imagine a better introductory book for software testers because this much needed text bypasses the theory that similar books inundate you with and goes straight to the essence of what testers spend most of their time doing: writing test plans and developing test cases. In fact, the first chapter (Tackling the Testing Maze) is the roadmap for the rest of the book, as well as the test process itself. The approach is modern in that it's aligned to iterative development life cycles, which is based on eight stages:
What I like about this book is the no-nonsense approach to developing a test outline from which the test plan(s) and test cases will be derived, and the way that this documentation is aligned to the real world. For example, due diligence in the form of meticulous attention to sign-offs and authorities to proceed is emphasized. This alone is a common failure point in many test organizations. I also like the way that the realities of the project are highlighted, especially the interactions with the development team and the integration of project considerations into the process - in particular, the schedule constraints that all testers must juggle while meeting quality goals.
- Baseline test
- Trend analysis
- Inventory combinations
- Stress the environment
Other areas that make this a realistic look at testing include the chapters on object-oriented and web testing, and the inclusion of security testing - especially the latter which has been neglected in many advanced books and is an important, but overlooked, aspect of the full test suite.
Because this is an introductory text the author uses case studies and copious examples to illustrate and reinforce concepts and activities. But most important, the focus is on activities that reflect what testers do and theory only when required. This makes the book interesting and will give to anyone who follows the approach solid skills that will increase their worth to their team as well as dramatically increase their professional knowledge and skills.
For new testers this is probably the most important book you can buy. If you're a test manager you'll find this book to be an ideal training tool, and if combined with Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel will give you a complete reference library. The approach in the Craig and Jaskiel book is completely consistent with the approach in this one, making both books all the more valuable.
In my next entry I am going to continue this theme and extend it with my thoughts on SQA, software process improvement and software reliability.
Wednesday, May 15, 2002
This goes much deeper than Microsoft's shenanigans. IBM shares the guilt, and based on past history Sun isn't exactly clean either. In this case they are the victim, but do you doubt that they would have been the perpetrator given the opportunity? No, this isn't a Microsoft or IBM sin, it's an indictment of the lack of ethics in our industry and it underscores the reason why we have anti-trust legislation to begin with. However, the courts should not bear the burden of sorting this mess out. We have a responsibility to just say no to technology based on proprietary standards. Until that happens we're going to get what we deserve, and it will be a regression to closed-systems and lack of interoperability.
On a Positive Note. If you develop in the J2EE environment you should be frequently visiting The ServerSide, which contains news, articles and other resources. Registration is free, and two great reasons to register are free PDF copies of:
- Mastering Enterprise JavaBeans (the same material that is contained in the paper book with the same title by Ed Roman, Scott W. Ambler, Tyler Jewell and Floyd Marinescu). The source code that goes with the book is also available for free download.
- EJB Design Patterns in PDF format, which is identical to the paper book with the same title by Floyd Marinescu and Ed Roman. Note: the PDF version of the book has not been put in the download section yet, but you can still get the source code.
The ServerSide has a sister site called The Middleware Company, which also requires registration and also has invaluable resources. Their article library is filled with whitepapers and articles that you'll find useful.
Tuesday, May 14, 2002
The potential for the two giants to erect a toll booth is tied to the likelihood that Web services protocols such as SOAP, WSDL, and UDDI--and the related ones to which the two companies hold patents or other intellectual property rights--will one day be as important as the standard protocols (such as TCP/IP and HTTP) on which the Internet is based today.A month later Mr. Berlind reports that IBM and Microsoft are not going unchallenged. His 7 May article titled Web Services Hero shows that both Hewlett-Packard and Apple are proactively challenging the moves by IBM and Microsoft. While Mr. Berlind's reporting is well written and researched, and his tenacious investigation is a true service, one of his readers, Gary Edwards, summed up the issues and threats in his Reader Talkback. This is important stuff and I think both David Berlind's articles and Mr. Edwards' thoughts merit a careful read and a lot of thought.
Yesterday I singled out Soft Java for its light, humorous approach to teaching Java. I found another site, Java Ranch, that uses the same approach and am now becoming interested in Java. One final note: If you are interested in CASE tools you'll like the collection of Freely Available CASE Tools that I stumbled upon by accident.
The reason I believe that the Craig and Jaskiel book will become the standard reference is based on:
- It not only proposes a preventive testing process (called STEP; Systematic Test and Evaluation Process), but is also aligned to IEEE standards for test documentation and uses IEEE terminology throughout. These accomplish three things: (1) it gives a test process that takes the entire life cycle into account and employs an approach that begins before a single line of code is written, (2) leverages established standards and shows how they can be successfully used in practice, and (3) uses established and standardized terminology.
- The STEP approach is based on risk management, which is missing other books on testing. The up front risk analysis in the test planning phase makes sense when you consider that testing is all about finding and removing defects, which represent risks to the software to be delivered. Finally, someone gets it right!
- The chapters on master and detailed test planning add clarity and consistency to these processes. If you've worked in more than one organization you'll understand the significance of this because it seems that no two organizations approach it the same way, and I have never seen an organization approach it in the logical manner in which it's outlined in this book.
- The same structure and consistency is applied to test implementation and execution - and the combined benefits will promote repeatability, which is a fundamental goal of testing.
- Forms, checklists and templates (unfortunately only in hard copy) that are provided are invaluable. If you tailor them to your own organization you'll have a ready-made set of testware that covers every facet of the QA process.
I also liked the chapters on test management (from a test manager's perspective) and improving the test process. If you are with an organization that is assessed against the CMM or are considering going in that direction, the brief piece on how to align the test process to the CMM is invaluable. If you are familiar with Test Process Improvement approach proposed by Koomen and Pol in Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing, you'll especially like the way that this book cross references STEP to TPI.
Obviously I will have much more to say about this book as I read through it in detail, and after I have I'll post a comprehensive review here. However, I found the book to be so impressive and compelling on the first scan through that I wanted to get the word out that this is, indeed, a book that is essential if you're involved in software testing.
Monday, May 13, 2002
- A structured quality cycle based on Plan-Do-Check-Act. This cycle is the foundation of continuous improvement, which is the theme of the book.
- Complete description of testing techniques. In this respect the book is an encyclopedia for software test professionals and a definitive reference.
- Comprehensive resource for forms and checklists (I wish these were also provided in soft copy on a CD ROM or author's web site, but they are not).
- Full view of metrics across every aspect of the development life cycle. In the same manner that the testing techniques are encyclopedic, the metrics are also an encyclopedia for SQA professionals.
I recently finished reading Business Rules Applied, which covers business rules from an implementation approach, and does so in great detail. If you are new to business rules you should first read Business Rules and Information Systems: Aligning IT with Business Goals by Tony Morgan, which is better for beginners. That book introduces business rules at a basic level.
This book expands Morgan's work by drilling down into details and exposing the nuances that a seasoned practitioner will appreciate. However, the main value of this book is the way Ms. von Halle steps you through the complexities of implementing business rules as an organizational methodology. This is not an easy task, but she manages to provide a complete and comprehensive approach that will guarantee success if carefully followed. I think the work breakdown structure alone that is provided in the book makes it essential to anyone who is tasked with implementing business rules.
In addition, the tables, checklists and documents and information from the book's web site add even more value. This is an important book about an important topic. It's not easy to read, but the diligent reader (assuming prior experience) will find everything he or she needs to know about business rules, the value proposition for using them, and how to implement them. It's the most authoritative book on the subject, and will probably remain so for years to come.
There's always some delightful site to be discovered, and the most recent is Soft Java, which is the creation of two women, Jeannie and Joy who are funny, slightly over the edge and have other similar qualities that will endear them to you. Their site is dedicated to teaching Java to the masses. I'm up to my eyeballs with my Oracle OCP training and am not about to add learning Java to my workload at this time, but when I do have the time and energy I might just return to their site and add Java to my skills.
Software Configuration Management. The clearest overview of SCM that I've found is on a University of Calgary page for a software engineering class. This page also has a PowerPoint presentation on software configuration management that is excellent.
Configuration Management for Software by Stephen B. Compton and Guy Conner is the best book I've read on SCM. This wonderful book was out of print, but a quick check on Amazon shows that it is once again available. If you get one book on SCM this is the one I recommend.
Another source of SCM information, along with software engineering processes by CMM key process area is the Systems Engineering Process Office maintained by the Space and Naval Warfare Systems Center, San Diego. Their document collection is mostly in MS Word and PowerPoint formats.
An Old Friend. Linda and I have a document that we frequently cite. It's titled Life Cycle Quality Gates and provides key metrics for every phase of the development life cycle, with attention paid to production (a phase that is too often ignored). Another of our old standbys, titled Configuration Management - The Big Picture, is a quick reference guide for configuration, change and release management. We've included both the technical and business value of each facet of configuration, change and release management, which is a starting point analyzing the ROI that will result from establishing and managing these critical processes.
Project Management. I've written three fairly comprehensive articles in the 9, 10 and 12 May entries in Postcards from the Revolution. If PM is a topic of interest you should read the articles.
Sunday, May 12, 2002
Julia's current research interests center around M-Commerce, so this collection of links and documents is dedicated to her:
- From IBM, a large collection of articles and papers about wireless technologies, and a related collection of papers about security. While many of the security papers are generic, some are specific to wireless and M-Commerce.
- Internet World's M-Commerce Portal.
- VoiceXML Review.
- A collection of papers (mostly in PDF format): On Designing M-Commerce Applications, Why M-Commerce Requires Robust Middleware, Next Generation of Communications Technology (M-Commerce architecture overview), Consistent M-Commerce Security on Top of GSM-based Data Protocols.
- Proceedings from Mobile Summit 2001 (sponsored by IST)
- Building Trust in Secure M-Commerce (MS Word format.
- V-Commerce Application Design (PDF format)
- Singapore Testing Mobile Location Services.
- Mobile Commerce World.
- M-Commerce Revolution (PDF format).
- Direct Payment Systems for M-Commerce.
- M-Commerce Readiness Checklist (online version), also in MS Word format.
- Wireless Web Magazine.
- Miscellaneous M-Commerce Articles and papers.
- Sun's end-to-end M-Commerce solutions based on Sun ONE.
- The Mobile Internet: How Japan Dialled up and the West Disconnected (see Linda's 17 December 2001 and my 4 April 2002 reviews).
- Mobile Commerce: Opportunities, Applications, and Technologies of Wireless Business
- M Commerce: Technologies, Services, and Business Models
- Managing Internet-Driven Change in International Telecommunications (also see the author's personal page, which contains numerous articles).
- The Wireless Application Protocol: Writing Applications for the Mobile Internet
- GPRS and 3G Wireless Applications: Professional Developer's Guide (see my 16 September 2001 review on Amazon)
- Wireless Application Protocol. The 100 slides in this presentation cover WAP in great detail. Regardless of whether you're designing, developing or managing WAP applications you should grab this presentation and study it.
- M-Commerce: Mobile Applications. All you need to know that's important is contained in this 36-slide presentation.
The first two chapters give reasons why complex event processing (CEP)is essential to the distributed systems that characterize supply chain, e-commerce and internet-enabled applications. They also sort out the key issues and present a paradigm for a global event cloud that is decomposed in subsequent chapters. Instead of providing an in-depth analysis of each chapter, which would make for a lengthly and boring review I'll give the highlights of what I liked:
- Architecture is an important theme throughout the book. In particular the Rapide architecture description language adds formality and structure. The key elements of Rapide are causal event modeling, event patterns/pattern matching and event pattern maps and constraints.
- Events, timing and causality, and their interrelationships, are thoroughly explained. These are the key to understanding the treatment of patterns, rules and constraints that follow, and for tackling the subsequent discussion of complex events and event hierarchies. This is slow reading, but the essence of the book.
- Event processing networks, which are a practical use of the knowledge imparted by this book. Moreover, the two case studies showed real world application of the concepts instead of abstract theory. They reinforced all of the key points made earlier in the book.
Subscribe to Posts [Atom]