Black Box Software Testing: Bug Advocacy

Black Box Software Testing
Bug Advocacy
How to
Win Friends,
influence programmers
and
SToMp BUGs.
ASSIGNED READING: Kaner, Bach, Pettichord, Bug Advocacy 179 Black Box Software Testing Copyright 2003 Cem Kaner & James Bach
Copyright Notice
These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.
Copyright
2003
Cem Kaner & James Bach
180
Software Errors: What is Quality?

Here are some of the traditional definitions: Fitness for use (Dr. Joseph M. Juran) The totality of features and characteristics of a product that bear on its ability to satisfy a given need (ASQ) Conformance with requirements (Philip Cosby) The total composite product and service characteristics of marketing, engineering, manufacturing and maintenance through which the product and service in use will meet expectations of the customer (Armand V. Feigenbaum) Note the absence of conforms to specifications.
Copyright
2003
181
Software Errors: What is Quality?

Juran distinguishes between Customer Satisfiers and Dissatisfiers as key dimensions of quality:
Customer Satisfiers
the right features adequate instruction
Dissatisfiers
unreliable hard to use too slow incompatible with the customers equipment
Copyright
2003
182
Software Errors: What Should We Report?

I like Gerald Weinbergs definition: Quality is value to some person
But consider the implication: Its appropriate to report any deviation from high quality as a software error. Therefore many issues will be reported that will be errors to some and non-errors to others.
Glen Myers definition: A software error is present when the program does not do what its user reasonably expects it to do.
Black Box Software Testing Copyright
2003
183
Quality is Multidimensional
Programming Glass Box Testing Multimedia Production Content Development User Interface Design Marketing Black Box Testing Writing Customer Service
Project Manager
Manufacturing
When you sit in a project team meeting, discussing a bug, a new feature, or some other issue in the project, you must understand that each person in the room has a different vision of what a quality product would be. Fixing bugs is just one issue. The next slide gives some examples.
2003
184
Quality is Multidimensional: Different People, Different Visions

Localization Manager: A good product is easy to translate and to modify to make it suitable for another country and culture. Few expereienced localization managers would consider acceptable a product that must be recompiled or relinked to be localized. Tech Writers: A high quality program is easily explainable. Aspects of the design that are confusing, unnecessarily inconsistent, or hard to describe are marks of bad quality. Marketing: Customer satisfiers are the things that drive people to buy the product and to tell their friends about it. A Marketing Manager who is trying to add new features to the product generally believes that he is trying to improve the product. Customer Service: Good products are supportable. They have been designed to help people solve their own problems or to get help quickly. Programmers: Great code is maintainable, well documented, easy to understand, well organized, fast and compact.
Copyright
2003
185
Software Errors: What Kind of Error?

You will report all of these types of problems, but its valuable to keep straight in your mind, and on the bug report, which type youre reporting. Coding Error: The program doesnt do what the programmer would expect it to do. Design Issue: Its doing what the programmer intended, but a reasonable customer would be confused or unhappy with it. Requirements Issue: The program is well designed and well implemented, but it wont meet one of the customers requirements. Documentation / Code Mismatch: Report this to the programmer (via a bug report) and to the writer (usually via a memo or a comment on the manuscript). Specification / Code Mismatch: Sometimes the spec is right; sometimes the code is right and the spec should be changed.
186
Copyright
2003
Software Errors: Why are there Errors?

New testers often conclude that the programmers on their project are incompetent or unprofessional.
This is counterproductive. It leads to infighting instead of communication, and it leads to squabbling over bugs instead of research and bug fixing.
And as we saw when we discussed private bug rates, programmers actually find and fix the large majority of their own bugs. Bugs come into the code for many reasons. Its worth considering some common systematic (as distinct from poor individual performance) factors. You will learn to vary your strategic approaches as you learn your companies weaknesses.
Copyright
2003
187

Bugs come into the code for many reasons: The major cause of error is that programmers deal with tasks that arent fully understood or fully defined. This is said in many different ways. For example:
Tom Gilb and Dick Bender quote industry-summary statistics that 80% of the errors, or 80% of the effort required to fix the errors, are caused by bad requirements; Roger Sherman recently summarized research at Microsoft that the most common underlying issue in bug reports involved a need for new code.
If you graduated from a Computer Science program, how much training did you have in task analysis? Requirements definition? Usability analysis? Negotiation and clear communication of negotiated agreements? Not much? Hmmmm . . . .
2003
188

Some companies drive their programmers too hard. They dont have enough time to design, bulletproof, or test their code. Another Sherman quote: Bad schedules are responsible for most quality problems. Late design changes result in last minute code changes, which are likely to have errors. Some third-party components introduce bugs. Your program might rely on a large suite of small components that display a specific type of object, filter data in a special way, drive a specific printer, etc. Many of these tools, bought from tool vendors or hardware vendors, are surprisingly buggy. Others work, but they arent fully compatible with common test automation tools. Failure to use source control tools creates characteristic bugs. For example, if a bug goes away, comes back, goes away, comes back, goes away, comes back, then ask how the programming staff makes sure its linking the most recent version of each module when it builds a version for you to test.
Copyright
2003
189

Some programs or tasks are inherently complex. Boris Beizer talks perceptively about the locality problem in software. Think about an underlying bug, and then about symptoms caused by the bug. When symptoms appear, theres no assurance that theyll be close in time, space, or severity to the underlying bug. They may appear much later, or when working with a different part of the program, and they may seem much more or much less serious than the bug. Some programmers (some platforms) work with poor tools. Weak compilers, style checkers, debuggers, profilers, etc. make it too easy to get bugs or too hard to find bugs. Similarly, some third party hardware, or its drivers, are non-standard and dont respond properly to standard system calls. Incompatibility with hardware is often cited as the largest single source of customer complaints into technical support groups. When one programmer tries to fix bugs, or otherwise modify another programmers code, theres lots of room for miscommunication and error. And, sometimes people just make mistakes.
2003
190
Bug Advocacy?
1. The point of testing is to find bugs.
2.
Bug reports are your primary work product.
3. The best tester isnt the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.
4. Programmers operate under time constraints and competing priorities. For
what people outside of the testing group will most notice and most remember of your work.
This is
A bug report is a tool that you use to sell the programmer on the idea of spending her time and energy to fix a bug.
Note: When I say the best tester is the one who gets the most bugs fixed, I am not encouraging bug counting metrics, which are almost always counterproductive. Instead, what I am suggesting is that the effective tester looks to the effect of the bug report, and tries to write it in a way that gives each bug its best chance of being fixed. Also, a bug report is successful if it enables an informed business decision. Sometimes, the best decision is to not fix the bug. The excellent bug report raises the issue and provides sufficient data for a good decision.
example, outside of the 8-hour workday, some programmers prefer sleeping and watching Star Wars to fixing bugs.
Copyright
2003
191
Selling Bugs
Time is in short supply. If you want to convince the programmer to spend his time fixing your bug, you may have to sell him on it. (Your bug? How can it be your bug? The programmer made it, not you, right? Its the programmers bug. Well, yes, but you found it so now its yours too.) Sales revolves around two fundamental objectives: Motivate the buyer (Make him WANT to fix the bug.) Overcome objections (Get past his excuses and reasons for not fixing the bug.)
Copyright
2003
192
Motivating the Bug Fixer

Some things that will often make programmers want to fix the bug:
It looks really bad. It looks like an interesting puzzle and piques the programmers curiosity. It will affect lots of people. Getting to it is trivially easy. It has embarrassed the company, or a bug like it embarrassed a competitor. One of its cousins embarrassed the company or a competitor. Management (that is, someone with influence) has said that they really want it fixed. Youve said that you want the bug fixed, and the programmer likes you, trusts your judgment, is susceptible to flattery from you, owes you a favor or accepted bribes from you.
2003
193
Overcoming Objections
These make programmers resist spending time on a bug: The programmer cant replicate the defect. Strange and complex set of steps required to induce the failure. Not enough information to know what steps are required, and it will take a lot of work to figure them out. The programmer doesnt understand the report. Unrealistic (e.g. corner case) It will take a lot of work to fix the defect. A fix will introduce too much risk into the code. No perceived customer impact Unimportant (no one will care if this is wrong: minor error or unused feature.) Thats not a bug, its a feature. Management doesnt care about bugs like this. The programmer doesnt like / trust you (or the customer who is complaining about the bug). Black Box Software Testing Copyright 2003 Cem Kaner & James Bach
194
Bug Advocacy
Motivating Bug Fixes By Better Researching The Failure Conditions
Copyright
2003
195
Motivating The Bug Fix: Looking At The Failure

Some vocabulary An error (or fault) is a design flaw or a deviation from a desired or intended state. An error wont yield a failure without the conditions that trigger it. Example, if the program yields 2+2=5 on the 10th time you use it, you wont see the error before or after the 10th use. The failure is the programs actual incorrect or missing behavior under the error-triggering conditions. A symptom might be a characteristic of a failure that helps you recognize that the program has failed. Defect is frequently used to refer to the failure or to the underlying error.
Nancy Leveson (Safeware) draws useful distinctions between errors, hazards, conditions, and failures.
2003
196
Motivating The Bug Fix: Looking At The Failure

VOCABULARY EXAMPLE
Heres a defective program

INPUT A INPUT B PRINT A/B
What is the fault? What is the critical condition? What will we see as the failure?
Copyright
2003
197
Motivating the Bug Fix

When you run a test and find a failure, youre looking at a symptom, not at the underlying fault. You may or may not have found the best example of a failure that can be caused by the underlying fault. Therefore you should do some follow-up work to try to prove that a defect:
is more serious than it first appears. is more general than it first appears.
Copyright
2003
198
Motivating the Bug Fix: Make it More Serious

LOOK FOR FOLLOW-UP ERRORS When you find a coding error, you have the program in a state that the programmer did not intend and probably did not expect. There might also be data with supposedly impossible values. The program is now in a vulnerable state. Keep testing it and you might find that the real impact of the underlying fault is a much worse failure, such as a system crash or corrupted data. I do three types of follow-up testing:
Vary my behavior (change the conditions by changing what I do) Vary the options and settings of the program (change the conditions by changing something about the program under test). Vary the software and hardware environment.
2003
199
Follow-Up: Vary Your Behavior

Keep using the program after you see the problem. Bring it to the failure case again (and again). If the program fails when you do X, then do X many times. Is there a cumulative impact? Try things that are related to the task that failed. For example, if the program unexpectedly but slightly scrolls the display when you add two numbers, try tests that affect adding or that affect the numbers. Do X, see the scroll. Do Y then do X, see the scroll. Do Z, then do X, see the scroll, etc. (If the scrolling gets worse or better in one of these tests, follow that up, youre getting useful information for debugging.) Try things that are related to the failure. If the failure is unexpected scrolling after adding, try scrolling first, then adding. Try repainting the screen, then adding. Try resizing the display of the numbers, then adding. Try entering the numbers more quickly or changing the speed of your activity in some other way. And try the usual exploratory testing techniques. So, for example, you might try some interference tests. Stop the program or pause it or swap it just as the program is failing. Or try it while the program is doing a background save. Does that cause data loss corruption along with this failure?
2003
200
Follow-Up: Vary Options and Settings

In this case, the steps to achieve the failure are taken as given. Try to reproduce the bug when the program is in a different state: Use a different database. Change the values of persistent variables. Change how the program uses memory. Change anything that looks like it might be relevant that allows you to change as an option. For example, suppose the program scrolls unexpectedly when you add two numbers. Maybe you can change the size of the program window, or the precision (or displayed number of digits) of the numbers, or background the activity of the spell checker.
2003
201
Follow-Up: Vary the Configuration

A bug might show a more serious failure if you run the program with less memory, a higher resolution printer, more (or fewer) device interrupts coming in etc. If there is anything involving timing, use a really slow (or very fast) computer, link, modem or printer, etc.. If there is a video problem, try other resolutions on the video card. Try displaying MUCH more (less) complex images. Note that we are not: checking standard configurations asking how broad the circumstances that produces the bug. What were asking is whether there is a particular configuration that will show the bug more spectacularly. Returning to the example (unexpected scrolling when you add two numbers), try things like: Different video resolutions Different mouse settings if you have a wheel mouse that does semi-automated scrolling An NTSC (television) signal output instead of a traditional (XGA or SVGA, etc.) monitor output.
Copyright
2003
202
Follow-up: Bug New to This Version?

In many projects, an old bug (from a previous shipping release of the program) might not be taken very seriously if there werent lots of customer complaints. (If you know its an old bug, check its complaint history.) The bug will be taken more seriously if it is new. You can argue that it should be treated as new if you can
find a new variation or a new symptom that didnt exist in the previous release. What you are showing is that the new versions code interacts with this error in new ways. Thats a new problem. This type of follow-up testing is especially important during a maintenance release that is just getting rid of a few bugs. Bugs wont be fixed unless they were (a) scheduled to be fixed because they are critical or (b) new side effects of the new bug fixing code.
Copyright
2003
203
Motivating the Bug Fix: Show it is More General

LOOK FOR CONFIGURATION DEPENDENCE Bugs that dont fail on the programmers machine are much less credible (to that programmer). If they are configuration dependent, the report will be much more credible if it identifies the configuration dependence directly (and so the programmer starts out with the expectation that it wont fail on all machines.)
Question: How many programmers does it take to change a light bulb? Answer: Whats the problem? The bulb at my desk works fine!
2003
204
LOOK FOR CONFIGURATION DEPENDENCE

In the ideal case (standard in many companies), test on 2 machines Do your main testing on Machine 1. Maybe this is your powerhouse: latest processor, newest updates to the operating system, fancy printer, video card, USB devices, huge hard disk, lots of RAM, cable modem, etc. When you find a defect, use Machine 1 as your bug reporting machine and replicate on Machine 2. Machine 2 is totally different. Different processor, different keyboard and keyboard driver, different video, barely enough RAM, slow, small hard drive, dial-up connection with a link that makes turtles look fast. Some people do their main testing on the turtle and use the power machine for replication. Write the steps, one by one, on the bug form at Machine 1. As you write them, try them on Machine 2. If you get the same failure, youve checked your bug report while you wrote it. (A valuable thing to do.) If you dont get the same failure, you have a configuration dependent bug. Time to do troubleshooting. But at least you know that you have to. AS A MATTER OF GENERAL GOOD PRACTICE, IT PAYS TO REPLICATE EVERY BUG ON A SECOND MACHINE.
2003
205

UNCORNER YOUR CORNER CASES We test at extreme values because these are the most likely places to show a defect. But once we find the defect, we dont have to stick with extreme value tests. Try mainstream values. These are easy settings that should pose no problem to the program. Do you replicate the bug? If yes, write it up, referring primarily to these mainstream settings. This will be a very credible bug report.
Copyright
2003
206

UNCORNER YOUR CORNER CASES If the mainstream values dont yield failure, but the extremes do, then do some troubleshooting around the extremes.
Is the bug tied to a single setting (a true corner case)? Or is there a small range of cases? What is it? In your report, identify the narrow range that yields failures. The range might be so narrow that the bug gets deferred. That might be the right decision. In some companies, the product has several hundred open bugs a few weeks before shipping. They have to decide which 300 to fix (the rest will be deferred). Your reports help the company choose the right 300 bugs to fix, and help people size the risks associated with the remaining ones.
Copyright
2003
207
Bug Advocacy
Overcoming
OBJECTIONS
By Better Researching The Failure Conditions
Copyright
2003
208
Overcoming Objections: Analysis of the Failure

Things that will make programmers resist spending their time on the bug:
The programmer cant replicate the defect.

Strange and complex set of steps required to induce the failure. Not enough information to know what steps are required, and it will take a lot of work to figure them out. The programmer doesnt understand the report. Unrealistic (e.g. corner case) Its a feature.
Copyright
2003
209
Objection, Objection: Non-Reproducible Errors

Always report non-reproducible errors. If you report them well, programmers can often figure out the underlying problem.
You must describe the failure as precisely as possible. If you can identify a display or a message well enough, the programmer can often identify a specific point in the code that the failure had to pass through.
When you realize that you cant reproduce the bug, write down everything you can remember. Do it now, before you forget even more. As you write, ask yourself whether youre sure that you did this step (or saw this thing) exactly as you are describing it. If not, say so. Draw these distinctions right away. The longer you wait, the more youll forget. Maybe the failure was a delayed reaction to something you did before starting this test or series of tests. Before you forget, note the tasks you did before running this test. Check the bug tracking system. Are there similar failures? Maybe you can find a pattern. Find ways to affect timing of the program or devices, Slow down, speed up. Talk to the programmer and/or read the code.
Copyright
2003
210
Non-Reproducible Errors
The fact that a bug is not reproducible is data. The program is telling you that you have a hole in your logic. You are not entertaining certain relevant conditions. Why not? See Watts Humphrey, Personal Software Process, for recommendations to programmers of a system for discovering and then eliminating characteristic errors from their code. A non-reproducible bug is a testers error, just like a design bug is a programmers error. Its valuable to develop a system for discovering your blind spots. To improve over time, keep track of the bugs youre missing and what conditions you are not attending to (or find too hard to manipulate). The following pages give a list of some conditions commonly ignored or missed by testers. Your personal list will be different in some ways, but maybe this is a good start. When you run into a irreproducible defect look at this list and ask whether any of these conditions could be the critical one. If it could, vary your tests on that basis and you might reproduce the failure.
----------------------------------------------------------------------------------------
(Note: Watts Humphrey suggested to me the idea of keeping a list of commonly missed conditions. It has been a valuable idea.)
Copyright
2003
211
Non-Reproducible Errors: Examples of Conditions Often Missed

Some problems have delayed effects: a memory leak might not show up until after you cut and paste 20 times.
stack corruption might not turn into a stack overflow until you do the same task many times. a wild pointer might not have an easily observable effect until hours after it was mis-set.
If you suspect that you have time-delayed failures, use tools such as videotape, capture programs, debuggers, debugloggers, or memory meters to record a long series of events over time.
Copyright
2003
212

I highlighted the first three in lecture because so many people have trouble with time-delayed bugs. Until you think backwards in time and ask how you could find a defect that has a delayed reaction effect, you wont be able to easily recreate these problems. The following pages give additional examples. There are plenty of other conditions that are relevant in your environment. Start with these but add others as you learn of them. How do you learn? Sometimes, someone will fix a bug that you reported as non-reproducible. Call the programmer, ask him how to reproduce it, what are the critical steps that you have to take? You need to know this anyway, so that you can confirm that a bug fix actually worked.
2003
213

The bug depends on the value of a hidden input variable. (Bob Stahl teaches this well.) In any test, there are the variables that we think are relevant and there is everything else. If the data you think are relevant dont help you reproduce the bug, ask what other variables were set, and what their values were. Some conditions are hidden and others are invisible. You cannot manipulate them and so it is harder to recognize that theyre present. You might have to talk with the programmer about what state variables or flags get set in the course of using a particular feature. Some conditions are catalysts. They make failures more likely to be seen. Example: low memory for a leak; slow machine for a race. But sometimes catalysts are more subtle, such as use of one feature that has a subtle interaction with another. Some bugs are predicated on corrupted data. They dont appear unless there are impossible configuration settings in the config files or impossible values in the database. What could you have done earlier today to corrupt this data?
Copyright
2003
214

The bug might appear only at a specific time of day or day of the month or year. Look for week-end, month-end, quarter-end and yearend bugs, for example. Programs have various degrees of data coupling. When two modules use the same variable, oddness can happen in the second module after the variable is changed by the first. (Books on structured design, such as Yourdon/Constantine often analyze different types of coupling in programs and discuss strengths and vulnerabilities that these can create.) In some programs, interrupts share data with main routines in ways that cause bugs that will only show up after a specific interrupt. Special cases appear in the code because of time or space optimizations or because the underlying algorithm for a function depends on the specific values fed to the function (talk to your programmer). The bug depends on you doing related tasks in a specific order.
Copyright
2003
215

The bug is caused by a race condition or other time-dependent event, such as:
An interrupt was received at an unexpected time. The program received a message from another device or system at an inappropriate time (e.g. after a time-out.) Data was received or changed at an unexpected time.
The bug is caused by an error in error-handling. You have to generate a previous error message or bug to set up the program for this one. Time-outs trigger a special class of multiprocessing error handling failures. These used to be mainly of interest to real-time applications, but they come up in client/server work and are very pesky. Process A sends a message to Process B and expects a response. B fails to respond. What should A do? What if B responds later?
Copyright
2003
216

Another inter-process error handling failure -- Process A sends a message to B and expects a response. B sends a response to a different message, or a new message of its own. What does A do? Youre being careful in your attempt to reproduce the bug, and youre typing too slowly to recreate it. The program might be showing an initial state bug, such as: The bug appears only the first time after you install the program (so it happens once on every machine.) The bug appears once after you load the program but wont appear again until you exit and reload the program. (See Testing Computer Softwares Appendixs discussion of initial state bugs.) The program may depend on one version of a DLL. A different program loads a different version of the same DLL into memory. Depending on which program is run first, the bug appears or doesnt.
Copyright
2003
217

The problem depends on a file that you think youve thrown away, but its actually still in the Trash (where the system can still find it). A program was incompletely deleted, or one of the current programs files was accidentally deleted when that other program was deleted. (Now that youve reloaded the program, the problem is gone.) The program was installed by being copied from a network drive, and the drive settings were inappropriate or some files were missing. (This is an invalid installation, but it happens on many customer sites.) The bug depends on co-resident software, such as a virus checker or some other process, running in the background. Some programs run in the background to intercept foreground programs failures. These may sometimes trigger failures (make errors appear more quickly).
Copyright
2003
218

You forgot some of the details of the test you ran, including the critical one(s) or you ran an automated test that lets you see that a crash occurred but doesnt tell you what happened. The bug depends on a crash or exit of an associated process. The program might appear only under a peak load, and be hard to reproduce because you cant bring the heavily loaded machine under debug control (perhaps its a customers system). On a multi-tasking or multi-user system, look for spikes in background activity. The bug occurred because a device that it was attempting to write to or read from was busy or unavailable. It might be caused by keyboard keybounce or by other hardware noise.
219
Copyright
2003

Code written for a cooperative multitasking system can be thoroughly confused, sometimes, when running on a preemptive multitasking system. (In the cooperative case, the foreground task surrenders control when it is ready. In the preemptive case, the operating system allocates time slices to processes. Control switches automatically when the foreground task has used up its time. The application is suspended until its next time slice. This switch occurs at an arbitrary point in the applications code, and that can cause failures. The bug occurs only the first time you run the program or the first time you do a task after booting the program. To recreate the bug, you might have to reinstall the program. If the program doesnt uninstall cleanly, you might have to install on a fresh machine (or restore a copy of your system taken before you installed this software) before you can see the problem.
220
Copyright
2003

The bug is specific to your machines hardware and system software configuration. (This common problem is hard to track down later, after youve changed something on your machine. Thats why good reporting practice involves replicating the bug on a second configuration.) The apparent bug is a side-effect of a hardware failure. For example, a flaky power supply creates irreproducible failures. Another example: one prototype system had a high rate of irreproducible firmware failures. Eventually, these were traced to a problem in the buildings air conditioning. The test lab wasnt being cooled, no fan was blowing on the unit under test, and prototype boards in the machine ran very hot. The machine was failing at high temperatures. Elves tinkered with your machine when you werent looking. There are several other ideas (focused on web testing) at http://www.logigear.com/whats_new.html#article
2003
221

The programmer cant replicate the defect.
Strange and complex set of steps required to induce the failure. Not enough information to know what steps are required, and it will take a lot of work to figure them out. The programmer doesnt understand the report.
Unrealistic (e.g. corner case) Its a feature!
Copyright
2003
222
Bug Advocacy
Writing the Bug Report
Copyright
2003
223
Bug Reporting Practice Example

You will be given a copy of a computer screen. Please write a description (all words, no pictures) of that screen When asked, please pass your description to your partner You will receive a description of a different screen from your partner. Please draw the screen that your partner is describing.
Copyright
2003
224
Reporting Errors
As soon as you run into a problem in the software, fill out a Problem Report form. In the well written report, you:
Explain how to reproduce the problem. Analyze the error so you can describe it in a minimum number of steps. Include all the steps. Make the report easy to understand. Keep your tone neutral and non-antagonistic. Keep it simple: one bug per report. If a sample test file is essential to reproducing a problem, reference it and attach the test file. To the extent that you have time, describe the dimensions of the bug and characterize it. Describe what events are and are not relevant to the bug. And what the results are (any characteristics of the failure) and how they varied across tests.
2003
225
The Problem Report Form

A typical form includes many of the following fields Problem report number: must be unique Reported by: original reporters name. Some forms add an editors name. Date reported: date of initial report Program (or component) name: the visible item under test Release number: like Release 2.0 Version (build) identifier: like version C or version 20000802a Configuration(s): h/w and s/w configs under which the bug was found and replicated Report type: e.g. coding error, design issue, documentation mismatch, suggestion, query Can reproduce: yes / no / sometimes / unknown. (Unknown can arise, for example, when the repro configuration is at a customer 226 site Testing available Black Box Software and not Copyright 2003 to the lab. ) Cem Kaner & James Bach

A typical form includes many of the following fields Severity: assigned by tester. Some variation on small / medium / large Priority: assigned by programmer/project manager Customer impact: often left blank. When used, typically filled in by tech support or someone else predicting actual customer reaction (such as support cost or sales impact) Problem summary: 1-line summary of the problem Key words: use these for searching later, anyone can add to key words at any time Problem description and how to reproduce it: step by step repro description Suggested fix: leave it blank unless you have something useful to say Assigned to: typically used by project manager to identify who has responsibility for researching/fixing the problem
2003
227

A typical form includes many of the following fields Status: Tester fills this in. Open / closed / dumpstersee previous slide on dumpsters. Resolution: The project manager owns this field. Common resolutions include:
Pending: the bug is still being worked on. Fixed: the programmer says its fixed. Now you should check it. Cannot reproduce: The programmer cant make the failure happen. You must add details, reset the resolution to Pending, and notify the programmer. Deferred: Its a bug, but well fix it later. As Designed: The program works as its supposed to. Need Info: The programmer needs more info from you. She has probably asked a question in the comments. Duplicate: This is just a repeat of another bug report (XREF it on this report.) Duplicates should not close until the duplicated bug closes. Withdrawn: The tester who reported this bug is withdrawing the report.
2003
228

A typical form includes many of the following fields Resolution version: build identifier Resolved by: programmer, project manager, tester (if withdrawn by tester), etc. Resolution tested by: originating tester, or a tester if originator was a non-tester Change history: datestamped list of all changes to the record, including name and fields changed. Comments: free-form, arbitrarily long field, typically accepts comments from anyone on the project. Testers programmers, tech support (in some companies) and others have an ongoing discussion of repro conditions, etc., until the bug is resolved. Closing comments (why a deferral is OK, or how it was fixed for example) go here.
This field is especially valuable for recording progress and difficulties with difficult or politically charged bugs. Write carefully. Just like e-mail and usenet postings, its easy to read a joke or a remark as a flame. Never flame. 229
Copyright
2003
The Problem Report Form: Further Reading

The best discussion in print of bug reporting and bug tracking system design is probably still the one in my book, Testing Computer Software, chapters 5 & 6. (Not because its so wonderful but because not enough good stuff has been written since.) Brian Marick has captured some useful material at his site, www.testingcraft.com. (You should get to know this site, and ideally, contribute to it. This is a collection point for examples.) Hung Quoc Nguyen (who co-authored TCS 2.0 and is working with us on 3.0) published TrackGear, a web based bug tracking system that has a lot of thought behind it. You can get a 30-day free eval at www.logigear.com. The Testing Tools FAQ lists other bug tracking software that you can get eval copies. The FAQ is linked from the main comp.software.testing FAQ at http://www.rstcorp.com/resources/hosted.html 230 Black Box Software Testing Copyright 2003 Cem Kaner & James Bach
Important Parts of the Report: Problem Summary

This one-line description of the problem is the most important part of the report. The project manager will use it in when reviewing the list of bugs that havent been fixed. Executives will read it when reviewing the list of bugs that wont be fixed. They might only spend additional time on bugs with interesting summaries. The ideal summary gives the reader enough information to help her decide whether to ask for more information. It should include: A brief description that is specific enough that the reader can visualize the failure. A brief indication of the limits or dependencies of the bug (how narrow or broad are the circumstances involved in this bug)? Some other indication of the severity (not a rating but helping the reader envision the consequences of the bug.)
2003
231
The Report: Can You Reproduce The Problem?

You may not see this on your form, but you should always provide this information. Never say its reproducible unless you have recreated the bug. (Always try to recreate the bug before writing the report.) If youve tried and tried but you cant recreate the bug, say No. Then explain what steps you tried in your attempt to recreate it. If the bug appears sporadically and you dont yet know why, say Sometimes and explain. You may not be able to try to replicate some bugs. Example: customerreported bugs where the setup is too hard to recreate. The following policy is not uncommon: If the tester says that a bug is reproducible and the programmer says it is not, then the tester has to recreate it in the presence of the programmer.
Copyright
2003
232
The Report--Description; How to Reproduce It.

First, describe the problem. Whats the bug? Dont rely on the summary to do this -- some reports will print this field without the summary. Next, go through the steps that you use to recreate this bug.
Start from a known place (e.g. boot the program) and Then describe each step until you hit the bug. NUMBER THE STEPS. Take it one step at a time. If anything interesting happens on the way, describe it. (You are giving people directions to a bug. Especially in long reports, people need landmarks.)
Describe the erroneous behavior and, if necessary, explain what should have happened. (Why is this a bug? Be clear.) List the environmental variables (config, etc.) that are not covered elsewhere in the bug tracking form. If you expect the reader to have any trouble reproducing the bug (special circumstances are required), be clear about them.
2003
233
The Report-- Description; How to Reproduce It.

It is essential keep the description focused: The first part of the description should be the shortest step-bystep statement of how to get to the problem. Add Notes after the description if you have them. Typical notes include: Comment that the bug wont show up if you do step X between step Y and step Z. Comment explaining your reasoning for running this test. Comment explaining why you think this is an interesting bug. Comment describing other variants of the bug.
Copyright
2003
234
Keeping the Report Simple

If you see two failures, write two reports.
Combining failures on one report creates problems: The summary description is typically vague. You say words like fails or doesnt work instead of describing the failure more vividly. This weakens the impact of the summary. The detailed report is typically lengthened. Its common to see bug reports that read like something written by an inept lawyer. Do this unless that happens in which case dont do this unless the first thing and then the testcase of the second part and sometimes you see this but if not then that. Even if the detailed report is rationally organized, it is longer (there are two failures and two sets of conditions, even if they are related) and therefore more intimidating. Youll often see one bug get fixed but not the other. When you report related problems on separate reports, it is a courtesy to cross-reference them.
2003
235
Keeping it Simple: Eliminate Unnecessary Steps (1)

Sometimes its not immediately obvious what steps can be dropped from a long sequence of steps in a bug. Look for critical steps -- Sometimes the first symptoms of an error are subtle. You have a list of the steps you took to show the error. Youre now trying to shorten the list. Look carefully for any hint of an error as you take each step -- A few things to look for: Error messages (you got a message 10 minutes ago. The program didnt fully recover from the error, and the problem you see now is caused by that poor recovery.) Delays or unexpectedly fast responses. Display oddities, such as a flash, a repainted screen, a cursor that jumps back and forth, multiple cursors, misaligned text, slightly distorted graphics, doubled characters, omitted characters, or display droppings (pixels that are still colored even though the character or graphic that contained them was erased or moved).
2003
236
Keeping it Simple: Eliminate Unnecessary Steps (2)

Sometimes the first indicator that the system is working differently is that it sounds a little different than normal. An in-use light or other indicator that a device is in use when nothing is being sent to it (or a light that is off when it shouldnt be). Debug messagesturn on the debug monitor on your system (if you have one) and see if/when a message is sent to it. If youve found what looks like a critical step, try to eliminate almost everything else from the bug report. Go directly from that step to the last one (or few) that shows the bug. If this doesnt work, try taking out individual steps or small groups of steps.
Copyright
2003
237
Keep it Simple: Put Variations After the Main Report

Suppose that the failure looks different under slightly different circumstances. For example, suppose that: The timing changes if you do two additional sub-tasks before hitting the final reproduction step The failure wont show up or is much less serious if you put something else at a specific place on the screen The printer prints different garbage (instead of the garbage you describe) if you make the file a few bytes longer This is all useful information for the programmer and you should include it. But to make the report clear: Start the report with a simple, step-by-step description of the shortest series of steps that you need to produce the failure. Identify the failure. (Say whatever you have to say about it, such as what it looks like or what impact it will have.) Then add a section that says ADDITIONAL CONDITIONS and describe, one by one, in this section the additional variations and the effect on the observed failure. 238 Black Box Software Testing Copyright 2003 Cem Kaner & James Bach

Things that will make programmers resist spending their time on the bug: The programmer cant replicate the defect. Strange and complex set of steps required to induce the failure. Not enough information to know what steps are required, and it will take a lot of work to figure them out. The programmer doesnt understand the report.
Unrealistic (e.g. corner case)

Its a feature!
Copyright
2003
239
Overcoming Objections: Unrealistic (e.g., Corner Conditions)

Some reports are inevitably dismissed as unrealistic (having no importance in real use). If youre dealing with an extreme value, do follow-up testing with less extreme values. If youre protesting a bug that has been left unfixed for several versions, realized that it has earned tenure in some peoples minds. Perhaps, though, customer complaints about this bug have simply never filtered through to developers. If your report of some other type of defect or design issue is dismissed as having no customer impact, ask yourself:
Hey, how do they know the customer impact?

Then check with people who might know:
-- Technical marketing -- Technical support -- Human factors -- Documentation -- Network administrators -- Training -- In-house power users -- Maybe sales
2003
240

The programmer cant replicate the defect. Strange and complex set of steps required to induce the failure. Not enough information to know what steps are required, and it will take a lot of work to figure them out. The programmer doesnt understand the report. Unrealistic (e.g. corner case)
Its a feature!
Later in the course, well think about this. The usual issues involve the costs of fixing bugs, the companys understanding of the definitions of bugs, and your personal credibility.
Copyright
2003
241
Bug Advocacy
Editing Bug Reports
Copyright
2003
242
Editing Bug Reports

Some groups have a second tester (usually a senior tester) review reported defects before they go to the programmer. The second tester:
checks that critical information is present and intelligible checks whether she can reproduce the bug asks whether the report might be simplified, generalized or strengthened.
If there are problems, she takes the bug back to the original reporter.
If the reporter was outside the test group, she simply checks basic facts with him. If the reporter was a tester, she points out problems with an objective of furthering the testers training.
2003
243
Editing Bug Reports

This tester might review: all defects all defects in her area all of her buddys defects. In designing a system like this, beware of overburdening the reviewing testers. The reviewer will often go through a learning curve (learning about parts of the system or types of tests that she hasnt studied before). This takes time. Additionally, you have to decide whether the reviewer is doing an actual reproduction of the test or thinking about the plausibility and understandability of the report when she reads it.
Copyright
2003
244
Assignment: Editing Bugs

Go through the IssueZilla bug database and find some bugs that look interesting Do an initial review of them Replicate them Revise the descriptions to make them clearer and more useful. Assignment: Give two improved bugs to another student Review two improved bugs from another student Send me copies of the improved bugs. Do NOT enter the revisions (or new reports) into IssueZilla until I have reviewed your work on this assignment.
245
Copyright
2003
Bug Advocacy
Advocating for bug fixes by alerting people to costs.

Assigned Reading: Kaner, Quality Cost Analysis: Benefits & Risks.
2003
246
Money Talks: Cost of Finding and Fixing Software Errors
This curve maps the traditionally expected increase of cost as you find and fix errors later and later in development.
2003
247

This is the most commonly taught cost curve in software engineering. Usually people describe it from the developers-eye view. That is, the discussion centers around how much it costs to find the bug how much it costs to fix the bug and how much it costs to distribute the bug fix. But sometimes, it pays to adopt the viewpoints of other stakeholders, who might stand to lose more money than the development and support organizations.
Copyright
2003
248

Costs escalate because more people in and out of the company are affected by bugs, and more severely affected, as the product gets closer to release. We all know the obvious stuff if we find bugs in requirements, we can fix them without having to recode anything; programmers who find their own bugs can fix them without taking time to file bug reports or explain them to someone else; it is hugely expensive to deal with bugs in the field (in customers hands). Along with this, there are many effects on other stakeholders in the company. For example, think of the marketing assistant who wastes days trying to create a demo, but cant because of bugs.
2003
249

It is important to recognize that this cost curve is predicated on a family of development practices.
When you see a curve that says,

Late changes are expensive you can reasonably respond in either of two ways: Make fewer late changes. This is the traditional recommendation Make it cheaper to make late changes. This is a key value of the agile development movement (see Becks Extreme Programming Explained, or go to www.agilealliance.org) In this testing course, I will push you to find ways to find bugs earlier, but my development philosophy is agile. Black Box Software Testing Copyright 2003 Cem Kaner & James Bach
250
Quality Cost Analysis

Quality Cost Measurement is a cost control system used to identify opportunities for reducing the controllable quality-related costs
The Cost of Quality is the total amount the company spends to achieve and cope with the quality of its product.
This includes the companys investments in improving quality, and its expenses arising from inadequate quality. A key goal of the quality engineer is to help the company minimize its cost of quality.
Refer to the paper, Quality Cost Analysis: Benefits & Risks.
2003
251
Quality-Related Costs
Prevention Appraisal
Cost of preventing Cost of inspection customer dissatisfaction, (testing, reviews, etc.). including errors or weaknesses in software, design, documentation, and support. Internal Failure External Failure Cost of dealing with Cost of dealing with errors discovered during errors that affect your development and testing. customers, after the Note that the company product is released. loses money as a user Black Box Software Testing (who cantCopyright 2003 Cem Kaner & James Bach make the
252
Examples of Quality Costs

Prevention Appraisal
Staff training Requirements analysis & early prototyping Fault-tolerant design Defensive programming Usability analysis Clear specification Accurate internal documentation Pre-purchase evaluation of the reliability of development tools
Internal Failure Bug fixes Regression testing Wasted in-house user time Wasted tester time Wasted writer time Wasted marketer time Wasted advertisements Direct cost of late shipment Opportunity cost of late shipment
Design review Code inspection Glass box testing Black box testing Training testers Beta testing Usability testing Pre-release out-of-box testing by customer service staff
External Failure
Lost sales and lost customer goodwill Technical support calls Writing answer books (for Support) Investigating complaints Supporting multiple versions in the field Refunds, recalls, warranty, liability costs Interim bug fix releases Shipping updated product PR to soften bad reviews Discounts to resellers
253
Copyright
2003
Customers Quality Costs

Seller: external costs
These illustrate costs absorbed by the seller that releases a defective product. Lost sales and lost customer goodwill Technical support calls Writing answer books (for Support) Investigating complaints Refunds, recalls, warranty, liability costs Government investigations Supporting multiple versions in the field Interim bug fix releases Shipping updated product PR to soften bad reviews Discounts to resellers
Customer: failure costs (sellers externalized costs)

These illustrate costs absorbed by the customer who buys a defective product. Wasted time Lost data Lost business Embarrassment Frustrated employees quit Failure during one-time-only tasks, e.g. demos to prospective customers Cost of replacing product Reconfiguring the system Cost of recovery software Tech support fees Injury / death
2003
254
Influencing Others Based on Costs

Its often impossible to fix every bug. But sometimes the development team will choose to not fix a bug based on their assessment of its risks for them, without thinking of the costs to other stakeholders in the company. Probable tech support cost. Risk to the customer. Risk to the customers data or equipment. Visibility in an area of interest to reviewers. Extent to which the bug detracts from the use of the program. How often will a customer see it? How many customers will see it? Does it block any testing tasks? Degree to which it will block OEM deals or other sales. To argue against a deferral, ask yourself which stakeholder(s) will pay the cost of keeping this bug. Flag the bug to them.
2003
255

Black Box Software Testing: Bug Advocacy

Uploaded by

Copyright:

Available Formats

Black Box Software Testing: Bug Advocacy

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Black Box Software Testing: Bug Advocacy

Uploaded by

Copyright:

Available Formats

Black Box Software Testing