CSCI 3308: Software Methods and Tools
Testing Notebook
1. Introduction
Lab 4 is a two week project worth 50 points called the "Testing Notebook". In this project, you will learn about two different testing methods, functional and structural testing, practically apply them to software and deepen your understanding about these two approaches.
Remember that you have the two lab sections to work on this work as well as the week in between to work on it from home. You and your lab partner(s) should be prepared to divvy up the work of lab among yourselves to work on outside the lab and then use the lab sections to bring each other up to date, work together to solve problems, and make use of the TA to help answer any remaining questions and/or issues.
2. Background
As you have experienced in the past labs, developing software incorporates a lot of testing. You write or modify a piece of program code and test it to make sure it creates the intended output. One could very well argue that testing will be the largest share of time you will spend in your software projects. Indeed, the Extreme Programming software methodology you have just learned about asks you to write test cases even before writing any code at all!
As software projects become more complex and developed by many developers in parallel, it makes sense to automate this software project: in a software developer team, you are modifying code that will be used not only by you but is called by many other programmers as well. An automated test suite will make sure that your changes to the code did not break your code but neither broke the programs of somebody else.
3. Functional Testing
One approach to testing software is called functional - or black-box - testing. Here, the tester does not know or see what a program internally looks like, instead she just knows what the program is supposed to do and what parameters the software takes as input.
An example would if I provide you a binary file and tell you that it takes a text file containing a list of three to seven values, sorts them and returns the sorted list. Since you can't look my the program - it is a black-box for you - you have to choose smart input values that might expose errors in my program.
Think about the simple specification I have given you. There is a set of boundary conditions you can extract from it. You definitely want to make sure to account for these, i.e. does the program sort a list of two, three, seven and eighth entries? Does it also sort five entries? What happens if two entries are the same etc.? It helps to make a plan first and visualize what kind of inputs the program would take and what boundary conditions there exist. These are usually the places where most things can go wrong, but other problems might exist as well. Therefore, make sure that you spread your test cases wisely over the spectrum of inputs and thoroughly test the program without practically testing every possible input value.
EZ Pay
Your boss has written a simple program that is supposed to process the time sheet of each employee at the end of the week for the payroll department. He tells you to make sure that the program does what it is to supposed to do and computes the correct wage for everyone in the department. All of you've got is the binary file and a list of features for the program.
Q1: Read the list of features and develop a plan for your testing using the process discussed in Lecture 19. Your plan should be divided into functional categories, each with their own specification items. Each item should have a list of categories and each category should have one or more test cases. You should eliminate any duplicate test cases that occur, using the process discussed in lecture. (You do not have to prioritize your test cases, as you should run them all to attempt to find as many bugs in the program as possible.) What critical boundary conditions have you identified? How will you test them? What else will you test?
After you have done your plan, create input values and test the program for errors (I have put it for you on the web). Like in reality, you don't know how many problem there might be in the program. So, stop when you feel you have tested the program well enough or think there are no more bugs left.
Q2: Submit a list of bugs you have identified and how you have identified them.
4. Structural Testing
As a developer, you usually have the source code of your application available which enables you to do a different type of testing: structural or white-box testing. Here, you can look at the code and write test cases that will test your program depending on how it works internally, for example if your program reaches certain statements for a set of given input values.
Q3: You as the developer are writing test cases for your application. Is this a good idea (not the fact that you are testing)? Might there be any problems with that?
There exist several test environments that help you with tests, a popular one for Java is called JUnit Test. As most of you have told me you are experienced Java programmers, we will do this white-box testing part in Java. If you need a refreshment in Java, feel free to read this short Java for C programmers document or Sun's Java Tutorial. But let's start easy.
Let's image I have the following little program called SimpleTest that I want to test with maximum condition coverage:
public class SimpleDemo {
int a, b;
boolean c;
public SimpleDemo(int a, int b, boolean c)
{
this.a = a;
this.b = b;
this.c = c;
dummyFunction();
}
private void dummyFunction()
{
if(c)
{
// Code
}else{
// Code
}
if(a < b)
{
// Code
}else{
// Code
}
if(a == b & c)
{
// Code
}else{
// Code
}
}
}
As you can see there are three conditions with boolean expressions and I would like to achieve the most condition coverage possible. My first test program looks like this:
public class SimpleDemoTest {
public static void main(String[] args) {
SimpleDemo s;
// Here I start my tests
s = new SimpleDemo(3, 4, true);
// The summary - how well did I do?
CondCoverage.printCoverageSummary();
}
}
I set a to 3, b to 4 and c to true. When I execute it I get the following report:
Only true:
c (L33)
a<b
c (L19)
Only false:
a==b
As expected we covered only part of the decisions. More specifically, c in line 33, and c in line 19 are only true. Further, a is always less than b and of course, therefore a is never equal to b Let's extend this a bit more and add the following line to our test and run it again:
s = new SimpleDemo(3, 4, false);
You see that our coverage went up:
Only true:
a<b
Only false:
a==b
Now there are just two more conditions that we did not check for yet. Feel free to play around with it a bit more for this beginning exercise. Here is again a web page where you can submit a your SimpleDemoTest class. Start with a the template above, but make sure to name your class SimpleDemoTest_YOURNAME where YOURNAME is indeed your name without spaces. (If it doesn't seem to work, check for non-visible characters you copy and pasted from your browser along with the code.) When you did achieve full condition coverage, please go on.
Now let's start with something bigger. As I told you before, there are packages that help you test your code. One popular test package for Java is called JUnit Test which is already shipped with Eclipse. As a preparation for this part, do the following steps:
1. Download this file:
do
2. Extract the contents of the downloaded file. At this point, you should have a folder/directory called white-box-testing that has four Java files (Board.java, BoardUnitTests.java, Logic.java, and LogicUnitTests.java), three gif files (red.gif, white.gif, and blue.gif) and a text file (Stories.txt).
3. Start up Eclipse.
4. From the "File" menu, select "New" -> "Project"
5. Select "Java Project" and Click "Next"
6. Put "white-box-testing" as the project name. Click "Finish".
7. Right click on your project name, select "New" -> "Class". Name it OldVersionWorkaround. There is now default package and in it the new Java file. Right click on the package, select "Import" and on the next screen "File system". Use the Browse button to select your "white-box-testing" directory. Select all the files for import. Click "Finish". The files should be imported into your default package.
8. At this point, Eclipse should report a number of errors in the LogicUnitTests. These happen because the library junit.jar is not yet part of the project. Clicking on one of the red "x" will give you the option "Add JUnit Libraries". Selecting this option will make the errors go away. Feel free to run the unit tests within eclipse.
9. Open LogicUnitTests.java and go to the class declaration (public class LogicUnitTests extends TestCase...). Right click on LogicUnitTests, in the menu select Refactor->Rename... In the rename dialog box change LogicUnitTests to LogicUnitTests_YOURNAME, where YOURNAME is the your name like in the example before. Click OK.
10. Paste the entire contents of LogicUnitTests_YOURNAME into the test case field of this page and click submit. The webpage will respond with condition coverage. More specifically the system will report conditions that are always false when executed and those that are always true when executed; these are the ones that are inadequately tested (at least as far as condition coverage is concerned).
 |
The files I just gave you are an almost
finished implementation of a game called
"Connect Four". In this game, two players who each have
chips of a certain color stack their chips into a board
you can see on the left side. They take turns and the
player who has four chips of her color adjacent
(vertical, horizontal or diagonal) wins. For more
details on this game, see the Wikipedia site about the game.
The text file Stories.txt tells you how the logic is
supposed to work.
|
So, what's in the files? If you right-click on Logic.java and select Run-As -> Java Application you can see that we have already implemented the board and part of the Logic of the game. You can click on a column and drop a chip in there.
Note that there is also a file named LogicUnitTests.java. If you right-click on in and select Run-As -> JUnit Tests it runs all the tests defined in this file. In the end you get a summary of which tests worked and did not work. Each function in this class is a particular test. Consider for example the following code:
public void testStartTurnBlue() {
assertTrue(l.getTurnColor() == l.Blue());
}
This is implementing a certain requirement that is described in Stories.txt:
Story "Start":
The game starts with the turn color as blue. ...
When a new game (the variable l holds the logic of this new game board) is created, the current color should be blue. Since this is the expected behavior, we wrap the statement assertTrue around it. If the expression within it is true, it continues, otherwise it throws an error. So, if we add the following test
public void testStartTurnBlue() {
assertTrue(l.getTurnColor() == l.Red());
}
The color of the first player is blue, the overall expression is false and assertTrue(false) therefore throws an exception. Go ahead change the code and run it. In a big software project, these unit tests will make sure that your code gets not broken when you work on it. Now change it back to blue. Note that if you explicitly want to test if a condition is wrong, you can use assertFalse() to do that.
When you run your unit tests on the web site, note that there is again the same output as before from the condition coverage checker. As you may notice, there is again several conditions that aren't properly covered. Don't be confused about that the coverage checker displays several times block named "Only true" / "false". It outputs the condition coverages after every test case as well as a final coverage at the end of the procedure. Look for the last pair of Only true / false.
Q4: Extend the current test cases to get the best code coverage possible. Use the coverage checker on the web site as much as you like. If you can't achieve full coverage, explain why this might not be possible. Include your test case file LogicUnitTests_YOURNAME.java in your submission.
Before we wrap up, think about two more questions:
Q5: Let's suppose you have written a test suite that achieves 100% decision coverage. Are these tests also guaranteed to give you 100% statement coverage? Discuss and provide examples if possible, why this might or might not be the case.
Q6: You have learned about several kinds of coverage metrics. Which metric would provide the best coverage metric in terms of the quality of your program, i.e. find the most bugs present in the software? Would you use this metric in practice then? Discuss advantages and disadvantages.
After you are done, tar and gzip your answers and upload them into the moodle.