Running the Tester
To use the tester, log on to ECF, compile your program (into an executable called either calculator or coins), and then (from the same directory as your program) type:
$ /share/copy/aps105s/A1/a1part1tester
or
$ /share/copy/aps105s/A1/a1part2tester
The tester will print out an introductory message, and begin testing your program:
Running Part 1 Tester...
Note: this tester only checks your formatting, and does not check your results.
We do not check any formatting of the resulting equation printed on the final line,
and we only check a single test case.
Interpreting the Results
If a test passes, the output will look like this:
Running test: Basic test
Test Passed!
If a test fails, you will get something like this:
Running test: Basic test
Test Failed. Here is the difference between the expected output and your output:
Please enter a number:
Please enter the operator (+, -, *, /, or #):
Please enter a number:
- The resulting equaion is "[equation]".
+ The resulting equation is "[equation]".
? +
This is output from a tool called diff, which is used to compare two files. In this case, it is being used to compare your output with the expected output of the program. Each line of its feedback starts with one of a few possible characters: "-", "+", "?", or a blank space. Blank spaces mean that there is no difference between your output and the expected output (i.e., that this line is correct). A "-" indicates that this line is from your output, and a "+" indicates that it's from the correct output. In this case, we have one "-" and one "+":
- The resulting equaion is "[equation]".
+ The resulting equation is "[equation]".
This means that the first line (with the "-") is from your output, and the second line (with the "+") is from the correct output. The two lines are shown side by side so that you can compare them to see what the differences are. In this case, the difference is the spelling of the word "equation".
Sometimes diff will provide more information about the difference that it found. For example, with the previous misspelling, we actually had three lines of output from diff:
- The resulting equaion is "[equation]".
+ The resulting equation is "[equation]".
? +
The third line begins with a "?". This indicates that this line of information from diff is not part of either file that it compared. Instead, this line shows more information about the difference(s) between the previous two lines. On this line, the "+" is being used to highlight the fact that the correct output has one more character than your output, and it is aligned to be under the location where the character is missing.
If you have a different number of lines in your output than in the expected output, you will see lines marked "+" without a matching set of "-" lines (or vice versa). For example, if you forgot to print the middle prompt:
Running test: Basic test
Test Failed. Here is the difference between the expected output and your output:
Please enter a number:
+ Please enter the operator (+, -, *, /, or #):
Please enter a number:
The resulting equation is "[equation]".
The "+" line (without any corresponding "-" line) means that this line occurred in the expected output, but not in your output.
The tester tries to be reasonably smart to figure out complicated mistakes in your output, but it is possible it might get confused if your output is very different than it expected. If you run into this, you should take the same approach that you use when dealing with compiler errors: start with the first mistake, and work down.
Equations
The tester does not check the formatting of your equation in Part 1. When it processes the output from your program, it first replaces everything between the quotation marks in the final line of output with the text [equation]. If it can't find any quotes in the last line, you will get an error message Missing quotes in last line.
Visualizing Whitespace
One potentially confusing aspect of diff is how it visualizes differences in whitespace. For example, let's say that we had an extra space at the end of a line:
Your guess is not correct.
instead of
Your guess is not correct.
The problem is that these two lines look identical to a human reading them, even though there is an extra space at the end of the first one (you can verify this by selecting the text). The fact that the lines look the same makes it hard to show the difference between them, since if diff simply said "these two lines differ", the user might have no idea what diff was talking about. The way that diff will visualize this type of difference is with the help of an underscore to show the extra space:
- Your guess is not correct.
? -
+ Your guess is not correct.
The underscore on the "?" line means that the line above it has a difference at that location. The difference is not readily apparent, since it's a whitespace character, but this is the best that diff can do to highlight it.
Final Comments
As noted by the tester when it runs, this is not an exhaustive test of your program, and passing the tester does not mean that your assignment is 100% correct. We are only testing a single case for each part of the assignment, and we are not checking your answers; we are merely checking the formatting of your output. For calculator, we ignore everything between the quotation marks in the final line, and we do not test the case where the user attempts to divide by 0. For coins, we ignore any Y and N characters.
Finally, if you think the tester is making a mistake (either rejecting output that you think is correct or passing output you know to be wrong), please let me know. The tester does a fair bit of work to try and interpret incorrect output, and it is always possible that it has bugs.