Assignment two questions and answers

Most of all, remember to "Keep It Simple". A more complex program is more likely to have bugs; it takes you longer to write; it is harder to maintain; it is unlikely to be more usable. Most cutesy features are not helpful, and are measurably hurtful.

In pragmatic assignment-writing terms, cutesy features don't get you extra marks, but the probabilistic expectation is that they will lose you marks on average, because they will introduce bugs which affect the working of the non-cutesy parts of the program.

That is to say: Wield your cleverness cleverly. Don't waste your cleverness in doing silly things.

Two pithy quotations:

"The superior pilot uses his superior judgement to avoid situations in which he has to demonstrate his superior skill."
(traditional pilot saying)

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan

More general advice:

Don't store text for any longer than or in any more of a complex data structure than you need to. Process it as you go.

Where possible, we even process data character by character, as in the example cat.c. However, myfgrep will need to process data line by line.

And this will already illustrate why any storing of data is inconvenient. You will want to choose a maximum line size. I suggest 1000. If you write it correctly, longer lines will simply be split, they won't cause you to exceed array bounds.

The directory /u/ajr/209/a2 on CDF contains several files you might find useful:

• readdir.c demonstrates reading directories with opendir() and readdir()
• lstat.c demonstrates finding some basic file information (mostly, the "ls -l" data) using lstat().
• time.c demonstrates a little working with "time_t" data values, and calling localtime() to break the time down into year/month/day/hours/minutes/seconds. (see "man 2 time" and "man 3 localtime")
• strcmp.c demonstrates the very simple use of strcmp()
• strstr.c demonstrates an even simpler use of strstr(), just looking to see if a string is contained in another, not paying much attention to the exact result (this is exactly what you need in myfgrep)
• decodeuid.c illustrates using the "pwent" ("password entry") suite to turn a numeric uid into a text logname (easy)
• decodegid.c illustrates the same thing for gids and group names (almost identical)

What you need to know about "structs" in C for this assignment was discussed in tutorial on October 18, and structs are described fully in http://www.dgp.toronto.edu/~ajr/209/notes/struct.html

Please see manpage.pdf and manpage-ajr.html

Please try out sample solutions (compiled programs only, so that you can't see my sample solution source code before the due date!), in /u/ajr/209/a2 on CDF.

I am happy to interpret compiler error messages (for CSC 209 students). Sometimes the compiler will emit error messages which you might find cryptic. I won't fix your assignment code for you, but I will tell you more clearly what a particular error message means. I will, sometimes, fix non-assignment-related code (although more frequently I'll give you hints instead).

The assignment handout does not necessarily specify all of the details of the required behaviour of your programs in all cases. I've tried to specify most things, but generally speaking, your programs are required to behave like standard unix tools. To answer some questions I've provided compiled sample implementations in /u/ajr/209/a2 on CDF.

For example, the (required) "usage" message has a very specific format, which you must adhere to. It is similar to the SYNOPSIS section of the man pages, with their meaning for square brackets (indicating that something is optional) and ellipses (indicating "one or more of"). You can take the usage messages from the behaviour of the example compiled programs in /u/ajr/209/a2 if you like; and very little variation is acceptable. The token immediately following the "usage:" string can be either argv[0] or the base program name.
[lsl has no usage message because there's no way to write an incorrect command-line for lsl.]

Error messages must be to stderr, not stdout. And where perror() is applicable, it is obligatory, rather than formulating your own error message. perror() writes to stderr.

In general, don't check whether operations will succeed; just try to do them and get an appropriate error if applicable. For example, if you're about to fopen() a file, don't do a stat() and try to determine whether the file exists and/or is readable. Just do the fopen() and check for error. This results in a simpler program, and also one which functions more correctly in the invariable case that you have omitted checking something so you think it's going to succeed but it doesn't. And there can always be unexpected i/o errors, etc.

Q: When I compile my program I get the following warning message: [...]
Is this ok?

A: No. Your program should compile with "gcc -Wall" with no warning or error messages. Almost all of the warning or error messages which gcc -Wall can output represent potentially-serious problems, and you need to fix them. I am willing to decode error messages by e-mail (although not generally to fix your bugs, obviously).

Here are some useful example programs for the basic structure of myfgrep.c.

First of all, I presented in lecture a basic 'cat' program which goes through its arguments and correctly calls fopen(), perror(), and fclose(). Almost all unix tools which take file name arguments will go through their file name arguments in this basic way.

Also, you should use the getopt() library function. This parses unix command-line options (beginning with a '-') in a standard way, taking care of various features such as the ability to put key letters either together in one argument (e.g. "ls -lc") or separately (e.g. "ls -l -c"), as well as dealing with "--", and so on. You should read the man page (don't use "getopt_long()" for this assignment -- that is a GNUism, albeit one which is increasing in popularity in the rest of the unix world -- anyway, it's not needed here and it's more complex), and you will also want to look at a demonstration of the use of getopt(). Please understand that program fully before copying any of it!

Also see ../notes/argv.html.

Q: Can I put some functions in a separate .c file and submit that too?

A: No. Your programs for this assignment are small enough that it isn't really worth it to separate them into multiple files.

Q: Can I submit a .h file so that I can declare some functions and/or variables?

A: No, just declare them at the top of your .c file (or wherever is appropriate). The purpose of .h files is to coordinate declarations across multiple files. Each of your files can be self-contained for this assignment.

Q: What is the difference between using fopen(), getc() or fgets() or fscanf(), then fclose(); as opposed to using open(), read(), and close()? Which should we use?

A: You should use the 'f' functions. (By which term I mean to include getc() -- i.e. you should feel free to use getc().)

The 'f' functions (fopen(), getc()/etc, fclose()) are part of the standard i/o library, which was built on top of the unix kernel calls (open(), read(), close()) for two reasons:

1. portability: The low-level file primitives work(ed) differently on different operating systems, but the stdio functions were designed to be implementable on top of any of them
2. buffering. When you do getc(), it uses read() to read a bunch of bytes, not just one, and then getc() gives you one at a time, until you exhaust the buffer and then it does another read(). Even on modern computers this makes a significant speed difference.

You can get the unix file descriptor underlying a FILE* with the fileno() function (that is, fileno(fp) is the file descriptor number). You can go the other way by using fdopen(), which creates all the FILE* stuff around an already-opened file identified by file descriptor number ("fd"). These are rarely necessary and won't be of use to us in this course.

Q: How do you print to the standard error?

A: Use fprintf(stderr, "format" ... ), or any other stdio function which accepts a value of type FILE*

Also, perror() prints its message to stderr.

Q: But when I do fprintf(stderr, "Hello, world\n"), I still see it on the screen.

A: Both stdout and stderr are initially connected to your terminal window, but they can be redirected independently.

If your program says fprintf(stderr, "Hello, world\n"), then if you run ./a.out >file, you'll still see "Hello, world" on the screen and it won't go into the file. This is the purpose of using stderr, as previously discussed in lectures.

Various getopt() questions:

See "man getopt". But typing "man getopt" gives you a tool for use in shell programming. So say "man 3 getopt". (And to be clear, you should be using getopt(), not getopt_long(), for assignment two.)

You want a colon in your third argument to getopt() for myfgrep. Specifically, your third parameter to getopt() should be the string "m:q" (or, equivalently, "qm:"). This means that the -m option has a required argument and the -q option does not. See the supplied example call of getopt() in getopt.c.

Here are some notes about getopt, of which you might want to read the "interface" section, after reading getopt.c above.

You are required to use getopt() for myfgrep.c, rather than parsing the command-line options yourself. All sorts of bizarre syntaxes are possible and will be dealt with automatically by getopt(). In the old days, everyone writing a unix tool parsed the options themselves, and the result was a lot of inconsistency as to whether or not you could do certain things (even including fundamentals such as combining options into one argument, e.g. writing "ls -qa" instead of "ls -q -a"). These days, everyone calls getopt(), and the users of your program may use a feature of standard option parsing which you didn't even know exists. This is good.

Be careful to use getopt() properly. Do not make assumptions as to the format of the input line. The standard unix command-line option format is actually extremely flexible in some ways. For example, these are all valid ways to execute your "myfgrep" command with '-m' value 17 and the '-q' flag on, with a pattern of "pattern" and a file named "file":

	myfgrep -m17 -q pattern file
	myfgrep -q -m17 pattern file
	myfgrep -qm17 pattern file
	myfgrep -qm 17 pattern file
	myfgrep -q -m 17 pattern file
	myfgrep -q -m 17 -- pattern file
	myfgrep -qm17 -- pattern file
	myfgrep -m1 -q -m2 -q -m3 -q -m4 -q -m5 -q -m17 -q pattern file

Don't use getopt() for lsl.c.

Q: If one of the files specified on the myfgrep command line cannot be opened, should we exit immediately or do we have to continue on through the rest of the arguments like cat does?

A: You have to call perror(), and you have to exit with a non-zero exit status eventually. So the easiest thing is just to exit right away. For myfgrep, it's ok (desirable, even) to process the remaining files which do exist, correctly; but it's not required for this assignment. You'll find that some standard unix tools proceed after error in this way and some don't.

The return type of getchar() and getc() is int, not char, and you can't store it in a char variable. With 8-bit chars, there are 257 possible return values: 0 to 255 indicating a byte of input (that's 256 possible values there), or -1 to indicate eof or error.

A value with 257 possible values cannot be stored in an 8-bit char. If you attempt to do so, e.g. if you have

	char c;
	while ((c = getc(fp)) != EOF) {

Q: The assignment handout says that "lsl" takes no command-line options. If the user does "./lsl -q", is that an error?

A: No, it is a request to do an "ls -l" of a file/directory named "-q" in the current directory. That is, this is not a special case. Keep It Simple.

Q: The assignment handout says that "lsl" acts like "ls -l". Does that mean we should avoid listing any file whose name begins with a dot?

A: No, you don't have to do this. Personally I think that that default behaviour of ls is silly. My solution skips only the files "." and ".." -- this is important for recursive traversal of the file system, although we're not doing that in lsl. For this assignment it's ok if you include or skip any files whose names begin with a dot, so long as you don't make it too complicated.

An example of reading a directory with opendir() / readdir() / closedir() can be found in the file readdir.c. The C "->" syntax was mentioned in tutorial on October 18th -- basically it means the same as Java's "." when used to select members of an object; and for this assignment, you only need to use it exactly as shown in supplied code examples. We'll do it in greater detail later, or see http://www.dgp.toronto.edu/~ajr/209/notes/struct.html

Q: What is a DIR* ? (the return value from opendir())

A: It is basically the same as a FILE*. In fact, an implementation of opendir() and friends which I've read the source code to just says "#define DIR FILE" in dirent.h. But some of them don't, so you should declare it correctly. But that's what it is -- it's the data associated with opendir()'s idea of the stream which is the open directory for reading. Readdir() then reads records from this structured file.

lstat() takes two parameters, as illustrated in lecture and in the example lstat.c The first parameter is the file path name. It can be an absolute or relative path name, etc.

The second parameter is a pointer to a "struct", which is simply a dataless class. That is, classes are structs plus functions plus data hiding rules (and C doesn't have classes, only structs). We will be discussing structs in detail in class shortly, but the above is all you need to know to use lstat(). As with scanf(), you pass lstat() a pointer to the struct because you are not trying to pass it the previous contents of that data structure but rather a pointer to where you want it to put its results.

lstat() returns an error status: 0 for success, -1 for failure. If lstat() succeeds, you can consult the members of that struct (the struct you passed a pointer to, as lstat()'s second parameter). A full list of its fields with descriptions can be found in the man page ("man 2 lstat"); one field you will be interested in for this assignment is st_mode, which contains information including that 'D' in our unix filesystem notes, and the permissions information. The st_mode member is commonly tested with a function from sys/stat.h which checks the bits of the mode corresponding to the file type, e.g. is it a directory, symlink, etc. As illustrated in the example lstat.c, call S_ISDIR(whatever.st_mode) to check whether it is a directory, and so on.

lstat() is declared in #include <sys/stat.h>. However, that file uses some type declarations in #include <sys/types.h>, so you have to #include that first, as shown in the above example. (Those system-oriented type declarations appear in <sys/types.h> rather than being replicated in several independent #include files in the sys directory. (Speaking of which, the directory for #include is /usr/include, if you're curious.))

You can read documentation about the lstat() system call by typing "man 2 lstat". Man page volume 1 is commands, volume 2 is system calls, and volume 3 is C library calls.

Q: What's the difference between stat() and lstat()?

A: For most directory-tree-traversing programs, including lsl, it's important to use lstat(), as follows.

For the most part, if you attempt to access a symbolic link, the kernel follows this symbolic link automatically, giving you instead the file that the symlink points to. If this weren't the case, then symlinks wouldn't mean what they do mean. A symlink is a stand-in for the pointed-to file.

But you can't have the kernel always following symlinks, only almost-always. For example, an ls -R, or du, would get very confused by symlinks if it called stat() rather than lstat(). In particular, if a symlink points to a parent directory, then to opendir that symlink and continue traversing from there will result in infinite recursion.

So when symlinks were introduced, a dozen or so programs needed to be modified to be able to continue to work in their presence. These days, many more programs need to be aware of symlinks. Anything which traverses a directory tree needs to treat symlinks-which-point-to-a-directory differently from directories. Programs such as "ls" need to collect information on the symlink, rather than the pointed-to file.

The way to do this is to call the special call "lstat()", which is like stat() so long as its parameter is not a symlink. If its parameter is a symlink, it does not follow the symlink, but rather, reports information about the symlink itself.

Thus "ls -l" calls lstat(), not stat(). There is an option '-L' to make it follow the symlinks, but otherwise it doesn't.

For more examples: "test -f" calls stat(), but "test -L" (check whether the file is a symlink) needs to call lstat().

C's "sizeof" operator does not give you the size of an array, in general. If you think there's no way to write a particular bit of code without using sizeof, then sizeof probably won't help you there, either. In particular,

void f(int *a)
{
    int i;
    for (i = 0; i < sizeof a; i++)   /* WRONG */
        ...
}

is completely wrong. It will not iterate the right number of times. The variable 'a' will have size 4 on the CDF machines, because that's how many bytes are used by a pointer. If you want to know the number of elements in the array which 'a' points to, you need to pass that value in as a second parameter, of type int.

Q: Can we assume a maximum path name length in lsl.c?

A: Well, sort of. You can set a maximum (make it at least, say, 1000 chars) so that you can declare your array, but if the path name is too long, you must print an appropriate message to stderr and exit; nothing can be permitted to make you exceed the array bounds.

Standard indentation is required in your C programs.

Your program must not exceed array bounds no matter what the user input (or command-line arguments).

Most cases of programs I see at this point in this course which contain lurking bugs of this nature are actually copying data entirely unnecessarily. Don't copy data when the original is just as good as the copy. For example, strings in the argv array can be used from that array directly, without copying the string data.

When writing your man pages, please note that blank lines are significant in the [nt]roff input format (quite unlike in HTML for example).

In the man page format, we don't use blank lines. If you need to leave some vertical space for the legibility of the troff source code (i.e. in your file, not in the result), you can use the null command, which is a dot alone on the line. Comments begin with \" (backslash-doublequote) and go to the end of the line, so you can write stuff like:

. \" cite analyze(1) here when it exists

Q: How do we see what the man page will look like formatted?

A: Use "nroff -man myfgrep.1". You probably want to pipe this to "less", which will not only give you paging facilities but will show underlining and bold. Some versions of "less" will automatically run nroff when the file name ends in .1 and it seems to be a man page.

Q: How do we run off a typeset man page?

A: With gnu troff (e.g. in linux), use "groff -man myfgrep.1 >myfgrep.ps". That is, groff's output by default is in postscript. You can view the file with a postscript previewer (e.g. ghostview) or you can print it on a postscript printer with lpr. You can convert it to a pdf file by typing "epstopdf myfgrep.ps", which creates a myfgrep.pdf.

Q: How long should my man page be?

A: Very short. There isn't a lot to say. Be brief, and avoid extraneous sections or information. For example, the error messages output by your program will be standard ones (via perror()), so omit the DIAGNOSTICS section.

Q: For the SYNOPSIS section, it says in manpage.pdf that leaving spaces within the brackets is an "historical practice" and "should perhaps be discouraged". So, do we put in the spaces or not?

A: Either way is fine for the assignment. I think the modern style tends towards leaving them out, but the "-man" macros might make it easier to put them in. Do be consistent within your SYNOPSIS line!

Q: It isn't showing my "..." line. (literally "..." -- three dots)

A: Lines beginning with "." are nroff commands. Commands which are not recognized are not diagnosed; thus, a line beginning with "..." is like a comment.

You can precede your line with the special character "\&". This is a character which is blank and of zero width, so it doesn't do anything... except that your line will no longer begin with a ".". This is the primary use of "\&" in nroff/troff.

Q: It doesn't say "Unix Programmer's Manual" or "Linux Programmer's Manual" at the top; how do I fix this?

A: Portably speaking, you don't. The system "-man" macros are supposed to supply an appropriate default title. If they don't, it's not your fault; your man page will look better on other systems. You could file a bug with your OS distributor, but don't complicate your own files over this.

In your man pages, avoid the GNUish sections "REPORTING BUGS" and "COPYRIGHT". You retain copyright over your code but you do not need to assert it in the man page.

Also avoid the GNU practice of writing "command [options] files" for the SYNOPSIS section. This has no information content. Write a standard SYNOPSIS section as in unix man pages.

Do not include both an AUTHOR and a HISTORY section. Stick with one or the other. Or neither.

Don't write things twice (e.g. AUTHOR: ajr; HISTORY: written by ajr), and don't include any blank sections or sections whose contents say something like "none known" or "nothing to say here".

You have nothing to say for a DIAGNOSTICS section, either. Leave it out. Short is good. Your program will emit standard diagnostics and the unix user is expected to know how those work (perhaps better than you yet do at this point in CSC 209! but you will soon).

The "template" at the end of the Henry Spencer document is your best overall reference for the content of the man pages.