CURSES TESTFRAME ---------------- 1. Introduction The curses library is a complex piece of software and, often, changes made to the library may introduce subtle bugs that are hidden by other actions so a visual check of the curses output may look correct in some circumstances and the bug only show itself after a certain sequence of actions. To assist with validating that changes made to the curses library have no undesired effects an automated test is needed to detect and highlight any changes in the curses application output stream. The programmer can then analyse the output changes and either correct a bug or update the automated test to accept the new output as valid. 2. Architecture The curses testframe consists of two separate programs connected by a number of pipes and a pseudo-tty (pty). The programs are called the director and the slave. The director reads a configuration file of tests to perform, passes these commands to the slave over a pipe and reads the pty for any output from the slave. Data from the slave is compared against expected output held in a file and any differences are highlighted to the tester. The slave is a curses application that is forked by the director on start up. It reads commands from the director over a pipe, these commands are calls to curses routines along with the parameters required for the call. The slave takes the parameters and uses them as arguments for the requested curses routine call. The return value from the curses routine is passed back to the director over another pipe, if the curses routine updates any passed by reference arguments then these are also passed back to the director for analysis. 3. Director The director has the following optional command line options: -v enables verbose output to assist debugging -s slave_path the director will execute slave_path as the slave process. The default is ./slave -t term Sets the TERM environment variable to term when executing the slave. The default is atf There is one mandatory command line parameter, that is a file name that contains the test command file. The test command file holds the calls required to exercise a particular curses routine and validate both the return codes from the routines and the output from the slave. The test language has a small number of commands, they are: assign: Assign a value to a variable. The syntax is: assign var_name value Where var_name is the name of the variable. Variable names are an arbitrary sequence of alphanumeric characters, the variable name must start with an alphabetic character. Value is the value to be assigned. The value can either be a numeric or a string type. Variables are created on first use and will be overwritten on each subsequent use. call, call2, call3, call4: All these are used to call curses routines, the only difference between then is the number of expected return values. Call expects one return value, call2 expects 2, call3 expects 3 and call4 expects four. Any parameters that are passed by reference and updated by the call are treated like returns. So, for example, calling the function getyx() which has three parameters, the window, a pointer to storage for y and a pointer to storage for x would be called like this: call3 OK 4 5 getyx $win1 Which calls getyx, the first (and possibly only) return is the return status of the function call, in this case we expect "OK" indicating that the call succeeded. The next two returns are the values of y and x respectively, the parameter $win1 is a variable that was assigned by a previous call. Any return can be assigned to a variable by including the variable name in a call return list. Variables are referenced in a call parameter list by prefixing the name with a $ character. All returns are validated against the expected values and an error raised if there is a mismatch. The only exception to this is when the return is assigned to a variable. Valid values for the returns list are: variable - assign the return to the given variable name. numeric - the value of the return must match the number given. string - an arbitrary sequence of characters enclosed in double quotes. ERR - expect an ERR return OK - expect an OK return NULL - expect a NULL pointer return NON_NULL - expect a pointer that is not NULL valued There is one special parameter that can be passed to a call, that is the label STDSCR. This parameter will be substituted by the value of stdscr when the function call is made. check: Validate the value of a variable. This allows a variable to be checked for an expected return after it has been assigned in a previous call. The syntax is: check var_name expected_result Where var_name is a variable previously assigned and expected_result is one of the valid return values listed in the above call section. compare: Compares the output stream from the slave against the contents of a file that contains the expected output. The syntax is: compare filename Where filename is the name of the file containing the expected output. The file can either be an absolute path or relative path. In the latter case the value of the environment variable CHECK_PATH will be prepended to the argument to provide the path to the file. The contents of this file will be compared byte by byte against the output from the slave, any differences in the output will be flagged. If the director is not in verbose mode then the first mismatch in the byte stream will cause the director to exit. comparend: Performs the same function as the above compare except that excess output from the slave is not discarded if there is more data from the slave than there is in the check file. This allows chaining of multiple check files. delay: Defines an inter-character delay to be inserted between characters being fed into the input of the slave. The syntax is: delay time Where time is the amount of time to delay in milliseconds. include: Include the contents of another test file, the parser will suspend reading the current file and read commands from the include file until the end of file of the include file is reached at which point it will continue reading the original file. Include files may be nested. The syntax is: include filename Where filename is the name of the file to include. If the filename is not an absolute path then the contents of the environment variable INCLUDE_PATH are prepended to the file name. input: Defines a string of characters that will be fed to the slave when a call requires input. Any unused input will be discarded after the call that required the input is called. The syntax is: input "string to pass" noinput: Normally the director will error if an input function is called without input being previously defined, this is to prevent input functions causing the test to hang waiting for input that never comes. If it is known that there is pending input for the slave then the noinput keyword can be used to flag that the input function has data available for it to read. The noinput command only applies to the next function call then behaviour reverts to the default. The testframe can define different types of strings, the type of string depends on the type of enclosing quotes. A null terminated string is indicated by enclosing double (") quotes. A byte string, one that is not null terminated and may contain the nul character within it is indicated by enclosing single (') quotes. A string of chtype character which are a combined attribute and character value is indicated by enclosing backticks (`), for this type of string pairs of bytes between the backticks are converted to an array of chtype, the first byte is the attribute and the second is the character. All strings defined will have a simple set of character substitutions performed on them when they are parsed. This allows the tester to embed some control characters into the string. Valid substitutions are: \e escape \n new line \r carriage return \t tab \\ \ character \nnn Where nnn is three octal digits, the character represented by the octal number will be inserted into the string. Any other invalid conversions will have the \ stripped and the subsequent characters inserted into the string. Integers may be specified by either a plain numeric (e.g. 12345) or by hexadecimal notation by prefixing the number with 0x (e.g. 0x3039). Internally, no distinction is made between the two formats and they can be freely intermixed. Integers and variables containing integers can have operations performed on them. Currently only bitwise ORing numbers together is supported. This can be done by separating a list of integers and variables with the pipe (|) symbol and enclosing the entire list in round brackets "()" like this: ( $var1 | 0x0100 | $var2 | 512 ) Variables and integer constants may be freely intermixed. The result of the operation can either be used as an argument for a call or can be used as an expected result for a call. In addition to all the curses calls being supported by the slave, there is one more special call called "drain". This call repeatedly called getch() until there are no more characters in stdin. The call assumes that the curses input is either in no delay or timed input mode otherwise the test will time out and fail. This call can be used to clear any pending input when testing testing a timed read to prevent the input being used in a later test. 4. Slave The user has no direct interaction with the slave process. The slave is forked off by the director communicates to the director over a set of pipes and a pseudo-tty connected to its standard i/o file descriptors. The slave executes the passed curses calls and passes back return values to the director. The slave automatically calls initscr() on start up.