Pipeline¶

Pipeline: Autograding Executable Files

Pipeline components allow you chain together tests for executable files within a few lines, without the headaches of typical executable testing.

class grade.pipeline.pipeline.Lambda(function: Callable[[any], any])¶: Execute arbitrary code in a Pipeline.

class grade.pipeline.pipeline.Pipeline(*callbacks: Callable[[grade.pipeline.completedprocess.CompletedProcess], grade.pipeline.completedprocess.CompletedProcess])¶

Takes callbacks and executes them sequentially when called.

Used for grading executable files or scripts, as it allows you to chain together Run() with things like AssertExitSuccess().

It does this by passing a CompletedProcess object to each successive callback, allowing each callback to modify or replace it during execution of the pipeline.

All Pipelines should start with a call to Run().

Raises an exception if any assertions or run commands fail.

Run¶

class grade.pipeline.run.Run(command, input=None, **kwargs)¶

Runs the given command.

This is the recommended way to start a Pipeline, as it generates a CompletedProcess object to pass down the chain.

You can pass anything to this that you would pass to subprocess.run(), plus one additional note: input can take a callable function. If you choose to pass a callable to input, you can make use of the CompletedProcess object.

An example of passing a callable to input:

>>> Pipeline(
    Run(['cat', 'README.md']),
    Run(['grep', 'pip'], input=lambda r: r.stdout),
    AssertStdoutMatches('python -m pip install grade')
)()

This essentially allows you to chain the output of previous calls to future calls in the pipeline.

Asserts¶

class grade.pipeline.asserts.AssertExitFailure¶: Asserts that the CompletedProcess exited unsuccessfully.

class grade.pipeline.asserts.AssertExitStatus(returncode: int)¶: Asserts that the program exited with a specific error code.

class grade.pipeline.asserts.AssertExitSuccess¶: Asserts that the CompletedProcess exited successfully.

class grade.pipeline.asserts.AssertFaster(duration)¶: Asserts that the most recent call to Run() was faster than duration.

class grade.pipeline.asserts.AssertStderrContains(strings: List[str])¶: Assert programs’ stderr contains all of the string(s) provided.

class grade.pipeline.asserts.AssertStderrMatches(stderr: str = None, filepath: str = None)¶

Asserts that the program’s stderr matches expected.

Expected stderr can be provided in 3 major ways:

Specify the stderr directly as a string
Specify the stderr as a filepath, which is read for stderr content
Specify neither, and it will look through the files contained in CompletedProcess.args for a file that has a counterpart named file.stderr, which it will take an use as stderr.

class grade.pipeline.asserts.AssertStderrRegex(pattern: str)¶: Asserts programs’ stderr contains the regex pattern provided.

class grade.pipeline.asserts.AssertStdoutContains(strings: List[str])¶: Asserts programs’ stdout contains all of the string(s) provided.

class grade.pipeline.asserts.AssertStdoutMatches(stdout: str = None, filepath: str = None)¶

Asserts that the program’s stdout matches expected.

Expected stdout can be provided in 3 major ways:

Specify the stdout directly as a string.
Specify the stdout as a filepath, which is read for stdout content
Specify neither, and it will look through the files contained in CompletedProcess.args for a file that has a counterpart named file.stdout, which it will take and use as stdout.

class grade.pipeline.asserts.AssertStdoutRegex(pattern: str)¶: Asserts programs’ stdout contains the regex pattern provided.

class grade.pipeline.asserts.AssertValgrindSuccess¶: Asserts that there are no valgrind errors reported.

class grade.pipeline.asserts.Check(callback: Callable[[grade.pipeline.completedprocess.CompletedProcess], grade.pipeline.completedprocess.CompletedProcess])¶

Prevents raising exceptions, alters returncode instead.

Wrap any assertion in this to make it a “check” instead of an assert. Check does not raise an exception, but rather alters the returncode to match what the assertion would have done.

class grade.pipeline.asserts.Not(callback: Callable[[grade.pipeline.completedprocess.CompletedProcess], grade.pipeline.completedprocess.CompletedProcess])¶

Negates the enclosed callback.

If the enclosed callback throws an exception, results are returned successfully. If the enclosed callback does not throw an exception, an assertion error is raised.

class grade.pipeline.asserts.Or(*callbacks)¶: Ensures that at least one of the callbacks given passes.

Partial Credit¶

class grade.pipeline.partialcredit.PartialCredit(pipelines: Iterator[grade.pipeline.pipeline.Pipeline], value: Union[int, List[int]])¶

Executes a list of pipelines, assigning credit for each successful run.

If value is a single integer, its’ value is distributed between all pipelines; if value is a list of integers, the values from the list are assigned sequentially to the pipelines and, if the length of the list is not equal to the number of pipelines, it wraps the values and repeats the first entries in the list.

Example: A = Pipeline B = Pipeline C = Pipeline PartialCredit([A, B, C], [1, 2]) A.max_score == 1 B.max_score == 2 C.max_score == 1

property score¶: Returns the aggregate score, raises exception if not run.

Write¶

class grade.pipeline.write.WriteOutputs(testcase)¶

Writes both outputs to files that can be inferred in a Pipeline.

This is mostly useful for creating testcases, it creates output files that can be inferred by the AssertStdoutMatches and AssertStderrMatches callbacks.

class grade.pipeline.write.WriteStderr(filepath: str = 'temp', overwrite: bool = True)¶

Writes the current stderr to the given file.

File defaults to ./temp

class grade.pipeline.write.WriteStdout(filepath: str = 'temp', overwrite: bool = True)¶

Writes the current stdout to the given file.

File defaults to ./temp