Introduction

clitest is a new kind of CLI testing tool that allows you to write tests for command-line applications using a simple syntax. It was born from the frustration with existing shell testing approaches.

It provides various flexible ways to verify command outputs, handle exit codes, manage test environment variables, and juggle processes. Built on top of Grok patterns, you can match complex patterns in the output with minimal effort.

Tests are as simple as:

$ echo "Hello, world!"
! Hello, world!

Features

Simple and readable test syntax
Support for pattern matching using grok patterns
Flexible output matching with multi-line support
Environment variable management
Control structures for complex test scenarios
Background process management
Temporary directory handling
Cleanup
Retry logic

Why clitest?

clitest makes it easy to write and maintain tests for command-line applications. Its syntax is designed to be concise, human-readable, and powerful, allowing you to express complex test scenarios without extra noise.

Installation

Prerequisites

Before installing clitest, make sure you have Rust and Cargo installed on your system. You can install them by following the instructions on the Rust installation page.

Installing clitest

The easiest way to install clitest is using Cargo:

cargo install clitest

This will download and compile clitest, making it available in your system's PATH.

Verifying the Installation

After installation, you can verify that clitest is properly installed by running:

clitest --version

This should display the version number of your clitest installation.

Updating clitest

To update clitest to the latest version, simply run the installation command again:

cargo install clitest

Uninstalling clitest

If you need to uninstall clitest, you can use Cargo's uninstall command:

cargo uninstall clitest

Basic Usage

Running Tests

To run tests using clitest, just pass the tests files to the clitest command:

clitest [options] [test-file] [test-file] ...

The test runner will exit with a non-zero exit code if any command does not match its expected output.

Test File Structure

Each test file should start with the shebang:

#!/usr/bin/env clitest --v0

The --v0 flag indicates that the test file uses version 0 of the syntax. This ensures backwards compatibility as the syntax evolves in future versions.

Basic Commands

Executing Commands

Commands are prefixed with $:

$ echo "Hello World"
! Hello World

You can split long commands across multiple lines using either backslashes or quotes:

$ echo "This is a very long command that \
spans multiple lines"
! This is a very long command that spans multiple lines

$ echo "This is another way to
split a command across lines"
! This is another way to
! split a command across lines

Comments

Comments start with # and are ignored during test execution:

# This is a comment
$ echo "Hello World"
! Hello World

Basic Output Matching

The simplest way to match output is using the ! pattern, which treats non-grok parts as literal text:

$ echo "Hello World"
! Hello World

Exit Codes

By default, clitest expects commands to exit with code 0. You can specify a different expected exit code using %EXIT:

$ exit 1
%EXIT 1

To expect a command to return a failing exit code (ie: non-zero):

$ exit 1
%EXIT fail

Or to accept any exit code (this will also accept a command that times out):

$ exit 1
%EXIT any

Pattern Matching

clitest provides two main types of pattern matching: auto-escaped patterns (!) and raw patterns (?). Each has its own use cases and syntax.

Auto-escaped Patterns (`!`)

Auto-escaped patterns treat non-grok parts as literal text, making them perfect for exact matches:

$ printf "[LOG] Hello, world!\n"
! [LOG] Hello, world!

Raw Patterns (`?`)

Raw patterns treat everything as a pattern, requiring special characters to be escaped with backslash:

$ printf "[LOG] Hello, world!\n"
? \[LOG\] Hello, world!

You can use ^ and $ anchors in raw patterns for exact line matching:

$ printf "  X  \n"
? ^  X  $

Grok Patterns

clitest supports grok patterns for flexible matching:

$ echo "Hello, anything"
? Hello, %{GREEDYDATA}

Common grok patterns:

%{DATA} - Matches any text
%{GREEDYDATA} - Matches any text greedily

You can also customize grok patterns by providing a name and value:

$ printf "[LOG] Hello, world!\n"
? \[%{log=(LOG)}\] %{GREEDYDATA}

Multi-line Matching

Auto-escaped Multi-line (`!!!`)

$ printf "a\nb\nc\n"
!!!
a
b
c
!!!

Raw Multi-line (`???`)

$ printf "a\nb\nc\n"
???
a
b
c
???

When using multi-line patterns, the indentation of the !!! or ??? lines is removed from all lines between them. This makes it easy to maintain proper indentation in your test files while matching unindented output:

$ printf "abc\n\ndef\n"
  !!!
  abc

  def
  !!!

Pattern Structures

Any Pattern (`*`)

The * pattern matches any number of lines lazily, completing when the next structure matches. It can be used at the start, middle, or end of patterns:

# Match any output
$ printf "a\nb\nc\n"
*

# Match start, any middle, end
$ printf "a\nb\nc\nd\ne\n"
! a
! b
*
! d
! e

# Match within repeat
$ printf "start\n1\n2\nend\nstart\n1\n2\nend\n"
repeat {
    ! start
    *
    ! end
}

Pattern Blocks

Pattern blocks allow you to combine multiple patterns in different ways:

Repeat

Match a pattern multiple times:

$ printf "a\nb\nc\n"
repeat {
    choice {
        ! a
        ! b
        ! c
    }
}

Choice

Match any one of the specified patterns:

$ echo "pattern1"
choice {
    ! pattern1
    ! pattern2
    ! pattern3
}

Unordered

Match patterns in any order:

$ printf "b\na\nc\n"
unordered {
    ! a
    ! b
    ! c
}

Sequence

Match patterns in strict order:

$ printf "a\nb\nc\n"
sequence {
    ! a
    ! b
    ! c
}

Optional

Make a pattern optional:

$ echo "optional output"
optional {
    ! optional output
}

Ignore

Ignore blocks are supported at the command and global level. Global ignore blocks are applied to all commands in the test, while command-level ignore blocks are applied to the command only.

Skip certain output:

$ printf "WARNING: Something happened\nHello World\n"
ignore {
    ? WARNING: %{DATA}
}
! Hello World

Reject

Reject blocks are supported at the command and global level. Global reject blocks are applied to all commands in the test, while command-level reject blocks are applied to the command only.

Ensure certain patterns don't appear:

$ echo "Hello World"
reject {
    ! ERROR
}
! Hello World

Conditional Patterns

You can use if blocks in patterns to conditionally match output:

if $TARGET_OS == "linux" {
    $ echo Linux specific output
    ? Linux specific %{GREEDYDATA}
}

Note that pattern if blocks and control if blocks have identical syntax, but one contains patterns and the other contains commands.

Pattern Examples

Matching Log Lines

$ printf "[INFO] User logged in\n[ERROR] Connection failed\n"
repeat {
    ! [%{WORD}] %{GREEDYDATA}
}

Matching Numbers

$ echo "Count: 42"
? Count: %{NUMBER}

Matching Dates

$ echo "Date: 20-03-2024"
? Date: %{DATE}

Grok patterns

Grok patterns are a way to parse text into structured data.

Syntax

A grok pattern is constructed using one of the formats:

%{PATTERN_NAME}: a standard named pattern
%{PATTERN_NAME=(regex)}: a custom pattern defined using a regular expression
%{PATTERN_NAME:field_name}: a standard named pattern with a named output
%{PATTERN_NAME:field_name=(regex)}: a custom pattern with a named output

Examples

The most basic pattern is %{DATA}, which matches any text lazily: as few times as possible for the remainder of the line to match. Alternatively, you can use %{GREEDYDATA} to greedily match any text, as many times as possible while allowing the remainder of the line to match.

$ echo "Hello, world!"
! Hello, %{DATA:what}!

Custom patterns are defined using the pattern command, after which the patterns are available for use in the tests.

pattern GREETING Hello|Goodbye

$ echo "[INFO] Hello, world!"
! [%{LOGLEVEL}] %{GREETING}, %{DATA}!

A custom pattern may also be defined inline:

$ echo "[DEBUG] Hello, world!"
! [%{CUSTOMLEVEL=INFO|DEBUG}] %{GREETING=(Hello|Goodbye)}, %{DATA}!

Custom patterns may be named and reused in a single line:

$ echo "[DEBUG] Hello, world!"
! [%{MY_WORD=(\w+)}] %{MY_WORD}, %{MY_WORD}!

Patterns may have named outputs. This feature is supported, but you cannot use the named outputs for any other purpose yet.

$ echo "[DEBUG] Hello, world!"
! [%{MY_WORD:word1=(\w+)}] %{MY_WORD:word2}, %{MY_WORD:word3}!

References

For further reading, see:

Tools

Some potential tools for working with grok patterns:

Control Structures

clitest provides several control structures to help you write complex test scenarios.

Quoting

Note that internal commands and control structures follow shell-style syntax, so quoting is significant.

Single quotes (') preserve the literal value of every character within the quotes. No characters inside single quotes have special meaning.

Double quotes (") preserve the literal value of most characters, but still allow for variable expansion (e.g., $VAR or ${VAR}).

Backslashes (\) can be used to escape the next character, preserving its literal meaning. This works both inside double quotes and unquoted text.

For Loops

The for block allows you to iterate over a list of values:

for OS in "linux" "macos" "windows" {
    $ uname -a | grep $OS
    %EXIT any
    optional {
        ! %{GREEDYDATA}
    }
}

Conditional Blocks

You can use if blocks to conditionally execute commands:

if TARGET_OS == "linux" {
    $ echo Linux specific output
    ! Linux specific output
}

Note that pattern if blocks and control if blocks have identical syntax, but one contains patterns and the other contains commands.

Background processes

Run commands in the background using background { }. When the block ends, the background process is automatically killed. If the test exits early (e.g., due to a failure), background processes are also killed.

Commands running in a background block have no explicit timeout, but you can set an explicit timeout for each command with %TIMEOUT if needed.

using tempdir;

background {
    $ python3 -m http.server 60801 2> server.log
    %EXIT any
}

$ echo "OK" > health

retry {
    $ curl -s http://localhost:60801/health
    ! OK
}

Deferred cleanup

Run commands after the block finishes. Multiple defer blocks are executed in reverse order (last in, first out):

defer {
    $ echo "Second cleanup"
    ! Second cleanup
}

defer {
    $ echo "First cleanup"
    ! First cleanup
}

$ echo "Running!"
! Running!

$ echo "Done!"
! Done!

Retry

Retry commands until they succeed or timeout:

retry {
    $ true
}

retry uses the global timeout for the whole retry block, but you can set a shorter timeout for the command itself with %TIMEOUT:

retry {
    $ true
    %TIMEOUT 100ms
}

Environment and Variables

clitest provides powerful features for managing environment variables and working directories.

Setting Variables

Using `%SET`

Capture command output into a variable:

$ printf "value\n"
%SET MY_VAR
*

Using `set`

Set environment variables directly:

set FOO bar;
set PATH "/usr/local/bin:$PATH";

Variable References

Basic Reference

Use $VAR to reference variables:

set FOO bar;
$ echo $FOO
! bar

Explicit Reference

Use ${VAR} when the variable name is followed by text:

set FOO bar;
$ echo ${FOO}123
! bar123

Working Directory Management

The working directory is managed through a special variable PWD. This can be set directory, or various commands can change it.

Changing Directory

Change the current working directory. The PWD is updated to the new directory for the duration of the test, unless another command changes it:

cd "subdir";

Using Temporary Directories

Create and use a temporary directory. The current working directory is automatically set to the temporary directory, and when the block ends, the temporary directory is automatically deleted.

using tempdir;

Creating New Directories

Create a new directory for testing. The current working directory is automatically set to the directory, and it is deleted when the block ends.

using new dir "subdir";

Using Existing Directories

Use an existing directory. The current working directory is automatically set to the directory, and it is not deleted when the block ends.

using tempdir;
$ mkdir -p subdir
using dir "subdir";

Special Variables

PWD

The PWD variable is special and controls the current working directory:

$ mktemp -d
%SET TEMP_DIR
*

# Set PWD to change working directory
$ echo $TEMP_DIR
%SET PWD
*

Environment Variable Examples

Combining Variables

set A 1;
set B 2;
set C "$A $B";
$ echo $C
! 1 2

Using Variables in Commands

set DIR "subdir";
using new dir "$DIR";
$ echo $PWD
! %{PATH}/subdir

Conditional Environment Setup

if $TARGET_OS == "linux" {
    set PATH "/usr/local/bin:$PATH";
}

Advanced Features

This chapter covers advanced features and best practices for using clitest effectively.

Complex Pattern Matching

Nested Patterns

Combine different pattern types for complex matching:

$ printf "a\nb\nc\nd\n"
sequence {
    ! a
    repeat {
        choice {
            ! b
            ! c
        }
    }
    ! d
}

Conditional Pattern Matching

Use conditions with patterns:

if $TARGET_OS == "linux" {
    $ echo Linux specific output
    ? Linux specific %{GREEDYDATA}
}

Process Management

Background Processes

Run and manage background processes, using retry to wait for the process to start:

using tempdir;

background {
    $ python3 -m http.server 60800 2> server.log
    %EXIT any
}

$ echo "OK" > health

retry {
    $ curl -s http://localhost:60800/health
    ! OK
}

# Test the server
$ curl -s http://localhost:60800/health
! OK

# Background processes are automatically killed

Process Cleanup

Ensure proper cleanup with defer:

defer {
    $ killall background-server
    %EXIT any
    *
}

background {
    $ python3 -m http.server 60800
    %EXIT any
}

$ echo 1
! 1

Timeouts

Set a timeout for a command:

$ sleep 60
%EXIT timeout
%TIMEOUT 100ms

Error Handling

Expected Failures

Test error conditions with %EXIT. %EXIT any will allow any exit code or signal, while %EXIT n will only allow exit code n:

$ false
%EXIT 1

Expecting Failures

If you want to verify that a pattern does not match, use %EXPECT_FAILURE. This can be useful in certain cases, but you should prefer a reject { } block if you just want to test that a certain pattern never matches.

$ echo "Hello World"
%EXPECT_FAILURE
! Wrong Output

Best Practices

Test Organization

Group related tests together and use descriptive comments.
Keep tests focused and atomic
Use variables for reusable values
Use ignore for noisy output, prefer a global ignore block over many, repeated ignore { } blocks.
Add defer blocks for cleanup immediately after allocation or creation of resources.

Example of Complex Test

# Global ignore/rejects
ignore {
    ! No configuration file found, creating default at %{GREEDYDATA}
}

reject {
    ! Critical error, corrupted configuration file.
}

# Test server startup and basic functionality
using tempdir;

# Start server in background
background {
    $ echo "{\"status\": \"success\"}" > api.json
    $ echo "OK" > health.json
    $ python3 -m http.server 60900 2> server.log
    %EXIT any
}

defer {
    $ rm server.log
}

# Wait for server to start
retry {
    $ curl -s http://localhost:60900/health.json
    ! OK
}

# Test main functionality
$ curl -s http://localhost:60900/api.json
! {"status": "success"}

# Verify logs
$ cat server.log
repeat {
    choice {
        ? %{IPORHOST} %{GREEDYDATA} code %{NUMBER}, %{GREEDYDATA}
        ? %{IPORHOST} %{GREEDYDATA} "GET /%{DATA} %{DATA}" %{NUMBER} -
    }
}

# Cleanup is automatic!

Keyboard shortcuts

clitest Documentation