Introduction

CLI/test is a new kind of CLI testing tool that allows you to write tests for command-line applications using a simple syntax. It was born from the frustration with existing shell testing approaches.

It provides various flexible ways to verify command outputs, handle exit codes, manage test environment variables, and juggle processes. Built on top of Grok patterns, you can match complex patterns in the output with minimal effort.

Tests are as simple as:

$ echo "Hello, world!"
! Hello, world!

Features

• Simple and readable test syntax
• Support for pattern matching using grok patterns
• Flexible output matching with multi-line support
• Environment variable management
• Control structures for complex test scenarios
• Background process management
• Temporary directory handling
• Cleanup
• Retry logic

Why CLI/test?

CLI/test makes it easy to write and maintain tests for command-line applications. Its syntax is designed to be concise, human-readable, and powerful, allowing you to express complex test scenarios without extra noise.

Installation

Prerequisites

Before installing CLI/test, make sure you have Rust and Cargo installed on your system. You can install them by following the instructions on the Rust installation page.

Installing CLI/test

The easiest way to install clitest is using Cargo:

cargo install clitest

This will download and compile clitest, making it available in your system's PATH.

Verifying the Installation

After installation, you can verify that CLI/test is properly installed by running:

clitest --version

This should display the version number of your CLI/test installation.

Updating CLI/test

To update clitest to the latest version, simply run the installation command again:

cargo install clitest

Uninstalling clitest

If you need to uninstall CLI/test, you can use Cargo's uninstall command:

cargo uninstall clitest

Basic Usage

Running Tests

To run tests using clitest, just pass the tests files to the clitest command:

clitest [options] [test-file] [test-file] ...

The test runner will exit with a non-zero exit code if any command does not match its expected output.

Test File Structure

Each test file should start with the shebang:

#!/usr/bin/env clitest --v0

The --v0 flag indicates that the test file uses version 0 of the syntax. This ensures backwards compatibility as the syntax evolves in future versions.

Basic Commands

Executing Commands

Commands are prefixed with $:

$ echo "Hello World"
! Hello World

You can split long commands across multiple lines using either backslashes or quotes:

$ echo This is a very long command that \
       spans multiple lines
! This is a very long command that spans multiple lines

$ echo "This is another way to
split a command across lines"
! This is another way to
! split a command across lines 

Special characters may need to be escaped using backslashes. See Quoting for more details.

Comments

Comments start with # and are ignored during test execution:

# This is a comment
$ echo "Hello World"
! Hello World

Basic Output Matching

The simplest way to match output is using the ! pattern, which treats non-grok parts as literal text:

$ echo "Hello World"
! Hello World

Exit Codes

By default, CLI/test expects commands to exit with code 0. You can specify a different expected exit code using %EXIT:

$ exit 1
%EXIT 1

To expect a command to return a failing exit code (ie: non-zero):

$ exit 1
%EXIT fail

Or to accept any exit code (this will also accept a command that times out):

$ exit 1
%EXIT any

Pattern Matching

CLI/test provides two main types of pattern matching: auto-escaped patterns (!) and raw patterns (?). Each has its own use cases and syntax.

Auto-escaped Patterns (!)

Auto-escaped patterns treat non-grok parts as literal text, making them perfect for exact matches:

$ printf "[LOG] Hello, world!\n"
! [LOG] Hello, world!

Raw Patterns (?)

Raw patterns treat everything as a pattern, requiring special characters to be escaped with backslash:

$ printf "[LOG] Hello, world!\n"
? \[LOG\] Hello, world!

You can use ^ and $ anchors in raw patterns for exact line matching:

$ printf "  X  \n"
? ^  X  $

Grok Patterns

CLI/test supports grok patterns for flexible matching:

$ echo "Hello, anything"
? Hello, %{GREEDYDATA}

Common grok patterns:

• %{DATA} - Matches any text
• %{GREEDYDATA} - Matches any text greedily

You can also customize grok patterns by providing a name and value:

$ printf "[LOG] Hello, world!\n"
? \[%{log=(LOG)}\] %{GREEDYDATA}

Multi-line Matching

Auto-escaped Multi-line (!!!)

$ printf "a\nb\nc\n"
!!!
a
b
c
!!!

Raw Multi-line (???)

$ printf "a\nb\nc\n"
???
a
b
c
???

When using multi-line patterns, the indentation of the !!! or ??? lines is removed from all lines between them. This makes it easy to maintain proper indentation in your test files while matching unindented output:

$ printf "abc\n\ndef\n"
  !!!
  abc

  def
  !!!

Literal Multi-line (""")

Literal multi-line blocks are similar to the other raw multi-line blocks, but they treat all text as literal text.

$ printf "We can match things that look grok-like:\n%%{GROKLIKE}\n"
"""
We can match things that look grok-like:
%{GROKLIKE}
"""

Pattern Structures

Any Pattern (*)

The * pattern matches any number of lines lazily, completing when the next structure matches. It can be used at the start, middle, or end of patterns:

# Match any output
$ printf "a\nb\nc\n"
*

# Match start, any middle, end
$ printf "a\nb\nc\nd\ne\n"
! a
! b
*
! d
! e

# Match within repeat
$ printf "start\n1\n2\nend\nstart\n1\n2\nend\n"
repeat {
    ! start
    *
    ! end
}

Pattern Blocks

Pattern blocks allow you to combine multiple patterns in different ways:

Repeat

Match a pattern multiple times:

$ printf "a\nb\nc\n"
repeat {
    choice {
        ! a
        ! b
        ! c
    }
}

Choice

Match any one of the specified patterns:

$ echo "pattern1"
choice {
    ! pattern1
    ! pattern2
    ! pattern3
}

Unordered

Match patterns in any order:

$ printf "b\na\nc\n"
unordered {
    ! a
    ! b
    ! c
}

Sequence

Match patterns in strict order:

$ printf "a\nb\nc\n"
sequence {
    ! a
    ! b
    ! c
}

Optional

Make a pattern optional:

$ echo "optional output"
optional {
    ! optional output
}

Not

Negative lookahead patterns are supported using not. The pattern will fail if the pattern matches when looking ahead. If the pattern does not match, it will succeed but consume no lines.

$ echo "Hello World"
not {
    ! ERROR
}
! Hello World

This can be useful for better targeting of reject lines:

$ echo "ERROR: We expect this one"
reject {
    # We don't want to reject this expected error, but any others should fail
    sequence {
        not {
            ! ERROR: We expect this one
        }
        ! ERROR: %{GREEDYDATA}
    }
}
# But remember that it doesn't get consumed!
! ERROR: We expect this one

Ignore

Ignore blocks are supported at the command and global level. Global ignore blocks are applied to all commands in the test, while command-level ignore blocks are applied to the command only.

Skip certain output:

$ printf "WARNING: Something happened\nHello World\n"
ignore {
    ? WARNING: %{DATA}
}
! Hello World

Reject

Reject blocks are supported at the command and global level. Global reject blocks are applied to all commands in the test, while command-level reject blocks are applied to the command only.

Ensure certain patterns don't appear:

$ echo "Hello World"
reject {
    ! ERROR
}
! Hello World

Conditional Patterns

You can use if blocks in patterns to conditionally match output:

$ echo `uname -s` specific output
if $TARGET_OS == "linux" {
    ! Linux specific output
}
if $TARGET_OS != "linux" {
    ! %{DATA} specific output
}

Note that pattern if blocks and control if blocks have identical syntax, but one contains patterns and the other contains commands.

Pattern Examples

Matching Log Lines

$ printf "[INFO] User logged in\n[ERROR] Connection failed\n"
repeat {
    ! [%{WORD}] %{GREEDYDATA}
}

Matching Numbers

$ echo "Count: 42"
? Count: %{NUMBER}

Matching Dates

$ echo "Date: 20-03-2024"
? Date: %{DATE}

Quoting

CLI/test uses shell-style quoting and variable expansion for commands, command-lines and control structures.

For Command-Lines ($)

Command-lines are the text that is passed to the shell. They are used to construct commands for the shell to execute.

For command-lines specified in the script, the entire line is passed as-is to the shell (/bin/sh by default), and all unescaping is handled by the shell itself.

The shell will use POSIX-compliant quoting and unescaping rules.

For Internal Commands and Control Structures (set, if, for, etc.)

Internal commands and control structures are not passed to the shell, and are processed by CLI/test itself.

Unescaping rules for characters (eg: \n, \xXX) are applied eagerly at parsing time, while variable references are lazily expanded at runtime.

Quoting Reference

Single Quotes (')

Single quotes preserve the literal value of every character within the quotes. No characters inside single quotes have special meaning, including the dollar sign used for variable references.

# Command-line
$ echo 'Hello $USER'
! Hello $USER

# Internal command
set MESSAGE 'Hello $USER';
$ echo $MESSAGE
! Hello $USER

Double Quotes (")

Double quotes preserve the literal value of most characters, but still allow for variable expansion (e.g., $VAR or ${VAR}).

set USER "username";

# Command-line
$ echo "Hello $USER"
! Hello username

# Internal command
set MESSAGE "Hello $USER";
$ echo $MESSAGE
! Hello username

Backslash Escaping (\)

Backslashes can be used to escape the next character, preserving its literal meaning. This works both inside double quotes and unquoted text.

You can use '$` to escape dollar signs so they do not participate in variable expansion in unquoted text or double-quoted strings.

$ echo "Hello \"World\""
! Hello "World"

$ echo Hello\ World
! Hello World

$ echo Hello \$WORLD
! Hello $WORLD

set MESSAGE "Hello \"World\"";
$ echo $MESSAGE
! Hello "World"

set MESSAGE "Hello\nWorld";
$ echo "$MESSAGE"
! Hello
! World

set MESSAGE "Hello \$WORLD";
$ echo $MESSAGE
! Hello $WORLD

Note that internal commands support additional escaped characters:

Multi-line Commands

You can split long command-lines across multiple lines using either backslashes or quotes:

$ echo "This is a very long command that \
spans multiple lines"
! This is a very long command that spans multiple lines

$ echo "This is another way to
split a command across lines"
! This is another way to
! split a command across lines 

Variable References

Basic Reference ($VAR)

Use $VAR to reference variables:

set FOO bar;
$ echo $FOO
! bar

Explicit Reference (${VAR})

Use ${VAR} when the variable name is followed by text:

set FOO bar;
$ echo ${FOO}123
! bar123

Quoting in Control Structures

Quoting is optional, but important in control structures like for loops and conditional blocks:

# All three of these are equivalent!
for OS in "linux" "macos" "windows" {
    $ uname -a | grep $OS
    %EXIT any
    *
}

for OS in linux macos windows {
    $ uname -a | grep $OS
    %EXIT any
    *
}

set LINUX "linux";
set WINDOWS "windows";
set MACOS "macos";

for OS in "$LINUX" "$MACOS" "$WINDOWS" {
    $ uname -a | grep $OS
    %EXIT any
    *
}

Grok patterns

Grok patterns are a way to parse text into structured data.

Syntax

A grok pattern is constructed using one of the formats:

• %{PATTERN_NAME}: a standard named pattern
• %{PATTERN_NAME=(regex)}: a custom pattern defined using a regular expression
• %{PATTERN_NAME:field_name}: a standard named pattern with a named output
• %{PATTERN_NAME:field_name=(regex)}: a custom pattern with a named output

Examples

The most basic pattern is %{DATA}, which matches any text lazily: as few times as possible for the remainder of the line to match. Alternatively, you can use %{GREEDYDATA} to greedily match any text, as many times as possible while allowing the remainder of the line to match.

$ echo "Hello, world!"
! Hello, %{DATA:what}!

Custom patterns are defined using the pattern command, after which the patterns are available for use in the tests.

pattern GREETING Hello|Goodbye;

$ echo "[INFO] Hello, world!"
! [%{LOGLEVEL}] %{GREETING}, %{DATA}!

A custom pattern may also be defined inline:

$ echo "[DEBUG] Hello, world!"
! [%{CUSTOMLEVEL=INFO|DEBUG}] %{GREETING=(Hello|Goodbye)}, %{DATA}!

Custom patterns may be named and reused in a single line:

$ echo "[DEBUG] Hello, world!"
! [%{MY_WORD=(\w+)}] %{MY_WORD}, %{MY_WORD}!

Expectations and Aliases

By default, patterns don't have aliases and may match any value. If any patterns are given aliases (eg: `%{PATTERN:alias}), all values of those aliases must match throughout the entire test output.

You can use the %EXPECT command to expect a specific value for a given alias. If no initial value is provided, the value is taken from the first match.

$ echo "[DEBUG] Hello, world!"
%EXPECT word1 "DEBUG"
%EXPECT word2 "Hello"
%EXPECT word3 "world"
! [%{MY_WORD:word1=(\w+)}] %{MY_WORD:word2}, %{MY_WORD:word3}!

$ printf "line\nline\nline\n"
! %{WORD:word}
! %{WORD:word}
! %{WORD:word}

References

For further reading, see:

• IBM Streams documentation
• Mezmo documentation
• EdgeDelta blog post

Tools

Some potential tools for working with grok patterns:

• Grok debugger
• Regex101

Control Structures

CLI/test provides several control structures to help you write complex test scenarios.

For Loops

The for block allows you to iterate over a list of values:

for OS in "linux" "macos" "windows" {
    $ uname -a | grep $OS
    %EXIT any
    optional {
        ! %{GREEDYDATA}
    }
}

Conditional Blocks

You can use if blocks to conditionally execute commands:

if TARGET_OS == "linux" {
    $ echo Linux specific output
    ! Linux specific output
}

This can also be used to exit the script early:

if TARGET_OS == "windows" {
    exit script;
}

# ... other commands ...

Note that pattern if blocks and control if blocks have identical syntax, but one contains patterns and the other contains commands.

Background processes

Run commands in the background using background { }. When the block ends, the background process is automatically killed. If the test exits early (e.g., due to a failure), background processes are also killed.

Commands running in a background block have no explicit timeout, but you can set an explicit timeout for each command with %TIMEOUT if needed.

using tempdir;

background {
    $ python3 -m http.server 60801 2> server.log
    %EXIT any
}

$ echo "OK" > health

retry {
    $ curl -s http://localhost:60801/health
    ! OK
}

Deferred cleanup

Run commands after the block finishes. Multiple defer blocks are executed in reverse order (last in, first out):

defer {
    $ echo "Second cleanup"
    ! Second cleanup
}

defer {
    $ echo "First cleanup"
    ! First cleanup
}

$ echo "Running!"
! Running!

$ echo "Done!"
! Done!

Retry

Retry commands until they succeed or timeout:

retry {
    $ true
}

retry uses the global timeout for the whole retry block, but you can set a shorter timeout for the command itself with %TIMEOUT:

retry {
    $ true
    %TIMEOUT 100ms
}

Early exit

You can exit a script early using exit script;. This will cause the script to exit with a success status while skipping the remaining commands. This is useful for skipping a test if a prerequisite is not met.

$ echo "will run"
! will run

if PREREQUISITE != "value" {
    exit script;
}

$ echo "won't run!"
! won't run!

Include

You can include another script into the current script using include "path/to/script.cli";.

include "include/included.cli";

# included.cli

# These patterns and variables are available in the outer script
pattern MY_PATTERN [abcd]+;
set VARIABLE "value";

# This is run at the time the script is included
$ echo "run in included script"
! run in included script

The included script is executed in the current script's context, so it can use the same variables and commands. The included script is treated as if it was a block in the outer script.

Variables

Variables in commands and control structures are lazily expanded using shell-style variable references.

set LINUX "linux";
set WINDOWS "windows";

for OS in "$LINUX" "$WINDOWS" {
    $ uname -a | grep $OS
    %EXIT any
    *
}

For more details, see Quoting.

Quoting

Strings in commands and control structures are eagerly unescaped.

set VARIABLE "This is a \"quoted\" string with a hex escape\x21";
$ echo $VARIABLE
! This is a "quoted" string with a hex escape!

For more details, see Quoting.

Environment and Variables

CLI/test provides powerful features for managing environment variables and working directories.

Setting Variables

Using %SET

%SET can be used to capture all or part of the command output into a variable.

Capture the entire command output into a variable:

$ printf "value\n"
%SET MY_VAR
*

You can also capture one or more grok captures into a variable. The value will be contructed from the grok captures and existing environment variables:

$ echo "Hello, world!"
%SET CAPTURED_GREETING "${greeting} ${word}"
! %{WORD:greeting}, %{WORD:word}!

$ echo "$CAPTURED_GREETING"
! Hello world

Using set

Set environment variables directly:

set FOO bar;
set PATH "/usr/local/bin:$PATH";

Variable References

Basic Reference

Use $VAR to reference variables:

set FOO bar;
$ echo $FOO
! bar

Explicit Reference

Use ${VAR} when the variable name is followed by text:

set FOO bar;
$ echo ${FOO}123
! bar123

Working Directory Management

The working directory is managed through a special variable PWD. This can be set directory, or various commands can change it.

Changing Directory

Change the current working directory. The PWD is updated to the new directory for the duration of the test, unless another command changes it:

cd "subdir";

Using Temporary Directories

Create and use a temporary directory. The current working directory is automatically set to the temporary directory, and when the block ends, the temporary directory is automatically deleted.

using tempdir;

Creating New Directories

Create a new directory for testing. The current working directory is automatically set to the directory, and it is deleted when the block ends.

using new dir "subdir";

Using Existing Directories

Use an existing directory. The current working directory is automatically set to the directory, and it is not deleted when the block ends.

using tempdir;
$ mkdir -p subdir
using dir "subdir";

Special Variables

PWD

The PWD variable is special and controls the current working directory:

$ mktemp -d
%SET TEMP_DIR
*

# Set PWD to change working directory
$ echo $TEMP_DIR
%SET PWD
*

Environment Variable Examples

Combining Variables

set A 1;
set B 2;
set C "$A $B";
$ echo $C
! 1 2

Using Variables in Commands

set DIR "subdir";
using new dir "$DIR";
$ echo $PWD
! %{PATH}/subdir

Conditional Environment Setup

if $TARGET_OS == "linux" {
    set PATH "/usr/local/bin:$PATH";
}

Advanced Features

This chapter covers advanced features and best practices for using CLI/test effectively.

Complex Pattern Matching

Nested Patterns

Combine different pattern types for complex matching:

$ printf "a\nb\nc\nd\n"
sequence {
    ! a
    repeat {
        choice {
            ! b
            ! c
        }
    }
    ! d
}

Conditional Pattern Matching

Use conditions with patterns:

if $TARGET_OS == "linux" {
    $ echo Linux specific output
    ? Linux specific %{GREEDYDATA}
}

Process Management

Background Processes

Run and manage background processes, using retry to wait for the process to start:

using tempdir;

background {
    $ python3 -m http.server 60800 2> server.log
    %EXIT any
}

$ echo "OK" > health

retry {
    $ curl -s http://localhost:60800/health
    ! OK
}

# Test the server
$ curl -s http://localhost:60800/health
! OK

# Background processes are automatically killed

Process Cleanup

Ensure proper cleanup with defer:

defer {
    $ killall background-server
    %EXIT any
    *
}

background {
    $ python3 -m http.server 60800
    %EXIT any
}

$ echo 1
! 1

Timeouts

Set a timeout for a command:

$ sleep 60
%EXIT timeout
%TIMEOUT 100ms

Error Handling

Expected Failures

Test error conditions with %EXIT. %EXIT any will allow any exit code or signal, while %EXIT n will only allow exit code n:

$ false
%EXIT 1

Expecting Failures

If you want to verify that a pattern does not match, use %EXPECT_FAILURE. This can be useful in certain cases, but you should prefer a reject { } block if you just want to test that a certain pattern never matches.

$ echo "Hello World"
%EXPECT_FAILURE
! Wrong Output

Best Practices

Test Organization

1. Group related tests together and use descriptive comments.
2. Keep tests focused and atomic
3. Use variables for reusable values
4. Use ignore for noisy output, prefer a global ignore block over many, repeated ignore { } blocks.
5. Add defer blocks for cleanup immediately after allocation or creation of resources.

Example of Complex Test

# Global ignore/rejects
ignore {
    ! No configuration file found, creating default at %{GREEDYDATA}
}

reject {
    ! Critical error, corrupted configuration file.
}

# Test server startup and basic functionality
using tempdir;

# Start server in background
background {
    $ echo "{\"status\": \"success\"}" > api.json
    $ echo "OK" > health.json
    $ python3 -m http.server 60900 2> server.log
    %EXIT any
}

defer {
    $ rm server.log
}

# Wait for server to start
retry {
    $ curl -s http://localhost:60900/health.json
    ! OK
}

# Test main functionality
$ curl -s http://localhost:60900/api.json
! {"status": "success"}

# Verify logs
$ cat server.log
repeat {
    choice {
        ? %{IPORHOST} %{GREEDYDATA} code %{NUMBER}, %{GREEDYDATA}
        ? %{IPORHOST} %{GREEDYDATA} "GET /%{DATA} %{DATA}" %{NUMBER} -
    }
}

# Cleanup is automatic!

Reference

This reference provides a overview of all CLI/test syntax elements organized by category.

Command Execution

Variables and Quoting

clitest uses shell-style variable references and quoting to delimit strings in commands and control structures.

Control Structures

Patterns

Common Grok Patterns

This is a subset of the grok patterns supported by clitest. See the full list of supported patterns at https://docs.rs/grok/latest/grok/patterns/index.html, including the full base patterns in the grok module: https://docs.rs/grok/latest/grok/patterns/grok/index.html.