Fuzzing

Fuzzing is the process of providing random data to programs to identify unexpected behavior, such as crashes and panics.

Fuzz tests can also be written as property tests that instead of seeking to identify panics and crashes, assert on some property remaining true. Fuzzing as demonstrated here and elsewhere in these docs will use principles from both property testing and fuzzing, but will only use the term fuzzing to refer to both.

The following steps can be used in any Stellar contract workspace. If experimenting, try them in the increment example. The contract has an increment function that increases a counter value by one on every invocation.

tip

See the Rust Fuzz Book for a general tutorial on using both cargo-fuzz and cargo-afl.

How to Write Fuzz Tests with cargo-fuzz

1. Install the nightly Rust toolchain. Nightly Rust is required to run cargo-fuzz.

   rustup install nightly

2. Install cargo-fuzz.

   cargo install --locked cargo-fuzz

3. Initialize a fuzz project by running the following command inside your contract directory.

   cargo fuzz init

4. Open the contract's Cargo.toml file. Add lib as a crate-type.

    [lib]
   -crate-type = ["cdylib"]
   +crate-type = ["lib", "cdylib"]

5. Open the generated fuzz/Cargo.toml file. Add the soroban-sdk dependency.

    [dependencies]
    libfuzzer-sys = "0.4"
   +soroban-sdk = { version = "*", features = ["testutils"] }

6. Open the generated fuzz/src/fuzz_target_1.rs file. It will look like the below.

   #![no_main]
   use libfuzzer_sys::fuzz_target;
   
   fuzz_target!(|data: &[u8]| {
       // fuzzed code goes here
   });

7. Fill out the fuzz_target! call with test setup and assertions. For example, for the increment example:

   #![no_main]
   use libfuzzer_sys::fuzz_target;
   use soroban_increment_with_fuzz_contract::{IncrementContract, IncrementContractClient};
   use soroban_sdk::{
       testutils::arbitrary::{arbitrary, Arbitrary},
       Env,
   };
   
   #[derive(Debug, Arbitrary)]
   pub struct Input {
       pub by: u64,
   }
   
   fuzz_target!(|input: Input| {
       let env = Env::default();
       let id = env.register(IncrementContract, ());
       let client = IncrementContractClient::new(&env, &id);
   
       let mut last: Option<u32> = None;
       for _ in input.by.. {
           match client.try_increment() {
               Ok(Ok(current)) => assert!(Some(current) > last),
               Err(Ok(_)) => {} // Expected error
               Ok(Err(_)) => panic!("success with wrong type returned"),
               Err(Err(_)) => panic!("unrecognised error"),
           }
       }
   });

8. Execute the fuzz target.

   cargo +nightly fuzz run --sanitizer=thread fuzz_target_1

   info

   If you're developing on MacOS you need to add the --sanitizer=thread flag in order to work around a known issue.

This test uses the same patterns used in unit tests and integration tests:

1. Create an environment, the Env.
2. Register the contract to be tested.
3. Invoke functions using a client.
4. Assert expectations.

tip

For a full detailed example, see the fuzzing example.

How to Write Fuzz Tests with cargo-afl

cargo-afl drives AFL++ and, unlike cargo-fuzz, runs on stable Rust.

1. Install cargo-afl. Installing it builds AFL++ from source, so a C compiler needs to be available.

   cargo install cargo-afl --locked

2. Configure the machine for fuzzing.

   cargo afl system-config

3. Open the contract's Cargo.toml file. Add lib as a crate-type. The fuzz target imports the contract as a Rust library.

    [lib]
   -crate-type = ["cdylib"]
   +crate-type = ["lib", "cdylib"]

4. Create a fuzz target crate, for example with cargo new --bin fuzz inside your contract's directory. Unlike a cargo-fuzz target, an AFL++ target depends on the arbitrary crate directly, because the fuzz! macro expands to code that refers to it by an absolute path, which only resolves if arbitrary is a direct dependency. Put the following in the new crate's Cargo.toml. It replaces the empty [dependencies] table that cargo new generated.

   [dependencies]
   afl = "0.18"
   arbitrary = { version = "~1.3.0", features = ["derive"] }
   soroban-sdk = { version = "*", features = ["testutils"] }
   # The contract to fuzz. Use the package name from its Cargo.toml.
   soroban-increment-contract = { path = ".." }
   
   [[bin]]
   name = "fuzz_target_1"
   path = "src/fuzz_target_1.rs"
   
   # Prevent this from interfering with the contract's workspace, if it has one.
   [workspace]
   members = ["."]

5. Write the fuzz target at src/fuzz_target_1.rs, and delete the default src/main.rs cargo new created. An AFL++ target is a regular binary crate with a main function that calls the afl::fuzz! macro. For example, for the increment example:

   use afl::fuzz;
   use arbitrary::Arbitrary;
   use soroban_increment_contract::{IncrementContract, IncrementContractClient};
   use soroban_sdk::Env;
   
   #[derive(Debug, Arbitrary)]
   pub struct Input {
       pub by: u8,
   }
   
   fn main() {
       fuzz!(|input: Input| {
           // Create the `Env` inside the closure, not outside: AFL++ reuses the
           // process for many inputs, and state created outside the closure
           // would leak from one input into the next.
           let env = Env::default();
           let id = env.register(IncrementContract, ());
           let client = IncrementContractClient::new(&env, &id);
   
           let mut last: Option<u32> = None;
           for _ in 0..input.by {
               match client.try_increment() {
                   Ok(Ok(current)) => {
                       assert!(Some(current) > last);
                       last = Some(current);
                   }
                   Err(Ok(_)) => {} // Expected error
                   Ok(Err(_)) => panic!("success with wrong type returned"),
                   Err(Err(_)) => panic!("unrecognised error"),
               }
           }
       });
   }

6. Build the target. The remaining steps also run from inside the fuzz crate.

   cd fuzz
   cargo afl build

7. Fuzz the target, pointing at an input directory containing at least one seed input and an output directory to write results to.

   mkdir in out
   echo -n '00000000' > in/seed
   cargo afl fuzz -i in -o out target/debug/fuzz_target_1

Crashing inputs are written to out/default/crashes/. The target reads an input on stdin when it isn't being driven by AFL++, so a crash can be replayed by feeding the file back in:

RUST_BACKTRACE=1 ./target/debug/fuzz_target_1 < out/default/crashes/id:000000*

How to Get Code Coverage of cargo-fuzz Tests

Getting code coverage data for fuzz tests requires some different tooling than when doing the same for regular Rust tests.

1. Run the cargo-fuzz target until it has produced a corpus.

   cargo +nightly fuzz run --sanitizer thread fuzz_target_1

2. Install the llvm-tools for the nightly compiler.

   rustup component add --toolchain nightly llvm-tools-preview

3. Run the fuzz coverage command that'll execute the corpus and write coverage data to the coverage directory in the profdata format.

   cargo +nightly fuzz coverage --sanitizer thread fuzz_target_1

4. Run the llvm-cov command to convert the profdata file to an lcov file.

   $(find $(rustc --print sysroot) -name llvm-cov) export \
       -instr-profile=fuzz/coverage/fuzz_target_1/coverage.profdata \
       -object target/$(rustc -vV | sed -n 's|host: ||p')/coverage/$(rustc -vV | sed -n 's|host: ||p')/release/fuzz_target_1 \
       --ignore-filename-regex "rustc" \
       -format=lcov \
       > lcov.info

   Load the lcov.info file into your IDE using its coverage feature. In VSCode this can be done by installing the Coverage Gutters extension and executing the Coverage Gutters: Watch command.

tip

To measure code coverage of regular Rust tests, see Code Coverage.