differential-fuzzer

Differential Fuzzer

Always load Debugging skill for reference

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

testing/differential-oracle/fuzzer/

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

• TIME_LIMIT_MINUTES - Total runtime (default: 1440 = 24h)
• PER_RUN_TIMEOUT_SECONDS - Per-run timeout (default: 1200 = 20min)
• NUM_STATEMENTS - Statements per run (default: 1000)
• LOG_TO_STDOUT - Print fuzzer output (default: false)
• GITHUB_TOKEN - For auto-filing issues
• SLACK_WEBHOOK_URL - For notifications

Output Files

All output goes to simulator-output/ directory:

File	Description
test.sql	All executed SQL statements. Failed statements prefixed with -- FAILED:, errors with -- ERROR:
schema.json	Database schema at end of run (or at failure)
test.db	Turso database file (only with --keep-files)
test-sqlite.db	SQLite database file (only with --keep-files)

Reproducing Errors

Always follow these steps

1. Find the seed and profile in the error output:

   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ..., weight_profile: Writes }
   

2. Re-run with that seed and profile (a seed only replays under the same profile):

   cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files
   

3. Read the minimized reproduction first. On an oracle failure the fuzzer
   writes these files to simulator-output/:

   • minimized.sql - a shrunken state script plus the shrunken failing
     statement, produced automatically. Start here.
   • turso-state.sql / sqlite-state.sql - each engine's full state as a
     replayable script, when you need more than the minimized version kept.
   • test.sql - every executed statement (the failing one is marked
     -- FAILED:). The minimizer falls back to replaying this history when
     the failure depends on how the state was built, not just its contents.
   • schema.json - table structure at failure time.

4. Probe the reproduction with differential_probe. It runs a
   statement-per-line script on Turso and SQLite side by side, prints both
   outcomes for every statement, marks divergences, and compares the final
   table contents. Exit code 1 means something diverged.

   cargo run -q -p differential-fuzzer --bin differential_probe -- \
       simulator-output/minimized.sql
   

   Use it instead of piping SQL into the two shells: the tursodb shell cannot
   ATTACH ':memory:' AS aux, so fuzzer reproductions with an aux schema
   only run correctly through the probe. Reading from stdin also works:
   echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differential_probe.

5. Bisect by editing the script. Copy minimized.sql, simplify one thing
   at a time (replace an expression with a constant, drop a column, drop a
   state line), and re-run the probe after each edit. The divergence marker
   tells you immediately whether the edit kept the bug. This loop usually
   ends at a one-line kernel you can hand to EXPLAIN on both engines.

6. Create a regression test in .sqltest (preferred) or .rs from the
   kernel. Always load the Debugging skill for reference.

Understanding Failures

Oracle Failure Types

1. Row set mismatch - Turso returned different rows than SQLite
2. Turso errored but SQLite succeeded - Turso rejected valid SQL
3. SQLite errored but Turso succeeded - Turso accepted invalid SQL
4. Schema mismatch - Tables/columns differ after DDL

Warning (non-fatal)

• Unordered LIMIT mismatch - LIMIT without ORDER BY may return different valid rows

Key Source Files

File	Purpose
main.rs	CLI parsing, entry point
runner.rs	Main simulation loop, executes statements on both DBs
oracle.rs	Compares Turso vs SQLite results
schema.rs	Introspects schema from both databases
memory/	In-memory IO for deterministic simulation

Tracing

Set RUST_LOG for more detailed output:

RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345