Version: 1.0
Date: 2025-11-21
Status: Specification for Implementation
Related: MAIN_DESIGN.md, wfst_programming_language_extensions.md
This document specifies a comprehensive benchmark suite for evaluating the 5-layer grammar correction pipeline. The benchmarks measure:
Target: 95% accuracy, <1s latency (balanced mode), <10MB memory overhead per correction.
Goal: Verify that corrections are semantically valid.
Benchmarks:
bench_layer1_completeness: All strings within edit distance are foundbench_layer1_soundness: All found strings are within edit distancebench_layer2_parse_validity: All corrected programs parse without errorsbench_layer3_type_validity: All corrected programs type-check successfullybench_layer4_semantic_validity: No undefined behavior or API misusebench_layer5_concurrency_safety: No deadlocks or race conditionsGoal: Measure latency, throughput, and resource usage.
Benchmarks:
bench_latency_per_layer: Time spent in each layerbench_throughput: Programs corrected per secondbench_memory_usage: Peak memory consumptionbench_cpu_utilization: Core usage efficiencybench_cache_hit_rate: LRU cache effectivenessGoal: Understand behavior as inputs grow.
Benchmarks:
bench_scaling_input_length: Latency vs. program size (LOC)bench_scaling_error_density: Latency vs. number of errorsbench_scaling_beam_width: Accuracy vs. beam widthbench_scaling_max_distance: Coverage vs. edit distance limitGoal: Assess correction quality beyond correctness.
Benchmarks:
bench_edit_distance_optimality: Is the correction minimal?bench_semantic_preservation: Does correction preserve original intent?bench_idiomaticity: Is the correction idiomatic Rholang?bench_ranking_quality: Are better corrections ranked higher?Generation: Programmatically inject errors into valid Rholang programs.
Error Types:
Typos (Layer 1):
let → ltefunction → functonif → iffreceive → recieveSyntax Errors (Layer 2):
let x = 42 let y = 10{{{let x = @@@Type Errors (Layer 3):
let x: String = 42let x =vec![1, "two"]Semantic Errors (Layer 4):
print(x); let x = 42;*ptr where ptr = nullopen(file); return;Concurrency Errors (Layer 5):
Size: 10,000 programs
LOC Distribution:
Source: Open-source Rholang repositories (RChain codebase, community contracts).
Collection:
Size: 1,000 real bug-fix pairs
Error Distribution (from preliminary analysis):
Purpose: Stress-test edge cases.
Examples:
Ambiguous corrections:
let x = 1;
let y = "one";
let z = ?; // Could be 1 or "one" depending on context
Multiple errors:
lte x: Strng = 42 + "hello" // 4 errors: lte, Strng, type mismatch, invalid op
Deeply nested structures:
for(a <- c1) { for(b <- c2) { for(c <- c3) { ... }}} // 10 levels deep
Size: 500 handcrafted adversarial examples
Precision:
P = TP / (TP + FP)
TP = Corrections that compile and match ground truth
FP = Corrections that compile but differ from ground truth
Recall:
R = TP / (TP + FN)
FN = Ground truth corrections not found by system
F1 Score:
F1 = 2 * (P * R) / (P + R)
Exact Match:
EM = # of exact string matches / # of test cases
Compile Rate:
CR = # of corrections that compile / # of attempts
Latency (milliseconds):
L = time(correction_end) - time(correction_start)
Throughput (corrections/second):
T = # of corrections / total_time
Memory (MB):
M = peak_memory_usage - baseline_memory
CPU Utilization (%):
CPU = (cpu_time / wall_time) * 100
Edit Distance Ratio:
EDR = edit_distance(original, correction) / edit_distance(original, ground_truth)
Ideal: EDR ≤ 1.0 (correction is as good or better than ground truth)
Semantic Similarity (using embeddings):
SemSim = cosine_similarity(embed(original), embed(correction))
Ideal: SemSim > 0.9
Ranking Accuracy (MRR - Mean Reciprocal Rank):
MRR = (1/N) * Σ(1 / rank_i)
Where rank_i = rank of ground truth in top-K results
Benchmark: bench_layer1
Test Cases:
let → ltereceive → recievefunction → functiom (m near n)initialize → initalze (2 errors)Metrics:
d are generateddExpected:
d \le 3$\mathcal{O}(n \times d^{2})$ (linear in n for fixed d)Implementation (Rust + Criterion):
use criterion::{black_box, criterion_group, criterion_main, Criterion};
fn bench_layer1_completeness(c: &mut Criterion) {
let test_cases = vec![
("let", "lte", 1),
("function", "functiom", 1),
("initialize", "initalze", 2),
];
c.bench_function("layer1_completeness", |b| {
b.iter(|| {
for (correct, input, max_dist) in &test_cases {
let lattice = build_error_lattice(black_box(input), *max_dist);
assert!(lattice_contains(black_box(&lattice), correct));
}
})
});
}
Benchmark: bench_layer2
Test Cases:
Metrics:
Expected:
Implementation:
fn bench_layer2_parse_time(c: &mut Criterion) {
let lattice = build_test_lattice(); // Pre-generated lattice with 100 candidates
c.bench_function("layer2_parse", |b| {
b.iter(|| {
let results = execute_layer2(black_box(&lattice));
assert!(results.layer_corrections.len() > 0);
})
});
}
Benchmark: bench_layer3
Test Cases:
i32 vs StringVec<T> with T=i32where T: CloneMetrics:
Expected:
Benchmark: bench_layer4
Test Cases:
Metrics:
Expected:
Benchmark: bench_layer5
Test Cases:
Metrics:
Expected:
bench_e2e_fast_modeConfiguration: Layers 1-2, beam width = 5
Test Set: 1,000 programs with single typos/syntax errors
Metrics:
Command:
cargo bench --bench e2e_fast -- --save-baseline fast_mode
bench_e2e_balanced_modeConfiguration: Layers 1-3, beam width = 20
Test Set: 1,000 programs with typos + type errors
Metrics:
bench_e2e_accurate_modeConfiguration: Layers 1-5, beam width = 50
Test Set: 1,000 programs with mixed errors
Metrics:
Benchmark: bench_latency_scaling
Test: Fix input length at $n \in$ {10, 50, 100, 500, 1000} LOC
Measure: Median latency for each n
Expected Complexity:
\mathcal{O}(n)$\mathcal{O}(n)$ (incremental parsing)\mathcal{O}(n)$\mathcal{O}(n^{2})$ (dataflow analysis)\mathcal{O}(n^{2})$ (graph analysis)Plot: Latency (ms) vs. LOC (log-log scale)
Benchmark: bench_throughput_beam
Test: Fix beam width $k \in$ {1, 5, 10, 20, 50, 100}
Measure: Corrections/sec for each k
Expected:
Plot: Throughput (corrections/s) vs. Beam Width
Benchmark: bench_memory_usage
Tool: Valgrind (Massif) or custom allocator tracking
Measure: Peak heap usage during correction
Expected:
Breakdown:
Benchmark: bench_cache_hit_rate
Test: Run 10,000 corrections with repeated inputs (Zipf distribution)
Measure: LRU cache hit rate
Expected:
Speedup:
Benchmark: bench_vs_baseline
Baseline: Standard compiler error messages (no auto-correction)
Measure:
Expected:
Benchmark: bench_vs_spell_check
Comparison: Standard spell checker (Aspell, Hunspell) adapted to code
Test Set: 1,000 programs with typos only (Layer 1 errors)
Metrics: | Metric | Spell Checker | Our Layer 1 | Improvement | |--------|---------------|-------------|-------------| | Accuracy | 65% | 90% | +25% | | Latency | 5ms | 12ms | -7ms | | False Positives | 20% | 5% | -15% |
Reason for Improvement: Context-aware (knows keywords, identifiers)
Benchmark: bench_vs_llm
Setup: GPT-4 with prompt "Fix this Rholang code:"
Test Set: 500 programs with mixed errors
Metrics: | Metric | GPT-4 | Our Pipeline | Notes | |--------|-------|--------------|-------| | Accuracy | 88% | 95% | GPT-4 sometimes "over-corrects" | | Latency | 2-5s | 1s | Our pipeline is faster | | Cost | $0.05/correction | $0 | Our tool is offline | | Explainability | Low | High | We show edit trace |
Use Criterion.rs for statistical rigor.
Features:
Example:
use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId};
fn bench_layers(c: &mut Criterion) {
let mut group = c.benchmark_group("layers");
for layer_id in 1..=5 {
group.bench_with_input(
BenchmarkId::from_parameter(layer_id),
&layer_id,
|b, &id| {
b.iter(|| execute_layer(id, &test_input));
},
);
}
group.finish();
}
criterion_group!(benches, bench_layers);
criterion_main!(benches);
Directory Structure:
benches/
├── data/
│ ├── synthetic/ # 10,000 generated programs
│ ├── real_world/ # 1,000 real bug-fix pairs
│ └── adversarial/ # 500 edge cases
├── bench_layer1.rs
├── bench_layer2.rs
├── ...
└── bench_e2e.rs
Loading:
lazy_static! {
static ref TEST_DATA: Vec<TestCase> = {
load_test_cases("benches/data/synthetic/*.rho")
};
}
Run benchmarks on consistent hardware:
Required:
Pinning (to reduce variance):
taskset -c 0 cargo bench --bench layer1
Generate HTML Report:
cargo bench
open target/criterion/report/index.html
Export to CSV (for further analysis):
cargo bench -- --save-baseline main
cd target/criterion
find . -name estimates.json | xargs cat > ../all_benchmarks.json
Regression Detection:
cargo bench -- --baseline main
# If performance regressed, Criterion will fail the build
| Benchmark | Fast Mode | Balanced Mode | Accurate Mode |
|---|---|---|---|
| Accuracy | 75% | 88% | 95% |
| Latency (p50) | 15ms | 85ms | 1s |
| Latency (p99) | 30ms | 150ms | 2s |
| Throughput | 67/s | 12/s | 0.9/s |
| Memory | 5MB | 10MB | 20MB |
| **CPU (%) | 150% | 200% | 350% |
| Layer | Time (ms) | % of Total |
|---|---|---|
| Layer 1 | 12 | 14% |
| Layer 2 | 45 | 53% |
| Layer 3 | 18 | 21% |
| Layer 4 | 8 | 9% |
| Layer 5 | 2 | 3% |
| Total | 85 | 100% |
Bottleneck: Layer 2 (parsing) due to lattice expansion.
Latency vs. Input Length (Balanced Mode):
n=10: 20ms
n=50: 85ms
n=100: 180ms
n=500: 950ms
n=1000: 2100ms
Fit: ~$\mathcal{O}(n^1.2)$ (subquadratic, better than naive $\mathcal{O}(n^{2})$)
GitHub Actions (example):
name: Benchmarks
on:
push:
branches: [main]
pull_request:
jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
- name: Run benchmarks
run: |
cargo bench --all --bench layer1 -- --save-baseline pr-${{ github.event.number }}
- name: Compare with main
run: |
cargo bench --all --bench layer1 -- --baseline main
- name: Upload results
uses: actions/upload-artifact@v3
with:
name: benchmark-results
path: target/criterion/
Threshold: Fail PR if latency increases >10%
Tool: critcmp (Criterion comparison tool)
cargo install critcmp
critcmp main pr-123
Output:
group main pr-123 diff
layer1 12.3ms 13.8ms +12.2% ⚠ REGRESSION
layer2 45.1ms 44.2ms -2.0% ✓
Database: Store benchmark results in PostgreSQL/SQLite
Schema:
CREATE TABLE benchmarks (
id SERIAL PRIMARY KEY,
commit_hash VARCHAR(40),
benchmark_name VARCHAR(255),
median_time_ms FLOAT,
stddev_ms FLOAT,
timestamp TIMESTAMP DEFAULT NOW()
);
Dashboard: Grafana visualization of latency trends over time.
This benchmark specification provides a comprehensive framework for evaluating the grammar correction pipeline across:
Implementation Priority:
Next Steps: Implement bench_layer1.rs using Criterion, validate against synthetic dataset, then proceed to Layers 2-5.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |