Technical documentation / Reference

CPU Hotspot Profiling Workflow

Maintainer workflow for issue #48 / PRD-0056. Default path is credential-free and offline: scripts/profile_cpu.py runs ignored Rust harness tests with a temporary synthetic MC_HOME, no auth.json, credential-looking environment variables removed from child processes, local SSE fixtures, local session/cache/skill data, and no live model-catalog refresh.

Requirements

• All platforms: Rust/Cargo, python3.
• macOS: built-in sample and spindump; optional Instruments.
• Linux: perf; optional FlameGraph and hyperfine.

Default credential-free run

python3 scripts/profile_cpu.py --repo . --all --min-seconds 3

Automatic wrapper artifacts:

target/profiling/issue-48/profile-results.json
target/profiling/issue-48/profile-summary.md
target/profiling/issue-48/<scenario>.log
target/profiling/issue-48/tui-render-pipeline.json

The wrapper does not collect .sample.txt files or idle PTY samples. Collect those separately using the manual macOS sampling workflow.

Run one scenario:

python3 scripts/profile_cpu.py --repo . --scenario provider_sse_parser --min-seconds 10
cargo test --release profile_cpu_sse_parser -- --ignored --nocapture

Self-test:

python3 scripts/profile_cpu.py --self-test

Mission Control render pipeline

Default render profiling is credential-free and uses Ratatui TestBackend; it does not enter raw mode, alternate screen, provider transport, network, or real stdout ANSI output. Historical baseline results live in 1.

Run full harness through Python wrapper:

python3 scripts/profile_cpu.py --repo . --scenario tui_render_pipeline --min-seconds 5

Run all render-pipeline scenarios once through the exact aggregate test (a partial filter also selects six individual tests):

cargo test --release --lib test_support::profile_harness::profile_tui_render_pipeline -- --exact --ignored --nocapture

Run one scenario directly:

cargo test --release profile_tui_render_pipeline_idle -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_streaming -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_heavy_transcript -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_scroll -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_resize -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_modal -- --ignored --nocapture

Direct Cargo runs print results to stdout; they do not write a JSON artifact. The Python wrapper writes:

target/profiling/issue-48/tui-render-pipeline.json

Backend modes are selected by PROFILE_TUI_RENDER_BACKEND:

PROFILE_TUI_RENDER_BACKEND=test cargo test --release --lib test_support::profile_harness::profile_tui_render_pipeline -- --exact --ignored --nocapture
PROFILE_TUI_RENDER_BACKEND=crossterm-sink cargo test --release --lib test_support::profile_harness::profile_tui_render_pipeline -- --exact --ignored --nocapture
PROFILE_TUI_RENDER_BACKEND=crossterm-real cargo test --release --lib test_support::profile_harness::profile_tui_render_pipeline -- --exact --ignored --nocapture

Metrics:

Interpretation rules:

1. Compare same backend, terminal area, Rust profile, scenario mix, and machine.
2. If render_draw_ms dominates p50 frame time, optimize Mission Control render/state projection first.
3. If diff_ms stays sub-millisecond while diff_cells is high, Ratatui diff is not blocking frame throughput.
4. If stdout_bytes, write_ms, or flush_ms rise in crossterm modes, terminal I/O may cap real-world FPS even when TestBackend is fast.
5. Treat TestBackend results as process-side render ceiling only. Real-world ceiling depends on terminal emulator write/flush/paint behavior.

Large-response and text-modal comparisons

These ignored, credential-free workflows compare current render paths with local reference algorithms, not historical binaries or terminal-emulator paint:

cargo test --release --locked --lib streaming_viewport_large_response_workload -- --ignored --nocapture
cargo test --release --locked --lib text_modal_large_document_workload -- --ignored --nocapture

Streaming uses independent states with identical synthetic events and alternates measurement order. It reports initial indexing separately, then medians across 30 appends. rebuild disables only completed-prefix reuse; full_reference materializes and measures every line. Text-modal results include Terminal::draw with TestBackend, across 21 top/middle/end draws.

Representative local results on macOS with Rust 1.98.1, release profile, September 2026:

The multiline Unicode cold index took 45.52 ms versus 38.24 ms for full formatting. Completed-prefix reuse—not faster cold Unicode wrapping—produces the steady append improvement. Reuse requires exact equality of the fully sanitized completed prefix and compatible row/layout identity. Unbroken responses retain full formatting because sparse rebuilding measured slower. Whole-source sanitization, cold indexing, final Markdown, explicit copying, and modal height/prefix scans remain full-work paths. Allocations and physical-terminal latency were not measured.

Scenarios

Issue #381 skill discovery benchmark

The dedicated release benchmark uses 55 selected skills plus three deterministic diagnostics across small, empty-body, representative-body, near-1-MiB, disabled, malformed, colliding, and oversized fixtures. The fixture contains 5,439,395 aggregate body bytes. That count describes generated fixture content; bytes read and allocations were not instrumented and are not claimed.

Both runs used PROFILE_CPU_ITERATIONS=25 cargo test --release profile_cpu_discover_skills -- --ignored --nocapture on a MacBook Pro Mac17,8 with Apple M5 Pro, 64 GB RAM, macOS 27.0 arm64, and rustc 1.98.0, release profile.

The dedicated result improved from 13.0 ms to 3.0 ms per iteration (about 77%). This is a same-machine synthetic comparison, not evidence of a user-visible startup regression or a hard budget. As secondary context only, the aggregate startup_discovery harness completed 25 iterations in 42 ms at the optimized commit (target/profiling/issue-381/startup-after.log); it also measures settings, instructions, sessions, and model-cache work, and the committed Issue #48 profile identifies session listing as dominant and skill discovery as secondary.

macOS sampling

Harness sample:

PROFILE_CPU_MIN_SECONDS=20 PROFILE_CPU_ITERATIONS=1 \
  cargo test --release profile_cpu_rendering_heavy_transcript -- --ignored --nocapture &
cargo_pid=$!
sleep 1
pgrep -P "$cargo_pid" -fl magi_code
test_pid=<pid-from-pgrep>
sample "$test_pid" 10 -mayDie -file target/profiling/issue-48/rendering-heavy-transcript.sample.txt
wait "$cargo_pid"

Idle Mission Control TUI real-terminal command:

MC_HOME="$(mktemp -d)" cargo run --bin magi-code -- --no-session
# In another terminal:
pgrep -fl "magi-code.*--no-session"
sample <pid> 30 -file target/profiling/issue-48/idle-tui.sample.txt
spindump <pid> 10 -file target/profiling/issue-48/idle-tui.spindump.txt

Idle rules: do not submit a prompt, do not run /login, do not configure provider credentials, do not refresh model catalogs, exit with /quit after sampling. A non-interactive /usr/bin/script PTY baseline is acceptable for issue #48 if it launches magi-code --no-session, keeps stdin open, uses isolated MC_HOME, captures sample, and is labeled as PTY rather than physical terminal evidence. Recorded baselines below retain the launch flags used at the time.

Linux perf parity

The Linux commands provide an equivalent workflow; they have not been run unless a Linux host is named in the baseline notes:

PROFILE_CPU_MIN_SECONDS=20 PROFILE_CPU_ITERATIONS=1 \
  perf record -F 99 -g -- cargo test --release profile_cpu_tui_streaming_simulation -- --ignored --nocapture
perf report --stdio > target/profiling/issue-48/tui-streaming.perf-report.txt
perf script > target/profiling/issue-48/tui-streaming.perf-script.txt

Optional FlameGraph:

perf script | stackcollapse-perf.pl > target/profiling/issue-48/tui-streaming.folded
flamegraph.pl target/profiling/issue-48/tui-streaming.folded > target/profiling/issue-48/tui-streaming.svg

Issue #48 historical baseline

Machine baseline from target/profiling/issue-48/profile-results.json generated on 2026-05-31T07:32:23Z. macOS stack samples were collected with /usr/bin/sample for all synthetic harness scenarios plus idle TUI under a non-interactive /usr/bin/script PTY. The Linux profiler was not run on this Darwin host.

Hotspot report

Historical findings from the Issue #48 baseline above. Source paths, symbols, and priorities reflect that run, not a current bottleneck assessment.

Historical CPU-reduction priority order

1. P1: reduce repeated transcript markdown/render projection in heavy transcript and streaming paths. Expected user impact: smoother long-answer TUI and lower redraw CPU.
2. P1: reduce activity visible-node clone/rebuild churn in streaming paths. Expected user impact: lower CPU during tool-heavy runs.
3. P2: inspect startup session listing cost when many JSONL files exist; keep session semantics unchanged.
4. P2: keep idle loop under observation with optional physical-terminal wakeup sample; PTY evidence shows low CPU and no P0 idle blocker.
5. P3: process cleanup polling changes are riskier because they touch safety semantics; optimize only with process-tree regression coverage.

Before/after comparison method

1. Use the same machine, branch base, and Rust profile, and the same terminal size for TUI checks.
2. Run:

python3 scripts/profile_cpu.py --repo . --all --min-seconds 10
cargo test --release profile_cpu -- --ignored --nocapture

1. Profile the changed area with a platform profiler:

sample <pid> 10 -file target/profiling/issue-48/<scenario>.sample.txt
# or Linux:
perf record -F 99 -g -- cargo test --release <scenario> -- --ignored --nocapture

1. Compare baseline table fields: scenario, date/time, machine/OS/arch, Rust version/profile, command, duration/iterations, primary metric, result, artifact path, notes.
2. Compare hotspot table fields: scenario, source file, function/symbol, metric type, metric value, profiler artifact, impact, confidence, follow-up priority.
3. Report delta as (after - before) / before, plus whether profiler top symbols moved away from target function.

Optional live-provider profiling

Live provider profiling is not part of the default workflow. Skip it unless credentials and network are intentionally available. Do not store provider captures, bearer-style headers, provider secret strings, account identifiers, authorization codes, or secret payloads in fixtures or artifacts.

Edit this page on GitHub ↗