7e2eb708a7
The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a persistent PVC, so every smoke probe builds incrementally on top of artefacts left by prior probes. File deletions (e.g., regime_conditional.rs inff00af68a) can leave dangling rmeta/object references that perturb downstream codegen between bisect runs — making the bisect result contingent on which order probes were submitted in, not on the source. Adds an optional `clean-cache` parameter (default `false`) which runs `cargo clean -p ml -p ml-dqn --release` before the compile step. Other crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean. Use case: re-runninga52d99613+ff00af68awith `--clean-cache` to verify the bisect under controlled build conditions. If the broken/clean status flips with clean cache, the regression is build-state-dependent rather than source-line; if it reproduces, the source regression is real and bisect is definitive. scripts/argo-smoke.sh exposes `--clean-cache` flag passing through to the workflow parameter.
89 lines
3.2 KiB
Bash
Executable File
89 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Run a smoke test on L40S via Argo Workflows — no profiler, no sanitizer.
|
|
#
|
|
# Use this when you want to validate that a smoke test passes against the
|
|
# full training-data PVC (27 months of OHLCV + MBP-10 + trades) without
|
|
# nsys profile overhead or sanitizer instrumentation. Iteration cycle is
|
|
# faster than argo-nsys.sh / argo-sanitizer.sh because there's no artefact
|
|
# upload step.
|
|
#
|
|
# Why L40S not local: the laptop has only 1 quarter of MBP-10/trades data,
|
|
# which truncates the fxcache below the 10-month minimum that the
|
|
# multi_fold_convergence smoke needs.
|
|
#
|
|
# Usage:
|
|
# ./scripts/argo-smoke.sh # multi_fold_convergence (default)
|
|
# ./scripts/argo-smoke.sh --test magnitude_distribution # different test
|
|
# ./scripts/argo-smoke.sh --ref <sha> # specific commit
|
|
# ./scripts/argo-smoke.sh --watch # follow logs
|
|
#
|
|
# Requires: argo CLI
|
|
|
|
set -euo pipefail
|
|
|
|
TEST_NAME="multi_fold_convergence::test_multi_fold_convergence"
|
|
COMMIT_REF="HEAD"
|
|
GPU_POOL=""
|
|
WATCH=false
|
|
CLEAN_CACHE="false"
|
|
|
|
usage() {
|
|
cat <<EOF
|
|
Usage: $(basename "$0") [OPTIONS]
|
|
|
|
Plain smoke-test runner on L40S (no profiler, no sanitizer).
|
|
|
|
Options:
|
|
--test <name> Smoke test name (default: $TEST_NAME)
|
|
Common values:
|
|
multi_fold_convergence::test_multi_fold_convergence
|
|
magnitude_distribution::test_magnitude_distribution
|
|
performance::test_real_data_single_epoch
|
|
iqn_quantile_monotonicity::iqn_multi_quantile_heads_produce_monotonic_estimates
|
|
--multi-fold Shortcut: --test multi_fold_convergence::test_multi_fold_convergence
|
|
(default — exercises fold-boundary code paths)
|
|
--ref <ref> Git ref to test (default: HEAD)
|
|
--gpu-pool <pool> GPU node pool (default: ci-training-l40s)
|
|
--clean-cache Wipe ml + ml-dqn artefacts before compile
|
|
(force fresh compile; deps stay cached). Use to
|
|
rule out build-cache contamination during bisect.
|
|
--watch Follow workflow logs
|
|
-h, --help Show this help
|
|
EOF
|
|
exit 0
|
|
}
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
--test) TEST_NAME="$2"; shift 2 ;;
|
|
--multi-fold) TEST_NAME="multi_fold_convergence::test_multi_fold_convergence"; shift ;;
|
|
--ref) COMMIT_REF="$2"; shift 2 ;;
|
|
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
|
|
--clean-cache) CLEAN_CACHE="true"; shift ;;
|
|
--watch) WATCH=true; shift ;;
|
|
-h|--help) usage ;;
|
|
*) echo "Unknown option: $1"; usage ;;
|
|
esac
|
|
done
|
|
|
|
if [ "$COMMIT_REF" = "HEAD" ]; then
|
|
COMMIT_REF=$(git rev-parse HEAD)
|
|
fi
|
|
|
|
CMD="argo submit -n foxhunt --from=wftmpl/smoke-test"
|
|
CMD="$CMD -p commit-ref=$COMMIT_REF"
|
|
CMD="$CMD -p test-name=$TEST_NAME"
|
|
CMD="$CMD -p clean-cache=$CLEAN_CACHE"
|
|
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
|
|
|
|
if $WATCH; then
|
|
CMD="$CMD --watch"
|
|
fi
|
|
|
|
echo "Submitting smoke workflow..."
|
|
echo " commit: $(echo "$COMMIT_REF" | cut -c1-8)"
|
|
echo " test: $TEST_NAME"
|
|
[[ "$CLEAN_CACHE" = "true" ]] && echo " clean: ml + ml-dqn target wipe before compile"
|
|
echo ""
|
|
eval "$CMD"
|