infra(argo): smoke-test --clean-cache parameter for build-cache-isolated bisect
The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a persistent PVC, so every smoke probe builds incrementally on top of artefacts left by prior probes. File deletions (e.g., regime_conditional.rs inff00af68a) can leave dangling rmeta/object references that perturb downstream codegen between bisect runs — making the bisect result contingent on which order probes were submitted in, not on the source. Adds an optional `clean-cache` parameter (default `false`) which runs `cargo clean -p ml -p ml-dqn --release` before the compile step. Other crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean. Use case: re-runninga52d99613+ff00af68awith `--clean-cache` to verify the bisect under controlled build conditions. If the broken/clean status flips with clean cache, the regression is build-state-dependent rather than source-line; if it reproduces, the source regression is real and bisect is definitive. scripts/argo-smoke.sh exposes `--clean-cache` flag passing through to the workflow parameter.
This commit is contained in:
@@ -25,6 +25,7 @@ TEST_NAME="multi_fold_convergence::test_multi_fold_convergence"
|
||||
COMMIT_REF="HEAD"
|
||||
GPU_POOL=""
|
||||
WATCH=false
|
||||
CLEAN_CACHE="false"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
@@ -43,6 +44,9 @@ Options:
|
||||
(default — exercises fold-boundary code paths)
|
||||
--ref <ref> Git ref to test (default: HEAD)
|
||||
--gpu-pool <pool> GPU node pool (default: ci-training-l40s)
|
||||
--clean-cache Wipe ml + ml-dqn artefacts before compile
|
||||
(force fresh compile; deps stay cached). Use to
|
||||
rule out build-cache contamination during bisect.
|
||||
--watch Follow workflow logs
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
@@ -55,6 +59,7 @@ while [[ $# -gt 0 ]]; do
|
||||
--multi-fold) TEST_NAME="multi_fold_convergence::test_multi_fold_convergence"; shift ;;
|
||||
--ref) COMMIT_REF="$2"; shift 2 ;;
|
||||
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
|
||||
--clean-cache) CLEAN_CACHE="true"; shift ;;
|
||||
--watch) WATCH=true; shift ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo "Unknown option: $1"; usage ;;
|
||||
@@ -68,6 +73,7 @@ fi
|
||||
CMD="argo submit -n foxhunt --from=wftmpl/smoke-test"
|
||||
CMD="$CMD -p commit-ref=$COMMIT_REF"
|
||||
CMD="$CMD -p test-name=$TEST_NAME"
|
||||
CMD="$CMD -p clean-cache=$CLEAN_CACHE"
|
||||
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
|
||||
|
||||
if $WATCH; then
|
||||
@@ -77,5 +83,6 @@ fi
|
||||
echo "Submitting smoke workflow..."
|
||||
echo " commit: $(echo "$COMMIT_REF" | cut -c1-8)"
|
||||
echo " test: $TEST_NAME"
|
||||
[[ "$CLEAN_CACHE" = "true" ]] && echo " clean: ml + ml-dqn target wipe before compile"
|
||||
echo ""
|
||||
eval "$CMD"
|
||||
|
||||
Reference in New Issue
Block a user