foxhunt/docs/archive/historical/SYSTEM_RESOURCE_MONITOR_REPORT.md

SYSTEM RESOURCE MONITOR REPORT - Agent 125

Generated: 2025-10-14 19:11:25 Monitoring Duration: 0 seconds Status: ✅ HEALTHY

---

Executive Summary

This report provides continuous system resource monitoring to prevent crashes during ML training.

Alert Summary

Resource	Alerts	Max Usage	Threshold	Status
Memory	0	57%	90%	✅ OK
Swap	0	0MB	6144MB	✅ OK
Disk	0	7%	85%	✅ OK

---

Current System Status

Memory Status

               total        used        free      shared  buff/cache   available
Mem:            31Gi        17Gi        10Gi       204Mi       3.3Gi        13Gi
Swap:          8.0Gi          0B       8.0Gi

Disk Status

Filesystem                Size  Used Avail Use% Mounted on
rpool/ROOT/ubuntu_e5jx5b  171G   11G  160G   7% /

Top Memory Consumers

jgrusew+   17758 69.2 20.3 99441876 6639384 pts/0 Rl+ 19:01   7:09 claude --permission-mode bypassPermissions --continue
jgrusew+   33147  101  2.9 2719368 971656 ?      Sl   19:10   0:35 /home/jgrusewski/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name ml --edition=2021 ml/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C panic=abort -C linker-plugin-lto -C codegen-units=1 --warn=clippy::wildcard_imports --deny=clippy::wildcard_enum_match_arm --deny=clippy::verbose_file_reads --warn=variant_size_differences --warn=clippy::used_underscore_binding --deny=clippy::use_debug --warn=clippy::unwrap_used --deny=clippy::unwrap_in_result --warn=unused_qualifications --warn=unused_lifetimes --warn=unused_import_braces --warn=unused_extern_crates --warn=unused_crate_dependencies --deny=clippy::unseparated_literal_suffix --warn=unsafe_code --warn=unreachable_pub --deny=clippy::unreachable --warn=clippy::unnecessary_wraps --deny=clippy::unnecessary_safety_doc --deny=clippy::unnecessary_safety_comment --warn=clippy::unnecessary_cast --deny=clippy::unimplemented --warn=clippy::undocumented_unsafe_blocks --deny=clippy::unchecked_duration_subtraction --warn=clippy::type_complexity --deny=clippy::try_err --warn=clippy::trivially_copy_pass_by_ref --warn=clippy::too_many_lines --warn=clippy::too_many_arguments --deny=clippy::todo --deny=clippy::tests_outside_test_module --deny=clippy::suspicious_xor_used_as_pow --deny=clippy::string_to_string --deny=clippy::string_slice --deny=clippy::string_add --deny=clippy::str_to_string --warn=clippy::single_match_else --warn=clippy::single_char_lifetime_names --warn=clippy::similar_names --deny=clippy::shadow_unrelated --deny=clippy::shadow_same --deny=clippy::shadow_reuse --deny=clippy::semicolon_inside_block --allow=clippy::self_named_module_files --deny=clippy::same_name_method --deny=clippy::rest_pat_in_fully_bound_structs --warn=clippy::redundant_clone --deny=clippy::rc_mutex --deny=clippy::rc_buffer --allow=clippy::pub_use --warn=clippy::print_stdout --warn=clippy::print_stderr --deny=clippy::partial_pub_fields --warn=clippy::panic --deny=clippy::out_of_bounds_indexing --deny=clippy::non_ascii_literal --deny=clippy::needless_raw_strings --deny=clippy::mutex_atomic --warn=clippy::multiple_unsafe_ops_per_block --deny=clippy::multiple_inherent_impl --deny=clippy::modulo_arithmetic --warn=clippy::module_name_repetitions --allow=clippy::mod_module_files --deny=clippy::mixed_read_write_in_expression --deny=clippy::missing_enforced_import_renames --allow=missing_docs --warn=clippy::missing_const_for_fn --deny=clippy::mem_forget --warn=clippy::map_err_ignore --warn=clippy::manual_let_else --deny=clippy::lossy_float_literal --deny=clippy::let_underscore_must_use --warn=clippy::large_types_passed_by_value --deny=clippy::large_include_file --warn=clippy::large_enum_variant --warn=clippy::integer_division --deny=clippy::inline_asm_x86_intel_syntax --deny=clippy::inline_asm_x86_att_syntax --deny=clippy::infinite_loop --warn=clippy::indexing_slicing --deny=clippy::impl_trait_in_params --deny=clippy::if_then_some_else_none --deny=clippy::host_endian_bytes --deny=clippy::get_unwrap --deny=clippy::format_push_string --deny=clippy::fn_to_numeric_cast_any --deny=clippy::float_cmp_const --warn=clippy::float_arithmetic --deny=clippy::filetype_is_file --warn=clippy::expect_used --deny=clippy::exit --deny=clippy::error_impl_error --warn=clippy::enum_variant_names --deny=clippy::empty_structs_with_brackets --deny=clippy::empty_drop --deny=clippy::else_if_without_else --warn=clippy::doc_markdown --deny=clippy::disallowed_script_idents --deny=clippy::deref_by_slicing --warn=clippy::default_numeric_fallback --deny=clippy::decimal_literal_representation --deny=clippy::dbg_macro --deny=clippy::create_dir --warn=clippy::cognitive_complexity --warn=clippy::clone_on_ref_ptr --deny=clippy::assertions_on_result_states --warn=clippy::as_conversions --warn=clippy::arithmetic_side_effects --cfg feature="default" --cfg feature="minimal-inference" --check-cfg cfg(docsrs,test) --check-cfg cfg(feature, values("arrayfire", "aws-config", "aws-credential-types", "aws-sdk-s3", "aws-types", "cuda", "default", "financial", "gc", "high-precision", "minimal-inference", "s3-storage", "simd", "urlencoding")) -C metadata=01101644b51a2412 -C extra-filename=-a7c38fb5468d485a --out-dir /home/jgrusewski/Work/foxhunt/target/release/deps -C incremental=/home/jgrusewski/Work/foxhunt/target/release/incremental -L dependency=/home/jgrusewski/Work/foxhunt/target/release/deps --extern anyhow=/home/jgrusewski/Work/foxhunt/target/release/deps/libanyhow-180e913d399bd5dd.rmeta --extern approx=/home/jgrusewski/Work/foxhunt/target/release/deps/libapprox-cbfa1ec09c1ff782.rmeta --extern async_trait=/home/jgrusewski/Work/foxhunt/target/release/deps/libasync_trait-db9f2d99224c01df.so --extern bincode=/home/jgrusewski/Work/foxhunt/target/release/deps/libbincode-6c24648171a3dab5.rmeta --extern candle_core=/home/jgrusewski/Work/foxhunt/target/release/deps/libcandle_core-c4c0751a15c8582d.rmeta --extern candle_nn=/home/jgrusewski/Work/foxhunt/target/release/deps/libcandle_nn-94794244e6d3be4e.rmeta --extern candle_optimisers=/home/jgrusewski/Work/foxhunt/target/release/deps/libcandle_optimisers-f2c8379b461e9f72.rmeta --extern chrono=/home/jgrusewski/Work/foxhunt/target/release/deps/libchrono-a31b326990c1faca.rmeta --extern clap=/home/jgrusewski/Work/foxhunt/target/release/deps/libclap-7b7981632d99c9bd.rmeta --extern common=/home/jgrusewski/Work/foxhunt/target/release/deps/libcommon-01517348b70d2779.rmeta --extern config=/home/jgrusewski/Work/foxhunt/target/release/deps/libconfig-ec4ac83fccc99f39.rmeta --extern crossbeam=/home/jgrusewski/Work/foxhunt/target/release/deps/libcrossbeam-eeaf6dbf902636c5.rmeta --extern dashmap=/home/jgrusewski/Work/foxhunt/target/release/deps/libdashmap-b06d923e6fe21011.rmeta --extern data=/home/jgrusewski/Work/foxhunt/target/release/deps/libdata-eee4875d79ce056b.rmeta --extern databento=/home/jgrusewski/Work/foxhunt/target/release/deps/libdatabento-a810b3cf14562d54.rmeta --extern dbn=/home/jgrusewski/Work/foxhunt/target/release/deps/libdbn-9b147dcf96d7d60c.rmeta --extern dotenv=/home/jgrusewski/Work/foxhunt/target/release/deps/libdotenv-cc006461563e7721.rmeta --extern fastrand=/home/jgrusewski/Work/foxhunt/target/release/deps/libfastrand-4a38212df9230956.rmeta --extern flate2=/home/jgrusewski/Work/foxhunt/target/release/deps/libflate2-b90b70aa324b1df5.rmeta --extern fs2=/home/jgrusewski/Work/foxhunt/target/release/deps/libfs2-c1a99bb5d87fa6ea.rmeta --extern futures=/home/jgrusewski/Work/foxhunt/target/release/deps/libfutures-32fcf9b8ac475ce5.rmeta --extern half=/home/jgrusewski/Work/foxhunt/target/release/deps/libhalf-f4f059af398a4a14.rmeta --extern lazy_static=/home/jgrusewski/Work/foxhunt/target/release/deps/liblazy_static-8280f5c32fdad539.rmeta --extern libc=/home/jgrusewski/Work/foxhunt/target/release/deps/liblibc-c9e013ee444aabe3.rmeta --extern lru=/home/jgrusewski/Work/foxhunt/target/release/deps/liblru-2a2d1a3f59956d87.rmeta --extern memmap2=/home/jgrusewski/Work/foxhunt/target/release/deps/libmemmap2-bbff928b2cb8c0c6.rmeta --extern nalgebra=/home/jgrusewski/Work/foxhunt/target/release/deps/libnalgebra-e551df1bda3fa581.rmeta --extern ndarray=/home/jgrusewski/Work/foxhunt/target/release/deps/libndarray-f23d8fced177dff0.rmeta --extern num=/home/jgrusewski/Work/foxhunt/target/release/deps/libnum-cb19870c91d6b218.rmeta --extern num_traits=/home/jgrusewski/Work/foxhunt/target/release/deps/libnum_traits-ebc67ff9ddc405d5.rmeta --extern num_cpus=/home/jgrusewski/Work/foxhunt/target/release/deps/libnum_cpus-f0316af4cb420fe7.rmeta --extern once_cell=/home/jgrusewski/Work/foxhunt/target/release/deps/libonce_cell-e75be84461a86d11.rmeta --extern parking_lot=/home/jgrusewski/Work/foxhunt/target/release/deps/libparking_lot-5906f3240fa856d3.rmeta --extern petgraph=/home/jgrusewski/Work/foxhunt/target/release/deps/libpetgraph-12b27b6940b83694.rmeta --extern prometheus=/home/jgrusewski/Work/foxhunt/target/release/deps/libprometheus-e97c616671e92821.rmeta --extern rand=/home/jgrusewski/Work/foxhunt/target/release/deps/librand-b45d57767a9ea88e.rmeta --extern rand_distr=/home/jgrusewski/Work/foxhunt/target/release/deps/librand_distr-45148a5f6a520556.rmeta --extern rayon=/home/jgrusewski/Work/foxhunt/target/release/deps/librayon-0ad2aaa3fd188326.rmeta --extern reqwest=/home/jgrusewski/Work/foxhunt/target/release/deps/libreqwest-a2ca063b87a0eb12.rmeta --extern risk=/home/jgrusewski/Work/foxhunt/target/release/deps/librisk-ebea8cb69337a3b5.rmeta --extern rust_decimal=/home/jgrusewski/Work/foxhunt/target/release/deps/librust_decimal-b6667ba314e7622c.rmeta --extern semver=/home/jgrusewski/Work/foxhunt/target/release/deps/libsemver-68f0373c3a7bc5e2.rmeta --extern serde=/home/jgrusewski/Work/foxhunt/target/release/deps/libserde-68fa479e5b8e757e.rmeta --extern serde_json=/home/jgrusewski/Work/foxhunt/target/release/deps/libserde_json-51b766013e2cd39c.rmeta --extern sha2=/home/jgrusewski/Work/foxhunt/target/release/deps/libsha2-02dccc93c0d63b3d.rmeta --extern sqlx=/home/jgrusewski/Work/foxhunt/target/release/deps/libsqlx-ef6c1c60c539227c.rmeta --extern statrs=/home/jgrusewski/Work/foxhunt/target/release/deps/libstatrs-a39a426b600e81e6.rmeta --extern storage=/home/jgrusewski/Work/foxhunt/target/release/deps/libstorage-ef34bf49087fa8b5.rmeta --extern structopt=/home/jgrusewski/Work/foxhunt/target/release/deps/libstructopt-6d3b3c02ef41e840.rmeta --extern sysinfo=/home/jgrusewski/Work/foxhunt/target/release/deps/libsysinfo-92176a11d65b95e0.rmeta --extern tempfile=/home/jgrusewski/Work/foxhunt/target/release/deps/libtempfile-0264fd6d0ee81903.rmeta --extern thiserror=/home/jgrusewski/Work/foxhunt/target/release/deps/libthiserror-1174f1f837a6df28.rmeta --extern tokio=/home/jgrusewski/Work/foxhunt/target/release/deps/libtokio-8d459cec4bf7c6f0.rmeta --extern tracing=/home/jgrusewski/Work/foxhunt/target/release/deps/libtracing-abe5e1a80aca75b6.rmeta --extern tracing_subscriber=/home/jgrusewski/Work/foxhunt/target/release/deps/libtracing_subscriber-5893092e2ec43f73.rmeta --extern trading_engine=/home/jgrusewski/Work/foxhunt/target/release/deps/libtrading_engine-769daa0e5e846502.rmeta --extern uuid=/home/jgrusewski/Work/foxhunt/target/release/deps/libuuid-bbf1a88f7219ae0f.rmeta -C link-arg=-Wl,-z,relro,-z,now -C link-arg=-Wl,--as-needed -C target-cpu=native -C target-feature=+avx2,+fma,+bmi2 -C opt-level=3 -C codegen-units=1 -L native=/home/jgrusewski/Work/foxhunt/target/release/build/ring-4461cc0d5e4e7d22/out -L native=/home/jgrusewski/Work/foxhunt/target/release/build/zstd-sys-96048fbd7afe6d09/out -L native=/home/jgrusewski/Work/foxhunt/target/release/build/lz4-sys-4b82d71b174ab8ea/out --cfg cpu_only_build
jgrusew+   25348  174  1.0 2550464 354012 ?      Rl   19:06   8:53 target/release/examples/train_tft_dbn --epochs 200 --learning-rate 0.001 --batch-size 32 --lookback-window 60 --forecast-horizon 10 --use-gpu --output-dir /home/jgrusewski/Work/foxhunt/ml/trained_models/production/tft
70         27036  0.1  0.5 8401252 195024 ?      Ss   19:07   0:00 postgres
472        28005  1.3  0.5 1798976 165340 ?      Ssl  19:07   0:02 grafana server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docker cfg:default.log.mode=console cfg:default.paths.data=/var/lib/grafana cfg:default.paths.logs=/var/log/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
jgrusew+   33851  7.5  0.4 158148 139300 ?       S    19:11   0:00 /home/jgrusewski/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo build --release -p ml --example tune_hyperparameters --features cuda
jgrusew+   32437  0.8  0.4 293036 139248 ?       Sl   19:10   0:00 /home/jgrusewski/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo run --release -p ml --example train_mamba2_dbn -- --epochs 200 --batch-size 16 --learning-rate 0.0001 --sequence-length 60 --hidden-dim 256 --state-dim 64 --data-dir test_data/real/databento/ml_training --output-dir ml/trained_models/production/mamba2 --use-gpu
nobody     26983  0.9  0.3 1600036 118136 ?      Ssl  19:07   0:02 /bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=15d --web.enable-lifecycle --query.max-concurrency=50
root       27000  0.4  0.2 1536780 97152 ?       Ssl  19:07   0:00 minio server /data --console-address :9001
dhcpcd     27172  0.4  0.2 1570752 93484 ?       Ssl  19:07   0:01 vault server -dev -dev-listen-address=0.0.0.0:8200

Active Training Processes

jgrusew+   25348  174  1.0 2550464 354012 ?      Rl   19:06   8:53 target/release/examples/train_tft_dbn --epochs 200 --learning-rate 0.001 --batch-size 32 --lookback-window 60 --forecast-horizon 10 --use-gpu --output-dir /home/jgrusewski/Work/foxhunt/ml/trained_models/production/tft
jgrusew+   32437  0.8  0.4 293036 139248 ?       Sl   19:10   0:00 /home/jgrusewski/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo run --release -p ml --example train_mamba2_dbn -- --epochs 200 --batch-size 16 --learning-rate 0.0001 --sequence-length 60 --hidden-dim 256 --state-dim 64 --data-dir test_data/real/databento/ml_training --output-dir ml/trained_models/production/mamba2 --use-gpu
jgrusew+   33851  7.4  0.4 158148 139300 ?       S    19:11   0:00 /home/jgrusewski/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo build --release -p ml --example tune_hyperparameters --features cuda

---

Resource Usage Timeline

Note: See system_resource_monitor.log for detailed timeline with timestamps

Memory Usage Pattern

• Peak Memory: 57%
• Alert Threshold: 90%
• Alerts Triggered: 0

Swap Usage Pattern

• Peak Swap: 0MB
• Alert Threshold: 6144MB
• Alerts Triggered: 0

Disk Usage Pattern

• Peak Disk: 7%
• Alert Threshold: 85%
• Alerts Triggered: 0

---

Monitoring Configuration

check_interval: 60s
thresholds:
  memory: 90%
  swap: 6144MB
  disk: 85%
alerts:
  memory: none
  swap: none
  disk: none

---

Recommendations

System Optimization

1. Memory Management:

   • Current available: 13Gi
   • Recommendation: ✅ Healthy - no action needed

2. Swap Usage:

   • Current swap: 0B
   • Recommendation: ✅ Minimal swap - good performance

3. Disk Space:

   • Available: 160G
   • Recommendation: ✅ Sufficient space

Training Optimization

1. Batch Size Tuning:

   • If memory >80%, reduce batch_size by 50%
   • If memory <50%, can increase batch_size by 50%

2. Gradient Checkpointing:

   • Enable for MAMBA-2/TFT models if memory >70%
   • Trades 30% more compute for 50% less memory

3. Mixed Precision:

   • Use fp16 instead of fp32 to halve memory usage
   • Minimal accuracy impact (<0.5% for most models)

---

Emergency Procedures

If Memory >90%

# 1. Kill non-essential processes
pkill -f 'chrome|firefox|slack' 2>/dev/null || true

# 2. Clear page cache (safe, no data loss)
sudo sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

# 3. Kill largest memory consumer (emergency only)
kill -9 $(ps aux --sort=-%mem | awk 'NR==2 {print $2}')

If Swap >6GB

# System is thrashing - immediate action required
# Kill the training process and restart with reduced batch size
pkill -f 'train_liquid|optuna'

# Wait for swap to clear
sleep 30

# Reduce batch size in config and restart

If Disk >85%

# Clean up old checkpoints
rm -rf ml/tuning_checkpoints/trial_*/checkpoint_epoch_*

# Remove old logs
find . -name '*.log' -mtime +7 -delete

# Clean cargo cache
cargo clean

# Check space again
df -h /

---

Continuous Monitoring

Status: 🔴 STOPPED

To continue monitoring:

# Start monitoring (runs in background)
nohup ./scripts/system_resource_monitor.sh monitor > /dev/null 2>&1 &

# Stop monitoring
kill $(cat /tmp/resource_monitor.pid 2>/dev/null)

To generate this report again:

./scripts/system_resource_monitor.sh report

---

Log File

Full monitoring log: system_resource_monitor.log

# View last 50 entries
tail -50 system_resource_monitor.log

# View all alerts
grep -E 'ALERT|WARNING|CRITICAL' system_resource_monitor.log

# View memory timeline
grep 'Memory:' system_resource_monitor.log

---

Last Updated: 2025-10-14 19:11:25 Next Check: In 60 seconds (if monitoring active) Generated by: Agent 125 - System Resource Monitor