ComputePulse
Predictive GPU-Cluster Intelligence & Failure-Risk Forecasting
A predictive GPU-cluster intelligence prototype that combines telemetry monitoring, LightGBM risk scoring, explainable alert presentation, and an operational dashboard.
Application Interface
Genuine interface screenshot captured from the deployed ComputePulse codebase.
Problem & Context
GPU clusters operate under intense thermal and memory workloads. Identifying early warning signals of node stress before job crashes occur helps infrastructure teams prioritize maintenance.
Created during a competitive hackathon and subsequently organized into a clean prototype demonstrating how predictive models and explainability can aid hardware operations.
Role & Contributions
Team Lead, Aletheia. Coordinated project scope and team delivery.
- Alibaba PAI GPU trace preprocessing and historical dashboard replay
- LightGBM classifier estimating failed or interrupted workload observations
- SHAP-based attribution for engineered CPU, GPU, memory-pressure, and I/O features
- Rule-based mitigation recommendations and alert triage UI
- Fully deployed demonstration web interface
Architecture & Engineering Decisions
A historical Alibaba trace pipeline that engineers workload and resource features, estimates failure risk with LightGBM, and exposes explanations and advisory recommendations through FastAPI and React.
Architecture Overview: ComputePulse multi-node telemetry stream, LightGBM fault risk scoring, and SHAP root-cause feature attribution.
- 1.Data Preparation: Processes historical workload and sensor records from the Alibaba PAI GPU trace.
- 2.Risk Estimator: LightGBM classifier for workload failure-risk estimation, with a separate Isolation Forest anomaly model.
- 3.Attribution Layer: TreeSHAP explanations for engineered workload and resource-utilization features.
- 4.Operator Dashboard: Interactive cluster node grid, time-series charts, and rule-based mitigation recommendations.
Decision Log & Trade-offs
Rationale: Decoupling the FastAPI analytics backend from the React visualization frontend allowed clean API contracts and independent development.
Trade-off: Required maintaining OpenAPI schemas across both services.
Rationale: Provides an immediate visual triage heuristic for operators before inspecting multi-sensor graphs.
Trade-off: Threshold boundaries require calibration depending on hardware tolerance.
Experimental Methodology & Evaluation
Evaluated on 100,014 processed historical observations using a stratified random row-level holdout of 20,003 observations. Related workloads or machines may occur in multiple partitions.
Evaluation Metrics & Targets
Recorded workload failure-risk evaluation and offline placement evidence on historical trace data. These results do not establish future-time or unseen-machine generalization.
Implemented and Verified Capabilities
- Built a dashboard that replays static historical trace data.
- Integrated risk estimates, anomaly flags, explanations, and offline placement recommendations.
- Packaged data preparation, model artifacts, evaluation reports, and backend services.
Limitations & Scope Constraints
Scientific Boundaries & Future Scope:
- •Current evaluation uses a random row-level split; chronological and workload/machine-grouped validation remain pending.
- •Workload mitigation recommendations are rule-based advisory suggestions rather than executed orchestrations.