From proof of concept to production: engineering the predictive maintenance platform

• The lifecycle — how the work moved from proof of concept through engineering, testing and acceptance to release.
• The deployed system — what runs in production: architecture, framework, model management, cloud cost, reliability and governance.
• Operating, adoption and improving — how the system is monitored, used, retrained and kept useful.

• Engineering: the workflow was rebuilt as modular components, each called through an API, with configuration kept apart from code (04).
• Testing: automated CI/CD tests on every change to the framework code, plus end-to-end regression testing of the whole workflow before release (08).
• User acceptance: IT, maintenance and operations teams tested the outputs on a small scale, on a few equipment models, before wider use.
• Phased release: coverage was rolled out gradually, with user testing along the way, rather than switched on fleet-wide at once.

The result was a repeatable, supportable workflow, not a model that depends on manual runs and one-off analysis.

Scheduled orchestration starts batch runs three to six times a day; the scheduler, the step APIs and unattended execution together form the orchestration layer. Units are grouped into batches by criticality: management ranked equipment by how critical or expensive it is, and the most critical batches run most often. Run frequency is an input parameter, kept separate from the code, so it can be adjusted without touching the framework.

The sensors stream continuously; each batch run processes the latest telemetry from the sensor data tables.

Each run first loads its parameter inputs (step 03): a configuration artifact with the run’s settings and parameters, and model artifacts with the trained models, both loaded from storage (06).

An anomaly score shows deviation from learned behavior; on its own it is not a confirmed mechanical diagnosis. That is why each alert carries its contributing signals and severity for a person to investigate.

The shared framework removes duplicated engineering work. It does not remove the need to assess the data, configure the analysis and validate performance for each new equipment class.

• Model registry: every model version is kept in a central registry with its artifacts, training metadata, tuning and hyperparameters, and a written commentary on what changed and why.
• Two artifacts per run: a configuration artifact supplies the model and configuration parameters, the sensor feature list (raw and engineered), and the hyperparameter and tuning settings; a model artifact supplies the trained model itself. Both are loaded from storage for each run, so every execution uses the current versions and can be traced back to them.
• Retraining: models are retrained every two months as a controlled process, separate from the production runs (07). Retraining is prompted mainly by user feedback and operating changes — a drop in performance, new data, a new equipment type — and in part by automated triggers.
• Evaluation before release: every retrained model is compared with the previous version on a window of historical data, showing how its predictions differ, and whether they improve, before it is released.

Retraining is a deliberate, documented step rather than an automated loop, so every change to a model is recorded and traceable.

• QA/QC as a development discipline: rigorous QA/QC and testing ran throughout development of the framework code, so the platform could reliably handle a varied, changing equipment fleet, model updates and unexpected data conditions. The discipline mattered more as the framework was extended across equipment configurations.
• Automated and regression testing: every change to the framework code is tested automatically (CI/CD) before it is merged, and the whole workflow was regression-tested end to end during the production deployment phase.
• Data-quality controls: missing, stuck or invalid readings are caught at the start of every run. Bad inputs are set aside and logged, and missing data is imputed where needed, so nothing invalid flows silently through the workflow.
• Controlled exits and informative error messages: when something fails, the run stops cleanly and says why, instead of going dark.
• Failure notifications and recovery: failures in the framework code trigger notifications and error logs, and a failed run can be retried or recovered on demand through the step APIs.
• Run logging and observability: every step, every API call and every function is logged to a separate log registry, and every kind of error encountered is recorded for troubleshooting and improvement.

The logs give a transparent view of how the system behaves in production, and the client’s data engineering teams found them especially useful.

System reliability and model quality are related but distinct: a run can succeed and still produce an unhelpful alert, and a good model can be undermined by missing inputs. Both are monitored.

• Authentication first: every run starts with a security and authentication step (step 02) before it reaches equipment or sensor data.
• Secrets management: keys, credentials and other secrets are managed outside the code.
• Separate environments: development, user acceptance and production were kept apart.
• Role-based access: who can see which dashboards, alerts and underlying data follows the client’s own roles and data policies.
• Data residency and retention: data and outputs stay inside the client’s cloud environment. Retention rules were refined during deployment: detailed prediction outputs are kept for about nine months, batch-run details and some reports for about two.
• Model lineage and auditability: the model registry and the run logs together record which model and configuration produced each result.