Real-time analytics depends on engineering work that occurs before any KPI or model runs: channel identity, units, clocks, wellbore and BHA context, state, quality, lineage, and recovery from delayed data. A correct calculation on the wrong wellbore or stale configuration is still a wrong result.

Normalize sources before calculations

“Rig data” is several datasets with different owners and update cycles:

Source Typical content
Rig control / EDR Block position, hook load, RPM, torque, pump activity, pressure
Directional Surveys, inclination, azimuth, toolface
Mud Mud weight, PV, YP, gel strengths, temperature-dependent properties
BHA and drillstring Bit, motor, stabilizers, drillpipe, dimensions
Well architecture Hole size, casing depth and ID, wellbore trajectory
Operational reporting Equipment and mud changes, casing runs, daily activity, notes

Real-time analytics needs to convert those sources into one coherent representation. That is fundamentally a data-engineering task. Different rigs and vendors may expose conceptually similar measurements under different:

  • tag names,
  • channel identifiers,
  • units,
  • scaling conventions.

A generic analytics model wants a canonical surface-torque channel, but Rig A might expose one tag, Rig B another, and Rig C two torque measurements from different systems. SPE/IADC-184743 describes rig-specific mapping layers that connect those sensor types and tags to a common data model without rewriting the analytics for each rig.[3] For example, a vendor tag may map to the canonical variable SURFACE_TORQUE.

Once that mapping is wrong, everything downstream can still execute perfectly. It will simply calculate the wrong thing.

Three fictional rig torque tags mapped to a canonical surface-torque variable, with one incorrect mapping producing a bad downstream result.

Tag mapping converts rig-specific names into engineering variables; a wrong map propagates into every dependent calculation.

Treat time and freshness separately

Real-time sensor streams usually arrive with timestamps, but different sources may update at:

  • 10 Hz,
  • 1 Hz,
  • 0.1 Hz,
  • irregular intervals.

SPE-191426 and SPE-191797 both emphasize the challenge of combining data with different frequencies.[1,2] If torque arrives at 10:00:00, RPM at 10:00:00.4, and ROP at 10:00:01, the software must define which values belong in one MSE calculation. The alignment strategy might use:

  • nearest observation,
  • bounded forward fill,
  • resampling,
  • interpolation,
  • aggregation.

That decision is part of the analytical method. A 12.1-ppg mud weight entered six hours ago still exists in the database, but the well may have been weighted up since then. Likewise, the database may contain:

  • a BHA,
  • a survey,
  • casing depth.

But the relevant question is:

Is this the latest valid context for the operation being analyzed?

Real-time systems therefore need the value, timestamp, source, and quality together. A stale value is not equivalent to a newly observed value with the same number.

Resolve changing well context

Imagine surface measurements are excellent. The hydraulic model receives:

  • accurate flow,
  • accurate SPP,
  • accurate RPM.

But the BHA record contains the previous bit nozzle configuration, so the model predicts the wrong pressure and creates an apparent hydraulic departure. SPE-181076 describes this class of failure: well and rig information needed by analytical models was sometimes incorrect or unavailable, and human-entered context could lag the operation.[4] Context quality is data quality.

Measured depth alone does not uniquely identify a physical location when multiple wellbores exist. Imagine:

  • Main bore: 14,000 ft MD.
  • Sidetrack: 14,000 ft MD.

Those represent different physical locations and drilling histories. Therefore, well plus MD may not be enough. The analytical key may also need the wellbore identity.

This is especially important when retrieving:

  • surveys,
  • plans,
  • BHA records,
  • offset intervals.

Without wellbore identity, perfectly valid data can be attached to the wrong trajectory.

Main wellbore and open-hole sidetrack with separate locations at the same 14,000 ft measured depth.

Measured depth is not globally unique inside a sidetracked well.

Carry quality and lineage through every product

SPE-181076 emphasizes the operational cost of bad drilling data. Its field trial focused on eight core surface channels and established a workflow to identify and improve:

  • missing data,
  • outliers,
  • bias errors.[5]

This order matters. A sophisticated MSE algorithm operating on a bad torque sensor creates a sophisticated wrong answer. The same applies to:

  • torque and drag,
  • hydraulics,
  • dysfunction detection,
  • machine learning.

There is little value in improving the downstream model while ignoring upstream measurement quality. Imagine an engineer asks:

Why was MSE high at 15,300 ft?

To answer properly, the system may need to reconstruct:

  • torque used,
  • WOB used,
  • RPM used,
  • ROP used,
  • bit size,
  • calculation version,
  • rig state.

If those relationships are not preserved, the final KPI becomes difficult to audit. A general pattern is:

$$Measurement \rightarrow Cleaned\ Measurement \rightarrow Derived\ Feature \rightarrow Model\ Output \rightarrow Recommendation$$

Each layer should retain enough lineage to trace backward.

Design for communications recovery

Rig connectivity can be intermittent. SPE-191797 describes local storage and store-forward capability so rig data could be retained and transferred after telecommunications recovered.[1] A real-time system should distinguish remote-connectivity failure from rig-acquisition failure. Where practical, local acquisition allows:

  • the rig to continue analyzing data,
  • the complete record to be preserved,
  • the office copy to catch up later.

Suppose a rig stores 30 minutes of data locally. The network returns. The buffered data is transmitted. Meanwhile, some recent samples have already reached the office through another path. The ingestion system now needs to avoid:

  • duplicate rows,
  • double-counted events,
  • out-of-order timestamps.

This is a classic data-engineering problem with very real drilling consequences. A duplicated connection event can alter:

  • ILT statistics,
  • stand counts,
  • time totals.

A duplicated sensor stream can distort:

  • distributions,
  • model inputs.

Therefore ingestion should ideally be idempotent:

replaying the same source data should not create a second version of the same observation unless the source itself represents a legitimate revision.

Represent the well, then run the pipeline

Database convenience should not override drilling reality. A useful drilling information hierarchy may include:

$$Operator \rightarrow Field \rightarrow Pad \rightarrow Well \rightarrow Wellbore \rightarrow BHA\ Run \rightarrow Interval \rightarrow Event$$

Not every application needs every level. But ignoring the hierarchy creates ambiguity later. This becomes especially important when:

  • wells are sidetracked,
  • multiple plans exist,
  • BHAs are rerun,
  • historical offsets are compared.

Industry data interfaces are extremely valuable. They make it possible to exchange rig information without building every connection from scratch. But a valid data stream still does not automatically answer:

  • Is this the correct channel?
  • Is the unit correct?
  • Is the sensor calibrated?
  • Is the timestamp aligned?
  • Is the value stale?
  • Which wellbore does it belong to?
  • Which BHA was active?

SPE-191797 itself describes an operational system that had to consume not just WITS/WITSML but also:

  • PLC data,
  • OPC-connected sources,
  • mudlogger information,
  • daily-report context.[1]

The practical conclusion is:

Data transport standards are an important layer of drilling data engineering, not a complete substitute for it.

A practical real-time pipeline can be summarized in ten stages:

Stage Required work
Acquire Receive rig-control, EDR, WITS/WITSML, MWD, mudlogging, and reporting data.
Identify Resolve the rig, well, wellbore, source, and channel.
Normalize Standardize variable names, units, and sign conventions.
Synchronize Align frequencies and apply explicit rules for latency and stale data.
Validate Detect missing, outlying, biased, frozen, or questionable values.
Attach context Resolve the active BHA, surveys, casing, mud properties, and formation where available.
Establish state Classify the current rig activity.
Calculate Run deterministic calculations, statistical features, and event models.
Persist Store measurements, context, outputs, and provenance.
Deliver Serve dashboards, alerts, APIs, reports, and controlled AI access.

DrillingMetrics Time Traces alongside active BHA configuration, survey-derived wellbore context, and rig-state classification.

The visible traces are one layer of the product; BHA configuration, survey-derived wellbore context, and rig-state classification determine what the streams mean.

Related technical resources

References

  1. Shahri, M., Kutlu, B., Thetford, T., Nelson, B., Wilson, T., Behounek, M., Ambrus, A., and Ashok, P. Adopting Physical Models in Real-Time Drilling Application: Wellbore Hydraulics. SPE-191797-MS, SPE Liquids-Rich Basins Conference, North America, Midland, Texas, 2018.

  2. Shahri, M., Wilson, T., Thetford, T., Nelson, B., Behounek, M., Ambrus, A., D'Angelo, J., and Ashok, P. Implementation of a Fully Automated Real-Time Torque and Drag Model for Improving Drilling Performance: Case Study. SPE-191426-MS, SPE Annual Technical Conference and Exhibition, Dallas, Texas, 2018.

  3. Behounek, M., Thetford, T., Yang, L., Hofer, E., White, M., Ashok, P., Ambrus, A., and Ramos, D. Human Factors Engineering in the Design and Deployment of a Novel Data Aggregation and Distribution System for Drilling Operations. SPE/IADC-184743-MS, SPE/IADC Drilling Conference and Exhibition, The Hague, Netherlands, 2017.

  4. Ashok, P., Ambrus, A., Ramos, D., Lutteringer, J., Behounek, M., Yang, Y. L., Thetford, T., and Weaver, T. A Step by Step Approach to Improving Data Quality in Drilling Operations: Field Trials in North America. SPE-181076-MS, SPE Intelligent Energy International Conference and Exhibition, Aberdeen, Scotland, 2016.

  5. Ashok, P., van Oort, E., and Ambrus, A. Automatic Sensor Data Validation: Improving the Quality and Reliability of Rig Data. SPE-163726-MS, SPE Digital Energy Conference and Exhibition, The Woodlands, Texas, 2013.