A statistical benchmark is trustworthy only when its population, analytical unit, measurement method, weighting, dispersion, and recency match the decision. Millions of sensor rows do not compensate for five non-comparable wells. Statistical precision and engineering relevance must be evaluated separately.
Count independent engineering units
Compare two benchmarks:
| Benchmark | Volume | Population |
|---|---|---|
| A | 40 lateral stands | Same formation and hole size; similar bit/BHA |
| B | 50,000 stands | Several formations, hole sizes, BHA systems, and years |
Benchmark B has far more data but may be much less relevant. This separates two concepts.
- Statistical precision: How stable is the estimate within the population that produced it?
- Engineering relevance: Does that population represent the drilling condition we are trying to evaluate?
A benchmark can have excellent statistical precision and poor engineering relevance. In operational drilling, relevance frequently matters more.
Statistical precision and engineering relevance are separate properties. A very stable estimate can still answer the wrong engineering question.
A rig recording at 1 Hz produces 3,600 observations per hour and 36,000 in ten hours. Those are not 36,000 independent drilling conditions. Adjacent samples are strongly related because the bit is:
- in the same well,
- often in the same formation,
- using the same BHA,
- under nearly the same parameters.
This is a classic problem in time-series statistics. Nearby observations are correlated. Counting every sensor sample as though it were an independent experiment creates an illusion of sample size. Drilling data is hierarchical: samples belong to intervals, stands, BHA runs, wells, and rigs.
Different questions require different analytical units, and a benchmark should identify the unit on which its comparison is based:
| Question | Useful analytical unit |
|---|---|
| Typical instantaneous rotary torque? | Sensor sample or short interval |
| ROP variation between stands? | Stand |
| Which BHA configuration performs better? | BHA run |
| Did one drilling program outperform another? | Independent well |
Balance comparability against sample size
If 18 candidate runs are filtered to the same bit model, motor bend, stabilizer spacing, mud weight, WOB range, and RPM range, only two may remain. The comparison is highly specific but the sample is weak. A broader population provides more data with less comparability; a narrower population improves comparability while increasing statistical uncertainty. A useful benchmark balances both.
The same ROP metric changes as the population narrows from the whole well to rotary drilling in the lateral. The filters are part of the benchmark.
There is no universal number of wells that makes a benchmark reliable. It depends on:
- variability,
- effect size,
- measurement quality,
- independence,
- similarity of wells,
- question being asked.
Five extremely comparable wells can provide useful operational guidance. Five wells cannot usually prove that a modest difference between two technologies is universally real. The field papers themselves illustrate this caution. SPE/IADC-194184 compared drilling performance across 25 wells on four rigs. Even with 25 wells, the authors explicitly cautioned that the dataset remained limited for hard conclusions and that many other variables affected the observed ROP differences. That is exactly the right statistical instinct.
Do not infer causes from descriptive cohorts
A descriptive comparison may show:
| Program | Average ROP |
|---|---|
| A | 190 ft/hr |
| B | 210 ft/hr |
We can state that Program B had higher observed average ROP. We cannot yet state that the new BHA caused a 20-ft/hr improvement because other changes might include:
- crew,
- formation,
- bit,
- mud,
- directional requirement,
- operating parameters.
This distinction is fundamental:
- Descriptive benchmark: What happened?
- Causal attribution: Why did it happen?
Historical drilling datasets are usually much stronger at the first than the second. In this hypothetical comparison, Rig A is faster within both formations:
| Formation | Rig A | Rig B |
|---|---|---|
| Soft | 220 ft/hr | 210 ft/hr |
| Hard | 140 ft/hr | 130 ft/hr |
The aggregate reverses when the formation mix differs:
| Rig | Soft-rock share | Hard-rock share | Weighted ROP |
|---|---|---|---|
| A | 20% | 80% | 156 ft/hr |
| B | 80% | 20% | 194 ft/hr |
The overall average makes Rig B look dramatically faster even though Rig A is faster within each formation. This is Simpson's paradox: the direction of an aggregate comparison changes when population mix is considered.
Make weighting and metric version explicit
Weighting also changes the answer. Take two stands:
| Stand | Footage | ROP | Drilling time |
|---|---|---|---|
| 1 | 100 ft | 200 ft/hr | 0.5 hr |
| 2 | 20 ft | 50 ft/hr | 0.4 hr |
The equal-weight stand average is 125 ft/hr, while 120 ft drilled in 0.9 hours gives an effective section ROP of about 133 ft/hr. Neither result is inherently wrong; they answer different questions.
- Equal-weight stand average: What does the average stand-level ROP value look like?
- Footage/time-derived section ROP: How quickly was the footage actually delivered?
Benchmark methodology should state which is being used. SPE-196020 provides an excellent example using tortuosity. The study found that Tortuosity Index values could vary materially with:
- survey interval,
- consistency of survey spacing.
Therefore two identical physical wellbores could appear different analytically if the measurement resolution differed. Before comparing a statistic, ensure it was measured the same way. Historical ROP may use one definition while the current system uses another; MSE version 1 may use surface RPM only while version 2 includes estimated motor RPM. Pooling them places different analytical definitions under one name.
A trustworthy historical benchmark should preserve the metric definition, filtering rules, and algorithm version wherever those differences affect interpretation.
Expose selection, spread, and recency
Another common practice is:
Benchmark against the top five wells.
That can be useful operationally. But it introduces selection bias. The top wells may have benefited from:
- unusually favorable geology,
- new bits,
- easier directional requirement,
- exceptionally clean hole conditions.
Selecting only the winners removes the conditions that produced average or poor outcomes. The resulting benchmark answers:
What has been achieved under the best observed combinations of circumstances?
It does not necessarily answer:
What should we reasonably expect on the current well?
Those are different uses. Two formations can share a median while having very different spread:
| Formation | P25 | Median | P75 |
|---|---|---|---|
| A | 190 ft/hr | 200 ft/hr | 210 ft/hr |
| B | 130 ft/hr | 200 ft/hr | 270 ft/hr |
Their operational predictability is completely different. A useful benchmark therefore includes:
- median,
- percentiles,
- sample count,
- perhaps spread.
A single target value hides the uncertainty around it. A benchmark of 500 stands may sound strong, but if 420 came from one well it may primarily reflect:
- one formation realization,
- one BHA,
- one crew,
- one bit run sequence.
A useful summary should report both 500 stands and 6 wells. That exposes the hierarchy of the evidence.
A field evolves as new bits, motors, RSS systems, drilling practices, and control systems are introduced. A benchmark containing wells drilled five years ago may represent historical rather than current capability. Old data is not necessarily useless; it may still show:
- formation behavior,
- failure patterns,
- mechanical limits.
But performance targets may need recency context. A benchmark should therefore consider:
Is the population technologically comparable to today's operation?
Benchmark reliability checklist
Before acting on a benchmark, verify:
| Check | Required disclosure |
|---|---|
| Metric | ROP, MSE, connection duration, or another defined quantity |
| Analytical unit | Sample, stand, run, or well |
| Population | Formation, hole section, BHA, and drilling mode |
| Independent evidence | Number of stands, runs, and wells |
| Measurement consistency | Calculation, units, filters, and survey resolution |
| Weighting | Time, footage, stand, or well |
| Dispersion | Typical range and variability |
| Selection | All qualifying wells or selected top performers |
| Recency | Material changes in bits, BHAs, or practices |
| Decision fit | Whether the benchmark answers the actual operational question |
Related technical resources
- The Average Trap: Comparing Drilling Performance With Distributions
- Practical Considerations in Offset-Well Benchmarking
- Using Percentiles and Distributions to Evaluate Drilling Performance
References
-
Behounek, M., Millican, B., Nelson, B., Wicks, M., Rintala, E., White, M., Thetford, T., Ashok, P., and Ramos, D. Change Management Challenges Deploying a Rig-Based Drilling Advisory System. SPE/IADC-194184-MS, SPE/IADC International Drilling Conference and Exhibition, The Hague, Netherlands, 2019.
-
Shahri, M., James, M., Vasicek, A., De Napoli, R., White, M., Behounek, M., D'Angelo, J., Ashok, P., and van Oort, E. Case Studies: Optimizing BHA Performance by Leveraging Data and Advanced Modeling. SPE-196020-MS, SPE Annual Technical Conference and Exhibition, Calgary, Alberta, 2019.