Fluency is not evidence of engineering reliability. An AI drilling assistant should be evaluated claim by claim: correct entity and context, correct source, reproducible calculation, calibrated inference, appropriate abstention, and consequence-aware recommendations. The test set should target failure modes that can produce confident but operationally dangerous answers.
Decompose the answer into verifiable claims
Modern language models are very good at producing:
- grammatically correct,
- technically familiar,
- well-structured
answers. That creates an unusual risk. A badly written engineering answer is easy to question. A wrong answer that sounds technically sophisticated may receive less scrutiny. For example:
Increasing MSE indicates worsening stick-slip.
That sentence sounds reasonable. But MSE can increase because of:
- harder formation,
- lower ROP,
- poor hole cleaning,
- bit degradation,
- several dysfunctions.
The problem is not language quality. The problem is unsupported diagnostic specificity. A reliability framework therefore cannot score answers primarily on:
- writing quality,
- terminology,
- similarity to a reference paragraph.
It has to evaluate the underlying engineering claims. Consider:
Current SPP is 4,920 psi, approximately 250 psi below the hydraulic-model prediction. Flow has remained near 650 gpm. The evidence supports a hydraulic abnormality, although the available data is insufficient to distinguish pump degradation from a drillstring washout.
That answer contains claim types that require different validation:
| Claim | Type | Primary check |
|---|---|---|
| Current SPP is 4,920 psi | Measurement | Read the correct live channel and timestamp. |
| SPP is about 250 psi below model | Calculation | Reproduce the residual from the disclosed inputs and model version. |
| Flow has remained near 650 gpm | Measurement and trend | Verify the channel, interval, and aggregation. |
| Evidence supports a hydraulic abnormality | Inference | Review whether the combined evidence justifies the event family. |
| Data cannot distinguish pump degradation from washout | Uncertainty | Confirm that discriminating evidence is unavailable or inconclusive. |
A useful AI evaluation framework therefore begins by decomposing the answer into typed claims.
The answer contains measured intervals, calculated changes, operating-state context, and an interpretation. Each claim can be checked independently.
A sidetracked well illustrates a wrong-context truth:
| Wellbore | Current recorded depth |
|---|---|
| Original bore | 14,850 ft MD |
| Sidetrack | 14,220 ft MD |
The engineer asks:
What is current bit depth on the sidetrack?
The AI answers:
14,850 ft.
That number may exist perfectly legitimately in the database. It is simply associated with the wrong wellbore. This is not a classical hallucination. It is an entity-resolution error. The answer looks particularly trustworthy because the value is real. A reliability test therefore needs to detect: wrong-context truth.
A value can be real and still be wrong when it comes from the wrong wellbore.
Test calculations and provenance separately
Now ask:
What is the modeled ECD under the current conditions?
If the system uses a deterministic hydraulic calculation, evaluation should verify:
- inputs,
- units,
- formula/model,
- output.
The LLM should not receive credit merely because its answer falls near an expected range. The strongest evaluation is:
Given identical inputs and model version, can the result be reproduced?
That creates an objective standard: evaluation can compare the AI-disclosed output against the result produced by the same tool, inputs, and model version. This is fundamentally different from evaluating free-form reasoning. If modeled ECD is 12.4 ppg, evaluation must also ask whether the model used:
- current mud rheology,
- current hole size,
- current flow,
- current BHA geometry?
If rheology was last updated six hours ago, the numerical calculation may be correct given the inputs while the operational answer remains uncertain. This leads to another reliability dimension: input validity. A correct calculation using stale inputs is not automatically a reliable current-state answer. Consider:
What did the driller report about the tight hole?
A useful answer might say:
The DDR describes tight hole while pulling through approximately 12,600–12,800 ft.
Evaluation should determine:
- Did that statement actually appear?
- Which report?
- Which well?
- Which date?
- Was the interval quoted or inferred?
The system should not receive full credit for producing the right general interpretation if its cited source does not support the claim. This makes provenance correctness a first-class evaluation target.
Score inference at the supported specificity
A typical inferred claim is:
The torque increase is likely related to worsening hole condition.
No database field may contain cause = poor hole condition. Evaluation should ask whether the evidence was stated correctly:
- torque increased,
- overpull increased,
- circulation had been limited.
and whether important competing explanations were considered:
- formation change,
- bit condition,
- BHA behavior.
The conclusion must also be proportional to the evidence. Compare:
Poor hole cleaning caused the torque increase.
with:
Worsening hole condition is one plausible explanation for the torque increase.
The second may be justified when the first is not. An assistant that correctly recognizes a hydraulic abnormality but labels it a washout has identified the domain and event family while overreaching on root cause. Evaluation should preserve those levels:
- Level 1: Did the system recognize a meaningful change?
- Level 2: Did it identify the correct engineering domain: hydraulic, mechanical, directional, or another?
- Level 3: Did it identify the correct event family?
- Level 4: Did it identify the specific root cause?
A useful test distinguishes broadly correct from specifically correct, which is more informative than binary pass/fail scoring.
Test abstention, data quality, and recommendations
A reliable AI drilling assistant should not answer every question with equal confidence. Testing only well-formed questions with complete data is insufficient. The evaluation set should deliberately include cases where:
- a required sensor is missing,
- two sources conflict,
- wellbore identity is ambiguous,
- relevant context is stale,
- no comparable offset exists.
Abstention testing asks whether the system recognizes when the requested conclusion is unsupported. An assistant that answers every difficult question with “insufficient evidence” will rarely hallucinate, but it will also be nearly useless. Reliability requires both correctness and appropriate uncertainty, not a high refusal rate. A useful system should answer when evidence is adequate, qualify when it is partial, and abstain when it is genuinely insufficient.
An assistant retrieves Flow = 320 gpm, although the pump is staged near 650 gpm and the flow channel has failed. It should not receive credit for presenting the database value as trustworthy actual flow. A field example in SPE-181076 showed measured pump output dropping to roughly half its expected value while:
- flow-out,
- standpipe pressure
remained unchanged. The broader evidence supported a sensor/data problem rather than a genuine reduction in circulation. An AI assistant operating above that data layer must inherit those quality distinctions. Now compare:
Increase RPM to 140.
That is different from saying:
Higher RPM has historically performed better in comparable intervals.
The first is a recommendation. It can affect the operation. Recommendation evaluation therefore needs additional checks. Is 140 RPM:
- within equipment limits?
- compatible with the BHA?
- acceptable given vibration behavior?
- consistent with current formation?
A recommendation should pass through operational constraints. Even if the assistant estimates:
Increasing RPM could improve ROP.
That does not automatically mean:
Increase RPM.
A decision also depends on:
- downside risk,
- dysfunction,
- wellbore quality,
- operational limits.
Evaluation should therefore distinguish an expected effect from a recommended action. The recommendation requires a broader evidence set.
Build tests around consequential failures
A useful test suite should contain more than random engineering questions. Include deliberately difficult cases.
- Wrong-well trap: Another well contains a more semantically similar report.
- Sidetrack trap: The same MD exists in two wellbores.
- Stale-data trap: A recent-looking value is no longer current.
- Planned-versus-actual trap: The plan differs from field execution.
- Sensor-fault trap: A plausible numerical value is quality flagged.
- Unit trap: One source uses different units.
- Missing-input trap: A calculation lacks required context.
- Ambiguous-cause trap: Evidence supports an event family but not a root cause.
- Conflicting-source trap: Two reports disagree.
These are far more valuable than testing only clean textbook cases. For every important claim, record:
| Claim | Type | Expected Source | Correct? | Source Correct? | Context Correct? | Certainty Appropriate? |
|---|---|---|---|---|---|---|
| SPP = 4,920 psi | Measured | Live rig data | Yes | Yes | Yes | Yes |
| Model SPP = 5,170 psi | Calculated | Hydraulic tool | Yes | Yes | Yes | Yes |
| Washout likely | Inferred | Combined evidence | Partial | N/A | Yes | No |
| Increase flow | Recommendation | Engineering reasoning | No | N/A | N/A | N/A |
This is far more diagnostic than a simple: Answer = fail. It shows exactly which layer failed.
A calibrated answer separates the most likely explanation from weaker alternatives and states which missing evidence prevents a stronger conclusion.
Evaluate the full tool chain
An engineering assistant may contain:
- entity resolver,
- retrieval system,
- database tools,
- deterministic calculations,
- LLM reasoning.
If the final answer is wrong, which component failed? Possible causes:
- Retrieval failure: Wrong record.
- Data failure: Bad sensor.
- Tool failure: Calculation returned an error.
- Context failure: Wrong wellbore.
- Reasoning failure: Correct evidence, wrong conclusion.
The evaluation harness should preserve this distinction. Otherwise every problem is mislabeled: LLM hallucination. That diagnosis is too broad to be useful.
Related technical resources
- Why Deterministic Engineering Calculations and LLM Reasoning Should Coexist in Drilling
- When “Unable to Determine” Is the Correct Output in Drilling Analytics
- Engineering Context in Drilling RAG: Why Semantic Similarity Is Not Enough
References
- Ashok, P., Ambrus, A., Ramos, D., Lutteringer, J., Behounek, M., Yang, Y. L., Thetford, T., and Weaver, T. A Step by Step Approach to Improving Data Quality in Drilling Operations: Field Trials in North America. SPE-181076-MS, SPE Intelligent Energy International Conference and Exhibition, Aberdeen, Scotland, 2016.