From Relay Test Set to Evidence Pack: Building an Audit-Ready Trail

The injection is the visible part of a relay test. The evidence is the part that survives the outage window, a handover, a maintenance interval, and an audit. A report that says “pass” without showing what was tested, against which settings, with what equipment, under what tolerance, is difficult to defend even if the relay really performed correctly.

The minimum chain

An evidence pack should let a reviewer answer six questions without interviewing the technician:

1. What asset was tested? Include station, bay, relay make/model, serial, firmware, hardware revision, and affected scheme.
2. What configuration was tested? Identify the approved, issued, installed, as-found, and as-left settings versions, including setting group and checksum where available.
3. What procedure and scope were used? Record the test method, cases, acceptance criteria, tolerance basis, exclusions, and reason for any deviation.
4. What equipment made the measurement? Retain test-set identity, software version, calibration status, connection checks, and time source where relevant.
5. What happened at each point? Keep expected value, measured value, unit, tolerance, verdict, waveform or trace reference, and operator comment.
6. Who reviewed the result? Keep edits, exceptions, approval, timestamp, and escalation path.

IEEE C37.233 and CIGRE Technical Brochure 637 provide useful industry anchors for protection testing and acceptance. They do not prescribe one database schema, but they make the reason for a complete record clear: the test method and evidence have to be repeatable and fit for the protection function.

As-found and as-left are different evidence

The as-found record tells you what condition the relay was in when the maintenance or commissioning activity began. The as-left record tells you what condition the engineer accepted after changes. Combining them into one “final” record loses information about drift, wrong settings, wiring problems, failed components, and the work required to return the scheme to service.

At minimum, keep:

• a read-only copy of the settings as found;
• the reason for any approved change;
• the engineer who authorized the change;
• a re-test of affected functions;
• the as-left export and checksum;
• the relationship between the two results.

This is especially important when a test passes only after a correction. A final pass can be correct and still hide a maintenance signal if the initial failure disappears from the record.

Raw evidence is not decoration

A rendered PDF is useful for handover, but it should point to the underlying data. Keep the raw COMTRADE record, test-set file, packet capture, settings export, and calculation inputs where the procedure requires them. Use stable identifiers rather than filenames alone; filenames change when folders are reorganized.

For each test point, make the relationship explicit:

asset -> settings version -> procedure -> test case -> expected value
      -> measured value -> tolerance basis -> raw file -> verdict
      -> deviation -> reviewer decision -> work order or close-out

That graph is more valuable than a long narrative because it lets a later engineer trace a number in a report back to a measurement and forward to the decision that followed it.

The frequent failure modes

The same weaknesses recur across relay-testing programmes:

• a copied test plan retains the previous relay’s identifier;
• the report uses a friendly asset name but not the serial or firmware;
• the expected time is printed without the equation, curve family, or tolerance basis;
• the test set’s calibration date is kept in a separate spreadsheet;
• the raw trace is overwritten when a re-test is performed;
• a failed point is replaced by a corrected point with no as-found record;
• GOOSE or binary evidence is summarized without the configuration version;
• an AI-generated explanation is stored without the source context or reviewer edit.

These are data-governance issues, not simply technician mistakes. A good system makes the correct record easier to produce and makes missing evidence visible before the report is issued.

How agentic AI can help without owning the record

An evidence-first agent can perform useful clerical and review work:

• compare the test plan asset identity with the settings export and work order;
• list required evidence that is absent or stale;
• find all tests that used a superseded settings revision;
• group deviations by relay function, station, or interval;
• draft a report section from cited raw measurements;
• generate a review queue for as-found failures, untested cases, and unresolved comments.

The assistant should never fill an absent value with a guess. It should label the gap, show the source records it did find, and abstain from a conclusion when the missing evidence matters. The reviewer must be able to inspect the original record and change or reject the draft.

PRC-005 is a boundary, not a product badge

NERC PRC-005-6 applies to entities and facilities within its scope and concerns maintenance programmes for protection systems, automatic reclosing, and sudden-pressure relaying. It is not a certification that a software product can grant. Other jurisdictions and utilities may use different rules, but the same principle applies: the programme owner must define the maintenance activity, interval, evidence, and responsible personnel.

The GridAPM PRC-005 evidence checklist is a starting point for organizing a record. It should be adapted to the actual procedure, jurisdiction, and asset scope. A ProtectionAI report can support a governed workflow; it cannot make an incomplete test compliant by changing its label.

Where ProtectionAI fits

ProtectionAI can be useful as a local workbench for organizing settings, test plans, simulator results, manuals, and report drafts. The product’s agentic assistant should be assessed on citation coverage, missing-evidence detection, reviewer effort, and traceability. Physical test evidence remains tied to the qualified instrument, procedure, and field activity.

For a pilot, select a small set of real historical records and ask the workbench to rebuild the evidence graph. Measure how many identifiers, settings revisions, raw files, and reviewer decisions are linked without manual searching. Then have an engineer verify every link. That is a more credible demonstration than a polished PDF produced from synthetic values.

References

• CIGRE Working Group B5.45. (2015). Acceptance, commissioning and field testing techniques for protection and automation systems (Technical Brochure No. 637). https://www.e-cigre.org/publications/detail/637-acceptance-commissioning-and-field-testing-techniques-for-protection-and-automation-systems.html
• Institute of Electrical and Electronics Engineers. (2023). IEEE guide for power system protection testing (IEEE Std C37.233-2023). https://standards.ieee.org/ieee/C37.233/6676/
• North American Electric Reliability Corporation. (2025). Protection system, automatic reclosing, and sudden pressure relaying maintenance (PRC-005-6). https://prod.nerc.com/standards/reliability-standards/prc/prc-005-6
• U.S. Department of the Interior, Bureau of Reclamation. (2011). Operation, maintenance, and field test procedures for protective relays and associated circuits (FIST 3-8). https://usbr.gov/power/data/fist/fist3_8/vol3-8.pdf
• U.S. Department of the Interior, Bureau of Reclamation. (2022). Management of protective relay settings (FIST 6-4). https://www.usbr.gov/power/data/fist/FIST_6-4_%281-2022%29.pdf
• Tabassi, E. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). https://doi.org/10.6028/NIST.AI.100-1