PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research
Abstract
Artificial intelligence is increasingly integrated into professional cybersecurity workflows, yet the cognitive effects of AI assistance on researcher behavior remain poorly understood. This paper introduces the PTRR Framework, a structured metacognitive instrument designed to measure and mitigate Automation Bias in AI-assisted vulnerability research. PTRR comprises four components: Prompts (interaction quality scoring), Time (temporal behavioral analysis), Results (outcome-based severity scoring), and Rubric (process independence and verification criteria). Three derived indices operationalize the core constructs: the Automation Bias Index (ABI), which quantifies uncritical reliance on AI output; the Cognitive Struggle Index (CSI), which measures productive independent effort; and the Tool Integration Intensity Score (TIIS), which captures deliberate multi-tool synthesis. The framework is grounded in Dual Process Theory and Cognitive Load Theory, which together predict that expertise moderates the relationship between AI interaction quality and research output. A preliminary single-participant case study conducted over 60 structured hours provides initial construct validity evidence. The case study documented a shift from a low-to-medium severity finding profile to a critical-dominant profile following PTRR-based workflow adoption, including seven critical-severity findings validated by independent program triage and three multi-layer escalation chains. Total accepted findings increased from 2 to 11 over equivalent time periods. These results motivate the formal multi-participant study design proposed here as future work, which employs a within-subject Phase A versus Phase B design, Hierarchical Linear Modeling, and independent human scoring with pre-defined inter-rater reliability thresholds.
Citation Information
@article{ziadsalah2026,
title={PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research},
author={Ziad Salah and Ashraf A. Mohamed},
journal={Scientific Reports},
year={2026},
doi={https://doi.org/10.21203/rs.3.rs-9247251/v1}
}
SinoXiv