New Criteria for Evaluating AI in Modern Cybersecurity

Public benchmarks for evaluating AI in cybersecurity have several systemic shortcomings. First, they often reflect the research priorities of their creators rather than the real operational needs of defenders. Another issue is saturation: when leading models begin approaching 100% performance in tests, those metrics lose their ability to highlight meaningful differences in capability and become little more than a formal threshold. In addition, there is a contamination risk when test data enters the broader ecosystem and models are optimized to pass specific evaluations instead of solving real security problems. Most importantly, these tests do not assess whether AI can detect adversary behavior in real-world telemetry.

Once an adversary gains access to the network, the defensive challenge becomes significantly more complex. Security teams must work through massive alert volumes, trying to separate real signals from noise. Response speed at this stage determines whether the threat is contained within minutes or allowed to persist over an extended period. Incident investigations require teams to reconstruct activity across endpoints, cloud environments, and Identity Security systems. The more time analysts spend on investigation, the more opportunity the adversary has for lateral movement, privilege escalation, and mission completion. That is why AI effectiveness should be measured by its ability to automate and accelerate these exact processes.

Detection engineering is foundational to deep investigations because it transforms knowledge of adversary behavior into reliable detection logic. At the same time, resilient defense requires proactive threat hunting to uncover adversaries operating below the visibility threshold of standard systems. Success in both areas depends on skilled analysts and accurate models of real attacker behavior—elements that public benchmarks largely fail to capture. Neither discipline produces simple binary outcomes, yet both are critical to infrastructure resilience. AI’s ability to reproduce adversary tactics with high fidelity makes it possible to validate detection coverage and expose weak points before attackers can exploit them.

Proactive defense depends on a clear understanding of adversary tooling and the sequence of their actions across the kill chain. AI trained on hypothetical scenarios or synthetic data has extremely limited practical value because it does not reflect the artifacts of real intrusions. By analyzing trillions of events every day across its global customer base, CrowdStrike builds its threat intelligence on real incidents. Advanced labs may develop powerful models, but without access to operational data from thousands of enterprise environments, those models remain largely theoretical. In addition, the right evaluation framework must adapt to the specific profile of each organization: a financial institution and a healthcare provider face different threat models, and no single standardized test can capture those differences.

AI for vulnerability discovery will continue evolving, but the defining question for the business remains whether systems can detect and stop adversaries after they establish a foothold in the network. Ultimately, true AI effectiveness should be measured exclusively through real telemetry, operational threat intelligence, and the ability of security tools to adapt to the realities of a specific infrastructure.

As a specialized Value Added Distributor (VAD) for cybersecurity and IT infrastructure solutions, iIT Distribution provides access to advanced CrowdStrike technologies. The expert iITD team delivers end-to-end project support, from comprehensive assessments and architecture design to the implementation of innovative protection systems. With deep understanding of enterprise environments and strong technical expertise, iIT Distribution helps partners and customers build a resilient security posture capable of withstanding today’s most sophisticated threats.