Catching card testing at scale with real-time features

In card testing attacks, fraudsters run automated scripts to validate card credentials. These attacks generally take one of two forms: verification, where bots iterate through lists of stolen credentials using low-amount transactions to identify active cards; and enumeration, where scripts systematically guess missing fields, such as CVVs or expiry dates, within specific BIN ranges.

This testing phase is a critical entry point for large-scale fraud across the financial ecosystem. Once a card is validated, it is resold or used for high-value purchases. Detecting this early is key: beyond the technical burden on merchants, successful card testing degrades authorization health, damages issuer relationships, and introduces verified, high-risk credentials into the broader payments industry. These cards often reappear as isolated purchases that lack the bulk scripted signature of the original attack, making them significantly harder to detect.

More obvious attacks reuse shopper identifiers across payments. Think of repeated usage of the same IP or (fake) email address. Larger bursts in velocity on these identifiers can be monitored. However, catching scripted fraud is becoming increasingly difficult because modern tactics bypass traditional entity-level tracking:

• Rotational attacks: Fraudsters continuously rotate details—such as IP addresses—to stay well below baseline velocity thresholds.

• Low-volume attacks: Fraudsters throttle their scripts, keeping transaction bursts too small to trigger rules at the individual merchant level.

Even highly capable ML models are constrained by their feature space. When features are limited to normalized velocity counts, the only way to catch these subtle attacks is to lower detection thresholds, which risks misclassifying genuine shoppers and driving up false positives. To break this limitation and protect legitimate traffic, we needed to enrich our feature space by combining entity tracking with traffic health in real time.

A fundamental strength of the system is the shift in feature engineering away from fine-grained shopper identifiers toward higher-level aggregated traffic dimensions, which we call subslices.

When fraudsters spread scripts across thousands of proxy IPs, tracking individual shopper identifiers can fail because the attack fragments into isolated events. Instead, we look at what the traffic is targeting. By grouping transactions across shared dimensions, such as a BIN range at the merchant level, we can aggregate distributed attempts into meaningful subslices where broader patterns become visible.

The key is to monitor the health and velocity of these subslices in real time. Card testing attacks can remain small at the individual merchant or shopper level while still creating measurable changes in the behavior of a particular subslice. By combining signals such as transaction volume, velocity, and payment health with historical baselines, the model can identify unusual patterns as they emerge.

These features need to be calculated continuously because timing is critical. The goal is not simply to identify that an attack happened; it is to detect and block the attack while the transactions are still coming in. Real-time subslice features give the Bot Attack model the context it needs to recognize distributed and low-volume attacks without relying on increasingly aggressive thresholds at the individual-entity level.

This approach makes it harder for fraudsters to evade detection simply by rotating identifiers or throttling their attacks. Rather than looking for a single obvious signal, the model can identify changes in the health and behavior of the traffic across multiple subslices, allowing us to intervene before card testing can lead to downstream fraud.

The biggest challenge in training models for card testing is the lack of explicit, reliable labels. Unlike chargeback fraud, where victims claim their losses with their bank (ground-truth feedback), card testing offers no explicit transaction-level signal. 

While issuers provide real-time decline codes indicative of testing, such as "invalid CVC" or "invalid credentials", these signals are not definitive proof of an attack. Declines can happen for legitimate reasons, like typos or expired cards. Additionally, scripts occasionally guess correctly, creating a small volume of approved transactions within an attack.

To overcome this lack of explicit labels, we proxy-label traffic subslices where statistical patterns confirm fraud is happening, providing clean training and evaluation data. Labeling is easier in hindsight, when knowing the complete picture of the attack. When combined with real-time subslice-health features, this training enables our model to accurately spot and block active attacks. But detecting these patterns is only half the challenge. To use them in production, we also needed to calculate the same complex subslice metrics consistently during both training and live inference.

Calculating real-time subslice health metrics presents a core engineering challenge: maintaining parity between offline training data and live production features. The model uses hundreds of features per transaction. When a feature is computed one way during training and another during live scoring, training-serving skew can degrade model performance in production.

To ensure high performance, we built a custom Feature Domain-Specific Language (DSL). Engineers define feature logic once, and the system automatically compiles it into execution pipelines for both offline training and online inference.

• Offline Feature Generation (Apache Spark): Compiles the DSL into Spark jobs that process historical payment data, computing rolling averages and baseline percentiles across billions of transactions to build point-in-time correct training sets.

• Online Streaming Inference (Apache Kafka + Apache Flink): Compiles the same DSL into Flink streaming pipelines that consume Kafka events, evaluating sliding time windows to update subslice health metrics live as attacks unfold.

Once calculated, Flink writes these streaming metrics to an Apache Cassandra feature store, where the risk engine fetches the latest subslice values to make real-time block/allow decisions before sending transactions to the issuer.

This unified flow eliminates training-serving skew and enables rapid feature development. With the DSL, a single engineer can test an idea on historical data in Spark, deploy the Flink pipeline, and push a live feature to production in a single PR, ensuring the system remains responsive to new fraud patterns.

While our custom DSL makes it incredibly fast to ship new real-time features, deploying these into production comes with risks. Upstream data changes, streaming delays, or shifting fraud distributions can cause features to drift, reducing model performance over time. To keep the system reliable, we monitor the pipeline across four main layers:

• Data Source Level: Confirms that incoming real-time events from Kafka match the structure and quality of historical training data.

• Feature Computation Level: Checks that live subslice features match expected offline values, catching any logic errors in the DSL.

• Model Prediction Level: Tracks live risk scores against offline distributions to catch shifts in model predictions.

• Decision Output Level: Monitors final block and allow rates to detect sudden shifts in overall payment decisions.

Even if a major incident triggers alerts across every layer at once, having this four-layer setup makes troubleshooting much faster. Instead of chasing down a vague drop in approval rates, teams get clear alerts that point straight to the root cause, so the model runs reliably with minimal manual digging.

Deploying this setup into our live Bot Attack model drove an immediate, measurable impact on fraud detection across Adyen’s platform. The system now scores millions of payments every day, increasing blocked card testing attacks by 28% on average, with zero impact on legitimate shoppers. This increased coverage captures subtle, low-volume, and proxy-rotated scripts that previously bypassed velocity thresholds, without increasing false positives. By combining real-time stream processing with subslice feature engineering, we have built a defense that can evolve alongside attacker tactics. And by stopping card testing at the source, we prevent downstream abuse and reduce overall fraud across Adyen’s global network.

Interested in joining our team?

Check out our careers page for all our open roles.

Explore open roles