Fighting fraud with rapid experimentation

The most effective way to block fraudsters using in-depth risk configuration testing.

Fighting payment fraud can be more challenging than you think. You work hard to ensure genuine customers have a seamless payment experience. Yet, criminals have the ability to adapt quickly and target your business using their technical knowledge.

The need for a sophisticated fraud management system is real. And getting the right balance between continuously optimizing your fraud protection without declining genuine orders (which at the time of purchase, sometimes look very suspicious) can be a real challenge.

Online businesses decline on average 2.5% of all payment attempts due to suspicion of fraud.

A 2019 report by the Merchant Risk Council[1] estimates that on average, online businesses decline 2.5% of all payment attempts due to suspicion of fraud. This number could equate to millions in lost revenue (if the declined payment was in fact not fraudulent).

So, how does a business continue to grow fast and protect customers without blocking too many genuine transactions? As fraud evolves it becomes crucial for any fraud management solution to detect, analyze, and adapt as fast as possible.

The power of A/B testing

The need for any business to continuously adjust and calibrate their fraud strategy is worth taking a closer look at. Randomly adjusting rules and scores, swapping out one model for another or integrating with additional third parties to validate customer data is an option, but usually not the most efficient route to take.

A/B testing isn’t a new concept but it is a proven methodology used to determine whether or not changes that we make to a configuration have an impact on any outcomes as measured by representative metrics.

A / B Testing illustration

A/B testing is a method used to decide if a variant impacts an outcome compared to a control.

In the risk and fraud space, it’s vital to track the impact of any change and how this affects false negatives and false positives. False negatives are the actual fraud your system has missed and that turn into chargebacks. False positives are actually good orders that are falsely classified as fraud, resulting in missed legitimate revenue.

The goal of A/B testing is to measure the impact of a change on said metrics and make an accurate decision as to whether or not the measured impact is “real” or simply due to chance (natural variation), ultimately guiding decisions as to whether or not changes should be deployed to all transactions.

In marketing and web analytics, A/B testing has found early adoption, but given the complexity and depth of available data in fraud management, A/B testing should be regarded as fundamental. A/B testing is a controlled experiment with two variants, A and B and is a way to compare two versions of a single variable typically by testing a subject's response to variable A against variable B, and determining which of the two variables is more effective.

Our approach to experimentation

Adyen’s team of global risk experts have always been able to use this experimentation framework internally, but we are excited to now make this feature available to our merchants.

Recent advances in our machine learning (ML) efforts really required a strong experimentation platform to monitor and show the associated impacts to users in real-time, effectively putting merchants in full control.

Any small change to a given set of risk settings can be split tested rather than simply replacing an existing configuration with a new one. Adyen runs all given configurations (we refer to the combination of a given set of risk settings as a profile) through its ML powered Risk Optimizer, effectively creating a hybrid solution that takes the best of both worlds - the transparency and control of a rule-based approach with the scalable and accurate predictions of machine learning.

The initial versions we tested with pilot merchants focused on key metrics like authorization and fraud rates, but we quickly learned that detailed insights needed to become a core component as well.

Supported by raw numbers, but also real-time graphs, users can drill down into the details of, for example, the exact decline reasons of the card-issuing banks to assess the impact of the applied changes.

Not only do we want to encourage continuous testing of new risk configurations we also believe strongly that successful fraud management is about making solid data-driven decisions as opposed to leaving this to opinions. Adyen’s experimentation feature is a part of this approach and will assist merchants in staying ahead of the curve.

Vimeo video will be rendered here

Now that you know the capabilities of our Experiments feature, let’s take a look at a real case study.

Experimentation put into practice: OLX’s success story

With a high volume of transactions, OLX Brazil is one of the largest marketplaces in the world. The company has always seen payment risk management as a challenge.

Together, OLX and Adyen developed a strategy based on machine learning technology and intelligent data usage. By analyzing the payment information base, the algorithm understood consumer behavior better and provided optimized insights across conversion, transaction approval, and how to get closer to the ‘best customer experience’.

With the newly learned knowledge, the next step was to review the risk profile used in OLX's anti-fraud payment protection tool. This profile consists of different data fields, such as location, email, average ticket, card data, shopping cart products, user history, and more. Each receives a score according to the risk of fraud it represents. This was profile ‘A’.

“After 8 weeks, the entire volume was migrated to the new risk profile and the end result was a 2.6% increase in the authorization rate, with maintenance of chargeback levels.”

Ligia Pires Trust and Safety Manager of OLX Brazil

A second payment risk profile, profile ‘B’, was then created by bringing together a new set of rules. The machine learning calibrated the scores of each profile according to the data generated by the company's sales history. "Since we had enough data, a powerful tool and customers always open to testing new solutions, we started a test in an A / B format," says Ligia Pires, Trust and Safety Manager of OLX Brazil. "The test consists of running the old and new anti-fraud profile in parallel, checking which one has the best results."

The first phase of the testing period lasted four weeks when only 10% of sales volume was sent to the new anti-fraud payment profile. When presenting good results, the volume increased to 25% for a further two weeks, and then to 50% in the last two weeks.

"Today, Adyen supports us in our payment initiatives and its technology allows us to have the best of both worlds: approved valid financial transactions and a robust anti-fraud system blocking illegitimate purchases," concludes Pires.

What experimentation can do for you

Adyen’s RevenueProtect risk product will have the new experiment feature as a premium risk add on once beta testing is complete. If you're interested in joining the pilot program to see what it can do, then get in touch.

For now, check out how the power of Adyen’s end-to-end payment system gives you more data in one place and puts you in control to test, learn, and optimize with experiments.


Fine-tune your risk management

Explore how to maintain the balance between blocking fraudsters and letting legitimate customers pay unhindered.

Go to guide

Sign up for the newsletter

By submitting this form, you acknowledge that you have reviewed the terms of our Privacy Statement and consent to the use of data in accordance therewith.

Are you looking for test card numbers?

Would you like to contact support?