Adyen Tech Academy: Taking Onboarding and Upskilling to The Next Level
In this blog post I am going to talk about the Log4j vulnerability and how Adyen responded; I will share some of the actions, mitigating factors and best coding practices from our Security team and experts in the domain.
Reported to the Apache Foundation by the Alibaba Cloud Security team, the Log4j vulnerability has initiated a global race to understand exposure, apply a fix as soon as possible, and evaluate impact across the wide number of systems and software applications that somehow rely (or include) the library.
The vulnerability, being so severe and ubiquitous, has been given its own name (Log4Shell), something that happened in the past for other notorious security threats (https://shellsharks.com/designer-vulnerabilities).
A malicious attacker can exploit a Log4j security flaw triggering Remote Code Execution (RCE), extracting sensitive data or causing a denial of service, typically providing certain input strings that are eventually logged by the application using Log4j.
Leveraging on JNDI capabilities a remote class can be injected into the server process and executed with a very undesirable outcome.
Java is a pillar of our core platform and several services are running on top of it - therefore we have taken the incident very seriously. People, processes and tools are our best assets and they have again delivered: Let me share some insights of what happened and what we have done.
Soon after the news started circulating, our teams were already exchanging messages, performing an initial assessment and discussing the first steps and actions. Although we have formal channels for managing security alerts, the AdyenWay is about direct, fast and effective interactions between teams (support, security, infrastructure, development, product). As such, detection and communication happened rapidly and efficiently.
We have a centralized way of understanding services, versions and dependencies. We didn’t limit it by looking at which Java application is affected, or which framework can be compromised, but we assessed the overall status of the platform and performed a full review to figure out every potential exploitable behavior and further detection techniques to be implemented.
Although Log4j and several Open Source frameworks depending on it were found in various parts of the Adyen Core platform, no breach or exploitation was recorded thanks to the mitigating factors already in place.
The mitigations were effective and included a mix of best practices and modern security measures:
Platform-wide patching and upgrade kicked off as soon as the latest Log4j version (v2.15.0 at that time) was made available after a remarkably fast and coordinated effort by the Log4j committers. Within 12 hours, from first becoming aware of the Apache Log4j vulnerability, the entire Adyen platform was upgraded and the fix was verified by Adyen’s Security team.
A second and third iteration were done when Log4j versions 2.16.0 and 2.17.0 became available. Although the mitigating factors and existing measures confirmed no exploitation would be possible we patched all our platform and supporting services diligently. We will keep a close look and monitor for further developments as new vulnerabilities or attack vectors could be identified.
Our merchants and partners are important to us and they are always our first priority: we informed relevant stakeholders about the situation and the activities undertaken to secure the platform and every integration.
"Communication can be informal but effective: there is at least one person coordinating actions, sometimes more depending on the domain or environment. We follow light but clear processes, and involve the right people at the right time."
Find below the timeline and main events of 10 December:
Engineering best practices are a form of mitigation in cases like this, so I would like to share tips and practices that are important to consider:
Make sure that you maintain your dependencies to avoid vulnerabilities, deprecation, and etc. Tools never solve all your problems, and dependency management is a complex topic in large codebases, but dependency scanners are easy to deploy in your CI/CD pipelines and there are amazing Open Source options that could help engineering teams to step up their visibility.
Make sure that your software can run in modern and maintained tech stacks. If you are using Log4j version 1 or running in Java SE 6, you will likely have a hard time trying to maintain your application and dependencies.
Avoid storing sensitive data (secret keys for example) in environment variables: there is a good possibility data gets exposed by an application crash or some unwanted logging (printenv), and definitely critical if the server is compromised. Integrate strong distributed secret stores instead.
Log only what is strictly necessary: capture user input if it brings value, keeping also in mind that confidential data might end up in the logging streams.
Revise the logging format: a popular pattern layout is `%-5p [%t]: %m%n` which looks harmless until you decide the user can somehow name the Java thread.
A final note to say THANK YOU to the Log4j contributors and all other developers, researchers and communities involved in addressing the security flaw and releasing a new and safer version of one of the most popular Java libraries.
At Adyen, we firmly believe in Open Source (we already sponsor and contribute to projects) and we plan to renew our commitment by open-sourcing more of our great software and supporting projects and committers who dedicate time and passion to those projects.
Stay safe and have a happy end of the year.
By submitting this form, you acknowledge that you have reviewed the terms of our Privacy Statement and consent to the use of data in accordance therewith.