Mask sensitive data in logs with logback
I need to be able to search an event for any one of a number of patterns and replace the text in the pattern with a masked value. This is a feature in our application intended to prevent sensitive information falling into the logs. As the information can be from a large variety of sources, it is not practical to apply filters on all the inputs. Besides there are uses for toString() beyond logging and I don't want toString() to uniformly mask for all calls (only logging).
I have tried using the %replace method in logback.xml:
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'f k\="pin">(.*?)</f','f k\="pin">**********</f'}%n</pattern>
This was successful (after replacing the angle brackets with character entities), but it can only replace a single pattern. I would also like to perform the equivalent of
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'pin=(.*?),','pin=**********,'}%n</pattern>
at the same time, but cannot. There is no way to mask two patterns in the one %replace.
The other way that has been loosely discussed on the interblags is extending something on the appender/encoder/layout hierarchy, but every attempt to intercept the ILoggingEvent has resulted in a collapse of the whole system, usually through instantiation errors or UnsupportedOperationException.
For example, I tried extending PatternLayout:
@Component("maskingPatternLayout")
public class MaskingPatternLayout extends PatternLayout {
@Autowired
private Environment env;
@Override
public String doLayout(ILoggingEvent event) {
String message=super.doLayout(event);
String patternsProperty = env.getProperty("bowdleriser.patterns");
if( patternsProperty != null ) {
String[] patterns = patternsProperty.split("|");
for (int i = 0; i < patterns.length; i++ ) {
Pattern pattern = Pattern.compile(patterns[i]);
Matcher matcher = pattern.matcher(event.getMessage());
matcher.replaceAll("*");
}
} else {
System.out.println("Bowdleriser not cleaning! Naughty strings are getting through!");
}
return message;
}
}
and then adjusting the logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.touchcorp.touchpoint.utils.MaskingPatternLayout">
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/touchpoint.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>logs/touchpoint.%i.log.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>3</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<encoder>
<layout class="com.touchcorp.touchpoint.utils.MaskingPatternLayout">
<pattern>%date{YYYY-MM-dd HH:mm:ss} %level [%thread] %logger{10} [%file:%line] %msg%n</pattern>
</layout>
</encoder>
</appender>
<logger name="com.touchcorp.touchpoint" level="DEBUG" />
<logger name="org.springframework.web.servlet.mvc" level="TRACE" />
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="STDOUT" />
</root>
</configuration>
I have tried many other insertions, so I was wondering if anyone has actually achieved what I am attempting and if they could provide any clues or a solution.
Solution 1:
You need to wrap layout using LayoutWrappingEncoder. And also I believe you cannot use spring here as logback is not managed by spring.
Here is the updated class.
public class MaskingPatternLayout extends PatternLayout {
private String patternsProperty;
public String getPatternsProperty() {
return patternsProperty;
}
public void setPatternsProperty(String patternsProperty) {
this.patternsProperty = patternsProperty;
}
@Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
if (patternsProperty != null) {
String[] patterns = patternsProperty.split("\\|");
for (int i = 0; i < patterns.length; i++) {
Pattern pattern = Pattern.compile(patterns[i]);
Matcher matcher = pattern.matcher(event.getMessage());
if (matcher.find()) {
message = matcher.replaceAll("*");
}
}
} else {
}
return message;
}
}
And sample logback.xml
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<file>c:/logs/kp-ws.log</file>
<append>true</append>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.kp.MaskingPatternLayout">
<patternsProperty>.*password.*|.*karthik.*</patternsProperty>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="fileAppender1" />
</root>
UPDATE
Here its better approach, set Pattern during init itself. such that we can avoid recreating Pattern again and again and this implementation is close to realistic usecase.
public class MaskingPatternLayout extends PatternLayout {
private String patternsProperty;
private Optional<Pattern> pattern;
public String getPatternsProperty() {
return patternsProperty;
}
public void setPatternsProperty(String patternsProperty) {
this.patternsProperty = patternsProperty;
if (this.patternsProperty != null) {
this.pattern = Optional.of(Pattern.compile(patternsProperty, Pattern.MULTILINE));
} else {
this.pattern = Optional.empty();
}
}
@Override
public String doLayout(ILoggingEvent event) {
final StringBuilder message = new StringBuilder(super.doLayout(event));
if (pattern.isPresent()) {
Matcher matcher = pattern.get().matcher(message);
while (matcher.find()) {
int group = 1;
while (group <= matcher.groupCount()) {
if (matcher.group(group) != null) {
for (int i = matcher.start(group); i < matcher.end(group); i++) {
message.setCharAt(i, '*');
}
}
group++;
}
}
}
return message.toString();
}
}
And the updated Configuration file.
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<file>c:/logs/kp-ws.log</file>
<append>true</append>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.kp.MaskingPatternLayout">
<patternsProperty>(password)|(karthik)</patternsProperty>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="fileAppender1" />
</root>
Output
My username=test and password=*******
Solution 2:
From the documentation:
replace(p){r, t}
The pattern p can be arbitrarily complex and in particular can contain multiple conversion keywords.
Facing same problem having to replace 2 patterns in a message, I just tried to chain so p is just an invocation of replace, in my case:
%replace( %replace(%msg){'regex1', 'replacement1'} ){'regex2', 'replacement2'}
Worked great, though I wonder if I'm pushing it a bit and p can be indeed that arbitrarily complex.