How to optimize gradle build performance regarding build duration and RAM usage?

You need to give more memory to the Gradle JVM, not to the compile task/JVM. One way to do so is via the GRADLE_OPTS environment variable (GRADLE_OPTS=-Xmx512m).

If using the Gradle Wrapper you can set DEFAULT_JVM_OPTS in gradlew like this:

DEFAULT_JVM_OPTS="-Xmx512m"

Set it in a similar fashion in gradlew.bat if you're on Windows:

set DEFAULT_JVM_OPTS=-Xmx512m

The Gradle Wrapper task can also be modified to include this automatically. This is how the Gradle developers have solved it:

wrapper {
    gradleVersion = '1.8'

    def jvmOpts = "-Xmx512m"
    inputs.property("jvmOpts", jvmOpts)
    doLast {
        def optsEnvVar = "DEFAULT_JVM_OPTS"
        scriptFile.write scriptFile.text.replace("$optsEnvVar=\"\"", "$optsEnvVar=\"$jvmOpts\"")
        batchScript.write batchScript.text.replace("set $optsEnvVar=", "set $optsEnvVar=$jvmOpts")
    }
}