Customize SparkContext using sparkConf.set(..) when using spark-shell

Solution 1:

Spark 2.0+

You should be able to use SparkSession.conf.set method to set some configuration option on runtime but it is mostly limited to SQL configuration.

Spark < 2.0

You can simply stop an existing context and create a new one:

import org.apache.spark.{SparkContext, SparkConf}

sc.stop()
val conf = new SparkConf().set("spark.executor.memory", "4g")
val sc = new SparkContext(conf)

As you can read in the official documentation:

Once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.

So as you can see stopping the context it is the only applicable option once shell has been started.

You can always use configuration files or --conf argument to spark-shell to set required parameters which will be used be the default context. In case of Kryo you should take a look at:

• spark.kryo.classesToRegister
• spark.kryo.registrator

See Compression and Serialization in Spark Configuration.