How to open spark web ui while running pyspark code in pycharm?
I am running pyspark program in pycharm local on windows 10 machine . I want to open spark web ui to monitor job and understand metrics showed over spark web ui . While running same code on jupyter I can access web ui but when I run program using pycharm I do not see option to access spark web ui , even I tried url local:4040 or localhost:4041 but , this did not work .
You can configure which port that ui can access in sparksession.
Example:
spark = SparkSession.builder.master("local").\
appName("Word Count").\
config("spark.driver.bindAddress","localhost").\
config("spark.ui.port","4050").\
getOrCreate()
Now once the spark session initialized then we can access spark ui in http://localhost:4050/jobs/
until the pycharm job finished executing
.
In case if the port is not able to access at the time then in your log spark prints where does we can access the spark ui.
Sample log:
2020-08-17 14:39:37 WARN Utils:66 - Service 'SparkUI' could not bind on port 4050. Attempting port 4051.