How to access a Databricks database with sparklyr
New in Azure Databricks environment, I discover the packages SparkR
and sparklyr
.
From my notebooks with SparkR
, I manage to connect to a database :
library(SparkR)
DW <- sql("select * from mydb.sometable")
It runs well, but SparkR syntax seems too far removed from classic R syntax (according to me).
So I wanted to try sparklyr
, but I can't access this same database :
library(sparklyr)
sc <- spark_connect(method="databricks")
test <- spark_read_table(sc, "mydb.sometable")
Error : org.apache.spark.sql.AnalysisException: It is not allowed to add database prefix ...
Is something wrong ?
Thanks !!
Solution 1:
If it helps anyone, here's what I found that seems to work.
- Setting the default data base
- Read table in default data base
library(sparklyr)
library(dplyr)
sc <- spark_connect(method="databricks")
tbl_change_db(sc, "mydb")
foo <- spark_read_table(sc,"sometable")