How to Configure the Spark SQL Hive Warehouse Directory

How to Configure the Spark SQL Hive Warehouse Directory

Since Spark 2.0.0, Spark references "spark.sql.warehouse.dir" as the default Spark SQL Hive Warehouse location. To create databases, both the "spark-defaults.conf" and "spark-thrift-sparkconf.conf" files should have a spark.sql.warehouse.dir property that has the same value as the hive.metastore.warehouse.dir property. The default value of this property is usually /apps/hive/warehouse:

spark.sql.warehouse.dir=/apps/hive/warehouse

On an Ambari cluster, select Spark2 > Configs, then use "Add Property" in "Custom spark2-defaults" and "Custom spark2-thrift-sparkconf" to add a spark.sql.warehouse.dir property with the value /apps/hive/warehouse. you can change this value to your desired value.

Now Click on "Add Property" and add your new value. Then Click on Save.