query job failed due to: An error occurred while calling o78034.count. : 2org.apache.spark.SparkException: Could not execute broadcast in 300 secs. 3You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or 4disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1 at 5org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast 6(BroadcastExchangeExec.scala:205) at .
“This is due to a limitation with Spark’s size estimator.
If the estimated size of one of the DataFrames is less than the
autoBroadcastJoinThreshold, Spark may use
BroadcastHashJoin to perform the join. If the available nodes do not have enough resources to accommodate the broadcast DataFrame, your job fails due to an out of memory error.”