@omar harb There is no separate workers when spark runs on YARN, it will run normal as a normal app within YARN in the form of YARN containers. please refer below doc for spark on yarn architecture. https://spark-summit.org/2014/wp-content/uploads/2014/07/Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf http://spark.apache.org/docs/latest/running-on-yarn.html ... View more

Can you please share your Ambari & HDP version? ... View more

Hi @Roberto Sancho Can you test the performance after disabling vectorization and running it on tez? set hive.vectorized.execution.enabled=false; set hive.vectorized.execution.reduce.enabled=false; set hive.execution.engine=tez; Also, please refer below doc for hive orc optimization. https://streever.atlassian.net/wiki/display/HADOOP/Optimizing+ORC+Files+for+Query+Performance ... View more

@Roberto Sancho Great.!! :), please accept the answer which helped you to close this thread. ... View more

@hari kiran You have lots of transmitted packet errors which will definitely cause performance degrade. It might be occurring due to many issues like faulty NIC, faulty cable,RJ5 connector, duplex setting or some other network layer things. Please ask your OS team to rectify this issue and see if you will get improvement in hdfs writes. Meantime can you share the duplex setting? ethtool eth0 ... View more

@Sri Bandaru Below should solve this error after adding in below path. Ambari -> HDFS -> Config -> Core Site. hadoop.proxyuser.HTTP.groups = * ... View more

@Roberto Sancho Did you added any external jar recently inside cluster or specifically in hive?. ... View more