Hi @adrijand Yeah, it seems some jar conflicts somewhere.  You are trying to load Hive 1.1.0  classes before the ones included with Spark, and as such, they might fail to reference a Hive configuration that didn't exist in 1.1.0. like below. : java.lang.NoSuchFieldError: METASTORE_CLIENT_SOCKET_LIFETIME at org.apache.spark.sql.hive.HiveUtils$.formatTimeVarsForHiveClient(HiveUtils.scala:195) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286) at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66) But here in the description mentioned you are using CDH 5.15v but in your log snippets it showing Apache Spark ( spark-2.3.0-bin-without-hadoop ) and Apache Hive (apache-hive-1.1.0-bin) version which is not a pre-built package version that comes along with CDH stack compatibility. Are you trying with building with varying versions of Hive which you would like to connect from a remote airflow docker container?  ... View more

Hi, @murali2425 @vchhipa It seems some dependency issue while building your custom NiFi Processor,  org.apache.nifi:nifi-standard-services-api-nar  dependency needs to be added in  pom.xml  of  nifi-*-nar  module. Ref here    <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-standard-services-api-nar</artifactId> <version> 1.11 .3 </version> <type>nar</type> </dependency>   Please modify your pom.xml and rebuild and see whether that fixes the issue. Please accept the answer you found most useful. ... View more

Hi @TimmehG,  Spark has a configurable metrics system based on the  Dropwizard Metrics Library . This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. The metrics are generated by sources embedded in the Spark codebase. They provide instrumentation for specific activities and Spark components. The metrics system is configured via a configuration file that Spark expects to be present at  $SPARK_HOME/conf/metrics.properties . A custom file location can be specified via the  spark.metrics.conf   configuration property . Instead of using the configuration file, a set of configuration parameters with prefix  spark.metrics.conf.  can be used.  I agree with you, running spark applications continuously & reliably is a challenging task, and a good performance monitoring system is needed. Several external tools can be used to help profile the performance of Spark jobs: Cluster-wide monitoring tools, such as   Ganglia, can provide insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia dashboard can quickly reveal whether a particular workload is disk-bound, network bound, or CPU bound. OS profiling tools such as   dstat,   iostat, and   iotop   can provide fine-grained profiling on individual nodes. JVM utilities such as   jstack   for providing stack traces,   jmap   for creating heap-dumps,   jstat   for reporting time-series statistics and   jconsole   for visually exploring various JVM properties are useful for those comfortable with JVM internals. For more insights you can refer to the below links: https://spark.apache.org/docs/latest/monitoring.html https://blog.cloudera.com/demystifying-spark-jobs-to-optimize-for-cost-and-performance/ https://www.infoq.com/articles/spark-application-monitoring-influxdb-grafana/ https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting   Please accept the answer you found most useful. ... View more

Can you verify one, whether did you followed all the steps listed in the documentation? https://docs.cloudera.com/runtime/7.1.1/ozone-storing-data/topics/ozone-setting-up-ozonefs.html ... View more

Hi @Ettery  Can you try to add those properties in   nifi.properties ? the   Docker configuration has been updated   to allow proxy whitelisting from the run command the host header protection is only enforced on "secured" NiFi instances. This should make it much easier for users to quickly deploy sandbox environments like you are doing in this case Even you can try with: -e NIFI_WEB_HTTP_HOST=<host>  in docker run command   docker run --name nifi -p 9090:9090 -d -e NIFI_WEB_HTTP_PORT='9090' -e NIFI_WEB_HTTP_HOST=<host> apache/nifi:latest   In GitHub   example configuration  and documentation for NiFi running behind a reverse proxy that you may be interested in.  For more detail refer stackoverflow1 and stackoverflow2 ... View more

You can try with spark-shell --conf spark.hadoop.hive.exec.max.dynamic.partitions=xxxxx.    $ spark-shell --conf spark.hadoop.hive.exec.max.dynamic.partitions=30000 Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hostname:port Spark context available as 'sc' (master = yarn, app id = application_xxxxxxxxxxxx_xxxx). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.x.x.x.x.x.x-xx /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112) Type in expressions to have them evaluated. Type :help for more information. scala> spark.sqlContext.getAllConfs.get("spark.hadoop.hive.exec.max.dynamic.partitions") res0: Option[String] = Some(30000)  Ref: SPARK-21574 ... View more

Please can you check with your internal Linux team / Network team for further support? Because it seems you have some internal connection while connecting the node from the  Intellij idea node. Once you resolve the connection issue we will check further.  ... View more

Can you add below property at <spark_home>/conf/hive-site.xml and <hive-home>/conf/hive-site.xml hive.exec.max.dynamic.partitions=2000   <name>hive.exec.max.dynamic.partitions</name> <value>2000</value> <description></description> Hope this helps. Please accept the answer and vote up if it did. Note : Restart HiveServer2 and Spark History Server if it didn't work.   -JD ... View more

Can you try this below article? https://saagie.zendesk.com/hc/en-us/articles/360021384151-Read-Write-files-from-HDFS ... View more

Hi @w12q12  So as per the below error in the log trace 20/04/21 18:20:50 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1067413441-127.0.0.1-1508775264580:blk_1073743149_2345 file=/data/ratings.csv at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:930) .... Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1067413441-127.0.0.1-1508775264580:blk_1073743149_2345 file=/data/ratings.csv at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:930) It seems that the namenode is not able to connect to the datanode when you ran the command. Please can you try to ping and telnet datanode and name node vice versa also check whether do you have any corrupt blocks and files in the cluster? ~JD ... View more

Hi @Gubbi  You can try the below script.   source_dir="/user/mft/inbound/" dest_dir="/users/aws/outbound/" for file in "$source_dir"*; do dest=$(echo $file | grep -o -P '(?<=_).*(?=.csv)'); mkdir -p "$dest_dir$dest"; mv $file "$dest_dir$dest"; done   Hope this helps. Please accept the answer and vote up if it did. ... View more

Try to clean up some old files from your disk or else add some more space into the disk.  ... View more

Hi  @Dharm,   What do you mean by bad health space here? Is HDFS Full and getting critical alerts either in Ambari or Cloudera Manager? Are you able to read and write on HDFS? Please can you elaborate as well it will be more helpful if you attach some screenshot to understand the issue?   Regards   ... View more

Hi  @JasmineD,   We might need to consider backing up the following: flow.xml.gz users.xml authorizations.xml All config files in NiFi conf directory NiFi local state from each node NiFi cluster state stored in zookeeper. Please make sure that you have stored the configuration passwords safely. NiFi relies on sensitive.props.key password to decrypt sensitive property values from flow.xml.gz file. If they do not know sensitive props key, they would need to manually clear all encoded values from flow.xml.gz. This action will clear all passwords in all components on the canvas. We need to re-enter all of them once NiFi was recovered. Also, if there are any local files that are required by the DataFlows, that would also need to be backed up as well. (i.e., Custom processor jars, user-built scripts, externally referenced config/jar files used by some processors, etc.).   Note: All the repositories in NiFi are backed up by default. Here is a good article to see how backup works in NiFi. https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418   Hope this helps. Please accept the answer and vote up if it did. ... View more

Hi @Tanred, Ideally,  it should run every day at 5 pm. Please, can you check the NIFI app logs of 2 consecutive days and see the log entry at that time. Else paste those error trace here for further analysis.  ... View more

@Michael Bronson Decommissioning is a process that supports removing components and their hosts from the cluster. You must decommission a master or slave running on a host before removing it or its host from service. Decommissioning helps you to prevent potential loss of data or disruption of service. Below HDP documentation for Ambari-2.6.1 help you to decommission a DataNode. When DataNode decommissioning process is finished, the status display changes to Decommissioned. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-operations/content/how_to_decommission_a_component.html I hope that the above answers your questions. ... View more

@Michael Bronson Below article help you to replacing faulty disks on datanode. https://community.hortonworks.com/articles/3131/replacing-disk-on-datanode-hosts.html Please accept the answer you found most useful. ... View more

@harish If you made update on that particular queue value then you can ignore it. Because yarn automatically made necessary changes after you updated that particular queue. ... View more

@harish There are some ordering policies to use different algorithms to decide which leaf queue should be scheduled the first. This ordering policy is set on the parent queue and its value is 'utilization' or 'priority-utilization'. (YARN-5864) You can refer below article for more info https://community.hortonworks.com/content/supportkb/174918/the-ordering-policy-setting-for-root-yarn-queue-ya.html Please accept the answer you found most useful. ... View more

@Harish More Please can you confirm whether new user having appropriate permission to access all database like old user, you can check all defined policies in Ranger/Sentry depend on your cluster security access configuration. ... View more

@Raghav Mp HDFS root scratch directory for Hive jobs, which gets created with write all (733) permission. For each connecting user, an HDFS scratch directory ${hive.exec.scratchdir}/ is created with ${hive.scratch.dir.permission}. To fix this issue, you can set below property in hive-site.xml, this is used for setting values for the entire Hive configuration <property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> <description>Scratch space for Hive jobs</description> </property> For more detail: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration Please accept the answer you found most useful. ... View more

@Akash Dixit Suppose below answer won't resolve, please share detailed error for troubleshoot. ... View more