Hi @ryu, then you might need to build some customize in-house monitoring scripts using Yarn APIs or other tools like Prometheus or Grafana for your use case. Please also refer to the below links for more insights https://www.programmersought.com/article/61565532790/ http://rokroskar.github.io/monitoring-spark-on-hadoop-with-prometheus-and-grafana.html https://www.linkedin.com/pulse/how-monitor-yarn-application-via-restful-api-wayne-zhu/         ... View more

Hi @joyabrata  I think you are looking in the Data Lake tab which is a different one, you can go to the Summary tab, then scroll down to FreeIPA   session then click Actions and get  Get FreeIPA Certificate   from the drop-down menu.  Hope this will help you.  ... View more

Hi @joyabrata Obtain the FreeIPA certificate of your environment: From the CDP Home Page navigate to   Management Console   >   Environments . Locate and select your environment from the list of available environments. Go to the Summary tab, scroll down to FreeIPA session then click Actions Select   Get FreeIPA Certificate   from the drop-down menu. The FreeIPA certificate downloads and follows this document  ... View more

Hi @ryu, Cloudera Manager trigger is what you need.  You can create it here: CM -> YARN -> Status -> Create trigger -> Edit manually   Examples: 1) It will alert if there are more than 50 applications in the pending state   Expression: IF (select total_apps_pending_across_yarn_pools WHERE entityName=$SERVICENAME and LAST( total_apps_pending_across_yarn_pools) > 50) DO health:concerning Metric Evaluation Window: 10 minutes   2) It will alert if more than 5 applications are failing   Expression: IF (select total_apps_failed_rate_across_yarn_pools WHERE entityName=$SERVICENAME and LAST( total_apps_failed_rate_across_yarn_pools) > 5) DO health:concerning   Here is the documentation about CM triggers: http://www.cloudera.com/documentation/enterprise/latest/topics/cm_dg_triggers.html   Here is the documentation about CM reports: https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_dg_reports.html ... View more

Hi, @murali2425 @vchhipa It seems some dependency issue while building your custom NiFi Processor,  org.apache.nifi:nifi-standard-services-api-nar  dependency needs to be added in  pom.xml  of  nifi-*-nar  module. Ref here    <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-standard-services-api-nar</artifactId> <version> 1.11 .3 </version> <type>nar</type> </dependency>   Please modify your pom.xml and rebuild and see whether that fixes the issue. Please accept the answer you found most useful. ... View more

Hi @TimmehG,  Spark has a configurable metrics system based on the  Dropwizard Metrics Library . This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. The metrics are generated by sources embedded in the Spark codebase. They provide instrumentation for specific activities and Spark components. The metrics system is configured via a configuration file that Spark expects to be present at  $SPARK_HOME/conf/metrics.properties . A custom file location can be specified via the  spark.metrics.conf   configuration property . Instead of using the configuration file, a set of configuration parameters with prefix  spark.metrics.conf.  can be used.  I agree with you, running spark applications continuously & reliably is a challenging task, and a good performance monitoring system is needed. Several external tools can be used to help profile the performance of Spark jobs: Cluster-wide monitoring tools, such as   Ganglia, can provide insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia dashboard can quickly reveal whether a particular workload is disk-bound, network bound, or CPU bound. OS profiling tools such as   dstat,   iostat, and   iotop   can provide fine-grained profiling on individual nodes. JVM utilities such as   jstack   for providing stack traces,   jmap   for creating heap-dumps,   jstat   for reporting time-series statistics and   jconsole   for visually exploring various JVM properties are useful for those comfortable with JVM internals. For more insights you can refer to the below links: https://spark.apache.org/docs/latest/monitoring.html https://blog.cloudera.com/demystifying-spark-jobs-to-optimize-for-cost-and-performance/ https://www.infoq.com/articles/spark-application-monitoring-influxdb-grafana/ https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting   Please accept the answer you found most useful. ... View more

Can you verify one, whether did you followed all the steps listed in the documentation? https://docs.cloudera.com/runtime/7.1.1/ozone-storing-data/topics/ozone-setting-up-ozonefs.html ... View more

Hi @Ettery  Can you try to add those properties in   nifi.properties ? the   Docker configuration has been updated   to allow proxy whitelisting from the run command the host header protection is only enforced on "secured" NiFi instances. This should make it much easier for users to quickly deploy sandbox environments like you are doing in this case Even you can try with: -e NIFI_WEB_HTTP_HOST=<host>  in docker run command   docker run --name nifi -p 9090:9090 -d -e NIFI_WEB_HTTP_PORT='9090' -e NIFI_WEB_HTTP_HOST=<host> apache/nifi:latest   In GitHub   example configuration  and documentation for NiFi running behind a reverse proxy that you may be interested in.  For more detail refer stackoverflow1 and stackoverflow2 ... View more

You can try with spark-shell --conf spark.hadoop.hive.exec.max.dynamic.partitions=xxxxx.    $ spark-shell --conf spark.hadoop.hive.exec.max.dynamic.partitions=30000 Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hostname:port Spark context available as 'sc' (master = yarn, app id = application_xxxxxxxxxxxx_xxxx). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.x.x.x.x.x.x-xx /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112) Type in expressions to have them evaluated. Type :help for more information. scala> spark.sqlContext.getAllConfs.get("spark.hadoop.hive.exec.max.dynamic.partitions") res0: Option[String] = Some(30000)  Ref: SPARK-21574 ... View more

Please can you check with your internal Linux team / Network team for further support? Because it seems you have some internal connection while connecting the node from the  Intellij idea node. Once you resolve the connection issue we will check further.  ... View more

Can you add below property at <spark_home>/conf/hive-site.xml and <hive-home>/conf/hive-site.xml hive.exec.max.dynamic.partitions=2000   <name>hive.exec.max.dynamic.partitions</name> <value>2000</value> <description></description> Hope this helps. Please accept the answer and vote up if it did. Note : Restart HiveServer2 and Spark History Server if it didn't work.   -JD ... View more

Can you try this below article? https://saagie.zendesk.com/hc/en-us/articles/360021384151-Read-Write-files-from-HDFS ... View more

Hi @w12q12  So as per the below error in the log trace 20/04/21 18:20:50 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1067413441-127.0.0.1-1508775264580:blk_1073743149_2345 file=/data/ratings.csv at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:930) .... Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1067413441-127.0.0.1-1508775264580:blk_1073743149_2345 file=/data/ratings.csv at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:930) It seems that the namenode is not able to connect to the datanode when you ran the command. Please can you try to ping and telnet datanode and name node vice versa also check whether do you have any corrupt blocks and files in the cluster? ~JD ... View more

Hi @Gubbi  You can try the below script.   source_dir="/user/mft/inbound/" dest_dir="/users/aws/outbound/" for file in "$source_dir"*; do dest=$(echo $file | grep -o -P '(?<=_).*(?=.csv)'); mkdir -p "$dest_dir$dest"; mv $file "$dest_dir$dest"; done   Hope this helps. Please accept the answer and vote up if it did. ... View more

Try to clean up some old files from your disk or else add some more space into the disk.  ... View more

Hi  @Dharm,   What do you mean by bad health space here? Is HDFS Full and getting critical alerts either in Ambari or Cloudera Manager? Are you able to read and write on HDFS? Please can you elaborate as well it will be more helpful if you attach some screenshot to understand the issue?   Regards   ... View more

Hi  @JasmineD,   We might need to consider backing up the following: flow.xml.gz users.xml authorizations.xml All config files in NiFi conf directory NiFi local state from each node NiFi cluster state stored in zookeeper. Please make sure that you have stored the configuration passwords safely. NiFi relies on sensitive.props.key password to decrypt sensitive property values from flow.xml.gz file. If they do not know sensitive props key, they would need to manually clear all encoded values from flow.xml.gz. This action will clear all passwords in all components on the canvas. We need to re-enter all of them once NiFi was recovered. Also, if there are any local files that are required by the DataFlows, that would also need to be backed up as well. (i.e., Custom processor jars, user-built scripts, externally referenced config/jar files used by some processors, etc.).   Note: All the repositories in NiFi are backed up by default. Here is a good article to see how backup works in NiFi. https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418   Hope this helps. Please accept the answer and vote up if it did. ... View more

Hi @Tanred, Ideally,  it should run every day at 5 pm. Please, can you check the NIFI app logs of 2 consecutive days and see the log entry at that time. Else paste those error trace here for further analysis.  ... View more

@Michael Bronson Decommissioning is a process that supports removing components and their hosts from the cluster. You must decommission a master or slave running on a host before removing it or its host from service. Decommissioning helps you to prevent potential loss of data or disruption of service. Below HDP documentation for Ambari-2.6.1 help you to decommission a DataNode. When DataNode decommissioning process is finished, the status display changes to Decommissioned. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-operations/content/how_to_decommission_a_component.html I hope that the above answers your questions. ... View more