Q&A

Hi, I am trying to figure out how to run command line (cmd) commands within Apache NiFi so that I can trigger or execute processors from the command line.

@mattburgess,@markpayne Hi All,I am using stateful variables to generate an incremental batchid value using update attribute processor,this runs in cluster and have set the processor to run on all nodes. But the batchid values generated are not in an incremental fashion,the processor is missing some values sometime or generating a duplicate value.Is it happening due to restart of the cluster and wiping off of stateful variables data? could you please suggest how can i persist stateful variables data? attached update attribute configuration for reference.Please suggest!!

I'm running CDH6.1 with HDFS underneath Solrcloud with a number of collections.  I'd like to create indexes on one cluster and move them using hadoop distcp to another cluster once the data ingest is complete for the collection.   An issue I've run into is that when creating collections via either 'solrctl collection --create' or via the API 'admin/collections?action=CREATE', the replicas aren't always named predictably. For example, on a 6 shard collection with replicationFactor of 1, I've seen anything from....    core_node3, core_node5, core_node7, core_node9, core_node11, core_node12 ...to....    core_node362, core_node363, core_node364, core_node365, core_node366, core_node367   Since these names end up being used in the HDFS based dataDir/ulogDir values for each replica, it means I have to do a bunch of HDFS renaming to get things to line up with the target cluster's collection's replica names.   I've recently started using createNodeS

I am trying to figure out why all my 3 tablet servers run out of memory, but it's hard to do. Configuration: 3 tablet servers, each has memory_limit_hard_bytes set to 8GB. All Kudu operations are performed via Impala JDBC. Symptoms: INSERT operations fail with errors like: Service unavailable: Soft memory limit exceeded (at 101.27% of capacity Tablet web interface (http://<kudu-tablet-server>:8050/mem-trackers) shows that all the memory is consumed: Total consumption 8.08G Memory limit 8.00G Percentage consumed 100.98% Another table on this web page displays details about memory consumption: Id Parent Limit Current Consumption Peak consumption root none none 1.00G 1.01G log_cache root 1.00G 835.7K 2.59M ... ... ... ... ...   However, when I try to sum up all entries in the "Current Consumption" column, I get much lower value: 2.6 GB. My question: who ate the remaining (8-2.6) = 5.4 GB?   How I sum up all the entries? - parse the HTML code of the w

Hello NiFi community,I have been investigating a problem where the most recent data provenance segment in a shut down nifi application session is not available when the nifi application is next started.My investigation has been to repeatedly (20+ times) stop nifi, start nifi (and let it generate a bit of data provenance), and then inspect the data provenance when the UI becomes available to confirm that all previous data provenance segments are accessible. Then repeat, and repeat, and repeat.At first, I was able to reproduce the problem fairly reliably ever second or third time that I stopped nifi and started it again. The data provenance generated during the previous nifi application session was not displayed in the current nifi application session. I could still see the .prov files in the provenance_repository on disk, so I concluded the most recent .prov file from the previous nifi application session had been corrupted.I then discovered this recommendation (https://docs.hortonwo

Hi, I installed Oracle virtual box and then Cloudera quick start ın it. I have to look for all letters because location of most of them have changed. For example / is on different place. Is there any solution?

Hi,   using director i am trying to create deployment with external mysql db. Got stuck with access issue . [2019-03-14 23:36:15.024 +0500] INFO [io-thread-4] - - - - - ssh:10.142.0.54: Installed:[2019-03-14 23:36:15.024 +0500] INFO [io-thread-4] - - - - - ssh:10.142.0.54: cloudera-manager-server.x86_64 0:6.1.1-853290.el7[2019-03-14 23:36:15.348 +0500] INFO [p-d9121c3c8c30-DefaultBootstrapDeploymentJob] b8c5a817-be73-497e-8ee2-891738075972 POST /api/d6.1/environments/gcp-env-01/deployments com.cloudera.launchpad.bootstrap.InstallMySQLDriver - c.c.launchpad.pipeline.AbstractJob: Installing MySQL driver on instance 6564aae4-b7cb-4b6a-a2e0-211cf08b457b[2019-03-14 23:36:15.354 +0500] INFO [p-d9121c3c8c30-DefaultBootstrapDeploymentJob] b8c5a817-be73-497e-8ee2-891738075972 POST /api/d6.1/environments/gcp-env-01/deployments com.cloudera.launchpad.bootstrap.InstallMySQLDriver - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-03-14T23:51:15.354+05:00 to discover SSH service endp

Here is the HDFS folder structure - /FLIGHT/2019/03/01/XYZ.tsv, /FLIGHT/2019/03/02/XYZ.tsv, /FLIGHT/2019/03/03/XYZ.tsv etc. While declaring HIVE external table LOCATION can we use something like /FLIGHT/{*}/{*}/{*}XYZ.tsv.Basically, we just have to read all files that match that specific pattern. Thanks for your feedback.

Does Cloudera Hive supports for Insert ,Update and Delete on Table?Which is best option for Analytics Batch processing which supports UPDATES and JOINS on tables?

After enabling Kerberos the Yarn ATS HBase Service quits working. Following the directions to destroy the service don't work due some sort of authentication issue. As you can see in my example I clearly have a Kerberos ticket for the yarn-ats users. I've also checked the Kerberos Mapping to ensure this principal is correct. I don't know what else to check.RULE:[1:$1@$0](yarn-ats-hdp31_cluster@DEV.EXAMPLE.ORG)s/.*/yarn-ats/[yarn-ats@hdp31-edge ~]$ klistTicket cache: FILE:/tmp/krb5cc_1001Default principal: yarn-ats-hdp31_cluster@DEV.EXAMPLE.ORGValid starting       Expires              Service principal03/15/2019 21:49:32  03/16/2019 21:49:32  krbtgt/DEV.EXAMPLE.ORG@DEV.EXAMPLE.ORG        renew until 03/22/2019 21:49:32[yarn-ats@hdp31-edge ~]$ yarn app -start ats-hbase19/03/15 21:49:41 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:805019/03/15 21:49:41 INFO client.A

I did shutdown all HDP Services and subsequently ambari-agents and ambari-server on a cluster due to some scheduled network maintainance.When starting the ambari-server and ambari-agents again, the ambari-agent on the headnode (also running ambari-server) did not register with the ambari-server (the server is not getting a request at all). The ambari-agent on the slavenode registers fine.Ambari version is 2.6.1.0 (recently upgraded according to HDP documentation, did work after that), HDP version 2.6.0.3Python 2.7.5 / CentOS7on headnode (ambari running as root):ambari-agent start producesVerifying Python version compatibility...Using python /usr/bin/pythonChecking for previously running Ambari Agent...Checking ambari-common dir...Starting ambari-agentVerifying ambari-agent process status...Ambari Agent successfully startedAgent PID at: /run/ambari-agent/ambari-agent.pidAgent out at: /var/log/ambari-agent/ambari-agent.outAgent log at: /var/log/ambari-agent/ambari-agent.logambari-agent s

Hi all, i was able to to login yesterday but when im trying to login today i'm facing this issue.Here is the output of ambari-server status from terminalScreen Shot of /var/log/ambari-server Also, here i attach the amabri-server.log ambari-server.txtAny help is appreciated a lot. Thanks in advance@Jay Kumar SenSharma @Geoffrey Shelton Okot

Hello -I have ambari version: 2.6.2.2-1 with IPA version: 4.6.4.Issue I am facing is when ambari pulls the keytab file for each host after creating the principle (step 2 of the enable kerberos) it does not take into account the encryption types field used. It pulls all the encryptions from the IPA server but the IPA server only supports: aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96.Looking at the code: https://github.com/apache/ambari/blob/c17ecd1b2d5e41e66533266c9f4d5880ef5bd948/ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/IPAKerberosOperationHandler.javaString[] createKeytabFileCommand = (StringUtils.isEmpty(encryptionTypeSpec))        ? new String[]{executableIpaGetKeytab, "-s", getAdminServerHost(true), "-p", principal, "-k", keytabFileDestinationPath}        : new String[]{executableIpaGetKeytab, "-s", getAdminServerHost(true), "-e", encryptionTypeSpec, "-p", principal, "-k", keytabFileDestinationPath};It sho

Hi, If I set the following property in Impala config for admission control and pool "default", in llama-site.xml: <property>   <name>llama.am.throttling.maximum.placed.reservations.root.default</name>   <value>4</value></property> Suppose I have 3 Impala nodes and all of them have the same configuration above.Does that mean I can run up to 12 concurrent queries (4 in each impalad) in default pool or just 4 overall? Thanks,Paulo. 

Using docker, I've succesfully installed HDP sandbox on my cloud-base CentOS 7 instance. Most services starts up and I see a lot of green in Ambari. Now, i am curious about why the deployed sandbox named my namenode: sandbox-hdp.hortonworks.com?i see that the same namenode is used in the screenshot attached to this post - as disclosed by the URL in the web-browser https://community.hortonworks.com/questions/155768/docker-sandbox-hdphortonworkscom50070-connection-r.htmlIs it expected that a user should tinker with the hosts file to translate that give name to the IP address of the node?That is strange since most of us do not own the hortonworks.com domain. Is this a business-as-usual way? is there a way to change the name of a namenode?

Hi,I have 3 versions of HDP in oracle virtual box. The versions are 2.5, 2.6.5 and 3.0. When i start VM for HDP 3.0 the Ambari UI works fine. But after that if i start VM for HDP lower versions 2.5 or 2.6.5 i can not open Ambari UI. It shows Server error. Based on the requirement i need to use different versions of HDP.How can i resolve this issue ?Thanks,Heta

hello,We have installed a secured hdp 3.1 cluster on Centos 7.5.Then we installed mpack in order to add a nifi single node.The unsecure version worked correctly (at least it displayed the ui correctly ) but upon activating ssl ( with auto generated certificate ) and activating kerberos for authentification, when connecting, we got the following error :Cannot replicate request to Node my_nifi_FDQN_node:9090 because the node is not connectedThis is strange because we use the secure version and connect to nifi via https://my_nifi_FDQN_node:9091/nifi/ and it should not try to connect to 9090.In nifi-user.log, we can see :2019-02-22 11:19:10,767 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for my_ldap_user2019-02-22 11:19:10,772 INFO [NiFi Web Server-21] o.a.n.w.a.c.IllegalClusterStateExceptionMapper org.apache.nifi.cluster.manager.exception.IllegalClusterStateException: Cannot replicate request to Node my_nifi_FDQN_node:9090 because the node is not co

Hello,Can anyone explain how to rise speed of "rest api"?Last month, the CPU load began to grow on ambari server and appeared alert "Ambari server performance rest api Critical 25,000ms"I have 4 CPU core and 32GB ram on ambari hostThe cluster is contain 25 nodes

I am managing CDH 5.6.0 consisting of 16 Linux servers. (1 eco, 2 namenodes, 13 datanodes) The Cloudera Hadoop has no trouble issue, but nowadays, the rate of analyzing process was increased. Anyone associated with analyzing can access CDH hive by using SQL communication program(ex, dbeaver).  What concerns me is termination of the CDH cluster, because, they send so many queries to the hive. Is there the way monitoring client IP accessing to hive?Thank you.

Hi. I am using Embedded PostgreSQL for CM. Backed up as described in the document below. https://www.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ag_backup_dbs.html#cmig_topic_5_6_3pg_dump -h hostname -p 7432 -U scm > /tmp/scm_server_db_backup.$(date +%Y%m%d) But trying to restore fails. [root@localhost ~]$ psql -w -h localhost -p 7432 -U scm -f /tmp/scm_server_db_backup.$(date +%Y%m%d)↓psql:scm_server_db_backup:54: ERROR: relation "audits" already existsALTER TABLEpsql:scm_server_db_backup:77: ERROR: relation "client_configs" already existsALTER TABLEpsql:scm_server_db_backup:89: ERROR: relation "client_configs_to_hosts" already existsALTER TABLE...psql:scm_server_db_backup:1036: ERROR: duplicate key value violates unique constraint "audits_pkey"DETAIL: Key (audit_id)=(1) already exists.CONTEXT: COPY audits, line 1psql:scm_server_db_backup:1056: ERROR: duplicate key value violates unique constraint "client_configs_pkey"DETAIL: Key (cl

I created a similar job as in : https://community.hortonworks.com/content/supportkb/48985/how-to-setup-oozie-shell-action.htmlOn execution the yarn log shows as success.But I can't see the /tmp/output file being created anywhere. I checked on the local linux host as well as hdfs.One question I have is in job.properties -oozie.coord.application.path=${nameNode}/user/root/apps/shellShould the configuration files (job.properties, coordinator.xml etc) reside in the above directory? I have them there.Not sure what is happening.Appreciate the insights.

All-   I have a cluster of 3 VM's within a lab. Memory/RAM is not an issue. I recently went to upgrade to Spark 2.3 which I have done before without issues. Originally I could not get Spark2 to start- it would fail on the 7th/8 configuration updates (Kafka). Unfortunately, I seem to be having numerous issues this time around. In researching my issue with Spark 2 not starting, I came across the following post: https://community.cloudera.com/t5/Cloudera-Manager-Installation/deployClientConfig-on-service-Spark-fails-in-CDH5/td-p/17640/page/2One of the employee posts said the following:We've seen this issue occassionally; you can work around the problem by deleting any empty alternatives under /var/lib/alternatives and restart cloudera-scm-agent ("service cloudera-scm-agent restart"). The agent will recreate any missing alternatives then.This was the "beginning of the end" for me. I did exactly what it stated, but upon restarting the agent I had numerous issues within the Cl

How we can copy recursive jar files from HDFS ( jar files are under sub folders ) to local folder?Exampleexport hdfs_folder=/app/lib export local_folder=/home/work_appwhile under /app/lib we have the following sub folder with the jar filesas/app/lib/folder_jar1/app/lib/folder_jar2Under each of above folder we have jar filesThe following command , will copy only the jar files under /app/lib but not the Jar files under the sub folders as /app/lib/folder_jar1 , /app/lib/folder_jar2 , Etc hadoop fs -copyToLocal $hdfs_folder/*.jar $local_folder

Hi, There is a program that uses Impyla to retrieve data from the local Impala daemon. cursor.execute("select * from table;")rows = cursor.fetchall() The table has 5 million rows, the number of columns is 9, the file size at the time of CSV conversion is about 200 MB.There are four data nodes.Memory is 32 GB. Despite just that much data, fetchall () takes over 200 seconds. Query execution ends in 0.2 seconds Why is it so slow? Do you have any ideas to speed up something?  Thanks! 

I am trying to build Impala on ARM/aarch64, But I did not find any useful guide/document to read. I noticed that there are some assembly code in Impala and third-party code, Has anyone tried building it on ARM?Best RegardsYork