Q&A

I have a workflow which loads JSON templates from SQL which contain attribute placeholders:{    "agent": "${hostname(true)}",    "startDate": "${startDate}",    "endDate": "${endDate}",    "data": "${data}"}These templates contain placeholder attributes which I'd like to replace with the value of real attributes in the Flowfile e.g. ${endDate}The SQL results are converted to JSON using ConvertAvroToJSON and then converted to an attribute named: analysisMethodArgs using EvaluateJsonPath.I've tried ReplaceText to replace the Flowfile content with the analysisMethodArgs attribute, but it still results in the placeholders in the text and not the actual attribute values.Is this possible?

Ambari 2.6 and HDP 2.6.3. The error is displayed while performing the following error:1) HDFS get operation.2) While aggregating and writing file on HDFS using pyspark. Error: "19/08/29 15:53:02 WARN hdfs.DFSClient: Failed to connect to /DN_IP:1019 for block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no length prefix available " We found the following links to resolve the above error.=> To set dfs.datanode.max.transfer.threads=81961) https://www.netiq.com/documentation/sentinel-82/admin/data/b1nbq4if.html (Performance Tuning Guidelines)2) https://github.com/hortonworks/structor/issues/7 (jmaron commented on Jul 28, 2014) Could you all please suggest shall i go ahead with this resolution?Does this configuration affects any other services? Thankyou

Hi    This is in continuation with my earlier question (https://community.cloudera.com/t5/Support-Questions/Services-fail-to-start-gets-stuck-on-HDP-3-0-sandbox/m-p/269541#M206914) about services failing to start after certain time. However following the suggestions I have stopped all the services and just trying to run zookeeper and Hive and still the Hive metastore service fails to start.       Any solution to this issue and what might be causing this?

This is my error logs ......    ==========================Creating target directory...==========================Command start time 2019-09-02 17:01:05Connection to master closed.SSH command execution finishedhost=master, exitcode=0Command end time 2019-09-02 17:01:05==========================Copying ambari sudo script...==========================Command start time 2019-09-02 17:01:05scp /var/lib/ambari-server/ambari-sudo.shhost=master, exitcode=0Command end time 2019-09-02 17:01:05==========================Copying common functions script...==========================Command start time 2019-09-02 17:01:05scp /usr/lib/python2.6/site-packages/ambari_commonshost=master, exitcode=0Command end time 2019-09-02 17:01:06==========================Copying OS type check script...==========================Command start time 2019-09-02 17:01:06scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.pyhost=master, exitcode=0Command end time 2019-09-02 17:01:06=======================

This query outputs NPE. The tasks with NPEs are retried, and most of the times (but not always) end up succeeding.I could not find a smaller query showing my problem so I give here my full query:select  s.ts_utc as sent_dowhour, o.ts_utc as open_dowhour, sum(count(s.ts_utc)) over(partition by s.ts_utc) as sent_countfrom vault.sent sleft join open o ono.id=s.idgroup by 1, 2My guess is that the construction sum(count(...)) over(partition by ...)has issues.When it fails, this is the output I get:Vertex failed, vertexName=Reducer 2, vertexId=vertex_1556016846110_42971_7_03, diagnostics=» Task failed, taskId=task_1556016846110_42971_7_03_000221, diagnostics=» TaskAttempt 0 failed, info=» Error: Error while running task ( failure ) : attempt_1556016846110_42971_7_03_000221_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunP

Using CM with CDF, using Parcels, trying to Add NiFi CA Service to cluster. Got the following error while trying to activate roles:at com.cloudera.cfg.tools.XmlTools.parseXml(XmlTools.java:122)     at com.cloudera.cfg.tools.XmlTools.parseHadoopXmlConf(XmlTools.java:51)     at com.cloudera.cfg.configuration.Configuration.getGlobals(Configuration.java:73)     at com.cloudera.cfg.configuration.Configuration.substitute(Configuration.java:89)     at com.cloudera.cfg.configuration.Configuration.load(Configuration.java:57)     at com.cloudera.cfg.JobProcessor.<init>(JobProcessor.java:45)     at com.cloudera.cfg.Main.main(Main.java:30) Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/59-nifitoolkitca-NIFI_TOOLKIT_SERVER/staging/globals.xml (No such file or directory)     at java.io.FileInputStream.open0(Native Method) &nb

Hello, We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better. Thanks.

I want to save events through Kafka into Hadoop ecosystem and also retrieve them with low latency. I can use hive3 or hbase. Which one should I use?

We have recently upgraded our Non Prod CPU resources, the upgraded cpu resources are visible in OS level .    but on Ambari Web UI in HOSTS section . old resources configurations showing.    how to refresh the resouce upgrade changes in WEB UI of ambari ? 

Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!Error Code: 0at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316)at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147)at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:207)at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:760)at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:265)at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:731)... 9 moreCaused by: java.sql.SQLException: Connections could not be acquired from the underlying database!at com.mchange.v2.sql.SqlUtils.toSQLException(SqlU

Hi Team,I was trying to read data from kafka topics by using hive table.SInce my data is in avro format so trying to create a hive using below statement:CREATE EXTERNAL TABLE kafka_t_avro   (`timestamp` timestamp , `page` string,  `newPage` boolean,   added int, deleted bigint, delta double)   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'   TBLPROPERTIES   ("kafka.topic" = "topic-name",   "kafka.bootstrap.servers"="hostname:9092",   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");It is giving me below error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found). I have added the below jars in my class path: kafka-handler-3.1.0.3.1.0.0-78.jar;hive-serde-0.10.0.jar;hive-exec-3.1.0.3.1.0.0-78.jar;@j

HI,   Since I am a novice in the Hadoop ecosystem and I have been trying to use the sandbox for HDP 3.0.1 in the past few days, the sandbox is behaving very strangely. In the beginning it used to work fine with all the services been started etc. just within a few minutes. However lately, the services most of the time fail to start or gets stuck at 55% or so for over an hour or even longer sometimes and still doesn't start. I have tried to uninstall and install again a couple of times but it doesn't help. Could you please explain why this might be happening and how to correct that?      

Hi,   There is an application to connect Hive with the the Hadoop ecosystem which I am trying to run with the HDP Sandbox but not been able to do so for sometime now. The functionality has been tested as far as I know I think with a properly established multi-node Hadoop ecosystem by an ex-teammate and since I want to test it and use it again, I was wondering if the sandbox provides such opportunity where a user can actually connect to such third party apps or is it just for testing the Hadoop ecosystem and there are some restrictions which could be causing the issues with the connectivity to third party connectivity?   The process will go like this:   I have a json config file (containing the connectivity and authentication parameters to the third party webapp I am trying to connect to) and a jar file which will run the service to connect the Hive and access it.    Thanks.

Does anyone know where the HDP documentation went? The https://docs.hortonworks.com/ gets redirected to https://docs.cloudera.com/index.html. And the link to HDP documentation URL mentioned there (https://docs.cloudera.com/HDPDocuments/index.html)  gets redirected back to https://docs.cloudera.com/index.html.   BTW, any direct link to original docs gives an HTTP 404 error message, such as: 404 Not Found Code: NoSuchKey Message: The specified key does not exist. Key: HDPDocuments/HDPDocuments/HDPDocuments/HDP3/HDP-3.1.0/authentication-with-kerberos/content/authe_spnego_enabling_browser_access_to_a_spnego_enabled_web_ui.html RequestId: 1F8B4A953909E2F5 HostId: UAdEmAmTrYb4pz3EPxI92SkgHo9kVUlwvXjCnWjRhFzJr3kbvUKHxE9mpDL1hLzdsI5HbGS2MlA=  

I am using CDH 5.16.1 and I hit the issue HDFS-8298 as discussed in the below link   https://issues.apache.org/jira/browse/HDFS-8298   Can someone let me know if this is yet fixed in any of the CDH versions?    

Hey, folks, I'm running a NIFI test on Hortonwork behind a firewall and I want you to help me, I have two nifi installed on my servers. ev01.pepe.local:9091mv01.pepe.local:9091 Generate my Internal CA and at the same time generate certificate for the external name that would be proxev01.pepe.com and also generate with SAN for the internal HOST names, now I'm doing a NAT for the firewall to the ports, it works for me and I have it with permissions for the ranger and ldap. it works for me but when I try to log in with the user I get an error Untrusted proxy CN=proxev01.pepe.com, OU=IT, O=PEPE, L=Madrid, ST=Madrid, C=ES. Could you help me? Thank you Translated with www.DeepL.com/Translator

I want to add 2 host in running Nifi server, which steps i have to follow.. please suggest you input

The reason why I am asking this is a little complicated. So I will not mention too much about it (unless somebody is interested). In fact, I have more than one question.   1. When Kerberos is enabled, does CM add service principals based on hostname shown/stored in CM or every host's actual hostname ? Otherwise, are they always the same ? 2. If hostnames in CM can be different from actual hostnames, how do I achieve that ? (I have tried changing database records and the host cannot connect to HostMonitor afterwards)   Thanks.

I use this command to start:sudo systemctl start cloudera-scm-server. I 'm using CDH-6.2.0-1.cdh6.2.0. and I use mysql as metastore.Then it tells me as below: ```2019-07-16 15:06:38,947 ERROR main:com.cloudera.server.cmf.Main: Server failed.org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version.at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java

Hi,Pretty new to NiFi and trying to understand the difference between Fetch,Get and List processors.List - As I understand, creates flow files with only metadata and not the data. This information can be further passed to downstream to read the file contents.I am pretty confused about Get/Fetch and which one to be used under what situation.

I'm fairly new to NiFi and trying to execute a Python script stored on Local FS using NiFi. There are couple of XLSB files stored in HDFS. I would want to build a NiFi flow that reads files from HDFS and passes the filename to Python script so that it can convert those to CSV and store it back to HDFS.What should be the flow I need to use to get the above working. I tried using ListHDFS -> ExecuteStream but dont know if that's correct. Also, how do I just test the output of ListHDFS to see the output.

is it possible to degrade hdp version from 3.1 to 2.7?

Deal all.   CM has below query to draw "Total slow get operations across RegionServers"   SELECT integral(total_slow_get_rate_across_regionservers) WHERE entityName = "hbase" AND category = SERVICE   What are the criteria for slow? 1sec ? or  hbase.ipc.warn.response.time?   Thanks in advance  

Hello Team, Is it mandatory to enable ssl for Nifi and Nifi registry in order to configure Ldap Authentication? I followed all steps mentioned in article https://docs.hortonworks.com/HDPDocuments/CFM/CFM-1.0.1/securing-cfm/content/cfm-configure-ldap.html When I open Nifi Ui no login window pops up. Regards, Hemanth 

where i need to check error and solve to launch the browser foe above ink