Solved

Does anyone know if it is possible to exclude the Kafka service from the primary "Cluster Start/Stop" sequence in Cloudera Manager?  We need to stop the CDH cluster to perform maintenance on the MySQL metadata server, which affects Hive, Impala, Sentry, Hue, HBase?, Oozie, Navigator, etc... all of which we use.  Kafka is part if this CDH cluster.To stop the cluster, I would like to use the cluster "start/stop" menu item on the main CM page just for peace of mind in maintaining the proper shut down and startup sequence, but that would include Kafka going down as well, which we would like to avoid if possible.  Any thoughts are appreciated.  Thanks!

Hello! I have a student attempting to install Mongo into Cloudera.  The following error messages are being received.  I am not a coder and am unsure how to assist.  I am learning the language as I go along! Any ideas? Thank you! 

I have a JSON data coming as {"x":"avc123.abc.com"} I would like to have a processor in nifi to extract only the integer (123) from the entire value and output it from the process secondly, I would like that integer value (123) to be added in the JSON event with different keys such as the final JSON event would be {"x":"avc123.abc.com" "y":"123" } I already tried routetext and extract text but it didn't work.

Hi, I have e simple txt file for test: 1;God of War 12;God of War 23;God of War 34;God of War 45;God of War Chains of Olympus6;God of War Ghost of Sparta7;God of War Ascension8;Batman Arkhan Asylum9;Batman Arkhan City10;Batman Arkhan Knight I want get the values and split with a ExtractText processor. I create two properties in processor, id and desc id value -> (.*?); desc value -> .*?;(.*?); When the process goes to matched relationship, it get only id value, and desc property comes blank

Hi, Just wanted to know if we can setup and use KNIME (A visual data analytics tool). I've tried to install in HDP 3.0 with the help of link "https://docs.knime.com/2018-12/server_installation_guide/index.html". Installation was successful but when I was trying to use link sandbox-hdf.hortonworks.com:8090, it was not connecting. In error log file I could see that "tomcat" was not able to connect localhost:8005. Not sure if this was the reason. Please let me know if anyone used KNIME in HDP sandbox. Or any Visual Data Analytics tool you have used.Thanks.

Hello,   I'm just starting off and was wondering if there's any concrete way of setting the compression when writing to a file in Spark?   I used to use option when writing files:   exampleDF.write.option("compression", "snappy").avro("output path")   but when I go to check where the Avro files are saved I can't tell from the name of the files whether they've been compressed or not. Also just to say this is after I've imported "com.databricks.spark.avro._" so I'm not having any trouble using Avro files.   Another way I've seen is to use "sqlContext.setConf" and these would be the commands I'd use in this instance:   import org.apache.spark.sql.hive.HiveContext val sqlContext = new HiveContext(sc) sqlContext.setConf("spark.sql.avro.compression.codec", "snappy") exampleDF.write.avro("output path")   Neither way causes any errors so I was wondering which would be the better way and if there are any other more reliable ways of setting the compression when

hi, what data type should we use for general home address in hive???

Hello All, First of all, I would like to thank community. I am trying my hand in NiFi and I came across a situation that confuses me. I am trying to read a csv file in NiFi using ExecuteScript(yes, I want to use this) processor. All the good records must be collected in one flowFile and the bad records are collected in one flowFile each (this way, I will easily know how many bad records exist.) and then route them to SUCCESS and FAILURE relations accordingly. The script works fine when all the records are good. I deliberately inserted one bad record to test exception and it is not working. As per my knowledge, NiFi should throw ValueError but it throws NullPointerException. Not sure what I am missing. I am new to this and learning. any guidance will be much appreciated. Input:1,2,3,41,A,3,41,2,3,4 Code import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCal

I have configured Ranger with Ranger-admin and Hadoop , HDFS  plugins.  I can access the basic url,curl -u rangeradmin:rangeradmin -i -X GET http://127.0.0.1:6080HTTP/1.1 302 FoundSet-Cookie: RANGERADMINSESSIONID=6A1E7311C911AE876D02359679222E24; Path=/; HttpOnlyX-Frame-Options: DENYCache-Control: no-cache, no-store, max-age=0, must-revalidatePragma: no-cacheExpires: 0X-XSS-Protection: 1; mode=blockX-Content-Type-Options: nosniffContent-Security-Policy: default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline';font-src 'self'Location: http://127.0.0.1:6080/login.jspContent-Length: 0Date: Thu, 05 Sep 2019 09:02:16 GMTServer: Apache Ranger but on accessing other urls it fails:  curl -u rangeradmin:rangeradmin -i -X GET http://127.0.0.1:6080/service/plugins/policies/service/1HTTP/1.1 401 UnauthorizedSet-Cookie: RANGERADMINSESSIONID=96A517FCD3CDC23354BFDFE3D22E6FE4; Path=/; HttpOnl

Yarn Application status was failed, and displayed follows.Diagnostics: Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143.And Yarn Log Entries includes following message.ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Driver XX.XX.XX.XX:46869 disassociated! Shutting down.INFO DiskBlockManager: Shutdown hook calledCould you tell me the checkpoint to solve this problem?Best Regards.

I have a workflow which loads JSON templates from SQL which contain attribute placeholders:{    "agent": "${hostname(true)}",    "startDate": "${startDate}",    "endDate": "${endDate}",    "data": "${data}"}These templates contain placeholder attributes which I'd like to replace with the value of real attributes in the Flowfile e.g. ${endDate}The SQL results are converted to JSON using ConvertAvroToJSON and then converted to an attribute named: analysisMethodArgs using EvaluateJsonPath.I've tried ReplaceText to replace the Flowfile content with the analysisMethodArgs attribute, but it still results in the placeholders in the text and not the actual attribute values.Is this possible?

Ambari 2.6 and HDP 2.6.3. The error is displayed while performing the following error:1) HDFS get operation.2) While aggregating and writing file on HDFS using pyspark. Error: "19/08/29 15:53:02 WARN hdfs.DFSClient: Failed to connect to /DN_IP:1019 for block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no length prefix available " We found the following links to resolve the above error.=> To set dfs.datanode.max.transfer.threads=81961) https://www.netiq.com/documentation/sentinel-82/admin/data/b1nbq4if.html (Performance Tuning Guidelines)2) https://github.com/hortonworks/structor/issues/7 (jmaron commented on Jul 28, 2014) Could you all please suggest shall i go ahead with this resolution?Does this configuration affects any other services? Thankyou

Hi    This is in continuation with my earlier question (https://community.cloudera.com/t5/Support-Questions/Services-fail-to-start-gets-stuck-on-HDP-3-0-sandbox/m-p/269541#M206914) about services failing to start after certain time. However following the suggestions I have stopped all the services and just trying to run zookeeper and Hive and still the Hive metastore service fails to start.       Any solution to this issue and what might be causing this?

This is my error logs ......    ==========================Creating target directory...==========================Command start time 2019-09-02 17:01:05Connection to master closed.SSH command execution finishedhost=master, exitcode=0Command end time 2019-09-02 17:01:05==========================Copying ambari sudo script...==========================Command start time 2019-09-02 17:01:05scp /var/lib/ambari-server/ambari-sudo.shhost=master, exitcode=0Command end time 2019-09-02 17:01:05==========================Copying common functions script...==========================Command start time 2019-09-02 17:01:05scp /usr/lib/python2.6/site-packages/ambari_commonshost=master, exitcode=0Command end time 2019-09-02 17:01:06==========================Copying OS type check script...==========================Command start time 2019-09-02 17:01:06scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.pyhost=master, exitcode=0Command end time 2019-09-02 17:01:06=======================

This query outputs NPE. The tasks with NPEs are retried, and most of the times (but not always) end up succeeding.I could not find a smaller query showing my problem so I give here my full query:select  s.ts_utc as sent_dowhour, o.ts_utc as open_dowhour, sum(count(s.ts_utc)) over(partition by s.ts_utc) as sent_countfrom vault.sent sleft join open o ono.id=s.idgroup by 1, 2My guess is that the construction sum(count(...)) over(partition by ...)has issues.When it fails, this is the output I get:Vertex failed, vertexName=Reducer 2, vertexId=vertex_1556016846110_42971_7_03, diagnostics=» Task failed, taskId=task_1556016846110_42971_7_03_000221, diagnostics=» TaskAttempt 0 failed, info=» Error: Error while running task ( failure ) : attempt_1556016846110_42971_7_03_000221_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunP

Using CM with CDF, using Parcels, trying to Add NiFi CA Service to cluster. Got the following error while trying to activate roles:at com.cloudera.cfg.tools.XmlTools.parseXml(XmlTools.java:122)     at com.cloudera.cfg.tools.XmlTools.parseHadoopXmlConf(XmlTools.java:51)     at com.cloudera.cfg.configuration.Configuration.getGlobals(Configuration.java:73)     at com.cloudera.cfg.configuration.Configuration.substitute(Configuration.java:89)     at com.cloudera.cfg.configuration.Configuration.load(Configuration.java:57)     at com.cloudera.cfg.JobProcessor.<init>(JobProcessor.java:45)     at com.cloudera.cfg.Main.main(Main.java:30) Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/59-nifitoolkitca-NIFI_TOOLKIT_SERVER/staging/globals.xml (No such file or directory)     at java.io.FileInputStream.open0(Native Method) &nb

Hello, We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better. Thanks.

I want to save events through Kafka into Hadoop ecosystem and also retrieve them with low latency. I can use hive3 or hbase. Which one should I use?

We have recently upgraded our Non Prod CPU resources, the upgraded cpu resources are visible in OS level .    but on Ambari Web UI in HOSTS section . old resources configurations showing.    how to refresh the resouce upgrade changes in WEB UI of ambari ? 

Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!Error Code: 0at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316)at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147)at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:207)at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:760)at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:265)at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:731)... 9 moreCaused by: java.sql.SQLException: Connections could not be acquired from the underlying database!at com.mchange.v2.sql.SqlUtils.toSQLException(SqlU

Hi Team,I was trying to read data from kafka topics by using hive table.SInce my data is in avro format so trying to create a hive using below statement:CREATE EXTERNAL TABLE kafka_t_avro   (`timestamp` timestamp , `page` string,  `newPage` boolean,   added int, deleted bigint, delta double)   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'   TBLPROPERTIES   ("kafka.topic" = "topic-name",   "kafka.bootstrap.servers"="hostname:9092",   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");It is giving me below error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found). I have added the below jars in my class path: kafka-handler-3.1.0.3.1.0.0-78.jar;hive-serde-0.10.0.jar;hive-exec-3.1.0.3.1.0.0-78.jar;@j

HI,   Since I am a novice in the Hadoop ecosystem and I have been trying to use the sandbox for HDP 3.0.1 in the past few days, the sandbox is behaving very strangely. In the beginning it used to work fine with all the services been started etc. just within a few minutes. However lately, the services most of the time fail to start or gets stuck at 55% or so for over an hour or even longer sometimes and still doesn't start. I have tried to uninstall and install again a couple of times but it doesn't help. Could you please explain why this might be happening and how to correct that?      

Hi,   There is an application to connect Hive with the the Hadoop ecosystem which I am trying to run with the HDP Sandbox but not been able to do so for sometime now. The functionality has been tested as far as I know I think with a properly established multi-node Hadoop ecosystem by an ex-teammate and since I want to test it and use it again, I was wondering if the sandbox provides such opportunity where a user can actually connect to such third party apps or is it just for testing the Hadoop ecosystem and there are some restrictions which could be causing the issues with the connectivity to third party connectivity?   The process will go like this:   I have a json config file (containing the connectivity and authentication parameters to the third party webapp I am trying to connect to) and a jar file which will run the service to connect the Hive and access it.    Thanks.

Does anyone know where the HDP documentation went? The https://docs.hortonworks.com/ gets redirected to https://docs.cloudera.com/index.html. And the link to HDP documentation URL mentioned there (https://docs.cloudera.com/HDPDocuments/index.html)  gets redirected back to https://docs.cloudera.com/index.html.   BTW, any direct link to original docs gives an HTTP 404 error message, such as: 404 Not Found Code: NoSuchKey Message: The specified key does not exist. Key: HDPDocuments/HDPDocuments/HDPDocuments/HDP3/HDP-3.1.0/authentication-with-kerberos/content/authe_spnego_enabling_browser_access_to_a_spnego_enabled_web_ui.html RequestId: 1F8B4A953909E2F5 HostId: UAdEmAmTrYb4pz3EPxI92SkgHo9kVUlwvXjCnWjRhFzJr3kbvUKHxE9mpDL1hLzdsI5HbGS2MlA=  

I am using CDH 5.16.1 and I hit the issue HDFS-8298 as discussed in the below link   https://issues.apache.org/jira/browse/HDFS-8298   Can someone let me know if this is yet fixed in any of the CDH versions?