Solved

We're in the process of upgrading from CDH 5 to CDH 6.3.2. We have a management application written in java which uses SparkLauncher to start spark jobs on yarn. We've updated libraries in both the management application and the spark applications from spark 1.x to 2.x When a spark job is started on yarn, it fails due to missing Hive libraries. Adding these libraries via SparkLauncher.addJar() results in the classes being unavailable from the desired classloader.   Code which adds libraries:     final String JAR_DIR = "/opt/cloudera/parcels/CDH/jars/";File jarDir = new File(JAR_DIR);for (String filename : jarDir.list()) {String jar = JAR_DIR + filename;log.info("adding jar: " + jar); launcher.addJar(jar);}     Resulting exception:     2020-05-20 00:21:01 [,,,] [] INFO ApplicationMaster:57 - Final app status: FAILED, exitCode: 15, (reason: User class threw exception: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgum

Update: template for flow titled CountryCodeLookup.xml is in GitHub: https://github.com/johndcal/NiFi-Tutorial-5-FilesHello,Can anyone help me with this?  Below is the data that I am trying to parse and the schema being used to parse it.  Should I be looking at something else?  Will provide more info if needed.Thanks,jc Data:name,loc_id,launch,norad_number,classificationNOAA 19,US,2009-02-06,33591,WeatherGOES 13,US,2006-05-24,29155,TVSAUDISAT 1C,SA,2002-12-20,27607,EducationMETEOR M2,RU,2014-07-08,40069,WeatherMEASAT 3,MY,2006-12-11,29648,TV Schema:{"type": "record","namespace": "satellites","name": "satdata_in","fields": [{ "name": "name", "type": "string" },{ "name": "loc_id", "type": "string" },{ "name": "launch", "type": "string" },{ "name": "norad_number", "type": "int" },{ "name": "classification", "type": "string" }]}

Hello. I have a HDF 3.2 cluster shared for different developing groups. It is running correctly but it's quickly growing bigger and bigger and we've decided to set up different isolated groups of NiFi nodes in order to prevent that a collapse on one project affects all the projects. Our first idea was to install new HDF clusters but we want to manage all the resources with one unique Ambari console and also we want to share other resources (Kafka, NiFi Registry) between all the groups, but I've read that older versions of Ambari doesn't support managing multiple clusters (I don't know if this is still true). What would be the recommended approach? It is possible to manage isolated groups in one unique cluster or multiple clusters managed by different ambari each is the only solution? PD: If upgrading to CDF last version is necessary, that's something we're open to (we're already planning to do this in the near future). Many thanks for your help 

I want to create roles for my hive database management in Hue, but when I try to create roles using Sentry in Hue, I keep on getting this message : Unable to obtain groups in user :I've already input my admin user to Sentry admin user

Hello guys!   I'm trying to run the following commands in Hive: use dataengsandbox;hive.mapred.supports.subdirectories=true;insert overwrite directory '/tmp/hmb/test' select max(id) from nodename;   If I remove the text marked in orange it works without issues, but I want my output to be written into a HDFS directory.   Here is the error that I've received: Error while compiling statement: FAILED: SemanticException No valid privileges User lrm0613 does not have privileges for QUERY The required privileges: Server=server1->URI=hdfs://nameservice1/tmp/hmb/test->action=*;

Hi all  I am running the following ambari API in order to find the CPU count on data node machine  curl -H "X-Requested-By: ambari" -u admin:admin -X GET http://master:8080/api/v1/clusters/hdp_test/hosts?fields=Hosts/host_name,Hosts/public_host_name,Hosts/cpu_count&host_compnents/HostRoles/component_name.in(DATANODE) we get the error about:  {  "status" : 400,  "message" : "The properties [host_compnents/HostRoles/component_name] specified in the request or predicate are not supported for the resource type Host."  The target is to find the number of cpu on data node machine as we can do it on datanode linux machine nproc16

Hi,I've got an INPUT-json that looks like this:{"vendor" : "myINC","beginTime" : "2020-01-21T10:00:00+08:00","userLabel" : "DB","Types" : [ {"Name" : "type001","subTypes" : [ {"subtype003" : {"counter1" : "0","counter5" : "0","counter54" : "0"}},{"subtype001": {"counter002": "0","counter5": "0","counter65": "0"}} ]}, {"Name" : "type002","subTypes" : [ {"subtype006" : {"counter1" : "0","counter5" : "0","counter54" : "0"}},{"subtype009" : {"counter002": "0","counter5": "0","counter65": "0"}}]} ]} and would need a OUTPUT-json like this: {"vendor" : "myINC","beginTime" : "2020-01-21T10:00:00+08:00","userLabel" : "DG","Types" : [ {"Name" : "type001","subType" : [ {"Name": "subtype003","Counters": [{"name": "counter1","value": "0"},{"name": "counter5","value": "0"},{"name": "counter54","value": "0"} ]},{"Name": "subtype001","Counters": [{"name": "counter002","value": "0"},{"name": "counter5","value": "0"},{"name": "counter65","value": "0"}]} ]}, {"Name" : "type002","subType" : [ {"

I have the following input formatINPUT.JSON  [ { "name": "adam", "age": 12, "address": { "city": "delhi", "country": "india", "zip": "110011" } }, { "name": "louis", "age": 23, "address": { "city": "goa", "country": "india", "zip": "110022" } }]  After applying jolt transformation i want to get the following outputDESIRED OUTPUT.JSON  [ { "name": "adam", "age": 12, "address": { "current_city": "delhi", //change here "current_country": "india", //change here "zipode": "110011" //change here } }, { "name": "louis", "age": 23, "address": { "current_city": "goa", //change here "current_country": "india", //change here "zipode": "110022"

HI All,   I am facing NIFI JSON to PutParquet avro schema issue, Please find below for the schema and actual incoming data format.   When I use below schema I am getting data but second record fields are getting missed and table looks not good.    And I want to achieve second record fields should come column by column, Can anyone help how to achieve that.       SCHEMA:   {"name": "MyClass","type": "record","namespace": "com.acme.avro","fields": [{"name": "preview","type": "boolean"},{"name": "offset","type": "int"},{"name": "result","type": {"name": "result","type": "record","fields": [{"name": "action","type": "string"},{"name": "app","type": "string"},{"name": "dest","type": "string"},{"name": "dest_bunit","type": "string"},{"name": "dest_category","type": "string"},{"name": "dest_ip","type": "string"},{"name": "dest_port","type": "string"},{"name": "dest_priority","type": "string"},{"name": "direction","type": "string"},{"name": "duration","ty

I have donwloaded HDP 3.0.1, and then installed in Virtualbox(its version greater than 5.1). But I am getting this error. Virtualization is enabled in BIOS. Also, Hyper -v was installed. What I will do? Note: I run Virtualbox as Adminastor.

Hi All, I am newbie to NIFI!Today I got one problem of duplicate records, Below is the following scenario:We need to get data from Splunk to HDFS in Parquet format so we created a Data Flow with NIFI  - I am getting data from Splunk using GetSplunk in JSON format and later putting in HDFS using the PutParquet processor with JsonTreeReader, AvroSchema.- I am successful in this but later I see there are duplicate records of each record, So seeking help here to fix this issue and below is the sample JSON record and PFA for the NIFI DataFlow screenshot. Thanks In Advance.  Example Data Record: [ {  "preview" : true,  "offset" : 0,  "result" : {    "action" : "allowed",    "app" : "",    "dest" : "xxxx.xx.xxx.xxx",    "dest_bunit" : "",    "dest_category" : "",    "dest_ip" : "xxx.xx.xxx.xxx",    "dest_port" : "xxx",    "dest_priorit

sqoop import \-Dhadoop.security.credential.provider.path=jceks://hdfs/user/lrm0613/mydb2.jceks \--connection-manager org.apache.sqoop.manager.SQLServerManager \--driver net.sourceforge.jtds.jdbc.Driver \--connect 'jdbc:jtds:sqlserver://SQLQP002:1433;useNTLMv2=true;domain=JNL_NT;databaseName=TC31Scheduler' \--username 'lrm0613' \--password-alias sqlserver2.password \--query 'select max(id) from NodeName a where $CONDITIONS' \--split-by a.id \--target-dir /user/lrm0613/queries_output/tt     Generates the following error: 20/06/10 16:37:51 ERROR tool.ImportTool: Import failed: We found column without column name. Please verify that you've entered all column names in your query if using free form query import (consider adding clause AS if you're using column transformation)   I've noticed that if I remove the max() function the job completes successfully. How can I make it work with the function.

I want to make several invokeHTTP requests with dates from a date range in its URL.So I have first date, for example 2020-05-01, and end date - 2020-05-31.How automatically create massive: 2020-05-01, 2020-05-02, ..., 2020-05-31 and put it into URL.Thanks in advance! 

Hi,  I read the document about the FAQ say i can submit a case then i will get a  download pass , but i do this , i get the following error ,SO how can i do can get the download password 

We are using Hue 4.6.0 with postgres as its backend and noticed that it performs/renders very slow as the number of active users increases. The webui after login taking several (often >1 or more) minutes to load, running query mostly takes even longer to start often showing red exclamation icon (The query is taking hanging or taking longer than usual) or clicking on different options in HUE renders very slow, taking over a minutes to load. One observation were made when this happens that the requests.active value under desktop/metrics goes > 12 or 15, it starts to slow down and higher the number, slower it becomes often just blank page. We have enabled the debug and most of the time costly operations are under notebook/api - execute/ check_status, get_logs or auto_complete, etc. access INFO 1.2.3.4 <username> - "POST /notebook/api/execute/hive HTTP/1.1" returned in 13895ms 200 1547 (mem: 1031mb)access INFO 1.2.3.4 <username> - "POST /n

Hi  I'm currently deploying  CDP-DC trial version on my dev enviroment using the latest Cloudera Manager 7.1.1 But at Add Cluster-Installation --> 4. Select repository, why do runtime version that i can select is only CDH 6.3.2 & CDH 5.16. Should't i be able to select cloudera runtime 7.1.1? Cloudera Manager 7.1.1 got only CDH 6.3 anf 5.16   I've tried 7.0.3 installation, and i got the option for Cloudera runtime 7.0.3. Cloiudera Manager 7.0.3 got runtime 7.0.3   it shouldn't be like this right?   Thanks

My datanode is currently giving me this warning below:  /dfs/dn is in a filesystem that reaches the threshold for free space.  I have a 2nd disk mounted on /data and I'm thinking of moving the datanode filesystem under /data to have free space, is it possible to do that with no significant effect in the cluster? because i have this same issue also with Namenode and I think if I changed the path there also, fsimage is still retail on the original path..

I want to control the number of containers running in parallel for a query, so that I can many queries parallely in a yarn queue. Yarn Queue Size: 200 GBApprox. Mappers / Containers: 50 I setting container size at 10GB by setting hive.tez.container.size=10240; Once the first query is triggered, the query consumes the whole queue (200 GB) and runs 20 containers parallely, and not allowing the other query to start due to unavailability of Yarn memory in the queue. I want help in indentifying parameters to control the number of containers running in parallel, so thay I can limit to 10. So at any point a query will consume only max 100GB  (10 Containers x 10 GB per container) of 200 GB yarn queue.

Hi,  after an upgrade from CM 5.11 to 5.13 the Cloudera Manager complains with a red excl mark: Cluster has stale Kerberos client configuration.   The cluster was all in green before upgrade and had no problem with kerberos configs (/etc/krb5.conf).   What is more concerning, that after opening this warning, three (gateway) nodes does not require upgrade, but the rest of them does:   Consider stopping roles on these hosts to ensure that they are updated by this command:ip-10-197-13-169.eu-west-1.compute.internal; ip-10-197-15-82.eu-west-1.compute.internal; ip-10-197-18-[113, 248].eu-west-1.compute.internal...     But the command is not there. What should I do? Stop the whole CDH and then rerun the deploy?   Thanks for the advise, T.  

I have a use case to read data from Cassandra where I provide inputs from outside NIFI[Flink Job], also instead of querying the given data for the scheduled time intervals, can we initiate the fetch whenever needed? Is these two use cases are possible with current NIFI system? 

hi, how can I stop my container logs on the nodemanagers deleted once the job finished? they seems be deleted once the job are finished, they exist it here /data/yarn/container-logs when running.  

Hello guys!  I hope your day is going well.  Something really weird (at least for me) is happening when I drop a table in Hive. 1.- A table called "Test" is created by a sqoop import job using "--create-hcatalog-table". See the full job below.2.-Once the import job completes (with a select * from the source), I drop the table without issues:drop table test purge;At this point I expect that all the information is deleted, gone, bye, finito, adios!3.-I run the same sqoop job again, but it not only loads the table with the fresh information from the source. It also shows me the previous information before the drop table! Some guys have told me that's because the table has been created as an external table by the sqoop job. But even that it was an external table, a drop table <table> purge; is used to get rid of the data, isn't it? Pls let me know your thoughts. Sqoop job:sqoop import \-Dhadoop.security.credential.provider.path=jceks://hdfs/use

I currently have NiFi setup and running on my master node. I have data coming in to my data node (node2). But I want to setup a NiFi processor to Get or Fetch the data from node2. Which processor do I use and how? Also I have data sorted in folders represented by date as name. So I have data within folders such as 2020-06-03 , 2020-06-04Is there a way to process the data within these efficiently?  

Dr.who is running on my CDH 6.3, I've seen its crontab also using YARN user:  I've already remove the files of this and I want to implement some restrictions using firewalld, how can I block this virus on running on my YARN 8088? do I need to block the 8088 port in all nodes? and what IP addresses do I need to insert for whitelisting? below is my current rules in firewalld: publictarget: defaulticmp-block-inversion: nointerfaces:sources:services: dhcpv6-client sshports: 8042/tcp 7191/tcp 2181/tcp 3181/tcp 4181/tcp 9010/tcp 8044/tcp 8041/tcp 8040/tcp 8091/tcp 9091/tcp 9995/tcp 9994/tcp 7184/tcp 7185/tcp 8084/tcp 8087/tcp 9087/tcp 9999/tcp 9998/tcp 9867/tcp 9866/tcp 9864/tcp 9865/tcpprotocols:masquerade: noforward-ports:source-ports:icmp-blocks:rich rules:rule family="ipv4" source address="195.3.146.118" reject Appreciate your help on this! thanks! 

I have few files as follows./source/abc.txt/source/xyz.txt/source/subdir/abc-sub.txt/source/subdir/xyz-sub.txt Currently, all the files are copied to the output dir/output/abc.txt/output/xyz.txt/output/abc-sub.txt/output/xyz-sub.txt Is it possible to re-create/maintain the original file hierarchy, as follows;/output/abc.txt/output/xyz.txt/output/subdir/abc-sub.txt/output/subdir/xyz-sub.txt