Q&A

HI All,   I am facing NIFI JSON to PutParquet avro schema issue, Please find below for the schema and actual incoming data format.   When I use below schema I am getting data but second record fields are getting missed and table looks not good.    And I want to achieve second record fields should come column by column, Can anyone help how to achieve that.       SCHEMA:   {"name": "MyClass","type": "record","namespace": "com.acme.avro","fields": [{"name": "preview","type": "boolean"},{"name": "offset","type": "int"},{"name": "result","type": {"name": "result","type": "record","fields": [{"name": "action","type": "string"},{"name": "app","type": "string"},{"name": "dest","type": "string"},{"name": "dest_bunit","type": "string"},{"name": "dest_category","type": "string"},{"name": "dest_ip","type": "string"},{"name": "dest_port","type": "string"},{"name": "dest_priority","type": "string"},{"name": "direction","type": "string"},{"name": "duration","ty

I have donwloaded HDP 3.0.1, and then installed in Virtualbox(its version greater than 5.1). But I am getting this error. Virtualization is enabled in BIOS. Also, Hyper -v was installed. What I will do? Note: I run Virtualbox as Adminastor.

Hi All, I am newbie to NIFI!Today I got one problem of duplicate records, Below is the following scenario:We need to get data from Splunk to HDFS in Parquet format so we created a Data Flow with NIFI  - I am getting data from Splunk using GetSplunk in JSON format and later putting in HDFS using the PutParquet processor with JsonTreeReader, AvroSchema.- I am successful in this but later I see there are duplicate records of each record, So seeking help here to fix this issue and below is the sample JSON record and PFA for the NIFI DataFlow screenshot. Thanks In Advance.  Example Data Record: [ {  "preview" : true,  "offset" : 0,  "result" : {    "action" : "allowed",    "app" : "",    "dest" : "xxxx.xx.xxx.xxx",    "dest_bunit" : "",    "dest_category" : "",    "dest_ip" : "xxx.xx.xxx.xxx",    "dest_port" : "xxx",    "dest_priorit

sqoop import \-Dhadoop.security.credential.provider.path=jceks://hdfs/user/lrm0613/mydb2.jceks \--connection-manager org.apache.sqoop.manager.SQLServerManager \--driver net.sourceforge.jtds.jdbc.Driver \--connect 'jdbc:jtds:sqlserver://SQLQP002:1433;useNTLMv2=true;domain=JNL_NT;databaseName=TC31Scheduler' \--username 'lrm0613' \--password-alias sqlserver2.password \--query 'select max(id) from NodeName a where $CONDITIONS' \--split-by a.id \--target-dir /user/lrm0613/queries_output/tt     Generates the following error: 20/06/10 16:37:51 ERROR tool.ImportTool: Import failed: We found column without column name. Please verify that you've entered all column names in your query if using free form query import (consider adding clause AS if you're using column transformation)   I've noticed that if I remove the max() function the job completes successfully. How can I make it work with the function.

I want to make several invokeHTTP requests with dates from a date range in its URL.So I have first date, for example 2020-05-01, and end date - 2020-05-31.How automatically create massive: 2020-05-01, 2020-05-02, ..., 2020-05-31 and put it into URL.Thanks in advance! 

Hi,  I read the document about the FAQ say i can submit a case then i will get a  download pass , but i do this , i get the following error ,SO how can i do can get the download password 

We are using Hue 4.6.0 with postgres as its backend and noticed that it performs/renders very slow as the number of active users increases. The webui after login taking several (often >1 or more) minutes to load, running query mostly takes even longer to start often showing red exclamation icon (The query is taking hanging or taking longer than usual) or clicking on different options in HUE renders very slow, taking over a minutes to load. One observation were made when this happens that the requests.active value under desktop/metrics goes > 12 or 15, it starts to slow down and higher the number, slower it becomes often just blank page. We have enabled the debug and most of the time costly operations are under notebook/api - execute/ check_status, get_logs or auto_complete, etc. access INFO 1.2.3.4 <username> - "POST /notebook/api/execute/hive HTTP/1.1" returned in 13895ms 200 1547 (mem: 1031mb)access INFO 1.2.3.4 <username> - "POST /n

Hi  I'm currently deploying  CDP-DC trial version on my dev enviroment using the latest Cloudera Manager 7.1.1 But at Add Cluster-Installation --> 4. Select repository, why do runtime version that i can select is only CDH 6.3.2 & CDH 5.16. Should't i be able to select cloudera runtime 7.1.1? Cloudera Manager 7.1.1 got only CDH 6.3 anf 5.16   I've tried 7.0.3 installation, and i got the option for Cloudera runtime 7.0.3. Cloiudera Manager 7.0.3 got runtime 7.0.3   it shouldn't be like this right?   Thanks

My datanode is currently giving me this warning below:  /dfs/dn is in a filesystem that reaches the threshold for free space.  I have a 2nd disk mounted on /data and I'm thinking of moving the datanode filesystem under /data to have free space, is it possible to do that with no significant effect in the cluster? because i have this same issue also with Namenode and I think if I changed the path there also, fsimage is still retail on the original path..

I want to control the number of containers running in parallel for a query, so that I can many queries parallely in a yarn queue. Yarn Queue Size: 200 GBApprox. Mappers / Containers: 50 I setting container size at 10GB by setting hive.tez.container.size=10240; Once the first query is triggered, the query consumes the whole queue (200 GB) and runs 20 containers parallely, and not allowing the other query to start due to unavailability of Yarn memory in the queue. I want help in indentifying parameters to control the number of containers running in parallel, so thay I can limit to 10. So at any point a query will consume only max 100GB  (10 Containers x 10 GB per container) of 200 GB yarn queue.

Hi,  after an upgrade from CM 5.11 to 5.13 the Cloudera Manager complains with a red excl mark: Cluster has stale Kerberos client configuration.   The cluster was all in green before upgrade and had no problem with kerberos configs (/etc/krb5.conf).   What is more concerning, that after opening this warning, three (gateway) nodes does not require upgrade, but the rest of them does:   Consider stopping roles on these hosts to ensure that they are updated by this command:ip-10-197-13-169.eu-west-1.compute.internal; ip-10-197-15-82.eu-west-1.compute.internal; ip-10-197-18-[113, 248].eu-west-1.compute.internal...     But the command is not there. What should I do? Stop the whole CDH and then rerun the deploy?   Thanks for the advise, T.  

I have a use case to read data from Cassandra where I provide inputs from outside NIFI[Flink Job], also instead of querying the given data for the scheduled time intervals, can we initiate the fetch whenever needed? Is these two use cases are possible with current NIFI system? 

hi, how can I stop my container logs on the nodemanagers deleted once the job finished? they seems be deleted once the job are finished, they exist it here /data/yarn/container-logs when running.  

Hello guys!  I hope your day is going well.  Something really weird (at least for me) is happening when I drop a table in Hive. 1.- A table called "Test" is created by a sqoop import job using "--create-hcatalog-table". See the full job below.2.-Once the import job completes (with a select * from the source), I drop the table without issues:drop table test purge;At this point I expect that all the information is deleted, gone, bye, finito, adios!3.-I run the same sqoop job again, but it not only loads the table with the fresh information from the source. It also shows me the previous information before the drop table! Some guys have told me that's because the table has been created as an external table by the sqoop job. But even that it was an external table, a drop table <table> purge; is used to get rid of the data, isn't it? Pls let me know your thoughts. Sqoop job:sqoop import \-Dhadoop.security.credential.provider.path=jceks://hdfs/use

I currently have NiFi setup and running on my master node. I have data coming in to my data node (node2). But I want to setup a NiFi processor to Get or Fetch the data from node2. Which processor do I use and how? Also I have data sorted in folders represented by date as name. So I have data within folders such as 2020-06-03 , 2020-06-04Is there a way to process the data within these efficiently?  

Dr.who is running on my CDH 6.3, I've seen its crontab also using YARN user:  I've already remove the files of this and I want to implement some restrictions using firewalld, how can I block this virus on running on my YARN 8088? do I need to block the 8088 port in all nodes? and what IP addresses do I need to insert for whitelisting? below is my current rules in firewalld: publictarget: defaulticmp-block-inversion: nointerfaces:sources:services: dhcpv6-client sshports: 8042/tcp 7191/tcp 2181/tcp 3181/tcp 4181/tcp 9010/tcp 8044/tcp 8041/tcp 8040/tcp 8091/tcp 9091/tcp 9995/tcp 9994/tcp 7184/tcp 7185/tcp 8084/tcp 8087/tcp 9087/tcp 9999/tcp 9998/tcp 9867/tcp 9866/tcp 9864/tcp 9865/tcpprotocols:masquerade: noforward-ports:source-ports:icmp-blocks:rich rules:rule family="ipv4" source address="195.3.146.118" reject Appreciate your help on this! thanks! 

I have few files as follows./source/abc.txt/source/xyz.txt/source/subdir/abc-sub.txt/source/subdir/xyz-sub.txt Currently, all the files are copied to the output dir/output/abc.txt/output/xyz.txt/output/abc-sub.txt/output/xyz-sub.txt Is it possible to re-create/maintain the original file hierarchy, as follows;/output/abc.txt/output/xyz.txt/output/subdir/abc-sub.txt/output/subdir/xyz-sub.txt  

Hello, I am using Sandbox HDP 3.0, my Zeppelin UI is not running. I have included the content of the log file, have everyone see it and help me fix it. Thanks very much!   File Log   INFO [2019-10-16 13:05:24,804] ({main} ZeppelinConfiguration.java[create]:121) - Load configuration from file:/etc/zeppelin/3.0.1.0-187/0/zeppelin-site.xmlINFO [2019-10-16 13:05:25,089] ({main} ZeppelinConfiguration.java[create]:129) - Server Host: 0.0.0.0INFO [2019-10-16 13:05:25,089] ({main} ZeppelinConfiguration.java[create]:131) - Server Port: 9995INFO [2019-10-16 13:05:25,090] ({main} ZeppelinConfiguration.java[create]:135) - Context Path: /INFO [2019-10-16 13:05:25,095] ({main} ZeppelinConfiguration.java[create]:136) - Zeppelin Version: 0.8.0INFO [2019-10-16 13:05:25,137] ({main} Log.java[initialized]:193) - Logging initialized @3285ms to org.eclipse.jetty.util.log.Slf4jLogINFO [2019-10-16 13:05:25,452] ({main} ZeppelinServer.java[setupWebAppContext]:387) - ZeppelinServer Webapp path: /usr/

hi everyone ,We want to put data in web app using API endpoint https://localhost:8000/api/event/${id}/attributes this is the API endpoint that i have used in invoke HTTP processor in Remote URL field with PUT method. we want to create new attribute with value and this attribute is a dictionary type so we have to pass the data in the dictionary format for this we have used updateattribute processor and attributetojson processor. updateAttribute processor snapAttributetojson processor snapand the below is out of attributetoJson processor (flowfile){"note":"note from nifi","attributes":"{\n \"workflow\":\"process\"\n }","id":"1ecfbcbb-38bf-49e9-a77e-a88w"} the error msg in invoke http processor { "code": 500, "errors": [ "Traceback (most recent call last):\n File \"/usr/local/lib/python3.6/site-packages/flask/app.py\", line 1949, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.6/site-packages/No value set as per my

Hi! I have a DBCPConnectionPool on NiFi 1.11.3 with this configuration:Normally it works ok, but sometimes the query get stuck on a PutSQL, no errors for minutes and then a timeout (very basic queries)I was trying some configurations on the pool but it seems I didn't quite understand the best way to troubleshoot this, I was thinking in each processor add a retry+counter+error. But don't know where to set the time out for queries from start of it and no per connection. Maybe a way is to not keep connections open until the query starts, how should I set the pool to generate the connection on request from a processor and terminate it before execution.I think it's a Athena problem with pools because I have no problem with another databases like postgres and mysql.Any ideas?? Thank you! 

        updateAttribute processor config and this is the output of AttributetoJson processor ....i want tags value in the list like ["UNIX01"] 

Hi, There was an issue of replication of Hive metadata to DAS; I was trying to resolve it by resetting tables by taking reference of "https://docs.cloudera.com/HDPDocuments/DAS/DAS-1.2.1/troubleshooting/content/das_replication_failure_in_event_processor.html". After executing SQL commands successfully; I was trying to start DAS. Under which "Data Analytics Studio Event Processor" was started successfully and running but "Data Analytics Studio Webapp" was started successfully but not running. Below is the configuration detailHDP - 3.0.1DAS - 1.0.2.0.0 Steps I used;1) Stops DAS from Ambari2) taken backup of /var/log/das log files and created empty files of event-processor.log, das-webapp.log files.3) Executed reset tables SQL commands to das database in postgres 4) Started DAS from Ambari. Other observations - 1) After start of DAS nothing is written in event-processor.log, das-webapp.log files.2) But can see das.db_replication_info was getting refreshed with id

Hello guys,I'm trying to run full import from a table in SQL Server. But sqoop is interpreting the data type incorrectly. Column in SQL Server: ErrorParams (nvarchar(max))Sqoop translates it to ErrorParams varchar, which generates an error. create table `dataengsandbox`.`task` (`id` INT,`recordid` int,`state` int,`prevstate` int,`instanceid` int,`parentjobid` int,`closed` boolean,`requeuecount` int,`autorequeuecount` int,`modifytime` string,`submittime` string,`starttime` string,`endtime` string,`createtime` string,`timestamp` binary,`pendingreason` int,`failurereason` int,`exitcode` int,`failednodeid` int,`errorcode` int,`errorparams` varchar,`output` string,`executionfailureretrycount` int)stored as parquet I would expect it to translate it to ErrorParams varchar(64) instead, or another value.FAILED: ParseException line 22:22 mismatched input ',' expecting ( near 'varchar' in primitive type specification Is there a way to make sqoop cast it correctly? Sqoop j

I have got a difficulty with PutEmail processor. It doesn't send (attach) messages more than 1 Kb.Is it a bug?

Hi Experts: Following the instruction as in https://docs.cloudera.com/documentation/enterprise/6/latest/topics/install_cm_cdh.html , we set up a test CDH 6.3.3 cluster. We also enabled Kerberos for security. The Hive, Impala, HBase command line clients all make connections and are functioning basically. Most parts of Hue works too, except the HBase Browser throws error - "Api Error: TSocket read 0 bytes". Please see below the Hue access.log and HBase ThriftServer log. What to check further to resolve the issue? Thank you!   /var/log/hue/access.log --------------------------- [02/Jun/2020 07:16:11 -0700] INFO 173.2.217.185 xe46 - "POST /hbase/api/getClusters HTTP/1.1" returned in 11ms[02/Jun/2020 07:16:11 -0700] INFO 173.2.217.185 xe46 - "POST /notebook/api/autocomplete/default HTTP/1.1" returned in 152ms[02/Jun/2020 07:16:12 -0700] INFO 173.2.217.185 xe46 - "POST /hbase/api/getTableList/HBase HTTP/1.1" returned in 96ms[02/Jun/2020 07:16:12 -0700] ERROR 173.2.217.185 xe46 - "P