Q&A

Hi there, I'm running hive with HDP 3.1.4, and with kerberos enabled.Currently there is a strange issue. Some sql could be success only if we run it with the system user `hive`, otherwise it fails with map error. A simple case is as below:1. Create a table named `test` with columns (id int, val string), and data [(1, '1'), (2, '2')].2. Run sql with user hive: `select * from test t join test t1 on t.id=t1.id;`, it succeeds as below:3. Run the same sql with some other user (e.g. test): `select * from test t join test t1 on t.id=t1.id;`, it fails with a vertex issue:```INFO : Compiling command(queryId=hive_20210119153558_48453ef9-4b06-48ed-b9f6-f64f82992d81): select * from test t join test t1 on t.id=t1.idINFO : Concurrency mode is disabled, not creating a lock managerINFO : Semantic Analysis Completed (retrial = false)INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t.id, type:int, comment:null), FieldSchema(name:t.val, type:string, comment:null), FieldS

Hello,   I am new to CDH and excuse if its a simple or an easy mistake.   I have shell action in oozie which is failing to read a file. I have simple wrapper script, reading a file and passing the value in file to the next step, but its failing saying file or directory not exist. I could read/see the same file in server.   Really appreicate your help on this.    below is my script   #!/bin/bash #to test a sample sqoop script   while read line; do tablename=`echo $line | cut -d'.' -f2` sqoop import --connect "jdbc:sqlserver://ppppppp:1433;database=yyyy" --username userid --password '******' --table $tablename --target-dir hdfs://nameservice1/d/dl/rlsolutions/raw/{$tablename} -m 1 --as-avrodatafile done < /d/dl/rlsolutions/code/oozie-workflows/rl_solutions/hiveScripts/tables.txt   error: ./sample3.sh: line 10: /d/dl/rlsolutions/code/oozie-workflows/rl_solutions/hiveScripts/tables.txt: No such file or directory  

HiI want to upgrade our cluster to HDP3.However I can not access to "public" repo describe in doc (for instance https://archive.cloudera.com/p/ambari/centos7/2.x/updates/2.7.5.0/ambari.repo) because an authentication is now required.Does the community edition still exist or do we need to subscribe a paying subscription even if we do not need support ?Thx for the answer.Regards.

%sqlselect * from enrichedEvents order by hoursDriven desc limit 20 Table or view not found: enrichedEvents; line 1 pos 14set zeppelin.spark.sql.stacktrace = true to see full stacktrace Why this error coming in HDP 2.6.0 where no issues with HDP 3.0 sandbox. Any confugration changes in Zeppelin notebo

I am writing a script to remove a worker node. So I need Ambari API to uninstall the clients running on the node (ZOOKEEPER HDFS YARN MAPREDUCE2 TEZ SPARK2 HIVE)

Hello,  I've looked everywhere and can't seem to find any posts about resolving this issue. I set this same type of config in Google Cloud and it works fine. However, something is wrong with our configuration in our environment and I can't seem to figure it out but I think it's some type of bug.Any suggestions are appreciated. Here's the scenario:Overview:   We have a nifi cluster (2 nodes), we have enabled all of the Site-to-Site settings on both nifi nodes. # Site to Site propertiesnifi.remote.input.host=abcdnifi.remote.input.secure=falsenifi.remote.input.socket.port=1234nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.remote.contents.cache.expiration=30 secs However, the minifi-app.log keeps getting the following errors when we try to start the flow.2020-12-14 19:19:19,595 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not

I am writing a script to remove a worker node. So I need Ambari api to put the node in maintenance mode before removing host components and deleting the node from the cluster.

I use ExecuteSQLRecord to run a query and write to CSV format. The table has 10M rows. Although I can split the output into multiple flow files, the query is executed by only a single thread and is very slow.  Is there a way to partition the query into multiple queries so that the next processor can run multiple concurrent tasks, each one process one partition? It would be like:GenerateTableFetch -> ExecuteSQLRecord (with concurrent tasks)The problem is that GenerateTableFetch only accepts table name as input. It does not accept customized queries. Please advise if you have solutions. I am new to NiFi. So I would like your details. Thank you in advance.

Hi,I have HDP 2.6.5 running in VirtualBox with the networkmode "bridgenetwork".I have opened port 50010 in the sandbox vm and the port is accessible.Now I want to create a text file from my Windows machine with a Java programme. Configuration conf = new Configuration();conf.set("dfs.client.use.datanode.hostname", "true");FileSystem fs = new DistributedFileSystem();fs.initialize(new URI("hdfs://sandbox-hdp.hortonworks.com:8020"), conf);FSDataOutputStream fdos=fs.create(new Path("/testing/file02.txt"), true);fdos.writeBytes("Test text for the txt file");fdos.flush();fdos.close();fs.close(); The text file is also created, but there is nothing in it and I get the following error: 21/01/13 19:13:00 INFO hdfs.DFSClient: Exception in createBlockOutputStreamjava.io.EOFException: Premature EOF: no length prefix availableat org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2294)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOu

Hello Experts, I have upgraded CDH 5.16 to CDP 7.2.4 Cloudera Manager only( not the runtime yet). After upgrade the Cloudera Manager server is running good including agent. But the Cloudera Agents on the rest of the nodes wont start when running $ sudo systemctl restart cloudera-scm-agentThe agents are still at 5.16 as the final step for upgrade does show the upgrade button for the agents. Before the upgrade the agents were running fine. Also the agents where shut down during the Cloudera Manager upgrade. How to start the agents and upgrade them to CDP 7.2.4 now. When trying to restart agents It just shows:$ sudo systemctl status cloudera-scm-agent● cloudera-scm-agent.service - LSB: Cloudera SCM AgentLoaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)Active: active (exited) since Tue 2021-01-12 17:52:10 EST; 7s agoDocs: man:systemd-sysv-generator(8)Process: 14093 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCE

I am new to NiFi, and I am trying to put some data into a postgresql database. I am trying to do it with the ConvertJSONtoSQL processor, unfortunately the JSON keyname and the columename of the table having not the same names (for example keyname = ServiceGroup and columnname = service_group).How can I map the key- and the columnname, so I can store the data in the tablename.

We want to deploy HDP 3.1.5 in production environmentWe have 3 server for masternode and 6 server for workernodeAnd we have plan component layout across 9 nodes above but we want to make sure where we need to place the service-client below 1. Yarn clientsFirst we've plan to install this to 9 nodes, does it okay or just install to 3 master nodes? Because as far as we know, yarn is needed for all nodes include resource managers and node managersOr is it just needed for launch yarn apps or anything else 2. Mapreduce2 clientsSame as above, we plan to install it to 9 nodes because it required for mapreduce jobsDo we need to install across 9 nodes? 3. Hive clientsWe've plan to install it to 3 master nodes, or we just need to install it to a master node?Is it just only needed for submit hive apps from beeline (cli)? 4. infra solr clientsWe just plan to install it to 9 nodes and we dont know enough to know how this client works 5. Kerberos clientsDoes all nodes need ke

HiWe are deploying on premise CDP 7.3 with CDSW 1.8 over RHEL 7.8 which comes with python 2.6 as default. Now we are planning to upgrade python to 3.6. Is there any specific steps I need to take in cluster config beside yum installation.  regards  

Hi All, I am new to solace and spark structured streaming. I would like to write a solace consumer using spark structured streaming and with the below code  var df = spark.readStream.format("solacestream").load I am getting java.lang.ClassNotFoundException: solacestream.DefaultSource Any hint of the format of the solace jms and how to pass in the connection parameters as option to the spark will be helpful. Thanks 

I want to obtain credentials by create support cacse in https://my.cloudera.com. But when i click the "Support caces" in https://my.cloudera.com/support.html , the next page show something : Access to page is restrictedAccessing the requested page requires special permissions. Someone can help me, thanks.

Hello Cloudera support. Is it possible to change the execution engine for hive in HDP-3.0.1.0 to spark?   Thank you,Amit  

Hello Community!Hope you're all doing fine. I'm beggining my Nifi studying and i'de like to replicate a process that i have running in other plataforms.My goal is to read a CSV and process it row by row to insert into a sql server table. So far so good, i was able to do this with the following: GetFile -> SplitRecord -> ConvertRecord (to json) -> ConvertJSONtoSQL -> UpdateAttribute (date conversion) -> PutSQL. But i need to validade if the record already exists in the destiny table. Can anyone help me with that? Already looked for but i wasn't able to find somethig like it. If you guys have a solution that is better and completely different, no problem, i'm open to any change. Thanks in advance!

Hi, i can't compress so many files.txt into a Parquet format to save in HDFS.How can i do that? @ApacheNifi 

I used the PutParquet processor with CSVReader to compress files.txt in parquet format and moving them to HDFS.Checking from the Browse Directory of Hadoop however the files keep .txt Is it normal? Are they saved in parquet format?Thank you! @ApacheNifi Configuration of PutParquet processor: Hadoop:

We made a mistake while configuring the storage of our worker nodes and we ended up adding a data mount point of a really small disk to the yarn.nodemanager.local-dirs property. This causes some of our very data-intensive spark jobs to fail as this disk runs out of space pretty quickly as these jobs generate a lot of staging data. Right now the value of this property is the following: /u01/hadoop/yarn/nm,/u02/hadoop/yarn/nm,/u03/hadoop/yarn/nm,/u04/hadoop/yarn/nm We just want to remove /u04/hadoop/yarn/nm from the property, all the other directories are mounted on disks with enough capacity.So, is it safe to just remove this directory from the yarn.nodemanager.local-dirs property? Or do we have to back up the data in /u04/hadoop/yarn/nm and create another data mount point using a disk with enough capacity and put that backup there?

hican I have the parcels located on an FS so the installation will include a copy instead of downloading from a remote URL?I believe it will be faster solutionthis is for testing cluster environmentsbetter still - can I copy the parcels to the /opt/cloudera/parcels myself - while I'm preparing the VM - will cdp know that the file is already there or do I need to update also the psql metadata db?thanksdoron

i have added on my cluster 1 node that is empty, and I want to add the keytrustee service in it but I encounter this when setting up : The Key Trustee Server service should not be added to a cluster with existing services.   does the key trustee server needs a separate new cluster or just a new empty node where it will be installed is okay?

Hi All, I have a scenario where I will get a number of records from csv file, the task is to read the csv and split each record as one file and save the each file and sheet name with the record name of the 1st column(exclude 1st row as header).   1. What I have done is  read the csv file using Getfile,  2. then used splittext processor, to split each record as one csv file by setting property Header line count to 1. 3. then need to extract the 1st record at column1, use that record(2nd row and 2nd column) value as file name and sheet name for the each individual file   Original csv file: after split the there should two files, one with the name ab123.csv and c35ks.csv and also sheet name also should be changed. ID Description status ab123 Eldon Base for stackable storage shelf, platinum   ab123.csv   ID Description status c35ks 1.7 Cubic Foot Compact "Cube" Office Refrigerators   c35ks.csv   How to get above out puts after the wo

Hi All,  I have nifi flow fthat fetches one json file and using the extractText processor, i want to store entire flowfile content(assume small json file) into one flowfile attribute. But (.*) is not extracting whole json and instead extracting only { . Please help with Java regular expression to extract full json afnd also share if any Java regular expression documentation which i can refer for Nifi extracttext processor.  I having json file which contains format like {"filename1":"insert into filename1","filename2":"insert into filename2","filename3":"insert into filename3"} I will retrieve the filenames in generic way/dynamic using JsonPath(${filename})  in updateattribute . Please assist to extract whole json content into one flow file attribute .  Thanks @Nifi@NifiDev

First let me say I've searched the board and I've seen the responses and I have a dilemma. Before my cluster establishes I can connect to my servers leveraging Initial Admin and the p12 that I've created. I've verified that all of my nodes are in my authorizations.xml file and reference my users.xml accurately as /proxy resources If I'm not attached to a node prior to joining the cluster the error appears like thisAfter clustering is complete attempt to connect yields the following response in the nifi-users.log2021-01-06 20:38:38,206 WARN [NiFi Web Server-157] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response.java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.I don't know where to go from here