Q&A

I'm passing a csv flowfile to ExecuteStreamCommand. How can I zip it with password and pass it to the next processor?   I don't know how to zip file from stdin and pass it to stdout. I'm using python

Raw data will be transferred from the source through an  HandleHttpRequest. How can I encrypt in motion this traffic?

I am logged into Ambari with 'admin' user.   Then I navigated to File View >> 'user' directory  >> Create a New Folder >> Enter name of folder and click on 'Add' button. But I am getting a server message: -   Permission denied: user=admin, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x   How can I provide write permission on user directory to write something or add a folder to it. Request to community to help me. Thanks.

All, while installing cdsw tcp-ingress controller and ds_reconsiler pods are failing due to connection failure with Grps server.. from below web service logs i see grps server status failing with port binding (description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,"grpc_status":14,"os_error":"invalid argument"}]}]}), any suggestions how to fix this...   [..] reated":"@1568398264.632135163","description":"No address added out of total 1 resolved","file":"../deps/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":349,"referenced_errors":[{"created":"@1568398264.632132593","description":"Failed to add port to server","file":"../deps/grpc/src/core/lib/iomgr/tcp_server_custom.cc","file_line":402,"referenced_errors":[{"created":"@1568398264.632131142","description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,"grpc_status":14,"os_error":"invalid argument"}]}]}E0913 18:11:0

Hi,   I tried Hortonworks HDP 3.0 image using oracle virutalbox. But the root/hadoop is not working for SSH access or browser ssh access? Need help in getting correct password or help in resetting it.   Thanks, Kalyan

Hi , I am running free trial cloudera in pseudo-distributed mod (One host, the server and agent are the same).i am trying now to enable tls/ssl for my cluster -Level 1 succedeed https://datastore.xxx.fr:7183 tuns correctly with no problem    When i configure level N2 of tls/ssl ,cloudera manager agent fails to connect !. In CM server fails to have heartbeat: when i checked :tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log I got ERROR : File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 292, in connect_sslreturn m2.ssl_connect(self.ssl, self._timeout)SSLError: certificate verify failed   The agent config files : server_host=datastore.xxx.fr server_port=7182   # Use TLS and certificate validation when connecting to the CM server.use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/x509/cmhost.pem   ---the cmhost.pem get my certifcate correctly ---when i run :   sudo openssl s_client -connect dat

Hello, how about everything I have a problem in hive at the moment of executing a query in hive with Hortonwork the next error I have is   SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskSQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskError while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask  

I have an EC2 instance running with the IPv4 Public IP 11.111.111.111.  The first thing I tried was to get NiFi running.     docker run --name nifi-standalone\ -p 8080:8080 \ -d \ apache/nifi:latest      At 11.111.111.111.111:8080/nifi/ I can reach NiFi. Great.   Now I wanted to setup a secure NiFi instance. Step 1: Create certificates with the toolkit.     ./bin/tls-toolkit.sh standalone -n '11.111.111.111' -C 'CN=admin,OU=nifi' -B SuperSecretPassword -o './standalone'       Step 2: Move keystore.jks and truststore.jks into a specific folder (here /home/ec2-user/project/nifi-standalone/certs).   Scenario 1: set NIFI_WEB_HTTP_HOST  I run the following docker command.     docker run --name nifi-ssl \ -v /home/ec2-user/project/nifi-standalone/certs:/opt/certs \ -v /home/ec2-user/project/nifi-standalone/conf:/opt/conf \ -p 8443:8443 \ -e NIFI_WEB_HTTPS_HOST=11.111.111.111 \ -e AUTH=tls \ -e KEYSTORE_P

Hi,   If I try and get the logs for an application like this: yarn logs -applicationId application_1575531060741_10424 The command fails because I am not running it as the application owner.  I need to run it like this: yarn logs -applicationId application_1575531060741_10424 -appOwner hive   The problem is I want to write out all the yarn logs to the os so I can ingest them into splunk.  If I try and figure out the appOwner for each application then this is awkward and time consuming even in a script.     Is there a better way to dump all the yarn logs to the os ?   Thanks      

Hi, We were trying to enable TLS on our cluster.We have followed cloudera steps ,but i think missed something.After enabling TLS,we were able to login to port 7183,but all  cloudera management services were disabled.   So we reverted back TLS changes from cloudera and removed all keys and certificates.After that given restart to the cloudera manager.   After this process,port 7180 started and login page came.But when i am trying to login then it is not redirecting to the GUI. when i have check the logs. then these exceptions are showing.   2019-12-17 13:24:29,320 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.2019-12-17 13:24:30,850 WARN ActionablesProvider-0:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORINGcom.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running   2019-12-17 13:24:50,238 WARN E

I have installed the HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) using Docker and Kerberized the cluster.   This works, i can kinit myuser and make an API request to Atlas:     curl --negotiate -u : sandbox-hdp.hortonworks.com:21000/api/atlas/admin/version       Now I would like to create a Ranger policy that includes my Kerberos user, however the autocomplete for users does not show any of my Kerberos users... I did read this post (https://community.cloudera.com/t5/Support-Questions/Autocompletion-of-names-not-working-in-Ranger/td-p/109988), however this problem seems to be slightly different.   What do I need to do to make Kerberos users show up in the Ranger policy creation UI?

Hello, I have some problems with NIFII made a simple dataflow, but it doesn't work ( getfile takes file it's okwhen update attribute  it's ok too and PutDataBaseRecord doesn't take any IN information what is the reason???I made a table in my postgresql database with the same attributes names   lately i began to take errors like that 

Hi Team, Trying to follow the Tutorial in Cloudera Search deployment for  CDH 6.2   Everything works until this command: solrctl collection --create test_collection -s 2 -c test_collection_config   we get the exception: { "responseHeader":{ "status":0, "QTime":33610}, "failure":{ "gis18.example.com:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://gis18.example.com:8983/solr: Error CREATEing SolrCore 'test_collection_shard1_replica_n1': Unable to create core [test_collection_shard1_replica_n1] Caused by: Permission denied: user=solr, access=WRITE, inode=\"/\":root:supergroup:drwxr-xr-x\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)\n\tat or

Hi,   I want to add headers to a csv file. I have used Replace text processor for this purpose. But final output just append headers to the first line. eg: Sample : 2-Dec-15,Mumbai,120 Expected output : date,city,temp2-Dec-15,Mumbai,120 Output : date,city,temp2-Dec-15,Mumbai,120   I have tried both ways by adding '\n' and 'shift+enter' after the header line   id,name,site,score\n   id,name,site,scoreshift+enter as the replacement value, but no luck instead \n and shift+enter was taken as a value.   Could anyone help to resolve this issue.

Hi, I'm newbie in NIFI and also in java services  yesterday I downloaded nifi-1.10.0 bin version  i run run-nifi.bin  and after i started to work with NIFI  I made a connection to postgree database but it doesn't work   also my colleague made the same thing and everything is ok. what can goes wrong with my NI FI ?   When I use this password in DBeaver  everything is OK. what have I do to make it right way?

https://community.hortonworks.com/articles/38149/how-to-create-and-register-custom-ambari-alerts.html doesn't seems to be working with ambari 2.7.1, stuck at step 3)No exception in ambari server or agent logs@Jay Kumar SenSharma

I am running following command in Zeppelin.First created hive context with following code - val hiveContext = new org.apache.spark.sql.SparkSession.Builder().getOrCreate() then I tried to load a file from HDFS with following code -val riskFactorDataFrame = spark.read.format("csv").option("header", "true").load("hdfs:///tmp/data/riskfactor1.csv")but I am getting following error message "org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://sandbox-hdp.hortonworks.com:8020/tmp/data/riskfactor1.csv;"I am quite new in Hadoop. Please help me figure out what wrong I am doing. 

Disclaimer: crosspost fromhttps://stackoverflow.com/questions/59359158/apache-nifi-1-10-how-to-execute-a-python-script-as-a-processor   In NiFi <= v1.9 I used the following NiFi setup: `GetTwitter` generates a JSON for every tweet gathered and attached to it there used to be a `ExecuteStreamCommand` that has the following parameters: This would allow me to get the flowfile as input, and I could analyse the content and do something (in this specific case I would put the content on Elasticsearch) - You could reach the same result if you do something like this `cat flowfile.json | python <scriptname>.py` But now in >= v1.10 it seems I cannot do the same. it gives me some error about parameters (it seems it wants to use the flowfile as parameter on calling the command/script instead, as it used to be, pass the flowfile as input.It gives me the following error:     2019-12-16 14:28:44,953 ERROR [Timer-Driven Process Thread-2] o.a.n.p.standard.ExecuteStreamCommand

Hi, I am trying to run nexgtech/knox as a container in a pod in a kubernetes cluster and i am getting the below   Exception in thread "main" java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.knox.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:68)at org.apache.knox.gateway.launcher.Invoker.invoke(Invoker.java:39)at org.apache.knox.gateway.launcher.Command.run(Command.java:99)at org.apache.knox.gateway.launcher.Launcher.run(Launcher.java:75)at org.apache.knox.gateway.launcher.Launcher.main(Launcher.java:52)Caused by: java.lang.NoSuchFieldError: DEFAULT_XML_TYPE_ATTRIBUTEat org.eclipse.persistence.jaxb.compiler.MappingsGenerator.setupInheritance(MappingsGenerator.java:2431)at org.eclipse.

  Hello Folks,    I am trying to debug Ambari alerts , I am not getting logs generated in /var/log/ambari-server/ directory.   Ambari Version 2.5.1.0   HDP -  1) Tried ambari-server restart  2) enabled debug mode mentioned in    tried steps mentioned in  https://community.cloudera.com/t5/Support-Questions/Not-able-to-receive-emails-via-Ambari-Alerts/td-p/122948   http://mail-archives.apache.org/mod_mbox/ambari-user/201504.mbox/%3CA0F1A095-D4F9-47CE-BEC7-12C4EB54E5EC@hortonworks.com%3E   @nsabharwal- Could you please provide some ways to debug      

Hello,  I create a new cluster with relaxed security and disabled TLS on all services. I'm able to connect on Hive/Impala using JDBC Connection (Squirrel), with no user and password using this conection string:  Impala: jdbc:impala://Impala_Daemon_Host:21050/default. Impala: jdbc:hive2://Hive_Gateway_Host:10000/default   Now I activated TLS, and create a user on SO (newuser1).  Now, I'm trying to connect using the same JDBC connection, and using the new user and password, with no sucess.   Any advice are welcome.  

Hello All,   I would need to use Path filter for searching subfolders starting by AX. AX1/ AX2/ XY1/ XY2/   I tried this in path filter: AX.*/   Is that correct? Thanks everybody

Is there a way to run spark-submit (spark v2.3.2 from HDP 3.1.0) while in a virtualenv? Have situation where have python file that uses python3 (and some specific libs) in a virtualenv (to isolate lib versions from rest of system). I would like to run this file with /bin/spark-submit, but attempting to do so I get... [me@myserver tests]$ source ../venv/bin/activate; /bin/spark-submit sparksubmit.test.py File "/bin/hdp-select", line 255 print "ERROR: Invalid package - " + name ^SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)?ls: cannot access /usr/hdp//hadoop/lib: No such file or directoryException in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx at org.apache.spark.launcher.Main.main(Main.java:118)# also tried...(ve

Dear All.   I having issue with creating hive table with hdfs data in Azure HDInsight cluster.   i am able to create hive table from BLOB Storage but not with HDFS Data.   as below example code for taking data from blob storage:- hive> CREATE EXTERNAL TABLE IF NOT EXISTS tsvtab (> name string,> region_code int,> sal int,> add string> )> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'> STORED AS TEXTFILE; load data inpath 'wasb://hadoophdinsightigi-2019-10-21t07-33-15-078z@hadooighdistorage.blob.core.windows.net/user/sshuser/data.tsv' into table tsvtab;   below taking data from hdfs but not working:-   hive> CREATE EXTERNAL TABLE IF NOT EXISTS tsvtab (> name string,> region_code int,> sal int,> add string> )> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'> STORED AS TEXTFILE; load data inpath 'wasb://hadoophdinsightigi-2019-10-21t07-33-15-078z@hadohdistorage.blob.core.windows.net//user/HadoopPOCDir/data.tsv' into ta

Hi, We are currently using merge content processor to merge kafka messages with "minimum number of entries" = 500. when 500 messages reached it will merge as single file. That use case works fine. whenever end of the day at 11:59, is there any queue pending in merge content processor of kafka messages needs to dynamically pushed before start to the new date. Kindly help me out from this kind of use case. Workflow :kafka_consumer -> merge_content_prcessor -> puthdfs@mburgess