Q&A

Hi Experts.  We recently upgraded our Nifi Version from 1.9.2 to 1.11.4 however many of our flows arent working as some processors arent available in the new version. One of them being the "Infer Avro Schema" . I am not sure why this should be the case as every new release should be backward compatible.  Is there a way to restore this processor onto the latest version as our code is impacted.  We can also see that this has been an issue and raised separately as well. Please find the link here http://apache-nifi-users-list.2361937.n4.nabble.com/GRPC-processors-missing-after-upgrade-to-1-11-3-td9161.html#a9166 cheers Abhinav

Hello, everyone,I'm trying to set up user authentication by username/password in Apache NiFi, but without using Kerberos or LDAP.her used nifi toolkit to generate ssl certificates, i have started nifi by https. but i want you to ask me to log in as minimally as possible without installing extras.is this possible? or is it required to have an external user management system? * i am using Ubuntu localhost, one node in my laptop for instance

In Data Science Workbench, I would like to create one file with a number of types/functions defined. Then, in a separate file, I'd like to be able to import those types/functions (the first file is essentially acting as a library that can be shared among many other scripts.) How do I go about this? In the docs, I see a mention of it being possible, but no example and the few obvious things I've tried haven't worked, so I'm clearly overlooking something simple.   Thanks in advance for any help!   - J.

Team , HUE UI caches username password .i have to restart HUE to prompt username and password . I made below change but it did not change . # UI customizations# -------------------[[custom]]# Top banner HTML code# e.g. (H4)Test Lab A2 Hue Services(/H4)## banner_top_html=# Login splash HTML code# e.g. WARNING: You are required to have authorization before you proceed## login_splash_html=## logo_svg=cacheable_ttl=0  RegardsBharad

There events error is: "Operation timed out. Template install timed out with this command id: 1546332222"Here's the complete sequence of events:   Operation timed out. Template install timed out with this command id: 15463322224/28/2020, 2:10:09 PMBuilding cluster; Cluster manager ip:xxx.xxx.xxx.xxx4/28/2020, 1:21:37 PMRetrying Create cluster operation from the failed step.4/28/2020, 1:21:37 PMCloudera Manager reported that node(s) data-lake-eastern-us-master0.dev-test.udum-f5f2.cloudera.site,data-lake-eastern-us-idbroker1.dev-test.udum-f5f2.cloudera.site status became HEALTHY.4/28/2020, 1:12:44 PMOperation timed out. Template install timed out with this command id: 15463322224/28/2020, 1:12:19 PMBuilding cluster; Cluster manager ip:xxx.xxx.xxx.xxx4/28/2020, 12:23:36 PMStarting cluster services4/28/2020, 12:21:14 PMStarting cluster services4/28/2020, 12:19:14 PMSetting up infrastructure metadata4/28/2020, 12:17:34 PMBootstrapping infrastructure cluster4/28/2020, 12:17:09

how to create new keytab if my previous keytab is expired?

I'm trying to run a sed command through a ExecuteCommandStream but I get the error pasted below. The ExecuteCommandStream has the input from a EvaluateXPath processor. How do I give the flowfile argument? I'm getting this error: ExecuteStreamCommand[id=9c7cfd48-0171-1000-1a59-064ce951124a] Transferring flow file StandardFlowFileRecord[uuid=e448ca5a-8dc8-4754-a0d4-549db92d1f79,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1587501858734-9104, container=default, section=912], offset=375435, length=-1],offset=0,name=DL_prelchim1971_20190302_142639_20190302_145642.xml,size=0] to nonzero status. Executable command sed ended in an error: Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]. 

I have scenario where I am getting the file(stream:flowfile of NIFI) as of type csv file, then creating the dataframe and dumping it thats it. But after creating the dataframe the structure of the file got disturbed, if I open the same flowfile on my disk i could see clear structure with columns separated with tab, but with python dataframe I am not getting the same structure, if I get same structure i can perform row manipulation. Here What I am doing:1: using ExecuteSQL processor, I am getting database record,2: then passing this record to ConvertRecord processor to convert this avro record type csv file separated by tab.convertRecordSetWriter settings...The output of the flowfile is ...3: Then the reading flowfile(step 2 folowfile) as python data using ExecuteStreamCommand, coz I am want to perform some action on the database record, to do this my record structure has been changed in data frame. 

I have Kerberos and sentry enabled for the cluster. Solr is also configured through sentry. The hue admin user is having all the access.   While testing out the collection creation and indexing using the hue indexes. I am getting the error 403. But I can see that collection being created, somehow the user which created the collection does not have access over the collection. Any other permission I need to add for this.  

Hi, After recreating hadoop cluster on HDInsight and connecting to the external metastore I get an error "Custom Ambari Metastore DB must be empty". Why I get this error? The goal was to keep all the metadata in external DB in case to preserve these data every time when I delete and recreate the cluster.

 i have got the below details other than normal host url,POST /j_security_checkContent-type: application/x-www-form-urlencodedj_username=<userid>&j_password=<password> 

hi all     we have two master machines    master01 master02     on both machines yarn resource manager is running as active/standby )   how to verify which machine is with active resource manager by ambari rest api ? 

 hi all the following ambari API will stop only the data-nodes on workers machines # curl -s -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop DataNodes","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME"},"query":"HostRoles/component_name=DATANODE&HostRoles/host_name.in(dn1.example.com,dn2.example.com,dn3.example.com)&HostRoles/maintenance_state=OFF"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' "h ttp://$AMBARI_FQDN:8080/api/v1/clusters/$CLUSTER_NAME/host_components" now we want to replace the list - n1.example.com,dn2.example.com,dn3.example.com with variable - DATANODES=n1.example.com,dn2.example.com,dn3.example.com so $DATANODES will replce the list in the API as the following  # curl -s -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop DataNodes","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME"},"query":"HostRoles/compone

Hi I appeared for exam yesterday its been 24 hours but no news from cloudera yet on the Result of my exam..How much time does it take to notify Result... Tried to search this on website but no luck neither getting a reply on email Please advise.Thanks,

hi all  in ambari we have the following hive components   how to stop Hive Metastore and HiveServer2 by cli  ?

This is a partition move from one volume to another in a single broker.  I have started a kafka-reaasign-partitions of a partition from one volume to another but it never completes. Has anyone faced this ? Kafka cluster is in a healthy state Controller is fine source and destination is also fine The replica is not a leader. So far havent seen any errors in logs  The directories look like the below. Here i am moving a partition from sdc to sdf volume. inodes also look fine. desind@xxx:~#> sudo du -sh /kafka/data/sdc/prod-events-4876G /kafka/data/sdc/prod-events-48 desind@xxx:~#> sudo du -sh /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future76G /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-futuredesind@xxx:~#> find /kafka/data -type d -name 'prod-events-48*'/kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future/kafka/data/sdc/prod-events-48   

I am trying to create hive table to read data from kafka topics.I am using CDH 6.2.0.I am adding the below jar before creating the table : kafka-handler-3.1.0.3.1.0.0-78.jar;hive-serde-0.10.0.jar;hive-metastore-0.9.0.jar; below is the create table statement: CREATE EXTERNAL TABLE kafka_table  (`timestamp` timestamp , `page` string,  `newPage` boolean,  added int, deleted bigint, delta double)  STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'  TBLPROPERTIES  ("kafka.topic" = "topic-name", "kafka.bootstrap.servers"="hostname:9092");  giving me below error:  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.DefaultHiveMetaHookorg/apache/hadoop/hive/metastore/DefaultHiveMetaHookError: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0) 

Hi All, We're currently running CDH 6.3.2 and trying to get Python 3.6.9 working on Spark working nodes. So far without success, though... As our cluster is built on RHEL7, default Python version is 2.7, which is EOL and doesn't have all the necessary libraries/modules. Original idea was to install Miniconda3 on all the nodes, create the environment py36 and point PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to the proper python executable. Apparently this doesn't work as expected, any PySpark job breaks with error message: Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings'Despite the fact that module "encodings" is out there in py36 environment. If we add spark.executorEnv.PYTHONPATH=/opt/miniconda3/envs/py36/lib/python3.6:/opt/miniconda3/envs/py36/lib/to spark-defaults.conf file in the project root, we becoming different error message:Traceback (most recent call last): File "/opt/miniconda3/en

I wanted to create a cluster in Ambari but unfortunately at the "Confirm hosts" step I could not register my nodes due to following exception:Exception: Registration failed due to: Cannot register host with not supported os type, hostname=ip6-localhost, serverOsType=ubuntu18, agentOsType=ubuntu18Is it because i am using ubuntu 18.04?Thank you.

I am trying to insert a document into mongodb using apache NiFi. I want to insert a document just from Flow File attributes but not the Flow File content.I have tried using PutMongo processor but it insert the content of the Flow file which I don't want it to insert. I need to insert a custom document by using the flow file's attributes.

Attached is the java stack. I am getting this error in the nifi.log whenever i am starting GetFile processor which is first processort in my list.    2020-04-07 01:17:01,371 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=51048885-0171-1000-9a41-00f55a11ae71] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IndexOutOfBoundsException: Index: 0, Size: 0: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.lang.IndexOutOfBoundsException: Index: 0, Size: 0at java.util.ArrayList.rangeCheck(ArrayList.java:657)at java.util.ArrayList.get(ArrayList.java:433)at org.apache.nifi.processors.standard.InvokeHTTP.setUpClient(InvokeHTTP.java:665)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:

Hello,I want to do a passwordless ssh login to my localhost.I tried the following steps many times but it asks for password when i try "ssh localhost".1. ssh-keygen2. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys3. chmod 700 ~/.ssh4. chmod 600 ~/.ssh/authorized_keysThis is screenshot of my terminal.I tried the same steps to login from localhost to udhav.fqdn.com, but even now it promts me for password. For the second case i followed steps provided in this link.https://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/I guess i am doing small mistake. Can anyone help me with this?Thank you.  

Hi All My Hive Server 2 is failing to start,Below is the complete log file of the failure,i am unable to figure out whats causing the failure of the service. 2020-03-25 08:33:56: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported! at org.apache.hadoop.hive.conf.HiveConf.validateExecutionEngine(HiveConf.java:5594) at org.apache.hadoop.hiv

Test CaseSetup  create table n1 (id STRING)stored as parquet;insert into n1 values ('-1234');insert into n1 values ('7890');  Resultsimpalad version 2.12.0-cdh5.15.2  SELECT `id`, CAST(`id` AS DECIMAL(38,0)) / 20 * 20 AS D0 , CAST(`id` AS DECIMAL(38,0)) / 20.0 * 20 AS D1FROM n1ORDER BY `id`;+----------+-------+---------+| id | d0 | d1 |+----------+-------+---------+| -1234 | -1220 | -1234.0 || 7890 | 7880 | 7890.0 |+----------+-------+---------+Fetched 4 row(s) in 0.56s  When we are casting the "id" column to a decimal without scale and dividing by an integer, even though the answer should have a scale (-1234 / 20 = -61.7) we lose the scale and the result is truncated to -61. This is then multiplied by 20 to give -1220 which is column d0. This does not get us back to the same value that we started with which was -1234. When we divide by a decimal value (20.0) the result of -61.7 is preserved, which gets

exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141)Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.la