Q&A

We have installed Spark2 in HDP 3.1 but when we are trying to access spark2 history server UI,we are getting below issue. HTTP ERROR 403Problem accessing /. Reason:java.lang.IllegalArgumentException log: spark-spark-org.apache.spark.deploy.history.HistoryServer-1-xxxxxx.visa.com.out 19/08/28 13:01:21 DEBUG AuthenticationFilter: Request [http://xxxxxxxx:18081/favicon.ico] triggering authentication. handler: class org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler19/08/28 13:01:21 DEBUG AuthenticationFilter: Authentication exception: java.lang.IllegalArgumentExceptionorg.apache.hadoop.security.authentication.client.AuthenticationException: java.lang.IllegalArgumentException at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:306) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536) at org.

I wanted to try with Nifi registry on Ubuntu today, but it failed to start. Nifi works well on the same machine. NiFi registry version is 0.5Java version is 13.0.1 The error log is below:2020-01-04 09:08:45,023 ERROR [main] o.springframework.boot.SpringApplication Application run failedjava.lang.IllegalStateException: Cannot load configuration class: org.apache.nifi.registry.security.authorization.AuthorizerFactoryat org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:413)....Caused by: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:177)at javax.xml.bind.ContextFinder.find(ContextFinder.java:364)at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:508)at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:465)at org.apache.nifi.registry.security.authorization.Author

Hi Guys,   I got a request to copy one of the large hive external table that has 72883 partitions from one database to another database.   I am new to the Hive database so I need your help in copying that table.  I have done the following steps so far and not sure about next step.    Do I need to add those 72883 partitions manually or is there another way to add those partitions to the target table.      Your early response is very much appreciated.   Steps: 1.  created the external table in new databases with the same definition of the source table,but with different location.   2.  Copy the CSV files of source table location  to target table location.       Regards ~Suresh D  

Hello I try to install CDM 6.3.1~1466458.ubuntu1804 on a server A to handle installation of HBASE on a server B. But I have a problem during the step "Install Agents".   I have found some post on the forum which speak about NOPASSWD to add into the sudoers file of the server B, but in my case I can established a ssh connexion from the server A to the server B in command line and run a sudo command without any password, so I think my servers was correctly configured.   Does anyone can help me ?

I have created new notebook in zeppelin but unable to open it. can some one can help on this. Attaching the screenshot

Hi,I Have a hadoop cluster 3.0.1 with 3 journalnodes, 1 nfsgateways node and 6 workernodes. I connected by ssh to the worker nodes today and realised by doing a "df -h" that one a the one local disk (/data/4) is around 94% used on every worker nodes whereas the others disk are between 50% and 65%...  The HDFS status on the another hand is the following:Disk Usage (DFS Used)          44.77%   28.1 TB / 62.8 TBDisk Usage (Non DFS Used)  14.97% 9.4 TB / 62.8 TBDisk Remaining                       40.26%  25.3 TB / 62.8 TB What are the the elements i should check to make sure that a full local disk won't create any issue?

Hi, I would like to route flow files  based on a column's value of a csv file. i have splitted a csv file into many csv files with the content: header and 1 row of the dataframe. so if the inital csv file consisted of 50 rows i will have 50 new csv files. i did this with the SplitRecord processor and a csv-reader/writer. The i have a UpdateAttribute processor to a UUID to these 50 files. Now, i would like to route those files based on a column's value. is this possible? Let's say column number 5 has got numeric values and want to route all flow files with a value bigger than 50. How can i achieve that? Thank you in advance for your help and advice

HiI've installed Apache NiFi through Apache Ambari on my HDP: Stack and Versions->Stack->Install... All configuration while installation I left to defaults. Installation was successful, however I was not able to access "http://sandbox-hdp.hortonworks.com:9090/nifi" out of box. So I added: server {listen 9090;server_name sandbox-hdp.hortonworks.com;location / {proxy_pass http://sandbox-hdp:9090;}} to my "HDP_3.0.1_docker-deploy-scripts_18120587fc7fb\sandbox\proxy\conf.d\http-hdp.conf" file and it started working properly. However now I have a problem when trying to change process configuration: when I click on Configure in process drop-down, I'm redirected to page with text "Please ensure the application is running and check the logs for any errors." In network I can see failed request to "http://sandbox-hdp:9090/nifi-api/processors/23ae900c-016f-1000-ffff-ffff8d289282". Any ideas how it can be fixed? Thanks

I was installing HDF cluster with an existing HDP cluster, the deploy option turned red and an error message arrived it says "If NiFi Certificate Authority is used, nifi.toolkit.tls.token must be set"   "This is a token that will be used by the NiFi Certificate Authority to verify the identity of NiFi nodes before issuing them certificates and by the NiFi nodes to verify the identity of the NiFi Certificate Authority. If relying on NiFi Certificate Authority, set this to a long, random value. For security purposes, password changes will not be shown in configuration version comparisons"     Help me fix this issue

I am a new NiFi and MongoDB user. I'm attempting to create an ingest process that first accepts a JSON file that will provide the information about a project (project submission id, project name, project submittor, project attachments (array of documents). I am able to parse the JSON information until I get to the array of project attachments. The project attachments field is an array with the name of one or more project documents (ex: attachment1.docx, attachment2.ppt, etc). The end goal is to put each project submission in MongoDB as a document. The project attachments field must be an array so I can reference them later when parsing the metadata from each of the attachment files. I want to be able to nest the metadata for each attachment to correspond to each attachment file in the array.    So far, I am the problem I am running into is that the final project attachments array somehow becomes a string instead of an array after passing through the AttributesToJSON processor

Hello, I would like to change my company information in my support profile. Could anyone help me? Thanks a lot!

Aloha. After a recent server migration, all of the hostnames on our cluster got a slight modification. After getting the Cloudera Manager Server to connect to it's required database correctly, it still wants to connect to the old hostnames for everything else in the cluster. Everything is already configured for them to work together, so it should be as simple as updating the URL to connect to the hostname (rather than reinstalling cloudera-scm-agent, etc). Is there a way to do this from within Cloudera Manager?How?

Hello,    I'm trying to pull data from twitter and push it to elasticsearch using nifi. I managed to download and run both of them. Do I need to have the Hortons sandbox installed too? Are there any software or packages I need to have them too?   I will really appreciate if anyone has tutorials or links on how to do it on ubuntu.

hi!, i ve been trying to download Hortonworks Sandbox but i m unable to do so. when ever i click on the download button it redirects me back to main download page.whenever i click on the download now button on the below link for HDP:https://www.cloudera.com/downloads/hortonworks-sandbox.htmlit redirects me back to the below page:https://www.cloudera.com/downloads.html i have been facing this issue since past week. i ll be grateful for help.

nifi registry security - able to login with certs but not getting login page when canceling certs    updated login-identity-provider.xml as below not sure what is missing  <identityProviders><provider><identifier>ldap-provider</identifier><class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class><property name="Authentication Strategy">SIMPLE</property><property name="Manager DN">xxxxxxxx</property><property name="Manager Password">>xxxxxxxx</property><property name="TLS - Keystore"></property><property name="TLS - Keystore Password"></property><property name="TLS - Keystore Type"></property><property name="TLS - Truststore"></property><property name="TLS - Truststore Password"></property><property name="TLS - Truststore Type"></property><property name="TLS - Client Auth"></property><property nam

I'm using CDH 6.2.0. and I found the version of Hive is 2.1.1.    What I want is to upgrade Hive to 3.0 and keep all the Hive data (keep the metastore is enough?), but I can't find any document for upgrading hive in CDH?   Is there any document or tutorial for that? Thanks a lot.   Kind regards.

Hi,  I am working on one of the Kafka use-cases in my project, where the scenario is as follows:Step 1:  I have to update records of a table (lets say CUSTOMER) in web portal Step 2:  A Spark Streaming job will be running which will capture the DES (Data Event Streaming) eventId related to the above. Step 3:  It will connect to the Broker at port 9092, pull the messages, process them and put them as records in one of RDBMS table.  This entire scenario is being executed in Spark Cluster setup in-side Kubernetes.  The same job is running perfectly fine outside Kubernetes.   But, inside Kubernetes, although the job is able to connect to Broker at port 9092 without any issues, it's not reading any real-time events and I am getting a loop of the below messages: 19/12/23 16:00:00.180 DEBUG Fetcher: [Consumer clientId=consumer-1, groupId=test] Added READ_UNCOMMITTED fetch request for partition table-update-0 at offset 102 t

Hi Here, I would like to ask for help on how to upgrade Hive from v.2.1 to v.3.x?Or is it possible to upgrade it manually?Currenty on latest CDH 6.2. Already saw this references;https://www.cloudera.com/documentation/enterprise/latest/topics/hive_install_upgrade_intro.html#hive_guide_install_upgrade_introhttps://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_packaging.html#cdh_packaging_60x

Hi all,   New in NiFi. Hence need guidance on achieving the desired result. Scenario: 1. Multiple .txt log files  2. each .txt log file contains many lines    Requirement: 1. Read each .txt log file and extract only those lines that has "Three.Link resp:". Below snippet for example is from abc.txt     09/10/18 20:06:07.581 DEBUG: (Radio) One.Link DONE09/10/18 20:06:07.963 34, 0, 0, 0, 0, -9909/10/18 20:06:08.591 DEBUG: (Radio) Two.Link request09/10/18 20:06:08.601 DEBUG: (Radio) Three.Link resp: -1 abc-34664 0 10-Sep 12:06:02 0 .00 .00 .00 .00 0 0 0 -99 -9909/10/18 20:06:08.601 DEBUG: (Radio) One.Link DONE09/10/18 20:06:08.983 34, 0, 0, 0, 0, -9909/10/18 20:06:09.600 DEBUG: (Radio) Two.Link request09/10/18 20:06:09.611 DEBUG: (Radio) Three.Link resp: -1 abc-34664 0 10-Sep 12:06:03 0 .00 .00 .00 .00 0 0 0 -99 -9909/10/18 20:06:09.611 DEBUG: (Radio) One.Link DONE     What I am trying to do is to extract only those lines from abc.txt which contains

Hi, guys:   I'm useing CDH6.1.1. When I query impala like "select sum([field]) from [table]", the impala daemon is correct,and the result is correct too. But when I change sql to "select sum([field]) from [table] group by", the impala daemon unexpected exit，and the ERROR log has nothing, but the "impala.FATAL" throw "F1222 llvm-codegen.cc:137] LLVM hit fatal error: Cannot select: intrinsic %llvm.x86.sse42.crc32.64.64". Thanks everyone for your help！log

If you are using distcp command for transferring data from one cluster to another cluster on regular basis in this scenario only new data will be copied on daily basis so how distcp keep tracks on it?

I am trying to repeat the work of Change Data Capture (CDC) with Apache NiFi. My testing environment is : Ubuntu 16.04;  MySQL and NiFi are installed in the OS system without using Docker, can access to MySQL via the terminal from another Windows PC in the same local network. However, when I just use the processor CaptureChangeMySQL and connect to the MySQL, it shows me the error:    The settings of the NiFi processors are: CDC MapCache: CaptureChangeMySQL:   Ubuntu 16.04:  Could anyone help to address the issues? Thank you very much.    

Hello   I am trying to add an ExecuteGroovyScript processor. A simple example test works fine, but when I try to add my script it does not initiate. Exception (on hoover over) is really brief: InstantiationException thrown by groovy invokehelper. Script uses some external jars. As far as I can see I added those to the classpath.    I am probably doing something stupid but I was wondering how to debug this issue? The log files do not seem to contain anything wrt this error which makes troubleshooting very difficult   All tips warmly welcomed!   Tx   Peter

Hello,   Iwant to execute .py from nifi, I am able to do that but when saving data in mongo from the script (using pymongo) I got an error: Executable command python3 ended in an error: Traceback (most recent call last): 'MYpath/MY.py', in line 104, in <module>  mycol.insert(document)   If I execute this .py in command is working fine.   Command Arguments Strategy   Command Arguments Property   Command Arguments   MYpath/MY.py;1000000000   Command Path   python3   Ignore STDIN   true   Working Directory   path   Argument Delimiter   ;   Output Destination Attribute   No value set   Max Attribute Length   200

Hi community,   Beyond the ConvertExcelToCSVProcessor, does Nifi offer other processors specific to MS Excel (.xls and/or .xlsx) like for example a specific processor to parse MS Excel files?   I know that is possible to do (and I already did) it with scripting (using for example the xlrd module of Python). However, here, I ask about the existence of pre-configured Nifi processors for MS Excel format.    Thanks in advance!