Solved

May I know the difference of Parcel and Packages upgrade? and How do i know what method i need to use? Do I based it on how the CDH installed previously?

Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here. Bootstrap log file :2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running2020-04-04

how do i get authentication credentials (username and password) for Ambari repositories ? 

Hi All   In my CDP environment,  Hive Server 2 is failing to start,Below is the log for the same     0-03-25 08:33:56: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported!   of the failure,i am unable to figure out whats causing the failure of the service.

I am trying to get the InvokeHTTP processor to download a csv, the URL is http://www.mambiente.madrid.es/opendata/horario.csv . Problem is, no matter what the options are in the processor, the response is always a 403 error. If I set the "Always Output Response" to true, the response body is: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>403 Forbidden</title></head><body><h1>Forbidden</h1><p>You don't have permission to access /opendata/horario.csvon this server.</p><hr><address>Apache/2.2.22 (Ubuntu) Server at www.mambiente.madrid.es Port 80</address></body></html>   Thing is, a simple "curl --request GET http://www.mambiente.madrid.es/opendata/horario.csv" works just fine, so I have no idea what the problem is.

Hello,   I have recently Kerberized our Hadoop cluster using Apache Ambari and HDFS seems to be working fine. However, YARN ResourceManager and TimelinseServer seem to not be able to communicate:   yarn-yarn-resourcemanager-XXXXXX.log:     2020-03-16 22:58:27,382 ERROR metrics.SystemMetricsPublisher (SystemMetricsPublisher.java:putEntity(549)) - Error when publishing entity [YARN_APPLICATION,application_1581534326709_0003]com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: <a href="<a href="http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn" target="_blank">http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn</a>" target="_blank"><a href="http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn</a" target="_blank">http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn</a</a>>, status: 403, message: org.apache.hadoop.secu

i'm new to NiFi and have developed simple dataflow which would read 3 files from on-prem and upload into S3 bucket. i am using ListFile, FetchFile and PutS3 processor groups for this case. When i trigger dataflow, CPU consumption is spiking >100% and NiFi gets crashed.After few minutes. it automatically comes down to normal. This is happening frequently. My onprem server is Linux and has 8 CPUs. Any idea what's going on  with this.

@MattWho :  I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. . All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix. For example, if today is 20200326,  /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file. /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.   Approach i followed : ListFile: Extract only the files whose name ends with "gz.20200326" UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"     RouteOnAttribute: Extract only the file which matches ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"   But when executing, looks l

Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this.

Hello Manager   We are in the process of implementing a data lake solution for different systems for data ingestion, reporting and analytics purpose.   I tried implementing Hortonworks Ambari 2.7.4 on CentOS 7, on single node. There were quite a few technical challenges faced during several installation attempts. I would like to know if your company offers Ambari installation plans, including pricing details.   Please provide the details about same, so we can analyse it and proceed further accordingly.   Thanks & Regards Harpreet Singh

Hello,   I'm new to Cloudera ..  I was trying to follow the instructions there: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/download_the_ambari_repo_lnx7.html   However, I got stuck with the step of downloading the repo ..  I tried to use my registered username & password in order to access it . but didn't work.   How can I get my access working properly there ? or it's limited to the enterprise (commercial) subscriptions ?

Hi,We are repeatedly getting alerts for NN Pause duration (The health test result for NAME_NODE_PAUSE_DURATION has become bad: Average time spent paused was 44.5 second(s) (74.25%) per minute over the previous 5 minute(s). Critical threshold: 60.00%.) CM / CDH - 5.16.2Current NN Heap Size - 4GBBlock Used - 5TB Request some assistance in fixing this. Thanks Amn   

Hello community!   I have a weird issue where we are on Windows 2016 server that also runs Microsoft SSIS services, and I have installed the Cloudera ODBC Driver for Apache Hive v2.6.4.1004.    When I install the ODBC with my personal account it will configure and test fine, but when we run a SSIS package using a different Run AS account that is not my personal AD account it fails with this error:   ERROR 1300 Connection::SQLDriverConnectW: [Cloudera][DriverSupport] (1030) Failed to decrypt data: Key not valid for use in specified state.   So my solution was to uninstall the ODBC driver on the windows server and re-install it using the Run As Service account we intend to use with SSIS.      I would like to know if this is intentional or I have a desktop driver when i should be using a server driver?   Normally i do not like to log on to servers and install drivers with Service accounts.   Any information would be greatly appreciate

@jsensharma  I followed all the steps but I m not seeing the custom alerts. I tried executing the script but it is throwing below error:    [root@xxxxx host_scripts]# python test_alert_disk_space.pyTraceback (most recent call last):File "test_alert_disk_space.py", line 8, in <module>from resource_management.core.logger import LoggerImportError: No module named resource_management.core.logger     Ambari Version 2.7.3.0 HDF: 3.4.1  

Hi All,   I enabled TLS for NIFI web UI ( CDF ) , while services are running fine on cluster I'm unable to access NIFI web UI from my browser. Below are the steps I followed please suggest what might be causing issue ?   I repeated below steps for all the machines in my nifi cluster   1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem) 2. Copy the JDK cacerts file to jssecacerts as follows:     sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts     3. import rootca cert into JKS store     sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem     4. Created JKS and imported host certificate in keystore.     $JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048

hi all what could be the reason that sometimes we get - REASON: Server not yet listening on http port 8080when we start the ambari serverand sometimes its start successfully ?   ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start............................................................DB configs consistency check: no errors and warnings were found.ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.   # ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/l

Currently I have a problem with configuring the LdapGroupsMapping, I cannot get the groups of the users and it generates the following error message with the command:   $hadoop org.apache.hadoop.security.UserGroupInformation 20/03/19 16:44:20 WARN security.LdapGroupsMapping: Failed to get groups for user c12345a (retry=0) by javax.naming.NamingException: [LDAP: error code 1 - 000004DC: LdapErr: DSID-0C090A4C, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, v3839]; remaining name 'DC=sub,DC=us,DC=domain,DC=local'   $hdfs dfsadmin -refreshUserToGroupsMappingsRefresh user to groups mapping successful   $hdfs groupsc12345a@SUB.US.DOMAIN.LOCAL :   the core-site.xml configuration is as follows -<property><name>hadoop.security.group.mapping</name><value>org.apache.hadoop.security.LdapGroupsMapping</value></property>-<property><name>hadoop.security.group.mapping.ldap.

I followed the instructions here for setting up and configuring the Impala driver on Linux.  When I test the connection using the isql command line utility I get the following error:     [S1000][unixODBC][Cloudera][DriverSupport] (1140) Unable to load CA certificates from certs file location. Please ensure the file exists, has read permission, and has the correct PEM format.[ISQL]ERROR: Could not SQLConnect       My odbc.ini looks like this:      [ODBC Data Sources][hadoop] Driver=Cloudera ODBC Driver for Impala 64-bit AllowHostNameCNMismatch=1 AllowSelfSignedServerCert=1 AuthMec=3 CheckCertRevocation=1 DelegateKrbCreds=0 Host={myhost} KrbFQDN=_HOST KrbRealm={myrealm} KrbServiceName=impala Min_TLS=1.2 Port=21050 ServicePrincipalCanonicalization=1 SSL=1 TrustedCerts=/opt/cloudera/impalaodbc/lib/64/cacerts.pem UseKeytab=0 UseOnlySSPI=1 UseSASL=1 UseSystemTrustStore=1       And the permission on /opt

We have an application continuously writing data with CSV format in a directory of HDFS. In our scenario, the data keep coming but not large in a batch. So the application keeps the files open and continuous writing them. After each batch of writing, it does a hard flush to make the data visible in the files and also increase the size of the files. As a result, there won't be too many small files and with the Impala "Refresh" command, the latest data can be seen immediately with CDH 5.16.1.   However, after the cluster is upgraded to CDH 6.3.1, the Impala "Refresh" command doesn't work. When a new file was created and written some data, the new data could be seen after refreshing, But afterwards, for the new coming data, even though I could directly see them through HDFS command and the size of the file was increased, I couldn't see them through the Impala SQL "Select". Only if the file was closed(the application was terminated), I could see the latest data  through

HDP support is ending in 2021. Is there any documentation for moving from HDP to cloudera distribution?

@MattWho please let me know what is missing    HDF 3.4 NIFI & NIFI Registry Integration (secured)   2 node (nifi1.abc.com, nifi2.abc.com) nifi cluster is secured  1 node (registry.abc.com) nifi registry is secured    generated client certs / server certs for nifi & registtry as below    sh /usr/hdf/current/nifi-toolkit/bin/tls-toolkit.sh standalone -B passwd-C 'CN=nifiadmin, OU=NIFI' -n 'nifi1.abc.com,nifi2.abc.com,registry.abc.com' --nifiDnPrefix 'CN=' --nifiDnSuffix ', OU=NIFI' -o /tmp/certs_ssl/ -K passwd -P passwd-S passwd   able to access registry with client cert (CN=nifiadmin, OU=NIFI) able to access nifi cluster with client cert (CN=nifiadmin, OU=NIFI)   created a bucket in the registry  Added Registry to nifi, but when versioning a processor group getting the below error      2020-03-27 19:31:22,367 INFO [NiFi Registry Web Server-19] o.a.n.r.w.s.NiFiRegistrySecurityConfig Identity in proxy chain

Hii All   When i runt he wget command fr CDP birany file installation,i am getting the below error in the log.can you please confirm why this is happening.   Iam using the below commands.   $ wget https://archive.cloudera.com/cm7/7.0.3/cloudera-manager-installer.bin   $ chmod u+x cloudera-manager-installer.bin   $ sudo ./cloudera-manager-installer.bin   https://archive.cloudera.com/cm7/7.0.3/redhat6/yum/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"Trying other mirror.Error: Cannot retrieve repository metadata (repomd.xml) for repository: cloudera-manager. Please verify its path and try again

I am facing an issue - in PutSQL I am sending a date value from XML-JSON->oracle table.   EvaluateJSOn ->RouteAttributes->AttributeJson->ConvertJsontoSQL -> UpdateAttrubites->PutSQL is my flow. But it is failing always with 

Hi,i want to trigger mail with oozie workflow name and start and end time.

For below sample xml file:<Company><Employee><Id>458790</Id><Name>Sameer</Name><Email>sameer@email.com</Email><Address><HouseNo>105</HouseNo><Street>Grand Road</Street><City>Bangalore</City><State>Karnataka</State><Pincode>560068</Pincode><Country>India</Country><Passport>Available</Passport><Visa/><Contact><Mobile>9909999999</Mobile><Phone>8044552266</Phone></Contact></Address></Employee><Employee><Id>458791</Id><Name>Gohar</Name><Email>Gohar@email.com</Email><Address><HouseNo>485</HouseNo><Street>Camac Street Road</Street><City>Mumbai</City><State>Maharastra</State><Pincode>400001</Pincode><Country>India</Country><Passport>Available</Passport><Visa/><Contac