Q&A

@MattWho :  I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. . All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix. For example, if today is 20200326,  /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file. /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.   Approach i followed : ListFile: Extract only the files whose name ends with "gz.20200326" UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"     RouteOnAttribute: Extract only the file which matches ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"   But when executing, looks l

Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this.

Hello Manager   We are in the process of implementing a data lake solution for different systems for data ingestion, reporting and analytics purpose.   I tried implementing Hortonworks Ambari 2.7.4 on CentOS 7, on single node. There were quite a few technical challenges faced during several installation attempts. I would like to know if your company offers Ambari installation plans, including pricing details.   Please provide the details about same, so we can analyse it and proceed further accordingly.   Thanks & Regards Harpreet Singh

Hello,   I'm new to Cloudera ..  I was trying to follow the instructions there: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/download_the_ambari_repo_lnx7.html   However, I got stuck with the step of downloading the repo ..  I tried to use my registered username & password in order to access it . but didn't work.   How can I get my access working properly there ? or it's limited to the enterprise (commercial) subscriptions ?

Hi,We are repeatedly getting alerts for NN Pause duration (The health test result for NAME_NODE_PAUSE_DURATION has become bad: Average time spent paused was 44.5 second(s) (74.25%) per minute over the previous 5 minute(s). Critical threshold: 60.00%.) CM / CDH - 5.16.2Current NN Heap Size - 4GBBlock Used - 5TB Request some assistance in fixing this. Thanks Amn   

Hello community!   I have a weird issue where we are on Windows 2016 server that also runs Microsoft SSIS services, and I have installed the Cloudera ODBC Driver for Apache Hive v2.6.4.1004.    When I install the ODBC with my personal account it will configure and test fine, but when we run a SSIS package using a different Run AS account that is not my personal AD account it fails with this error:   ERROR 1300 Connection::SQLDriverConnectW: [Cloudera][DriverSupport] (1030) Failed to decrypt data: Key not valid for use in specified state.   So my solution was to uninstall the ODBC driver on the windows server and re-install it using the Run As Service account we intend to use with SSIS.      I would like to know if this is intentional or I have a desktop driver when i should be using a server driver?   Normally i do not like to log on to servers and install drivers with Service accounts.   Any information would be greatly appreciate

@jsensharma  I followed all the steps but I m not seeing the custom alerts. I tried executing the script but it is throwing below error:    [root@xxxxx host_scripts]# python test_alert_disk_space.pyTraceback (most recent call last):File "test_alert_disk_space.py", line 8, in <module>from resource_management.core.logger import LoggerImportError: No module named resource_management.core.logger     Ambari Version 2.7.3.0 HDF: 3.4.1  

Hi All,   I enabled TLS for NIFI web UI ( CDF ) , while services are running fine on cluster I'm unable to access NIFI web UI from my browser. Below are the steps I followed please suggest what might be causing issue ?   I repeated below steps for all the machines in my nifi cluster   1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem) 2. Copy the JDK cacerts file to jssecacerts as follows:     sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts     3. import rootca cert into JKS store     sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem     4. Created JKS and imported host certificate in keystore.     $JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048

hi all what could be the reason that sometimes we get - REASON: Server not yet listening on http port 8080when we start the ambari serverand sometimes its start successfully ?   ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start............................................................DB configs consistency check: no errors and warnings were found.ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.   # ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/l

Currently I have a problem with configuring the LdapGroupsMapping, I cannot get the groups of the users and it generates the following error message with the command:   $hadoop org.apache.hadoop.security.UserGroupInformation 20/03/19 16:44:20 WARN security.LdapGroupsMapping: Failed to get groups for user c12345a (retry=0) by javax.naming.NamingException: [LDAP: error code 1 - 000004DC: LdapErr: DSID-0C090A4C, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, v3839]; remaining name 'DC=sub,DC=us,DC=domain,DC=local'   $hdfs dfsadmin -refreshUserToGroupsMappingsRefresh user to groups mapping successful   $hdfs groupsc12345a@SUB.US.DOMAIN.LOCAL :   the core-site.xml configuration is as follows -<property><name>hadoop.security.group.mapping</name><value>org.apache.hadoop.security.LdapGroupsMapping</value></property>-<property><name>hadoop.security.group.mapping.ldap.

I followed the instructions here for setting up and configuring the Impala driver on Linux.  When I test the connection using the isql command line utility I get the following error:     [S1000][unixODBC][Cloudera][DriverSupport] (1140) Unable to load CA certificates from certs file location. Please ensure the file exists, has read permission, and has the correct PEM format.[ISQL]ERROR: Could not SQLConnect       My odbc.ini looks like this:      [ODBC Data Sources][hadoop] Driver=Cloudera ODBC Driver for Impala 64-bit AllowHostNameCNMismatch=1 AllowSelfSignedServerCert=1 AuthMec=3 CheckCertRevocation=1 DelegateKrbCreds=0 Host={myhost} KrbFQDN=_HOST KrbRealm={myrealm} KrbServiceName=impala Min_TLS=1.2 Port=21050 ServicePrincipalCanonicalization=1 SSL=1 TrustedCerts=/opt/cloudera/impalaodbc/lib/64/cacerts.pem UseKeytab=0 UseOnlySSPI=1 UseSASL=1 UseSystemTrustStore=1       And the permission on /opt

We have an application continuously writing data with CSV format in a directory of HDFS. In our scenario, the data keep coming but not large in a batch. So the application keeps the files open and continuous writing them. After each batch of writing, it does a hard flush to make the data visible in the files and also increase the size of the files. As a result, there won't be too many small files and with the Impala "Refresh" command, the latest data can be seen immediately with CDH 5.16.1.   However, after the cluster is upgraded to CDH 6.3.1, the Impala "Refresh" command doesn't work. When a new file was created and written some data, the new data could be seen after refreshing, But afterwards, for the new coming data, even though I could directly see them through HDFS command and the size of the file was increased, I couldn't see them through the Impala SQL "Select". Only if the file was closed(the application was terminated), I could see the latest data  through

HDP support is ending in 2021. Is there any documentation for moving from HDP to cloudera distribution?

@MattWho please let me know what is missing    HDF 3.4 NIFI & NIFI Registry Integration (secured)   2 node (nifi1.abc.com, nifi2.abc.com) nifi cluster is secured  1 node (registry.abc.com) nifi registry is secured    generated client certs / server certs for nifi & registtry as below    sh /usr/hdf/current/nifi-toolkit/bin/tls-toolkit.sh standalone -B passwd-C 'CN=nifiadmin, OU=NIFI' -n 'nifi1.abc.com,nifi2.abc.com,registry.abc.com' --nifiDnPrefix 'CN=' --nifiDnSuffix ', OU=NIFI' -o /tmp/certs_ssl/ -K passwd -P passwd-S passwd   able to access registry with client cert (CN=nifiadmin, OU=NIFI) able to access nifi cluster with client cert (CN=nifiadmin, OU=NIFI)   created a bucket in the registry  Added Registry to nifi, but when versioning a processor group getting the below error      2020-03-27 19:31:22,367 INFO [NiFi Registry Web Server-19] o.a.n.r.w.s.NiFiRegistrySecurityConfig Identity in proxy chain

Hii All   When i runt he wget command fr CDP birany file installation,i am getting the below error in the log.can you please confirm why this is happening.   Iam using the below commands.   $ wget https://archive.cloudera.com/cm7/7.0.3/cloudera-manager-installer.bin   $ chmod u+x cloudera-manager-installer.bin   $ sudo ./cloudera-manager-installer.bin   https://archive.cloudera.com/cm7/7.0.3/redhat6/yum/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"Trying other mirror.Error: Cannot retrieve repository metadata (repomd.xml) for repository: cloudera-manager. Please verify its path and try again

I am facing an issue - in PutSQL I am sending a date value from XML-JSON->oracle table.   EvaluateJSOn ->RouteAttributes->AttributeJson->ConvertJsontoSQL -> UpdateAttrubites->PutSQL is my flow. But it is failing always with 

Hi,i want to trigger mail with oozie workflow name and start and end time.

For below sample xml file:<Company><Employee><Id>458790</Id><Name>Sameer</Name><Email>sameer@email.com</Email><Address><HouseNo>105</HouseNo><Street>Grand Road</Street><City>Bangalore</City><State>Karnataka</State><Pincode>560068</Pincode><Country>India</Country><Passport>Available</Passport><Visa/><Contact><Mobile>9909999999</Mobile><Phone>8044552266</Phone></Contact></Address></Employee><Employee><Id>458791</Id><Name>Gohar</Name><Email>Gohar@email.com</Email><Address><HouseNo>485</HouseNo><Street>Camac Street Road</Street><City>Mumbai</City><State>Maharastra</State><Pincode>400001</Pincode><Country>India</Country><Passport>Available</Passport><Visa/><Contac

Hi,   I did nifi and mysql setup in stand alone machine, where I have scenario the input will be in excel form, i need to create table in mysql db. I have seen processors where we can do UPDATE, INSERT, or DELETE SQL, does create will be possible?   Also Do I need to write custom processor to extract the column names from the excel?    

I have a problem when init Hive view. It inits ds_jobimpl_xx table but miss add ds_jobimpl_xx_id_seq to ambari_sequences. I go through all Hive view code but can not find any info about those. Any expert can help me to clarify it?  Thank you.

org.apache.ambari.view.PersistenceException: Caught exception trying to store view entity JobImpl{id='null, owner='admin, hiveQueryId='null, dagId='null, queryId='null} org.apache.ambari.view.PersistenceException: Caught exception trying to store view entity JobImpl{id='null, owner='admin, hiveQueryId='null, dagId='null, queryId='null}at org.apache.ambari.server.view.persistence.DataStoreImpl.throwPersistenceException(DataStoreImpl.java:666)at org.apache.ambari.server.view.persistence.DataStoreImpl.store(DataStoreImpl.java:166)at org.apache.ambari.view.hive20.persistence.DataStoreStorage.store(DataStoreStorage.java:61)at org.apache.ambari.view.hive20.resources.CRUDResourceManager.save(CRUDResourceManager.java:117)at org.apache.ambari.view.hive20.resources.PersonalCRUDResourceManager.save(PersonalCRUDResourceManager.java:68)at org.apache.ambari.view.hive20.resources.PersonalCRUDResourceManager.save(PersonalCRUDResourceManager.java:34)at org.apache.ambari.view.hive20.resources.CRUDResour

Hi friends,   I have CDH 6.2.1 setted up in my local server. I can login to cloudera manager to monitor all the components and its running status. Meanwhile im trying to found how to access Cloudera navigator and stuck with it. Can you guide me how to access the same.   Even i followed the below steps, it dint help as i couldnt locate the navigator How can I access the Cloudera Navigator console? The Cloudera Navigator console can be accessed from the Cloudera Manager Admin Console or directly on the Navigator Metadata Server instance. Using the Cloudera Manager Admin Console as a starting point requires the Cloudera Manager roles of either Navigator Administrator or Full Administrator. From the Cloudera Manager Admin Console: Select Clusters > Cluster-n. Select Cloudera Navigator from the menu. To access the Cloudera Navigator console directly: Open the browser to the host name of the node in the cluster that runs the Cloudera Navigator Metad

I need help with config of  wait module. I am trying to execute lookup and if not matched , retry loop is entered with wait 1sec. second loop the message is dropped. THe mechanism works, but I worry of numbers provided by wait module. it shows 8mil files IN, and 134k files out.  Is this a correct setup ?Why and where is the 8mil IN comming from? How to avoid this?I worry it will crash when more traffic comes in. Thank you  

@MattWho  unable to log in seems authorization issue whereas cert authentication is successful. for initial admin cert user.     HDF 3.4 nifi new cluster self-signed ssl enabled - unable to login nifi with cert "Insufficient Permissions"   Insufficient Permissions home No applicable policies could be found. Contact the system administrator.   nifi-user.log:   2020-03-26 04:47:13,898 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=nifiadmin, OU=NIFI) GET https://lpappnifixx:9091/nifi-api/flow/current-user (source ip: 72.191.44.165)2020-03-26 04:47:13,899 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=nifiadmin, OU=NIFI2020-03-26 04:47:13,901 INFO [NiFi Web Server-17] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=nifiadmin, OU=NIFI], groups[] does not have permission to access the requested resource. No applicable policies could be found. Returning Forbidden respons

HI Can someone let me know if CDH 5.15 and CDH 5.16 is supported on Ubuntu 18? The below doc does not mention anything about Ubuntu 18 so just confirming https://docs.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#c516_supported_os Thanks,