Q&A

Hello,I am working with 3 nodes CFM cluster. the version of CFM is 1.0.1.  The below pic shows the steps: the below pic is config of ExecuteSQL:the below pic is lookup service:the below is the config of dbcpconnetionpool:The behavior is the ExcuteSQL processor will execute very slowly seem hanged.  but sometime execute very fast. I believe that is database issue. So my questions is:1. what is the impact? is dbcp connections pool issue of config? or is the dbcp connections pool lookup wrong behavior?  2. I'm afraid the impact of this will be magnified. when increased database connection pool to 500+.Could help me to answer my questions?ThanksPaul

Hi Team, Greetings....!!My cluster license got expired and still it was showing "Cloudera Enterprise ".Is any problem using after license expiration ?Can some one suggest me how to downgrade to express edition ? Thanks,Vinod

Hey, so I want to create a flow that creates a putemail trigger when NO files are present in a hdfs path for a certain date range: my current flow is GETHDFS->  ROUTEONATTRIBUTE(has the proper date matching criteria) -->MATCHED(should be getting files here) -->UNMATCHED(when this outputs zero I want to receive an email alert)    RouteOnAttribute Logic: ${Path:substring(0,6):equals(${now():toNumber():minus(86400000):format('yyyyMM')})}   Pretty much if I find no files within the current month(last 20 days) then I need to receive an email alert. How do I get an alert when there are no files that fit the criteria?    @Former Member  Thank you matt sorry for yet another question.

I tried to install HortonWorks HDP Sandbox with Docker on a Windows10 Machine, as described here: https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html   I executed the commandsh docker-deploy-hdp30.shin a Git Bash console window, having admin rights. It seems that pulling the image worked but I got following error:   C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: An unexpected error occurred while sharing drive.   =>How can I resolve that? Does the docker image expect that a certain drive exists?   Some more output from the console:   ... 4b9047c5fbbb: Download complete951bdea65c93: Pull complete4b9047c5fbbb: Pull complete773156407aae: Pull completed8524176841d: Pull completeDigest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9eStatus: Downloaded newer image for hortonworks/sandbox-proxy:1.0docker.io/hortonworks/sandbox-proxy:1.0+ '[' hdp == hdf ']'+ '[' hdp == hdp ']'+

Apache Nifi PutHiveQL How do I extract error message into flowfile attribute when failure occurs?

Hi,  We are running CDH 5.14.0 version of cloudera.There are multiple warnings coming related to log data directory for only one Data node( hadoop-slave-1). please see warnings.   for the same node,root / dir space is showing critical. can anyone suggest what files i should delete from "/ ,so that i can clear these alarms .   Thanks Devendra    

java.lang.ClassNotFoundException    error while running/usr/hcp/2.0.0.0-4/metron/bin/threatintel_taxii_load.sh SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/phoenix/phoenix-5.0.0.3.1.4.0-315-server.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Exception in thread "main" org.codehaus.jackson.map.JsonMappingException: org.apache.metron.stellar.dsl.VariableResolver (through reference chain: org.apache.metron.dataloads.extractor.ExtractorHandler["extractor"])at org.codehaus.jackson.map.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:345)at org.codehaus.jackson.map.deser.SettableBeanProperty._throwAsIOE(SettableBeanProper

Nifi DatadogReportingTask attribute "API Key" is neither sensitive nor it supports env variables. This field is a sensitive key and I dont want it to show the attribute value on the UI.   Please let me know how can this be achieved? Also what is the reason to not keep this field sensitive or "expression language compatible".

Attempting to set LDAP/AD users for Ranger (v1.2.0) following the docs (https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/configuring-ranger-authe-with-unix-ldap-ad/content/configuring_ranger_authentication_with_unix_ldap_or_ad.html) and this older video (https://www.youtube.com/watch?v=2aZ9GBhCOhA), but when looking at the Ranger Users tab in the Ranger UI, seeing only the original Unix users. In the UI I see...  Looking at the usersync logs, near the tail I see: ....06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization started06 Dec 2019 14:21:51 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldap://172.18.4.42:389, ldapBindDn: UID=hwldap,OU=Users,OU=HortonworksUsers,DC=ucera,DC=local, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: dc=hadoop,dc=apache,dc=org, userSearchBase: [dc=ucera,dc=local], userSearchScope: 2, userObj

hi there,   I have created a S2S reporting task to the same instance of nifi:    After enabing it, the port "Bulletin" can get error message however the bulletin board raise warining the message continueously  :   2019-12-06 20:35:13,274 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@7a616d8a Unable to refresh Remote Group's peers due to null   Why and any idea to fix it?

Hello,  I install impala/kudu on HDP 2.6.2.2:  impalad version 2.7.0-cdh5-IMPALA_KUDU-cdh5 RELEASE (build 48f1ad385382cd90dbaed53b174965251d91d088)  With impala-shell i created database "imp_kudu" with single table "test_table" When I try to open JDBC connection for "ImpalaJDBC4.jar" driver using url: jdbc:impala://myserver.com:21050/imp_kudu;AuthMech=0 I have the following error Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: Operation timed out (Connection timed out). What kind of problem does this indicate and how can I resolve it?

Hi,I am trying to remove all new lines from base64encoded string using replaceText processor.Properties for replaceText processor is as follows:Search Value : \\nReplacement Value:$1Character Set: UTF-8Maximum Buffer Size: 10MBReplacement Strategy: Regex ReplaceEvaluation Mode: Line-by-Line But this not working.  

I had installed HDP 3.0 which has Spark 2.3.1. I was running sql commands in Zeppelin notebook successfully but suddenly Windows update restarted the machine. When I opened the Zeppelin notebook again and ran the first command %spark2val hiveContext = new org.apache.spark.sql.SparkSession.Builder().getOrCreate() I got following error logs: -    java.lang.NullPointerException at org.apache.thrift.transport.TSocket.open(TSocket.java:209) at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51) at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37) at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60) at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) at org.apac

Hi All, when using the MergeContent processor I'm getting an avro.SchemaParseException due to $ characters in a nested attribute name ($.Snapshot.Events.['$values'][0].SportsUrn) in a json file. I have tried to fix it with the UpdateAttribute processor, but it won't work. What would be the best way to handle this exception? Your support is highly appreciated! Thanks

I need to know how I can increase the RAM size (Java Heap Size) of my NameNode.

Hi    I am trying to do an incremental load from API using the API end_time tag.    Trying to use the below syntax in RouteAttribute processor to fetch only records updated till yesterday night - 23:59:59  Could you tell me what is the exact syntax for checking end_time here?  ${end_time:lt(${now():toNumber():minus(86400000) })}

HI All,can somebody let me know which version of HDP supports Spark 2.4? and is that good idea to go to Spark 2.4 instead of 2.3?My Current versions: HDP 2.6.3.0Apache Ambari 2.6.0.0 HDFS 2.7.3 YARN 2.7.3MapReduce2 2.7.3Spark2 2.2.0Any suggestions are welcome.Regards,Manjunath P N

https://docs.cloudera.com/documentation/enterprise/6/6.1/topics/admin_ha_hiveserver2.html#concept_u4b_c5d_wv My cluster version is Cloudera CDH 6.1 Express Edition. I try set Hive HA.I added new instance.And I set the address on HiveServer2 Load Balancer property.But Load Balancer server didn't open proxy port.Doesn't Managed cluster start proxy server?Do I have to configure proxy?Managed cluster means Enterprise Edition?If that is, Can I use L4 instead of haproxy?    

I'm trying to create a Ranger policy for Atlas in a kerberized cluster using HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) but receive a 401 (unable to create policy) notification.   I'm trying to figure out which user to use following the "Learning the Ropes of the HDP Sandbox" but I can't make sense of it.   - "raj_ops", "maria_dev", "amy_ds" give the error stated above - "holger_gov" cannot login to Ranger UI - "admin" (following the "Admin password reset" procedure) can login to Ambari but not to Ranger UI   Any guidance would be greatly appreciated.

I am learning Hadoop. I want to know can i deploy more than one edge node in my cluster? If yes, how to do this and on which hosts should i keep the edge nodes? Currently i have one edge node in my cluster on a dedicated machine.  

hi all     we have 5 kafka brokers    we can stop or start the kafka service by API  but how to stop specific kafka broker by REST API?   for example , in this pic we can see 5 kafka brokers , and we want to stop/start only the second kafka broker by REST API    

When I connect to 'http://xxx.xxx.xxx:7180/cmf/login' I can see the login screen but any user/password combination that I use fails.   I've tried to change the "users" table on the "scm" database to "ffa2eb4251b38e069e968890cb2bcdb6229982322f5ed2470bf96231fe4c39c8" and -4382599614486590865 to try to reset the passwd of admin but nothing works.   cloudera-scm-server.log adds a new line for each login try: "INFO scm-web-114:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: 'admin' from 111.222.333.111"   If I try some random user the error is the same (minus the user).   Could Kerberos or Sentry have anything to do with this?   Thank you Gonçalo Redondo      

Hi, I am trying to fetch all files with .txt extension from a list of files present in s3 bucket using nifi. Is there any way to fetch the file based on the format of file and what processors we will use here. Can any one explain me with an example as i am new to this.     Thanks  Sunil  

hello, i have a server intel® Xeon(R) CPU E5-2630 v3 @ 2.40GHz × 32 with 1007gb memory and 20tb hardisck i have a docker in this server, i created a 25 container cloudera (quickstart)  18 container of 25 is running the others stop it and i will create more than 50 container but now i can't how can solve this problem plz if you can help me in this issue 

Hi, this has been troubling for months.. I simply can't find any CANVAS ID (Processor or ControllerService)  on the exported XML template 😞   Example - on my canvas I see : Name PutS3Object Id 0c6637cc-bea1-1c43-a023-921b68e400f5 Type PutS3Object 1.0.0   Now i created template for the PG, and theres no 0c6637cc-bea1-1c43-a023-921b68e400f5 on the xml ?!   This is very annoying, why i can't see any IDs on canvas on the exported xml ?   We have cluster of 3 nodes and PG is versioned on NiFi registry.   Any suggestions in how to locate on XML template any Processor or ControllerService using their ID as visible on canvas editor ?   Thanks, Emanuel O.