Solved

I have removed files from HDFS by skipping trash but still it shows same disk useage

Very new to Nifi and regex. I have a test txt file with a mock log file in.Format along the lines of: srcip=10.10.10.10 timestamp=152532431 action="denied" What I need is to match against the word and then return everything after the '=' until the next space character. Any help would be appreciated.

I am wanted to split this txt file into separate flow files at the spaces.I am very new to nifi and regex and am unsure of which processor and properties I will need. For example: srcip=10.10.10.10 timestamp=152532431 action="denied" logver=12 tz="UTC+0" logid="0000012" dstip=12.12.12.12This snippet is part of a long line that I need to split up at the spacesSo here I would have 6 separate flow files. Any help appreciated.

[hdfs@xyz ~]$ beelineBeeline version 1.2.1000.2.6.3.0-235 by Apache Hivebeeline> !connect jdbc:hive2://xyz:10000/default;Connecting to jdbc:hive2://xyz:10000/default;Enter username for jdbc:hive2://xyz:10000/default;: *************Enter password for jdbc:hive2://xyz:10000/default;: *************20/01/08 19:08:10 [main]: WARN jdbc.HiveConnection: Failed to connect to xyz:10000Error: Could not open client transport with JDBC Uri: jdbc:hive2://xyz:10000/default;: Peer indicated failure: Error validating the login (state=08S01,code=0)0: jdbc:hive2://xyz:10000/default (closed)>

"Map 0% reduce 0%" is not displayed when I execute Sqoop And Tracking URL is  UNASSIGNED ---------- --- Virtual Machine: Cloudera Quickstart Virtual Machine (Using Oracle VM Virtualbox, download URL: https://www.cloudera.com/downloads/quickstart_vms/5-13.html)(1) Version: 5.13.0-0(2) Number of processors: 2(3) Controller: IDE Controller(4) Memory space: 11400 MB(5) Hard drive: 64.00 GB storage, solid state drive(6) OS: Red Hat, 64-bit(7) MySQL 5.1.73(8) HBase 1.2.0-cdh5.13.0(9) Hadoop 2.6.0-cdh5.13.0(10) Sqoop 1.4.6-cdh5.13.0 ---------- ...20/01/16 13:13:08 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hadoop/conf:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar:/usr/lib/had

Hello,    i have a problem to start hue after enabling AUTO_TLS Feature. Mysql Connection seems to be blocked and flush hosts doenot help.    Can you help me?   Thanks,   Simon

Hi, im trying to upload a CSV to a RedShift database using GetFile->PutDatabaseRecord i'm getting this error: PutDatabaseRecord:CSVReader (my file got headers, this way and not hardcoding the schema makes everything more easily scalable):Null string: NULL CSV:A,B,C,D,E,F,G,H,I2020-01-15,AX,COD245,NO,PASS,R,,,2020-01-15,AX,COD235,YES,PASS,R,,, Table with same fields names.Any idea what i'm doing wrong? Is there any way to look for the query it's making, that will make this more easy to debug.Thank you!  

I am using HDP. The inode in the following code is a managed hive table. # id zeppelinuid=1017(zeppelin) gid=1003(hadoop) groups=1003(hadoop),1005(zeppelin)# sudo -u zeppelin hadoop fs -ls /warehouse/tablespace/managed/hive/test1ls: Permission denied: user=zeppelin, access=READ_EXECUTE, inode="/warehouse/tablespace/managed/hive/test1":hive:hadoop:drwxrwx--- The user zeppelin is in hadoop group, which has full permisison on the hdfs folder. So why do I get the permission error?

Hi, I am starting a new career in AI as a solution architect, and I need to know what exactly the knowledge needed for such a role, what should I know, which product should I start work on and use and what is the best way to start with Cloudera product.     Kindly I need advice on this matter ASAP. Thanks in advance.

Is cloudera express still an open source/free? please if anyone knows about this help.

HelloI receive an error while enabling kerberos on ambari as below;i have installed krb5-kdc krb5-admin-server and config krb5.conf, kdc.conf and kadm5.acl then created new principle (as attached)Note when i wrote the realm name in the kdc file in uppercase letter i got an error while using kadmin.local master key cannot be fetch, it only works in lowercase letterAlso when i try to restart the krb5 services, it said service can't be found although it is running so i restart the server insteadLast thing when i installed krb5-kdc krb5-admin-server the /var/kerberos folder didn't create automatically and i had to create it manually.Please help me solve this issue, thank you in advanced.KADM5.acl.txtKDC.conf.txtKRB5.conf.txt

Nifi - HandleHttpRequest processor throwing an exception saying "Failed to Initialize the server". Functionality wise it is working fine but I always see the below error message.  Will be able to post the request using curl command to this processor and it is accepting the request without any issue. Please help me to get rid of this error? Thanks. Error message:13:42:37 EST- hostname:port - ERRORHandleHttpRequest[id=123] failed to process session due to failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to Initialize the server. Configuration:

  since metrics take space on the local disk ,   we want to move to distributed mode , so metrics data will be on HDFS and not on local disk   we start to work according to the link - https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode   we are using ambari cluster 2.6.4 with active/stand by name node     so we start the configuration    first we set in ambari the following     ams-site, Set timeline.metrics.service.operation.mode = distributed   now  we need to Edit ams-hbase-site,   hdfs://<nameservice-id>/user/ams/hbase    the question is - how to find the nameservice-id ?   since we have active/stand by name node   then how to know the - nameservice-id value?            

Hello,    we have a a CDH5.16.2 Cluster with Hue connected to ActiveDirectory.  We can login and access Hive as an example.    But the automatic creation of a user is not working.  The error Message is : [09/Jan/2020 17:57:10 +0100] views ERROR Could not create home directory at login for *****.Traceback (most recent call last):File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/desktop/core/src/desktop/auth/views.py", line 129, in dt_loginensure_home_directory(request.fs, user)File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/apps/useradmin/src/useradmin/views.py", line 811, in ensure_home_directoryfs.do_as_user(username, fs.create_home_dir, home_directory)File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 102, in do_as_userreturn fn(*args, **kwargs)File "/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 1

Hi All, I have a fresh Ambari eco system installed in my VMware Vsphere, assume its hostname is ambari01.example.com and IP Address is 192.168.1.10, I cloned it to ambari02 and changed IP 192.168.1.11. but unable to start services in ambari02.example, I think all eco system configuration files need to be modified, but not sure what all the files need to be modified. Might be some guys already tried this, can someone assist me. Thanks Niruu

Hi, I'm trying to install Ambari 2.6 on CentOS7 x64. I have HDP 2.6 already installed. These versions are compatible to each other per Horton Support Matrix.   When trying to create a cluster through wizard, the host registration is failing, below is the log error and I've also tried all common troubleshooting I could find online = stop, reset, start server = change hostname = disable firewall = change python code (https://community.cloudera.com/t5/Support-Questions/Ambari-server-host-register-failed/td-p/233413)   None of it seems to help. Looking forward to any help.   06 Jan 2020 22:53:19,011 INFO [Thread-23] AbstractPoolBackedDataSource:212 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connec

Is it possible to add a second user on hadoop cluster like the spark user?

Hi, I am receiving the flowfile from ConsumerKafka in json form.     Then I am using EvaluateJsonPath to extract the json data once the data matched, i want call RouteOnAttribute to call script, but the expression which i have used  in EvaluateJsonPath   processor looks to be wrong and throwing out unmatched info. Plz help me out to add right expression to extract  the fields  

Hi,   I am adding ranger in Cloudera runtime(CDP) but i am getting below error while adding ranger service 'db_password' not found in /var/run/cloudera-scm-agent/process/1546333400-ranger-RANGER_ADMIN-SetupRangerCommand/install.properties I have created ranger db, ranger username, mysql password password but still i am getting this error

Using CentOS7 Cloudera 5.16 version installations... got below error   Failed to install Cloudera-manager-agent package   Attaching full logs from the CM.

I have recently enabled NameNode HA, ResourceManager HA, and Hive HA on an HDP 2.6 cluster by following the instructions provided here:  https://docs.cloudera.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-operations/content/ch_managing_service_high_availability.html.   Everything continues to work fine if I stop all HDP components on host where the primary NameNode and ResourceManager are running by logging into Ambari, selecting that host, and then selecting the Host Action "Stop All Components".  But as soon as I power down that host, beeline requests become intolerably slow.  For example, a beeline request for "show tables;" takes about 20 minutes to complete.  The request will hang at this point for an extended period of time: Setting property: [silent, false]issuing: !connect jdbc:hive2://nn2:10000/intrepid hdfs [passwd stripped]Connecting to jdbc:hive2://nn2:10000/intrepid (nn2 is the host name where the standby NameNode is running.)   After a long

I just download HDP 3.0.1.  I can log into Ambari as maria_dev, but I can't use putty (or Windows 10 ssh) to log in from a terminal (using Virtual Box, here)   I know the root password is hadoop, but when I try to connect, I get the error message "remote side unexpectedly closed network connection."  I tried: ssh maria_dev@localhost -p 4200   I also tried ssh root@localhost -p 4200 -pwd hadoop   ...same error.  I'm never even given the chance to log in as either user, maria_dev nor root, so I can't even enter the password hadoop to change the root password.  Didn't have this problem in HDP 2.5

Prv account part of AD group is not getting sync'd to Ambari and default access is not working. 2020-01-07T15:18:15.345Z, User(null), RemoteIp(10.176.18.143), Operation(User login), Roles(), Status(Failed), Reason(Authentication required)2020-01-07T15:18:15.346Z, User(null), RemoteIp(10.176.18.143), Operation(User login), Roles(), Status(Failed), Reason(Invalid JWT token) Normal lanid works fine and both are part of same AD group which has admin privileges.   the login is direct and not via knox , another weird thing noticed is prv works fine for knox login

After enabling Ranger Plugin for Hive, I was running a "dfs -ls" in hive/ beeline to list hdfs files. I'm getting the following error :jdbc:hive2://vserver69901.example.com> dfs -ls ;Error: Error while processing statement: Permission denied: user [ambari] does not have privilege for [DFS] command (state=,code=1)Do we need to update/enable  any other properties. hadoop fs -ls works without any issuesThe user has admin access in Ranger. The storage is on Isilon , so the HDFS Ranger plugin cannot be enabled. 2015-11-12 17:07:31,980 INFO  [HiveServer2-Handler-Pool: Thread-46]: operation.Operation (HiveCommandOperation.java:setupSessionIO(69)) - Putting temp output to file /tmp/hive/012f5aa7-fa31-4fb2-8cd5-4f3fe3f3120624919258595801789.pipeout2015-11-12 17:07:31,981 ERROR [HiveServer2-Handler-Pool: Thread-46]: processors.CommandUtil (CommandUtil.java:authorizeCommand(66)) - Error authorizing command [-ls]org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:

Hi,   I am running Spark jobs on YARN, using HDP 3.1.1.0-78 version.    I have set the Spark Scheduler Mode to FAIR by setting the parameter "spark.scheduler.mode" to FAIR.  The fairscheduler.xml is as follows:   I have also configured my program to use "production" pool.      Upon running the job, it has been observed that although 4 stages are running, only 1 stage run under "production" and rest 3 run under "default" pool.    So, at any point of time, I am able to make sure that only 2 tasks are running in parallel.  If I want to make sure that 3 tasks or more run in parallel, then 2 tasks should run under "production" and rest 2 should run under "default".    Is there any programmatic way to achieve that, by setting configuration parameters?    Any inputs will be really helpful.    Thanks and Regards, Sudhindra