Member since
10-20-2016
106
Posts
0
Kudos Received
0
Solutions
05-24-2022
08:11 AM
Hi! Did you could able to solve it?? Im facing the same problem with the same scenario.
... View more
03-02-2022
10:11 PM
1 Kudo
Stop CDH Services and Stop Cloudera Manager Management Services. Import the new kerberos account. You will need an admin account on the KDC for this: CM UI -> Administration -> Security -> Kerberos credentials -> "Import Kerberos Account Manager Credentials" Enter username and password Click Import button Re-generate missing principals if the previous step was successful CM UI -> Administration -> Security -> "Kerberos credentials" Click the button "Generate Missing Credentials" Wait until credentials have been generated Start Cloudera Manager Management Services Start CDH Services
... View more
12-29-2021
08:42 AM
@ebeb , as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
01-05-2021
11:06 AM
@saivenkatg55 My Assumptions You already executed the HDP environment preparation. If not see prepare the environment https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/prepare_the_environment.html You are running on Linux [RedHat, Centos] and you have root access! Note: Replace test.ambari.com with the output of your $ hostname -f Re-adapt to fit your cluster # root password = welcome1
# hostname = test.ambari.com
# ranger user and password is the same Steps Install the MySQL connector if not installed [Optional] # yum install -y mysql-connector-java Shutdown Ambari # ambari-server stop Re-run the below command it won't hurt # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Backup the ambari server properties file # cp /etc/ambari-server/conf/ambari.properties /etc/ambari-server/conf/ambari.properties.bak Change the timeout of the ambari server # echo 'server.startup.web.timeout=120' >> /etc/ambari-server/conf/ambari.properties
# echo 'server.jdbc.connection-pool.acquisition-size=5' >> /etc/ambari-server/conf/ambari.properties
# echo 'server.jdbc.connection-pool.max-age=0' >> /etc/ambari-server/conf/ambari.properties
# echo 'server.jdbc.connection-pool.max-idle-time=14400' >> /etc/ambari-server/conf/ambari.properties
# echo 'server.jdbc.connection-pool.max-idle-time-excess=0' >> /etc/ambari-server/conf/ambari.properties
# echo 'server.jdbc.connection-pool.idle-test-interval=7200' >> /etc/ambari-server/conf/ambari.properties Recreate a new ranger schema & Database # mysql -u root -pwelcome1
CREATE USER 'rangernew'@'%' IDENTIFIED BY 'rangernew';
GRANT ALL PRIVILEGES ON *.* TO 'rangernew'@'localhost';
CREATE USER 'rangernew'@'%' IDENTIFIED BY 'rangernew';
GRANT ALL PRIVILEGES ON rangernew.* TO 'rangernew'@'%';
GRANT ALL PRIVILEGES ON rangernew.* TO 'rangernew'@'localhost' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON rangernew.* to 'rangernew'@'localhost' identified by 'rangernew';
GRANT ALL PRIVILEGES ON rangernew.* to 'rangernew'@'test.ambari.com' identified by 'rangernew';
GRANT ALL PRIVILEGES ON rangernew.* TO 'rangernew'@'test.ambari.com';
GRANT ALL PRIVILEGES ON rangernew.* TO 'rangernew'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
quit; Create the new ranger database # mysql -u rangernew -prangernew
create database rangernew;
show databases;
quit; Start the ambari server # ambari-server start
......Desired output.........
..................
.................
Ambari Server 'start' completed successfully. For ranger Ambari UI setup Use the hostname in this example test.ambari.com and the corresponding passwords Test the Ranger DB connectivity The connection test should succeed if it does then you can now start Ranger successfully. Drop the old Ranger DB # mysql -u root -pwelcome1
mysql> Drop database old_Ranger_name; The above steps should resolve your Ranger issue. Was your question answered? If so make sure to mark the answer as the accepted solution. If you find a reply useful, Kudos this answer by hitting the thumbs up button.
... View more
11-16-2020
10:45 PM
There should be materialized view using this table as underlying table, which is not allowing the table to get dropped. try dropping the MV and drop the external table.
... View more
09-04-2020
10:14 PM
@saivenkatg55 Use the below commands to remove the background operation entry in ambari database. # select task_id,role,role_command from host_role_command where status='IN_PROGRESS'; The above command will list all IN_PROGRESS status , You can also check for QUEUED or PENDING STATE by replacing 'IN_PROGRESS' with QUEUED or PENDING # update host_role_command set status='ABORTED' where status='QUEUED'; Use the above command to change the state to ABORTED
... View more
06-18-2020
09:44 AM
Hi, Have you enabled dynamic resource allocation in spark? https://docs.cloudera.com/runtime/7.1.0/configuring-spark/topics/spark-dynamic-resource-allocation-properties.html Thanks AKR
... View more
05-07-2020
06:59 AM
Hi Sai, Could you please share the Entrie Node manager logs for further analysis? Thanks AKR
... View more
03-16-2020
04:45 PM
@saivenkatg55 Sorry to hear you are having space issues with your content repository. The most common reason for space issues is because there are still active FlowFiles referencing the content claims. Since a content claim cannot be moved to archive sub-directory or deleted until there are no FlowFiles referencing that claim, even a small FlowFile can still queued somewhere within a dataflow can result in a large claim not being able to be removed. I recommend using the NiFi Summary UI (Global menu --> Summary) to locate connections with flowfiles just sitting in them not getting processed. Look at the connections tab and click on "queue" to sort connections based on queued FlowFiles. A connection with queued FlowFiles, but shows 0 for both "In/Size" and "Out/Size" are what I would be looking for. This indicates in the last 5 minutes that queue has not changed in amount of queued FlowFiles. You can use the got to arrow to far right to jump to that connection on the canvas. If that data is not needed (just left over in some non active dataflow), right click on the connection to empty the queue. See if after clearing some queues the content repo usage drops. Is is also possible that not enough file handles exist for your NiFi service user making clean-up not to work efficiently. I recommend increasing the open files limit and process limits for your NiFi service user. Check to see if your flowfile_repository is large or if you have content claims moved to archive sub-directories that have not yet been purged. Does restart of NiFi which would release file handles trigger some cleanup of the repo(s) on startup? It is also dangerous to have all your NiFi repos co-located on the same disk for risk of corruption to your flowfile repository which can lead to data loss. The flowfile_repository should always be on its own disk, the content_repository should be on its own disk, and the provenance_repository should be on its own disk. The database repository can exist on a disk used for other NiFi files (config files, local state, etc.) https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999 Here are some additional articles that may help you: https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 https://community.cloudera.com/t5/Community-Articles/How-to-determine-which-FlowFiles-are-associated-to-the-same/ta-p/249185 Hope this helps, Matt
... View more
02-25-2020
11:05 AM
@saivenkatg55 You need to literally use ./keystore.p12 in your command instead of just keystore.p12 curl --cert-type P12 --cert ./keystore.p12:password --cacert nifi-cert.pem -v https://w0lxqhdp04:9091/nifi-api/flow/search-results?q= Hope this helps, Matt
... View more
02-10-2020
07:59 AM
@saivenkatg55 That could be a memory issue on your cluster. Can you share the below config set spark.executor.memory
set yarn.nodemanager.resource.memory-mb
set yarn.scheduler.maximum-allocation-mb Here are some links to help How to calculate node and executors memory in Apache Spark after adjusting that share the new output
... View more
01-30-2020
04:14 AM
@Shelton we are still experiencing slowness in ambari UI after enabling the above properties.
... View more
01-23-2020
05:26 AM
@Lewis @Shelton @Prakashcit Hi All, I did ranger re-installation and now it is working, but if i integrate hive with ldap, again getting same null pointer exception. Note: we integrated hive with ldap not ranger, just creating same ldap user in ranger (local) and trying to match it with LDAP user. whether it will create any problem? Kindly advice on this. select * from asop.test; Error: Error while compiling statement: FAILED: NullPointerException null (state=42000,code=40000)
... View more
01-22-2020
12:39 PM
@saivenkatg55 Your Out Of Memory (OOM) issue is occurring in the NiFi toolkit script and not in NiFi itself. Ambari will execute the NiFi toolkit before the NiFi service itself is ever started. If the NiFi toolkit fails, Ambari does not proceed any further. I can see you are running HDF 3.2.0 which exposes the heap memory settings under the NiFi configs in Ambari: Locate the above line and increase your heap settings here. Once the toolkit script finished running which must occur before NiFi is even started, that JVM is gone. So the memory allocation set here are short lived and will not impact available memory for your NiFi. Hope this helps, Matt
... View more
01-17-2020
02:07 AM
@ramineni Please find the netcat O/P [hdfs@w0lxdhdp01 ~]$ nc -v w0lxqhdp01 8020 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 10.49.70.13:8020.
... View more
01-06-2020
06:26 AM
@Shelton do u have any idea on this getting unknown exception while accessing the query from spark sql Time taken: 0.216 seconds, Fetched 318 row(s) 20/01/06 09:16:49 INFO SparkSQLCLIDriver: Time taken: 0.216 seconds, Fetched 318 row(s) spark-sql> select * from snapshot_table_list; 20/01/06 09:16:57 INFO ContextCleaner: Cleaned accumulator 0 20/01/06 09:16:57 INFO ContextCleaner: Cleaned accumulator 1 20/01/06 09:16:57 INFO ContextCleaner: Cleaned accumulator 2 20/01/06 09:16:58 INFO HiveMetastoreCatalog: Inferring case-sensitive schema for table project.snapshot_table_list_ext (inference mode: INFER_AND_SAVE) 20/01/06 09:16:58 INFO deprecation: No unit for dfs.client.datanode-restart.timeout(30) assuming SECONDS 20/01/06 09:16:58 ERROR SparkSQLDriver: Failed in [select * from snapshot_table_list] java.lang.IllegalArgumentException: java.net.UnknownHostException: datalakedev at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:445) at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:353) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:287)
... View more
01-06-2020
02:41 AM
@Shelton Any update on this? looks like it is looking for some java packages java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-4657625312215122883.8 (Permission denied)] can we install it externally?
... View more
01-01-2020
11:04 AM
@saivenkatg55 You didn't respond to this answer, do you still need help or it was resolved if so please do accept and close the thread.
... View more
12-27-2019
07:09 AM
1 Kudo
@saivenkatg55 A very common reason for UI slowness is JVM Garbage Collection (GC). All GC events are stop-the-world events whether it is a partial or full GC event. Partial/young GC is normal and healthy, but if it is being triggered back to back non stop or is running against a very large configured JVM heap it can take time to complete. You can enable some GC logging in your NiFi bootstrap.conf file so you can see how often GC is running to attempt to free space in your NiFi JVM. To do this you need to add some additional java,arg.<unique num>= entries in your NiFi bootstrap.conf as follows: java.arg.20=-XX:+PrintGCDetails
java.arg.21=-XX:+PrintGCTimeStamps
java.arg.22=-XX:+PrintGCDateStamps
java.arg.23=-Xloggc:<file> The last entry allows you to specific a separate log file for this output to be written in to rather than stdout. NiFi does store information about component status in heap memory. This is the info you can see on any component (processor, connection, process group, etc.) when you right click on it and select "view status history" from the displayed context menu. You'll notice that these component report status for a number of data points. When your restart your NiFi, everything in the JVM heap memory is gone. So over the next 24 hours (default data point retention) the JVM heap will be holding a full set of status points again. You can adjust the component status history buffer size and datapoint frequency to reduce heap usage here if this status history is not that important to you via the following properties in the nifi.properties file: nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min above represents defaults. For every status history point for every single component, NiFi will retain 1440 status points (recording 1 point every 1 min). This totals 24 hours worth of status history for every status point. Changing the buffer to 288 and frequency to 5 minutes will reduce number of points retained by 80% while still giving your 24 hours worth of points. The dataflows you build may result in high heap usage triggering a lot of heap pressure. Those NiFi components that can result in high heap usage are documented. From the NiFi Global menu in the upper right corner of the NIFi UI, select "Help". You will see the complete list of components on the left had side. When you select a component, details about that component will be displayed. One of the details is "System Resource Considerations". For example, here is the system resource considerations for the MergeContent processor: You may need to make adjustments to your dataflow designs to reduce heap usage. NiFi also holds FlowFile metadata for queued FlowFiles in heap memory. NiFi Does have a configurable swap threshold (which is applied per connection) to help with heap usage here. When a queue grows too large, FlowFile metatdata in excess of the configured swap threshold will be written to disk. The swapping in and swapping out of FlowFiles from disk can affect dataflow performance. NiFi's default backpressure object thresholds settings for connections is set low enough that swapping would typically never occur. However, if you have lots and lots of connections with queued FlowFiles, that heap usage can add up. This article I wrote may help you here: https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166 ----- Other than heap usage, component validation can affect NiFi UI responsiveness. Here is an article i wrote about that: https://community.cloudera.com/t5/Community-Articles/HDF-NiFi-Improving-the-performance-of-your-UI/ta-p/248211 Here is another useful article you may want to read: https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999 Hope this helps you with some direction to help improve your NiFi UI responsiveness/performance, Matt
... View more
12-20-2019
07:29 AM
@saivenkatg55 NiFi musty be secured before you can use any form of authentication and authorization. Along with securing NiFi you can configure NiFi's login-identity-providers.xml to use the "ldap-provider" for user authentication. Here are some useful links: Securing NiFi: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/nifi-authentication/content/nifi_authentication.html Setting up ldap-provider: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/nifi-security/content/ldap_login_identity_provider.html Setting up authorization: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/nifi-authorization/content/nifi_authorization.html Hope this helps, Matt
... View more
12-19-2019
02:12 PM
@saivenkatg55 This "Exiting with status 1: java.io.IOException: Problem starting http server" error should be linked to your other question I just have responded to https://community.cloudera.com/t5/Support-Questions/Unable-to-start-the-node-manager/td-p/286013 If this is resolved then the java.io.IOException shouldn't occur HTH
... View more
11-12-2019
06:22 AM
Hi, We need to see the cluster Resource usage during the time frame when the jobs are in accepted state. If the entire memory and allocated vcores has been used the job does not have suffiicient resources to run the application. We need to look for the RM webui and Scheduler screenshot of all the queues to view the usage of the cluster. Thanks AKR
... View more
11-01-2019
02:26 PM
@saivenkatg55 Check your message
... View more
10-23-2019
11:30 PM
@paras Please check the above curl O/P
... View more
10-15-2019
10:13 AM
@Shelton not able to execute any of the HDFS command due to kerberos. hadoop fs -ls / 19/10/15 13:12:55 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/15 13:12:55 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/15 13:12:55 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort hostname:8020 , LocalHost:localPort hostname/10.49.70.18:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over hostname10.49.70.14:8020 after 1 failover attempts. Trying to failover after sleeping for 1171ms
... View more
10-04-2019
01:05 PM
@saivenkatg55 What are the permission on that file? It should be -rw-r--r-- 1 yarn hadoop $ ls /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-<host_name>.org.out Permissions should be # chmod 644 /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-<host_name>.org.out Ownership should be yarn:hadoop
... View more
09-30-2019
04:32 AM
@saivenkatg55 Good to know that your issue is resolved. If your question is answered then, Please make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
05-29-2019
03:41 PM
@sai venkatesh Any updates?
... View more
05-10-2019
02:49 PM
please find the output [root@edge conf]# /usr/bin/hdp-select | grep hdfs hadoop-hdfs-datanode - 2.4.3.0-227 hadoop-hdfs-journalnode - 2.4.3.0-227 hadoop-hdfs-namenode - 2.4.3.0-227 hadoop-hdfs-nfs3 - 2.4.3.0-227 hadoop-hdfs-portmap - 2.4.3.0-227 hadoop-hdfs-secondarynamenode - 2.4.3.0-227 [root@node1 ~]# /usr/bin/hdp-select | grep hdfs hadoop-hdfs-datanode - 2.4.3.0-227 hadoop-hdfs-journalnode - 2.4.3.0-227 hadoop-hdfs-namenode - 2.4.3.0-227 hadoop-hdfs-nfs3 - 2.4.3.0-227 hadoop-hdfs-portmap - 2.4.3.0-227 hadoop-hdfs-secondarynamenode - 2.4.3.0-227 [root@node1 ~]# [root@edge conf]# yum info hadoop-yarn^C [root@edge conf]# yum info hadoop-hdfs Loaded plugins: fastestmirror, security Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Loading mirror speeds from cached hostfile Available Packages Name : hadoop-hdfs Arch : noarch Version : 2.7.1.2.4.3.0 Release : 227.el6 Size : 2.5 k Repo : HDP-2.4 Summary : hadoop-hdfs Distro virtual package License : APL2 Description : hadoop-hdfs-2.7.1.2.4.3.0 virtual package
... View more