Member since
03-06-2020
398
Posts
54
Kudos Received
35
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
87 | 11-21-2024 10:12 PM | |
822 | 07-23-2024 10:52 PM | |
1071 | 05-16-2024 12:27 AM | |
3012 | 05-01-2024 04:50 AM | |
1335 | 03-19-2024 09:23 AM |
07-23-2024
10:52 PM
1 Kudo
Hi @therealsrikanth You can follow this if you do not need CM or ambari. Step 1: Install Hadoop Download Hadoop: Download the latest stable release of Hadoop from the Apache Hadoop website. tar -xzf hadoop-3.3.4.tar.gz sudo mv hadoop-3.3.4 /usr/local/hadoop 2. Configure Hadoop Environment Variables: Add the following lines to your .bashrc or .profile file. export HADOOP_HOME=/usr/local/hadoop export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin 3. Edit Configuration Files: Edit the core configuration files in $HADOOP_CONF_DIR. core-site.xml: <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master-node:9000</value> </property> </configuration> hdfs-site.xml: <configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///usr/local/hadoop/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///usr/local/hadoop/hdfs/datanode</value> </property> </configuration> mapred-site.xml: <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> yarn-site.xml: <configuration> <property> <name>yarn.resourcemanager.address</name> <value>master-node:8032</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> 4. Format the NameNode: hdfs namenode -format 5. Start Hadoop Services: start-dfs.sh start-yarn.sh Step 2: Install Zookeeper: 1. Download and Extract Zookeeper: https://downloads.apache.org/zookeeper tar -xzf apache-zookeeper-3.8.1-bin.tar.gz sudo mv apache-zookeeper-3.8.1-bin /usr/local/zookeeper 2. Configure Zookeeper: Create a configuration file at /usr/local/zookeeper/conf/zoo.cfg tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=master-node1:2888:3888 server.2=master-node2:2888:3888 server.3=slave-node1:2888:3888 3. Start Zookeeper: /usr/local/zookeeper/bin/zkServer.sh start Step 3: Install HBase: 1. Download HBase: Download the latest stable release of HBase from the Apache HBase website. tar -xzf hbase-2.4.16-bin.tar.gz sudo mv hbase-2.4.16 /usr/local/hbase 2. Configure HBase: hbase-site.xml <configuration> <property> <name>hbase.rootdir</name> <value>hdfs://master-node:9000/hbase</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>master-node1,master-node2,slave-node1</value> </property> </configuration> 3. Start HBase: /usr/local/hbase/bin/start-hbase.sh Step 4: Verify Installation: Check the Hadoop services using the web interfaces: NameNode: http://master-node:9870 ResourceManager: http://master-node:8088 HBase: http://master-node:16010 Additional Resources Apache Hadoop Documentation Apache Zookeeper Documentation Apache HBase Documentation Regards, Chethan YM
... View more
06-11-2024
10:29 PM
1 Kudo
Hi @rizalt The error is because you have not provided keytab path here the command should look like below: > klist -k example.keytab To create the keytab you can refer any of below steps: $ ktutil
ktutil: addent -password -p myusername@FEDORAPROJECT.ORG -k 42 -f
Password for myusername@FEDORAPROJECT.ORG:
ktutil: wkt /tmp/kt/fedora.keytab
ktutil: q Then kinit -kt /tmp/kt/fedora.keytab myusername@FEDORAPROJECT.ORG Note: Replace the username and REALM as per your cluster configurations. Regards, Chethan YM
... View more
05-30-2024
10:24 PM
Hi @jayes May i know how you are exporting the table into HDFS? what is the command?
... View more
05-16-2024
12:27 AM
2 Kudos
Hi @d_liu_ The error shows that problem with the network connectivity between the Hive server (ip-172-19-36-68) and the HDFS namenode (ip-172-19-36-94.ap-southeast-2.compute.internal) on port 8020. So as per your comment above HDFS node does not exist in your cluster right? May i know how you are running this query like from beeline or Hue or from any third party tool? Search the above host in all the configuration files(hive-site.xml,hive-env.sh,hive.metastore.uris etc..) Try to search the above host name at client side from where you are running the query and there is a chances if you have any custom scripts or config files causing the issue, so check if you have any custom scripts to run the jobs. Regards, Chethan YM
... View more
05-15-2024
02:46 AM
1 Kudo
@hadoopranger 1. Verify if the JDBC connection string is valid and correct. 2. Try to use the latest JDBC driver version that is available. 3. Check the HS2 servers that you are connecting are accepting the connections and in good health. 4. Check if beeline is working or not. 5. If all correct enable the JDBC driver trace level logging and get the more details on the error. https://docs.cloudera.com/documentation/other/connectors/hive-jdbc/2-6-15/Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf Regards, Chethan YM
... View more
05-15-2024
02:34 AM
1 Kudo
Hello @hadoopranger Do provide the complete error stack trace from hue log and error screenshot from Hue UI to check the same. Try restarting impala, hue and see if any difference. Regards, Chethan YM
... View more
05-14-2024
01:25 AM
1 Kudo
@vlallana Its a generic error you just need to compare the configurations/settings between prod and dev clusters and check the Impala server status and it is ready to accept the connections. try to use the latest available driver and check if there is any network connectivity issue, if everything good enable the driver trace level logging[1] and repro the issue then review the logs for more details. Regards, Chethan YM [1]. https://docs.cloudera.com/documentation/other/connectors/impala-odbc/2-6-11/Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf
... View more
05-01-2024
05:28 AM
@Anderosn 1. If the content of your flow file is too large to be inserted into a single CLOB column, you can split it into smaller chunks and insert each chunk into the database separately. 2. Instead of storing the content in a CLOB column, you can consider storing it in a BLOB (Binary Large Object) column in your database. BLOB columns can store binary data, including large files, without the size limitations of CLOB columns. 3. Store the content of the flow file in an external storage system (e.g., HDFS, Amazon S3) and then insert the reference (e.g., file path or URL) into the database. This approach can be useful if the database has limitations on the size of CLOB or BLOB columns 4. If ExecuteScript is not approved, consider using an external script or application to perform the insertion into the database. You can trigger the script or application from NiFi using ExecuteProcess or InvokeHTTP processors Regards, Chethan YM
... View more
05-01-2024
04:50 AM
1 Kudo
@rsheikh Ensure that the Kerberos configuration (krb5.ini) is correctly set up on your Windows Server 2019 machine. The krb5.ini file should contain the necessary realm and KDC (Key Distribution Center) information for your Kerberos setup. Set the java.security.auth.login.config environment variable to point to the JAAS (Java Authentication and Authorization Service) configuration file (jaas.conf). This file defines the login modules used for authentication. Verify that the realm and principal settings in krb5.ini match the configuration of your Kerberos environment. Regards, Chethan YM
... View more
03-19-2024
09:23 AM
1 Kudo
Hi @Muskan You can set it in CM Impala advance configs -> Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve) Regards, Chethan YM
... View more