@vishal dutt Yup, you need to create them on all the nodes. If you are using the LDAP, please bind those nodes to LDAP so that if you do id janu all the nodes you should able to see her id. ... View more

@vishal dutt When you run the 1st query it's not triggering any MapReduce program. It's directly reading the data but when you define a logic it's triggering mapreduce program to perform aggregations. When it's running mapreduce program, user id need to be present on all the Resource Managers. Hope this helps you and let me know if you need any further information. ... View more

@John Adams While running ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar you need to find where mysql-connector-java.jar is located and point to that directory. Below URL will help you. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-administration/content/using_ambari_with_mysql.html Database Requirements Ambari requires a relational database to store information about the cluster configuration and topology. If you install HDP Stack with Hive or Oozie, they also require a relational database. The following table outlines these database requirements: Component Databases Description Ambari PostgreSQL 9.1.13+,9.3, 9.4*** MariaDB 10.2.9* MySQL 5.7**** Oracle 11gr2 Oracle 12c** By default, Ambari installs an instance of PostgreSQL on the Ambari Server host. Hortonworks supports any version of Postgres that is automatically installed during the Ambari Server installation. Optionally, you can use an existing instance of PostgreSQL, MySQL or Oracle and the versions of supported existing database instances are listed here. Support Versions: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_support-matrices/content/ch_matrices-ambari.html Hope this helps. ... View more

@Josh Nicholson If you are storing your Ranger Hive logs to HDFS and running doAs=false, you can build a hive table on to of ranger-hive logs and start querying. Example: select requser,count(*) from ranger_audit_event_json_tmp where TO_DATE(evttime)>='2018-05-10' group by requser; Above query gives you number queries ran by with respective to each user from 2018-05-10. Couple of links for creating tables: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/amb_infra_arch_n_purge_command_line_operations.html https://community.hortonworks.com/articles/60802/ranger-audit-in-hive-table-a-sample-approach-1.html This worked for me, hope this helps you ... View more

@Venkat Metastore connection options: Argument Description --meta-connect <jdbc-uri> Specifies the JDBC connect string used to connect to the metastore By default, a private metastore is instantiated in $HOME/.sqoop. If you have configured a hosted metastore with the sqoop-metastore tool, you can connect to it by specifying the --meta-connect argument. This is a JDBC connect string just like the ones used to connect to databases for import. In conf/sqoop-site.xml, you can configure sqoop.metastore.client.autoconnect.url with this address, so you do not have to supply --meta-connect to use a remote metastore. This parameter can also be modified to move the private metastore to a location on your filesystem other than your home directory. If you configure sqoop.metastore.client.enable.autoconnect with the value false, then you must explicitly supply --meta-connect. Hope this helps ... View more

@Mike Wong Yes ... View more

@Mike Wong When you restart the Services, it should automatically get updated if you have added the disk to the same mount. ... View more

@Bharath N Try to perform the following steps on the failed DataNode: Get the list of DataNode directories from /etc/hadoop/conf/hdfs-site.xml using the following command: $ grep -A1 dfs.datanode.data.dir /etc/hadoop/conf/hdfs-site.xml <name>dfs.datanode.data.dir</name> <value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data, /data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data,/data5/hadoop/hdfs/data,/data6/hadoop/hdfs/data, /data7/hadoop/hdfs/data,/data8/hadoop/hdfs/data,/data9/hadoop/hdfs/data</value> Get datanodeUuid by grepping the DataNode log: $ grep "datanodeUuid=" /var/log/hadoop/hdfs/hadoop-hdfs-datanode-$(hostname).log | head -n 1 | perl -ne '/datanodeUuid=(.*?),/ && print "$1\n"' 1dacef53-aee2-4906-a9ca-4a6629f21347 Copy over a VERSION file from one of the <dfs.datanode.data.dir>/current/ directories of a healthy running DataNode: $ scp <healthy datanode host>:<dfs.datanode.data.dir>/current/VERSION ./ Modify the datanodeUuid in the VERSION file with the datanodeUuid from the above grep search: $ sed -i.bak -E 's|(datanodeUuid)=(.*$)|\1=1dacef53-aee2-4906-a9ca-4a6629f21347|' VERSION Blank out the storageID= property in the VERSION file: $ sed -i.bak -E 's|(storageID)=(.*$)|\1=|' VERSION Copy this modified VERSION file to the current/ path of every directory listed in dfs.datanode.data.dir property of hdfs-site.xml: $ for i in {0..9}; do cp VERSION /data$i/hadoop/hdfs/data/current/; done Change permissions on this VERSION file to be owned by hdfs:hdfs with permissions 644: $ for i in {0..9}; do chown hdfs:hdfs /data$i/hadoop/hdfs/data/current/VERSION; done $ for i in {0..9}; do chmod 664 /data$i/hadoop/hdfs/data/current/VERSION; done One more level down, there is a different VERSION file located under the Block Pool current folder at: /data0/hadoop/hdfs/data/current/BP-*/current/VERSION This file does not need to be modified -- just place then in the appropriate directories. Copy over this particular VERSION file from a healthy DataNode into the current/BP-*/current/ folder for each directory listed in dfs.datanode.data.dir of hdfs-site.xml: $ scp <healthy datanode host>:<dfs.datanode.data.dir>/current/BP-*/current/VERSION ./VERSION2 $ for i in {0..9}; do cp VERSION2 /data$i/hadoop/hdfs/data/current/BP-*/current/VERSION; done Change permissions on this VERSION file to be owned by hdfs:hdfs with permissions 644: $ for i in {0..9}; do chown hdfs:hdfs /data$i/hadoop/hdfs/data/current/BP-*/current/VERSION; done $ for i in {0..9}; do chmod 664 /data$i/hadoop/hdfs/data/current/BP-*/current/VERSION; done Restart DataNode from Ambari. The VERSION file located at <dfs.datanode.data.dir>/current/VERSION will have its storageID repopulated with a regenerated ID. If any data is not an issue (say, for example, the node was previously in a different cluster, or was out of service for an extended time), then delete all data and directories in the dfs.datanode.data.dir (keep that directory, though), restart the data node daemon or servic ... View more

@vishal dutt Spark driver is not able to find the sqljdbc.jar in class path. When using spark-submit , the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. That list is included in the driver and executor classpaths. Directory expansion does not work with --jars . Else 1) Provide the spark.driver.extraClassPath =/usr/hdp/hive/lib/mysql-connector-java.jar 2) Provide the spark.executor.extraClassPath = /usr/hdp/hive/lib/mysql-connector-java.jar. 3) Add Sqljdbc.jar to the Spark Classpath or add it using -jar option. Hope this helps you. ... View more

@Jorge Florencio Hope this article would help you: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-upgrade-ppc/content/upgrading_log_rotation_configuration.html If you are planning to change through log4j, even below article is good. https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.html Hope this helps you. ... View more

@Alpesh Virani Please try to use Ranger. You should be able to do that: Hope this link helps you : https://hortonworks.com/blog/best-practices-for-hive-authorization-using-apache-ranger-in-hdp-2-2/ ... View more

@Lokesh Mukku Seems like root user doesn't have HDFS home directory. Please create HDFS home for root user and try again. It should work. Below are steps to create home directory: sudo -u hdfs hadoop fs -mkdir /user/root sudo -u hdfs hadoop fs -chown root /user/root ... View more

@Sami Ahmad Using Phoenix: Use Salting to increase read/write performance Salting can significantly increase read/write performance by pre-splitting the data into multiple regions. Although Salting will yield better performance in most scenarios. Example: CREATE TABLE TEST (HOST VARCHAR NOT NULL PRIMARY KEY, DESCRIPTION VARCHAR) SALT_BUCKETS=16 Note: Ideally for a 16 region server cluster with quad-core CPUs, choose salt buckets between 32-64 for optimal performance. Per-split table Salting does automatic table splitting but in case you want to exactly control where table split occurs with out adding extra byte or change row key order then you can pre-split a table. Example: CREATE TABLE TEST (HOST VARCHAR NOT NULL PRIMARY KEY, DESCRIPTION VARCHAR) SPLIT ON ('CS','EU','NA') Use multiple column families Column family contains related data in separate files. If you query use selected columns then it make sense to group those columns together in a column family to improve read performance. Example: Following create table DDL will create two column faimiles A and B. CREATE TABLE TEST (MYKEY VARCHAR NOT NULL PRIMARY KEY, A.COL1 VARCHAR, A.COL2 VARCHAR, B.COL3 VARCHAR) Article: https://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/ ... View more

@pk reddy Seems like it's unable to assign resources. Please check resource utilization. ... View more

@vishal dutt Glad your problem is fixed ... View more

@pk reddy Can you see multiple outputs? Can you please provide logs? Once you run a job in tez, the session/container with Application master will be there for sometime, that doesn't mean it's running all the time. ... View more

@vishal dutt Please refer to this link which will give all the information. https://cwiki.apache.org/confluence/display/RANGER/Hive+Commands+to+Ranger+Permission+Mapping ... View more

@kanna k Try to set below parameters, hope it should work: add to ambari.properties: client.threadpool.size.max = 50 and then ambari-server restart ... View more

@Sebastian Fröhlich I suggest based Hive View 2 based on my experience with Views. Hive View 2 is much secured, sustainable and durable as it has more functionality. For better understand, you can refer below URL's: https://hortonworks.com/blog/3-great-reasons-to-try-hive-view-2-0/ https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-views/content/ch_using_hive_view.html Hope this helps you. ... View more

@Nikunj Solanki Please try to increase the sandbox memory, it will start. Else need to wait for longer time to start for the 1st time. Even I had similar issue but started after sometime. ... View more

@Yassine Did get chance to check below URL: 1. Added the following property to the core-site.xml on the client machine (secure cluster): <property> <name>ipc.client.fallback-to-simple-auth-allowed</name> <value>true</value> </property> 2. Added the following rule to the core-site.xml on the source cluster (non-secure) RULE:[1:$1@$0](.*@<replace with REALM name>)s/@.*// After adding the above properties you should be able to perform distcp successfully. Link: https://community.hortonworks.com/questions/294/running-distcp-between-two-cluster-one-kerberized.html Hope this helps you. ... View more

@Jose Correa Technically I should store only last two fsimages in the directory. May I know what you are referring to old fsimages means? You can delete them if they haven't been updated for days, but I would suggest better check with support team if that cluster has critical data. Example: Here is an example of an HDFS metadata directory taken from a NameNode. This shows the output of running the tree command on the metadata directory, which is configured by setting dfs.namenode.name.dir in hdfs-site.xml. data/dfs/name ├── current │ ├── VERSION │ ├── edits_0000000000000000001-0000000000000000007 │ ├── edits_0000000000000000008-0000000000000000015 │ ├── edits_0000000000000000016-0000000000000000022 │ ├── edits_0000000000000000023-0000000000000000029 │ ├── edits_0000000000000000030-0000000000000000030 │ ├── edits_0000000000000000031-0000000000000000031 │ ├── edits_inprogress_0000000000000000032 │ ├── fsimage_0000000000000000030 │ ├── fsimage_0000000000000000030.md5 │ ├── fsimage_0000000000000000031 │ ├── fsimage_0000000000000000031.md5 │ └── seen_txid └── in_use.lock fsimage_end transaction ID – This contains the complete metadata image up through . Each fsimage file also has a corresponding .md5 file containing a MD5 checksum, which HDFS uses to guard against disk corruption. Hope this helps. ... View more

@suresh krish There won't be any problem in adding 'disable.quoting.for.sv' to whitelist. Did you try setting system parameter at runtime and executing your query? beeline -u 'jdbc:hive2://localhost:10000/' --outputformat=csv2 -e 'set system:disable.quoting.for.sv=false; <query>' ... View more