Member since
09-26-2015
48
Posts
29
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6607 | 10-25-2016 12:53 PM | |
7651 | 10-22-2016 10:22 PM | |
4545 | 10-22-2016 09:34 PM | |
5203 | 10-21-2016 09:56 PM | |
2257 | 07-17-2016 05:26 PM |
07-14-2016
07:28 PM
@Krishna Srinivas Have you tried the Falcon mirroring feature ? Instead of cluster to cluster replication, you can try replicating to different directories in the same cluster. http://hortonworks.com/hadoop-tutorial/mirroring-datasets-between-hadoop-clusters-with-apache-falcon/ https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_data_governance/content/section_mirroring_data_falcon.html https://falcon.apache.org/HDFSDR.html
... View more
07-14-2016
03:59 PM
@ANSARI FAHEEM AHMED Are you referring to Hive/Tez job container sizes ? If yes, you can go to hive CLI and try set hive.tez.container.size; or if it a mapreduce job , you can try the same set command for mapreduce mapper or reducer mb.memory properties. if it is a generic YARN container size for any particular YARN application, then the containers are JVM processes, you can use yarn application commands to get the application attempt id and using the application attempt id, you can list the containers running for that APP. Doing a ps aux | grep <container pid> should give you enough details about the container size.
... View more
07-14-2016
03:43 PM
@ANSARI FAHEEM AHMED Are you referring to the running jobs on the cluster, as they acquire resources on YARN, the namenode heap is increasing ? If yes, your processes might be under the hood making lot of namenode requests which might be the reason for increase in namenode heap size as well. The namenode heap size will eventually get down after it hits Garbage Collection.
... View more
05-23-2016
02:33 PM
@Manoj Dhake Look at the below link for Atlas 0.7 http://atlas.incubator.apache.org/Bridge-Falcon.html Hope this helps!
... View more
04-05-2016
05:41 PM
Is there any data encryption option for Spark Thrift Server ?
... View more
Labels:
01-13-2016
02:58 PM
3 Kudos
We need to set up a HDP cluster based on Isilon storage and customer is asking how much impact would it have on the CPU usage on Isilon nodes ? Currently the Isilon cluster is shared with other work loads as well. What are our experiences around this ? Would the namenode operations in Isilon cause lot of CPU spikes which results in degradation in performance with other workloads on Isilon ?
... View more
Labels:
- Labels:
-
Apache Hive
11-17-2015
07:46 PM
Trying to import table data from Sybase table to Hive using the below command: sqoop import --verbose --driver com.sybase.jdbc4.jdbc.SybDriver --connect jdbc:sybase:Tds:dbgbl-tst:8032/DATABASE=trim_bw --username hrongali -P --table trim_bw..account --hive-database trim_bw --hive-table account --hive-import -m 1 Sqoop is generating the below alias(AS trim_bw..account) which is failing to execute in Sybase and the below exception is thrown: 2015-11-17 14:29:48,511 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: SELECT col_1, col_2, col_3, col_4 FROM trim_bw..account AS trim_bw..account WHERE ( 1=1 ) AND ( 1=1 )
2015-11-17 14:29:48,514 ERROR [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception:
com.sybase.jdbc4.jdbc.SybSQLException: Incorrect syntax near '.'.
at com.sybase.jdbc4.tds.Tds.processEed(Tds.java:4084)
at com.sybase.jdbc4.tds.Tds.nextResult(Tds.java:3174)
at com.sybase.jdbc4.tds.Tds.getResultSetResult(Tds.java:3940)
at com.sybase.jdbc4.tds.TdsCursor.open(TdsCursor.java:328)
at com.sybase.jdbc4.jdbc.SybStatement.executeQuery(SybStatement.java:2370)
at com.sybase.jdbc4.jdbc.SybPreparedStatement.executeQuery(SybPreparedStatement.java:264)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
Note: Tried with out giving the database name in the --table parameter, but the table object is not being recognized with that convention
... View more
Labels:
- Labels:
-
Apache Sqoop
11-12-2015
04:00 PM
3 Kudos
Sort Bucket the Hive table and Read the bucketed Hive table in Mapreduce program and hit Hbase when the Key changes. Requires programming effor, but very effective. Bucketing the Hive table will make sure that a particular key goes to only one bucket, so you hit Hbase Once for a particular key.
... View more
11-09-2015
04:58 PM
2 Kudos
The below Blog provides very good guideline too: http://hortonworks.com/blog/best-practices-for-hive-authorization-using-apache-ranger-in-hdp-2-2/
... View more
11-07-2015
09:35 PM
1 Kudo
Thanks Pardeep !
... View more
- « Previous
- Next »