@RANKESH NATH Architecture wise it is not a good option as spark is distributed processing framework (running on multiple nodes) and spark writing to MYSQL on a single node will create performance bottleneck as MYSQL is not distributed.If you want to make full use of Sparks distributed processing capability then it is advisable to use either distributed file system or distributed database. ... View more

@MyeongHwan Oh Do you mean transferring data to Hadoop data node. You can use your own test data in Oracle and import it using sqoop as there are no specific data datasets or benchmarking results for sqoop. Also, There are few common performance improvement techniques for Sqoop e.g. split-by and boundary-query, direct, fetch-size, num-mapper Please find below links as they are good starting points https://community.hortonworks.com/articles/70258/sqoop-performance-tuning.html https://kb.informatica.com/h2l/HowTo%20Library/1/0930-SqoopPerformanceTuningGuidelines-H2L.pdf http://www.xmsxmx.com/performance-tuning-data-load-into-hadoop-with-sqoop/ https://dzone.com/articles/apache-sqoop-performance-tuning If this answer your question, please vote/accept best answer. ... View more

Can you send a sample file. I will try it on my machine. ... View more

Can you please verify the back-pressure threshold of previous processor, which means once the threshold is reached the components right before the queue will no longer be triggered to run anymore until the queue goes below the threshold. You can see it is red in color. You can increase the threshold limit. ... View more

@Artem Ervits This is a great article Do we have performance comparison of HIVE on HDFS vs HIVE on HBase. Is it advisable to come up with Hive on Hbase in production for large datasets? ... View more

@Vedant Jain I used to work in healthcare domain. We used to parse HL7 file and used mapping file to map hl7 attributes to store into HBase entities based on below models. http://www.fhims.org/content/420A62FD03B6_root.html https://www.ibm.com/support/knowledgecenter/en/SS9NBR_9.1.0/com.ibm.ima.toc/welcome_version.html While designing the model different approached can be followed based on access patterns. ... View more

@Balakumar Balasundaram Does the explanation and link I provided address your question? If so, please "accept" the answer to close the posting. ... View more

@priyal patel Does the explanation provided address your question? If so, please "accept" the answer to close the posting. ... View more

@priyal patel You can look at the application logs from the failed application. To fetch them run the following command as the user who ran the Sqoop command. yarn logs -applicationId application_1496289796598_0013 > appln_logs.txt appln_logs.txt would contain more details on the errors. Please post them here If you can't figure out. ... View more

@priyal patel Sqoop - Sqoop is used to move data from an existing RDBMS to Hadoop (or vice versa). Once the data is imported from Sqoop initially (i.e initial load is performed), the incremental data (i.e the data which is updated in RDBMS) is not updated automatically. It needs to be imported incrementally using incremental imports . https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports Flume - Flume was the main tool previously to ingest log files, events, flat files, csv, etc. Flume has recently fallen out of favour and is often being replaced with HDF(Hortonworks DataFlow) /NiFi. HDF(Hortonworks DataFlow)/NiFi - NiFi provides a visual user interface with more than 180+ processors for collecting data from various sources (Sensors,geo-location devices, machine, logs, files, feeds etc) & perform simple event processing (e.g. parsing, filtering etc) & delivering to storage platforms such as HDP in a secure environment. Kafka is a distributed fault tolerant messaging system which lets you to publish and subscribe to streams of records. Generally it is used while doing real time stream processing where real time messages are buffered in Kafka and are consumed by storm or spark streaming. The right tool for the job depends on your use case. Here is another good write up on the same subject: https://community.hortonworks.com/questions/23337/best-tools-to-ingest-data-to-hadoop.html As always, if you find this post useful, please accept the answer. ... View more

@Balakumar Balasundaram The main disadvantage of RDDs is that they don’t perform particularly well whenever Spark needs to distribute the data within the cluster, or write the data to disk, it does so using Java serialization by default (although it is possible to use Kryo as a faster alternative in most cases). The overhead of serializing individual Java and Scala objects is expensive and requires sending both data and structure between nodes (each serialized object contains the class structure as well as the values). There is also the overhead of garbage collection that results from creating and destroying individual objects. So in Spark 1.3 DataFrame API was introduced which seeks to improve the performance and scalability of Spark. The DataFrame API introduces the concept of a schema to describe the data, allowing Spark to manage the schema and only pass data between nodes, in a much more efficient way than using Java serialization. There are also advantages when performing computations in a single process as Spark can serialize the data into off-heap storage in a binary format and then perform many transformations directly on this off-heap memory, avoiding the garbage-collection costs associated with constructing individual objects for each row in the data set. Because Spark understands the schema, there is no need to use Java serialization to encode the data. Query plans are created for execution using Spark catalyst optimiser. After an optimised execution plan is prepared going through some steps, the final execution happens internally on RDDs only but thats completely hidden from the users. Please find below the list of useful blogs : https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html http://www.adsquare.com/comparing-performance-of-spark-dataframes-api-to-spark-rdd/ https://www.youtube.com/watch?v=1a4pgYzeFwE http://why-not-learn-something.blogspot.in/2016/07/apache-spark-rdd-vs-dataframe-vs-dataset.html https://www.youtube.com/watch?v=pZQsDloGB4w ... View more

@Sonu Sonu You can use FCC Conversion API to get fips code from lat & long For example: http://data.fcc.gov/api/block/2010/find?latitude=40.0&longitude=-85 Also, Please check out http://www.datasciencetoolkit.org/ a ready to use virtual machine(VM) for geocoding and reverse geocoding, it provides useful information including FIPS codes. i hope it will help. ... View more

@Bala Vignesh N V Capacity planning depends on multiple factors - - Amount of data that needs needs to be stored, the incremental data growth rate for next 2-3 years & data retention period - Kind of processing Real time or Batch - Use cases - Based on the use case, Workload patterns can be derived - Balance workload(Work load equally distributed - CPU bound, Disk I/O Bound, Network I/O bound) , Compute intensive (CPU bound - involving complex algorithms, NLP, HPCC etc ) or I/O intensive - Jobs requiring very less compute power and more of I/O (Archival use case have lots of cold data). If you don’t know the workload pattern then it is recommended to start with balanced workload - The SLA’s for the system. You need to consider all the above factors. It really depends on what you want to do with the data & need to look at each and every single piece. For e.g. Ingesting data from different sources (Sqoop,Flume, NiFi etc.), transformations if any ( pig/hive ) ,consumption ( hive ) or real time processing - Distributed message queue (Kafka ), Storage (HBase), . The best strategy in my opinion is to setup a development cluster and test it out, then scale it up. Hadoop is designed in a way that most tasks scale linearly with resources allocated to them. Below 2 links are good starting point : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_cluster-planning/content/ch_hardware-recommendations_chapter.html http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_cluster-planning/content/index.html ... View more