If you'd like to generate some data to test out the HDP/HDF platforms at a larger scale, you can use the following GitHub repository: https://github.com/anarasimham/data-gen This will allow you to generate two types of data: Point-of-sale (POS) transactions, containing data such as transaction amount, time stamp, store ID, employee ID, part SKU, and quantity of product. These are transactions you make at a store when you are checking out. For simplicity's sake, this assumes each shopper only buys one product (potentially greater than 1 in quantity) Automotive manufacturing parts production records, simulating the completion of parts in an assembly line. Imagine a warehouse completing different components of a car, such as the hood, front bumper, etc. at different points in time and those parts being tested for heat and vibration thresholds. This data will contain a timestamp of when the part was produced, thresholds for heat & vibration, values as tested for heat & vibration, quanity of produced part, a "short name" identifier for the part, a notes field, and a part location Full details of both schemas are documented in the code in file datagen/datagen.py at the repository above. The application is able to generate data and insert into one of two supported locations: Hive MySQL You will need to configure the table by running one of the scripts in the mysql folder after connecting to the desired server and the desired database as the desired user. Once that is done, you can copy the inserter/mysql.passwd.template file into inserter/mysql.passwd and edit it to provide the correct details. If you'd like to insert into Hive, do the same with the hive.passwd.template file. After editing, you can execute using the following command: python main_manf.py 10 mysql This will insert 10 rows of manufacturing data into the configured MySQL database table. At this point, you're ready to explore your data in greater detail. Possible next steps include using NiFi to pull the data out of MySQL and push into Druid for a dashboard-style data lookup workflow. You can also push into Hive for ad-hoc analyses. These activities are out of scope for this article but are suggestions to think about. ... View more

Assumptions: -You have a running HDP cluster with Sqoop installed -Basic knowledge of Sqoop and its parameters Ingesting SAP HANA data with Sqoop To ingest SAP HANA data, all you need is a JDBC driver. To the HDP platform, HANA is just another database - drop the JDBC driver in and you can plug & play. 1. Download the JDBC driver. This driver is not publicly available - it is only available to customers using the SAP HANA product. Find it on their members-only website and download it. 2. Drop the JDBC driver into Sqoop's lib directory. For me, this is located at /usr/hdp/current/sqoop-client/lib 3. Execute a Sqoop import. This command has many variations and many command-line parameters, but the following is one such example. sqoop import --connect "jdbc:sap://<HANA_SERVER>:30015" --driver com.sap.db.jdbc.Driver --username <YOUR_USERNAME> --password <PASSWORD> --table "<TABLE_NAME>" --target-dir=/path/to/hdfs/dir -m 1 -- --schema "<YOUR_SCHEMA_NAME>" The '-m 1' argument will limit Sqoop to using one thread, so don't use this if you want parallelism. You'll need to use the --split-by argument and give it a column name to be able to parallelize the import work. If all goes well, Sqoop should start importing the data into your target directory. Happy Sqooping! ... View more