@nejm hadjmbarek Set below property in oozie-site.xml to * to resolve this issue hadoop.proxyuser.oozie.host ... View more

@nejm hadjmbarek You can achieve this in two ways. 1. Create a wrapper shell script and call "pig <pig script path>" inside it. After that you can create an Unix cron entry to schedule it as per your requirement. 2. Another way is through Oozie scheduler, for that you can either create a pig action and along with recurring a coordinator service(see below link) or you can also create an Oozie shell action and call the same wrapper shell script inside your Oozie shell action( point 1). http://rogerhosto.com/apache-oozie-shell-script-example/ https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a3.2.3_Pig_Action http://blog.cloudera.com/blog/2013/01/how-to-schedule-recurring-hadoop-jobs-with-apache-oozie/ Thanks ... View more

Hi @nejm hadjmbarek, I'm Posting my code, which is working Fine: import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.PreparedStatement; import java.sql.Statement; public class phoenix_hbase { public static void main(String[] args) throws SQLException { @SuppressWarnings("unused") Statement stmt = null; ResultSet rset = null; try { Class.forName("org.apache.phoenix.jdbc.PhoenixDriver"); } catch (ClassNotFoundException e1) { System.out.println("Exception Loading Driver"); e1.printStackTrace(); } try { Connection con = DriverManager.getConnection("jdbc:phoenix:172.31.124.43:2181:/hbase-unsecure"); //172.31.124.43 is the adress of VM, not needed if ur running the program from vm itself stmt = con.createStatement(); PreparedStatement statement = con.prepareStatement("select * from javatest"); rset = statement.executeQuery(); while (rset.next()) { System.out.println(rset.getString("mycolumn")); } statement.close(); con.close(); } catch(Exception e) { System.out.println(e.getMessage()); } } } ... View more

@nejm hadj First I’ll answer your question and then I’ll make my recommendation. Answer: The name of the file does not matter. When setting up a Hive external table just specify the data source as the folder that will contain all the files (regardless of names). Details on setting up and external table: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/moving_data_from_hdfs_to_hive_external_table_method.html Details on reading/parsing JSON files into Hive: http://hortonworks.com/blog/howto-use-hive-to-sqlize-your-own-tweets-part-two-loading-hive-sql-queries/ (alternatively, you can convert JSON to CSV within NiFi. To do so, follow the NiFi portion of this example https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.html) Recommendation: HDFS prefers large files with many entries as opposed to many files with small entries. The main reason being that for each file landed on HDFS, file information is saved in the NameNode (in memory). If you’re putting each twitter message in a separate file you will quickly fill up your NameNodes’s memory and overload the server. I suggest you aggregate multiple messages into one file before writing to HDFS. This can be done with the MergeContent processor in Nifi. Take a look at the below screenshots showing how it would be set up. Also, take a look at the NiFi Twitter_Dashboard.xml example template (https://raw.githubusercontent.com/abajwa-hw/ambari-nifi-service/master/demofiles/Twitter_Dashboard.xml). You can import this into your NiFi by by clicking on Templates (third icon from right) which will launch the 'Nifi Flow templates' popup, and selecting the file. ... View more