Hi I have to design a spark streaming application with below use case. I am looking for best possible approach for this. I have application which pushing data into 1000+ different topics each has different purpose . Spark streaming will receive data from each topic and after processing it will write back to corresponding another topic. Ex. Input Type 1 Topic --> Spark Streaming --> Output Type 1 Topic Input Type 2 Topic --> Spark Streaming --> Output Type 2 Topic Input Type 3 Topic --> Spark Streaming --> Output Type 3 Topic . . . Input Type N Topic --> Spark Streaming --> Output Type N Topic and so on. I need to answer following questions. 1. Is it a good idea to launch 1000+ spark streaming application per topic basis ? Or I should have one streaming application for all topics as processing logic going to be same ? 2. If one streaming context , then how will I determine which RDD belongs to which Kafka topic , so that after processing I can write it back to its corresponding OUTPUT Topic? 3. Client may add/delete topic from Kafka , how do dynamically handle in Spark streaming ? 4. How do I restart job automatically on failure ? Any other issue you guys see here ? ... View more

Hi I have HDF cluster with 3 Nifi instance which lunches jobs(Hive/Spark) on HDP cluster. Usually nifi writes all information to different repositories available on local machine. My question is - Does nifi writes any data,provenance information or does spilling on HDP nodes (ex. data nodes in HDP cluster) while accessing HDFS,Hive or Spark services ? Thanks ... View more

Hi I have installed HDF 2.1.2 with 3 node Nifi 1.1 cluster and I am trying to configure controller service in it. I created a simple ExecuteHQL processor which depends on ThriftConnectionPool controller service. I dragged ExecuteProcessor on canvas and as soon as I clicked on gear icon for configuring controller service , it throws runtime exception in log. Attempting request for (anonymous) GET http://<hostname>:9090/nifi-api/flow/process-groups/83490f62-015c-1000-0000-00004da8f033/controller-servicesjavax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error Anyone faced this? ... View more

Hi I was reading about HDFS encryption at rest and found that when we create a encryption zone in hdfs , folder should be empty or should not be present. Is there any way in which i can convert existing folders into encryption zone which are already having some folders/data into it ? Note : I am using Ranger KMS as key management server. ... View more

Hi I am trying to impersonate kerberos in order to connect to hive but it is giving gss init exception. <code>UserGroupInformation ugi = kinit.generateKerberosTicket(configResources, keytab, principal); serGroupInformation ugiProxy = UserGroupInformation.createProxyUser("shashi", ugi.getCurrentUser()); ugiProxy.doAs( new PrivilegedExceptionAction<Void>(){ @Override public Void run() throws Exception { Connection con = DriverManager.getConnection("jdbc:hive2://quickstart.cloudera:10000/default;principal=hive/quickstart.cloudera@CLOUDERA", "shashi", ""); Statement stmt = con.createStatement(); String sql = "show databases "; ResultSet res = stmt.executeQuery(sql); if (res.next()) { System.out.println("DB names ---- >" +res.getString(1)); } makeHiveJdbcConnection(); return null; } But getting following exception . <code>java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://quickstart.cloudera:10000/default;principal=hive/quickstart.cloudera@CLOUDERA: GSS initiate failed Any pointer for this issue? ... View more

Hi I am trying to understand NiFi data flow mechanism . I read that Nifi has flow file which holds content and metadata (flow file attribute). So I wanted to understand if I have 1 TB of data placed on edge node and would like to pass it to Nifi processors , is it going to load everything into memory to be used by processor? ... View more

Hi I created a sample controller service - 'MyControllerService' and packaged it into nar and pasted into nifi lib directory. I restarted nifi service to see changes. I was able to see MyControllerService in Controller setting. After that i made small label change into controller service and followed same process but changes are not getting affected. Even i removed nar files from nifi/lib just to check if it getting removed from list. That too not happening. Even I dont see any exception in nifi/log. Any pointer for this issue? ... View more

Hi I was designing on my own custom processor. I added couple of simple property descriptor into it with simple non-empty validators. I was looking for a validator by which I can add multiple values into one property descriptor. Something like below. My property descriptor will have multi value selection option. Does anyone know how can I achieve it ? Thanks Shashi ... View more

Hi I am trying to create HDFS Admin super user. I referred below for another super user creation. Create HDFS Admin User I followed exact steps but after running hdfs dfsadmin -report report: Access denied for user abc. Superuser privilege is required. Any pointer here? how should I debug this? ... View more

Hi I want to run my NiFi application using ec2-user rather than default nifi user. I changed run.as=ec2-user in bootstrap.conf but it did not worked .It is not allowing me to start Nifi application getting following error while staring nifi service. ./nifi.sh start nifi.sh: JAVA_HOME not set; results may vary Java home: NiFi home: /opt/nifi/current Bootstrap Config File: /opt/nifi/current/conf/bootstrap.conf Error: Could not find or load main class org.apache.nifi.bootstrap.RunNiFi Any pointer to this issue? Thanks Shashi ... View more