@Mateusz Grabowski Check in the components installed and I am sure smartsense is still installed. Hortonworks will not be collecting data but smartsense is still doing its job. Check in the list of components you are installing and I am sure it is still installed. ... View more

@ROHIT AILA I understand. That's why, I think you need to call kafka-consumer-group.sh script from your java program to get what you are looking for. ... View more

@Wolfgang nobody Can you give "yarn" user the right to impersonate in core-site.xml? Try the following: <property> <name>hadoop.proxyuser.yarn.groups</name> <value>group your user is part of. Can be comma separated list or '*' (without quotes) for all</value> <description>Allow the superuser yarn to impersonate any members of the group mentioned in group list</description> </property> <property> <name>hadoop.proxyuser.yarn.hosts</name> <value>host1,host2</value> <description>The superuser can connect only from host1 and host2 to impersonate a user. Make it '*' so it can connect from any host</description> </property> ... View more

@ROHIT AILA Have you tried using consumer.position(TopicPartition partition)? or run the following script from your java program. You can also run the script with zookeeper instead of bootsrap-server. kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --new-consumer --group groupname --describe ... View more

@Tamil Selvan K Cloudbreak is used to provision, configure and elastically grow clusters in cloud. You would use Ambari to get cluster details you are looking for, which includes cluster name, configuration values and so on. And if you need to get these values without using the interface, you can use Ambari API. So, what you are looking for is available through Ambari and not cloudbreak. ... View more

@Prabhu Muthaiyan Here is how you would do it. first in your spark-env.sh set HADOOP_CONF_DIR to where your hdfs-site.xml, core-site.xml and hive-site.xml exist, such that your program when it runs is able to pick up these files and know how to connect to Hive. Then you basically code similar to below import org.apache.spark.sql.SparkSession; SparkSession spark = SparkSession .builder() .appName("Java Spark SQL basic example") .config("spark.some.config.option", "some-value") .enableHiveSupport() .getOrCreate(); DataSet<Row> emp1 = spark.sql("SELECT col1, col2, col3 from emp1 where <condition goes here>"); emp1.write().saveAsTable("emp2") ; //or use this emp1.write().mode("append").saveAsTable("emp2") ; you can have write modes which are following: SaveMode.Overwrite : overwrite the existing data. - SaveMode.Append : append the data. - SaveMode.Ignore : ignore the operation (i.e. no-op). - SaveMode.ErrorIfExists : default option, throw an exception at runtime ... View more

@Prabhu Muthaiyan Here is how you would do it. first in your spark-env.sh set HADOOP_CONF_DIR to where your hdfs-site.xml, core-site.xml and hive-site.xml exist, such that your program when it runs is able to pick up these files and know how to connect to Hive. Then you basically code similar to below import org.apache.spark.sql.SparkSession; SparkSession spark = SparkSession .builder() .appName("Java Spark SQL basic example") .config("spark.some.config.option", "some-value") .enableHiveSupport() .getOrCreate(); DataSet<Row> emp1 = spark.sql("SELECT col1, col2, col3 from emp1 where <condition goes here>"); emp1.write().saveAsTable("emp2") ; //or use this emp1.write().mode("append").saveAsTable("emp2") ; you can have write modes which are following: SaveMode.Overwrite : overwrite the existing data. - SaveMode.Append : append the data. - SaveMode.Ignore : ignore the operation (i.e. no-op). - SaveMode.ErrorIfExists : default option, throw an exception at runtime ... View more

@sally sally Is this the right XML you have copied here? It is not a valid XML. ... View more