Hi, I need to get output like total-count-of-records-InTable, count(max(date)) using Hive. Where count(max(date)) ----> gives the count of records loaded from last run. Each record has current-run-date column. I know to get both separately, but not sure to get the output together... Could anyone help me on this... Appreciate your help.. Thanks, Jee ... View more

Hi there, I have got a project where I need to decide the storage system and then schema design for the below problem scenario. The below is the data which we receive as CSV file for every 1 hour with 1TB of data. recipe_id,recipe_name, description, ingredient, active, updated_date, created_date 1, pasta, Italian pasta, tomato sauce, true, 2018-01-09 10:00:57, 2018-01-10 13:00:57 1, pasta, null, cheese, true, 2018-01-09 10:10:57, 2018-01-10 13:00:57 2, lasagna, layered lasagna, cheese, true, 2018-01-09 10:00:57, 2018-01-10 13:00:57 2, lasagna, layered lasagna, blue cheese, false, 2018-01-09 10:00:57, 2018-01-10 13:00:57 …. Requirement: 1. XXX need to show a page with list of all the recipes and when user clicks on each of the recipe they want to show the Recipe page with their ingredients. They also want user to further click into each of the ingredient and see all the recipes linked to that ingredients. Create a data model which can store this data to allow user to do the activities mentioned above. This data model needs to support millions of read per second. Which persistence system will be the best for this scenario? Write a Spark Job in Scala which can takes the CSV shown above and store that in the storage system of your choice using the data model you discussed above. I know Hive will not support this real time requirement, because it is for batch processing only and there is no random access concept. I think I can go with Hbase but I am not sure about the data model. If i know the data model, I can write an application on Spark using Scala to export the data to Hbase as per the data model. Could anyone please help with the storage system and the data model? Appreciate your help! Thanks, Krishna ... View more

Hi There, SBT gives error like "The java installation you have is not up to date sbt requires at least version 1.6+, you have version 0" , even though I have JDK 1.8 installed. If I give "java -version" in windows command prompt, it displays java 1.8. How to make SBT to point this JDK? Thanks, Jeeva ... View more

Hi there, I am trying to execute hdfs groups <username> command on commandline, but it gives error like below. I could execute the same command on DEV region to see the list of groups that the user belongs to, but QA gives error. FYI, we are using ISILON(Hadoop) cluster in QA region and HDP 2.5.3. Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unknown protocol: org.apache.hadoop.tools.GetUserMappingsProtocol at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy8.getGroupsForUser(Unknown Source) at org.apache.hadoop.tools.protocolPB.GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser(GetUserMappingsProtocolClientSideTranslatorPB.java:57) at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96) Thanks, jee ... View more

Hi there, I have created a directory /tmp/ran-test with access permission(400) in HDFS. I tried to copy a file to the directory and got permission denied error. But, my colleague could able to copy files to the directory without any error. I am wondering now how it works. I would appreciate if somebody give their explanation. For your information, We are using Active Directory. Thanks, Jee ... View more

Hi there, In my work place, HDFS plugin for Ranger has been enabled. I created a policy for the source in hdfs /tmp/ranger_test(which has access permission 400 in HDFS). I can see that the policy has been synced in ranger Plugins tab. But, It is not showing up any audit logs and it does not enforce the ranger policy while accessing the directory in hdfs. For your information, the audit log is enabled. Don't know the reason why it does not work in the way i expected. No audit logs are displayed in ranger Audit tab. But audit are enable for hdfs to solr. Please let me know what would be the reason and how to troubleshoot it. Thanks, kJ ... View more

Hi there, In my working place, We have Datameer server Installed and it uses Git-Plugin. The Git repository has been installed in the Datameer server. Because as mentioned in the Datameer documentation, (To use this Git plug-in, you need to install Git on the same machine as the Datameer server. The Datameer service depends on the Git repository for constant writes, so a low response time is necessary. As a result, the repository folder needs to be locally set up on Datameer server and cannot be remote) the git repository cannot be locally. This is the link for the documentation. https://www.datameer.com/documentation/display/DAS50/Using+the+Git+Versioning+Plug-in But, our organization uses BitBucket. So, how to create a Repository in BitBucket and sync the Datameer-server's repo with BitBucket repo. Because, if the datameer server goes down, we can still have the Bitbucket repo. Also, keeping the repository history indatameer-sever may affect the server performance. I would appreciate If you share your knowledge on this. Thanks, Jee ... View more

Hi therem I am pretty new to Datameer. I just want to Know that What is Datameer? How Datameer is related to Hadoop? How Git is used with Datameer? Thanks in advance. ... View more

Hi there, I know how to Kafka broker configuration properties in Ambari. But, Not sure, How to check Consumer properties in Amabari. I would appreciate your reply on this. Thanks in advance. ... View more

Hi there, I am just executing some KafkaTestCases on console. Zookeeper and Kafka brokers are started(I am seeing this in Ambari). There are 3 Zookeepers and 2 Kafka-Brokers are running. I ran "bin/kafka-console-producer.sh --broker-list EdgeNodeAddress:xxxx --topic test21" Note: EdgeNodeAddress is the console where i am issuing the producer console command and I ran " bin/kafka-console-consumer.sh --zookeeper zookeeperaddress:2181 --topic test21 --from-beginning" As soon as i ran the Kafka-Console-Consumer.sh script, It keeps on generating message like {metadata.broker.list=xxxxxxx2.xxxx:7903,xxxxxxxx2.xxxxxx:7903, request.timeout.ms=30000, client.id=console-consumer-1927, security.protocol=PLAINTEXT} This message is keeps on generating until i stop it myself. It goes on like "while loop without meeting condition" I troubleshooted zookeeper zkcli.sh to check if i could see the broker list(ls /brokers/ids) and it shows the broker ids [0,1] I am not sure what is the issue. Please share your knowledge on this ASAP. Thanks in advance. ... View more

Hi there, I am running kafka on console for testing purpose. While executing kafka-consumer.sh, I am getting a message continuously without stopping like below: {metadata.broker.list=xxxxxxxv.xxxxxxx.com:7xx3,dpxxxxxxx.bxxxx.com:7xxx, request.timeout.ms=30000, client.id=console-consumer-xx36x, security.protocol=PLAINTEXT} metadata.broker.list=xxxxxxxv.xxxxxxx.com:7xx3,dpxxxxxxx.bxxxx.com:7xxx, request.timeout.ms=30000, client.id=console-consumer-xx36x, security.protocol=PLAINTEXT} metadata.broker.list=xxxxxxxv.xxxxxxx.com:7xx3,dpxxxxxxx.bxxxx.com:7xxx, request.timeout.ms=30000, client.id=console-consumer-xx36x, security.protocol=PLAINTEXT} This message keeps on going until i stop this. Please share your knowledge on this to fix this issue. Thanks in advance. ... View more

Hi there, I am running sqoop to import data from Netezza to HDFS. I am getting error like below, NzSQLException: LDAP authentication failed for user 'JKRIS03' Please help me how to overcome this issue. @Frank Lu @Benjamin Leonhardi @ @fra ... View more

Hi there, I am running sqoop cmd to connect to netezza database. so i need to add the connector jar. In my workplace, they don't want me to put in sqoop/lib dir. So, what are the other way i can add the jar so that sqoop can use it. Now i am getting error like couldn't find the connector. The connertor in not available in sqoop/lib dir. I don't have permission to the sqoop/lib dir. Please let me know ASAP. Thanks ... View more