Hi Team, We need to move TBs of data from RDBMS to HBase. Using HDP 2.6.5 in AWS. What is the best possible way to do this activity? As of now I could find below options: Option-1 : Use Sqoop to migrate. But can be slow because it uses "put" command to load data. But advantage is that its simple to right and can handle new line characters which we have in some fields in RDBMS Option-2 : Create delimited files from RDBMS by taking care of new line somehow (not sure). Then use the HBase bulk load utility to load data in HBase Options-3: Create delimited from RDBMS by taking care of new line somehow (not sure). Then use Spark HBase SHC connector to load data in HBase. Can you guys please guide on the best option or if any other option I missed out. Thanks ... View more

Hi Team, We have a use case where data from source system will get migrated to AWS S3 buckets in some specific format as files and need to load the data to HBase table. We are using HDP 2.6.5. Can you please help in providing the solution for this problem case? Note that we will have data in TBs in S3. ... View more

Hi Team, We are using HDP 2.6.5 and we are trying to test Spark and HBase integration but we are facing issues while testing. We are following the link https://github.com/hortonworks-spark/shc. Everytime we are facing connection refused from the Zookeeper. Please help on this issue as we are stuck. We need to build DF over HBase tables. Note that the HDP cluster is Kerberized. Below is the screen dump: 18/09/05 08:47:38 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 18/09/05 08:47:38 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 18/09/05 08:47:38 WARN RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid 18/09/05 08:47:39 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused ... View more

Hello Team, We have a large number of small files in HDFS and periodically we want to merge small files as handling too many small files in HDFS can be an issue. Looking for advice on how to handle that. One possible solution is to use Haddop Archive but that would result in changing the Hive external table file location. Situation is we will get small files every day and we want to merge all the files for a month into one large file. Thanks and Regards, Rajdip ... View more

Hello Team, Looking for your advice for this issue. We have an use case where almost 50TB of files need to be moved from local file server to HDFS. The files are kept under multiple folders in local file system and we need to maintain a similar HDFS folder structure. Looking for suggestions, any utilities through which we achieve this ojective. Let me know in case you need any more information. Thanks and Regards, Rajdip ... View more

Hello Guys, We have couple of use cases to implement related to source being a Mainframe system where data are kept as large files. These data needs to be offloaded to Hadoop both in Batch and Real Time mechanism for further downstream analytics. Wanted to know on implementation of below processes : 1) Batch load of Mainframe data (both as files and from DB2) to Hadoop 2) Real Time ingestion mechanism (incremental) of Mainframe data (both as file and from DB2) to Hadoop. Looking for your help to come with the architecture and ideas for implementation. Thanks and Regards, Rajdip ... View more

Hi Guys, We are setting up a HDP-2.6.0.3 cluster on 4 nodes having CentOS6.5. While setting up through Ambari we are getting below error. Please note that we are using local repository for set up. Need your URGENT help. Spark2 client install issue: stderr: /var/lib/ambari-agent/data/errors-450.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 64, in <module> MysqlServer().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 33, in install self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 605, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 54, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 51, in install_package self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 86, in checked_call_with_retries return self._call_with_retries(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 98, in _call_with_retries code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-server' returned 1. Error: Nothing to dostdout: /var/lib/ambari-agent/data/output-450.txt 2017-06-13 14:16:23,830 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6 2017-06-13 14:16:23,830 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf User Group mapping (user_group) is missing in the hostLevelParams 2017-06-13 14:16:23,831 - Group['livy'] {} 2017-06-13 14:16:23,832 - Group['spark'] {} 2017-06-13 14:16:23,832 - Group['hadoop'] {} 2017-06-13 14:16:23,833 - Group['users'] {} 2017-06-13 14:16:23,833 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,833 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,834 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,834 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,835 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-06-13 14:16:23,835 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,836 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,836 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,837 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-06-13 14:16:23,837 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,837 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,838 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,838 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,839 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-06-13 14:16:23,839 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-06-13 14:16:23,840 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2017-06-13 14:16:23,846 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2017-06-13 14:16:23,846 - Group['hdfs'] {} 2017-06-13 14:16:23,846 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']} 2017-06-13 14:16:23,847 - FS Type: 2017-06-13 14:16:23,847 - Directory['/etc/hadoop'] {'mode': 0755} 2017-06-13 14:16:23,865 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2017-06-13 14:16:23,865 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2017-06-13 14:16:23,878 - Initializing 2 repositories 2017-06-13 14:16:23,879 - Repository['HDP-2.6'] {'base_url': 'http://3.209.124.205/HDP/centos6', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None} 2017-06-13 14:16:23,886 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.6]\nname=HDP-2.6\nbaseurl=http://3.209.124.205/HDP/centos6\n\npath=/\nenabled=1\ngpgcheck=0'} 2017-06-13 14:16:23,887 - Repository['HDP-UTILS-1.1.0.21'] {'base_url': 'http://3.209.124.205/HDP-UTILS-1.1.0.21', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None} 2017-06-13 14:16:23,891 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.21]\nname=HDP-UTILS-1.1.0.21\nbaseurl=http://3.209.124.205/HDP-UTILS-1.1.0.21\n\npath=/\nenabled=1\ngpgcheck=0'} 2017-06-13 14:16:23,892 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-06-13 14:16:23,936 - Skipping installation of existing package unzip 2017-06-13 14:16:23,936 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-06-13 14:16:23,946 - Skipping installation of existing package curl 2017-06-13 14:16:23,946 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-06-13 14:16:23,956 - Skipping installation of existing package hdp-select 2017-06-13 14:16:24,104 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-06-13 14:16:24,109 - call['ambari-python-wrap /usr/bin/hdp-select status hive-server2'] {'timeout': 20} 2017-06-13 14:16:24,126 - call returned (0, 'hive-server2 - 2.6.0.3-8') 2017-06-13 14:16:24,127 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6 2017-06-13 14:16:24,130 - File['/var/lib/ambari-agent/cred/lib/CredentialUtil.jar'] {'content': DownloadSource('http://nn.tcsgegdc.com:8080/resources/CredentialUtil.jar'), 'mode': 0755} 2017-06-13 14:16:24,131 - Not downloading the file from http://nn.tcsgegdc.com:8080/resources/CredentialUtil.jar, because /var/lib/ambari-agent/tmp/CredentialUtil.jar already exists 2017-06-13 14:16:24,132 - checked_call[('/usr/java/jdk1.8.0_131/bin/java', '-cp', '/var/lib/ambari-agent/cred/lib/*', 'org.apache.ambari.server.credentialapi.CredentialUtil', 'get', 'javax.jdo.option.ConnectionPassword', '-provider', 'jceks://file/var/lib/ambari-agent/cred/conf/hive/hive-site.jceks')] {} 2017-06-13 14:16:24,683 - checked_call returned (0, 'SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".\nSLF4J: Defaulting to no-operation (NOP) logger implementation\nSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.\nJun 13, 2017 2:16:24 PM org.apache.hadoop.util.NativeCodeLoader <clinit>\nWARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\nhive') 2017-06-13 14:16:24,688 - Package['mysql-server'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-06-13 14:16:24,739 - Installing package mysql-server ('/usr/bin/yum -d 0 -e 0 -y install mysql-server') 2017-06-13 14:16:24,992 - Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-server' returned 1. Error: Nothing to do 2017-06-13 14:16:24,993 - Failed to install package mysql-server. Executing '/usr/bin/yum clean metadata' 2017-06-13 14:16:25,093 - Retrying to install package mysql-server after 30 seconds Command failed after 1 tries ... View more

Hi, We are using HDP-2.5.3 and facing issues in sqoop incremental import. Detailed explanation as below: 1) Used sqoop to import initial data. 2) In source (Oracle), we have some changes like 1 data is inserted and 2 data got updated. 3) Using incremental we are trying to get the changes merged but its not happening. Screenshot for updated data: Sqoop command for incremental load. But while using this command data is not getting merged. Note that from above screen shot record witth value 8000 is the new record and other 2 are updated records. sqoop import -Dmapreduce.job.user.classpath.first=true --connect jdbc:oracle:thin:@//3.209.124.204:1521/eqlmpoc --fetch-size 1000 --username utility_test --password utility_test --table UTILITY_TEST.EMP_DETAILS --check-column HIREDATE --incremental lastmodified --last-value "2017-05-28 16:01:12" --target-dir hdfs://nn.tcsgegdc.com:8020/data/dlingestion/DEMO/FINAL/EMP_DETAILS --merge-key SSN -m 1 --class-name UTILITY_TEST.EMP_DETAILS ... View more

Hi Guys, We are trying to use Oozie for our ETL process by connecting to Hive2 but while working with it facing issues during the jobs. Need your help to resolve the issue. HDP-2.5.3 is deployed in our cluster and below the workflow.xml file: <workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1"> <start to='hive2tab'/> <action name='hive2tab'> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>aa.bb.com:8050</job-tracker> <name-node>hdfs://cc.dd.com:8020</name-node> <jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url> <password>ZZZZ</password> <script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script> </hive2> <ok to='end'/> <error to='end'/> </action> <kill name='kill'> <message>JOb_failed</message> </kill> <end name='end'/> </workflow-app> This is a simple job to execute a .hql script. However, when running this job we can see the below error in the Oozie log: 2017-05-18 17:25:32,510 WARN Hive2ActionExecutor:523 - SERVER[dn2.tcsgegdc.com] USER[tcs_ge_user] GROUP[-] TOKEN[] APP[Meg_Test1] JOB[0000004-170518130535586-oozie-oozi-W] ACTION[0000004-170518130535586-oozie-oozi-W@Hive1_Table] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2212) ... 9 more Please help us in resolving this issue as this is very important for us. Thanks and Regards, Rajdip ... View more