Member since
02-03-2016
123
Posts
23
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1495 | 04-13-2017 08:09 AM |
09-13-2018
02:46 PM
Hi Team, We need to move TBs of data from RDBMS to HBase. Using HDP 2.6.5 in AWS. What is the best possible way to do this activity? As of now I could find below options: Option-1 : Use Sqoop to migrate. But can be slow because it uses "put" command to load data. But advantage is that its simple to right and can handle new line characters which we have in some fields in RDBMS Option-2 : Create delimited files from RDBMS by taking care of new line somehow (not sure). Then use the HBase bulk load utility to load data in HBase Options-3: Create delimited from RDBMS by taking care of new line somehow (not sure). Then use Spark HBase SHC connector to load data in HBase. Can you guys please guide on the best option or if any other option I missed out. Thanks
... View more
- Tags:
- Data Processing
- HBase
Labels:
- Labels:
-
Apache HBase
09-05-2018
01:05 PM
Hi Team, We are using HDP 2.6.5 and we are trying to test Spark and HBase integration but we are facing issues while testing. We are following the link https://github.com/hortonworks-spark/shc. Everytime we are facing connection refused from the Zookeeper. Please help on this issue as we are stuck. We need to build DF over HBase tables. Note that the HDP cluster is Kerberized. Below is the screen dump: 18/09/05 08:47:38 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 18/09/05 08:47:38 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 18/09/05 08:47:38 WARN RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid 18/09/05 08:47:39 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Spark
03-22-2018
01:23 PM
Hello Team, We have a large number of small files in HDFS and periodically we want to merge small files as handling too many small files in HDFS can be an issue. Looking for advice on how to handle that. One possible solution is to use Haddop Archive but that would result in changing the Hive external table file location. Situation is we will get small files every day and we want to merge all the files for a month into one large file. Thanks and Regards, Rajdip
... View more
Labels:
- Labels:
-
Apache Hadoop
03-22-2018
01:16 PM
Hello Team, Looking for your advice for this issue. We have an use case where almost 50TB of files need to be moved from local file server to HDFS. The files are kept under multiple folders in local file system and we need to maintain a similar HDFS folder structure. Looking for suggestions, any utilities through which we achieve this ojective. Let me know in case you need any more information. Thanks and Regards, Rajdip
... View more
Labels:
- Labels:
-
Apache Hadoop
11-09-2017
06:23 AM
Can you guys please post some guidance here?
... View more
11-08-2017
06:44 AM
Hi Guys, Really look for some guidance here. Appreciate your help.
... View more
11-07-2017
07:27 AM
Hello Guys, We have couple of use cases to implement related to source being a Mainframe system where data are kept as large files. These data needs to be offloaded to Hadoop both in Batch and Real Time mechanism for further downstream analytics. Wanted to know on implementation of below processes : 1) Batch load of Mainframe data (both as files and from DB2) to Hadoop 2) Real Time ingestion mechanism (incremental) of Mainframe data (both as file and from DB2) to Hadoop. Looking for your help to come with the architecture and ideas for implementation. Thanks and Regards, Rajdip
... View more
Labels:
- Labels:
-
Apache Hadoop
06-13-2017
10:44 AM
Hi Team, After addressing the 2 issues we are able to progress with the cluster set up activity but facing below error under the step "Check YARN" in one of the nodes. Detailed log below. Really looking for your help as we are not getting any help over the community. Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 181, in <module>
ServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 117, in service_check
user=params.smokeuser,
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 --queue default' returned 2. 17/06/13 15:56:38 INFO distributedshell.Client: Initializing Client
17/06/13 15:56:38 INFO distributedshell.Client: Running Client
17/06/13 15:56:39 INFO client.RMProxy: Connecting to ResourceManager at nn.tcsgegdc.com/3.209.124.205:8050
17/06/13 15:56:39 INFO client.AHSProxy: Connecting to Application History server at nn.tcsgegdc.com/3.209.124.205:10200
17/06/13 15:56:39 INFO distributedshell.Client: Got Cluster metric info from ASM, numNodeManagers=3
.........................................................................
17/06/13 15:56:40 INFO impl.YarnClientImpl: Submitted application application_1497349747602_0006
17/06/13 15:56:41 INFO distributedshell.Client: Got application report from ASM for, appId=6, clientToAMToken=null, appDiagnostics=AM container is launched, waiting for AM container to Register with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, appStartTime=1497349916566, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=http://nn.tcsgegdc.com:8088/proxy/application_1497349747602_0006/, appUser=ambari-qa
17/06/13 15:56:42 INFO distributedshell.Client: Got application report from ASM for, appId=6, clientToAMToken=null, appDiagnostics=AM container is launched, waiting for AM container to Register with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, appStartTime=1497349916566, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=http://nn.tcsgegdc.com:8088/proxy/application_1497349747602_0006/, appUser=ambari-qa
17/06/13 15:56:43 INFO distributedshell.Client: Got application report from ASM for, appId=6, clientToAMToken=null, appDiagnostics=AM container is launched, waiting for AM container to Register with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, appStartTime=1497349916566, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=http://nn.tcsgegdc.com:8088/proxy/application_1497349747602_0006/, appUser=ambari-qa
17/06/13 15:56:44 INFO distributedshell.Client: Got application report from ASM for, appId=6, clientToAMToken=null, appDiagnostics=, appMasterHost=dn3.tcsgegdc.com/3.209.124.208, appQueue=default, appMasterRpcPort=-1, appStartTime=1497349916566, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=http://nn.tcsgegdc.com:8088/proxy/application_1497349747602_0006/, appUser=ambari-qa
17/06/13 15:56:45 INFO distributedshell.Client: Got application report from ASM for, appId=6, clientToAMToken=null, appDiagnostics=, appMasterHost=dn3.tcsgegdc.com/3.209.124.208, appQueue=default, appMasterRpcPort=-1, appStartTime=1497349916566, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=http://nn.tcsgegdc.com:8088/proxy/application_1497349747602_0006/, appUser=ambari-qa
17/06/13 15:56:46 INFO distributedshell.Client: Got application report from ASM for, appId=6, clientToAMToken=null, appDiagnostics=Diagnostics., total=1, completed=1, allocated=1, failed=1, appMasterHost=dn3.tcsgegdc.com/3.209.124.208, appQueue=default, appMasterRpcPort=-1, appStartTime=1497349916566, yarnAppState=FINISHED, distributedFinalState=FAILED, appTrackingUrl=http://nn.tcsgegdc.com:8088/proxy/application_1497349747602_0006/, appUser=ambari-qa
17/06/13 15:56:46 INFO distributedshell.Client: Application did finished unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring loop
17/06/13 15:56:46 ERROR distributedshell.Client: Application failed to complete successfully
... View more
06-13-2017
10:21 AM
MySQL server issue we have handled now. But Spark2 issue still persists. For the time being we are shifting to Spark but need your help in resolving the spark2 issue.
... View more
06-13-2017
09:53 AM
Hi Guys, Looking for your help here. We are stuck with the cluster set up. Thanks and Regards, Rajdip
... View more
06-13-2017
08:51 AM
Sorry for wrong log. Spark2 client install error log as below. stderr: /var/lib/ambari-agent/data/errors-414.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_client.py", line 60, in <module>
SparkClient().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_client.py", line 36, in install
self.configure(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 117, in locking_configure
original_configure(obj, *args, **kw)
File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_client.py", line 42, in configure
setup_spark(env, 'client', upgrade_type=upgrade_type, action = 'config')
File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py", line 56, in setup_spark
mode=0644
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/properties_file.py", line 54, in action_create
mode = self.resource.mode
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 120, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/spark2-client/conf/spark-defaults.conf'] failed, parent directory /usr/hdp/current/spark2-client/conf doesn't exist
... View more
06-13-2017
08:49 AM
Also in another we are getting below error. MySQL server install error: stderr: /var/lib/ambari-agent/data/errors-450.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 64, in <module>
MysqlServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 33, in install
self.install_packages(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 605, in install_packages
retry_count=agent_stack_retry_count)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 54, in action_install
self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 51, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 86, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 98, in _call_with_retries
code, out = func(cmd, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-server' returned 1. Error: Nothing to do
... View more
06-13-2017
08:47 AM
Hi Guys, We are setting up a HDP-2.6.0.3 cluster on 4 nodes having CentOS6.5. While setting up through Ambari we are getting below error. Please note that we are using local repository for set up. Need your URGENT help. Spark2 client install issue: stderr: /var/lib/ambari-agent/data/errors-450.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 64, in <module>
MysqlServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 33, in install
self.install_packages(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 605, in install_packages
retry_count=agent_stack_retry_count)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 54, in action_install
self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 51, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 86, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 98, in _call_with_retries
code, out = func(cmd, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-server' returned 1. Error: Nothing to dostdout: /var/lib/ambari-agent/data/output-450.txt
2017-06-13 14:16:23,830 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6
2017-06-13 14:16:23,830 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2017-06-13 14:16:23,831 - Group['livy'] {}
2017-06-13 14:16:23,832 - Group['spark'] {}
2017-06-13 14:16:23,832 - Group['hadoop'] {}
2017-06-13 14:16:23,833 - Group['users'] {}
2017-06-13 14:16:23,833 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,833 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,834 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,834 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,835 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-06-13 14:16:23,835 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,836 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,836 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,837 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-06-13 14:16:23,837 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,837 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,838 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,838 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,839 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-13 14:16:23,839 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-06-13 14:16:23,840 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-06-13 14:16:23,846 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-06-13 14:16:23,846 - Group['hdfs'] {}
2017-06-13 14:16:23,846 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2017-06-13 14:16:23,847 - FS Type:
2017-06-13 14:16:23,847 - Directory['/etc/hadoop'] {'mode': 0755}
2017-06-13 14:16:23,865 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2017-06-13 14:16:23,865 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-06-13 14:16:23,878 - Initializing 2 repositories
2017-06-13 14:16:23,879 - Repository['HDP-2.6'] {'base_url': 'http://3.209.124.205/HDP/centos6', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None}
2017-06-13 14:16:23,886 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.6]\nname=HDP-2.6\nbaseurl=http://3.209.124.205/HDP/centos6\n\npath=/\nenabled=1\ngpgcheck=0'}
2017-06-13 14:16:23,887 - Repository['HDP-UTILS-1.1.0.21'] {'base_url': 'http://3.209.124.205/HDP-UTILS-1.1.0.21', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2017-06-13 14:16:23,891 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.21]\nname=HDP-UTILS-1.1.0.21\nbaseurl=http://3.209.124.205/HDP-UTILS-1.1.0.21\n\npath=/\nenabled=1\ngpgcheck=0'}
2017-06-13 14:16:23,892 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-06-13 14:16:23,936 - Skipping installation of existing package unzip
2017-06-13 14:16:23,936 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-06-13 14:16:23,946 - Skipping installation of existing package curl
2017-06-13 14:16:23,946 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-06-13 14:16:23,956 - Skipping installation of existing package hdp-select
2017-06-13 14:16:24,104 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-06-13 14:16:24,109 - call['ambari-python-wrap /usr/bin/hdp-select status hive-server2'] {'timeout': 20}
2017-06-13 14:16:24,126 - call returned (0, 'hive-server2 - 2.6.0.3-8')
2017-06-13 14:16:24,127 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6
2017-06-13 14:16:24,130 - File['/var/lib/ambari-agent/cred/lib/CredentialUtil.jar'] {'content': DownloadSource('http://nn.tcsgegdc.com:8080/resources/CredentialUtil.jar'), 'mode': 0755}
2017-06-13 14:16:24,131 - Not downloading the file from http://nn.tcsgegdc.com:8080/resources/CredentialUtil.jar, because /var/lib/ambari-agent/tmp/CredentialUtil.jar already exists
2017-06-13 14:16:24,132 - checked_call[('/usr/java/jdk1.8.0_131/bin/java', '-cp', '/var/lib/ambari-agent/cred/lib/*', 'org.apache.ambari.server.credentialapi.CredentialUtil', 'get', 'javax.jdo.option.ConnectionPassword', '-provider', 'jceks://file/var/lib/ambari-agent/cred/conf/hive/hive-site.jceks')] {}
2017-06-13 14:16:24,683 - checked_call returned (0, 'SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".\nSLF4J: Defaulting to no-operation (NOP) logger implementation\nSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.\nJun 13, 2017 2:16:24 PM org.apache.hadoop.util.NativeCodeLoader <clinit>\nWARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\nhive')
2017-06-13 14:16:24,688 - Package['mysql-server'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-06-13 14:16:24,739 - Installing package mysql-server ('/usr/bin/yum -d 0 -e 0 -y install mysql-server')
2017-06-13 14:16:24,992 - Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-server' returned 1. Error: Nothing to do
2017-06-13 14:16:24,993 - Failed to install package mysql-server. Executing '/usr/bin/yum clean metadata'
2017-06-13 14:16:25,093 - Retrying to install package mysql-server after 30 seconds
Command failed after 1 tries
... View more
Labels:
05-22-2017
10:43 AM
Hi, We are using HDP-2.5.3 and facing issues in sqoop incremental import. Detailed explanation as below: 1) Used sqoop to import initial data. 2) In source (Oracle), we have some changes like 1 data is inserted and 2 data got updated. 3) Using incremental we are trying to get the changes merged but its not happening. Screenshot for updated data: Sqoop command for incremental load. But while using this command data is not getting merged. Note that from above screen shot record witth value 8000 is the new record and other 2 are updated records. sqoop import -Dmapreduce.job.user.classpath.first=true --connect jdbc:oracle:thin:@//3.209.124.204:1521/eqlmpoc --fetch-size 1000 --username utility_test --password utility_test --table UTILITY_TEST.EMP_DETAILS --check-column HIREDATE --incremental lastmodified --last-value "2017-05-28 16:01:12" --target-dir hdfs://nn.tcsgegdc.com:8020/data/dlingestion/DEMO/FINAL/EMP_DETAILS --merge-key SSN -m 1 --class-name UTILITY_TEST.EMP_DETAILS
... View more
Labels:
- Labels:
-
Apache Sqoop
05-19-2017
06:48 PM
Since this is the first time we are doing so we using the workflow XML file directly while running the job. We added the oozie.use.system.libpath=true parameter the XML along with the other ones as in above comments. SO, do you want to add just this parameter and remove the other ? Also can you share a XML file which works on HDP-2.5.3 while connecting to beeline with auth over non-kerberized cluster? Please suggest.
... View more
05-19-2017
06:31 PM
We can try running these commands next week Monday. Should we run this command as the oozie user in the cluster? Also please note we are running the oozie job from individual developer passwords.
... View more
05-19-2017
06:29 PM
Hi @Kuldeep Kulkarni we tried that also but didn't work. Even we uninstalled the service and installed Oozie again. We also changed the oozie library permission to 755. All these steps resulted in same error. What we should do now? Using HDP 2.5.3 cluster. Is this an issue in this version? Please guide us.
... View more
05-19-2017
02:34 PM
Hi Guys, Can you please guide us? Not able to progress with it at all. Really looking for some help.
... View more
05-19-2017
10:00 AM
@Kuldeep Kulkarni we have deleted the oozie service and added the service again from Ambari. But now below command doesn't give the list of jars that are there in the oozie sharedlib oozie admin -oozie http://dn2.tcsgegdc.com:11000/oozie -shareliblist hive* > hive_shareliblist.txt 2>&1 oozie admin -oozie http://dn2.tcsgegdc.com:11000/oozie -shareliblist hive2* > hive2_shareliblist_OLD.txt 2>&1 We have tried lot of changes as has been suggested at the other HCC links, but nothing is working. Really need some help here. We think its an issue of HDP.
... View more
05-19-2017
08:16 AM
Hi Guys, help is really appreciated. Totally stuck with it for both Hive and Hive2.
... View more
05-19-2017
07:31 AM
Hi @Kuldeep Kulkarni, using Hive instead of Hive2 from oozie is also not working. When we are using Hive getting the below error: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found Really looking for some help here as we are totally stuck.
... View more
05-19-2017
06:46 AM
Hi @Kuldeep Kulkarni jar file as you mentioned is there in the path hdfs://nn.tcsgegdc.com:8020/user/oozie/share/lib/lib_20170419171320/hive2/oozie-sharelib-hive2-4.2.0.2.5.3.0-37.jar and have added the parameters as you mentioned in the workflow xml. Please note that we are not using job properties file yet. After the changes we are still facing the same issue of : java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found Changed xml : <workflow-app name='Raj' xmlns="uri:oozie:workflow:0.1">
<start to='hive2tab'/>
<action name='hive2tab'>
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<job-tracker>dn1.tcsgegdc.com:8050</job-tracker>
<name-node>hdfs://nn.tcsgegdc.com:8020</name-node>
<configuration>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
<property>
<name>oozie.libpath</name>
<value>hdfs://nn.tcsgegdc.com:8020/user/oozie/share/lib/lib_20170419171320/hive2</value>
</property>
</configuration>
<jdbc-url>jdbc:hive2://3.209.124.207:10000/demo</jdbc-url>
<password>tcs_ge_user</password>
<script>hdfs://nn.tcsgegdc.com:8020/user/tcs_ge_user/oozie_test/tab3.hql</script>
</hive2>
<ok to='end'/>
<error to='end'/>
</action>
<kill name='kill'>
<message>JOb_failed</message>
</kill>
<end name='end'/>
</workflow-app>
... View more
05-18-2017
02:42 PM
Really need your help to resolve the issue. Because we have planned to use Hive2 for our use case.
... View more
05-18-2017
01:45 PM
<workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1">
<start to='hive2tab'/>
<action name='hive2tab'>
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<job-tracker>aa.bb.com:8050</job-tracker>
<name-node>hdfs://cc.dd.com:8020</name-node>
<jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url>
<password>ZZZZ</password>
<script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script>
</hive2>
<ok to='end'/>
<error to='end'/>
</action>
<kill name='kill'>
<message>JOb_failed</message>
</kill>
<end name='end'/>
</workflow-app>
... View more
05-18-2017
01:41 PM
Hi Guys, We are trying to use Oozie for our ETL process by connecting to Hive2 but while working with it facing issues during the jobs. Need your help to resolve the issue. HDP-2.5.3 is deployed in our cluster and below the workflow.xml file: <workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1"> <start to='hive2tab'/> <action name='hive2tab'> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>aa.bb.com:8050</job-tracker> <name-node>hdfs://cc.dd.com:8020</name-node> <jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url> <password>ZZZZ</password> <script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script> </hive2> <ok to='end'/> <error to='end'/> </action> <kill name='kill'> <message>JOb_failed</message> </kill> <end name='end'/> </workflow-app> This is a simple job to execute a .hql script. However, when running this job we can see the below error in the Oozie log: 2017-05-18 17:25:32,510 WARN Hive2ActionExecutor:523 - SERVER[dn2.tcsgegdc.com] USER[tcs_ge_user] GROUP[-] TOKEN[] APP[Meg_Test1] JOB[0000004-170518130535586-oozie-oozi-W] ACTION[0000004-170518130535586-oozie-oozi-W@Hive1_Table] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2212)
... 9 more Please help us in resolving this issue as this is very important for us. Thanks and Regards, Rajdip
... View more
Labels:
- Labels:
-
Apache Oozie
05-15-2017
06:20 AM
Hi Guys, Can you please guide for this issue? Really looking for some guidance here.
... View more
05-14-2017
05:18 PM
Hello Guys, really looking for some guidance here on how to handle the scenario?
... View more