Member since
02-21-2016
3
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8824 | 09-13-2018 08:51 PM |
09-13-2018
08:51 PM
2 Kudos
Mahesh you nailed it! To get this to work I created the file "beeline-hs2-connection.xml" in ${HIVE_CONF_DIR}, i.e. /etc/hive/conf, and I included the credentials there. See https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients
... View more
08-29-2018
04:27 PM
I believe I have the same exact problem. I get more details if I set debug logging prior to running sqoop.
export HADOOP_ROOT_LOGGER=DEBUG,console
Note that my cluster in not Kerberized. Here are the debug messages I get while it appears to be hung. But the job still never finishes. It keeps repeating the messages at about 5 minute intervals.
18/08/29 09:13:23 INFO hive.HiveImport: Connecting to jdbc:hive2://node1:2181,node1:2181,node1:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 18/08/29 09:13:24 DEBUG impl.LeaseRenewer: Lease renewer daemon for [] with renew id 1 executed
18/08/29 09:13:27 DEBUG impl.LeaseRenewer: Lease renewer daemon for [] with renew id 1 expired
18/08/29 09:13:27 DEBUG impl.LeaseRenewer: Lease renewer daemon for [] with renew id 1 exited
18/08/29 09:13:47 DEBUG ipc.Client: IPC Client (505973878) connection to node2/172.21.0.11:41588 from agileanalytics: closed
18/08/29 09:13:47 DEBUG ipc.Client: IPC Client (505973878) connection to node2/172.21.0.11:41588 from agileanalytics: stopped, remaining connections 1
18/08/29 09:13:48 DEBUG ipc.Client: IPC Client (505973878) connection to node2/172.21.0.10:8020 from agileanalytics: closed
18/08/29 09:13:48 DEBUG ipc.Client: IPC Client (505973878) connection to node2/172.21.0.10:8020 from agileanalytics: stopped, remaining connections 0
18/08/29 09:18:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Initialize connection to node node3:6667 (id: 1002 rack: null) for sending metadata request
18/08/29 09:18:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Initiating connection to node node3:6667 (id: 1002 rack: null)
18/08/29 09:18:18 DEBUG metrics.Metrics: Added sensor with name node-1002.bytes-sent
18/08/29 09:18:18 DEBUG metrics.Metrics: Added sensor with name node-1002.bytes-received
18/08/29 09:18:18 DEBUG metrics.Metrics: Added sensor with name node-1002.latency
18/08/29 09:18:18 DEBUG network.Selector: [Producer clientId=producer-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1002
18/08/29 09:18:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Completed connection to node 1002. Fetching API versions.
18/08/29 09:18:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Initiating API versions fetch from node 1002.
18/08/29 09:18:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Sending metadata request (type=MetadataRequest, topics=ATLAS_HOOK) to node node3:6667 (id: 1001 rack: null)
18/08/29 09:18:18 DEBUG clients.Metadata: Removing unused topic ATLAS_HOOK from the metadata list, expiryMs 1535559498233 now 1535559498257
18/08/29 09:18:18 DEBUG clients.Metadata: Updated cluster metadata version 3 to Cluster(id = 9eDSa9DhRMOTMZguD63M-w, nodes = [node3:6667 (id: 1002 rack: null), node3:6667 (id: 1003 rack: null), node3:6667 (id: 1001 rack: null)], partitions = [Partition(topic = ATLAS_HOOK, partition = 0, leader = 1001, replicas = [1001], isr = [1001], offlineReplicas = [])]) 18/08/29 09:18:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Recorded API versions for node 1002: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 6 [usable: 6], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 [usable: 0], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0])
18/08/29 09:22:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Node -1 disconnected.
18/08/29 09:22:18 DEBUG clients.NetworkClient: [Producer clientId=producer-1] Sending metadata request (type=MetadataRequest, topics=) to node node3:6667 (id: 1002 rack: null)
18/08/29 09:22:18 DEBUG clients.Metadata: Updated cluster metadata version 4 to Cluster(id = 9eDSa9DhRMOTMZguD63M-w, nodes = [node3:6667 (id: 1003 rack: null), node3:6667 (id: 1002 rack: null), node3:6667 (id: 1001 rack: null)], partitions = [])
... View more