Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive error

Hive error

Explorer

How to move the files from hdfs to hive dynamically. please help me

5 REPLIES 5

Re: Hive error

Rising Star

Can you please explain more on your use-case. What you mean by dynamic?

If you just want to move the hdfs files to hive table, you may use move(-mv) command (hdfs dfs -mv)

You may also create external table on top of existing hdfs files too, no need to move.

Re: Hive error

Explorer

Thanks @ nyadav

I'm Receiving set of csv files from ftp to HDFS and i need load the files data into hive tables

- if csv files contains different columns. how can i create dynamic columns in hive tables and load the data

Thanks in advance

Re: Hive error

Rising Star

@Saleema,

You can move files to some hdfs dir where hive has write permission, for example, /user/[usrname] and then create external table via hive CLI like the below:

create external table (i int)

row formate delimited

fields terminated by ','

stored as textfile

location '/user/[usrname]';

More details: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/moving_data_from...

Highlighted

Re: Hive error

Explorer

thanks for u r reply, i did this one but my problem is how to move set of .csv files in hdfs folder with different columns to hive table dynamically it is possible.

Re: Hive error

Explorer

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStore Client at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:494) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1533) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(Retry ingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:104) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:3000) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3019) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:475) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1531) ... 14 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:187) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:426) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaS toreClient.java:236) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>( SessionHiveMetaStoreClient.java:74) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1531) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(Retry ingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:104) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:3000) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3019) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:475) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.net.ConnectException: Connection refused: connect at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja va:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket Impl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java :188) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.thrift.transport.TSocket.open(TSocket.java:182) ... 22 more ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:472) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaS toreClient.java:236) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>( SessionHiveMetaStoreClient.java:74) ... 19 more C:\Windows\System32>

Don't have an account?
Coming from Hortonworks? Activate your account here