Member since
04-08-2014
16
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2476 | 11-07-2014 01:06 AM |
02-11-2017
01:00 PM
1 Kudo
Hi. I'm trying to connect to HBase from Spark using this connector https://github.com/hortonworks-spark/shc This is my code: def catalog = s"""{
|"table":{"namespace":"default", "name":"terminals"},
|"rowkey":"key",
|"columns":{
|"col0":{"cf":"rowkey", "col":"key", "type":"string"},
|"col1":{"cf":"tinfo", "col":"status", "type":"int"},
|"col2":{"cf":"tinfo", "col":"latitude", "type":"double"},
|"col2":{"cf":"tinfo", "col":"longitude", "type":"double"}
|}
|}""".stripMargin
def withCatalog(cat: String): DataFrame = {
sqlContext
.read
.options(Map(HBaseTableCatalog.tableCatalog->cat))
.format("org.apache.spark.sql.execution.datasources.hbase")
.load()
}
val df = withCatalog(catalog)
df.show()
val dfFilter = df.filter($"col0".isin("1212121"))
parsed.join(dfFilter, parsed("terminal_id") === dfFilter("col0")).show() but when I try to execute with spark-submit: spark-submit --class com.location.userTransactionMain --master local[*] --files /etc/hbase/conf/hbase-site.xml userTransactionAppScala-assembly-1.0.jar It returns an error:
Exception in thread "main" java.lang.NullPointerException: Please define 'tableCoder' in your catalog. If there is an Avro records/schema in your catalog, please explicitly define 'coder' in its corresponding column.
at org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:223)
at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:77)
at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) Can someone help me? Thanks!!!!
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Spark
12-02-2014
03:09 AM
Hi All I have the same problem. I've seen the log and I don't see something. CDH 5.3.1 2014-12-02 12:03:04,837 INFO org.apache.solr.servlet.ProxyUserFilter: Loading proxyuser settings [solr.security.proxyuser.hue.hosts]=[*]
2014-12-02 12:03:04,837 INFO org.apache.solr.servlet.ProxyUserFilter: Loading proxyuser settings [solr.security.proxyuser.hue.groups]=[*]
2014-12-02 12:03:04,953 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-12-02 12:03:04,981 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret' configuration not set, using a random value as secret
2014-12-02 12:03:05,028 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http client, config:
2014-12-02 12:03:05,041 INFO org.apache.solr.servlet.SolrDispatchFilter: SolrDispatchFilter.init()
2014-12-02 12:03:05,084 INFO org.apache.solr.core.SolrResourceLoader: No /solr/home in JNDI
2014-12-02 12:03:05,084 INFO org.apache.solr.core.SolrResourceLoader: using system property solr.solr.home: /var/lib/solr
2014-12-02 12:03:05,084 INFO org.apache.solr.core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/lib/solr/'
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.1.3--1, built on 09/17/2014 02:09 GMT
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hd02
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_55
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-oracle-cloudera/jre
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/bigtop-tomcat/bin/bootstrap.jar
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/lib/solr/
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.2.0-4-amd64
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=solr
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/run/solr
2014-12-02 12:03:05,308 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/run/cloudera-scm-agent/process/984-solr-SOLR_SERVER
2014-12-02 12:03:05,309 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hd01:2181/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6c30aec7
2014-12-02 12:03:05,326 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2014-12-02 12:03:05,329 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hd01/192.168.20.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-02 12:03:05,334 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hd01/192.168.20.1:2181, initiating session
2014-12-02 12:03:05,345 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hd01/192.168.20.1:2181, sessionid = 0x14a0a9b2d300066, negotiated timeout = 30000
2014-12-02 12:03:05,348 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6c30aec7 name:ZooKeeperConnection Watcher:hd01:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2014-12-02 12:03:05,348 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2014-12-02 12:03:05,353 INFO org.apache.solr.core.ConfigSolr: Loading solr.xml from ZooKeeper
2014-12-02 12:03:05,459 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr
2014-12-02 12:03:05,459 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment
2014-12-02 12:03:05,459 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps
2014-12-02 12:03:05,460 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr
2014-12-02 12:03:05,460 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/js
2014-12-02 12:03:05,460 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/js/scripts
2014-12-02 12:03:05,461 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/js/lib
2014-12-02 12:03:05,461 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF
2014-12-02 12:03:05,461 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib
2014-12-02 12:03:05,464 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/css
2014-12-02 12:03:05,464 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/css/styles
2014-12-02 12:03:05,465 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/img
2014-12-02 12:03:05,465 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/img/ico
2014-12-02 12:03:05,468 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/img/filetypes
2014-12-02 12:03:05,471 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/META-INF
2014-12-02 12:03:05,471 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/solr/tpl
2014-12-02 12:03:05,472 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/webapps/ROOT
2014-12-02 12:03:05,472 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/work
2014-12-02 12:03:05,472 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/work/Catalina
2014-12-02 12:03:05,472 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/work/Catalina/localhost
2014-12-02 12:03:05,472 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/work/Catalina/localhost/solr
2014-12-02 12:03:05,472 INFO org.apache.solr.core.SolrCoreDiscoverer: Looking for cores in /var/lib/solr/tomcat-deployment/conf
2014-12-02 12:03:05,480 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14a0a9b2d300066 closed
2014-12-02 12:03:05,480 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2014-12-02 12:03:05,487 INFO org.apache.solr.core.CoreContainer: New CoreContainer 1045865347
2014-12-02 12:03:05,487 INFO org.apache.solr.core.CoreContainer: Loading cores into CoreContainer [instanceDir=/var/lib/solr/]
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting socketTimeout to: 0
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting urlScheme to: http://
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting connTimeout to: 0
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maxConnectionsPerHost to: 20
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting corePoolSize to: 0
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maximumPoolSize to: 2147483647
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maxThreadIdleTime to: 5
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting sizeOfQueue to: -1
2014-12-02 12:03:05,508 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting fairnessPolicy to: false
2014-12-02 12:03:05,509 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
2014-12-02 12:03:05,654 INFO org.apache.solr.logging.LogWatcher: Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2014-12-02 12:03:05,655 INFO org.apache.solr.core.ZkContainer: Zookeeper client=hd01:2181/solr
2014-12-02 12:03:05,664 INFO org.apache.solr.cloud.ZkController: zkHost includes chroot
2014-12-02 12:03:05,664 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hd01:2181 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1b878da9
2014-12-02 12:03:05,665 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2014-12-02 12:03:05,666 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hd01/192.168.20.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-02 12:03:05,666 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hd01/192.168.20.1:2181, initiating session
2014-12-02 12:03:05,677 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hd01/192.168.20.1:2181, sessionid = 0x14a0a9b2d30006d, negotiated timeout = 60000
2014-12-02 12:03:05,678 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@1b878da9 name:ZooKeeperConnection Watcher:hd01:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2014-12-02 12:03:05,678 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2014-12-02 12:03:05,689 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14a0a9b2d30006d closed
2014-12-02 12:03:05,689 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2014-12-02 12:03:05,690 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
2014-12-02 12:03:05,691 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hd01:2181/solr sessionTimeout=15000 watcher=org.apache.solr.common.cloud.ConnectionManager@2c1a4849
2014-12-02 12:03:05,692 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2014-12-02 12:03:05,692 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hd01/192.168.20.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-02 12:03:05,692 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hd01/192.168.20.1:2181, initiating session
2014-12-02 12:03:05,700 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hd01/192.168.20.1:2181, sessionid = 0x14a0a9b2d30006f, negotiated timeout = 15000
2014-12-02 12:03:05,700 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@2c1a4849 name:ZooKeeperConnection Watcher:hd01:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2014-12-02 12:03:05,700 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2014-12-02 12:03:05,736 INFO org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper...
2014-12-02 12:03:06,694 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (1)
2014-12-02 12:03:06,754 INFO org.apache.solr.cloud.ZkController: Register node as live in ZooKeeper:/live_nodes/hd02:8983_solr
2014-12-02 12:03:06,760 INFO org.apache.solr.common.cloud.SolrZkClient: makePath: /live_nodes/hd02:8983_solr
2014-12-02 12:03:06,771 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (3)
2014-12-02 12:03:06,778 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (4)
2014-12-02 12:03:06,824 INFO org.apache.solr.servlet.SolrDispatchFilter: user.dir=/run/cloudera-scm-agent/process/984-solr-SOLR_SERVER
2014-12-02 12:03:06,824 INFO org.apache.solr.servlet.SolrDispatchFilter: SolrDispatchFilter.init() done
2014-12-02 12:03:25,745 INFO org.apache.solr.servlet.SolrDispatchFilter: [admin] webapp=null path=/admin/cores params={action=STATUS&wt=json} status=0 QTime=51
... View more
11-17-2014
06:21 AM
Hi!!! Is it possible to execute UDAF java function in Impala? Thanks!!! Regards Miguel Angel.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
11-07-2014
01:06 AM
Hi. I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!
... View more
11-06-2014
02:09 PM
Hi Is it possible write a buffer into HDFS from UDF Impala (C++ or Java)? So I'd like simulate UDTF in Impala. First call to UDF to write output in HDFS. Later, another query reads these data from HDFS. If it is possible, it will be with libhdfs. How can I install libhdfs-dev in debian wheezy? Thanks!!! Regards.
... View more
Labels:
- Labels:
-
Apache Impala
-
HDFS
04-09-2014
08:15 AM
Yes. Use Impala or Hive to convert to Parquet stream from Flume is a good option, although it would be nice to have it natively. Thanks!!!! Miguel Angel.
... View more
04-08-2014
06:39 AM
Hi. Is it possible configure Apache Flume to save my logs in HDFS with Parquet? Thanks very much!!!! Miguel Angel.
... View more
Labels:
- Labels:
-
Apache Flume
-
HDFS