Member since
06-09-2016
185
Posts
22
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1151 | 04-21-2017 07:57 AM | |
764 | 04-18-2017 07:07 AM | |
1609 | 02-27-2017 05:41 AM | |
440 | 12-09-2016 11:05 AM | |
670 | 11-24-2016 11:20 AM |
08-10-2017
07:15 AM
Hi All, I am running the hdf sandbox and trying to do a sample - couple of challenges 1. how to see the logs - when I ssh into the docker container by ssh root@172.17.0.1 - I don't see any /usr/hdf or /var/log/registry directories 2. while adding a avro schema - gets error ' Given schema is invalid' - attached screenshot Thanks, Avijeet
... View more
Labels:
07-24-2017
08:28 AM
Hi All, is there a demo app for HDF3.0 with SAM pipeline, that can be imported. Thanks, Avijeet
... View more
07-21-2017
06:21 AM
Thanks @Jay SenSharma I will read more about maintenance mode. I learnt about what HDF includes, was wondering why in sandbox the other services are bundled..just to make demos work?
... View more
07-21-2017
05:48 AM
Hi All, I am looking at the HDF3.0 sandbox - couple of observations 1. It included HDFS, YARN, MR2, HBASE etc. - I thought these are not part of HDF so can HDF be all services except HIVE, Spark 2. In Ambari I see a lot of services with the briefcase with a + symbol - what exactly this symbol means - I started a few services still it shows the same symbol instead of green tick Thanks, Avijeet
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)
05-31-2017
05:10 AM
Hi All, I set up Kylin2.0 on hdp2.5 - built the sample sales cube. I wanted to see the hbase table, which table does kylin stores data? hbase(main):005:0> list_namespace_tables 'default' TABLE ATLAS_ENTITY_AUDIT_EVENTS KYLIN_ANX1S4IYMZ atlas_titan kylin_metadata kylin_metadata_acl kylin_metadata_user 6 row(s) in 0.0190 seconds Thanks, Avijeet
... View more
- Tags:
- Hadoop Core
- kylin
Labels:
- Labels:
-
Apache HBase
05-12-2017
05:51 AM
Hi All, I tried HIVE insert/update/delete operations, I see that it launches a tez job and bit slow. I have a couple of questions - 1. when we use hive streaming processors (in nifi, streamsets) does it insert record by record? I believe no. just to confirm. 2. it seems bit risky to design an acid hive table, is there an approach where it can be used (such as IOT ) safely without having locking/concurrency issues etc. Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Hive
05-11-2017
06:26 AM
@Vaisakh Sundaram pls find attached, I had to hide the host names, but hive2 url is correct, I checked with beeline.
... View more
05-10-2017
02:43 PM
@Scott Shaw yeah it simply says cannot connect - I tried localhost, 127.0.01, hostname...same error.
... View more
05-10-2017
02:34 PM
@ScottShaw no luck, is using hive user OK, or setting up some other user is better?
... View more
05-10-2017
01:45 PM
Hi @Scott Shaw, hive server 2 is running file, i can reach by beeline, hive etc. This is happening in 2 hdp2.5 clusters, same issue.
... View more
05-10-2017
11:34 AM
Hi All, While enabling ambari hive view (unsecured hadoop cluster) - as hive user - getting error Service 'hiveserver' check failed: Cannot connect to hive I have disabled 'Run as end user instead of Hive user' on HIVE Config and proxy user hive.users and hosts are set to * pls suggest if there is some way to fix it. Service 'hiveserver' check failed:
java.lang.Exception: Cannot connect to hive
java.lang.Exception: Cannot connect to hive
at org.apache.ambari.view.hive2.actor.message.job.ExecutionFailed.<init>(ExecutionFailed.java:28)
at org.apache.ambari.view.hive2.actor.JdbcConnector.notifyConnectFailure(JdbcConnector.java:385)
at org.apache.ambari.view.hive2.actor.JdbcConnector.connect(JdbcConnector.java:422)
at org.apache.ambari.view.hive2.actor.JdbcConnector.handleNonLifecycleMessage(JdbcConnector.java:179)
at org.apache.ambari.view.hive2.actor.JdbcConnector.handleMessage(JdbcConnector.java:171)
at org.apache.ambari.view.hive2.actor.HiveActor.onReceive(HiveActor.java:38)
at akka.actor.UntypedActor$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487) Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
05-04-2017
04:59 AM
Thanks @Alejandro Fernandez It will be great if you could share what technology/language it uses underline, if any. also custom ambari-services. How are those built.
... View more
04-25-2017
04:42 AM
Thanks @Matt Clarke - so if I pick one large file and write to hdfs - it will create 1 flowfile in content repository and will keep it for some duration until we clean it, same with one SQL read writing into nifi.
... View more
04-21-2017
07:57 AM
I deleted this directory under /stormdist - and the error went away
... View more
04-18-2017
07:07 AM
This issue got resolved by using maven-shade-plugin as mentioned here http://storm.apache.org/releases/0.10.0/storm-hdfs.html
... View more
04-17-2017
10:02 AM
HI All, while trying the twitter example on storm - getting this error I am using 1.0.1 on hdp2.5 I compiled the classes with 1.0.1 for storm-core and storm-hdfs also provided exclusion for hadoop-hdfs and hadoop-client in pom.xml 2017-04-17 09:52:33.863 STDERR [INFO] WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-04-17 09:52:34.408 o.a.s.h.b.HdfsBolt [INFO] Preparing HDFS Bolt...
2017-04-17 09:52:34.586 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Error preparing HdfsBolt: No FileSystem for scheme: hdfs
at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:123) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] pls let me know if any other information required. Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Storm
04-17-2017
06:24 AM
Hi, I tried to run the Wordcount example storm-starter-0.0.1-storm-0.9.0.1.jar on storm in hdp2.5 The supervisor keeps going down and doesn't run giving below errro java.io.FileNotFoundException: File '/datadrive/hadoop/storm/supervisor/stormdist/WordCount-2-1491988512/stormconf.ser' does not exist
at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292) I killed the topology still, same error persisits. pls suggest. Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Storm
04-13-2017
09:25 AM
Hi All, Hadoop uses kerberos for authentication. What would be a best way to implement SSO authentication for big data applications such as web-apps, BI tools etc. which talk to hadoop. One uniform solution across a diverse set of products and apps. Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Kerberos
04-13-2017
08:14 AM
1 Kudo
What error it throws? Thanks.
... View more
04-13-2017
08:03 AM
1 Kudo
Hi All, what would be the side effects of doing batch ingestion through nifi. Lets say large files copies, if we ingest through nifi, how would it behave? same with large db copies. Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache NiFi
04-12-2017
09:34 AM
Hi All, The communication between hive to atlas through Kafka - is it transactional in nature. If Atlas update fails, does it affect the hive operation in any way? Does all messages remain in kafka untill processed or discrded? Thanks, Avijeet
... View more
- Tags:
- Atlas
- Data Processing
Labels:
- Labels:
-
Apache Atlas
04-12-2017
09:30 AM
Hi All, by default hive-atlas integration is enabled when we set up HDP2.5. And this integration needs kafka, hbase etc. as a side effect Is there a way to enable / disable this integration easily. Thanks, Avijeet
... View more
- Tags:
- Data Processing
- Hive
Labels:
- Labels:
-
Apache Hive
04-12-2017
09:08 AM
Hi All, With all the DevOps tools available (Docker, swram, kubernetes, Ansible, Puppet, Chef etc.) around configuration management, cluster management, container clustering etc. I was wondering where does Amabri fits in, how is it developed? What about Ambari blue prints? Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Ambari
04-05-2017
05:44 AM
I had the same issue, increasing the heap space for HDFS helped. Also apparently a reboot was required to disable selinux - as the command sestatus was showing enabled.
... View more
03-30-2017
10:40 AM
Hi @Tom McCuch Great article, I have one question - when we say "ARCHIVE, which has high storage density (petabyte of storage) but little compute power" Is its a different kind of HDD - can you please give examples how it is different from the regular HDD. Thanks, Avijeet
... View more
03-09-2017
11:18 AM
1 Kudo
Hi All, I am able to resolve this issue now by changing the sequence of transformation - however the tweetes table is not getting created. twt.foreachRDD(rdd => rdd.map(status => Tweet(status.getCreatedAt().getTime()/1000, status.getText(), status.getUser().getScreenName())).toDF().registerTempTable("tweets")) select * from tweets limit 10 Table not found: tweets; line 1 pos 14
set zeppelin.spark.sql.stacktrace = true to see full stacktrace Please suggest. Thanks, Avijeet
... View more
03-09-2017
05:34 AM
Hi All, The zeppelin architecture is clear based on this link https://community.hortonworks.com/content/kbentry/70658/how-to-diagnose-zeppelin.html however is there more information detailing the dependency management part, maven repositores, steps to maange this correctly. Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Zeppelin
03-08-2017
08:32 AM
Hi, It worked after I added imports again in the script. Thanks, Avijeet
... View more
03-08-2017
08:29 AM
Hi All, I am running the twitter zeppelin notepad - getting the below error I am using local mode for spark - As spark streaming doesn't work on YARN - please suggest steps to start a distributed spark master if local will not work. val conf = new SparkConf().setAppName("avi_twitter")setMaster("local") org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122) Thanks, Avijeet
... View more
Labels:
- Labels:
-
Apache Spark