Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
debananda_sahoo
debananda_sahoo
New Contributor

Caused by: java.lang.IllegalArgumentException: Unr...

by New Contributor debananda_sahoo in Support Questions
‎08-28-2019 02:27 AM
‎08-28-2019 02:27 AM
I am getting this error while running the below code in CDH 6.2   import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.DataFrame import org.apache.spark.sql.Row import org.apache.spark.sql.Column import org.apache.spark.sql.streaming.StreamingQueryException import org.apache.spark.sql.types._ import java.io.PrintStream import org.apache.spark.SparkConf import org.apache.spark.SparkContext import java.io.File import org.apache.spark.sql.{Row, SaveMode, SparkSession} object test3 { def main(args: Array[String]){ val warehouseLocation = new File("spark-warehouse").getAbsolutePath val spark = SparkSession .builder() .appName("Spark Hive Example") .config("spark.sql.warehouse.dir", warehouseLocation) .enableHiveSupport() .getOrCreate() import spark.implicits._ import spark.sql sql("SELECT * FROM default.test").show() }} ... View more
Labels:

not getting any data in kafka consumer using spark...

by New Contributor debananda_sahoo in Support Questions
‎05-27-2019 06:54 AM
‎05-27-2019 06:54 AM
I have created a topic and ingest some sample data using producer . But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic >abc >xyz >mno spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2" scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load() scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination() Here no data is coming. ... View more

not getting any data in kafka consumer using spark...

by New Contributor debananda_sahoo in Support Questions
‎05-27-2019 06:53 AM
‎05-27-2019 06:53 AM
I have created a topic and ingest some sample data using producer . But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic >abc >xyz >mno spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2" scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load() scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination() Here no data is coming. ... View more

Re: How to Convert XML file into CSV file using NI...

by New Contributor debananda_sahoo in Support Questions
‎04-29-2019 10:17 AM
‎04-29-2019 10:17 AM
@Andrew Lim Is there anything possible for nested xml to multiple csvs ?Attached is the nested xml file ad.xml ... View more

Re: Is it possible to convert nested XML to csv/te...

by New Contributor debananda_sahoo in Support Questions
‎04-25-2019 09:42 AM
‎04-25-2019 09:42 AM
@Turker TUNALI Is there anything possible for this xml to csv conversion ? ... View more

Re: Is it possible to convert nested XML to csv/te...

by New Contributor debananda_sahoo in Support Questions
‎04-25-2019 09:41 AM
‎04-25-2019 09:41 AM
@Matt Burgess Is there anything possible for this xml to csv conversion ? ... View more

Re: How to Convert XML file into CSV file using NI...

by New Contributor debananda_sahoo in Support Questions
‎04-23-2019 11:20 AM
‎04-23-2019 11:20 AM
Hi Andrew , this works for single xml but I have nested xml. How to convert nested xml to multiple csvs ? ... View more

Re: flatten json using nifi

by New Contributor debananda_sahoo in Support Questions
‎04-22-2019 11:58 AM
‎04-22-2019 11:58 AM
could you please share the nifi template for this. I am not able to achieve this. ... View more

Re: How to Convert XML file into CSV file using NI...

by New Contributor debananda_sahoo in Support Questions
‎04-05-2019 09:13 AM
‎04-05-2019 09:13 AM
Hi Andrew, I am not able to convert xml to csv as suggested by you to do the inverse of this. Could you please provide the nifi xml file so that I can upload the template and check. ... View more

ERROR [KafkaServer id=1001] Fatal error during Kaf...

by New Contributor debananda_sahoo in Support Questions
‎04-02-2019 09:47 AM
‎04-02-2019 09:47 AM
After restarting kafka cluster one of the node is not starting up and throwing this error. ERROR [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.NoSuchMethodError: org.apache.kafka.common.metrics.Sensor.add(Lorg/apache/kafka/common/MetricName;Lorg/apache/kafka/common/metrics/MeasurableStat;)Z at kafka.server.ClientQuotaManager.<init>(ClientQuotaManager.scala:180) at kafka.server.QuotaFactory$.instantiate(QuotaFactory.scala:64) at kafka.server.KafkaServer.startup(KafkaServer.scala:231) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:75) at kafka.Kafka.main(Kafka.scala) [2019-04-02 12:24:05,939] INFO [KafkaServer id=1001] shutting down (kafka.server.KafkaServer) [2019-04-02 12:24:05,941] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient) [2019-04-02 12:24:05,944] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient) [2019-04-02 12:24:05,947] INFO [KafkaServer id=1001] shut down completed (kafka.server.KafkaServer) [2019-04-02 12:24:05,948] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable) ... View more
Labels:

Re: Change Data Capture (CDC) with Apache NiFi (Pa...

by New Contributor debananda_sahoo in Community Articles
‎03-25-2019 01:49 PM
‎03-25-2019 01:49 PM
We need to read sql server transnational logs for CDC instead of hitting DB using jdbc . Is that possible ? ... View more

java.lang.IllegalArgumentException: No running LLA...

by New Contributor debananda_sahoo in Support Questions
‎01-11-2019 01:50 PM
‎01-11-2019 01:50 PM
[root@hadoopmaster ~]# spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar scala> import com.hortonworks.hwc.HiveWarehouseSession import com.hortonworks.hwc.HiveWarehouseSession scala> import com.hortonworks.hwc.HiveWarehouseSession._ import com.hortonworks.hwc.HiveWarehouseSession._ scala> val hive = HiveWarehouseSession.session(spark).build() hive: com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl = com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl@5da2966 scala> hive.setDatabase("default") scala> val df = hive.executeQuery("select * from dd1") df: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [col: int] scala> df.show() 19/01/11 15:35:24 ERROR LlapBaseInputFormat: Closing connection due to error shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration ... View more

Re: Insert overwite query on LLAP is not working b...

by New Contributor debananda_sahoo in Support Questions
‎01-04-2019 11:12 AM
‎01-04-2019 11:12 AM
any one get solution for this ? ... View more

Re: How to resolve Unable to read HiveServer2 conf...

by New Contributor debananda_sahoo in Support Questions
‎12-21-2018 01:37 PM
‎12-21-2018 01:37 PM
@Sidharth Kumar How you resolved this ? Can you share the exact steps or command. In my case is showing like below. hive.server2.zookeeper.namespace=hiveserver2 hive.zookeeper.namespace=hive_zookeeper_namespace ... View more

Data is not loaded in hive using “Hive warehouse c...

by New Contributor debananda_sahoo in Support Questions
‎11-27-2018 02:56 PM
‎11-27-2018 02:56 PM
I used the below command to connect Hive Warehouse Connector. spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar Then I executed the below query to create a table >>> spark.sql('create table C_DAMAGE_join stored as orc as select tdc.* from T_DAMAGE_CODE CD LEFT JOIN C_DAMAGE TDC ON CD.ID = TDC.DAMAGE_CODE_ID') Now the issue is table created in hive with only metadata , without any data. ... View more

Data is not loaded in hive using “Hive warehouse c...

by New Contributor debananda_sahoo in Support Questions
‎11-27-2018 07:02 AM
‎11-27-2018 07:02 AM
I used the below command to connect Hive Warehouse Connector. spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar Then I executed the below query to create a table >>> spark.sql('create table C_DAMAGE_join stored as orc as select tdc.* from T_DAMAGE_CODE CD LEFT JOIN C_DAMAGE TDC ON CD.ID = TDC.DAMAGE_CODE_ID') Now the issue is table created in hive with only metadata , without any data. ... View more

WARN cli.LlapStatusServiceDriver: Watch mode enabl...

by New Contributor debananda_sahoo in Support Questions
‎11-14-2018 11:31 AM
‎11-14-2018 11:31 AM
After setting up llap I am not able to start HIVESERVER2 INTERACTIVE and its throwing below errors. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. LLAP status unknown -------------------------------------------------------------------------------- WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.") resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED. ... View more

Error while connecting Oracle DB using spark in HD...

by New Contributor debananda_sahoo in Support Questions
‎09-27-2018 01:36 PM
‎09-27-2018 01:36 PM
I am running the below code to load oracle data to hive but I am getting the error mentioned below. spark-submit --driver-class-path /app/ojdbc5.jar /app/raci.py Driver stacktrace: 18/09/27 16:30:47 INFO DAGScheduler: Job 0 failed: saveAsTable at NativeMethodAccessorImpl.java:0, took 5.097862 s 18/09/27 16:30:47 ERROR FileFormatWriter: Aborting job null. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, hadoopslave2, executor 1): java.lang.ClassNotFoundException: oracle.jdbc.OracleDriver ... View more
Labels:

Re: TypeError: jdbc() takes at least 3 arguments (...

by New Contributor debananda_sahoo in Support Questions
‎08-14-2018 01:32 PM
‎08-14-2018 01:32 PM
Its working now. The format should be "(SELECT * FROM T_DISTRICT_TYPE_test) as abc". Without alias its not working ... View more

Re: TypeError: jdbc() takes at least 3 arguments (...

by New Contributor debananda_sahoo in Support Questions
‎08-14-2018 12:40 PM
‎08-14-2018 12:40 PM
@Harald Berghoff I am not getting a clear syntax for this. The below code is working fine for table but not for a sql query. I want to load from select query with some where condition , not the complete table. If possible could you please modify the code to support sql query instead of tables. import os from pyspark import SparkConf,SparkContext from pyspark.sql import HiveContext from pyspark.sql import SparkSession from pyspark.sql import Row spark = (SparkSession .builder .appName("data_import") .config("spark.dynamicAllocation.enabled", "true") .config("spark.shuffle.service.enabled", "true") .config("spark.sql.parquet.writeLegacyFormat","true") .enableHiveSupport() .getOrCreate()) lst = list(["T_DISTRICT_TYPE_test"]); for tbl in lst: df = spark.read.jdbc("jdbc:sqlserver://10.24.25.25;database=CORE_13_2_TEST;username=core;password=password",tbl) df.write.format("orc").save("/tmp/orc_query_output_"+tbl) df.write.mode('append').format('orc').saveAsTable(tbl) ... View more

TypeError: jdbc() takes at least 3 arguments (2 gi...

by New Contributor debananda_sahoo in Support Questions
‎08-14-2018 10:50 AM
‎08-14-2018 10:50 AM
Hi, I am running below code to fetch data from sql server tables and loading it to hive tables. import os from pyspark import SparkConf,SparkContext from pyspark.sql import HiveContext from pyspark.sql import SparkSession from pyspark.sql import Row spark = (SparkSession .builder .appName("data_import") .config("spark.dynamicAllocation.enabled", "true") .config("spark.shuffle.service.enabled", "true") .config("spark.sql.parquet.writeLegacyFormat","true") .enableHiveSupport() .getOrCreate()) df = spark.read.jdbc("jdbc:sqlserver://10.24.25.25;database=CORE_13_2_TEST;username=core;password=password;table=(select * from T_DISTRICT_TYPE_test)") df.write.mode('append').format('orc').saveAsTable(test) But I am getting below error while running this. df = spark.read.jdbc("jdbc:sqlserver://10.24.25.25;database=CORE_13_2_TEST;username=core;password=password;table=(select * from T_DISTRICT_TYPE_test)") TypeError: jdbc() takes at least 3 arguments (2 given) ... View more
Labels:

Re: incremental update in hive using sqoop

by New Contributor debananda_sahoo in Support Questions
‎08-10-2018 07:15 AM
‎08-10-2018 07:15 AM
@Sandeep Nemuri This is also not working. Every time a new record is inserted instead of updating the existing record. It means the table in hive now has 2 records , both old and new with the same ID ... View more

incremental update in hive using sqoop

by New Contributor debananda_sahoo in Support Questions
‎08-09-2018 11:46 AM
‎08-09-2018 11:46 AM
How to do incremental update when I am loading data from SQL server to hive tables using sqoop without creating extra temporary tables ? Incremental insert is working using below command. --incremental append --check-column id --last-value 5 But update is not working using below --incremental lastmodified --check-column UPDATE_DATE --last-value '2018-07-19 16:14:38' ... View more

Re: AttributeError in Spark

by New Contributor debananda_sahoo in Support Questions
‎07-18-2018 11:59 AM
‎07-18-2018 11:59 AM
@Felix Albani There is still some issue. Tables were exist in hive but I am not able to access it. Its showing below error while I am doing a select * from table. <small>java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1531811351810_0064_1_00, diagnostics=[Task failed, taskId=task_1531811351810_0064_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/t_currency/part-00000-2feb31ba-70a4-40a0-a64f-e976b8dd587a-c000.snappy.parquet at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) <strong>Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/t_currency/part-00000-2feb31ba-70a4-40a0-a64f-e976b8dd587a-c000.snappy.parquet</strong> at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:196) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:135) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:101) at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149) at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80) at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:674) at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:633) at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:145) at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:405) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:124) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)</small> ... View more

Re: AttributeError in Spark 2.3

by New Contributor debananda_sahoo in Support Questions
‎07-17-2018 01:03 PM
‎07-17-2018 01:03 PM
Thanks Felix for your quick response. It worked. Thanks a lot. ... View more

Re: AttributeError in Spark

by New Contributor debananda_sahoo in Support Questions
‎07-17-2018 01:03 PM
‎07-17-2018 01:03 PM
Thanks Felix for your quick response. It worked. Thanks a lot. ... View more
Contact Me
Online
Offline
Last Visited
‎10-25-2019 09:06 AM
Public Statistics
Date Registered ‎04-09-2018 01:48 PM
Last Visited ‎10-25-2019 09:06 AM
Total Messages Posted 35