About debananda_sahoo

debananda_sahoo · ‎08-28-2019

I am getting this error while running the below code in CDH 6.2 import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.DataFrame import org.apache.spark.sql.Row import org.apache.spark.sql.Column import org.apache.spark.sql.streaming.StreamingQueryException import org.apache.spark.sql.types._ import java.io.PrintStream import org.apache.spark.SparkConf import org.apache.spark.SparkContext import java.io.File import org.apache.spark.sql.{Row, SaveMode, SparkSession} object test3 { def main(args: Array[String]){ val warehouseLocation = new File("spark-warehouse").getAbsolutePath val spark = SparkSession .builder() .appName("Spark Hive Example") .config("spark.sql.warehouse.dir", warehouseLocation) .enableHiveSupport() .getOrCreate() import spark.implicits._ import spark.sql sql("SELECT * FROM default.test").show() }}

debananda_sahoo · ‎05-27-2019

I have created a topic and ingest some sample data using producer . But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic >abc >xyz >mno spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2" scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load() scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination() Here no data is coming.

debananda_sahoo · ‎05-27-2019

I have created a topic and ingest some sample data using producer . But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic >abc >xyz >mno spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2" scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load() scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination() Here no data is coming.

debananda_sahoo · ‎04-29-2019

@Andrew Lim Is there anything possible for nested xml to multiple csvs ?Attached is the nested xml file ad.xml

debananda_sahoo · ‎04-25-2019

@Turker TUNALI Is there anything possible for this xml to csv conversion ?

debananda_sahoo · ‎04-25-2019

@Matt Burgess Is there anything possible for this xml to csv conversion ?

debananda_sahoo · ‎04-23-2019

Hi Andrew , this works for single xml but I have nested xml. How to convert nested xml to multiple csvs ?

debananda_sahoo · ‎04-23-2019

debananda_sahoo · ‎04-23-2019

debananda_sahoo · ‎04-22-2019

could you please share the nifi template for this. I am not able to achieve this.

debananda_sahoo · ‎04-05-2019

Hi Andrew, I am not able to convert xml to csv as suggested by you to do the inverse of this. Could you please provide the nifi xml file so that I can upload the template and check.

debananda_sahoo · ‎04-02-2019

After restarting kafka cluster one of the node is not starting up and throwing this error. ERROR [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.NoSuchMethodError: org.apache.kafka.common.metrics.Sensor.add(Lorg/apache/kafka/common/MetricName;Lorg/apache/kafka/common/metrics/MeasurableStat;)Z at kafka.server.ClientQuotaManager.<init>(ClientQuotaManager.scala:180) at kafka.server.QuotaFactory$.instantiate(QuotaFactory.scala:64) at kafka.server.KafkaServer.startup(KafkaServer.scala:231) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:75) at kafka.Kafka.main(Kafka.scala) [2019-04-02 12:24:05,939] INFO [KafkaServer id=1001] shutting down (kafka.server.KafkaServer) [2019-04-02 12:24:05,941] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient) [2019-04-02 12:24:05,944] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient) [2019-04-02 12:24:05,947] INFO [KafkaServer id=1001] shut down completed (kafka.server.KafkaServer) [2019-04-02 12:24:05,948] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)

debananda_sahoo · ‎03-25-2019

We need to read sql server transnational logs for CDC instead of hitting DB using jdbc . Is that possible ?

debananda_sahoo · ‎03-25-2019

debananda_sahoo · ‎03-25-2019

debananda_sahoo · ‎01-11-2019

[root@hadoopmaster ~]# spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar scala> import com.hortonworks.hwc.HiveWarehouseSession import com.hortonworks.hwc.HiveWarehouseSession scala> import com.hortonworks.hwc.HiveWarehouseSession._ import com.hortonworks.hwc.HiveWarehouseSession._ scala> val hive = HiveWarehouseSession.session(spark).build() hive: com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl = com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl@5da2966 scala> hive.setDatabase("default") scala> val df = hive.executeQuery("select * from dd1") df: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [col: int] scala> df.show() 19/01/11 15:35:24 ERROR LlapBaseInputFormat: Closing connection due to error shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration

debananda_sahoo · ‎01-04-2019

any one get solution for this ?

debananda_sahoo · ‎12-21-2018

@Sidharth Kumar How you resolved this ? Can you share the exact steps or command. In my case is showing like below. hive.server2.zookeeper.namespace=hiveserver2 hive.zookeeper.namespace=hive_zookeeper_namespace

debananda_sahoo · ‎11-27-2018

I used the below command to connect Hive Warehouse Connector. spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar Then I executed the below query to create a table >>> spark.sql('create table C_DAMAGE_join stored as orc as select tdc.* from T_DAMAGE_CODE CD LEFT JOIN C_DAMAGE TDC ON CD.ID = TDC.DAMAGE_CODE_ID') Now the issue is table created in hive with only metadata , without any data.

debananda_sahoo · ‎11-27-2018

I used the below command to connect Hive Warehouse Connector. spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar Then I executed the below query to create a table >>> spark.sql('create table C_DAMAGE_join stored as orc as select tdc.* from T_DAMAGE_CODE CD LEFT JOIN C_DAMAGE TDC ON CD.ID = TDC.DAMAGE_CODE_ID') Now the issue is table created in hive with only metadata , without any data.

debananda_sahoo · ‎11-14-2018

After setting up llap I am not able to start HIVESERVER2 INTERACTIVE and its throwing below errors. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. LLAP status unknown -------------------------------------------------------------------------------- WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.") resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.

debananda_sahoo · ‎09-27-2018

I am running the below code to load oracle data to hive but I am getting the error mentioned below. spark-submit --driver-class-path /app/ojdbc5.jar /app/raci.py Driver stacktrace: 18/09/27 16:30:47 INFO DAGScheduler: Job 0 failed: saveAsTable at NativeMethodAccessorImpl.java:0, took 5.097862 s 18/09/27 16:30:47 ERROR FileFormatWriter: Aborting job null. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, hadoopslave2, executor 1): java.lang.ClassNotFoundException: oracle.jdbc.OracleDriver

debananda_sahoo · ‎08-14-2018

Its working now. The format should be "(SELECT * FROM T_DISTRICT_TYPE_test) as abc". Without alias its not working

debananda_sahoo · ‎08-14-2018

@Harald Berghoff I am not getting a clear syntax for this. The below code is working fine for table but not for a sql query. I want to load from select query with some where condition , not the complete table. If possible could you please modify the code to support sql query instead of tables. import os from pyspark import SparkConf,SparkContext from pyspark.sql import HiveContext from pyspark.sql import SparkSession from pyspark.sql import Row spark = (SparkSession .builder .appName("data_import") .config("spark.dynamicAllocation.enabled", "true") .config("spark.shuffle.service.enabled", "true") .config("spark.sql.parquet.writeLegacyFormat","true") .enableHiveSupport() .getOrCreate()) lst = list(["T_DISTRICT_TYPE_test"]); for tbl in lst: df = spark.read.jdbc("jdbc:sqlserver://10.24.25.25;database=CORE_13_2_TEST;username=core;password=password",tbl) df.write.format("orc").save("/tmp/orc_query_output_"+tbl) df.write.mode('append').format('orc').saveAsTable(tbl)

debananda_sahoo · ‎08-14-2018

Hi, I am running below code to fetch data from sql server tables and loading it to hive tables. import os from pyspark import SparkConf,SparkContext from pyspark.sql import HiveContext from pyspark.sql import SparkSession from pyspark.sql import Row spark = (SparkSession .builder .appName("data_import") .config("spark.dynamicAllocation.enabled", "true") .config("spark.shuffle.service.enabled", "true") .config("spark.sql.parquet.writeLegacyFormat","true") .enableHiveSupport() .getOrCreate()) df = spark.read.jdbc("jdbc:sqlserver://10.24.25.25;database=CORE_13_2_TEST;username=core;password=password;table=(select * from T_DISTRICT_TYPE_test)") df.write.mode('append').format('orc').saveAsTable(test) But I am getting below error while running this. df = spark.read.jdbc("jdbc:sqlserver://10.24.25.25;database=CORE_13_2_TEST;username=core;password=password;table=(select * from T_DISTRICT_TYPE_test)") TypeError: jdbc() takes at least 3 arguments (2 given)

debananda_sahoo · ‎08-10-2018

@Sandeep Nemuri This is also not working. Every time a new record is inserted instead of updating the existing record. It means the table in hive now has 2 records , both old and new with the same ID

debananda_sahoo · ‎08-09-2018

How to do incremental update when I am loading data from SQL server to hive tables using sqoop without creating extra temporary tables ? Incremental insert is working using below command. --incremental append --check-column id --last-value 5 But update is not working using below --incremental lastmodified --check-column UPDATE_DATE --last-value '2018-07-19 16:14:38'

debananda_sahoo · ‎07-18-2018

@Felix Albani There is still some issue. Tables were exist in hive but I am not able to access it. Its showing below error while I am doing a select * from table. <small>java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1531811351810_0064_1_00, diagnostics=[Task failed, taskId=task_1531811351810_0064_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/t_currency/part-00000-2feb31ba-70a4-40a0-a64f-e976b8dd587a-c000.snappy.parquet at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) <strong>Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/t_currency/part-00000-2feb31ba-70a4-40a0-a64f-e976b8dd587a-c000.snappy.parquet</strong> at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:196) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:135) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:101) at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149) at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80) at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:674) at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:633) at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:145) at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:405) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:124) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)</small>

debananda_sahoo · ‎07-17-2018

Thanks Felix for your quick response. It worked. Thanks a lot.

debananda_sahoo · ‎07-17-2018

Thanks Felix for your quick response. It worked. Thanks a lot.

Online	Offline
Last Visited	‎10-25-2019 09:06 AM

Date Registered	‎04-09-2018 01:48 PM
Last Visited	‎10-25-2019 09:06 AM
Total Messages Posted	35

Caused by: java.lang.IllegalArgumentException: Unr...

not getting any data in kafka consumer using spark...

not getting any data in kafka consumer using spark...

Re: How to Convert XML file into CSV file using NI...

Re: Is it possible to convert nested XML to csv/te...

Re: Is it possible to convert nested XML to csv/te...

Re: How to Convert XML file into CSV file using NI...

Is it possible to convert nested XML to csv/text d...

Is it possible to convert nested XML to csv/text d...

Re: flatten json using nifi

Re: How to Convert XML file into CSV file using NI...

ERROR [KafkaServer id=1001] Fatal error during Kaf...

Re: Change Data Capture (CDC) with Apache NiFi (Pa...

Can Nifi be used for CDC from sql server by readin...

Can Nifi be used for CDC from sql server by readin...

java.lang.IllegalArgumentException: No running LLA...

Re: Insert overwite query on LLAP is not working b...

Re: How to resolve Unable to read HiveServer2 conf...

Data is not loaded in hive using “Hive warehouse c...

Data is not loaded in hive using “Hive warehouse c...

WARN cli.LlapStatusServiceDriver: Watch mode enabl...

Error while connecting Oracle DB using spark in HD...

Re: TypeError: jdbc() takes at least 3 arguments (...

Re: TypeError: jdbc() takes at least 3 arguments (...

TypeError: jdbc() takes at least 3 arguments (2 gi...

Re: incremental update in hive using sqoop

incremental update in hive using sqoop

Re: AttributeError in Spark

Re: AttributeError in Spark 2.3

Re: AttributeError in Spark