I am working on an application where I am making use of StreamingContext as well as SPark SQLContext. Now in the same application I am writing HiveContext as well, but it is throwing error saying rg.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at: ALthough I have set spark.driver.allowMultipleContexts = true in SparkConfig, but no luck. Could you please tell me how to proceed on this? ... View more

I have Spark which keeps on writing text data in a file in HDFS each day in a single folder. Folder Structure in HDFS 05th April /user/abc/2017-04-05/a.txt /user/abc/2017-04-05/b.txt /user/abc/2017-04-05/c.txt /user/abc/2017-04-05/d.txt 06th April /user/abc/2017-04-06/a.txt /user/abc/2017-04-06/b.txt /user/abc/2017-04-06/c.txt /user/abc/2017-04-06/d.txt I have also created HIVE table which maps the data from HDFS. I want to map HIVE and HDFS in such a way that ,as new data is added in new folder, it automatically maps to HIVE table. Everyday I have to manually alter the table to update the date. My Hive Table is create using below command: CREATE EXTERNAL TABLE retail_transaction (SysDate STRING, Id STRING, startTime STRING, endTimeSTRING, isSale STRING) PARTITIONED BY (dateval string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION '/user/abc'; To map: I use alter table below query every day to map newly added data ALTER TABLE retail_transaction add partition (dateval = '2017-04-04') location '/user/abc/2017-04-04'; How to automate this process. ... View more

I have installed Spark on RedHat Centos 6. Installed:Java 1.8,spark-2.1.0-bin-hadoop2.7,Scala 2.12 Environment Variable set for Hadoop config HADOOP_CONF_DIR Hadoop directory contains hdfs-site.xml , core- site.xml While executing I am getting below Warning and I am not able to write HDFS 17/03/27 03:48:18 INFO Utils: Successfully started service 'SparkUI' on port 4040. 17/03/27 03:48:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.4.124.113:4040 17/03/27 03:48:18 INFO SparkContext: Added JAR file:/storm/Teja/spark/target/uber-spark_kafka-0.0.1-SNAPSHOT.jar at spark://10.4.124.113:50101/jars/uber-spark_kafka-0.0.1-SNAPSHOT.jar with timestamp 1490600898913 17/03/27 03:48:20 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 17/03/27 03:48:20 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 17/03/27 03:48:21 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 17/03/27 03:48:22 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 17/03/27 03:48:23 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime ... View more

I have to do a real-time transfer of data from CouchBase to Kafka ... View more

I tried simple use case. GetFile processor will pick file PutFile will put those files. in PutFile configuration I have given a non-existent path, I am expecting NiFi to create path structure with user configured in bootstrap.conf NiFi is creating directory structure with root user. While checking status of NiFi it says: -bash-4.1# sh nifi.sh status Java home: /storm/prady/jdk1.8.0_111 NiFi home: /storm/prady/nifi-1.1.1 Bootstrap Config File: /storm/prady/nifi-1.1.1/conf/bootstrap.conf Error: Could not find or load main class org.apache.nifi.bootstrap.RunNiFi ... View more

I have configured a path in PutHDFS its throwing an authentication error LOG: 2017-03-03 01:52:29,200 DEBUG [IPC Client (1496249304) connection to dnn01.com/10.4.151.88:8020 from root] org.apache.hadoop.ipc.Client IPC Client (1496249304) connection to dnn01.com/10.4.151.88:8020 from root got value #39976 2017-03-03 01:52:29,201 TRACE [Timer-Driven Process Thread-7] org.apache.hadoop.ipc.ProtobufRpcEngine 105: Exception <- dnn01.com/10.4.151.88:8020: create {org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode="/raw/externaltbls/falcon/testing/.1PUGETSLA_PO810.P0125.EDIINV.P20150125.107.20160304025143629.gz":hdfs:hdfs:drwxrwxr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) 2017-03-03 01:52:29,201 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=015a1010-9c64-1ed3-c39b-d19ab2dfe19b] Failed to write to HDFS due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=015a1010-9c64-1ed3-c39b-d19ab2dfe19b]: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/raw/externaltbls/falcon/testing/.1PUGETSLA_PO810.P0125.EDIINV.P20150125.107.20160304025143629.gz":hdfs:hdfs:drwxrwxr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) ... View more

I little confused in understanding each one of them clearly. Can someone here help me in this. For Example: I have set following attribue: Max Select:2 Run Schedule: 30 sec Polling Interval: 0 sec In the source directory I have many files (say 10000), and I am writing these files to HDFS What would be output/expected behavior. ... View more

While running PutEmail processor in Apache NiFi, if there is an error then we are able to see that error by clicking red notification. Can we send this error to specified email ... View more