Member since
01-30-2017
49
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1492 | 02-23-2017 07:54 AM |
04-15-2017
03:42 AM
Thank you for your response. Is there any way to repair this file? Downloading again this file (11GB) is painful task
... View more
04-14-2017
11:55 AM
I am trying
to run HDP 2.5 in VMWare player by selecting “Open a Virtual Machine” button
but every time it is throwing an error saying: Failed to open virtual machine: SHA1 digest of
file
HDP_2.5_docker_vmware_25_10_2016_08_59_25_hdp_2_5_0_0_1245_ambari_2_4_0_0_1225-disk1.vmdk
does not match manifest. Please
provide a solution to me how to run it successfully ?
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
04-07-2017
09:38 AM
I am working on an application where I am making use of StreamingContext as well as SPark SQLContext. Now in the same application I am writing HiveContext as well, but it is throwing error saying rg.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at: ALthough I have set spark.driver.allowMultipleContexts = true in SparkConfig, but no luck. Could you please tell me how to proceed on this?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
04-06-2017
12:06 PM
In Sprig I am running Spark Application. Now I want to connect to HIVE to run HIVE query in Spring Suite itself. How to do this? I learned that HiveContext could be used but am clueless how to use this?
... View more
- Tags:
- hdp-2.4
- HIVECONTEXT
- Spark
- spring
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
04-05-2017
10:08 AM
I have Spark which keeps on writing text data in a file in HDFS each day in a single folder. Folder Structure in HDFS 05th April /user/abc/2017-04-05/a.txt /user/abc/2017-04-05/b.txt /user/abc/2017-04-05/c.txt /user/abc/2017-04-05/d.txt 06th April /user/abc/2017-04-06/a.txt /user/abc/2017-04-06/b.txt /user/abc/2017-04-06/c.txt /user/abc/2017-04-06/d.txt I have also created HIVE table which maps the data from HDFS. I want to map HIVE and HDFS in such a way that ,as new data is added in new folder, it automatically maps to HIVE table. Everyday I have to manually alter the table to update the date. My Hive Table is create using below command: CREATE EXTERNAL TABLE retail_transaction
(SysDate STRING,
Id STRING,
startTime STRING,
endTimeSTRING,
isSale STRING) PARTITIONED BY (dateval string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE LOCATION '/user/abc'; To map: I use alter table below query every day to map newly added data
ALTER TABLE retail_transaction add partition (dateval = '2017-04-04') location '/user/abc/2017-04-04'; How to automate this process.
... View more
- Tags:
- HDFS
- hdp-2.5.0
- Hive
- hive-metastore
- hql
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
03-27-2017
08:50 AM
I have installed Spark on RedHat Centos 6. Installed:Java 1.8,spark-2.1.0-bin-hadoop2.7,Scala 2.12 Environment Variable set for Hadoop config HADOOP_CONF_DIR Hadoop directory contains hdfs-site.xml , core- site.xml While executing I am getting below Warning and I am not able to write HDFS 17/03/27 03:48:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/03/27 03:48:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.4.124.113:4040
17/03/27 03:48:18 INFO SparkContext: Added JAR file:/storm/Teja/spark/target/uber-spark_kafka-0.0.1-SNAPSHOT.jar at spark://10.4.124.113:50101/jars/uber-spark_kafka-0.0.1-SNAPSHOT.jar with timestamp 1490600898913
17/03/27 03:48:20 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
17/03/27 03:48:20 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/03/27 03:48:21 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
17/03/27 03:48:22 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
17/03/27 03:48:23 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime
... View more
Labels:
- Labels:
-
Apache Spark
03-22-2017
11:09 AM
I have to do a real-time transfer of data from CouchBase to Kafka
... View more
Labels:
- Labels:
-
Apache Kafka
03-09-2017
02:17 PM
Its running as original user. that's problem
... View more
03-09-2017
01:52 PM
@Matt Clarke Thanks Matt, I tried this but its not working. Above use case is not working. I am expecting PutFile to create new directory with configured user (configured user in bootstrap.conf). But again and again its creating file/folder using logged in user
... View more
03-09-2017
04:19 AM
I tried simple use case. GetFile processor will pick file PutFile will put those files. in PutFile configuration I have given a non-existent path, I am expecting NiFi to create path structure with user configured in bootstrap.conf NiFi is creating directory structure with root user. While checking status of NiFi it says: -bash-4.1# sh nifi.sh status Java home: /storm/prady/jdk1.8.0_111
NiFi home: /storm/prady/nifi-1.1.1 Bootstrap Config File: /storm/prady/nifi-1.1.1/conf/bootstrap.conf Error: Could not find or load main class org.apache.nifi.bootstrap.RunNiFi
... View more
Labels:
- Labels:
-
Apache NiFi
03-03-2017
02:54 PM
@Matt Clarke Hi Matt, Thank you I understood it now, I will go through documentation to learn more on this. Thanks
... View more
03-03-2017
02:36 PM
@Matt Burgess Thank your for your response. Which all processor supports dynamic properties. Any link would be helpful. Thanks,
... View more
03-03-2017
02:11 PM
PFA image of same
... View more
03-03-2017
02:09 PM
1 Kudo
If I am adding a property from UI it is giving me an error saying property is not valid Although using UpdateAttribute processor we update the existing attribute but I am looking for ways to add new property in a processor say 'GetFile' and I want to add a property say 'nickname' PFA image of same
... View more
Labels:
- Labels:
-
Apache NiFi
03-03-2017
09:36 AM
Thank you for your response @Dan Chaffelson. we are not using any complex system like Kerberos, LDAP etc. I am running NiFi on a linux system where I logged in as root user. There is another path where I am able to write to HDFS but as soon as I change the path in HDFS to some other then issue appears. To me it seems like an authentication issue, but where do we configure to instruct NiFi to write as root user?
... View more
03-03-2017
07:07 AM
I have configured a path in PutHDFS its throwing an authentication error LOG: 2017-03-03 01:52:29,200 DEBUG [IPC Client (1496249304) connection to dnn01.com/10.4.151.88:8020 from root] org.apache.hadoop.ipc.Client IPC Client (1496249304) connection to dnn01.com/10.4.151.88:8020 from root got value #39976
2017-03-03 01:52:29,201 TRACE [Timer-Driven Process Thread-7] org.apache.hadoop.ipc.ProtobufRpcEngine 105: Exception <- dnn01.com/10.4.151.88:8020: create {org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode="/raw/externaltbls/falcon/testing/.1PUGETSLA_PO810.P0125.EDIINV.P20150125.107.20160304025143629.gz":hdfs:hdfs:drwxrwxr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) 2017-03-03 01:52:29,201 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=015a1010-9c64-1ed3-c39b-d19ab2dfe19b] Failed to write to HDFS due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=015a1010-9c64-1ed3-c39b-d19ab2dfe19b]: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/raw/externaltbls/falcon/testing/.1PUGETSLA_PO810.P0125.EDIINV.P20150125.107.20160304025143629.gz":hdfs:hdfs:drwxrwxr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
... View more
Labels:
- Labels:
-
Apache NiFi
03-01-2017
12:40 PM
I little confused in understanding each one of them clearly. Can someone here help me in this. For Example: I have set following attribue: Max Select:2 Run Schedule: 30 sec Polling Interval: 0 sec In the source directory I have many files (say 10000), and I am writing these files to HDFS What would be output/expected behavior.
... View more
Labels:
- Labels:
-
Apache NiFi
02-28-2017
05:38 AM
@Matt Clarke You are saying making a processor specific entry into logback.xml. ? Any example of this?
... View more
02-24-2017
11:36 AM
Thanks Arun, you are right. I did it using TailFile. @Arun A K
... View more
02-24-2017
07:43 AM
My Usecase: Pick log data and send it to PutSplunk. Solution I am trying is: I used TailFile processor which will have details of path of log file. And it will keep fetching the updated data from this file and will put in FlowFile routed to PutSplunk. As in the image attached, its not picking any more data and seems like hung, although the data is queued up. and the processors are in running state. Also added the screen dump of Processor Configuration
... View more
Labels:
- Labels:
-
Apache NiFi
02-24-2017
06:31 AM
I thought of one solution: I will use FetchFile processor and I will put the path of LOG file in its configuration. Will that work? (FetchFile will keep fetching the new content in the file and putting those content to destination which is PutSplunk in my case)
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2017
07:54 AM
Thanks @Bryan Bende @Timothy Spann, @ozhurakousky foryour reply. It was some configuration issue. While trying to put file into Splunk, I was using web-port (8081 in my case) of splunk in configuration of PutSplunk. When I pointed my PutSplunk configuration to TCP port of Splunk(In Splunk setting go to Data Inputs -> Click on TCP and enter details as instructed to create a new TCP input port of Splunk) it started working properly.
... View more
02-23-2017
05:57 AM
@Bryan Thanks Bryn. I will try this and will let you know if this works or not.
... View more
02-22-2017
11:12 AM
@Bryan Bende Hi Bryan, How to use logback.xml for logging into Splunk. Any suggestion or link or example would be helpful. Thanks,
... View more
02-22-2017
10:11 AM
My NiFi data model looks like GetFile--> PutSplunk -->PutEmail(on success one email and on failure different email) While executing the processor, I am able see email in both scenario. But when I go to Splunk server, I am not able to track file.
... View more
Labels:
- Labels:
-
Apache NiFi
02-21-2017
01:10 PM
@ozhurakousky Yeah sure. I have uploaded the image now.
... View more
02-21-2017
01:00 PM
1 Kudo
When I Cleared whole queue and tried again. Now again its saying something similar, ArrayIndexOutOfBounds exception
... View more
Labels:
- Labels:
-
Apache NiFi
02-21-2017
12:38 PM
Hi Peter, I already did that. Am I still missing something?
... View more
02-21-2017
11:00 AM
While running PutEmail processor in Apache NiFi, if there is an error then we are able to see that error by clicking red notification. Can we send this error to specified email
... View more
Labels:
- Labels:
-
Apache NiFi