Vj1989
Explorer
My Accepted Solutions
Show All

Apache NiFi PutParquet Error

by Explorer in Support Questions
‎06-09-2020 11:18 PM
‎06-09-2020 11:18 PM
Any idea on why we get this error while writing files to HDFS using PutParquet Processor? at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:498) ... 28 common frames omitted 2020-06-10 12:20:24,729 INFO [NiFi Web Server-495935] o.a.n.c.s.StandardProcessScheduler Starting PutParquet[id=92ceb89d-0172-1000-35ad-31dcdeb7b51d] 2020-06-10 12:20:24,729 INFO [NiFi Web Server-495935] o.a.n.controller.StandardProcessorNode Starting PutParquet[id=92ceb89d-0172-1000-35ad-31dcdeb7b51d] 2020-06-10 12:20:24,784 INFO [Timer-Driven Process Thread-7] o.a.hadoop.security.UserGroupInformation Login successful for user 13010840 using keytab file /home/aiadmin/aiadmin.keytab 2020-06-10 12:20:24,791 INFO [Timer-Driven Process Thread-7] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutParquet[id=92ceb89d-0172-1000-35ad-31dcdeb7b51d] to run with 1 threads 2020-06-10 12:20:24,885 ERROR [reader] net.schmizz.sshj.transport.TransportImpl Dying because - Broken transport; encountered EOF net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF at net.schmizz.sshj.transport.Reader.run(Reader.java:57) 2020-06-10 12:20:24,938 INFO [Thread-513233] o.a.h.h.p.d.sasl.SaslDataTransferClient SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-06-10 12:20:24,940 INFO [Thread-513233] org.apache.hadoop.hdfs.DataStreamer Exception in createBlockOutputStream blk_2184945431_1111306988 java.io.EOFException: null at java.io.DataInputStream.readByte(DataInputStream.java:267) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240) at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221) at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:203) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:193) at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1731) at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataS at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716) PutParquet[id=92ceb89d-0172-1000-35ad-31dcdeb7b51d] Failed to write due to Could not get block locations. Source file "<<directory>>.27037720888019658" - Aborting...block==null: java.io.IOException: Could not get block locations. Source file "<<directory>>.27037720888019658" - Aborting...block==null ... View more
Labels:

Re: ExecuteScript with python 3

by Explorer in Support Questions
‎05-25-2020 05:32 PM
‎05-25-2020 05:32 PM
Why don't you try with ExecuteStreamCommand as shown here: https://www.youtube.com/watch?v=rUJOS6LT5Jk&t=184s ... View more

Re: Merge empty Flow files based on attribute in n...

by Explorer in Support Questions
‎05-03-2020 01:53 PM
‎05-03-2020 01:53 PM
Can you please send the screenshot of the configuration done in correlation attribute name? ... View more

Re: How to run a python script with dependencies i...

by Explorer in Support Questions
‎04-26-2020 05:46 AM
‎04-26-2020 05:46 AM
You may follow this video for the complete instruction.   https://www.youtube.com/watch?v=rUJOS6LT5Jk     ... View more

Re: NiFi ExecuteCommandStream with sed

by Explorer in Support Questions
‎04-26-2020 05:43 AM
‎04-26-2020 05:43 AM
You have to provide full path of your script in "Command Path" Attribute.  All the flowfile arguments can be passed in "Command Arguments" Property.   In below example, I am considering that I have a shell script created under /tmp directory with the name yourscript.sh This script should have all the logics inside to handle receiving arguments and other logics.      Script Content (/tmp/yourscript.sh)   #!/bin/bash echo $1   Expected Output: Just print the name of the FlowFile inside the flow file content.   You may check this video for complete detail: https://www.youtube.com/watch?v=rUJOS6LT5Jk&t=27s ... View more

Re: Nifi RouteOnAttribute with variable attributes

by Explorer in Support Questions
‎04-26-2020 05:30 AM
‎04-26-2020 05:30 AM
Hi Markushg,   When you want to apply condition on any number of attributes based on a pattern match. Then you may consider using anyMatchingAttribute NiFi Function.   Here is the link to the documentation: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#anyattribute   In your case, You may have to do something like as below: ${anyMatchingAttribute("syslog.structuredData*os"):equals('linux69')} ... View more

Re: Apache Nifi Docker container volumes location

by Explorer in Support Questions
‎04-26-2020 05:22 AM
‎04-26-2020 05:22 AM
NiFi Templates are stored within flow.xml.gz file, you can refer to below screenshot from NIFi Documentation.     flow.xml.gz is stored in the <</conf >>directory. In your case I can see the mount point is pointing to <</opt/nifi/nifi-current/conf >>directory   The best way to manage Templates are from "Global Menu"->Templates   ... View more

Re: ERROR while loading CSV to PostgreSQL using Pu...

by Explorer in Support Questions
‎04-11-2020 01:49 PM
‎04-11-2020 01:49 PM
The table name is different in than the one which you are trying to load data.   Please notice carefully that the error in Nifi is complaining about below table, whereas in your create table there is no prefix with the name "op"   op_working_time_nifi   ... View more

Re: How to learn Nifi practically for beginners .....

by Explorer in Support Questions
‎04-10-2020 10:00 PM
‎04-10-2020 10:00 PM
Here is a link of 5.5 hours of free Apache NiFi course.    https://www.udemy.com/course/apache-nifi-latest-course/?couponCode=LATESTNIFI ... View more

Re: Detect duplicate based on 2 attributes per sec...

by Explorer in Support Questions
‎04-10-2020 04:02 PM
‎04-10-2020 04:02 PM
Sure.  Fortunately in this case I was able to explore answer myself. I would remember to open a new thread from next time.  ... View more

Re: Extracting NiFi Provenance Data using SiteToSi...

by Explorer in Community Articles
‎04-10-2020 03:39 PM
‎04-10-2020 03:39 PM
To all the members who are asking if it is still valid, the answer is yes this seems to be still valid. I have extracted provenance data by connecting to same instance of NiFi rather than having multiple NiFi instance.    https://www.youtube.com/watch?v=2v-H3WeaCkc ... View more

Re: How to Read from H2 datanase in NiFi

by Explorer in Support Questions
‎04-09-2020 01:29 PM
‎04-09-2020 01:29 PM
You connect to H2 database like any other database using JDB connection. Here is the link where you can find H2 driver and documentation. www.h2database.com/html/download.html   By the way, I have also uploaded a video explaining step by step process on connecting to NiFi H2 database. You may check it out here.  https://www.youtube.com/watch?v=tsAR2f4uGK4 ... View more

Re: Merge multiple json with the same id in Nifi?

by Explorer in Support Questions
‎03-14-2020 10:51 AM
‎03-14-2020 10:51 AM
Refer to this thread for similar question: https://community.cloudera.com/t5/Support-Questions/merge-too-csv-files-in-nifi/m-p/229259#M191116 ... View more

Re: Merge multiple json with the same id in Nifi?

by Explorer in Support Questions
‎03-14-2020 09:38 AM
‎03-14-2020 09:38 AM
What you are trying to achieve is can be done using NiFi Lookup Processor. Please explore more on that. ... View more

Re: Detect duplicate based on 2 attributes per sec...

by Explorer in Support Questions
‎03-11-2020 09:37 PM
‎03-11-2020 09:37 PM
Can " Age Off Duration" be set to 7 days to detect any duplicate files coming in last 7 days? What will be the performance impact on that? ... View more

Re: Move file from http cloud serve to another ser...

by Explorer in Support Questions
‎02-19-2020 09:37 AM
‎02-19-2020 09:37 AM
You may use below commands for this.    wget http://77.17.6.166/tls/file/nn.csv scp nn.csv username@77.17.6.122:/tls/file/hello.csv ... View more

Re: Unknown host exception on nameservice

by Explorer in Support Questions
‎02-19-2020 09:29 AM
‎02-19-2020 09:29 AM
Do you have entries of all the nodes under /etc/hosts file? ... View more

Re: By default Metastore & Hiveserver2 not running...

by Explorer in Support Questions
‎02-19-2020 09:26 AM
‎02-19-2020 09:26 AM
Please run below command to setup auto startup of hive service:   sudo systemctl enable hive-server2    Do let me know the results. Thanks. ... View more

Re: Unable to start hive metastore

by Explorer in Support Questions
‎02-19-2020 09:20 AM
‎02-19-2020 09:20 AM
Writing this so that it can help someone in future: I was installing Hive and getting error that It hive metastore wasn't able to connect, and I successfully resolved the error by recreating the hive metastore database.   Someone the user which was created in mysql Hive metastore wasn't working properly and not able to authenticate. So I dropped metastore DB, Dropped User. Recreated Metastore DB, Recreated User, Granted all privileges and then it was working without issues. ... View more

Re: How to Extract All Hive Tables DDL

by Explorer in Community Articles
‎02-19-2020 09:15 AM
‎02-19-2020 09:15 AM
Well you can write a small Unix script to do that. Here are the steps breaukp.   Step 1: Get the list of all the databases, commands for that is and redirect the output to any temporary file (e.g. /tmp/databases)   hive -e "show databases;" >> /tmp/databases   Step 2: Loop through each database to get the list of tables by using "show tables" and redirect the output to temporary file (e.g. /tmp/tables)   for f in `cat /tmp/databases` do echo "Database name is $f , Table Names are as below:" hive -e "show tables;" >> /tmp/tables done     ... View more
Contact Me
Online
Offline
Last Visited
‎07-22-2020 09:20 AM
Public Statistics
Date Registered ‎02-19-2020 09:11 AM
Last Visited ‎07-22-2020 09:20 AM
Total Messages Posted 22