Member since
11-25-2016
62
Posts
2
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1932 | 09-28-2017 08:08 AM | |
| 2140 | 09-28-2017 08:02 AM | |
| 2737 | 09-28-2017 07:18 AM | |
| 2129 | 03-20-2017 01:12 PM |
06-20-2017
09:54 AM
Does it mean, if i have 3 node hdp, i can add nifi with the same node?
... View more
06-19-2017
10:34 PM
Hi Dan, I can not make any adjustment to the windows server, would you recommend that i put the nifi in windows for production?
... View more
06-19-2017
01:13 PM
use case: 1. NiFi was installed in Linux 2. The only way to access data from windows is via shared folder. Question: 1. Is it best practise to mount all 300+ to my nifi server (standalone or clustered)? 2. What other solution to support all 300+ windows shared dir?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
06-13-2017
03:45 AM
@Bryan Bende Thanks! Slow HDFS Put cause by a slow BlockReceiver. 2017-06-05 13:17:34,210 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
... View more
06-05-2017
11:53 PM
@tsokorai Upon checking, I'm having a slow blockreceiver. Server 3
2017-06-05 13:17:34,210 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 13:17:41,421 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6378ms (threshold=300ms)
2017-06-05 13:17:46,730 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 13:17:47,431 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 13:17:47,966 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 13:17:55,893 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6379ms (threshold=300ms)
2017-06-05 13:18:29,615 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6472ms (threshold=300ms)
2017-06-05 13:18:35,108 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6379ms (threshold=300ms)
2017-06-05 13:18:35,994 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6059ms (threshold=300ms)
2017-06-05 13:19:16,052 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6320ms (threshold=300ms)
2017-06-05 13:19:22,431 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6059ms (threshold=300ms)
2017-06-05 13:20:02,761 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6299ms (threshold=300ms)
2017-06-05 13:20:09,140 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6059ms (threshold=300ms)
2017-06-05 13:20:13,156 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:18:31,029 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6440ms (threshold=300ms)
2017-06-05 14:19:29,885 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6348ms (threshold=300ms)
2017-06-05 14:19:44,370 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6348ms (threshold=300ms)
2017-06-05 14:21:17,524 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 14:21:47,190 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 14:22:01,013 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 14:38:36,960 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6346ms (threshold=300ms)
2017-06-05 14:38:52,316 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6347ms (threshold=300ms)
2017-06-05 14:39:22,903 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:42:13,357 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:42:27,970 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:43:12,248 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:43:27,225 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6379ms (threshold=300ms)
2017-06-05 15:06:50,003 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 15:07:05,381 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-06 04:50:36,118 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6286ms (threshold=300ms)
Server 2
2017-06-05 13:20:26,755 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6326ms (threshold=300ms)
2017-06-05 13:20:33,072 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5996ms (threshold=300ms)
2017-06-05 13:20:33,366 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6379ms (threshold=300ms)
2017-06-05 14:19:06,478 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:19:21,001 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:19:35,321 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:21:53,394 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:22:36,143 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:22:50,619 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:38:33,079 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 14:39:27,543 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:40:01,897 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 14:43:02,861 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:43:17,503 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 15:06:20,526 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 15:06:35,291 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-05 15:06:56,859 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 15:07:38,618 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6347ms (threshold=300ms)
2017-06-05 15:19:27,961 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 15:30:20,450 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6316ms (threshold=300ms)
2017-06-06 04:50:49,974 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6303ms (threshold=300ms)
Server 1
2017-06-05 14:43:44,558 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 14:43:50,876 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6037ms (threshold=300ms)
2017-06-05 14:43:59,473 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6379ms (threshold=300ms)
2017-06-05 14:44:05,853 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6060ms (threshold=300ms)
2017-06-05 15:06:32,905 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6378ms (threshold=300ms)
2017-06-05 15:06:39,222 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6036ms (threshold=300ms)
2017-06-05 15:06:47,670 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6378ms (threshold=300ms)
2017-06-05 15:06:53,987 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5997ms (threshold=300ms)
2017-06-05 15:07:09,238 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6378ms (threshold=300ms)
2017-06-05 15:07:15,555 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5997ms (threshold=300ms)
2017-06-05 15:07:22,323 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6317ms (threshold=300ms)
2017-06-05 15:07:28,640 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5996ms (threshold=300ms)
2017-06-05 15:07:37,701 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6315ms (threshold=300ms)
2017-06-05 15:07:44,018 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5997ms (threshold=300ms)
2017-06-05 15:07:50,966 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6379ms (threshold=300ms)
2017-06-05 15:07:57,314 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6029ms (threshold=300ms)
2017-06-05 15:19:40,342 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6408ms (threshold=300ms)
2017-06-05 15:19:46,659 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5997ms (threshold=300ms)
2017-06-05 15:30:32,832 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 6410ms (threshold=300ms)
2017-06-05 15:30:39,150 WARN datanode.DataNode (BlockReceiver.java:receivePacket(571)) - Slow BlockReceiver write packet to mirror took 5997ms (threshold=300ms)
... View more
06-05-2017
07:51 AM
@tsokorai Hi Tomas, Upon check trace, it takes 13-15secs Sending DataTransferOp.
17/06/05 15:06:26 TRACE datatransfer.DataTransferProtocol: Sending DataTransferOp OpWriteBlockProto: header {
17/06/05 15:06:41 TRACE datatransfer.DataTransferProtocol: Sending DataTransferOp OpWriteBlockProto: header
17/06/05 15:07:02 TRACE datatransfer.DataTransferProtocol: Sending DataTransferOp OpWriteBlockProto: header {
17/06/05 15:07:16 TRACE datatransfer.DataTransferProtocol: Sending DataTransferOp OpWriteBlockProto: header {
... View more
05-26-2017
01:30 PM
@Matt Clarke Images are broken.
... View more
05-19-2017
06:33 AM
3 node HDP cluster with 10Gb network 3 X 2 TB (7200rpm) ea node master (node 1), worker2 (node 2), worker3 (node 3) why is it slow when putting a file in hdfs? [client1@master1 txt]$ ls -l
total 1203488
-rw-rw-r--. 1 client1 client1 4314198 Feb 20 11:51 random1.txt
-rw-rw-r--. 1 client1 client1 482564279 Nov 2 2016 random1.log
-rw-rw-r--. 1 client1 client1 745472547 May 15 15:03 random1.log.txt
-rwxrwxrwx. 1 client1 client1 183 May 15 14:05 transfer2hdfs.sh
[client1@master1 txt]$ ./transfer2hdfs.sh random1.txt
Fri May 19 09:05:56 +08 2017
Fri May 19 09:06:10 +08 2017
14
[client1@master1 txt]$ ./transfer2hdfs.sh random1.log.txt
Fri May 19 09:08:11 +08 2017
Fri May 19 09:09:31 +08 2017
80
[client1@master1 txt]$ ./transfer2hdfs.sh random1.log
Fri May 19 09:09:41 +08 2017
Fri May 19 09:10:35 +08 2017
54
... View more
Labels:
05-05-2017
04:22 AM
@Bryan Bende
Btw, most of the file is small around 10~50 mb, Upon testing: 350 Files, 10 mb per file, 1Gbps Network Client to Server (SSH) - 36 secs
[admin@client1 data]$ cat transfer.logs
Fri May 5 09:42:40 +08 2017
Fri May 5 09:43:16 +08 2017
36 secs Server to HDFS - 4,479 secs or 1.27 Hrs
[hadoop@master1 data]$ ./puthdfs-stdf.sh
Fri May 5 09:55:53 +08 2017
Fri May 5 11:10:32 +08 2017
4479 sec [client@master1 data]$ cat puthdfs-stdf.sh
start=`date +%s`
echo `date`
hdfs dfs -put *.txt /data/test/testing/
echo `date`
end=`date +%s`
runtime=$((end-start))
echo $runtime
I haven't install Hadoop client in NIFI Server (a windows server)
... View more
05-04-2017
08:46 AM
Having Problem with the slowness of PutHDFS to HDFS.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache NiFi