Member since
02-02-2018
20
Posts
0
Kudos Received
0
Solutions
12-03-2018
03:37 PM
Thanks @Kei
Miyauchi. I've upgraded my HDF as well. Now I'll just try to use PutHive3Streaming. Thank you!
... View more
11-30-2018
04:31 PM
Was this issue solved? I need the answer for 2) as well
... View more
11-26-2018
03:48 PM
I'm facing the same issue with a similar environment (HDP 3.0 Nifi 1.5). Was it resolved?
... View more
10-19-2018
02:15 PM
Hey @Geoffrey Shelton Okot Sorry for the long delay, and thanks for your help! It seems that I didn't have to run the beeline commands, since no scripts were generated (that's my understanding, at least). Now I'm facing another issue preventing me from finishing this upgrade. I reported it here if you want to take a look (any help is welcome!): https://community.hortonworks.com/questions/224430/host-is-not-sending-heartbeats-after-ambari-upgrad.html?childToView=224540#comment-224540 Thank you very much!
... View more
10-18-2018
02:37 PM
It seems my external link was removed, so I'm posting it again just in case: https://drive.google.com/drive/folders/1EDC2lri6r9Sqg-XRT2I7gCAncroWiJCl?usp=sharing
... View more
10-17-2018
05:11 PM
Thanks for your reply, @Jay Kumar SenSharma 1. > rpm -qa | grep ambari
ambari-metrics-hadoop-sink-2.7.1.0-169.x86_64
ambari-agent-2.7.1.0-169.x86_64
ambari-metrics-grafana-2.7.1.0-169.x86_64
ambari-metrics-monitor-2.7.1.0-169.x86_64
ambari-server-2.7.1.0-169.x86_64
ambari-metrics-collector-2.7.1.0-169.x86_64 2. Attached (agent and server logs) 3. > telnet hadooptest1 8440
Trying 10.165.0.11...
Connected to hadooptest1.
Escape character is '^]'. I double-checked here, and the agent has the correct server IP. I uploaded the log files here, if you need to take a look: AmbariLogs Thanks again for your help!
... View more
10-16-2018
06:50 PM
Hello my friends, I was wondering if you could help me with something. We're upgrading our environment to HDP 3.0.1, and for that we had to upgrade Ambari to 2.7.1.0. After the Ambari upgrade, the HDP 3.0.1 installation is not working due to critical errors with the hosts. Aside from the server host, all other 3 we have in test environment are not sending heartbeats. I've tried many solutions already posted here in the community, but nothing seems to work. I've already restarted the server, checked the hosts IPs, and upgraded the other services (Metrics and SmartSense) to the latest version. Before the upgrade, everything was working fine, so I'm not sure if this is a known issue or if I'm just missing something. Any ideas? Thanks in advance. Cheers
... View more
Labels:
- Labels:
-
Apache Ambari
10-12-2018
10:01 PM
Nevermind. I ran the command without the -execute, and it worked fine. Now I'm a little bit confused regarding step 7 of the pre-upgrade documentation. Can you help me with that? "7. Login to Beeline as the Hive service user, and run each generated script to prepare the cluster for upgrading. The Hive service user is usually the hive user. This is hive by default. If you don’t know which user is the Hive service user in your cluster, go to the Ambari Web UI and click Cluster Admin > Service Accounts, and then look for Hive User." I agree that this documentation needs some updates.
... View more
10-12-2018
04:48 PM
I'm having the same issue here. After changing the folders, I'm still getting this exception: 2018-10-12T12:51:14,010 WARN [main] hive.ql.metadata.Hive - Failed to register all functions. org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: Invalid method name: 'get_all_functions' Do you have any idea what's happening here? Thanks in advance! Cheers
... View more
07-27-2018
03:33 PM
Matt, thanks a lot for all your help. I was able to refactor my dataflow, reducing the number of groups and keeping everything simple in a single dynamic flow. Just to elaborate a little bit better, here's what I did. Data coming in CSV format separated by pipes. e.g.: (transaction #, sequence #, table code) 123|456|35| 123|456|36| 123|456|100| First I split the flowfile into multiple ones using SplitText >> then I used the ExtractText processor to grab the 3rd field (table code) >> LookupAttribute setting the user-defined-field schema.name (to be used by AvroSchemaRegistry controller service) >> Push the data to Kafka and Hive using the appropriate processors. Thanks a lot!
... View more
07-25-2018
06:47 PM
Hi Matt. First of all, thank you so much for the explanation. My scenario currently falls into the 3rd one you described: I have multiple table codes coming in a single flowfile. Could you please elaborate on how to use the PartitionRecord processor? I tried here using the CSVReader and CSVSetWriter controller services, but they ask for an Avro schema as well. All the tables' structures I'm working with right now have only the 3 first fields in common (the last one being the table code). The rest of them varies, so I got a little bit confused on how to set this avro schema.
... View more
07-25-2018
05:53 PM
Hi experts, Good day! I've been using Nifi for a couple of months, so I'm still learning lots of new things every day. I'm building a dataflow to get csv data (separated by pipes - '|' ) and push it to different targets (e.g. Hive, SQL Server, and Kafka). The project started fine but the dataflow started getting bigger and bigger and now I'm finding it difficult to manage. I just wanted to ask for some help understanding if I'm currently working with the best possible scenario. More details below. I'm getting data from a ListenHTTP processor. Data comes as csv separated by pipes. One of the fields is a code that identifies which table the data should be pushed to, so I've created one process group for each "table". Here's where I think the dataflow gets complicated. Each of those groups (23, to be precise) has 4 other groups, each responsible to push data to a specific target. Since I have a Hive dataflow inside these groups, I had to create the Avro schema defining the structure for each table. I was just wondering if I could substitute this dataflow with a single one that evaluates the code in the csv, and "chooses" the correct avro schema to be used. I did some research but couldn't progress further. If there's a way to do it, I could simply substitute those 23 groups with a single dynamic dataflow. Hopefully you can help me with this scenario. Thanks in advance! Sincerely, Cesar Rodrigues
... View more
Labels:
- Labels:
-
Apache NiFi
02-26-2018
04:17 PM
Hi, guys, I have a data flow in Nifi that gets a file from the server and converts it to Avro to stream data to Hive. In this flow, I have some sensitive information that I need to hash (SHA2_512). I checked that Nifi has a couple of processors to work with hash, but it seems they only do this for the whole file. Is there a way to hash a specific field? Before converting to Avro, my flow files are coming from the server as fields separated by pipes ('|'). Thanks in advance! Cheers
... View more
Labels:
- Labels:
-
Apache NiFi
02-14-2018
06:17 PM
Thanks, @Matt Burgess! This helped a lot 😉
... View more
02-07-2018
08:08 PM
@Abdelkrim Hadjidj Thank you! Could you please provide more details on how to use the schema registry? I'm having some trouble with that.
... View more
02-07-2018
08:06 PM
@Matt Burgess I've never used the Avro Schema before. Could you please explain how to name the fields in it? I checked the documentation, but it's a little bit confusing. Thanks in advance!
... View more
02-07-2018
05:45 PM
Hi, guys, So I have an incoming FlowFile with content text delimited by pipes ('|'), and I want to send this information to several destinations. To convert it to JSON, for example, I know I can use the AttributesToJSON processor, but how exactly can I access the FlowFile content and convert them to attributes? e.g. original FlowFile content: 1234567891285|37797|1| the brown fox FlowFile attributes (after converting): id = 1234567891285 sequence = 37797 category = 1 text = the brown fox ... and after that I could use AttributesToJSON to generate my JSON file. Any ideas on how to achieve this? Thanks in advance! Cheers.
... View more
Labels:
- Labels:
-
Apache NiFi
02-05-2018
02:03 PM
@Shu, thank you very much. It worked perfectly!
... View more
02-02-2018
09:28 PM
Hello, guys, I'm trying to use Nifi to split a text file into 2 other files. I just have a problem with that: I need to split them based on their category type. e.g. FlowFile content: Some fixed text |1| more text Another field |8| more text Last one |1| more text With that, I'd like to split this file into, for example: first FlowFile: Some fixed text |1| more text Last one |1| more text second FlowFile: Another field |8| more text Do you guys have any idea on how to accomplish that using Nifi? I appreciate any help you can provide. Thanks in advance, Cheers!
... View more
Labels:
- Labels:
-
Apache NiFi