About iraz098

sandyy006 · ‎06-04-2018

@irfan aziz Glad that it helped! Please click on "Accept" to mark this thread as closed.

iraz098 · ‎05-31-2018

Hi Below is the code of my spark stream however, the output contains extra characters one line before and one after the actual output data. Here is one output. I dont know why i am getting these "::::::::::::::" out/local-1527763545000/part-00000 :::::::::::::: (u'windows', 1) (u'mac', 1) (u'all', 1) :::::::::::::: from pyspark import SparkContext from pyspark.streaming import StreamingContext sc = SparkContext("local[2]", "WordCount") ssc = StreamingContext(sc, 15) words=ssc.textFileStream("file:///home/aziz/in") wordss = words.flatMap(lambda line: line.split(" ")) pairs = wordss.map(lambda word: (word, 1)) wordCounts = pairs.reduceByKey(lambda x, y: x + y) wordCounts.saveAsTextFiles("file:///home/aziz/out/local") ssc.start() # Start the computation ssc.awaitTermination() # Wait for the computation to terminate

iraz098 · ‎10-20-2017

You can search it from the console using command $ locate *hive-hcatalog-core*.jar

EricL · ‎09-17-2017

Impala does not support DATE data type, please refer to Cloudera doc: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_langref_unsupported.html You mentioned you changed to timestamp, it also failed, can you please let me know how you made the change?

mhegedus · ‎02-08-2017

Do you have any exceptions when you run the above client? Try running the Flume agent with the extra option -Dflume.root.logger=DEBUG,console

Zsombor_ · ‎02-02-2017

This should work: URL url = new URL("http://192.xxx.xx.21:44444"); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); conn.setRequestMethod("POST"); conn.setDoInput(true); conn.setDoOutput(true); OutputStream os = conn.getOutputStream(); DataOutputStream out = new DataOutputStream(conn.getOutputStream()); out.writeChars(ar); out.flush(); out.close(); os.close(); conn.connect();

iraz098 · ‎01-16-2017

@mhegedus Thank you. Yes changing the handler help. However, i had to use the unicode('utf-8') function for the text otherwise it was failing with error. Now i have to look for some python library to parse the values.

mhegedus · ‎01-04-2017

You have to send an array of JSONEvents otherwise the handler will fail to deserialize the events. An event must have at least a body and the body must be a string. You can also add optional headers. See the event specification in the user guide. import requests import json a = [{'body': 'my 1st event data'}, {'body': 'my 2nd event data'}] requests.post('http://localhost:44444', data=json.dumps(a)) You can also use GET method but still have to specify data to send.

iraz098 · ‎01-04-2017

@Jay SenSharma Thank you Jay but this http://hdp.localdomain:8989 is my http-source url. So i thought i could just post anything to it and it should ingest and deliver to the sink in my case hdfs. How will i know what type of data i can post.

SandyCT · ‎10-07-2017

I am trying to run a similar flume agent with HTTP source on the VM installed on my local computer. Not sure if the path and port is correct. When I ran my agent like this: ./flume-ng agent -n HttpAgent -c conf -f flume.conf it is stuck at the point http-source started. Not sure what's happening and not able to see the flume logs either! HttpAgent.sinks.hdfs-sink.hdfs.path = hdfs://quickstart.cloudera:8020/user/cloudera/output/http_flume Also if you can explain what will my output file be like when this works? Any suggestions?

TimothySpann · ‎07-27-2018

Don't use Flume anymore it is remove from HDP 3 Also you can't use samples.open... the correct URL is api.open... You also need to sign up for a real API key Use NiFi.

iraz098 · ‎01-02-2017

So finally i resolved the issue. The problem was with the .jar file and source type. I used this link http://blog.hubacek.uk/streaming-tweets-into-hadoop-part-ii/ to download the jar file. When i did not care much about the flume_classpath this time and also it placed the jar in and instead of hdp source i used the cloudera source for the agent config. TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource Now it works perfect enuf to get data into hdfs.

iraz098 · ‎01-04-2017

yes it was typo with the channel spelling.

jsensharma · ‎12-21-2016

@irfan aziz You mentioned "That was failing the resource manager with the error given below." However the output that you posted is INFO message. DO you see any error as well.

KuldeepK · ‎12-29-2016

Thank you so much @irfan aziz for the confirmation. I'm accepting answer given by @Michael Young. Please feel free to accept appropriate answer if required.

iraz098 · ‎12-19-2016

I managed to install cloudera manager. However, now that i want to install the other tools on a single node cluster with 8gb ram. after selecting the host through the web portal of cdh manager, the installation fails. From the logs when i filter on the error i get the following ERROR: relation "cm_version" does not exist I dont know where is the issue. Somebody wrote it was due to low ram however, i have plenty of ram. I dont know where is the issue.

geochap · ‎12-03-2016

Have you tried using facets? E.g. something like this should generate the facet values for a custom prop named test: /api/v9/interactive/entities?query=up_test:*&facetFields=up_test&limit=0 Not sure how it behaves if you have a lot of unique values for the property...probably just returns the most frequently used ones.

Mathivanan · ‎10-26-2016

you can create filter like this, but only issue , if you want not accessed in 30 days, you have do custom date range with before 30 days (if today date is 10/26/16, you have put date range 01/01/16 to 09/26/16).

iraz098 · ‎10-04-2016

The navigator will show that tables as deleted. Since this data is collected in Cloudera Navigator's own server. It must have its own retention period. It is possible that to make some changes to the collection script so that it should remove those rows that have a table deleted info.

iraz098 · ‎10-04-2016

Hi I am new to cloudera and trying to understand and set some standard procedure for metadata collection. Can someone share tell me what are the advantages of using tag and property-name (business metadata) and cases where we can use them. I could define tag and get the entities than use property-name. If you know any good link on use cases of cloudera meta data please share.

Online	Offline
Last Visited	‎05-31-2018 07:15 AM

Date Registered	‎10-04-2016 02:57 AM
Last Visited	‎05-31-2018 07:15 AM
Total Messages Posted	30
Total Kudos Received	3

Re: Can we load csv from local directory to Hbase ...

Spark stream output contains extra characters

Re: Adding Hive SerDe jar on SparkSQL Thrift Serve...

Re: Unsupported Datatype

Re: What should be the data handler for the flume ...

Re: What is the format to write to the dataoutputs...

Re: How can i post the response.text data to flume...

Re: is my url of http-source of flume agent correc...

Re: The flume agent http-source starts without err...

Re: Http-source flume into hdfs is not working.? ...

Re: How to use flume http-source to POST data from...

Re: What is the issue with flume agent configurati...

Re: How to configure the flume.conf file of an age...

Re: How important is the host command for installa...

Re: What could be the issue with my installation? ...

ERROR: relation "cm_version" does not exist

Re: Cloudera Navigator - Getting distinct values f...

Re: Cloudera navigator report for user

Re: How to keep Cloudera Navigator synchronized wi...

What is the difference between tag and property-va...