About ClouderaVMissue

ClouderaVMissue · ‎10-30-2018

You can build the custom processor on top of the nifi 1.2 version only and keep the nar file with the same name. It would work the same way. I have done it in my code.

ClouderaVMissue · ‎09-05-2018

Adding few more details to the query as in the current scenario , the data is encrypted using AWS KMS-Managed Keys (SSE-KMS) only using client side encryption. The client (Nifi) need to download the encrypted object from Amazon S3 along with the cipher blob version of the data encryption key stored as object metadata. The client then sends the cipher blob to AWS KMS to get the plain-text version of the key so that it can decrypt the object data. Please have a look to the option 1 of the below link: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html Not sure if we can do this with the existing Nifi processor with some customization OR we need to create a new processor altogether. Please suggest.

ClouderaVMissue · ‎09-04-2018

Currently i have created a process group which would get the events from SQS via getSQS processor and retrieve the S3 data from fetchS3Object processor. But now data is encrypted before placing in S3 and my Nifi need to decrypt the file collected using an AWS key which would be changing every 3 month so it should refresh the key by making an API call to AWS KMS. I didn't find any document to encounter this scenario. Please share some hint or processor which could handle this situation.

ClouderaVMissue · ‎08-25-2018

Current Nifi flow in which the above mentioned error is thrown in the GetSQS processor. Thanks, Bandhu Gupta

ClouderaVMissue · ‎08-25-2018

I am facing connectivity issue from Apache Nifi to Amazon SQS & S3: I have given the details in the below link: https://community.hortonworks.com/questions/214811/connection-issue-in-integrating-apache-nifi-with-a.html Thanks in Advance. Please suggest.

ClouderaVMissue · ‎08-25-2018

Facing issue when trying to connect with SQS from Nifi using the getSQS and fetchS3Object processor. Once all the details are entered and the processor is started , it throws an error as: 04:04:13 AEST - nifi-$nodename:9091 - ERROR GetSQS[id=ef15f694-0164-1000-0000-00002ea13324] Failed to receive messages from Amazon SQS due to com.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset

ClouderaVMissue · ‎05-18-2018

I have a file with multiple json seperated by ',' (comma) . I am trying to load one json as in one row of a table having only one column. So if my file has 10 json seperated by ',' , i want to load 10 rows in hive table with a single column having the entire json. Currently when i am creating the hive table , all the data is loaded in to 1 row in a column and then i can't apply get_json_object udf function to parse the data.My Json data also have a comma in between the fields. Please suggest how to create the hive table to handle this scenario.

ClouderaVMissue · ‎03-24-2017

I am using Upper & lower function in my hive query . So just wanted to know if that effects the performance .

ClouderaVMissue · ‎02-02-2017

Is there a way to add gear configuration button on the ambari 2.2.2.0 as well

ClouderaVMissue · ‎02-02-2017

I am currently using Ambari 2.2.2.0 and don't want to upgrade my ambari so is there a way to add this option in the current Ambari

ClouderaVMissue · ‎11-14-2016

Hi All, We are in the process of assessing what is required to scale/uplift/upsize the our current Hadoop cluster to cater to a greater number of concurrent users – we’re looking for your help to understand and review the assumptions, approach, and gain broader practice input from the experts on the considerations .We are using Zeppelin from which user would be running Queries using hiveserver2 . Our current cluster size , specification and few considerations regarding the use cases are listed below. 1)8 node cluster – 2 master node + 6 slave nodes (8 vcpu+1 TB disk space+64 GB RAM) Total data size is around 1.5 TB Replication factor is 3 Initially there are 50 users on Zeppelin but going forward , there would be 150 users from which 15-20 concurrent users. In order to maintain a good performance , we want to evaluate the upsize estimates to support parallel queries in cluster through 20+ concurrent user on Zeppelin.

ClouderaVMissue · ‎11-06-2016

I have increased the heap size to 1638 MB , Metastore heap size to 8052 Mb , Client heap size to 1024 mb but still i am getting this random behavior . Please suggest.

ClouderaVMissue · ‎09-26-2016

<property> <name>hive.security.metastore.authorization.manager</name> <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value> </property> <property> <name>hive.server2.allow.user.substitution</name> <value>true</value> </property> <property> <name>hive.server2.authentication</name> <value>NONE</value> </property> <property> <name>hive.server2.authentication.spnego.keytab</name> <value>HTTP/_HOST@EXAMPLE.COM</value> </property> <property> <name>hive.server2.authentication.spnego.principal</name> <value>/etc/security/keytabs/spnego.service.keytab</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.logging.operation.enabled</name> <value>true</value> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/tmp/hive/operation_logs</value> </property> <property> <name>hive.server2.support.dynamic.service.discovery</name> <value>true</value> </property> <property> <name>hive.server2.table.type.mapping</name> <value>CLASSIC</value> </property> <property> <name>hive.server2.tez.default.queues</name> <value>default</value> </property> <property> <name>hive.server2.tez.initialize.default.sessions</name> <value>false</value> </property> <property> <name>hive.server2.tez.sessions.per.default.queue</name> <value>1</value> </property> <property> <name>hive.server2.thrift.http.path</name> <value>cliservice</value> </property> <property> <name>hive.server2.thrift.http.port</name> <value>10001</value> </property> <property> <name>hive.server2.thrift.max.worker.threads</name> <value>500</value> </property> <property> <name>hive.server2.thrift.port</name> <value>10000</value> </property> <property> <name>hive.server2.thrift.sasl.qop</name> <value>auth</value> </property> <property> <name>hive.server2.transport.mode</name> <value>binary</value> </property> <property> <name>hive.server2.use.SSL</name> <value>false</value> </property> <property> <name>hive.server2.zookeeper.namespace</name> <value>hiveserver2</value> </property>

ClouderaVMissue · ‎09-26-2016

This is the hive error shown below. 21545:2016-09-21 18:34:04,785 ERROR [HiveServer2-Handler-Pool: Thread-78]: metadata.Hive (Hive.java:getTable(1121)) - Table XXXXX not found: default.XXXXX table not found 21622:2016-09-21 18:34:05,004 ERROR [HiveServer2-Handler-Pool: Thread-78]: hdfs.KeyProviderCache (KeyProviderCache.java:createKeyProviderURI(87)) - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!

ClouderaVMissue · ‎09-21-2016

i have seen a regular behavior of hive server2 getting stopped while we are using hive from Qliksense or multiple query from Zeppelin. I am using HDP-2.4.2.0 .I am getting the following error. Connection failed on host XXX:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py", line 200, in execute check_command_timeout=int(check_command_timeout)) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hive_check.py", line 74, in check_thrift_port_sasl timeout=check_command_timeout) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) Fail: Execution of '! beeline -u 'jdbc:hive2://XXXXX:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. stdin: is not a tty stdin: is not a tty Error: Could not open client transport with JDBC Uri: jdbc:hive2://xxxx:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://lxdb4231.dc.corp.telstra.com:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)

ClouderaVMissue · ‎07-11-2016

Yeah i ignored the warning and used /var for all of the rest operation and it worked without any issue.

ClouderaVMissue · ‎07-05-2016

Any Responses ?

ClouderaVMissue · ‎07-01-2016

I need to integrate the Qlik sense with Hive server2 running in my cluster

ClouderaVMissue · ‎06-29-2016

I would recommend to test it on IE and also verify that you don't have a pop up blocked in IE. I think it should work then.After changing these settings ,please restart the IE.

ClouderaVMissue · ‎06-29-2016

I am giving access to my users but didn't want to allow the access of Interpreter tab and Configuration tab.

ClouderaVMissue · ‎06-28-2016

I am using Zeppelin 0.6 and hdp 2.4.2 and i am working on a live projecct , we have some user specific Notebook but we are not finding a way to restrict the user to a particular schema in hive. We don't have LDAP authentication but still wanted to implement such security . We are looking out a way in which user can be restricted for some of the databases or Schemas using Apache Ranger.

ClouderaVMissue · ‎06-10-2016

We are facing the above error while we were using the below query sqoop import --connect jdbc:oracle:thin:@XX:1521/DATABASENAME --username USER --password PWD --table SCHEMANAME.TABLENAME --hive-import --hive-table TABLENAME --hive-overwrite --num-mappers 1 --as-parquetfile It's an issue when we are using Parquet and trying to ingest data in hive only because if we do the ingestion in hdfs with parquet , it gets completed.

ClouderaVMissue · ‎06-09-2016

We are only getting the above error if we are using parquet otherwise my table get pulled in hive easily .Please keep in mind that we have installed the HDP without internet so it's highly possible that we missed some thing.

ClouderaVMissue · ‎06-09-2016

sqoop import --connect jdbc:oracle:thin:@XXX:XXXX/YYYY --username YYYYY --password YYYYY --table A.BBBB --hive-import --hive-database default --hive-table test15 --as-parquetfile -m 1 Job job_1465371735536_0055 failed with state FAILED due to: Job commit failed: java.lang.IllegalArgumentException: Wrong FS: ____file:/tmp/default/.temp/job_1465371735536_0055/mr/job_1465371735536_0055/b402d4ba-1a16-46bc-92c6-91fe141070d2.parquet, expected: hdfs://lxapp5524.dc.corp.telstra.com:8020 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)

ClouderaVMissue · ‎06-08-2016

I am getting this issue when using sqoop with parquet

ClouderaVMissue · ‎06-08-2016

This is our machine for POC and creating a single node cluster so we installed Ambari and when we are adding our machine as a host , it gets added with the above warnings. Please suggest.

ClouderaVMissue · ‎06-08-2016

error.pngissue-sandpit.pngI have attached the snapshot , please suggest as currently we are installing HDP 2.4.0.0 on a test environment but Ambari shows this warning.

ClouderaVMissue · ‎01-05-2016

I have data in a SQL Server RDBMS. The data is in French and I need to save that data on hdfs. I also need the data translated into English.

Online	Offline
Last Visited	‎05-18-2016 11:34 PM

Date Registered	‎04-18-2016 10:51 PM
Last Visited	‎05-18-2016 11:34 PM
Total Messages Posted	28
Total Kudos Received	3

Re: Not enough disk space on host . A minimum of 2...

Re: Building a custom controller service against n...

Re: Integrate Apache Nifi with encrypted data in A...

Integrate Apache Nifi with encrypted data in Amazo...

Re: Connection Issue in integrating Apache NiFi wi...

Re: Connecting to AWS S3 from NiFi

Connection Issue in integrating Apache NiFi with A...

Loading json in hive

Does adding the upper function slows down the quer...

Re: How to add tsv option in Hive View Upload tab ...

How to add tsv option in Hive View Upload tab alon...

Uplift estimation of hadoop cluster

Re: Hive Server2 service getting stopped

Re: Hive Server2 service getting stopped

Re: Hive Server2 service getting stopped

Hive Server2 service getting stopped

Re: Not enough disk space on host . A minimum of 2...

Re: How to disable the Interpreter tab and for al...

How to integrate Qlik sense with hiveserver2

Re: How to provide user based authentication in Ze...

How to disable the Interpreter tab and for all of...

Zeppelin security and integration with Apache Rang...

Re: Issue when using parquet org.kitesdk.data.Data...

Re: Issue when using parquet org.kitesdk.data.Data...

Re: Issue when using parquet org.kitesdk.data.Data...

Issue when using parquet org.kitesdk.data.DatasetN...

Re: Not enough disk space on host . A minimum of 2...

Not enough disk space on host . A minimum of 2GB i...

Is there any tool in Hadoop which can do the langu...