Solved

I have a relatively large CSV (~80GB) I need to transform into multiple JSON documents/records. I'm using a ConvertRecord processor with a CSVReader and AvroRecordSetWriter and that's where my CSV gets stuck. What's the best approach? Break up the CSV prior to converting it or try to get more horsepower on the server? Server Mem: 16GBCores: 4Maximum Timer Driven Thread Count : 16Java Min/Max Heap: 2GB/10GB

Hi  I am trying to import a single table from sqoop and i get this error:   Warning: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]19/06/18 16:25:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.019/06/18 16:25:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instea

Hi,Data is in files in the following format 2019-06-15T15:43:12 ( yyyy-MM-ddThh:mm:ss )When I do a select * on the table , it is displayed as NULLHive version - 1.2.1ALTER TABLE table SET SERDEPROPERTIES ("timestamp.formats"="yyyy-MM-dd'T'HH:mm:ss"); - did not helpFollowed the links,https://issues.apache.org/jira/browse/HIVE-9298https://www.ericlin.me/2017/03/alternative-timestamp-support-in-hive-iso-8601/Could you please help on this. Thanks

Hello. We're on version CDH 5.12.2.On Cloudera Manager, Impala -> Query , I see query that is "Executing" but query status is "Finished".query_duration was 42/51 mins. User is running query on Hue. I've searched through official papers, google it, but getting nowhere.Did user left Hue before getting query result? Can anyone give me explanation on this? Thank you! 

I have completed the installation of CDH 5.16.1 on Ubuntu 16.04 LTS with OpenJDK 8.   My employer has developed a UI using Banana (currently packaged with Fusion) that I am required to run on a VM hosted in my machine. The Banana UI needs Solr both to operate and to store data collections that can be visualised. In this case, the Solr instance within the VM uses ports 8983 and 7574. The Banana UI also has a unique security feature developed by my employer (which I have to familirise myself with) that does not yet allow me to host the entire contents of the VM on my server which also hosts the Cloudera CDH instance.   Due to the above arrangement, it is not possible for me to startup Cloudera CDH (though Cloudera Manager is still running). This has proven to be a difficulty since I need to use Solr and also use the various tools (Spark etc).   Could you tell me how I could change the default port for Cloudera Solr (from 8983 to an appropriate port) and for Cloudera Zookee

Hi, I have a dashboard application (using Zoomdata) in which we submit queries to a Impala resource pool with limited number of concurrent requests. While we load the dashboard, some queries might get queued if conc. requests limit is reached.When I change the dashboard time range, I noticed the running queries are properly cancelled, but the queued ones stay in the queue and are going to execute before the new queries submitted. Do you know if there is any Impala JDBC driver property or Impala server setting to be able to cancel queued queries and prevent them from running?

I'm testing Hue - SAML integration.SAML login now works, but all users land in Hue with "default" group.Is there a way to assign Hue groups and permissions based on SAML attributes?

I tried sudo apt-get install lucidworks-hdpsearch. That also gives the same error. My OS is ubuntu 16.04 and I have ambari 2.6 installed.Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/SOLR/6.6.2/package/scripts/solr.py", line 139, in <module> Solr().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/SOLR/6.6.2/package/scripts/solr.py", line 20, in install self.install_packages(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 821, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_manag

I am calling an API which returns a JSON object as the body of the response as a flow file. I want to count the total records in the flow file.I tried record processors like CalculateRecordStats needs a record reader and that in-turn needs a schema. But I don't have a schema. So Is there a better way to do it?{"result":[ { "key1": "value1", "key2": "value2" }, { "key1": "value1", "key2": "value2" }]}This is the sample JSON the api response body returns. The records are the items of the list which is the value of the "result" keyIn the above sample there are 2 records (2 items in the list). So that is what I want to calculate.You can also answer here

hi allwe have ambari cluster - HDP version - 2.6.4on one of the master machine we cant start the name nodeand we notice that ls  /hadoop/hdfs/journal/hdfsha/current/ | grep edits_inprogress  edits_inprogress_0000000000018783114.emptywe not have the edits_inprogress_xxxxxx filewhat we have is only the file - edits_inprogress_0000000000018783114.emptyany idea how to recover the - edits_inprogress_xxxxxx file2019-06-13 19:45:42,473 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond.        at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createNewUniqueEpoch(QuorumJournalManager.java:183)        at

http://archive.cloudera.com/gplextras5/parcels/5.16.1/  new gplextras parcel has bug.as follows：impala/lib/libimpalalzo.so: undefined symbol: _ZN4base21ThreadCollisionWarner5LeaveEv

I use chmod to give others the read permission on the home/maria_dev directory. However, the user hive still cannot read the directory.[maria_dev@sandbox-hdp ~]$ ls -lddrwx---r-- 1 maria_dev maria_dev 4096 Mar  5 05:35 .[maria_dev@sandbox-hdp ~]$ sudo su hive[hive@sandbox-hdp maria_dev]$ lsls: cannot open directory .: Permission denied

Hi,Can you provide the procedure for backing up ranger kms? I've backed up the ranger and ranger audit databases using the postgres dump command (pg_dump), but I'm running into an issue when I try the same procedure for the rangerkms database. I tried the following command and got the error shown below. To access the ranger kms I have to login using the keyadmin user. I was thinking that this is my issue, that I'm trying to create a backup of this database using the wrong user? Any thoughts are suggestions are appreciated as I haven't been able to find any support articles on this. Thanks in advance...pg_dump -U rangeradmin rangerkms > rangerkms.sqlPassword:pg_dump: [archiver (db)] query failed: ERROR: permission denied for relation ranger_masterkeypg_dump: [archiver (db)] query was: LOCK TABLE public.ranger_masterkey IN ACCESS SHARE MODE

In my cluster Event server is faild to start.When I restart it it shows the below error message : ---------Process Status:This role's process failed to start.Cloudera Manager Descriptor AgeNot enough data to test: Test of whether the Cloudera Manager descriptor is up to dateUnknown : Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date.--------------- Someone sugesated to increase Java heap size of Event server 1 GB. I can see its already 1 GB. Can you guys help me why its failed to start , what other things I can check?? 

Hi!How can I configure InvokeHTTP to return comma separated flow file? At the moment it is returning file in below format (tab separated):RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772103971RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772104117RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772101000Thanks in advance!

Hi,I wanted to know whether it is possible to use HDP and HDF simultaneously on same virtual machine. Actually wanted to use HDF Nifi/Kafka (producer) services to fetch data from source and wanted to trigger Kafka(consumer) at HDP.Facing issue when both HDF/HDP are running not able to use localhost:1080

Hi,Need help here:When I am doing "ip addr show" in HDP 3.0 shell in box; it is showing only eth0 and lo. Though in NETWORK options I have two network adapters 1) NAT2) NETNETWORKIn my VirtualBox NET NETWORK setting I have given NETNETWORK 11.0.2.0/24. My concern is when I am doing ip addr show my NETNETWORK ip 11.0.2.# should be displayed, which is not displaying currently.Do I need to have some setting changes in HDP 3.0. Or HDP 3.0 doesn't sense network adapter other than NAT?

Hello,what is the purpose of Spark2_Jobhistoryserver_Process? . Under what circumstances we get alert for 'Spark2_Jobhistoryserver_Process' apart from stopping the service.

Hello, where I can find updated information upon how to install a small cluster in Azure? Preferably a video.I understand that using cloudbreak looks like the most promising way. After I have my cluster, is it possible to store the parameters and be able to launch the cluster on Azure at will, without having to spend a lot of time configuring the same things?Or even better do you know if there is a way to launch already preconfigured clusters from a blueprint? Say for example that I have a script which automatically provides 5 or 6 VMs on Azure and then launce HDP automatically?Thank you

0: jdbc:hive2://localhost:21050/defalt> select 1 * (NULL);null+-------------+--+| 1 * (null) |+-------------+--++-------------+--+ vs. 0: jdbc:hive2://localhost:21050/defalt> select 1 * (CASE WHEN '1' IS NULL THEN NULL ELSE NULL END);Error: AnalysisException: Arithmetic operation requires numeric operands: 1 * (CASE WHEN '1' IS NULL THEN NULL ELSE NULL END) (state=HY000,code=0) Is this a bug in Impala? Impala version is 2.11 (CDH 5.14.4)   Same statement works with Hive.

Hello, im currently an intern and was given a task to automate a hadoop cluster. It should auto start the component once its down and alert the user (email or similar) that the node was autostarted. How would i go about doing this? Thanks in advance!

Hello everyone,   I’m newer to Hive so I’m assuming that I’m doing something wrong. I’ve configured Hive to require LDAP BasedDN authentication (not AD) and configured TLS for HiveServer2.   With beeline I’m able to authenticate with an interactive user prompt. For example: beeline> !connect jdbc:hive2://<serverName>:10000/default;ssl=true;sslTrustStore=<trustStorePath>.jks;trustStorePassword=<trustStorePassword> Login with username and password: Enter username for jdbc:hive2://…Enter password for jdbc:hive2://… What I would like to do is connect with username/password in the connection string. I see example to configure the connection string with TLS or username/password, but haven’t been able find an example of both. I’ve attempted combining settings and I’m doing something wrong.   This is an example of what I've attempted: beeline -u jdbc:hive2://<serverName>:10000/default;ssl=true;sslTrustStore=<truststorePath>.jks;trustStorePassword

HI All,I have deployed zeppelin in my Ubuntu instance (My private IP: 20.20.20.213) using the tgz file. I have changed the zeppelin-en.sh.template to zeppelin-en.sh and zeppelin-site.xml.template to zeppelin-site.xml. Also, I have configured my port in the site-xml file as 8090 and the server addrs as 0.0.0.0 default only. On checking the log file i get the following code:I even tried to change the server addrs to 20.20.30.213 but still the same error I am getting. Please help!ubuntu@ip-20-20-30-213:~/zeppelin/logs$ cat zeppelin-root-ip-20-20-30-213.log INFO [2019-06-12 17:24:41,675] ({main} ZeppelinConfiguration.java[create]:121)                                                                              - Load configuration from file:/home/ubuntu/zeppelin/conf/zeppelin-site.xml INF

I run a sqoop import command  sqoop import --connect jdbc:oracle:thin:.... --username ....  --password ...--query "SELECT $CHOOSE_COLUMN_NAMES FROM $OTABLE where $OWHERE and \$CONDITIONS"  ... through unix shell. If the command takes a long time  how can i kill through shell ? Thanks in advance. 

I am trying to install components using a python script. Based loosely upon the community example I found here:https://github.com/datagulp/hdp_utilities/blob/master/autoScale.shAt the step where I get a list of components from another compute node, then try to replicate, I can add each component onto a node with a POST successfully. I can see the components being added. But when I try to then do a PUT to set the state of each one to INSTALLED, it fails. curl -u user1 -i -H "X-Requested-By:ambari" -X PUT -d '{"HostRoles": {"state":"INSTALLED"}}' http://my-ambari-host:8080/api/v1/clusters/my-cluster/hosts/my-host/host_components/ATLAS_CLIENT{ "status" : 400, "message" : "java.lang.IllegalArgumentException: Missing KDC administrator credentials.\nThe KDC administrator credentials must be set as a persisted or temporary credential resource.This may be done by issuing a POST to the /api/v1/clusters/:clusterName/credentials/kdc.admin.credential API entry point with the following payload:\n