Q&A

I reduced the replication factor of my HDFS data from 3 to 2, but I noticed that the total data size and storage space are still not decreasing as expected

Hello, I am experimenting with LDAP integration, which I managed to make it work in Hue. Now, I switched the 'backend' property back to 'desktop.auth.backend.AllowFirstUserDjango.Backend', restarted Hue and I can login with the local user as before LDAP. However, on the User Admin page I still see "Add/Sync LDAP user" and not the local "Add user". Which other option prevents the local functionality from showing up? Thank you,Gerasimos

We have kafka cluster with 3 kafka brokers nodes and 3 zookperes serverskafka version - 10.1 ( hortonworks )from my understanding since all meta data is located on the zookeeper servers , and kafka brokers are using this data ( kafka talk with zookeeper server via port 2181 ) I just wondering if each kafka machine talk with other kafka in the cluster , or maybe kafka are get/put the data only on/from the zookeepers servers ?So dose kafka service need to communicate with other kafka in the cluster ? , Or maybe kafka machines get all is need only from the zookeepers server ?

I am using the "Hortonworks Data Platform (HDP®) 3.0.1 on Hortonworks Sandbox" using Docker in Hyper-V on Windows 10 1803 box. Pleas note have lots of disk space. memory, and cores on this machine. I have scan the forums and web so far nothing has worked. Here is the output from df -h:[root@sandbox-hdp ~]# df -hFilesystem Size Used Avail Use% Mounted onoverlay 59G 55G 1.4G 98% /tmpfs 64M 0 64M 0% /dev/dev/sda1 59G 55G 1.4G 98% /etc/hostsshm 64M 4.0K 64M 1% /dev/shmtmpfs 9.7G 355M 9.4G 4% /runtmpfs 2.0G 0 2.0G 0% /run/user/1003tmpfs 2.0G 0 2.0G 0% /run/user/0tmpfs 2.0G 0 2.0G 0% /run/user/1004tmpfs 2.0G 0 2.0G 0% /run/user/1011tmpfs 2.0G 0 2.0G 0% /run/user/1009tmpfs 2.0G 0 2.0G 0% /run/user/1019tmpfs 2.0G 0 2.0G 0% /run/user/1002tmpfs 2.0G 0 2.0G 0% /run/user/1021tmpfs 2.0G 0 2.0G 0% /run/user/1001tmpfs 2.

Hi All,I have a stringString:some text with an ip 111.111.111.111 and a decimal 11.2323232 and some text here and then an int 1 and then some HTTP/1.1 with a 503 request and then another ip 222.222.222.222 and some imaginary 999.999.999.999I want to output all the ip addresses in comma saperated. I tried the belowselect regexp_replace(regexp_replace(String,'[^(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})]',' '),'\\s+',',');+------------------------------------------------------------------------+--+| _c0 |+------------------------------------------------------------------------+--+| ,111.111.111.111,11.2323232,1,1.1,503,222.222.222.222,999.999.999.999 |+------------------------------------------------------------------------+--+Expected output is : 111.111.111.111,222.222.222.222,999.999.999Could you please help me

I am trying to follow the steps at https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_getting-started-with-stream-analytics/content/ch_prepare-your-environment.htmlby creating a new kafka topic, then adding a new schema in the SchemaRegistry. However, when I try to add a new schema (following the instructions on the page) I get the following error when I click on Save in Schema Registry:An exception was thrown while processing request with message: [SchemaMetadata not found with the schemaName: raw-truck_events_avro]I am a newbie with this tool - so I don't know where to look next to see what is causing this error. Your help is greatly appreciated!

I am trying to install HDP 3.0 version on a 6 node cluster where 5 nodes act as ambari-agent and 1 node as server.Here is the log file:stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms_server.py", line 137, in <module> KmsServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms_server.py", line 51, in install kms.setup_kms_db() File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms.py", line 68, in setup_kms_db copy_jdbc_connector(kms_home) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms.py", line 359, in copy_jdbc_connector Please run 'ambari-server setup --jdbc-db={db_name} --jdbc-driver={path_to_jdbc} on server host.'".format(params.db_flavo

I want to be able to tag a list of Hive columns in Atlas (HDP 2.6.5) from a Python script. For this I need to find the guid of the Hive column I'm targetting. But I can't seem to tame the Atlas REST API to do that.Getting all the Hive columns in the data lake is easy enough:curl -u myaccount -i -H "Content-Type: application/json" -X GET 'https://atlasnode.domain.com:21000/api/atlas/discovery/search/dsl?query=hive_column+where+__state=%27ACTIVE%27+select+qualifiedName,name,__guid'That works (the number of Hive columns is limited to 100 by default):{ "requestId": "pool-2-thread-6 - 7e8ec2ee-a94b-4e37-b851-57b1b3df6e29", "query": "hive_column where __state='ACTIVE' select qualifiedName,name,__guid", "queryType": "dsl", "count": 100, "results": [ { "$typeName$": "__tempQueryResultStruct5", "qualifiedName": "mytestdb.employee.name@CLUSTERNAME", "__guid": "809c84d5-e065-45f4-81c8-fcdb7cf81560", "name": "name" }, { "$typeName$": "__tempQueryResultStruct5

When I execute quires that return large results in the phoenix shell (sqlline.py), I get the following error:org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 33164, already closed?Which will probably be resolved by the solution posted here. However, I do not have permission to edit the hbase-site.xml so I need to pass the settings as command line arguments when starting the phoenix shell. Previously we have used the `--conf hbase.rpc.timeout=xxx` syntax on other jobs, but I can't find a way to pass that into the phoenix shell. So my question is: Can I pass those settings as arguments when starting the sqlline.py or can I set them while already inside?

I have a table that I'm querying with an array<String> column and queries are extremely slow when accessing the complex type.  The table is partitioned and Impala is doing a broadcast cross join on the "array" component and scanning ALL partitions, wherein the base part of the query, it's pruning the partitions based on the partition key and only scanning the partition I'm querying on.     Here is the query:   select rta.transaction_purchase_id, rta.cigarette_transaction_flag, rta.non_cig_merch_transaction_flag, bow.itemfrom wdl_atomic.retail_transaction_attribute rta, wdl_atomic.retail_transaction_attribute.retail_offering_material_group_distinct_list bowwhere rta.fiscal_period_id = 2019001;   This is the explain plan:   Explain String Max Per-Host Resource Reservation: Memory=0B Per-Host Resource Estimates: Memory=44.61GB PLAN-ROOT SINK | 04:EXCHANGE [UNPARTITIONED] | 02:NESTED LOOP JOIN [CROSS JOIN, BROADCAST] | |--03:EXCHANGE [BROADCAST] |

oozie-error.jpgHi,I installed hdp 2.6.5 cluster using two machines. It's not a kerberized environment. I'm using Chrome as a browser.When Opening Oozie UI i get the following message :Oozie web console is disabled.To enable Oozie web console install the Ext JS library.How can I enable the UI? Many thanks in advance.Regards,

Assistance required to replace double quotes with single quotes in Nifi Replace Text processor, Please assist.Input: "162","298","99","1.5","3","7.46","0","0.53","1"Output: '162','98','99','1.5','3','7.46','0','0.53','1'Post 'Replace Text' i will pass on the values to Putsql processor which inserts data into Oracle, where double quotes for column value is not accepted.

Windows 10 / VMware Workstation Pro 15:Uninstalled HDP 2.6.5 which was working in Bridged mode. Installed HDP 3.0.1 as per https://hortonworks.com/tutorial/sandbox-deployment-and-install-guide/section/2/No errors; unable to access splash page on http://192.168.1.9:1080 but can access SSH terminal using putty on 192.168.1.9:2222Tried adding Host-only network adapter, no joy!Tried reinstalling VMWare, no joy!With my limited experience I can see no other services using 1080 or 8080 ports.Please help

Can anyone please share with me the Ambari installation documentation on Ubuntu 16.04?

Dear All, I am using this sqoop command for importing Oracle Table into Hive Table sqoop import --connect jdbc:oracle:thin:@123.123.123.112:1500:COL --username admin -P --table db.table1 --hive-import --hive-overwrite --create-hive-table --hive-database hive_db --hive-table table1 --split-by primary_Key After Importing from oracle to Hive Some rows and columns are entirely Showing 'NULL' . ORACLE TABLE HIVE TABLE -ISSUE 1. PRIMARY_KEY (NOT NULL COLUMN)2. There is 0 'NULL' Value 1. SOME ROWS AND COLUMNS ARE NULL2. 35 NULL (ROWS and COLUMS) How to clear it?

step 1 get rsa key from umbari host # cat /root/.ssh/id_rsa step 2 Amabri web UI choose Hosts...Action Add Host. enter ambari.example.com in Target Hosts Field Paste the key from step 1 into where it asks for 'ssh prv key' step 3 click button Register and confirm.... returns Failed ==========================Creating target directory...==========================Command start time 2019-02-05 14:23:08root@ambari.example.com: Permission denied (publickey,password). SSH command execution finishedhost=ambari.example.com, exitcode=255 Command end time 2019-02-05 14:23:08ERROR: Bootstrap of host ambari.example.com fails because previous action finished with non-zero exit code (255) ERROR MESSAGE: root@ambari.example.com: Permission denied (publickey,password).STDOUT: root@ambari.example.com: Permission denied (publickey,password). OK

I found it necessary to troubleshoot a problem enabling kerberos to modify the ambari server and cluster boxes /etc/hostname from name.example.com to name.corp.example.comI also edited the /etc/hosts on all to reflect accordingly.I also ran ambari-server setup-security and option 1 to DISABLE the https.Enter choice, (1-5): 1 Do you want to disable HTTPS [y/n] (n)? y Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership... NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start")restarted the ambari service #ambari-server restart|Now I get the login page on port 8- butif I try to login as admin the screen just freezes and mouse cursor becomes circle with line through it.Any suggestions welcome to get back to admin able to log in.

I have a NiFi job starting with a GetFile processor configured to run on primary node to avoid duplicate flowfiles. After a NiFi node restart, causing a Primary node re-election, the GetFile processor has created two flowfiles (one expected and one duplicate). I suppose that the GetFile was configured to run with the pre-election primary node and post-election one too, causing those two flowfiles (both flowfiles were processed by different nodes according to NiFi data provenance). Is there a way to avoid this behavior, and is it a NiFi bug ? NB: While the job was running, coordinator and primary nodes were the same. Thanks a lot.

The support matrix states PostgreSQL 10.2 or 9.6 are supported..... Is this a hard and fast rule or can we use 11.1 or 10.2 ???We are getting ready to stand up a new cluster and I wanted to stay on a fairly current version.

Hi,  is it supported to submit a new cluster template to Cloudera Altus Director and use an existing Cloudera Manager installation?  I had a Director template for Cluster A and CM A and it was deployed sucessfully. As the Director server was not needed any more, it was deleted (so the operations is managed by CM, but no changes were made in cluster config). Now I would like to create Cluster B also with Director, but to point to an existing Cloudera Manager A (and thus using the same KDC)    Is this supported?   Thanks    

Dear all!   I have kerberized cluster on Cloudera Express 5.9.3, Java Version: 1.7.0_67. Since I turned on HDFS Data At Rest Encryption using Java Keystore KMS I recieve an error in Pig job    Failed to renew token: Kind: kms-dt, Service: 10.6.1.44:16000, Ident: (kms-dt owner=hdfs, renewer=yarn, realUser=, issueDate=1547451323413, maxDate=1548056123413, sequenceNumber=2, masterKeyId=2)   Yarn log:   Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect Caused by: java.security.UnrecoverableKeyException: Password verification failed   I have checked the pessword by doing keytool -list, the password is correct.   Please, help me to resolve this problem.  

Hello,I have enable kerberos on cluster. I can successfully connect to consumer through kafka-konsole-consumer.But when i try to connect to topic through NiFi ConsumeKafka processor i have error:WARN [Timer-Driven Process Thread-8] o.a.n.p.kafka.pubsub.ConsumeKafka_1_0 ConsumeKafka_1_0[id=504e5811-0168-1000-0000-000024c83cc5] Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@51911c24. Will roll back session and discard any partially received data. WARN [kafka-kerberos-refresh-thread-nifi/*@*] o.a.k.c.security.kerberos.KerberosLogin [Principal=nifi/*@*]: TGT renewal thread has been interrupted and will exit.

hi all we are trying to delete service from ambari and we get 500 ERRORwhat is needed in order to fix this issue ?we have ambari cluster with HDP version - 2.6.4 and ambatri - 2.6.1

Is there a way to let Atlas discover existing Hive tables (in HDP 2.6.5)? I've got a couple of Hive external tables that existed before Atlas was enabled. It didn't pick them up, but Atlas does find new tables.I'm trying to get one Hive table in Atlas, but it's not exactly easy to do that for tables, columns, storagedesc and references. There are very few (simple) examples out there. I'm getting one ObjectId is not valid AtlasObjectId error after the other.

Greetings, We are using Cloudera Director, with databases stored on AWS RDS.   Director was using the IP address of the RDS database, as opposed to the hostname. I recently enabled Multi-AZ on the RDS database, and now it has multiple IP addresses.   How can I update the config for the external database in Director so that new clusters are created against the RDS database's hostname instead of the IP address?I tried the unsupported database update mechanism via the director API at `PUT /api/v12/environments/{environment}/databaseServers/{externalDatabaseServer}` (filled in obviously) with a bunch of different json body contents. However, director just responds with a `UnsupportedExternalDatabaseServerUpdateException` regardless of the format I give.  I've tried enabling debug logging to see if there is additional information in application.log, but I can't seem to find the good stuff.   I tried opening director's database, but the tables con