Solved

I'm trying to create a Ranger policy for Atlas in a kerberized cluster using HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) but receive a 401 (unable to create policy) notification.   I'm trying to figure out which user to use following the "Learning the Ropes of the HDP Sandbox" but I can't make sense of it.   - "raj_ops", "maria_dev", "amy_ds" give the error stated above - "holger_gov" cannot login to Ranger UI - "admin" (following the "Admin password reset" procedure) can login to Ambari but not to Ranger UI   Any guidance would be greatly appreciated.

I am learning Hadoop. I want to know can i deploy more than one edge node in my cluster? If yes, how to do this and on which hosts should i keep the edge nodes? Currently i have one edge node in my cluster on a dedicated machine.  

hi all     we have 5 kafka brokers    we can stop or start the kafka service by API  but how to stop specific kafka broker by REST API?   for example , in this pic we can see 5 kafka brokers , and we want to stop/start only the second kafka broker by REST API    

When I connect to 'http://xxx.xxx.xxx:7180/cmf/login' I can see the login screen but any user/password combination that I use fails.   I've tried to change the "users" table on the "scm" database to "ffa2eb4251b38e069e968890cb2bcdb6229982322f5ed2470bf96231fe4c39c8" and -4382599614486590865 to try to reset the passwd of admin but nothing works.   cloudera-scm-server.log adds a new line for each login try: "INFO scm-web-114:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: 'admin' from 111.222.333.111"   If I try some random user the error is the same (minus the user).   Could Kerberos or Sentry have anything to do with this?   Thank you Gonçalo Redondo      

Hi, I am trying to fetch all files with .txt extension from a list of files present in s3 bucket using nifi. Is there any way to fetch the file based on the format of file and what processors we will use here. Can any one explain me with an example as i am new to this.     Thanks  Sunil  

hello, i have a server intel® Xeon(R) CPU E5-2630 v3 @ 2.40GHz × 32 with 1007gb memory and 20tb hardisck i have a docker in this server, i created a 25 container cloudera (quickstart)  18 container of 25 is running the others stop it and i will create more than 50 container but now i can't how can solve this problem plz if you can help me in this issue 

Hi, this has been troubling for months.. I simply can't find any CANVAS ID (Processor or ControllerService)  on the exported XML template 😞   Example - on my canvas I see : Name PutS3Object Id 0c6637cc-bea1-1c43-a023-921b68e400f5 Type PutS3Object 1.0.0   Now i created template for the PG, and theres no 0c6637cc-bea1-1c43-a023-921b68e400f5 on the xml ?!   This is very annoying, why i can't see any IDs on canvas on the exported xml ?   We have cluster of 3 nodes and PG is versioned on NiFi registry.   Any suggestions in how to locate on XML template any Processor or ControllerService using their ID as visible on canvas editor ?   Thanks, Emanuel O.

I'm using PublishKafkaRecord processor. Everything works fine, but if I stop the processor and multiple flowfiles stuck in queue when I run the processor again if any of files is invalid all files get into failed queue. Is there any batch setting or any other way to solve this?

I am trying to read a csv file and put those records into a postgres table. To do it, I have used some tips given Loading a simple CSV file using nifi.   My template is : GetFile -> UpdateAttribute -> PutDatabaseRecord   In UpdateAttribute I have used `schema.name` -> `postgres database`   In PutDatabaseRecord ->   CSVReader   AvroSchemaRegistry   DBCPConnectionPool   My CSV file ``` userid,username 1,'sachith' ```   When I put this to my source folder, it goes to PutDatabaseRecord and gives following error   > None of the fields in the record map to columns defined by the nifi.nifiTest table.   First I used MySQL just with GetFile -> PutDatabaseRecord, it was working properly without error.   What am I missing here? I am pretty new for Nifi.

Hi Support Team,   We are planning to setup cloudera on one of our cloud environment.We have RHEL 7.7 installed on our servers. Does currently Cloudera support RHEL 7.7 version ??? or any possibility in near future to support latest RHEL version.   i also want to know what happened if we deploy current cloudera versions on RHEL 7.7.   Thanks,      

Hi, I tried to insert some data in postgresql db using PUTSQL processor. Below example of sql statement in putSql properties:   INSERT INTO myTable(date_response, success, code, payload_response) values ('${now()}', '${status:toNumber()}', '${code:toNumber()}', ${payload});    The problem is when this insert is executed, i have this error: org.postgresql.util.PSQLException: ERROR: column "zxlbavuzvmlvm2x6vedsemrdstzjrnnnzxlbavuzvmlvm2x6swpvz2v5qwlhv1f" does not exist. What happen ? .... the payload_response column name is replace by the value of my flow file attribute ${payload}. Why ? any hints ?

Getting error when inserting a csv-file into a MSSQL database using the PutDatabaseRecord (1.9.2) processor in Nifi.ListFile → FetchFile → PutDatabaseRecord     The schema used:     { "type" : "record", "name" : "mibas_t005_talkgroup", "fields" : [ { "name" : "talkgroup_ref", "type" : ["null","long"] },{ "name" : "fleet_ref", "type" : ["null","long"] },{ "name" : "talkgroup_id", "type" : ["null","long"] },{ "name" : "talkgroup_name", "type" : ["null","string"] },{ "name" : "group_type", "type" : ["null","string"] },{ "name" : "dispatcher_ref", "type" : ["null","string"] },{ "name" : "multigroup_ref", "type" : ["null","string"] },{ "name" : "owner_", "type" : ["null","string"] },{ "name" : "fne_state", "type" : ["null","string"] },{ "name" : "addl_attrvar", "type" : ["null","string"] },{ "name" : "last_user_modified", "type" : ["null","string"] },{ "name" : "time_stamp", "type" : ["null", { "type":"long", "logicaltype":"timestamp-millis"} ]},{ "name" : "fne_oi

I have installed the HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) using Docker and Kerberized the cluster.    I can list the principles from the keytab file like this:   klist -kte /etc/security/keytabs/atlas.service.keytab  Which gives the following output:   Keytab name: FILE:/etc/security/keytabs/atlas.service.keytabKVNO Timestamp Principal---- ------------------- ------------------------------------------------------ 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (des-cbc-md5) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (des3-cbc-sha1) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (arcfour-hmac) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (aes256-cts-hmac-sha1-96) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (aes128-cts-hmac-sha1-96)     Please forgive my stupidity, but how can I now login to Atlas UI?   I've tr

hi all   we want to identify kafka alert from ambari by rest api   Example when we get alert     what is the AMBARI REST API in order to verify alert on kafka broker01 or kafka broker02 , etc 

Hey, folks,I know how to remove and add a server what I don't know how I can see the data replicating in the new node. someone could help me. Greetings

Hi team,Iam creating materialized view for a table ,for that getting below error.can any one help on this? create materialized view car1_mv as select * from cars;Error: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for materialized view cannot be enabled if the materialized view uses non-transactional tables (state=42000,code=40000)

we are runs spark application on hadoop cluster ( HDP version - 2.6.5 from hortonworks )   from the logs we can see the following Diagnostics   User: airflowApplication Type: SPARKUser class threw exception: org.apache.hadoop.security.AccessControlException: Permission denied. user=airflow is not the owner of inode=alapati not clearly what we need to search in `HDFS` in order to find why we get Permission denied

Hi folks,   We have been struggling for the past 3 weeks trying to secure a cluster. We have 3 nodes. They were working ok in a cluster (but not secured). When trying to secure those (and following the guide suggested here), we came across the following message when trying to access the UI:   javax.net.ssl.SSLPeerUnverifiedException: Hostname <my_fqdn_is_here> not verified: certificate: sha256/716mOuXyoAKqzNrXrNnG2ozHXwN8WWJsVxzWzfQzpNV= DN: CN=xxx-xxxx-xxxx.xxx.xxx.net, OU=XXXXXXXXX XXXXXX, O=XXXXX, L=XXXXXX, ST=XXXXXX XXXXX, C=CA subjectAltNames: []    We've been going over and over the configs and nothing seems to point to the right direction.Would anyone point us to at least the right direction? 

Hello I'm new in Big data and I'm trying to create a cluster, but I have an issue with the registration of hosts, I saw a lot of posts about this problem but the problem is persist (You find below the Registration Log for the ambari server):   ========================== Creating target directory... ========================== Command start time 2019-11-18 10:23:23 Connection to ambari.server closed. SSH command execution finished host=ambari.server, exitcode=0 Command end time 2019-11-18 10:23:23 ========================== Copying ambari sudo script... ========================== Command start time 2019-11-18 10:23:23 scp /var/lib/ambari-server/ambari-sudo.sh host=ambari.server, exitcode=0 Command end time 2019-11-18 10:23:24 ========================== Copying common functions script... ========================== Command start time 2019-11-18 10:23:24 scp /usr/lib/python2.6/site-packages/ambari_commons host=ambari.server, exitcode=0 Command end time 2019-11-

#/usr/jdk64/jdk1.8.0_112/bin/java -cp/usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java-5.1.37.jarorg.apache.ambari.server.DBConnectionVerification "jdbc:mysql://xxxxxxx:3306/ambari" "ambari" "xxxxx" com.mysql.jdbc.DriverERROR: Unable to connect to the DB. Please check DB connection properties.com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communicationslink failureThe last packet sent successfully to the server was 0 milliseconds ago. Thedriver has not received any packets from the server.#ambari-server setup --jdbc-db=mysql--jdbc-driver=/usr/share/java/mysql-connector-java-5.1.37.jarUsing python /usr/bin/pythonSetup ambari-serverCopying /usr/share/java/mysql-connector-java-5.1.37.jar to/var/lib/ambari-server/resources/mysql-connector-java-5.1.37.jarCreating symlink/var/lib/ambari-server/resources/mysql-connector-java-5.1.37.jar to/var/lib/ambari-server/resources/mysql-connector-java.jarIf you are updating existing jdbc driver jar for mys

I'm attempting to enable authentication on my NiFi server that is in a cluster with several other hosts. If I enabled HTTPS for webUI authentication, without changing any of the other hosts on the cluster, will this cause any complications with their communication with one another?

Hello everyone,, When we assign Operate the Component to a process group, it allows users to start/stop/disable process group, but it doesn't allow the user to start/stop the Processors inside the Process group. How can we enable Operate the Component so that the users can start/stop the Processors as opposed to the all the processors inside the Process Group? Steps Taken:1. Go inside the process Group2. Click on Access Policy icon3. Click "Operate the component" in the drop down4. If needed, click either add or override link at the top of the page5. Assign a "User Group" to this policy Testing1. Login as a user in the User Group2. Navigate to Process Group3. You will be able to start or stop4. Click on Processor5. User unable to start or stop (Problem} 

Hi,   My Ambari server is offline (ambari.properties file is empty/virgin - don't ask 😞 ) but my cluster is functional, that is, HDFS is fine and so on. My Question is this, can I rebuild my Ambari server and "import" the running HDP cluster or do I have to start from scratch?   Thanks,   john

Hello All, We have applications in our data organization like Oracle Data Integrator and Jupyter Hub. Those set of applications require up to date jars, core-site.xml, hdfs-site xml etc to integrate properly with the cluster. Our Application Architects suggests we should install those applications' clients and/or agents on edge nodes of our cluster. I don't want to do this because: *I might need to deal with conflicting environment requirements. *I will have limited control over server resource consumption *Application level operations such as upgrades might have adverse affects on the cluster I am considering to offer mounting jars/conf directories of edge nodes on read only mode to application servers. Any suggestions and evaluations about both architectures or alternatives are more than welcome. Best regards

sudo apt-get install libmysql-javaReading package lists... DoneBuilding dependency treeReading state information... Donelibmysql-java is already the newest version (5.1.45-1).0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded.     /usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://XXXXXX:3306/ambari" "ambari" "password" com.mysql.jdbc.DriverERROR: Unable to connect to the DB. Please check DB connection properties.java.lang.ClassNotFoundException: com.mysql.jdbc.Driver