Solved

Due to Amabri "DB configs consistency check failed" .. I have deleted all the service configs of HBASE from ambari DB as suggested in one of the articles https://community.hortonworks.com/questions/69240/ambar-server-start-failes-with-db-configs-consiste.html but after that .. I am able to start ambari server without errors but unable to see hbase service in Amabri UI

Hi, I need some help with JOLT-syntax.JSON-Input{  "nummer": "22",  "id": "177",  "table": [    {      "zn": 1,      "stfflbisart": 10    },    {      "zn": 2,      "stfflbisart": 50    }  ]}JOLT-Specification[{    "operation": "shift",    "spec": {      "id": "ID",      "nummer": "Nummer",      "table": {        "*": {          "zn": "ArtikelPreise_Pos.[#2].ZeileNr",          "stfflbisart": "ArtikelPreise_Pos.[#2].StaffelBis"        }      }    }}, {    "operation": "default",    "spec": {      "Default_Kopf": "${VAR_KD}"    }    }]JSON-Output{  "ID" : "177",  "Nummer" : "22",  "ArtikelPreise_Pos" : [ {    "Zeil

Hi All,I'm trying to upgrade Ambari server and agents from 2.7.1 to 2.7.3 using below link.https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade/content/upgrade_ambari.html But it is failing in "yum upgrade ambari-server". I have tried cleaning up repository & cache but still facing same issue.    [root@host ~]# yum upgrade ambari-server    Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager    No packages marked for update[root@host conf]# yum repolist | grep ambariambari-2.7.3.0                                      ambari Version - amba     13[root@host conf]#Below is content of ambari.repo[root@host yum.repos.d]# cat ambari.repo#VERSION_NUMBER=2.7.3.0-139[ambari-2.7.3.0]#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.jsonname=ambari Version - ambari-2.7.3.0baseurl=http://public-

2019-06-27 03:39:42,688 - Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 546, in <module> CheckHost().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 207, in actionexecute raise Fail(error_message) resource_management.core.exceptions.Fail: Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties. com.mysql.jdbc.exception

hi allcan you upgrade HDP 2.6.4 to 3.1 and stay with ambari 2.6.1?

is it safe to remove the /tmp/hive/hive folder? ( from hdfs )ashdfs dfs -rm -r /tmp/hive/hivethe reason for that because under /tmp/hive/hive we have thousand of files and we cant delete therm hdfs dfs -ls /tmp/hive/Found 7 itemsdrwx------   - admin     hdfs          0 2019-03-05 12:00 /tmp/hive/admindrwx------   - drt   hdfs          0 2019-06-16 14:02 /tmp/hive/drtdrwx------   - ambari-qa hdfs          0 2019-06-16 15:11 /tmp/hive/ambari-qadrwx------   - anonymous hdfs          0 2019-06-16 08:57 /tmp/hive/anonymousdrwx------   - hdfs      hdfs          0 2019-06-13 08:42 /tmp/hive/hdfsdrwx------   - hive      hdfs          0 2019-06-13 10:58 /tmp/hive/hivedrwx------   - root      hdfs          0 2018-07-17 23:37

Hi there! I'm try to deploy Apache Ranger ranger-tagsync module without Apache Atlas.I'm enable option "TAG_SOURCE_FILE_ENABLED = true" but can't understand the way of creating needed tags.json file.In my plan, my own deamon will simply generate this file for tagging hive-columns giving tag-based policies opportunity masking this columns for usergroups.Mostly i'am follow next article:https://community.hortonworks.com/articles/135895/how-to-mask-columns-in-hive-with-atlas-and-ranger.htmlbut it's a problem to create working file-prototype.I'am need worked examples of this file or Atlas REST output with same actions.Thank you.

I am new to hortonworks . I have created a shell script with the Sqoop import command . I want to execute the .sh script to run the sqoop import command .I have tried the following :1. I have uploaded the .sh script in a folder /home/scripts/sqoop_test.sh2. The permission : -rwxr-xr-x 1 root root 215 Jun 25 18:44 sqoop_test.sh3. Then I tried to run the file : ./sqoop_test.sh4. I am getting the error : Permission denied: user=root, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-xMy query is :1. Do I need to upload the .sh script in a folder within HDFS2. How do I setup the permission for the script so that it runs successfully3. How to execute .sh file from within hdfs so that I do not get the permission denied errorThanks

Hi,I have NiFi 1.7.1 running as a service on Ubuntu 18.04.The service is running under root.The instance is set up to gather modbus data on a network using a custom processor, and send the data via S2S to another instance on a separate machine. The instance is also gathering other data such as nifi resources and sending them via S2S as well.My issue is that for some reason, when there is no user logged in to ubuntu, the flow of modbus data stops. Either via the ubuntu GUI or SSH. when I log back in, the flow starts back up. I cannot check if the service is running directly as I would need to log in for that, but the non modbus data continues to flow via S2S so I think the service is running.There doesn't seem to be anything obvious in the nifi-app.log, or bootstrap logs.Has anybody encountered unusual behaviour when the OS is not logged in?Thank you

Hello, I've set a cluster with CDH 6.20, 1 master 4 slaves, with the parcels way. Everything is working quite good, except the connection between Hue and S3.I've followed the instructions from this guide The hosts are running on Ubuntu 18.04, Python version is 2.7.15rc1S When I try to navigate my buckets in Hue filebrowser, I get this error in the GUI : "Unknown error occurred". On the Hue server logs, here is the complete error :Internal Server Error: /filebrowser/view=S3A:/Traceback (most recent call last):File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/exception.py", line 41, in innerresponse = get_response(request)File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/base.py", line 249, in _legacy_get_responseresponse = self._get_response(request)File "/opt/cloudera/parce

Hi , we are in the process of creating HDP 2.6 cluster where in RHEL OS will be integrated with AD for authentication. We will using AD as the KDC. My question if we create a local UNIX user called HIVEUSER and use any BI tool to connect to HIVE using this user, will the local user be able to get authenticated and access Hive tables in kerberized cluster?or the HIVEUSER should be in AD?

Hi,We are getting below warnings while starting hive shell from CLI.$hive19/06/20 07:53:14 WARN conf.HiveConf: HiveConf of name hive.mapred.strict does not exist19/06/20 07:53:14 WARN conf.HiveConf: HiveConf of name hive.mapred.supports.subdirectories does not existlog4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppenderhive>Hive version - 1.2As per I know, we can safely ignore these warnings but, we need to get rid off these.Could you please help on this. Thanks

Hi. On an Oracle BDA system where several network interfaces exist on each host, how can I bind Sentry to wildcard address 0.0.0.0 ? I've already tried by adding:<property>  <name>sentry.service.server.rpcaddress</name>  <value>0.0.0.0</value></property>to Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml but it's stil not working (I've double checked that generated sentry-site.xml file contains the property). Any help is appreciated. Thanks in advance. 

Facing this issue with the namenode down. It is a parcel based CDH 5.15.2 installation on AWS.   9:15:18.262 PM ERROR NameNode Failed to start namenode.java.io.FileNotFoundException: File does not exist: /user/spark/spark2ApplicationHistory/.5e9b4c52-032a-4469-b278-80aa6254cfdf at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:903) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:756) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:324) at org.apac

Hi,I have a simple question about querying complex types.  The schema, raw table and desired result is listed below.  Could someone help me craft this query in Impala? CDH 5.15 I can get it to work in Hive, but it seems like Impala uses a join-like syntax to access complex columns.  Many Thanks!!schema: snum int items map<string, float>raw parquet table: 1, {"a": 1.3, "b", 9.1} 2, {"c", 84.9, "b", 1.1} 3, {} 4, {"x", 42.5,"p", 0.08}desired result: 1,a,1.3 1,b,9.1 2,c,84.9 2,b,1.1 3,null,null 4,x,42.5 4,p,0.08  

Can anyone tell me the way to install JanusGraph on Ambari. I tried the link https://github.com/JanusGraph/janusgraph-ambari but it didn't work out for me. I have Ubuntu 14.04 and Ambari 2.6 and have already installed Solr 6.6.2 on Ambari. Is there a way to make a connection between the two? Or, is there any other alternative to this (JanusGraph) which might run on Ambari with the same functionality?Actually JanusGraph installed properly on Ambari using the link above. It also starts without any error, but then within 10 seconds it automatically closes. During the final check it says "can't connect to port 8182". What can be the fault in this case?

Hi!I need to execute following awk command on flow file via ExecuteStreamCommand processorawk -F"," '{OFS=","; $4=substr($4,1,19); print $0}'When run it throws an error: Executable command awk ended in an error: awk: fatal: cannot open file `print $0}" for reading (No such file or directory).Flow file comes from ListFTP -> FetchFTP -> ExecuteStreamCommand Could you please help to solve this issue!!Thank you!!When running the same awk command on server it works fine:awk -F"," '{OFS=","; $4=substr($4,1,19); print $0}' myfile.csv

I wrote a java code to insert few thousand records by creating batch, I was not getting good performance from PreparedStatements so I switched to Statements and see if there is some performance difference.However after getting error and reading Impala doc I came to know executebatch() is not supported for statements but I am able to use addBatch() so its kind a strange.Here I am posting some of my code-  private static Connection connectViaDS() throws Exception { Connection connection = null; Class.forName("com.cloudera.impala.jdbc41.DataSource"); connection = DriverManager.getConnection(CONNECTION_URL); return connection; }//inside main--// I have declared all the variables,I have lots of other data as well so it may create more confusion try { connection = connectViaDS(); connection.setAutoCommit(false); Statement statement = connection.createStatement(); Timestamp ed = new Timestamp(e_date);

Hi Experts,   Need your suggestions:   How to enforce a user to submit a job in dedicated resource pool if nothing is mentioned during submission.   Whenever a user submits a  job (without mentioning any queue name) , it should go in a dedicated queue defined by Admins.   Can we control it through YARN ACLs.

Hi, When trying to import table from oracle database to hive from hue qui i get the following error when i am trying to setup the jdbc connection: Connection Failed: Error accessing the database: An error occurred while calling z:java.lang.Class.forName. : java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver at java.net.URLClassLoader$1.run(URLClassLoader.java:360) at java.net.URLClassLoader$1.run(URLClassLoader.java:349) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:348) at java.lang.ClassLoader.loadClass(ClassLoader.java:430) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:323) at java.lang.ClassLoader.loadClass(ClassLoader.java:363) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMet

Hi there, colleaguesPast several days i tried to implement integration apache ranger and ranger-kafka plugin.My versions:kafka: 2.12_2.2.0ranger: 1.2.0 , ranger-plugin 1.2.0My current problem:as far as i understand, kafka cant find the ranger-created audit files that i made with enable-kafka-plugin.sh script (ranger-kafka-audit.xml and ranger-kafka-security.xml). I sold this files everywhere in file system but this class still cant find it and cause fatal error:[2019-06-06 13:22:44,584] INFO getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding /usr/lib/kafka/libs/ranger-kafka-plugin-impl/guava-17.0.jar (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)[2019-06-06 13:22:44,584] INFO getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding /usr/lib/kafka/libs/ranger-kafka-plugin-impl/ranger-plugins-common-1.2.0.jar (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)[2019-06-06 13:22:44,615] INFO [Transaction Marker

so i have a flume agent  with a spooldir channel that got too full (too many files error). the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again. but now i have some flume-formatted data that i would like to ingest.can i move the files into the spooldir 10% at a time fx?would i have to read the files and add them to the flume source?  additional notes:the reason for the spool dir was a missing sink (another flume agent- turned off)

Hello all,My system administrator wants to check the Impala Query's execution time.I can check the each fragment's total time in[Cloudera Manager > Impala > Query > Query details].But I can't find the total of query execution time. Please anyone tell me where can I check it. Thanks

I am working on a NIFI flow that begins with an InvokeHTTP processor.On the occasion that there is a connection timeout (Failure condition), I wish to assign the InvokeHTTP processor Remote URL value to a new property and have this property passed as data (attribute) through the "Failure" path to another processor. Is it possible to do within the same InvokeHTTP processor?Can anyone help?

My nifi flow is not progressing nor showing any error:Flow is for :fetch a test.json fiel from s3 and populate its content in a table in postgresql db.lists3object-> fetchs3object->JoltTrasformationJSON -> ........ ->PutSQL.The lists3object alone lists key as my test.json in "view status"I'm not able to paste screenshots here.Though all processors got started, none of them are doing any transactions (in & out ) both are 0 bytes and i dont see any errors also.Last message logged in nifi-app.log is:2019-06-14 07:26:48,864 INFO [NiFi Web Server-92] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.aws.s3.ListS32019-06-14 07:26:48,864 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] to run2019-06-14 07:26:49,166 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowCon