Q&A

while installing HDP 3.1 Cluster at stage 9 Install Start & Test.. showing below error   Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-374.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-374.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']2019-09-23 16:54:26,245 - Reporting component version failed ================================================= DataNode Installstderr: /var/lib/ambari-agent/data/errors-423.txt Command aborted. Reason: 'Server considered task failed and automatically aborted it'stdout: /var/lib/ambari-agent/data/output-423.txt Command aborted. Reason: 'Server considered task failed and automatically aborted it' Command failed after 1 tries

Good Afternoon,   My organization is having trouble setting up an ODBC on Redhat and I think it might have something to do with the SSL configuration.   The error we receive is: "FATAL ERROR : Signal Received: SIGSEGV (11)" but when our administrator looked into the trace it seemed to point to SSL.   Is there a way to configure trust store password? I think this might be the item we are missing from our driver configuration. Settings below for a normal JDBC string we use in other applications and our current driver settings.   JDBC:  jdbc:hive2://{host}:{port}/;ssl=true;sslTrustStore={certs.jks};trustStorePassword=changeit   Driver: [HIVE]Description=Cloudera ODBC Driver for Apache Hive (64-bit) DSNDriver=/opt/cloudera/hiveodbc/lib/64/libclouderahiveodbc64.soHiveServerType=2Host=[host]Port=[post]AuthMech=3ThriftTransport=1UID=[username] PWD=[password]TrustedCerts=[cert.pem]KrbHostFQDN=_HOSTKrbServiceName=hiveKrbRealm=[ORG]SSL=1   Thank you! Aaron

Hello Team,   We are planning to use Sentry for implementing authorization for Impala Tables. In prerequisites for installing Sentry last point suggest to implement Kerberos authentication but point number 6 mentions that either Kerberos or LDAP is required. This is a bit confusing, can you help in clarifying if it is necessary to implement kerberos authentication for Configuring Sentry.     Regards Parth   

  Hello, I´m trying using sqoop, to import / export but I have the following Error... It is on a Sandbox Hortonworks 2.2.4.2. could you help me please ? Thank You -       [root@sandbox results]# sqoop import --connect jdbc:mysql://127.0.0.1/sqoop --username root --table result --target-dir /user/results10/Warning: /usr/hdp/current/accumulo-client does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.19/09/25 15:06:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.4.2-219/09/25 15:06:55 INFO manager.SqlManager: Using default fetchSize of 100019/09/25 15:06:55 INFO tool.CodeGenTool: Beginning code generationSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.

Hello everyone. When cloudera manager installs the agent on the nodes, I receive the heartbeat error. I look the log an I see this: >>[04/Sep/2019 12:53:55 +0000] 744 Monitor-HostMonitor throttling_logger ERROR ntpq: ntpq -np: not synchronized to any server All host can see the others, the Cloudera Server host has ntp configured, and if I execute ntpq -np on the client where I try to install the agent, it runs and ntp is working.   Could anyone help me?   Thanks

Hi,I'm trying to understand a little more about what these options do behind the scenes when pressed?  I found this post, but it never got any responses (http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/What-is-under-the-hood-of-Hue-Impala-editor-Clear-cache/m-p/59814#M2432)Clear CacheThis is the one I am not too sure what happensPerform Incremental metadata UpdateI assume this issues a refresh command for all tables within the current database which is been viewed?  If no database is veiwed does it do it for everything?Invalidate All metadata and rebuild indexSame as above, but rather than refresh it invalidates full metadata?  If no database is veiwed does it do it for everything?

Attempting to install HDP 3.1.0 via Ambari 2.7.3 using an existing postgresql DB after following the docs here (https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/configuring_postgresql_for_ranger.html) and entering the commands below:   ---------- dbname=hive postgres=postgres user=hive passwd=hive echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $postgres superuser;" | sudo -u $postgres psql -U postgres     dbname=oozie postgres=postgres user=oozie passwd=oozie echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter u

I am getting the below error while trying to start hbase 2.2.0 shell.I have downloaded the `hbase-2.2.0 binary tar.gz` and extracted into a Windows x64 environment 16GB RAM.As the below error message suggests, it is arising out of `Line 81` of `%HBASE_HOME%\bin\hirb.rb` due to the `org.apache.log4j.Level` class being not found I set the HBase environment variables in `%HBASE_HOME%\conf\hbase-env.cmd` as well as added all the `%HBASE_HOME%\lib` jars into `HBASE_CLASSPATH` (as shown below) but getting the same error > %HBASE_HOME%\conf\hbase-env.cmd     set JAVA_HOME=c:\Program Files\Java\jdk1.8.0_101set HBASE_CLASSPATH=%HBASE_HOME%\lib;%HBASE_HOME%\lib\client-facing-thirdparty;%HBASE_HOME%\lib\ruby;%HBASE_HOME%\lib\zkcli;%HBASE_HOME%\lib\shaded-clientsset HBASE_HEAPSIZE=8000set HBASE_OPTS="-XX:+UseConcMarkSweepGC" "-Djava.net.preferIPv4Stack=true"set SERVER_GC_OPTS="-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCDateStamps" %HBASE_GC_OPTS%set CLIENT_GC_OPTS="-verbose:gc" "

I would like to drop multiple partitions of an impala table at once.Can I use the following statement  in order to delete all the partitions of January ? ALTER TABLE schema.Table  DROP IF EXISTS PARTITION (part_col='201801*') PURGE ; Can I use Star Wildcard  just as unix?

i installed hbase while running the hbase its shows below error     ./start-hbase.shJava HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.running master, logging to /usr/local/Hbase/bin/../logs/hbase-root-master-gaian-lap386.com.outJava HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option PermSize; support was removed in 8.0Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0  

Hi, I have a CSV file where the last column has pipe-delimited text which needs to be split up on pipe and I have to pivot it into rows. The last column can have any number of values with pipe-delimited and it varies by each row. How can we accomplish this in NIFI? Ex:  A, B,  C|DX, Y,  X|Y|Z|B The above file has to be converted to  A, B, CA, B, DX, Y, XX, Y, YX, Y, ZX, Y, B

    Dear friends and colleges   we little worry about the future of HDP versions for now the latest HDP is 3.1 but what next ?      

Zip command is not working in Execute Stream Command but working in Execute Command. Please let me know what is wrong with the below. Command Arguments  -r -j /opt/nifi/nifi-1.9.2/data/print/CIF_206.zip /opt/nifi/nifi-1.9.2/data/print/CIF_206/ Command Path    zip Ingore STDIN  true

Hi, I have a csv file with dynamic columns and headers and there are multiple columns with same header name "SKIP"  that needs to be removed from the file before ingesting them into the database using PutDatabaseRecord. How can I delete the multiple columns with header name "SKIP'?

We have an end-user who runs queries usually with Impala/HUE.  He is receiving this error, which I assume will require some changes in tuning parameters.  But where do we start to investigate here?  The error message is: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed: Failed to get minimum memory reservation of 68.00 MB on daemon chia1.haas.berkeley.edu:22000 for query e74ef8d9b9215369:4994cbde00000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage: Process: Limit=97.35 GB Total=80.62 GB Peak=85.72 GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=4.24 GB Buffer Pool: Unused Reservation: Total=-4.24 GB Free Disk IO Buffers: Total=1.41 GB Peak=1.99 GB RequestPool=root.hue: Total=78.60 GB Peak=82.56 GB Query(2f4b5cff11212907:886aa1400000000): Reservation=77.88 GB ReservationLimit=7

Hi, I am in need of SQL Server lookup, found this library https://github.com/mrcsparker/nifi-sqllookup-services-bundle    but there are no good examples especially when dealing with incoming CSV text file.  Appreciate if anybody can point me to an article or sample workflow that does something similar. Thanks in advance

I have installed cdsw trail version before purchasing the product but stuck with errors. 500 gig block device  and using default /var/lib/ for application.  CDSW disks provisioned via ESX/VMware.  CDSW starts fine in CM but failed to launch containers. Here is the status.    Failed events are below for web-5dddb59597-pclf2  (redacted host name for security reasons). Not sure why tcp-ingress-controller-7d65df6d8f-qqxzn  is starting , i am not using TLS for CDSW Any help is highly appreciated.    NAME READY STATUS RESTARTS AGEcdsw-compute-pod-evaluator-566f7664df-rqm2j 1/1 Running 0 9m38scron-6dd99c7b78-kh4rw 1/1 Running 0 9m40sdb-86bbb69b54-nxz4z 1/1 Running 0 9m40sdb-migrate-46715e4-nf5dh 0/1 Completed 0 9m40sds-cdh-client-578fc69b9-hcqv7 1/1 Running 0 9m38sds-operator-66ff9c4b78-r6bk7 1/1 Running 0 9m38sds-reconciler-64bccdd574-vhhpz 0/1 CrashLoopBackOff 4 9m38sds-vfs-648f9bddf-7n6hg 1/1 Running 0 9m38simage-puller-xwbcs 1/1 Running 0

I just started using cm_client python for swagger v32. I wonder if I missed something because I found some simple task seemed quite hard to accomplish as a new API user.   For example, I am trying to use the API to set a Yarn configuration mapreduce.job.split.metainfo.maxsize. In CM GUI I can go to YARN -   Configuration - search for such a setting and then update it, quite simple. However, in order to do it in API, I was not sure which "update_config" I should use because there are many of them. I tried "serviceResourceAPi.update_service_config( )" for yarn service but the specific setting was not there. then I thought I should try update_role_config or update_role_config_group. Then I realized I had to inpect the many "read_config" output manually to find out which API may be relavent to the parameter. I could not do it programly because I was not sure  how the  setting/parameter would be  called in API - maybe  mapreduce.job.split.metainfo.maxsize

How to resolve this issue. please tell me step by step process   File /var/log/ambari-server/ambari-server.log: Spoiler (Highlight to read) 2019-09-15 05:21:14,273 WARN [C3P0PooledConnectionPoolManager[identityToken->1hgeumca5eo2un51maxtev|d554c5f]-HelperThread-#0] BasicResourcePool:220 - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@a21c74 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.2019-09-15 05:21:14,298 ERROR [main] AmbariServer:1114 - Failed to run the Ambari Serverjavax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!Error Code: 0at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:815)at or

Hi, folks, You can tell me if it's possible to use Ambari with Two Active Directories.   For example   dc01.test.comdc02.test.com   With the Ambari server, there is some way to use this option in High availability.   Greetings

Is there a way to store intermediate query results in Beeline - Hivesomething like below in pseudo code. within a beeline sessionbeeline -u  -e" set var1 = select max(date) from tableA; set var2 = select count(emps) from tableB;set var3 = select min(date) from tableC;Insert into table TableD select var1, var2, var2" --...

Below article shows how to restart clients on Specific host. https://cwiki.apache.org/confluence/display/AMBARI/Restarting+host+components+via+the+APIIs there any way we can perform the same on all the hosts in the cluster at one time.Thank you in advance.

I am running spark 2.4 on Azure HDI version 4.0, which includes HDP 3.0.After the upgrade, we are unable to use spark-hbase connector version 1.1.1-2.1-s_2.11, it fails with below errorjava.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; at org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257) at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:80) at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) at withCatalog(<console>:42) ... 56 elidedAs per

The non-interactive setup of Ambari seems to be broken. It looks like the /etc/ambari-server/conf/ambari.properties file is not updated when using the ambari-server setup CLI flags (non-interactive setup).   Has anyone ever had the same issue?   Environment CentOS Linux release 7.6.1810 (Core) http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0 repo for ambari-server Steps to reproduce sudo yum install -y postgresql-jdbc ambari-serversudo ambari-server setup --help Using python /usr/bin/pythonSetup ambari-serverUsage: ambari-server.py action [options]Options:-h, --help show this help message and exit-v, --verbose Print verbose status messages-s, --silent Silently accepts default prompt values. For db-cleanupcommand, silent mode will stop ambari server.Database options (command need to include all options):--database=DBMS Database to useembedded|oracle|mysql|mssql|postgres|sqlanywhere--databasehost=DATABASE_HOSTHostname of database server--

Hello Team, All of sudden service monitor was went down and when i checked the logs i can see connection refused error in logs.I have tried to restart all the CMS services and this time all went down.What i was did is i have deleted CMS and trying to re-add the CMS to cloudera manager,This time i am not able to add the CMS and when i am trying to adding it active monitor and reports manager cant able to find databases and DB's in test connection stage.Can some one please respond ASAP and it would be greatfull. Thanks,Vinod