Solved

Need to know how to differentiate headless and service keytab.. any specific commands

 if there is postgresql install then also can i install MySQL 2. If i want to add Service Nifi & Spark in existing ambari server then which i have to go first.

Hi, We are using CDH 5.14. Webhdfs is enabled by default on cloudera manger and i am able to list HDFS data through webhdfs protocol.   [root@hadoop-slave-1 ~]# curl -i -L "http://172.16.0.10:50070/webhdfs/v1/?op=LISTSTATUS"HTTP/1.1 200 OKCache-Control: no-cacheExpires: Fri, 27 Sep 2019 13:52:56 GMTDate: Fri, 27 Sep 2019 13:52:56 GMTPragma: no-cacheExpires: Fri, 27 Sep 2019 13:52:56 GMTDate: Fri, 27 Sep 2019 13:52:56 GMTPragma: no-cacheContent-Type: application/jsonX-FRAME-OPTIONS: SAMEORIGINTransfer-Encoding: chunked {"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":33381,"group":"supergroup","length":0,"modificationTime":1569326622576,"owner":"hdfs","pathSuffix":"cli","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":10,"fileId":17365,"group":"supergroup","length":0,"modificationTime":1569589022824,"owner":"hdfs","pathSuffix":"hadoop","permission":"755","replication":0,"storage

Hi Team,   I am using cloudera manager.In this i am trying to give an access for particular services to the new users. Like i am a new user and i needed only spark and pig so once i added in cloudera as a user so i can read and write for these two services only not all the services.   For that i have tried to sentry option but it is not full fill my requirement. Can you give me or provide me any solution for this.   Thanks, Vikas,.

Couldn't connect to the cluster { "status" : 404, "message" : "The requested resource doesn't exist: Cluster not found, clusterName=

Where would someone increase the database connection pool for Director? Would this be in application.properties or in /etc/default/cloudera-director-server?   I've noticed after tuning the amount of threads for tasks and the amount of instances in parallel in the application.properties that if I try to provision 4 clusters of 100 nodes each at around the same time or with a small delay that Director has issues with not being able to get database connections causing failures.   To pre-empt this I have migrated to a serverless aurora with a minimum of 8 units which from the cloudwatch charts shows no spikes in connections yet Director still fails pipeline errors with the following:     java.util.concurrent.ExecutionException: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection   I guess what I'm try

Recently upgraded Director from 6.1 to 6.2.1 and noticed a lot more spam in the logs with the following trace:   [2019-07-22 17:17:27.632 +0000] ERROR [task-thread-10] - - - - - com.cloudera.api.ext.ClouderaManagerClientProxy: API call to Cloudera Manager failed, gathering informationjavax.ws.rs.NotFoundException: HTTP 404 Not Found at sun.reflect.GeneratedConstructorAccessor246.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:544) at org.apache.cxf.jaxrs.client.ClientProxyImpl.checkResponse(ClientProxyImpl.java:326) at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:891) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:797) at org.apache.cxf.jaxrs.client.ClientProxy

Hey, folks, I have an error when executing kafka when executing a message for the topic. kafka-console-producer.sh --broker-list w01.s03.hortonweb.com:6667 -topic TestNYC error. WARN [Producer clientId=console-producer] Bootstrap broker w01.s03.hortonweb.com:6667 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) I have 3 Brokers, which are working and is configured according to the parameters. just a topic that I just realized. I have 2 network cards one internal and external to netstat I see that port 6667 is listening to the internal. but still executing the command with the internal IP kafka-console-producer.sh --broker-list 192.168.0.9:6667 -topic TestNYC I still get the same mistake. Someone could help me. Greetings  

We have a Cluster running CDH 5.15.1 on 150+ nodes. Cluster nodes storage capacity varies (Mix of various node sizes, 10 TB, 13 TB, 25 TB, 48 TB).  Nodes are connected with a 10G network and presently running with a physical topology of Rack awareness.  Issue is, while few racks have 18 servers others are just having 1 or 2 servers on them. we have been advised to move away from this nonuniform rack awareness topology to a Logical Rack awareness topology which will help us put together UNIFORM no. of nodes on each rack.   I would like to understand, if the rack awareness topology is changed, would HDFS kick-off immediate Block movement or the new topology will only be effective for NEW blocks written on the cluster. 

while installing HDP 3.1 Cluster at stage 9 Install Start & Test.. showing below error   Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-374.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-374.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']2019-09-23 16:54:26,245 - Reporting component version failed ================================================= DataNode Installstderr: /var/lib/ambari-agent/data/errors-423.txt Command aborted. Reason: 'Server considered task failed and automatically aborted it'stdout: /var/lib/ambari-agent/data/output-423.txt Command aborted. Reason: 'Server considered task failed and automatically aborted it' Command failed after 1 tries

Good Afternoon,   My organization is having trouble setting up an ODBC on Redhat and I think it might have something to do with the SSL configuration.   The error we receive is: "FATAL ERROR : Signal Received: SIGSEGV (11)" but when our administrator looked into the trace it seemed to point to SSL.   Is there a way to configure trust store password? I think this might be the item we are missing from our driver configuration. Settings below for a normal JDBC string we use in other applications and our current driver settings.   JDBC:  jdbc:hive2://{host}:{port}/;ssl=true;sslTrustStore={certs.jks};trustStorePassword=changeit   Driver: [HIVE]Description=Cloudera ODBC Driver for Apache Hive (64-bit) DSNDriver=/opt/cloudera/hiveodbc/lib/64/libclouderahiveodbc64.soHiveServerType=2Host=[host]Port=[post]AuthMech=3ThriftTransport=1UID=[username] PWD=[password]TrustedCerts=[cert.pem]KrbHostFQDN=_HOSTKrbServiceName=hiveKrbRealm=[ORG]SSL=1   Thank you! Aaron

Hello Team,   We are planning to use Sentry for implementing authorization for Impala Tables. In prerequisites for installing Sentry last point suggest to implement Kerberos authentication but point number 6 mentions that either Kerberos or LDAP is required. This is a bit confusing, can you help in clarifying if it is necessary to implement kerberos authentication for Configuring Sentry.     Regards Parth   

  Hello, I´m trying using sqoop, to import / export but I have the following Error... It is on a Sandbox Hortonworks 2.2.4.2. could you help me please ? Thank You -       [root@sandbox results]# sqoop import --connect jdbc:mysql://127.0.0.1/sqoop --username root --table result --target-dir /user/results10/Warning: /usr/hdp/current/accumulo-client does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.19/09/25 15:06:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.4.2-219/09/25 15:06:55 INFO manager.SqlManager: Using default fetchSize of 100019/09/25 15:06:55 INFO tool.CodeGenTool: Beginning code generationSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.

Hello everyone. When cloudera manager installs the agent on the nodes, I receive the heartbeat error. I look the log an I see this: >>[04/Sep/2019 12:53:55 +0000] 744 Monitor-HostMonitor throttling_logger ERROR ntpq: ntpq -np: not synchronized to any server All host can see the others, the Cloudera Server host has ntp configured, and if I execute ntpq -np on the client where I try to install the agent, it runs and ntp is working.   Could anyone help me?   Thanks

Hi,I'm trying to understand a little more about what these options do behind the scenes when pressed?  I found this post, but it never got any responses (http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/What-is-under-the-hood-of-Hue-Impala-editor-Clear-cache/m-p/59814#M2432)Clear CacheThis is the one I am not too sure what happensPerform Incremental metadata UpdateI assume this issues a refresh command for all tables within the current database which is been viewed?  If no database is veiwed does it do it for everything?Invalidate All metadata and rebuild indexSame as above, but rather than refresh it invalidates full metadata?  If no database is veiwed does it do it for everything?

Attempting to install HDP 3.1.0 via Ambari 2.7.3 using an existing postgresql DB after following the docs here (https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/configuring_postgresql_for_ranger.html) and entering the commands below:   ---------- dbname=hive postgres=postgres user=hive passwd=hive echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter user $postgres superuser;" | sudo -u $postgres psql -U postgres     dbname=oozie postgres=postgres user=oozie passwd=oozie echo "CREATE DATABASE $dbname;" | sudo -u $postgres psql -U postgres echo "CREATE USER $user WITH PASSWORD '$passwd';" | sudo -u $postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | sudo -u $postgres psql -U postgres echo "alter u

I am getting the below error while trying to start hbase 2.2.0 shell.I have downloaded the `hbase-2.2.0 binary tar.gz` and extracted into a Windows x64 environment 16GB RAM.As the below error message suggests, it is arising out of `Line 81` of `%HBASE_HOME%\bin\hirb.rb` due to the `org.apache.log4j.Level` class being not found I set the HBase environment variables in `%HBASE_HOME%\conf\hbase-env.cmd` as well as added all the `%HBASE_HOME%\lib` jars into `HBASE_CLASSPATH` (as shown below) but getting the same error > %HBASE_HOME%\conf\hbase-env.cmd     set JAVA_HOME=c:\Program Files\Java\jdk1.8.0_101set HBASE_CLASSPATH=%HBASE_HOME%\lib;%HBASE_HOME%\lib\client-facing-thirdparty;%HBASE_HOME%\lib\ruby;%HBASE_HOME%\lib\zkcli;%HBASE_HOME%\lib\shaded-clientsset HBASE_HEAPSIZE=8000set HBASE_OPTS="-XX:+UseConcMarkSweepGC" "-Djava.net.preferIPv4Stack=true"set SERVER_GC_OPTS="-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCDateStamps" %HBASE_GC_OPTS%set CLIENT_GC_OPTS="-verbose:gc" "

I would like to drop multiple partitions of an impala table at once.Can I use the following statement  in order to delete all the partitions of January ? ALTER TABLE schema.Table  DROP IF EXISTS PARTITION (part_col='201801*') PURGE ; Can I use Star Wildcard  just as unix?

i installed hbase while running the hbase its shows below error     ./start-hbase.shJava HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.running master, logging to /usr/local/Hbase/bin/../logs/hbase-root-master-gaian-lap386.com.outJava HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option PermSize; support was removed in 8.0Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0  

Hi, I have a CSV file where the last column has pipe-delimited text which needs to be split up on pipe and I have to pivot it into rows. The last column can have any number of values with pipe-delimited and it varies by each row. How can we accomplish this in NIFI? Ex:  A, B,  C|DX, Y,  X|Y|Z|B The above file has to be converted to  A, B, CA, B, DX, Y, XX, Y, YX, Y, ZX, Y, B

    Dear friends and colleges   we little worry about the future of HDP versions for now the latest HDP is 3.1 but what next ?      

Zip command is not working in Execute Stream Command but working in Execute Command. Please let me know what is wrong with the below. Command Arguments  -r -j /opt/nifi/nifi-1.9.2/data/print/CIF_206.zip /opt/nifi/nifi-1.9.2/data/print/CIF_206/ Command Path    zip Ingore STDIN  true

Hi, I have a csv file with dynamic columns and headers and there are multiple columns with same header name "SKIP"  that needs to be removed from the file before ingesting them into the database using PutDatabaseRecord. How can I delete the multiple columns with header name "SKIP'?

We have an end-user who runs queries usually with Impala/HUE.  He is receiving this error, which I assume will require some changes in tuning parameters.  But where do we start to investigate here?  The error message is: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed: Failed to get minimum memory reservation of 68.00 MB on daemon chia1.haas.berkeley.edu:22000 for query e74ef8d9b9215369:4994cbde00000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage: Process: Limit=97.35 GB Total=80.62 GB Peak=85.72 GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=4.24 GB Buffer Pool: Unused Reservation: Total=-4.24 GB Free Disk IO Buffers: Total=1.41 GB Peak=1.99 GB RequestPool=root.hue: Total=78.60 GB Peak=82.56 GB Query(2f4b5cff11212907:886aa1400000000): Reservation=77.88 GB ReservationLimit=7

Hi, I am in need of SQL Server lookup, found this library https://github.com/mrcsparker/nifi-sqllookup-services-bundle    but there are no good examples especially when dealing with incoming CSV text file.  Appreciate if anybody can point me to an article or sample workflow that does something similar. Thanks in advance