Q&A

Hi all,I have to import a mysql table into hive environment.The table has a "notes" column with free text, including the possibility for the user to insert CRLF. when I try to sqoop import that table, it seems the sqoop command interprets this CRLF as a end of row.... Then everything, from that point on, is not correctly imported: I mean that the CRLF is the end of the row and the first column of the next row in filled in with the "rest" of free text and so on I use this command: sqoop import --connect jdbc:mysql://myserver:3306/mydb --username myusername --password mypwd--table mytable --hive-import --hive-table mytable --hive-database mydb --as-textfile--warehouse-dir /user/mydb --fields-terminated-by '|' --direct -m 1 --driver com.mysql.jdbc.Driver

Hello all,We are installing 5 nodes HDF cluster using local repository and facing below issue .its seems look like not pointing to right local repo. Can someone please help me .Thanks in advance .-VickyCannot find a valid baseurl for repo: HDF-3.3-repo-152Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 37, in <module> BeforeInstallHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 33, in hook install_packages() File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/shared_initialization.py", line 37, in install_packages retry_count=params.agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_managemen

Hi All,I need help . I am in the process of installing HDF 3.3.1 cluster and the Ambari version is 2.7.3 My Installation fails while configuring the Ranger with Mysql DBshell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Error occurred during initialization of VM java.lang.Error: Properties init: Could not determine current working directory. at java.lang.System.initProperties(Native Method) at java.lang.System.initializeSystemClass(System.java:1119)Thanks in advance.-Vicky

I am looking for new auto start function in which I wan't to start kafka when it fails or stale for sometime and it gets started after some interval. Currently, auto start is giving functionality to start service or reboot which I don't wan't. In my case kafka service get stop's after midnight and there is no one to start it hence it affects the business.any suggestions? I have already tried shell script but due to security issues I couldn't deploy shell scripts.

If it is possible, Could you please share steps that how I can do it.Thanks.Versions:Sqoop1.4.7 NiFi1.8.0

Hi ,While connect power bi to cloudera db getting 1107 error . Can I have the solution.

Hello,  I'm trying to run a bash script located at a HDFS folder via oozie workflow. This script has a lot o configurations to be done by the user, and they are passed to the script via a file the script has to read and is put at the same HDFS folder as the script. The script cannot read it, though, and I wonder if I should comply with some specific path configurations for this to work on HDFS.  When this code here is reached:  if [ ! -e `pwd`/vars.sh ] ; then echo "Unable to continue: file vars.sh doesn't exist!" exit 1fithe script exits. Is the "pwd" a problem? I know that it pressuposes that the oozie workflow would respect the path of execution of my script. Is this a mistake? Thanks! 

Hi,  Executing "hdfs dfs -ls /", we are getting aKrbException: Fail to create credential. (63) - No service credseven with a valid kerberos ticket on a particular Kerberos realm. We've been looking into the source code of the JVM and we think we might find the cause with DEBUG level. According to http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html, there is a loglevel option to the hdfs command. So we launch it like this: hdfs dfs -ls / --loglevel DEBUG But that seems to have no effect (only WARN is getting printed). How can we control the loglevel? Thanks in advance

Hello,We are using Cloudbreak rc-2.9 with LDAP integration, and I'd like to know if it's possible to 1) create workspace for other users in automated way2) assigning these users to Workspace (without them logging in yet)3) setting default workspace for userI wasn't able to found anything like that in API docs, maybe some hacking through SQL against CB repository should help?Thanks!

I have requirement to modify the email subject from Ambari for certain alerts. For example, anything related to HDFS, I need to have "CRITICAL HDFS" in the subject line. I went through this link : https://community.hortonworks.com/questions/198815/ambari-email-alert-format.htmlbut it does not talk about customizing the subject for certain alerts.Has anyone has this setup in your environment ?

We have tried several attempts to get the hortonworks platform to work with kerberos using the ambari automated install. the errors are below. any advice to offer is appreciated. Tried to install manually the krb5-user but seems I am getting broken package errors, not sure of that was a good approach afer the automated failed.2019-01-28 22:35:23,105 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}2019-01-28 22:35:23,142 - Skipping installation of existing package unzip2019-01-28 22:35:23,143 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}2019-01-28 22:35:23,177 - Skipping installation of existing package curl2019-01-28 22:35:23,177 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}2019-01-28 22:35:23,211 - Skipping installation of existing package hdp-select2019-01-28 22:35:23,214 - The repository with version 3.1.0.0-78 for this command has been marked as resolved. It will be used to report the v

I've installed CDH-5.16.1-1.cdh5.16.1.p0.3 on SLES 12.3 in "single user mode". I have some services runningcloucera-scm-servercloudera-scm-agentCloudera Management Service (Alert Publisher, Event Server, Host Monitor, Service Monitor)ZookeeperWhen I try to start HDFS, Secondary Name Node and Data Node seems ok, but Name Node fails with this error:  2019-01-25 17:21:52,016 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.apache.hadoop.ipc.Server.start(Server.java:2696) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.start(NameNodeRpcServer.java:448) at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:713) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692) at org.

Hello, I am trying to totally remove Kerberos and re-enable it. I have followed all the rollback steps as described in several posts: https://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos/td-p/19654https://stackoverflow.com/questions/29744821/how-to-disable-hadoop-kerberoshttp://bigdata-tips.blogspot.com/2017/03/how-to-disable-kerberos-in-cloudera-cdh.html I seems that kerberos is indeed disabled BUT I cannot get the Enable Kerberos button back:  What should I do to re-enable it? Thank you,Gerasimos 

I want to create a table with the complex type removed from the avro data in the same schema. This is because Impala does not skipping complex types. Platform is CDH 6.0.1 For Example : Employee(raw data) - name : string - age : int - additional-info : map<string, string>Employee(Hive table 1) - name : string - age : int - additional-info : map<string, string>Employee_For_Implala(Hive table 2) - name : string - age : int Pipeline : KafkaProducer(Avro Bytes) - Kafka - Flume - HDFS - Hive(Impala) Flume : KafkaSource - Channel - Sink(AvroEventSerializer$Builder) I tried changing the sink(serializer.schemaURL, remove Complex type field) but it failed. I am trying to use morphine now. But this is also failing. Is there a better way?

hi allwe have 3 kafka brokers ( HDP version - 2.6.0.1 and ambari verdion - 2.6.0 ) we cant start them from ambari , and we saw the following errors on ambari-agent logsthe error about - Fail: Configuration parameter 'kafka-env' was not found in configurations dictionary!what chould be the problem here ?Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file imp.load_source('__main__', script) File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py", line 141, in <module> KafkaBroker().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py", line 128, in status import status_params File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/status_params.py", li

Hello Team,After enabling Kerberos on HDP-2.6 cluster, Atlas metadata server is not starting. Getting following error:java exceptionERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 503 Error: HTTP 503 at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1236) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7857) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1999) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1981) at org.apache.hadoop.hbase.protobuf.generated.ClientPro

Hello Team,After enabling kerberos on HDP-2.6, Hive metadata server is failing. Getting following error in hive metastore log:2019-01-25 03:58:28,880 ERROR [pool-7-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849) at org.apac

Hi, how do i get last Friday in  Impala? Thx

I am interested in learning to use Atlas but am having trouble installing it. I have posted a question on stackoverflow:https://stackoverflow.com/questions/54296453/getting-errors-while-installing-apache-atlas-standalone-on-windows-10but it doesn't look like the Atlas community is active there. Thank you for any help.

When upgrading Ambari from 2.7.1 to 2.7.3, I get these messages with no mention of failure or success:Preparing to unpack .../ambari-server_2.7.3.0-139_amd64.deb ...Backing up Ambari properties: /etc/ambari-server/conf/ambari.properties -> /etc/ambari-server/conf/ambari.properties.rpmsaveBacking up Ambari properties: /var/lib/ambari-server/ambari-env.sh -> /var/lib/ambari-server/ambari-env.sh.rpmsaveBacking up JAAS login file: /etc/ambari-server/conf/krb5JAASLogin.conf -> /etc/ambari-server/conf/krb5JAASLogin.conf.rpmsaveBacking up stacks directory: /var/lib/ambari-server/resources/stacks -> /var/lib/ambari-server/resources/stacks_25_01_19_17_08.oldBacking up common-services directory: /var/lib/ambari-server/resources/common-services -> /var/lib/ambari-server/resources/common-services_25_01_19_17_08.oldBacking up mpacks directory: /var/lib/ambari-server/resources/mpacks -> /var/lib/ambari-server/resources/mpacks_25_01_19_17_08.oldBacking up Ambari view jars: /var/lib/

hi allI want to validate the balanced of the topics , include replica and IsrI will give some example of wrong balanced ( master01 is the zookeper server , and we defined 3 replicate )on that example we can see that kafka brokers are missing from some Isr/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper master01:2181 --topic mno.de.pola.trump Topic:mno.de.pola.trump PartitionCount:100 ReplicationFactor:3 Configs: Topic:mno.de.pola.trump Partition: 0 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 Topic:mno.de.pola.trump Partition: 1 Leader: 1018 Replicas: 1018,1016,1017 Isr: 1018,1017,1016 Topic:mno.de.pola.trump Partition: 2 Leader: 1016 Replicas: 1016,1017,1018 Isr: 1016,1018 Topic:mno.de.pola.trump Partition: 3 Leader: 1017 Replicas: 1017,1016,1018 Isr: 1017,1018,1016 Topic:mno.de.pola.trump Partition: 4 Leader: 1018 Replicas: 1018,1017,1016 Isr: 1018,1017,1016 Topic:mno.de.pola.trump Partition: 5 Leader: 1016

I am trying s simple workflow ExecuteSQL -> PutSFTP, I want my flowfiles to be with .avro extension as well as I want to add either a prefix or suffix to the file name. For example if I am extracting employee data then I want flowfile name should be employee<timestamp/randomnumber>.avro@Shu @Matt Burgess @Neeraj Sabharwal

Hi All, If user connected Namenode through Gatewaynode, suddenly namenode goesdown, after few seconds standbynode became active node. After that standby namenode became active namenode,he can able to access without getting downtime after connecting to namenode which ip address he can able to see?

hi all we run the following command on kafka machine in order to view/print the topics names we can see that its fails onCaused by: java.net.BindException: Address already in use but without to print also the port number./kafka-topics.sh --zookeeper zookeeper1:2181 --list To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files>where num_of_file > 0GC log rotation is turned offException in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPrem

 when I run the command service cloudera-scm-server startShows failed Urgent help needed  as I have stuck with issue since last 2 days org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)        at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:392)        at com.cloudera.server.cmf.Main.<init>(Main.java:242)        at com.cloudera.server.cmf.Main.main(Main.java:216)Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory