Solved

I want to create a table with the complex type removed from the avro data in the same schema. This is because Impala does not skipping complex types. Platform is CDH 6.0.1 For Example : Employee(raw data) - name : string - age : int - additional-info : map<string, string>Employee(Hive table 1) - name : string - age : int - additional-info : map<string, string>Employee_For_Implala(Hive table 2) - name : string - age : int Pipeline : KafkaProducer(Avro Bytes) - Kafka - Flume - HDFS - Hive(Impala) Flume : KafkaSource - Channel - Sink(AvroEventSerializer$Builder) I tried changing the sink(serializer.schemaURL, remove Complex type field) but it failed. I am trying to use morphine now. But this is also failing. Is there a better way?

hi allwe have 3 kafka brokers ( HDP version - 2.6.0.1 and ambari verdion - 2.6.0 ) we cant start them from ambari , and we saw the following errors on ambari-agent logsthe error about - Fail: Configuration parameter 'kafka-env' was not found in configurations dictionary!what chould be the problem here ?Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file imp.load_source('__main__', script) File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py", line 141, in <module> KafkaBroker().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py", line 128, in status import status_params File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/status_params.py", li

Hello Team,After enabling Kerberos on HDP-2.6 cluster, Atlas metadata server is not starting. Getting following error:java exceptionERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 503 Error: HTTP 503 at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1236) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7857) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1999) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1981) at org.apache.hadoop.hbase.protobuf.generated.ClientPro

Hello Team,After enabling kerberos on HDP-2.6, Hive metadata server is failing. Getting following error in hive metastore log:2019-01-25 03:58:28,880 ERROR [pool-7-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849) at org.apac

Hi, how do i get last Friday in  Impala? Thx

I am interested in learning to use Atlas but am having trouble installing it. I have posted a question on stackoverflow:https://stackoverflow.com/questions/54296453/getting-errors-while-installing-apache-atlas-standalone-on-windows-10but it doesn't look like the Atlas community is active there. Thank you for any help.

When upgrading Ambari from 2.7.1 to 2.7.3, I get these messages with no mention of failure or success:Preparing to unpack .../ambari-server_2.7.3.0-139_amd64.deb ...Backing up Ambari properties: /etc/ambari-server/conf/ambari.properties -> /etc/ambari-server/conf/ambari.properties.rpmsaveBacking up Ambari properties: /var/lib/ambari-server/ambari-env.sh -> /var/lib/ambari-server/ambari-env.sh.rpmsaveBacking up JAAS login file: /etc/ambari-server/conf/krb5JAASLogin.conf -> /etc/ambari-server/conf/krb5JAASLogin.conf.rpmsaveBacking up stacks directory: /var/lib/ambari-server/resources/stacks -> /var/lib/ambari-server/resources/stacks_25_01_19_17_08.oldBacking up common-services directory: /var/lib/ambari-server/resources/common-services -> /var/lib/ambari-server/resources/common-services_25_01_19_17_08.oldBacking up mpacks directory: /var/lib/ambari-server/resources/mpacks -> /var/lib/ambari-server/resources/mpacks_25_01_19_17_08.oldBacking up Ambari view jars: /var/lib/

hi allI want to validate the balanced of the topics , include replica and IsrI will give some example of wrong balanced ( master01 is the zookeper server , and we defined 3 replicate )on that example we can see that kafka brokers are missing from some Isr/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper master01:2181 --topic mno.de.pola.trump Topic:mno.de.pola.trump PartitionCount:100 ReplicationFactor:3 Configs: Topic:mno.de.pola.trump Partition: 0 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 Topic:mno.de.pola.trump Partition: 1 Leader: 1018 Replicas: 1018,1016,1017 Isr: 1018,1017,1016 Topic:mno.de.pola.trump Partition: 2 Leader: 1016 Replicas: 1016,1017,1018 Isr: 1016,1018 Topic:mno.de.pola.trump Partition: 3 Leader: 1017 Replicas: 1017,1016,1018 Isr: 1017,1018,1016 Topic:mno.de.pola.trump Partition: 4 Leader: 1018 Replicas: 1018,1017,1016 Isr: 1018,1017,1016 Topic:mno.de.pola.trump Partition: 5 Leader: 1016

I am trying s simple workflow ExecuteSQL -> PutSFTP, I want my flowfiles to be with .avro extension as well as I want to add either a prefix or suffix to the file name. For example if I am extracting employee data then I want flowfile name should be employee<timestamp/randomnumber>.avro@Shu @Matt Burgess @Neeraj Sabharwal

Hi All, If user connected Namenode through Gatewaynode, suddenly namenode goesdown, after few seconds standbynode became active node. After that standby namenode became active namenode,he can able to access without getting downtime after connecting to namenode which ip address he can able to see?

hi all we run the following command on kafka machine in order to view/print the topics names we can see that its fails onCaused by: java.net.BindException: Address already in use but without to print also the port number./kafka-topics.sh --zookeeper zookeeper1:2181 --list To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files>where num_of_file > 0GC log rotation is turned offException in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPrem

 when I run the command service cloudera-scm-server startShows failed Urgent help needed  as I have stuck with issue since last 2 days org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)        at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:392)        at com.cloudera.server.cmf.Main.<init>(Main.java:242)        at com.cloudera.server.cmf.Main.main(Main.java:216)Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory   

hi there, Below spark-sql comand failed to run on an edgenode with only spark-client installed spark-sql --executor-cores 2 --num-executors 2 -e 'select profitcenter, count(1) from default.tableX group by profitcenter' --master yarn --name spark_sql_test1 --verbose --queue defaultThe error log is:19/01/23 19:18:24 ERROR ApplicationMaster: Failed to connect to driver at emr-header-1.cluster-92503:35638, retrying ...19/01/23 19:18:24 ERROR ApplicationMaster: Uncaught exception:org.apache.spark.SparkException: Failed to connect to driver! at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:672) at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:532) at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$runImpl(ApplicationMaster.scala:347) at org.apache.spark.deploy.yarn.ApplicationMaster$anonfun$run$2.apply$mcV$sp(ApplicationMaster.sca

If I were planning on decommisioning a node, is it possible to drain the flume channels? If I disable the sink, Flume doesn't proceed to do anything and if I change the batch size on the Kafka source to 0, it still fetches data.

I just upgraded ambari to 2.7.3 on my cluster. The cluster had previously been configured with kerberos, and running correctly. Now whenever I try to modify components I get an "admin session expired" dialog. I enter my admin credentials (kerberos) and get an invalid KDC admin error returned. I verified that the principal I entered has admin rights by logging into kadmin on the command line. The principal I am entering is the one I used when I initially configured the cluster to use kerberos. Why would this not work anymore, and how can I fix it?

hello all,I need to pull files from linux server, so at first i used listFile and fetchFile processors.The files are being writing and when they are done, nifi need to pull them.Unfotunatly, listFile timestamp already list the file but fetchFile did not pull them because they are still being writing (i added last modified limitation). so the files left in the server. To solve this problem, i started using getSftp which is really good, but it does not include file size attribute.I can see the file size at DETAILS tab in the flowFile but cant access it.Do you have any suggestion?

hello all, Trying to update a  table where date_num is  a  date(timestamp) and "day" is the field i want to update but i am getting errorsAnyhting wrong with the query below? update anp_catdidai1_working.dim_bsp_dateset day=day(date_num)

Hi,   I noticed that the time it takes for load data to run in order to load a single file into a partitioned, internal, parquet table is proportional to the partition and table size.   Then I did an experiment in which I run load time for the same parquet file (renaming it uniquely), which has around 1MB in size, until one partition was filled up with 10k files. Then, I continue filling up the next partition, until 5 partitions were full with 10k files each (full partition has around 12GB in size).   The results are the following:    It is noticeable that load data time increases slightly inside a partition and the next load data time continues from where the previous partition left.   Obs: we are using Ibis framework to do the load data.   Could you please explain a little bit about the Impala load data internals in order to justify increase in time as the table grows?   Thanks, Paulo.

Hello,I'm looking for ways of enforcing specific cloud configurations for CB template to limit and simplify user experience. We'd like to pre-configure regions, VPCs, subnets, encryption configuration and so on. We are using rc-2.9 branch of Cloudbreak.Thanks!

By default with ambari installation, Zeppelin is set to have yarn client mode for Spark Interpreter which means the driver runs in the same host of Zeppelin Server. This incur high memory pressure on the Zeppelin Server host especially when Spark Interpreter is ran in isolated mode.I'm trying to switch to yarn-cluster mode which would let yarn decide on where spark driver should be executed depending of the available resources in the cluster. This mode is supported by Zeppelin since the version 0.8.0 but I'm facing the following issue https://issues.apache.org/jira/browse/ZEPPELIN-3633. Basically, the node where yarn decided to run spark driver doesn't have zeppelin installed so is unable to start. There is a fix on Zeppelin's github https://github.com/apache/zeppelin/pull/3181 but I can't find the files that I need to change. Any chance that this can be fixed easily or should I just install zeppelin on every nodes?

Hi,there are many log files in /var/log/hadoop/hdfs, and file name like hdfs-audit.log.2018-06-01,and some file name like hadoop-hdfs-namenode-hostname are very large.may i delete it all? (this host running namenode, ranger, datanode flume and node manager.)and if any configure can setting those log as auto cycling or auto delete? Thanks

Hello, every one.Currently, I try to use Presto in docker on YARN.I have 3 node manager hosts and 2 resource maneger hosts (this is because resource manager is HA) in my HDP cluster.I completed creating docker containers on all node manager hosts and was able to run Presto server (one of three is presto-coordinator and remain are presto-workers) by using overlay network.Also, I know that YARN supports docker containers and can manage docker containers resource (for example: CPU Usages).Now, in my above use case, I want to know how to configure docker containers resource management by YARN.Thank you.

HI all, A simple question:  It's safe to delete replication logs under /user/hdfs/.cm/ ? Thanks in advance.Ivan

Hi,  I have a good old GSS initiate failed Keberos error message when my application (JAVA) tries to connect to Impala via JDBC. I tried to eliminate all the usuall root causes but this time I think I missed something, because it still not connects:      Cannot connect: connection refused: Java::JavaSql::SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed. The settings are quite standard, MIT Kerberos, kerberized Impala with SSL, here are the params for the JDBC:   -> KDC host: ip-10-85-150-11.eu-west-1.compute.internal-> ImpaladD host: ip-10-85-150-6.eu-west-1.compute.internalJDBC params:-> Host: ip-10-85-150-6.eu-west-1.compute.internal -> Port: 21050-> Additional params in URL: KrbHostFQDN=ip-10-85-150-11.eu-west-1.compute.internal;KrbRealm=HADOOP.DEV.REALM.LOCALL;KrbServiceName=impala;SSL=1;CAIssuedCe

hi allwe want to verify the name nodes status from ambari by API ( namenode adtive/stand by under hdfs )capture.pngis it possible to create API for that?