Solved

When i am running below select unix_timestamp('20190131 05:00:00',"yyyyMMdd HH:mm:ss"); output is 1548910800  but when i am running same thing on a table on a column which is big int select min(IPID.BK_EFF_STRT_DT) ,max(IPID.BK_EFF_STRT_DT) from  ipid; o/p is below +--------------------------+--------------------------+| min(ipid.bk_eff_strt_dt) | max(ipid.bk_eff_strt_dt) |+--------------------------+--------------------------+| 1512709200000 | 1548910800000 why is extra 0 added in 2nd case Also select cast(1548910800000 div 1000 as timestamp) + interval (1548910800000 % 1000) milliseconds; returns : 2019-01-31 05:00:00 This is causing issue when i am trying something like below it returns 0 rows: select count(*) from ipidwhere unix_timestamp('20190124',"yyyyMMdd") between IPID.BK_EFF_STRT_DT AND IPID.BK_EFF_END_DT;

When I try to add a GCP credential in the UI I get this error:Failed to verify the credential: Could not verify credential! 403 ForbiddenOn console I get this:# cb credential create from-file --cli-input-json credential.json --name cloudbreakkeyEnter Password: ERROR: status code: 400, message: Failed to verify the credential: Could not verify credential! 403 Forbiddencb doctor:# cbd doctor===> Deployer doctor: Checks your environment, and reports a diagnose.uname: Linux cbd-deployment-vm 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linuxlocal version:2.9.0latest release:2.9.0docker command exists: OKdocker client version: 1.13.1docker client version: 1.13.1ping 8.8.8.8 on host: OKping github.com on host: OKping 8.8.8.8 in container: [ERROR]ping github.com in container: [ERROR] I assume there must be some networking issue but don't know what to look forEDIT: In the logs I get this stacktrace:com.google.api.client.googleapis.json.GoogleJs

I created a table using Regex Serde in Hive (shown below). In Heu it returns that the creation of the table was successful. However, when I try to return the table (SELECT * FROM pricefile_edited) or view the table in heu, it doesn't work and I get the Error (also below).  The data is a 130 character (each line) with no delimiter. Does anyone know what seems to be the problem, and help? Thanks SELECT * FROM pricefile_edited;CREATE EXTERNAL TABLE pricefile_edited(field1 STRING,field2 STRING,field3 STRING,field4 STRING,field5 STRING,field6 STRING, field7 STRING,field8 STRING,field9 STRING,field10 STRING,field11 STRING,field12 STRING,field13 STRING,field14 STRING,field15 STRING,field16 STRING,field17 STRING,field18 STRING)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = "(\\.{12})(\\.{1})(\\.{1})(\\.{24})(\\.{6})(\\.{6})(\\.{13})(\\.{6})(\\.{1})(\\.{4})(\\.{1})(\\.{3})(\\.{17})(\\.{9})(\\.{12})(\\.{1})(\\.{1})(\\.{12})")LOCATION '/use

I'm currently building a customed cdh6.1.0 docker image.Based on centos7:latest.Installed net-tools, openssh-server, openssh-clients and etc.The privileged is set to true to allow systemd boot up.All is fine(though cost me a week to get through) and now I've got images: 1. image installed cloudera-scm-server (used as cloudera manager)2. image installed dnsmasqude to provide dns resolution instead of editing the /etc/hosts (used as dns)3. image loaded with latest cdh6.1.0/cm6.1.0/gplextra6.1.0 repos serving on port 8900 (used as local repo)4. image installed only sshd and ntpd services (used for deployment, run under privileged mode) Then I started a test cluster to deploy CDH on some replicates of the 4th images. Though there were a lot of errors bursted into logs (such as some file cannot be found which in fact exists and well written). After a short duration, the installation completes, the detail should the installation script completed. However the status showed that

I have the following example JSON in a flowfile (data has been randomly imputed for this example) :{"TrackingRequestFirstIp":"192.10.01.01","TrackingRequestLastIp":"100.120.123.456","TrackingRequestCreationTime":"05:49:55","TrackingRequestCreationDate":"20171013","TrackingRequestTrackingNumber":"12349876"}Here is my AVRO schema:{  "name": "IpReports",  "type": "record",  "namespace": "com.acme.avro",  "fields": [    {"name": "DataSourceInformation",       "type":{        "type":"map",         "values": {          "name":"DataSourceInformation",          "type":"record",           "fields": [            {"name":"DataSourceUuid", "type":["null","string"], "default":null},          &n

I have a cluster of 350+ nodes ( Edge Nodes, Ingestion Nodes, Management Nodes & Worker Nodes) and all these managed by Ambari. All the Management nodes and Worker nodes are up-to-date with the latest Krb5.conf file. However the Edge Nodes and Ingestion Nodes Krb5.conf file is not updated and missing an entry. Hence My question is, If I modify the krb5.conf file in the Edge nodes & Ingestion nodes Manually to match with the same krb5.conf file in the remaining nodes, what would be the impact? Which all components would require restart? How do i go about syncing them up?No master services or client services are installed on these nodes. However they are part of the cluster. Any help would be appreciated.

Hi All,We have created a Hive database and have some external tables on with files placed in S3 bucket. Our Application accesses this data using Impala.Tables are not partitioned and the files are in text/csv format.How should we collect stats using Impala or Hive ?Is it possible to collect stats for Hive tables using Impala and HIve tables stats should only be created to hive. Appreciate your help. Regards.Sandeep Suman

  I need to change my Community Nickname but the option is disabled, is there a way I can change my nickname ??

Hi Friends,I have little doubt about Impala using fair schedulre for launching job. I am reading about Impala since 3 months but I never come across about Impala using fair scheduler instead it has its own mechanism for resource allocation. Is there any situation where Impala uses fair scheduler during execution of query?I have one more doubt about IMPALA using YARN. I want to the scenario/condition when we have to use YARN with IMPALA, becuase IMPALA has its own execution engine. When we have to use YARN with IMPALA? I think Llama is meant for that only.

Hello All, Could someone appraise me the significance of Impala Audit & Lineage Logs. We are facing space issue and would like  to know whether we can delete some of those old log files, also the default setting for Impala Daemon Maximum Lineage Log File Size & Impala Daemon Maximum Audit Log File Size is 5000 can this be lowered to 1000 etc. RegardsWert

Hi Friends,I am trying to run IMPALA queries using query options. I am trying to analyze the resultant query attributes. What I have observed that when I run the query for the first time, it took long time than I run the query second time or onwards.I want to know the reason behind difference in time?Does this happen only when table statistics are not available or it happens all the time.Is my observation right? 

Hi,I'm having a little bit of a problem when trying to create a Database and when trying to create a Table.I was following the Hortonworks tutorial for Loading and Querying Data with Hadoop, but when I try to create new Databases or new Tables the Database option show me no results, even if the query to create them was successul.Queries:Database:The Database "TCC" that I created don't appear either.

I'm trying to work with Spark and Hive for HDP 3.0. As I see in some articles, now we have to use Hive Warehouse Connector. Everything works fine except for one proble.I can't write a dataframe directlyh from Spark to Hive using Hive Warehouse connector.I have the following code: val spark = SparkSession      .builder()      .appName("Spark Hive Test Job")      .config("spark.sql.hive.hiveserver2.jdbc.url", "jdbc:hive2://s01.ndtstu.local:2181,m01.ndtstu.local:2181,m02.ndtstu.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;user=hive;password=hive?")      .getOrCreate()                val hive = HiveWarehouseSession.session(spark).build()    hive.createDatabase("testADD", true);    hive.setDatabase("testADD")    hive.createTable("tabletest").ifNotExists()        .column("k", "int")      &nb

Hi, Can I create a parameterized view in impala somethig like the below pseudo code: Create View MyView     as     SELECT col1, col2 col3     FROM table_one     WHERE startdate = ${date1}     and enddate         = ${date2}     ...   

After enable Kerberos, all the links (Job history server web UI, Solr UI, etc.), asking for Authorization Kerberos.png

I need to change the property of default_sql_interpreter  to Impala in place of hive so by default Impala query editor get loaded. I check hue.ini but didn't find that. Will that be available in hue database configuration.  

Hi all,I have 6 datanodes on my hortonworks cluster (HDP 2.6.3) and one of them is 91% full. The others are "only" 65% full.Il don't understand why the replication is not homogeneous and how i can fix it ?I check the file system and the same difference is observed :On a safe node :# pwd/grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized# du -h . --summarize1.9T    .On the unsafe node :# pwd/grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized# du -h . --summarize2.7T    .Same things on each DataNode directories.Thanks for your help.Mathieu

Hi Guys,I am trying to sync ldap users, I am getting below error, any idea, please?====================Review Settings====================Primary LDAP Host : xxxxxxxxxxxxx001Primary LDAP Port (636): 636Secondary LDAP Host <Optional>: xxxxxxxxxxx002Secondary LDAP Port <Optional> (636): 636Use SSL [true/false] (true): trueUser object class (posixUser): posixUserUser ID attribute (uid): uidGroup object class (posixGroup): posixGroupGroup name attribute (cn): cnGroup member attribute (memberUid): memberUidDistinguished name attribute (dn): dnSearch Base (dc=xyzcompany,dc=com): dc=xyzcompany,dc=comReferral method [follow/ignore] (follow): followBind anonymously [true/false] (false): falseHandling behavior for username collisions [convert/skip] for LDAP sync (skip): skipForce lower-case user names [true/false] (true): trueResults from LDAP are paginated when requested [true/false] (false): falseambari.ldap.connectivity.bind_dn: uid=svcTDPlookup,ou=serviceaccounts,ou

Hi.  We have a client who has 2 clusters.  On the security cluster, they have sensitive data that they redact and copy to the analysis cluster.  For security reasons, they would like to minimize the number of open ports on the security cluster.  We have successfully tested using distcp from the shell to copy the data with port 8020 open.  They would now like to automate the process through oozie.  In testing, we have run into an error that port 8042 (Node Manager External Port) is not open.     We do not understand why distcp works fine without port 8042 available when run through the shell but fails when called through Oozie.   Any help would be appreciated.  Thanks.   Henry

we need to partition our Hive Table based on date. Date/Month/Year is it better to use int or string for the partition types.ex:CREATE EXTERNAL TABLE partition (id string, event timestamp and so on)PARTITIONED BY (year INT, month INT, day INT)Stored as Parquetvs CREATE EXTERNAL TABLE partition (id string, event timestamp and so on)PARTITIONED BY (year string, month string, day string)Stored as Parquet Noticed that we couldn't do queries like:... where day > 10 with the string option

The following ODBC download page version has the description "C6 Compatible".　https://www.cloudera.com/downloads/connectors/hive/odbc/2-6-1.html　　Hive ODBC Connector 2.5.25 (C6 Compatible)　https://www.cloudera.com/downloads/connectors/impala/odbc/2-6-2.html　　Impala ODBC Connector 2.5.42 (C6 Compatible)What does "C6 compatible" mean?

We have a highly scalable kafka cluster with 11 nodes and another highly scalable Nifi cluster with 11 nodes that we are currently testing. These are created from our custom blueprint. They are deployed in Azure.We find that everytime we stop and restart the cluster a couple of the nodes go unhealthy and we are unable to recover.Do let us know how to resolve the issue.

Hi All, if we add a few additional DataNodes to our cluster, and then execute a rebalance, do the dependent services (Hive, Impala, SOLR, Flume, etc) require an immediate restart or can they run with a stale configuration for a week or so until we can schedule it?   The DataNodes will include the following roles: HBase RegionServer HDFS DataNode Impala Daemon Solr Server YARN NodeManager Thank you.

Hello,I have an attribute log_source that holds a path like D:\dir1\dir2\dir3I need to get only dir2 but I'm unable to escape the slash with ${log_source:getDelimitedField(3,'\')}I've tried several things unsuccessfully.

I sure hope someone can help me, because otherwise I'm sunk. I'm using the ListS3 processor to feed a FetchS3 processor, but when I list my test directory, which contains one file, I get 4 messages in my queue instead of one. (I also get 4 messages with the directory name, but I filter that out.) The 4 messages have the same filename and I can't find any reason why this is happening.I searched the site but can find no mention of duplicate messages (or flows) from the ListS3 processor. Which is odd because it has been so easy to create this condition. Multiple developers encountered it. It wasn't just one flow. I encountered it when creating flows on a Linux server, version 1.6, and on my local Windows 10 machine, version 1.8. Does it every time with various buckets.If I examine the queue in a connection between processors, I can see the 4 messages and that they all have the same filename. The weirdest thing is that position is the same for each of the 4 messages. When I list the direct