Member since
09-18-2015
3274
Posts
1153
Kudos Received
425
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
985 | 11-01-2016 05:43 PM | |
2074 | 11-01-2016 05:36 PM | |
1904 | 07-01-2016 03:20 PM | |
3206 | 05-25-2016 11:36 AM | |
1317 | 05-24-2016 05:27 PM |
04-07-2021
05:55 PM
I know it's an old post but looks issue is unresolved yet. I came to following bash version: function hdfsMoveMerge()
{
src=$1
DEST=$2
if [[ "$SRC" == "$DEST" ]]
then
echo "source and dest is same "
return 1
else
echo "source and dest is not same "
fi
hadoop fs -find $SRC | xargs -n 10 hadoop fs -ls -d | grep -v '^d'| awk '{print $8}' | while read file ;
do
newFP=${file/$SRC/${DEST}};
if hadoop fs -mkdir -p $(dirname $newFP ) ;
then
cnt=0
if hadoop fs -test -f ${newFP}
then
((cnt++))
while hadoop fs -test -f "${newFP}_copy($cnt)"
do
((cnt++))
done
hadoop fs -mv "${file}" "${newFP}_copy($cnt)";
else
hadoop fs -mv "${file}" "${newFP}";
fi
fi
done
}
... View more
01-24-2021
01:41 AM
I was just able to confirm that the update command listed is Postgresql database flavor.
... View more
12-08-2020
01:49 PM
Hi, I have permenantly deleted the data. Is this anyway we can recover the data?
... View more
12-07-2020
11:36 AM
2020 Update, what are the preferred data quality tools compatible with CDH for Hive,Hbase and Solr? Our team is looking at Apache Griffin. Regards, Nithya Koka
... View more
12-05-2020
12:22 PM
check this video for solution https://www.youtube.com/watch?v=cNebUgKxKqw
... View more
12-01-2020
03:47 AM
I'm trying to run a dag with airflow 1.10.12 and HDP 3.0.0 when i run the dag it gets stuck in ```Connecting to jdbc:hive2://[Server2_FQDN]:2181,[Server1_FQDN]:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2``` when i run ```beeline -u "jdbc:hive2://[Server1_FQDN]:2181,[Server2_FQDN]:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"``` from shell, it connect to hive with no problem. I've also made a connection like this ``` Conn Id * hive_jdbc ------------- Conn Type ------------- Connection URL jdbc:hive2://centosserver.son.ir:2181,centosclient.son.ir:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 ------------- Login hive ------------- Password ****** ------------- Driver Path /usr/hdp/3.0.0.0-1634/hive/jdbc/hive-jdbc-3.1.0.3.0.0.0-1634-standalone.jar ------------- Driver Class org.apache.hive.jdbc.HiveDriver ``` and I'm not using kerberos I've also added ```hive.security.authorization.sqlstd.confwhitelist.append``` in the ambari ```Custom hive-site``` ``` radoop\.operation\.id|mapred\.job\.name||airflow\.ctx\.dag_id|airflow\.ctx\.task_id|airflow\.ctx\.execution_date|airflow\.ctx\.dag_run_id|airflow\.ctx\.dag_owner|airflow\.ctx\.dag_email|hive\.warehouse\.subdir\.inherit\.perms|hive\.exec\.max\.dynamic\.partitions|hive\.exec\.max\.dynamic\.partitions\.pernode|spark\.app\.name ``` any suggestions? I'm desperate, I've tried every way that i know but still nothing @nsabharwal @agillan @msumbul1 @deepesh1
... View more
10-06-2020
09:17 AM
@praveen_bora Java UDF's in Impala work differently than Hive. Since Hive is Java based, it will load all the jars in the Aux directory to the classpath, so any dependency jars are also picked up. Impala backends are C++ based, and it only picks up the specific jar configured when the UDF is defined, so you will need to bundle all dependency jars inside the UDF jar as well so the UDF is self contained in a single jar. There are certain restrictions of using UDFs for Hive in Impala. These restrictions are listed under "Using Hive UDFs with Impala" section in the link . https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_udf.html
... View more
08-13-2020
09:08 AM
@torafca5 Could you please try downloading the jar from the below link, http://www.congiu.net/hive-json-serde/1.3.8/hdp23/json-serde-1.3.8-jar-with-dependencies.jar Once the jar is downloaded, move the jar to the location /usr/hdp/3.0.1.0-187/hive/lib. Please place the jar on all the nodes hosting Hive services. Also, please make sure you are not using LLAP(HiveserverInteractive) to connect to the hive. add jar command does not work with LLAP. implementing the above recommendation should help overcome this issue.
... View more
08-11-2020
01:40 AM
I did this in root user, found the file and changed it there. But, how to change it for each node?
... View more
07-13-2020
04:11 AM
Hello, I have the same issue. I want to permanently store a custom mapping. How could you resolve your issue? regards
... View more
06-30-2020
01:06 AM
I see that you use Active DIrecyory Did you use the below property? +++ < property > < name > hive.server2.authentication.ldap.Domain </ name > < value > AD_Domain </ value > </ property > +++
... View more
06-11-2020
01:27 PM
Our installation had the password hash in another table. update ambari.user_authentication set authentication_key='538916f8943ec225d97a9a86a2c6ec0818c1cd400e09e03b660fdaaec4af29ddbb6f2b1033b81b00' where user_id='1' Note: user_id=1 was the admin in my case.
... View more
06-08-2020
12:19 AM
@gfeng As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.
... View more
05-13-2020
11:56 PM
Yes this is possible. You need to kinit with the username that has been granted access to the SQL server DB and tables. integrated security passes your credentials to the SQL server using kerberos "jdbc:sqlserver://sername.domain.co.za:1433;integratedSecurity=true;databaseName=SCHEMA;authenticationScheme=JavaKerberos;" This worked for me.
... View more
05-09-2020
01:20 AM
Hi bigdatasuperno, I need a beeline shell script which should connect hive url and print the no. of tables in the db and print the record count of each table in the output file. Your help will greatly appreciated. I belongs to different technology. Regards, Vin
... View more
05-06-2020
03:24 AM
This could be permission issue. you can see the hive server2 log for the error. Log will be in /var/log/hive on the node to which you connect the hive
... View more
05-03-2020
03:56 PM
20-05-04 04:25:52 DEBUG ClientCnxn:818 - Reading reply sessionid:0x171dc5f77f00040, packet:: clientPath:null serverPath:null finished:false header:: 23,3 replyHeader:: 23,1099,0 request:: '/hbase-unsecure,F response:: s{116,116,1477381693974,1477381693974,0,24,0,0,0,18,966} 2020-05-04 04:25:52 DEBUG ClientCnxn:717 - Got ping response for sessionid: 0x171dc5f77f00040 after 4ms 2020-05-04 04:25:52 DEBUG ClientCnxn:818 - Reading reply sessionid:0x171dc5f77f00040, packet:: clientPath:null serverPath:null finished:false header:: 24,4 replyHeader:: 24,1099,0 request:: '/hbase-unsecure/master,F response:: #ffffffff000146d61737465723a3136303030ffffffe330ffffff86ffffffafffffffd2352bffffffc850425546a23a1773616e64626f782e686f72746f6e776f726b732e636f6d10ffffff807d18ffffff95ffffffcaffffffbdffffffe3ffffff9d2e10018ffffff8a7d,s{938,938,1588541230259,1588541230259,0,0,0,104106568998453260,71,0,938} 2020-05-04 04:25:52 DEBUG RpcClientImpl:351 - Use SIMPLE authentication for service MasterService, sasl=false 2020-05-04 04:25:52 DEBUG RpcClientImpl:716 - Connecting to sandbox.hortonworks.com/127.0.0.1:16000 2020-05-04 04:25:54 INFO RpcRetryingCaller:132 - Call exception, tries=10, retries=35, started=64734 ms ago, cancelled=false, msg=
... View more
03-31-2020
03:06 AM
you should install ambari-server and ambari-agent on the first node wich you want to install hdfs service for example . the other nodes install ambari-agent only . dont forget to change (ambari-agent.ini ) hostname and the hosts file with (ip and hostname of all machines.
... View more
03-23-2020
02:14 AM
hello, This wokarround didn't work for me. Configured the LDAP setup so that the BaseDN matches only 1 entry. calling " ambari-server sync-ldap --existing" didn't remove all existing LDAP Users and groups rather it deleted 2 only. may be i missed something, but after running the setup do we need to restart ambari-server? What should be the expected behaviour when runing the "ambari-server sync-ldap --all" and the BaseDN pointing to a single AD entry? The doc states the following for option '--exisiting' : "Users will be removed from Ambari if they no longer exist in LDAP, and group membership in Ambari will be updated to match LDAP". Since AD users still exist that would have no effect to remove the users even if baseDN points to single entry. What we are looking for (HDP2.6.5) is to remove all LDAP synced users other than these specified in --users users.txt and --groups group.txt. It looks like there is no such tool and we have to resort to manually use ambari APIs somehow. One thing i'm not sure is how are the lowercased alias being handled, since during the first sync we had the default value 'true' to force lower case, and now changed it to 'false' looking forward your insights
... View more
03-06-2020
07:22 AM
@Ham As this is an older post you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.
... View more
02-10-2020
08:47 AM
Awesome.It worked for me.
... View more
02-04-2020
04:02 PM
@asmarz,
One of our members posted a reply on how to add users in the thread you posted a similar question to later the same day.
As this is an older thread which was previously marked 'Solved', you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity for you to provide details specific to your environment about what you did in an attempt to add the relevant user accounts that could aid others in providing a more relevant, accurate answer to your question.
... View more
01-23-2020
11:46 AM
Okay so I wrote an example nifi process to do it https://www.datainmotion.dev/2020/01/flank-stack-nifi-processor-for-kafka.html
... View more
12-19-2019
01:53 PM
@Augustine ,
Thanks for the feedback. I have corrected the answer and thank you for reporting this.
Cheers,
Li
... View more
12-17-2019
07:48 AM
Hi All . here is more Details about above :- https://community.cloudera.com/t5/Support-Questions/HDInsight-Vs-HDP-Service-on-Azure-Vs-HDP-on-Azure-IaaS/m-p/166424 Thanks HadoopHelp
... View more
11-28-2019
12:05 AM
Hi, the link seems broken. Can you share with us the working one? Thanks.
... View more
10-30-2019
08:08 AM
Moved this question from the original thread from Feb 2016.
11-09-2018
06:23 AM
@Venu Shanmukappa how did u add the proxy.. can u pls explain
... View more
12-12-2016
10:18 AM
Hello, Kindly remove pam_profiles in below given lines and restart the drill. It will work. drill.exec {
security.user.auth {
enabled: true,
packages += "org.apache.drill.exec.rpc.user.security",
impl: "pam",
pam_profiles: [ "sudo", "login" ]
}
} Thanks Michael N.
... View more
11-01-2016
05:31 PM
Demo
Extract data from images and store in HDFS. Documents with size less than 10mb stores into HBase.
Document > 10mb lands into HDFS with metadata into HBase
Part 1 - https://www.linkedin.com/pulse/cds-content-data-store-nosql-part-1-co-dev-neeraj-sabharwal
... View more
- Find more articles tagged with:
- Data Processing
- HBase
- How-ToTutorial
- image-extract
Labels: