About WELI

WELI · ‎01-03-2016

Anyone knows when HDF 1.1 will be available?

WELI · ‎01-03-2016

After adding org.xerial.snappy.tempdir to a newly created directory with rwx permissions, spark works fine now.

WELI · ‎12-17-2015

Hi Pardeep, Thanks fro taking time to answer my question. Even I changed location to /tmp/data.tsv, still got the same error. There are only two statements int the pig script: A = LOAD '/tmp/data.tsv' USING PigStorage ('\t') AS (id:chararray,c1:chararray,c2:chararray); --DUMP A STORE A INTO 't3' USING org.apache.hive.hcatalog.pig.HCatStorer(); And from the output of the script, there are two MR jobs ran: pig -useHCatalog simple_bulkload.pig WARNING: Use "yarn jar" to launch YARN applications. 15/12/17 16:24:40 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 15/12/17 16:24:40 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 15/12/17 16:24:40 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType 2015-12-17 16:24:40,232 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.3.2.0-2950 (rexported) compiled Sep 30 2015, 19:39:20 2015-12-17 16:24:40,232 [main] INFO org.apache.pig.Main - Logging error messages to: /home/weli/pig_1450391080230.log 2015-12-17 16:24:40,797 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/weli/.pigbootup not found 2015-12-17 16:24:40,893 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://HdpTest 2015-12-17 16:24:42,192 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://stlhd05t.centene.com:9083 2015-12-17 16:24:42,229 [main] INFO hive.metastore - Connected to metastore. 2015-12-17 16:24:42,682 [main] INFO org.apache.hadoop.hive.hbase.HBaseStorageHandler - Configuring output job properties 2015-12-17 16:24:42,803 [main] INFO org.apache.hadoop.hive.hbase.HBaseStorageHandler - Configuring output job properties 2015-12-17 16:24:42,867 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2015-12-17 16:24:42,899 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code. 2015-12-17 16:24:42,932 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]} 2015-12-17 16:24:43,040 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2015-12-17 16:24:43,062 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2015-12-17 16:24:43,062 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2015-12-17 16:24:43,357 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://stlhd06t.centene.com:8188/ws/v1/timeline/ 2015-12-17 16:24:43,442 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stlhd06t.centene.com/172.19.26.26:8050 2015-12-17 16:24:43,566 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job 2015-12-17 16:24:43,569 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2015-12-17 16:24:43,570 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process 2015-12-17 16:24:43,754 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-metastore-1.2.1.2.3.2.0-2950.jar to DistributedCache through /tmp/temp-1114090506/tmp144129392/hive-metastore-1.2.1.2.3.2.0-2950.jar 2015-12-17 16:24:43,785 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive/lib/libthrift-0.9.2.jar to DistributedCache through /tmp/temp-1114090506/tmp-1993882981/libthrift-0.9.2.jar 2015-12-17 16:24:43,892 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to DistributedCache through /tmp/temp-1114090506/tmp1975882158/hive-exec-1.2.1.2.3.2.0-2950.jar 2015-12-17 16:24:43,914 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive/lib/libfb303-0.9.2.jar to DistributedCache through /tmp/temp-1114090506/tmp2019681710/libfb303-0.9.2.jar 2015-12-17 16:24:43,938 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive/lib/jdo-api-3.0.1.jar to DistributedCache through /tmp/temp-1114090506/tmp-17824286/jdo-api-3.0.1.jar 2015-12-17 16:24:43,959 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-hbase-handler-1.2.1.2.3.2.0-2950.jar to DistributedCache through /tmp/temp-1114090506/tmp-1130216789/hive-hbase-handler-1.2.1.2.3.2.0-2950.jar 2015-12-17 16:24:43,981 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive-hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1.2.3.2.0-2950.jar to DistributedCache through /tmp/temp-1114090506/tmp-1592124709/hive-hcatalog-core-1.2.1.2.3.2.0-2950.jar 2015-12-17 16:24:44,004 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/hive-hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-1.2.1.2.3.2.0-2950.jar to DistributedCache through /tmp/temp-1114090506/tmp-1500851776/hive-hcatalog-pig-adapter-1.2.1.2.3.2.0-2950.jar 2015-12-17 16:24:44,043 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/pig/pig-0.15.0.2.3.2.0-2950-core-h2.jar to DistributedCache through /tmp/temp-1114090506/tmp374893069/pig-0.15.0.2.3.2.0-2950-core-h2.jar 2015-12-17 16:24:44,066 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/pig/lib/automaton-1.11-8.jar to DistributedCache through /tmp/temp-1114090506/tmp1964188417/automaton-1.11-8.jar 2015-12-17 16:24:44,088 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/hdp/2.3.2.0-2950/pig/lib/antlr-runtime-3.4.jar to DistributedCache through /tmp/temp-1114090506/tmp775990025/antlr-runtime-3.4.jar 2015-12-17 16:24:44,110 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2015-12-17 16:24:44,117 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code. 2015-12-17 16:24:44,117 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche 2015-12-17 16:24:44,117 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize [] 2015-12-17 16:24:44,153 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2015-12-17 16:24:44,255 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://stlhd06t.centene.com:8188/ws/v1/timeline/ 2015-12-17 16:24:44,255 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stlhd06t.centene.com/172.19.26.26:8050 2015-12-17 16:24:44,261 [JobControl] WARN org.apache.hadoop.hdfs.ClientContext - Existing client context 'default' does not match requested configuration. Existing: shortCircuitStreamsCacheSize = 4096, shortCircuitStreamsCacheExpiryMs = 300000, shortCircuitMmapCacheSize = 256, shortCircuitMmapCacheExpiryMs = 3600000, shortCircuitMmapCacheRetryTimeout = 300000, shortCircuitCacheStaleThresholdMs = 1800000, socketCacheCapacity = 16, socketCacheExpiry = 3000, shortCircuitLocalReads = true, useLegacyBlockReaderLocal = false, domainSocketDataTraffic = false, shortCircuitSharedMemoryWatcherInterruptCheckMs = 60000, keyProviderCacheExpiryMs = 864000000, Requested: shortCircuitStreamsCacheSize = 4096, shortCircuitStreamsCacheExpiryMs = 300000, shortCircuitMmapCacheSize = 256, shortCircuitMmapCacheExpiryMs = 3600000, shortCircuitMmapCacheRetryTimeout = 300000, shortCircuitCacheStaleThresholdMs = 1800000, socketCacheCapacity = 16, socketCacheExpiry = 3000, shortCircuitLocalReads = false, useLegacyBlockReaderLocal = false, domainSocketDataTraffic = false, shortCircuitSharedMemoryWatcherInterruptCheckMs = 60000, keyProviderCacheExpiryMs = 864000000 2015-12-17 16:24:44,322 [JobControl] INFO org.apache.hadoop.hive.hbase.HBaseStorageHandler - Configuring output job properties 2015-12-17 16:24:44,391 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2015-12-17 16:24:44,459 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2015-12-17 16:24:44,459 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 2015-12-17 16:24:44,473 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2015-12-17 16:24:44,541 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1 2015-12-17 16:24:44,627 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1449253347993_0033 2015-12-17 16:24:44,743 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources. 2015-12-17 16:24:44,819 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1449253347993_0033 2015-12-17 16:24:44,843 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://stlhd06t.centene.com:8088/proxy/application_1449253347993_0033/ 2015-12-17 16:24:44,844 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1449253347993_0033 2015-12-17 16:24:44,844 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases A 2015-12-17 16:24:44,844 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: A[1,4],A[-1,-1] C: R: 2015-12-17 16:24:44,855 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2015-12-17 16:24:44,855 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1449253347993_0033] 2015-12-17 16:24:54,881 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! ======================================================================================= Specify -stop_on_failure if you want Pig to stop immediately on failure. 2015-12-17 16:24:54,881 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1449253347993_0033 has failed! Stop running all dependent jobs 2015-12-17 16:24:54,881 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete ----------------------------------------------------------------------------------- 2015-12-17 16:24:54,968 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://stlhd06t.centene.com:8188/ws/v1/timeline/ 2015-12-17 16:24:54,969 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stlhd06t.centene.com/172.19.26.26:8050 2015-12-17 16:24:55,079 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://stlhd06t.centene.com:8188/ws/v1/timeline/ 2015-12-17 16:24:55,079 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at stlhd06t.centene.com/172.19.26.26:8050 2015-12-17 16:24:55,104 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed! ------------------------------------------- 1 failed ----------------------------------------------------------------------- 2015-12-17 16:24:55,105 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.7.1.2.3.2.0-2950 0.15.0.2.3.2.0-2950 weli 2015-12-17 16:24:43 2015-12-17 16:24:55 UNKNOWN Failed! Failed Jobs: JobId Alias Feature Message Outputs job_1449253347993_0033 A MAP_ONLY Message: Job failed! t3, Input(s): Failed to read data from "/tmp/data.tsv" Output(s): Failed to produce result in "t3" Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_1449253347993_0033 2015-12-17 16:24:55,105 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! 2015-12-17 16:24:55,110 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job job_1449253347993_0033 failed, hadoop does not return any error message Details at logfile: /home/weli/pig_1450391080230.log 2015-12-17 16:24:55,125 [main] INFO org.apache.pig.Main - Pig script completed in 15 seconds and 4 milliseconds (15004 ms) If I uncomment DUMP A, I can see the files are read out correctly! Can you spot anything wrong? Thanks,

WELI · ‎12-16-2015

Following guide here, the last step pig -useHCatalog simple.bulkload.pig give error message: pig read the file correctly (the DUMP command verified): 2015-12-16 11:46:58,012 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code. 2015-12-16 11:46:58,022 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2015-12-16 11:46:58,022 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process :1 (row1,c1,c2) (row2,c1,c2) (row3,c1,c2) (row4,c1,c2) (row5,c1,c2) (row6,c1,c2) (row7,c1,c2) 2015-12-16 11:46:58,225 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://stlhd05t.centene.com:9083 2015-12-16 11:46:58,286 [main] INFO hive.metastore - Connected to metastore. Then at the end ============== HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.7.1.2.3.2.0-2950 0.15.0.2.3.2.0-2950 weli 2015-12-16 11:46:58 2015-12-16 11:47:10 UNKNOWN Failed! Failed Jobs: JobId Alias Feature Message Outputs job_1449253347993_0030 A MAP_ONLY Message: Job failed! t3, Input(s): Failed to read data from "hdfs:///tmp/data.tsv" Output(s): Failed to produce result in "t3" Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_1449253347993_0030 2015-12-16 11:47:10,093 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! 2015-12-16 11:47:10,097 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job job_1449253347993_0030 failed, hadoop does not return any error message Details at logfile: /home/weli/pig_1450288003370.log 2015-12-16 11:47:10,113 [main] INFO org.apache.pig.Main - Pig script completed in 26 seconds and 846 milliseconds (26846 ms) Why the records are not written into hbase table?

WELI · ‎12-16-2015

For NN with HA, what should you put for NameNode URL value?

WELI · ‎12-15-2015

To vperiasamy, but on namenode url, you still need to specify the active namenode url like hdfs://myactive.nn.com:8020 , instead of the HA sevice name like: hdfs://myhdpcluster

WELI · ‎12-14-2015

HDP-2.3.2.0-2950 Spark: 1.4.1.2.3 $ rpm -qa |grep -i snappy snappy-devel-1.1.0-3.el7.x86_64 snappy-1.1.0-3.el7.x86_64 Hortonwork support suggests us to apply spark 1.5.1 TP repo. But our Unix admin needs a tar.gz file to set up a local repo. Anyone knows the link to tar file?

WELI · ‎12-09-2015

Thank you.

WELI · ‎12-09-2015

Neeraj, You are right the password got messed up. After correcting it, it starts to sync users/groups with AD. Mike

WELI · ‎12-08-2015

Yes. I can use LDAP browser\Editor using the same credential.

WELI · ‎12-08-2015

But I am sure my user and credentials are right, since I can use the credential in LDAP browser.

WELI · ‎12-08-2015

After changing to server:port ,it works. Thank you!

WELI · ‎12-08-2015

ambari-server-20151207.zip--setup----- sudo /usr/sbin/ambari-server setup-ldap Using python/usr/bin/python2.7 Setting up LDAP properties... Primary URL* {host:port} (ldap://ldap.xxxxxxx.com:389): ldap://ldap.xxxxx.com:389 Secondary URL {host:port} : Use SSL* [true/false] (false): false User object class* (user): user User name attribute* (sAMAccountName): sAMAccountName Group object class* (group): group Group name attribute* (cn): cn Group member attribute* (member): Distinguished name attribute* (DistinguishedName): Base DN* (dc=CENTENE,dc=com): Referral method [follow/ignore] (follow): Bind anonymously* [true/false] (false): Manager DN* (CN=xxxxxx,OU=LDAP,DC=xxxxxxx,DC=com): Enter Manager Password* : Re-enter password: ==================== Review Settings ==================== authentication.ldap.managerDn: CN=xxxuser,OU=LDAP,DC=xxxxxx,DC=com authentication.ldap.managerPassword: ***** Save settings [y/n] (y)? y Saving...done Ambari Server 'setup-ldap' completed successfully. ====Log file====================== see attachment.

WELI · ‎12-07-2015

This is the ldap url, did not see anything wrong with it: authentication.ldap.primaryUrl=ldap.xxxx.com:389

WELI · ‎12-07-2015

Thanks all for providing possible reasons and solutions. I verified the user and password, they are correct. The only thing I can think of now is that whether from SysAdmin/Unix side, they need doing something, like to grant these users/goups to access the boxes??

WELI · ‎12-07-2015

07 Dec 2015 11:33:12 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder 07 Dec 2015 11:33:13 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder created 07 Dec 2015 11:33:13 INFO UserGroupSync [UnixUserSyncThread] - initializing source: org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder 07 Dec 2015 11:33:13 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 07 Dec 2015 11:33:13 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 07 Dec 2015 11:33:13 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization started 07 Dec 2015 11:33:13 ERROR UserGroupSync [UnixUserSyncThread] - Failed to initialize UserGroup source/sink. Will retry after 30000 milliseconds. Error details: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C0903CF, comment: AcceptSecurityContext error, data 52e, v2580^@] at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3135) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3081) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2883) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2797) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.createLdapContext(LdapUserGroupBuilder.java:149) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:262) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745)

WELI · ‎12-07-2015

"href" : "http://127.0.0.1:8080/api/v1/ldap_sync_events/17", "Event" : { "id" : 17, "specs" : [ { "names" : "HDP_Admin,HDP_Developer,HDP_Users", "sync_type" : "specific", "principal_type" : "groups" }, { "names" : "user1,user2,user3", "sync_type" : "specific", "principal_type" : "users" } ], "status" : "ERROR", "status_detail" : "Caught exception running LDAP sync. Invalid name: /ldap.xxxxx.com:389; nested exception is javax.naming.InvalidNameException: Invalid name: /ldap.xxx.com:389", "summary" : { "groups" : { "created" : null, "removed" : null, "updated" : null }, "memberships" : { "created" : null, "removed" : null }, "users" : { "created" : null, "removed" : null, "updated" : null } }, "sync_time" : { "end" : 1449508883644, "start" : 1449508883642 }

WELI · ‎12-04-2015

Thank you Shane. Find out our company security policy block execution on tmp directory. So We created a directory for hbase.tmp.dir and points it to that one in Ambari setting, it worked. Thanks,

WELI · ‎12-03-2015

On HDP 2.3.1, when trying to start hbase shell, got error message: $ hbase shell java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tmp: /tmp/jffi7007828132073784430.t mp: failed to map segment from shared object: Operation not permitted at com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:90) at com.kenai.jffi.Foreign.getInstance(Foreign.java:95) at com.kenai.jffi.Library.openLibrary(Library.java:151) at com.kenai.jffi.Library.getCachedInstance(Library.java:125) at com.kenai.jaffl.provider.jffi.Library.loadNativeLibraries(Library.java:66) at com.kenai.jaffl.provider.jffi.Library.getNativeLibraries(Library.java:56) at com.kenai.jaffl.provider.jffi.Library.getSymbolAddress(Library.java:35) at com.kenai.jaffl.provider.jffi.Library.findSymbolAddress(Library.java:45) at com.kenai.jaffl.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:188) at com.kenai.jaffl.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:110) at com.kenai.jaffl.provider.jffi.Provider.loadLibrary(Provider.java:31) at com.kenai.jaffl.provider.jffi.Provider.loadLibrary(Provider.java:25) at com.kenai.jaffl.Library.loadLibrary(Library.java:76) at org.jruby.ext.posix.POSIXFactory$LinuxLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:108) at org.jruby.ext.posix.POSIXFactory$LinuxLibCProvider.getLibC(POSIXFactory.java:112) at org.jruby.ext.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:30) at org.jruby.ext.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:17) at org.jruby.ext.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:70) at org.jruby.ext.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:31) at org.jruby.ext.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:29) at org.jruby.ext.posix.LazyPOSIX.posix(LazyPOSIX.java:25) at org.jruby.ext.posix.LazyPOSIX.isatty(LazyPOSIX.java:159) at org.jruby.RubyIO.tty_p(RubyIO.java:1897) at org.jruby.RubyIO$i$0$0$tty_p.call(RubyIO$i$0$0$tty_p.gen:65535) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63) at org.jruby.ast.IfNode.interpret(IfNode.java:111) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147) at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:183) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at org.jruby.ast.VCallNode.interpret(VCallNode.java:86) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169) at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302) at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:148) at org.jruby.RubyClass.newInstance(RubyClass.java:822) at org.jruby.RubyClass$i$newInstance.call(RubyClass$i$newInstance.gen:65535) at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrNBlock.call(JavaMethod.java:249) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at usr.hdp.$2_dot_3_dot_2_dot_0_minus_2950.hbase.bin.hirb.__file__(/usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:128) at usr.hdp.$2_dot_3_dot_2_dot_0_minus_2950.hbase.bin.hirb.load(/usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb) at org.jruby.Ruby.runScript(Ruby.java:697) at org.jruby.Ruby.runScript(Ruby.java:690) at org.jruby.Ruby.runNormally(Ruby.java:597) at org.jruby.Ruby.runFromMain(Ruby.java:446) at org.jruby.Main.doRunFromMain(Main.java:369) at org.jruby.Main.internalRun(Main.java:258) at org.jruby.Main.run(Main.java:224) at org.jruby.Main.run(Main.java:208) at org.jruby.Main.main(Main.java:188) Caused by: java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tmp: /tmp/jffi7007828132073784430.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at com.kenai.jffi.Init.loadFromJar(Init.java:164) at com.kenai.jffi.Init.load(Init.java:78) at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49) at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45) at com.kenai.jffi.Foreign.getInstance(Foreign.java:95) at com.kenai.jffi.Internals.getErrnoSaveFunction(Internals.java:44) at com.kenai.jaffl.provider.jffi.StubCompiler.getErrnoSaveFunction(StubCompiler.java:68) at com.kenai.jaffl.provider.jffi.StubCompiler.<clinit>(StubCompiler.java:18) at com.kenai.jaffl.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:146) ... 50 more SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBind er.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBi nder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Foreign.java:90:in `getForeign': java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tm p: /tmp/jffi7007828132073784430.tmp: failed to map segment from shared object: Operation not permitted from Foreign.java:95:in `getInstance' from Library.java:151:in `openLibrary' from Library.java:125:in `getCachedInstance' from Library.java:66:in `loadNativeLibraries' from Library.java:56:in `getNativeLibraries' from Library.java:35:in `getSymbolAddress' from Library.java:45:in `findSymbolAddress' from DefaultInvokerFactory.java:51:in `createInvoker' from Library.java:27:in `getInvoker' from NativeInvocationHandler.java:90:in `createInvoker' from NativeInvocationHandler.java:74:in `getInvoker' from NativeInvocationHandler.java:110:in `invoke' from null:-1:in `isatty' from BaseNativePOSIX.java:300:in `isatty' from LazyPOSIX.java:159:in `isatty' from RubyIO.java:1897:in `tty_p' from RubyIO$i$0$0$tty_p.gen:65535:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from CallNoArgNode.java:63:in `interpret' from IfNode.java:111:in `interpret' from NewlineNode.java:104:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:147:in `call' from DefaultMethod.java:183:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from VCallNode.java:86:in `interpret' from NewlineNode.java:104:in `interpret' from BlockNode.java:71:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:169:in `call' from DefaultMethod.java:191:in `call' from CachingCallSite.java:302:in `cacheAndCall' from CachingCallSite.java:144:in `callBlock' from CachingCallSite.java:148:in `call' from RubyClass.java:822:in `newInstance' from RubyClass$i$newInstance.gen:65535:in `call' from JavaMethod.java:249:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:128:in `__file__' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:-1:in `load' from Ruby.java:697:in `runScript' from Ruby.java:690:in `runScript' from Ruby.java:597:in `runNormally' from Ruby.java:446:in `runFromMain' from Main.java:369:in `doRunFromMain' from Main.java:258:in `internalRun' from Main.java:224:in `run' from Main.java:208:in `run' from Main.java:188:in `main' Caused by: ClassLoader.java:-2:in `load': java.lang.UnsatisfiedLinkError: /tmp/jffi7007828132073784430.tmp: /tmp/jffi700782813207378443 0.tmp: failed to map segment from shared object: Operation not permitted from ClassLoader.java:1937:in `loadLibrary0' from ClassLoader.java:1822:in `loadLibrary' from Runtime.java:809:in `load0' from System.java:1086:in `load' from Init.java:164:in `loadFromJar' from Init.java:78:in `load' from Foreign.java:49:in `getInstanceHolder' from Foreign.java:45:in `<clinit>' from Foreign.java:95:in `getInstance' from Internals.java:44:in `getErrnoSaveFunction' from StubCompiler.java:68:in `getErrnoSaveFunction' from StubCompiler.java:18:in `<clinit>' from AsmLibraryLoader.java:146:in `generateInterfaceImpl' from AsmLibraryLoader.java:110:in `loadLibrary' from Provider.java:31:in `loadLibrary' from Provider.java:25:in `loadLibrary' from Library.java:76:in `loadLibrary' from POSIXFactory.java:108:in `<clinit>' from POSIXFactory.java:112:in `getLibC' from BaseNativePOSIX.java:30:in `<init>' from LinuxPOSIX.java:17:in `<init>' from POSIXFactory.java:70:in `loadLinuxPOSIX' from POSIXFactory.java:31:in `loadPOSIX' from LazyPOSIX.java:29:in `loadPOSIX' from LazyPOSIX.java:25:in `posix' from LazyPOSIX.java:159:in `isatty' from RubyIO.java:1897:in `tty_p' from RubyIO$i$0$0$tty_p.gen:65535:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from CallNoArgNode.java:63:in `interpret' from IfNode.java:111:in `interpret' from NewlineNode.java:104:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:147:in `call' from DefaultMethod.java:183:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from VCallNode.java:86:in `interpret' from NewlineNode.java:104:in `interpret' from BlockNode.java:71:in `interpret' from ASTInterpreter.java:74:in `INTERPRET_METHOD' from InterpretedMethod.java:169:in `call' from DefaultMethod.java:191:in `call' from CachingCallSite.java:302:in `cacheAndCall' from CachingCallSite.java:144:in `callBlock' from CachingCallSite.java:148:in `call' from RubyClass.java:822:in `newInstance' from RubyClass$i$newInstance.gen:65535:in `call' from JavaMethod.java:249:in `call' from CachingCallSite.java:292:in `cacheAndCall' from CachingCallSite.java:135:in `call' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:128:in `__file__' from /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:-1:in `load' from Ruby.java:697:in `runScript' from Ruby.java:690:in `runScript' from Ruby.java:597:in `runNormally' from Ruby.java:446:in `runFromMain' from Main.java:369:in `doRunFromMain' from Main.java:258:in `internalRun' from Main.java:224:in `run' from Main.java:208:in `run' from Main.java:188:in `main'

WELI · ‎11-25-2015

Following Neeraji suggestions, I stopped AMS first, made user no ams processes running, then restart AMS. Metric Monitors (agents) now all started, and the Metrics Collector is still stuck at 35% .... Saw the following output :

WELI · ‎11-24-2015

using ruok command to check, all 3 zookeeper processes are up and running: ./check_zookeeper.ksh imok imok imok

WELI · ‎11-24-2015

When trying to restart AMS from Ambari UI, I check the AMS log file: ambar-metric-monitor.out on Ambari sever host node: ----------------------------------------- 2015-11-24 13:47:30,987 [WARNING] emitter.py:80 - Retrying after 5 ... 2015-11-24 13:48:35,989 [INFO] emitter.py:91 - server: http://xxxx06t.xxxx.com:6188/ws/v1/timeline/metrics 2015-11-24 13:48:35,990 [WARNING] emitter.py:74 - Error sending metrics to server. <urlopen error [Errno 111] Connection refused> on the another node, which is the primary namenode and also as one of zookeeper nodes, the log file of AMS : ------------------------------------------------------------------------------------------------------------ 2015-11-24 13:43:50,822 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2015-11-24 13:43:50,822 WARN org.apache.zookeeper.ClientCnxn: Session 0x1513af962d30005 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2015-11-24 13:43:51,217 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2015-11-24 13:43:51,217 WARN org.apache.zookeeper.ClientCnxn: Session 0x1513af962d30005 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) The hbase-ams-master-xxx06t.xxxx.com.log has the same error message: -------------------------------------------------------------------------------------------- 2015-11-24 13:30:10,202 WARN [main-SendThread(localhost:61181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2015-11-24 13:30:10,206 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

WELI · ‎11-24-2015

Nothing is returned from from netstat -anp |grep 6188 when running on namenode. On Ambari server node (which is also our edge node), the output from the command is: [ambari@xxxxx~]$ netstat -anp |grep 6188 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) unix 3 [ ] STREAM CONNECTED 215461888 - /var/lib/sss/pipes/private/sbus-dp_centene.com.39654 unix 3 [ ] STREAM CONNECTED 215461883 - /var/lib/sss/pipes/private/sbus-dp_centene.com.39654 Thank you, Neeraj for your quick response.

WELI · ‎11-24-2015

Had a disk full issue. After making some space in /var then trying to restart Metric Collector from Ambari, got error: ----------error --------------- 2015-11-24 11:35:54,281 [INFO] controller.py:110 - Adding event to cache, : {u'metrics': [], u'collect_every': u'15'} 2015-11-24 11:35:54,281 [INFO] main.py:65 - Starting Server RPC Thread: /usr/lib/python2.6/site-packages/resource_monitoring/main.py start 2015-11-24 11:35:54,281 [INFO] controller.py:57 - Running Controller thread: Thread-1 2015-11-24 11:35:54,282 [INFO] emitter.py:45 - Running Emitter thread: Thread-2 2015-11-24 11:35:54,282 [INFO] emitter.py:65 - Nothing to emit, resume waiting. 2015-11-24 11:36:54,283 [INFO] emitter.py:91 - server: http://xxxxxxx.com:6188/ws/v1/timeline/metrics 2015-11-24 11:37:44,334 [WARNING] emitter.py:74 - Error sending metrics to server. timed out 2015-11-24 11:37:44,334 [WARNING] emitter.py:80 - Retrying after 5 ...

WELI · ‎11-20-2015

Yes after we use full path. And we also change owner to root. Then worked. Thanks,

WELI · ‎11-20-2015

We started SmartSense server fine. But for the agents, they started but after a few seconds, went down again. See the screen capture for error in hst-agent.log. smartsense-error-in-agent-hst.pdf O/S: RHEL 7, HDP 2.3.2.0-2950 Ambari 2.1.2

WELI · ‎11-20-2015

We changed the rpm's owner to root and we copy and paste the full path, now it works. Thank you for your help.

WELI · ‎11-20-2015

When running the command " hst add-to-ambari " , following the prompt we added the path to rpm and receive the error. We are using ambari 2.1.2 and HDP 2.3.2.0-2950, and the rpm is: smartsense-hst-1.1.0-119.x86_64.rpm. Our O/S is RHEL 7.

Online	Offline
Last Visited	‎01-03-2016 04:29 AM

Date Registered	‎11-17-2015 09:37 PM
Last Visited	‎01-03-2016 04:29 AM
Total Messages Posted	28
Total Kudos Received	9

Re: How to run Nifi without Visual Designer ? Is t...

Re: I get an IllegalArgumentException error when t...

Re: pig -useHCatalog simple.bulkload.pig failed

pig -useHCatalog simple.bulkload.pig failed

Re: Ranger HA - Documentation

Re: Ranger HA - Documentation

Re: I get an IllegalArgumentException error when t...

Re: Ranger user sync error: javax.naming.Authenti...

Re: Ranger user sync error: javax.naming.Authenti...

Re: Ranger user sync error: javax.naming.Authenti...

Re: Ranger user sync error: javax.naming.Authenti...

Re: Ambari Ldap sync error: javax.naming.InvalidN...

Re: Ambari Ldap sync error: javax.naming.InvalidN...

Re: Ambari Ldap sync error: javax.naming.InvalidN...

Re: Ranger user sync error: javax.naming.Authenti...

Ranger user sync error: javax.naming.Authenticati...

Ambari Ldap sync error: javax.naming.InvalidNameE...

Re: HBase shell error: java.lang.UnsatisfiedLinkEr...

HBase shell error: java.lang.UnsatisfiedLinkError

Re: Ambari Metric Collector: Error sending metric ...

Re: Ambari Metric Collector: Error sending metric ...

Re: Ambari Metric Collector: Error sending metric ...

Re: Ambari Metric Collector: Error sending metric ...

Ambari Metric Collector: Error sending metric to s...

Re: Error: Unsupported distributable during SmartS...

SmartSense Agent failed to stay up

Re: Error: Unsupported distributable during SmartS...

Error: Unsupported distributable during SmartSense...