Community Articles

SYMPTOM:user complains that he is seeing some special characters (^@^@^@^@^@^@^) in GC logs similar to thisconcurrent-mark-sweep perm gen total 29888K, used 29792K [0x00000007e0000000, 0x00000007e1d30000, 0x0000000800000000)^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2016-12-10T00:09:31.648-0500: 912.497: [GC2016-12-10T00:09:31.648-0500: 912.497: [ParNew: 574321K->15632K(629120K), 0.0088960 secs] 602402K->43713K(2027264K), 0.0090030 secs]ROOT CAUSE:there was some query which was running on hiveserver2 in mr mode which spins Map Join act

SYMPTOM: HDFS service is not able to start and throwing below error in logs -python error " File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 124, in create_ranger_repository" ERROR: Ambari operation logs shows below message -stderr: /var/lib/ambari-agent/data/errors-22280.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 433, in <module> NameNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 524, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 102, in start namenode(action="start", hdfs_binary=hdfs_binary, upgrade_type=upgrade_type, env=en

SYMPTOM: Ambari smoke test fails for hbase service. Below is the current scenario - Ranger is installed in the cluster HBase policy has been enabledambari-qa user has the privileges correctly defined in the HBase policyERROR: ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=ambari-qa, scope=default, params=[namespace=default,table=default:ambarismoketest,family=family],action=CREATE)2015-10-27 09:52:03,342 ERROR [main] client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.TableNotFoundException: Table 'ambarismoketest' was not found, got: XXXXX01. ROOT CAUSE: If the Ranger co-processor is not correctly defined in the HBase configuration, the smoke test from Ambari would fail. Any table creation as non-hbase user could also fail.RESOLUTION:Verify the Ranger configuration for HBase. Ensure that the following properties are set correctly and that the co-processors include Ranger classeshbase.coprocessor.master.classeshbase

SYMPTOM: Ranger plugin is enabled for HIVE. Restarting HIVE service its not able to start and stucking on below error ERROR: 2015-10-15 13:02:51,683 - u"File['/var/lib/ambari-agent/data/tmp/ojdbc6.jar']" {'content': DownloadSource('http://sjcservicenode04-prod.xxxinternal.com:8080/resources//oracle-jdbc-driver.jar')} 2015-10-15 13:02:51,796 - Not downloading the file from http://sjcservicenode04-prod.xxxinternal.com:8080/resources//oracle-jdbc-driver.jar, because /var/lib/ambari-agent/data/tmp/oracle-jdbc-driver.jar already exists 2015-10-15 13:02:51,996 - call['hdp-select status hadoop-client'] {'timeout': 20} ROOT CAUSE: Ranger Hive policies http url calls were taking forever to return results Ranger makes a lot of calls to urllib2.urlopen(request) that don't have a timeout in Ambari 2.0 Opened Ambari BUG to put time=5 in the ranger_function.py file. https://hortonworks.jira.com/browse/BUG-46275   RESOLUTION: 1) Edit /usr/lib/python2.6/site-packages/resource-management/librar

Scenario: Lets say you have 2 ranger admin instances configured in your cluster. Now if you want to enable Ranger HA, you need to delete one of the ranger admin instance[since Ranger HA admin wizard will create 2nd instance of ranger for you] In such case you need to remove one instance of Ranger admin which is already installed. The following steps will guide you how to remove Ranger Admin instance using Ambari API Backup Ambari Server database [https://ambari.apache.org/current/installing-hadoop-using-ambari/content/ambari-chap11-1.html] Stop the Ranger service using Ambari. In case the Ranger Admin fails to stop, try stopping the Ranger Service as follows using Ambari API: curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' \http://xxx.hostname:8080/api/v1/clusters/TEST/services/RANGERcurl -u admin:admin -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop Service"},

SYMPTOM: User was not able to browse ambari UI after ambari server restart. Ambari version : 2.1.2Below was the error seen in logsERROR: 06 Jul 2016 09:40:26,505 ERROR [Stack Version Loading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.1 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out) 06 Jul 2016 09:40:26,506 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.2 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:28,508 ERROR [Stack VersionLoading Thread] LatestRepoCallable:93 - Could not load the URI for stack HDP-2.2 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (connect timed out)06 Jul 2016 09:40:28,509 INFO [Stack Version Loading Thread] LatestRepoCallable:74 - Loading latest URL info for stack HDP-2.3 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 06 Jul 2016 09:40:30,511 ERROR [Stack Version Loading Thread] Lat

IntroductionThe producer sends data directly to the broker that is the leader for the partition without any intervening routing tier.Optimization ApproachBatching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger batches in a single request. The batching can be configured to accumulate no more than a fixed number of messages and to wait no longer than some fixed latency bound (say 64k or 10 ms). This allows the accumulation of more bytes to send, and few larger I/O operations on the servers. This buffering is configurable and gives a mechanism to trade off a small amount of additional latency for better throughput.In order to find the optimal batch size and latency, iterative test supported by producer statistics monitoring is needed.Enable MonitoringStart the producer with the JMX parameters enabled:JMX_PORT=10102 bin/kafka-console-producer.sh --broker-list localhost:9092--topic testtopic

While accessing the Zeppelin View is shows the "zeppelin service is not running". But when we check at the back end then we find that the Zeppelin server was running. PID file has the correct PID information mentioned in it.- As we see that Zepplin Notebok is running via ambari as well. But still when we try to access the Zeppelin View it shows failure saying "Zeppelin service is not running".We will need to check the Zeppelin log to see if it shows any error? We see the following kind of error:INFO [2016-12-01 09:42:08,926] ({main} ZeppelinServer.java[setupWebAppContext]:266) - ZeppelinServer Webapp path: /usr/hdp/current/zeppelin-server/webapps INFO [2016-12-01 09:42:09,331] ({main} ZeppelinServer.java[main]:114) - Starting zeppelin server INFO [2016-12-01 09:42:09,333] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210 WARN [2016-12-01 09:42:09,367] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@69b794e2{/,null,null}{/usr/hdp/curr

Ambari allows its users to have different configurations for different hosts for different components via configuration groups. Ambari initially assigns all hosts in your cluster to one, default configuration group for each service you install. When using Configuration groups, it enforces configuration properties that allow override, based on installed components for the selected service and group.For more information on this please refer to: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_config_groups.htmlHere we will try to make a very small change in "flume-conf" content and will apply it to specific host ("erie3.example.com") where as all the other hosts will be using the default "flume-conf" configuration. We can simply use the following ambari API to list all the config_groups.http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/config_groupsExample:http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups.Step-1).

SYMPTOMS: Sometimes the Hive CLI hangs and the gives no response. At the same time, the /var/log/hive/hivemetastore.log reports timeoutsERROR [<hostname>-47]: txn.TxnHandler (TxnHandler.java:getDbConn(984)) - There is a problem with aconnection from the pool, retrying(rc=9): Timed out waiting for a free available connection.(SQLState=08001,ErrorCode=0) java.sql.SQLException: Timed out waiting for a free available connection.ROOT CAUSE: The Hive Metastore service is waiting for the database connections to get free. At the time of error, not enough concurrent connections were available. RESOLUTION: (Note: This is for mysql DB only).To fix this problem, the number of max database connection should be increased.1. Open the /etc/my.cnf in the text editor.vi /etc/my.cnf2. Under [mysqld] section, add:max_connections = 2503. Save the file and restart the mysqld serviceservice mysqld restart

HDP Stack Version: 2.4.0SYMPTOMS: WARN Error while fetching metadata [{TopicMetadata for topic <topic-name> -> No partition metadatafor topic <topic-name> due to kafka.common.TopicAuthorizationException}] for topic <topic-name>: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) .... ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: <topic-name> (kafka.producer.async.DefaultEventHandler) ROOT CAUSE: At the moment we can’t use user/group based access to authorize Kafka access over a non-secure channel. This is because it is not possible to assert client’s identity over the non-secure channel. It is ip based in a non-secure environment. Reference: https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin See: Authorizing Kafka access over non-authenticated channel via Ranger RESOLUTION: The policy doesn't work if, IP address is not configured. This address will

Run hive shell debug so it prints detailed error: hive -hiveconf hive.log.file=hivecli_tez.log -hiveconf hive.log.dir=/tmp/hivecli -hiveconf hive.execution.engine=tez -hiveconf hive.root.logger=DEBUG,DRFA SYMTOMS: DEBUG [main]: amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1152)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: null; Status Code: 403; Error Code: 403 Forbidden; Request ID: 85BA6566D33A519B), S3 Extended Request ID: 228pqAjcCjTHo+ExpZ+86INHAhkIeE+DQoicPLkan8GDaraxsklIuHwK3f+QmjtIBzw/z5OSWaM= WARN [main]: avro.AvroSerDe (AvroSerDe.java:determineSchemaOrReturnErrorSchema(169)) - Encountered AvroSerdeException determining schema. Returning signal schema to indicate problem org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read schema from given path: s3a://<file-path> WORKAROUND: 01. mkdir joda-backup 02. cd /tmp 03. wget http://central.maven.org/maven2/joda-time/joda-time/2.8.1/joda-time-

SYMTOMPS:Ambari server log shows:WARN [qtp-client-41858] ObjectGraphWalker:209 - The configured limit of 1,000 object references was reached while attempting to calculate the size of the object graph. Severe performance degradation could occur if the sizing operation continues. This can be avoided by setting the CacheManger or Cache <sizeOfPolicy> elements maxDepthExceededBehavior to "abort" or adding stop points with @IgnoreSizeOf annotations. If performance degradation is NOT an issue at the configured limit, raisethe limit value using the CacheManager or Cache <sizeOfPolicy> elements maxDepth attribute. For more information, see the Ehcache configuration documentation. WORK-AROUND Disable cache by setting below property in in /etc/ambari-server/conf/ambari.propertiesserver.timeline.metrics.cache.disabled = true ROOT CAUSE See: AMBARI-13517 Disabling cache is not going have adverse effect. It was introduced in Ambari 2.1.2 to create a Caching layer that provides sli

Setting up a Websocket Client and Server with Apache NiFi 1.1.I wanted to test out the new WebSocket Listener in Apache NiFi 1.1, but I needed a server to serve up my HTML client. So I ran that web server with NiFi as well. So my full solution is hosted and runs through Apache NiFi.This simple WebSockets Server and Client does the hello world of web sockets, Echo! Whatever the client sends, we send it back.My Suggested Use Cases for WebSocketsWebSocket Client to Slack InterfaceWebSocket Client to Email WebSocket Chat stored to Apache PhoenixWebSocket To Communicate From Mobile Web AppsWebSocket To Send Detailed Log Details From Enterprise Web Applications Directly To Log Ingest Platform, bypassing local filesystem.Step 1: HandleHTTPRequest This accepts the HTTP calls from browsersStep 2: ExecuteStreamCommand Returns HTML page (could do getfile or any number of other ways of getting the HTML as a flowfile)Step 3: HandleHttpResponse this serves up our web page to browser

Steps:Oozie server timezone For Ambari users, login and navigate to Custom Oozie-site.Ambari > Oozie > configs > custom oozie-site Add Property "oozie.processing.timezone=GMT-0500"oozie.processing.timezone Default value=UTCOozie server timezone. Valid values are UTC and GMT(+/-)####, for example 'GMT+0530' would be India timezone. All dates parsed and generated dates by Oozie Coordinator/Bundle will be done in the specified timezone. The default value of 'UTC' should not be changed under normal circumstances. If for any reason is changed, note that GMT(+/-)#### timezones do not observe DST changes. Save and Restart Oozie ServiceOozie Web ConsoleTo View the Job in EST Timezone on Oozie Web Console, follow below steps. Open Oozie Web Console on

PROBLEM: Unable to start Resource Manager which fails with below errors:-STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9e75108092247d96ce7d70839b6945e9eba2a0b7; compiled by 'jenkins' on 2014-11-04T04:31ZSTARTUP_MSG: java = 1.7.0_67************************************************************/2014-11-04 08:41:08,705 INFO resourcemanager.ResourceManager (SignalLogger.java:register(91)) - registered UNIX signal handlers for [TERM, HUP, INT]2014-11-04 08:41:10,636 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to loginorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to loginat org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main

SYMPTOM: RM is down due to below error. Earlier we were suspicion the ulimit could be culprit though we have increased it to 128K. But still no luck.ERROR: 2016-07-25 12:19:47,125 WARN security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(873)) - Unable to add the application to the delegation token renewer. java.lang.OutOfMemoryError: unable to create new native thread. Below was few steps followed -1. Checked the error and saw that previously the same issue and increasing ulimit resolved the issue. 2. Checked the ulimit and lsof output - $ulimit -n 131072 $lsof |grep yarn |wc 1726 15553 242741 3. Checked the heap size for yarn process which was set to 8Gb and looks good.Below error was displayed in RM out.log fileJava HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f89641cf000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)## There is insufficient memory for the Java Runtime Environment to continue.# Native memory

SYMPTOM:The following jmx tool command is failing with "port already in user" error/usr/hdp/current/kafka-broker/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec --jmx-url service:jmx:rmi:///jndi/rmi://`hostname`:1099/jmxrmiROOT CAUSE:JMX tool invoke the kafka-run-class.sh after reading kafka-env.sh, since we have already have export $JMX_PORT=1099 there then it tried to bind the same host while invoking the jmx tool which result into error.WORKAROUND:NARESOLUTION:edit kafka-run-class.shReplace section # JMX port to use if [ $JMX_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" fi To the following: Changed security protocol to the correct ones and kafka cli script as follows. # JMX port to use if [ $ISKAFKASERVER = "true" ]; then JMX_REMOTE_PORT=$JMX_PORT else JMX_REMOTE_PORT=$CLIENT_JMX_PORT fi if [ $JMX_REMOTE_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management

Consider increasing network capacity to overcome the challenge caused due to non locality of blocks.Create configuration groups of datanodes exclusively for HBASE, disabling HDFS balancer on this group and allow only hbase balancer. Follow this url Host_Config_Groups to create host config groups.Few temporary workarounds can also be applied if problem is severe and need immediate attention :-Disable HDFS balancer permanently on the cluster and run it manually on need basis. (Please spin a support case and have the situation discussed before implementing this workaround.)In case the performance issue needs to be fixed post running of HDFS Balancer, a major compaction could be manually initiated. For performance gains, major compaction is run on off peak hours such as weekends. This article Compaction_Best_Practices is a recommended read here.Scheduling major compaction after scheduled balancer rather than vice versa.HDFS although has introduced concept of "favored nodes" feature but HBa

PROBLEM Running sqoop import command in direct mode for accessing a Netezza data warehouse appliance hangs at 100% mapsqoop import --options-file sqoop_opts_file.opt...INFO mapreduce.Job: Running job: job_1465914632244_0005INFO mapreduce.Job: Job job_1465914632244_0005 running in uber mode : falseINFO mapreduce.Job: map 0% reduce 0%INFO mapreduce.Job: map 25% reduce 0%INFO mapreduce.Job: map 50% reduce 0%INFO mapreduce.Job: map 100% reduce 0% The sqoop_opts_file.opt had the following options :-connectjdbc:netezza://xxxxxxxxxxxxxxxxxxxxxx:5480/--usernameXXXX --password***************--direct--direct-split-size1000000--compress--tabletable_name--target-dir/user/root/table_name--verbose Yarn logs show the below errorsERROR [Thread-14] org.apache.sqoop.mapreduce.db.netezza.NetezzaJDBCStatementRunner: Unable to execute external table exportorg.netezza.error.NzSQLException: ERROR: found delim ',' in a data field, specify escapeChar '\' option in the external table definition RESOLUTION

PROBLEM :The example workflow submitted by users was failing with the below ClassNotFound exceptionsjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.example.SampleMapper not found ROOT CAUSE : These classes are found in oozie-examples-<version>.jar file, which is expected to be present in the lib folder of the job. RESOLUTION :Add the oozie-examples-<version>.jar file to the lib folder of the job and then submit the job.

PROBLEMWhen we query hbase tables through hive, it always creates a fetch task instead of running a MR task. The parameterhive.fetch.task.conversion.threshold controls whether a fetch task would run or a Map Reduce. If hive.fetch.task.conversion.thresholdis less than the table size, it will use MapReduce Job. The default value of the above parameter is 1GB.Create a 'hbase_hive' external table in hive, make sure the hbase table is more than 1GB. [root@node1 ~]# hadoop fs -du -s -h /apps/hbase/data/data/default/hbase-hive 3.4 G /apps/hbase/data/data/default/hbase-hiveFrom beeline analyze the explain plan, which launches a fetch task instead of Map Reduce job, even when the size of the table is more than 1GB0: jdbc:hive2://node1.hwxblr.com:10000/> explain select * from hbase_hive where key = '111111A111111' ; +----------------------------------------------------------------------------------------------------------+--+ | Explain | +------------------------------------------------------

SYMPTOM: User has latest HDP integrated with kerberos. While starting the datanode user gets the message: Login failure for dn/host1@EXAMPLE.NET from keytab /etc/security/keytabs/dn.service.keytab. But the principal is dn/host1.bc@EXAMPLE.NETWhere host1 is the hostname of the datanode host andEXAMPLE.NET is the REALM name.ERROR: The output of klist command is as below -$klist -kt /etc/security/keytabs/dn.service.keytab Keytab name: FILE:/etc/security/keytabs/dn.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 0 12/21/2016 10:38:13 dn/host1.bc@EXAMPLE.NETIn logs it shows - dn/host1@EXAMPLE.NETWhere as it should show - dn/host1.bc@EXAMPLE.NET ROOT CAUSE: This is issue with entries in /etc/host file.RESOLUTION: User has below entry in /etc/hosts file -<ipaddress> <hostname> <FQDN> <FQDN> Now the order is changed to <ipaddress> <FQDN> <hostname> <FQDN> Which resol

SYMPTOM: there are lots of heap dump file created under the directory controlled by java flag -XX:HeapDumpPath, user complains that HiveServer2 has generated these dump since this flag was setup by him for HiveServer2.looking at the HiveServer2 logs, there was no trace for startup and stop services. even there was no sign of any type of failure in the logs.ROOT CAUSE:At very preliminary analysis of heap dump I examine the process arguments which looks like this hive     12345 1234  9.8 27937992 26020536 ?   Sl   Dec01 19887:12  \_ /etc/alternatives/java_sdk_1.8.0/bin/java -Xmx24576m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhdp.          version=2.3.4.20-3 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.2.-258/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.   path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.-258/h

Question: We added the following settings to our knox topology so that knox may search in ldap for the groups that a user is in, so that we can efficiently control knox permissions with ranger.<!-- changes needed for group sync--> <param> <name>main.ldapGroupContextFactory</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> </param> <param> <name>main.ldapRealm.authorizationEnabled</name> <value>true</value> </param> <param> <name>main.ldapRealm.groupSearchBase</name> <value>OU=Infrastructure,OU=EXAMPLE,DC=test,DC=local</value> </param> <param> <name>main.ldapRealm.groupObjectClass</name> <value>group</value> </param> <param> <name>main.ldapRealm.groupIdAttribute</name> <value>cn</value> </param> Our search base and attribute settings are the same as those we have in our ha