@Ani73    Azure controls access to different port through the firewall easily configured from the networking pane. You should locate the VM running Ambari Home --> Resource Group-->filter type and look for the Ambari VM  and under settings locate the networking it will open a network interface pane on the right where you set the inbound and outbound rules. See attached screenshot   If you are not a network expert the best would be to lock the inbound port rules  with source *  and Destination * to your IP which you can get from the URL www.whatismyipaddress.com ... View more

@moubaba  Here is a fantastic document by Artem Ervits  Ambari Views REST API Overview Hope that answers your query    Happy hadooping ... View more

@RjsChw  The error you are encountering is Oracle related you should validate the username password against the oracle database. If you have an oracle client installed on your laptop do the below there are many variations. But let your DBA give you the username/password for the database you are trying to export. Variant 1 sqlplus /nolog Connect user/password@dbname Variant 2 sqlplus user@orcl Variant 3 sqlplus user/password@hostname:port/sid ----------- ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: ORA-01017: invalid username/password; logon denied java.sql.SQLException: ORA-01017: invalid username/password; logon denied   Having said that Sqoop on the command line will display your password in clear text, that's not secure so below is a way to encrypt your password so that your sqoop jobs are secured from prying eyes, to do that you will use the hadoop package to encrypt your oracle credential.   In the below example I am creating a password for my fictitious  testDB and using the name in the alias to easily identify it from the 100's of db's. In the below example I am using MySQL database the alias doesn't matter ie oracle.testDB.alias or db2.testDB.alias, the most important is the password it should match the password of the user of the Oracle/MySQL/dbs user   Encrypting SQOOP password Generating the jceks file you MUST provide a path to your hdfs home, create one before executing this command $ hadoop credential create mysql.testDB.alias -provider jceks://hdfs/user/george/mysql.testDB.password.jceks Enter password: [database_password] Enter password again: [database_password] mysql.testDB.alias has been successfully created. org.apache.hadoop.security.alias.JavaKeyStoreProvider has been updated. Validating the encyrpted password creation The encrypted password jceks will be MUST be written to your hdfs home $ hdfs dfs -ls /user/george Found 1 items -rwx------ 3 george hdfs 503 2018-09-02 01:40 /user/george/mysql.testDB.password.jceks   Running the sqoop with the jceks alias Assumption my mysql database test is running on host is pomme.cloudera.com port 3306 $ sqoop import -Dhadoop.security.credential.provider.path=jceks://hdfs/user/george/mysql.testDB.password.jceks --driver com.mysql.jdbc.Driver --connect jdbc:mysql://pomme.cloudera.com:3306/test --username george --password-alias mysql.testDB.alias --table "customer" --target-dir /user/george/test Success output Warning: /usr/hdp/2.6.2.0-205/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 18/09/02 02:08:04 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.2.0-205 18/09/02 02:08:06 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time. 18/09/02 02:08:06 INFO manager.SqlManager: Using default fetchSize of 1000 18/09/02 02:08:06 INFO tool.CodeGenTool: Beginning code generation .......... some text removed here.............. 18/09/02 02:08:18 INFO mapreduce.Job: The url to track the job: http://pomme.cloudera.com:8088/proxy/application_1535835049607_0002/ 18/09/02 02:08:18 INFO mapreduce.Job: Running job: job_1535835049607_0002 18/09/02 02:08:55 INFO mapreduce.Job: Job job_1535835049607_0002 running in uber mode : false Total megabyte-milliseconds taken by all map tasks=917431296 Map-Reduce Framework Map input records=2170 Map output records=2170 Input split bytes=396 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=944 CPU time spent (ms)=11690 Physical memory (bytes) snapshot=669270016 Virtual memory (bytes) snapshot=18275794944 Total committed heap usage (bytes)=331350016 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=243892 18/09/02 02:11:48 INFO mapreduce.ImportJobBase: Transferred 238.1758 KB in 218.8164 seconds (1.0885 KB/sec) 18/09/02 02:11:48 INFO mapreduce.ImportJobBase: Retrieved 2170 records.   Sqoop import in hdfs Check the export was successful $ hdfs dfs -ls /user/george/test Found 5 items -rw-r--r-- 3 george hdfs 0 2018-09-02 02:11 /user/george/test/_SUCCESS -rw-r--r-- 3 george hdfs 60298 2018-09-02 02:10 /user/george/test/part-m-00000 -rw-r--r-- 3 george hdfs 60894 2018-09-02 02:10 /user/george/test/part-m-00001 -rw-r--r-- 3 george hdfs 62050 2018-09-02 02:11 /user/george/test/part-m-00002 -rw-r--r-- 3 george hdfs 60650 2018-09-02 02:11 /user/george/test/part-m-00003 Check the values in he splits $ hdfs dfs -cat /user/george/test/part-m-00000 1,Julian Stuart,sagittis.felis@sedhendrerit.com,Suspendisse Tristique Neque Associates,9230 Metus. Av.,Pemberton,Mexico 2,Ferris Fulton,condimentum@morbitristique.co.uk,Nunc Ltd,256-788 At Avenue,Northampton,China 3,Byron Irwin,adipiscing.Mauris@DonecnibhQuisque.edu,Nascetur Ridiculus Foundation,4042 Non, St.,Gattatico,Lithuania ..........................some text removed ................ 18,Peter Middleton,purus.Nullam.scelerisque@egetdictumplacerat.com,Erat In Consectetuer Associates,1618 Donec St.,Grand Island,Thailand   Voila      ... View more

@nirajp  Either way HIVE or Beeline you MUST provide username /password to authenticate to be able to execute any SQL statement against the DB. See below examples Hive CLI [hive@calgary ~]$ hive .......... SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://calgary.canada.ca:2181,ottawa.canada.ca:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Enter username for jdbc:hive2://calgary.canada.ca:2181,ottawa.canada.ca:2181/default: hive Enter password for jdbc:hive2://calgary.canada.ca:2181,ottawa.canada.ca:2181/default: ****   Beeline Connection [hive@london ~]$ beeline Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive beeline> ! connect jdbc:hive2://london.tesco.co.uk:10000/;principal=hive/london.tesco.co.uk@TESCO.CO.UK Connecting to jdbc:hive2://london.tesco.co.uk:10000/;principal=hive/london.tesco.co.uk@TESCO.CO.UK Enter username for jdbc:hive2://london.tesco.co.uk:10000/;principal=hive/london.tesco.co.uk@TESCO.CO.UK:xxxxx Enter password for jdbc:hive2://london.tesco.co.uk:10000/;principal=hive/london.tesco.co.uk@TESCO.CO.UK:xxxxx Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37) Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://london.tesco.co.uk:10000/> show databases; +----------------+--+ | database_name | +----------------+--+ | default                   | | uxbribge               | | White_city          | +----------------+--+ 3 rows selected (2.863 seconds)   If you have ranger plugin enable for hive then you will have authorization centrally handles by Ranger. HTH    ... View more

@saivenkatg55    This is nor a silver bullet but worth trying,your ambari database could be overwhelmed scanning through old data to help you narrow down the problem please do the below steps   Stop the Ambari Server by using ambari-server stop. # ambari-server stop   Run db-purge-history Use the cotrrect data format for your server # ambari-server db-purge-history --cluster-name [PROD] --from-date 2016-04-01   Start the Ambari Server: by using ambari-server start . # ambari-server start Ref: Tuning Ambari server performance Please revert l ... View more

@Elephanta    Okay, now with that information I get a better understanding and picture. By default, HDP 2.6 has a replication factor of 3  so it's looking to place the other 2 copies on different data nodes that the existing unless you create new files with a replication factor of 1 you will continue to get the unreplicated block errors 🙂 but now that you know it's manageable. Maybe next time you delete files in hdfs use the -skipTrash option hdfs dfs -rm -skipTrash /path/to/hdfs/file/to/remove/permanently or emtpyting the existing .Trash Options: Change the replication factor of a file  at creatio: hdfs dfs –setrep –w 1  /user/hdfs/file.txt   Or  change the replication factor of a directory  hdfs dfs -setrep -R 1  /user/hdfs/your_dir Changing the replication factor for a directory will only affect the existing files and the new files under the directory will get created with the default replication factor  in dfs.replication from hdfs-site.xml  maybe in your case this is what you should change to 1 as this takes effect cluster-wide for your Dev environment Happy hadooping        ... View more