Q&A

sqoop import --connect "jdbc:jtds:sqlserver://BJHBLSNR:1496;useNTLMv2=true;domain=BST;databaseName=HTG" --username DM -P  --table PROVIDER_STG --split-by PROV_CODE_BK --target-dir /test/dd -m 1    ERROR tool.ImportTool: Import failed: org.apache.hadoop.security.AccessControlException: Permission denied: user=cloudera_user, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x  

Hi, I tried looking in to the community but could not get proper answer for this question. How can I change the block size for the existing files in HDFS? I want to increase the block size. I see the solution as distcp and I understood that we have to use distcp to move the files, folders and subfolders to a new temporary location with new block size and then remove the files, folders, etc for which block size has to be increased and copy the files from temporary location back to original location. The above methodology might have side effects such as overhead of HDFS by adding duplicate copies of the files and change in permissions while copying files from temporary location and etc. Is they any way which is efficient enough to replace the existing files with the same name and same privileges but with increased block size? Thanks to all for your time on this question.

Zookeeper 3-node cluster in docker (DockerHub zookeeper:3.6.1). Individual services deployed as stack. docker-compose.yml:zookeeper1: image: zookeeper:3.6.1 hostname: "zookeeper1" environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=zookeeper3:2888:3888;2181 ALLOW_ANONYMOUS_LOGIN: "yes" ZOO_4LW_COMMANDS_WHITELIST: "*" ZOO_STANDALONE_ENABLED: "false" ZOO_INIT_LIMIT: 20 ZOO_SYNC_LIMIT: 10 networks: - kafka-net volumes: - zookeeper-data:/data/ - zookeeper-datalog:/datalog/ zookeeper2: image: zookeeper:3.6.1 hostname: "zookeeper2" environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zookeeper3:2888:3888;2181 ALLOW_ANONYMOUS_LOGIN: "yes" ZOO_4LW_COMMANDS_WHITELIST: "*" ZOO_STANDALONE_ENABLED: "false" ZOO_INIT_LIMIT: 20 ZOO_SYNC_LIMIT: 10 networks:

Hello, I'm using a HDP 3.1 Kerberized HDP and I've a problem when using "regenerating keytabs" button : I've 16 regenerated keytabs and 5 keytabs not regenerated. The 5 keytabs that are not updated :- ambari-infra-sol.service.keytab- hive.service.keytab- smokeuser.headless.keytab- spnego.service.keytab- yarn.service.keytab I have no error in the logs regarding problem with keytab generation or keytab deployment. I can restart all the services and everything is working fine but how can I have ALL keytabs regenerated ? thanks for your help,    

Hi,I recently installed ranger by building it from source.We also have ambari installed and was wondering if anyone can help redirect me to somewhere that explains how to manage Ranger using Ambari. Thanks,

I need your advice, I need to insert data into salesforce but I have a lookup to do to verify the existence of the data before adding it.Me, I had considered creating a tempo database (PosgresSql) in order to put the contents of the salesforce object being the object of lookup to verify first and then to make the join to update the contents. in salesforce.1- NIFI to PosgresSql2- Salesforce to PosgreSql3- Join data to SalesforceWhat do you think? what do you think is the best method?

Hello,I am trying to download a nifi template which has around 260 processors. Whenever I try to download the template in chrome browser, the downloading stops after 320kb and throws me an error which says "Failed - Network Error". Since there are around 260 processors, it is understandable that the template size is considerably big however I can download other smaller templates from the same Nifi instance. I even tried downloading the template with other browsers but the error is same.Is there any configurations which needs to be done from the server side? Any insights are appreciated@mburgess @bbende @TimothySpann 

Hi all,we are executing pyspark and spark-submit to kerberized CDH 5.15v from remote airflow docker container not managed by CDH CM node, e.g. airflow container is not in CDH env. Versions of hive, spark and java are the same as on CDH. There is a valid kerberos ticket before executing spark-submit or pyspark.Python script: from pyspark.sql import SparkSession, functions as Fspark = SparkSession.builder.enableHiveSupport().appName('appName').getOrCreate()sa_df=spark.sql("SELECT * FROM lnz_ch.lnz_cfg_codebook") Error is: To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.3.0 /_/Using Python version 3.6.12 (default, Oct 13 2020 21:45:01)SparkSession available as 'spark'.>>> from pyspark.sql import SparkSession, functions as F>>> spark = SparkSession.builder.enableHiveSupport().app

Hi, I am having trouble getting the data provenance displaying in a clustered environment in Nifi. I have set all the required access policies and still the data provenance is empty. Any suggestions at this point is very much appreciated. 

What is the default character set used within Nifi while extracting from Oracle using executeSQL ?I have a use case of 2 database in 2 different character sets, was just curious, is there any way to identify the incoming character set & then over-ride it based on the target ?

Hi all,I am curious to know if the BDR schedules sync deletes as well? Meaning, if some files are deleted from the source, will the next BDR run delete those files from the destination as well to make the source and destination exact copies?Thanks!

Hi,  I have Cloudera Enterprise Data Hub Edition trial version installed in my environment. So I am getting an message like below when I login into my cloudera manager. You have 3 days left to try Cloudera Enterprise Data Hub Edition.After that, you will need a valid license, or you can revert to using Cloudera Express. Your cluster will remain unaffected.So will I be able to login to my cloudera manager after the license expire?,will this effect my Hadoop cluster?How can I revert to using Cloudera Express? Thanks You!

Hi, we are running a HDP 3.1 production cluster with 2 Master Nodes and 54 Data Nodes;Spark version is 2.3 and yarn is the cluster manager. Each data node has about 250GB RAM and 56 Cores. We are using a combination of Nifi and Spark to set up a Hive DWH as follows:Nifi picks input csv files from source and loads them to HDFS in parquet formatSpark jobs picks these files and loads them to hive managed tablesAnother set of Spark jobs aggregates the data in the hive managed tables to hourly and daily hive managed tables.I noticed that my Spark jobs were very slow at times when writing to hive. I ran a test where i ran the same spark-submit job several times and it took between 10-13 minutes (fast) to 35-40 minutes (slow) to run the same job with the exact same parameters. My spark submit job: sudo /usr/hdp/3.1.0.0-78/spark2/bin/spark-submit --files /etc/hive/3.1.0.0-78/0/hive-site.xml,/etc/hadoop/3.1.0.0-78/0/core-site.xml,/etc/hadoop/3.1.0.0-78/0/hdfs-site.xml --driver-class-p

Hi,          I've pretty much read through all the articles and questions about this, but i'm still having issues. the ldap user is definitely there and i can query through ldap port:389.  I've set up the HUE ldap to authenticate against my openldap server, it works great. But i couldn't get the impala ldap setup to work. I'm using CDH 6.3.    I configured impala ldap settings through CM gui, services restarted okay after the config   Advanced Configuratoin snippet(safety valve) :--ldap_passwords_in_clear_ok=true Enable LDAP Authentication: checked LDAP URL: ldap://myldapFQDN:389 LDAP BaseDN: ou=users,dc=ldap,dc=xxx,dc=com     Error when using cloudera impala jdbc driver to connect, it worked with AuthMech=0 before I enabled LDAP drv <- RJDBC::JDBC("com.cloudera.impala.jdbc4.Driver", "/opt/jars/ImpalaJDBC4.jar")con <- dbConnect(drv, "jdbc:impala://impalaDaemonIP:21050;AuthMech=3;UID=myuser;PWD=mypassword") Error in .jca

Hello allI have task and not find solutions, have to comparing configuration between cluster (CDH 6.1.1) and everything i found shows using Apache Ambari but i dont have this tool and must using different way. Do you know this tool/script??

Hello All,I need your support , I have Cloudera version 5.15 connected to Data source Oracle 11.2.0.3 and Database Team want to Upgrade it to Oracle 19.3C, can someone tell me if My cloudera version will still compatible with the new Oracle DB version or not ? I searched lot in google but didn't find what I want. Thank you in Advance

Hello,I had my CCA122 exam during which I was not able to solve two questions due to repo was not accessible. I tried manually and even from browser, I was getting not found and 403 for HDP and ambari repo. Same was informed and noted by the proctor.As per Cloudera exam policy, if any technical issue is encountered my exam should be reset but they have marked me failed.What could I do in such situation? As per my understanding, Repo should be managed by Cloudera during exam

I am trying to copy CSV files from my local directory into a SQL Server database running in my local machine by using Apache NiFi. I am new to the tool and I have been spending few days googling and building my flow. I managed to connect to source and destination but still I am not able to populate the database since I get the following error: "None of the fields in the record map to the columns defined by the tablename table." I have been struggling with this for a while and I have not been able to find a solution in the Web.Any hint would be highly appreciated. Here are further details. I have built a simple flow using GetFile and PutDatabaseRecord processors My input is a simple table with 8 columns My configurations for GetCSV process are here (I have added the input directory and left the rest as default) The configuration for PutDatabaseRecord process is here (I have referred to the CSVReader and DBCPConnectionPool controller services, used the

How to enable Authentication for Hadoop HTTP web-consoles? running a cluster with older software cloudera-manager-server-5.15.0-1.cm5150.p0.62.el6.x86_64hadoop-httpfs-2.6.0+cdh5.15.0+2783-1.cdh5.15.0.p0.52.el6.x86_64cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el6.x86_64cloudera-manager-agent-5.15.0-1.cm5150.p0.62.el6.x86_64 

Hi, as we all know CDH 6.3.3 and subsequent versions are not available anymore under "Express licensing" model. Yet, I was in the process of setting up a 6.3.1 installation and apparently this is not possible, because a valid authentication (enterprise) is required. What am I doing wrong? Below, a screenshot from the Installation Guide:  But when I try to import the 6.3.1 Repository (WITHOUT Username and Password as stated on the documentation itself for version < 6.3.3 I get the following error:     # rpm --import https://archive.cloudera.com/p/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-clouderacurl: (22) The requested URL returned error: 401 Authentication requirederrore: https://archive.cloudera.com/p/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera: lettura importazione fallita(2).     What am I doing wrong? @Cloudera1 Is this a mistake I'm committing or is it a problem with Cloudera implementing their policies in a wro

I installed Cloudera Quickstart VM 5.13 on virtualbox and I'm trying to start hadoop server with the command sudo service hadoop-hdfs-namenode startBut i'm getting these two errors: log4j:ERROR Could not find value for key log4j.appender.RFAlog4j:ERROR Could not instantiate appender named "RFA" This is the full output of the command: [root@quickstart etc]# sudo service hadoop-hdfs-namenode startstarting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.outlog4j:ERROR Could not find value for key log4j.appender.RFAlog4j:ERROR Could not instantiate appender named "RFA".log4j:WARN No appenders could be found for logger.(org.apache.hadoop.hdfs.server.namenode.NameNode).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Failed to start Hadoop namenode. Return value: 1 [FAILED] Can anyone can help me with this issue?

Server returned HTTP response code: 401 for URL: https://archive.cloudera.com/p/cm6/6.3.1/allkeys.asc

Hi Cloudera Community, greetings from Florida I am very new to the new Cloudera Technologies. What I need is working with Impala and get used to that technology. Before, I used to download the Cloudera VM and that is it. All the technologies came installed already.  Would you please help me by saying step by step what I need in order to have a working environment in which I can use Impala?How that is work?Is there any other free IDE different than web based HUE? If not, that is OK. How much will it cost to have a small dataset and run some queries? But mainly I need to have it everything set up.  Thanks for your help and time VV

When running Hive queries from NiFi 1.10 that contain backticks, I get the following error: 2019-12-23 15:17:00,191 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=075b3a1e-7632-1647-68d9-338231b5921b] Failed to parse query: select 1 as `asdf` due to java.lang.NullPointerException: I thought Hive queries allowed backticks to escape column names, so I'm not sure why NiFi can't parse this.  The actual query runs fine on the Hive server, and I get a valid flow file with the query results, but it still raises a red NiFi bulletin (which we would prefer not to have if there is not a real problem). The Hive server running the query is the one that comes with HDP 2.5.

Hi Team,Please help me on the below issueI am ingesting the Data From Oracle to PostgresSql Using NiFi.We have huge data and continuously generating from sources, I want to update attributes and delete attributes using NiFi My Flow is: QueryDatabaseRecord-->UpdateAttribute-->PutDatabaseRecordwhen matched record found on the final table then define which action you need to take either Update (or) Deleteif the record not matched in the final dataset then insert the record.Appreciate your help.Thank you,