Q&A

I am trying to stop the VM VirtualBox to be able to change the base memory but it is still greyed out. Can i enable the change please? Thank you

Hello I trying to put some data from my CSV to my database but i got some error.  Settings UpdateRecord Settings about CSVRecordSetWriter My csv file:RegionCountryItemTypeSalesChannelOrderPriorityOrderDateOrderIDShipDateUnitsSoldUnitPriceUnitCostTotalRevenueTotalCostTotalProfitSub-Saharan AfricaChadOffice SuppliesOnlineL1/27/201129249452302/12/20114484651.21524.962920025.642353920.64566105.00EuropeLatviaBeveragesOnlineC12/28/20153618255491/23/2016107547.4531.7951008.7534174.2516834.50Middle East and North AfricaPakistanVegetablesOfflineC1/13/201114151576702/01/20116515154.0690.931003700.90592408.95411291.95Sub-Saharan AfricaDemocratic Republic of the CongoHouseholdOnlineC09/11/201250036400510/06/20127683668.27502.545134318.413861014.821273303.59EuropeCzech RepublicBeveragesOnlineC10/27/201512748159112/05/2015349147.4531.79165647.95110978.8954669.06

Hi,I have SiteToSiteBulletinReportingTask setup to capture bulletin information for a standalone secure nifi instance. the reporting task is pointing to an input port as expected but when I start the task it doesnt work and it keeps giving this warning:SiteToSiteBulletinReportingTask[id=e99b03f1-0173-1000-c7ef-4e475ed2416d] org.apache.nifi.remote.client.PeerSelector@24677b11 Unable to refresh Remote Group's peers due to response code 403:Forbidden with explanation: null"The reporting task is setup as follows:         The SSL context service is setup with the same values for the keystore and truststore as in the nifi.properties file which is working to secure my instance. I have added the policy "receive data via site-to-site" to the input port "S2S_Bulletins_rec" . The user created by securing the instance has the policy "retrieve site-to-site details". The error doesn't give much details. Can you please help? Thank you. 

Good morning everybody, I have the following scenario of returning an API through two streams =Flow 1 ({"cod": 1}{"cod": 2}{"cod": 3}) Flow 2 ({"error": "error 1"}{"error": "error 2"}{"error": "error 3"}) Each error has its cod and I need to send it to my database in this way through a procedure through "ExecuteSQL" =procedure (cod => '$ {cod}',                  error => '$ {error}'); I am trying to do this through "MergeContent" but it is not working. What Nifi is doing is as follows ={"cod": 1} {"cod": 2} {"cod": 3} {"error": "error 1"} {"error": "error 2"} {"error": "error 3" } Not dividing the shipments by code, let's say and passing to "ExecuteSQL" all at once, when Cod + Error would have to be sent one at a time. Follow all my flow = This is my "MergeContent" =  This is my "ExecuteSQL" = Can someone help me?

I am building a log analysis planform to monitor spark jobs on a yarn cluster and I want to get a clear idea about spark/yarn logging. I have searched a lot about this and these are the confusions I have.The directory specified in spark.eventLog.dir or spark.history.fs.logDirectory get stored all the application master logs and through log4j.properties in spark conf we can customize those logs ?In default all data nodes output their executor logs to a folder in /var/log/. with log-aggregation enabled you can get those executer logs to the spark.eventLog.dir location as well?I've managed to set up a 3 node virtual hadoop yarn cluster, spark installed in the master node. When I'm running spark in client mode I'm thinking this node becomes the application master node. I'm a beginner to Big data and appreciate any effort to help me out with these confusions.

Good afternoon, I am using the "RouteOnAttribute" processor to do a loop validating a condition that is as follows = ${cont:lt(${CONTADOR})} But I am not able to make the processor understand the property "CONTADOR" Could you help me please?      

NIFI 1.13.2 has nodes randomly restarting with connections refused.cluster information12 nodes 8 cores 32 gb rambootstrap.confjava=javarun.as=lib.dir=./libconf.dir=./confgraceful.shutdown.seconds=20java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=truejava.arg.2=-Xms28gjava.arg.3=-Xmx28gjava.arg.4=-Djava.net.preferIPv4Stack=truejava.arg.5=-Dsun.net.http.allowRestrictedHeaders=truejava.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocoljava.arg.7=-XX:ReservedCodeCacheSize=256mjava.arg.8=-XX:CodeCacheMinimumFreeSpace=10mjava.arg.9=-XX:+UseCodeCacheFlushingjava.arg.14=-Djava.awt.headless=truenifi.bootstrap.sensitive.key=java.arg.15=-Djava.security.egd=file:/dev/urandomjava.arg.16=-Djavax.security.auth.useSubjectCredsOnly=truejava.arg.17=-Dzookeeper.admin.enableServer=falsejava.arg.snappy=-Dorg.xerial.snappy.tempdir=/opt/nifi/tmpnotification.services.file=./conf/bootstrap-notification-services.xmlnotification.max.attempts=5# cluster node properties (only configure for cluster nodes

Is there any way to integrate ambari Grafana with ldap or Integrate Garafana with ranger for access control?Ambari version -2.75HDF -3.5

Hey I need to find the yarn logs on the local disk. Where I can find the logs ? Also can I see the log id manually in log files?

Good afternoon everyone, I have a problem with my flow in the processor "JoltTransformJSON" when I run the processor "ExecuteSQL" and no Oracle record returns to me.   I would like your help to know what to do in this case. Thank you so much.

Hi,  We have scenario to read file feed containing different data schema for lines and then write to multiple tables. We are trying to setup error handling and consolidated all success and failure count for different types of data. our data flow is almost as follows,getFile -> splitText -> RouteText -> ConvertToJson -> JsonToSql -> PutSql Can you please help with some suggestion. Thanks in advance. Regards,Rahul

Current setup, there are 2 computer machine1) First machine (IP of 192.168.4.1)It contain the NiFi setup as show in the image below:Inside the NiFi setup, I get a file from a local directory and I invoke a HTTP POST to send over to the second machine which hit the Wildfly Restful Webservice endpoint. Below image show the configuration I have done inside the Invoke HTTP: (Note: the Remote URL: http://192.168.3.1:8081/JaxRS-Nifi/restapi/multipart/upload ) 2) Second machine (IP of 192.168.3.1) It contain a Wildfly Restful Webservice which allow my first machine to hit the webservice endpoint.Error that I am facing:  I have a problem which the first machines doesn't send a HTTP POST over to the Second machines. I am currently looking for solutions on what are the different configurations should I do to make a NiFi HTTP over to hit the Restful Webservice endpoint? Thanks. Appreciate if there is anyone who can help me.   

Since yesterday afternoon, i am not able to access internet in cloudera (Oracle VM ). I keep getting errorsCloudera: Firefox can't establish a connection to the server at localhost:8889Two days ago things were working fine. and I don't think it is an internet problem on my end since i have been using it all day. Any idea? Thanks

Hello, We have a strange issue here. We are on CDH 6.3.0 and have sentry in place for authorizations. A user is trying to execute queries from Hue on Impala Editor and fails with Authorization errors. However, same query works fine via Hive and also via Impala shell. It fails only from Hue editor. We have:1. Refreshed metadata2. Invalidated metadata3. Verified permissions are in place in Sentry4. There are no logs generated (I checked in Hue, Sentry Server and Catalog server). Please suggest if any other place should also be checked. Example query:select * from DB.table limit 5; Kindly help to diagnose and resolve this issue. ThanksSnm1523

I am trying to connect Hive from spark with JDBC connection. While I use spark-submit option from terminal, it's completed successfully. But from Oozie workflow it's showing ERROR  Unable to read HiveServer2 configs from ZooKeeper Here is my code snippet  val sparkSession: SparkSession = SparkSession .builder().appName("Spark hive" ) .getOrCreate() val jdbcUrl = "jdbc:hive2://master1.cloudera.com:2181," + "master2.cloudera.com:2181,master3.cloudera.com:2181/default;" + "serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" val driverClass = "org.apache.hive.jdbc.HiveDriver" try { val cls = Class.forName(driverClass) println(cls.getName) } catch { case e: Throwable => e.printStackTrace() } val connection = DriverManager.getConnection(jdbcUrl, null,null) println(connection.isClosed) val statement: Statement = connection.createStatement() val table = "default.external_tab" val sql = "SELECT * FROM " + table

Hello,Can any one please help me understand that is CDH still is an open source Hadoop distributed platform offered by Cloudera as of 12th May 2021.Challenges faced : I am legacy customer of CDH 6.0.0 open source for about an year. If CDH is going away from open source to subscriptions, what does that mean for legacy customers [Patch fix, bugs, small upgrades] ?Note, The Official home page of CDH still calls CDH platform to be 100% open source but when we go to Downloads for upgraded versions, it's asking for subscription. You guys really need to update the documentations for customers.https://www.cloudera.com/products/open-source/apache-hadoop/key-cdh-components.html

Hi experts,I was wondering if there is a place where it shows the enhancements between each hadoop component such as hadoop, yarn, hive, hbase, etc. So I am trying to find a reason to upgrade our hadoop clusters or wait for a newer version to come out. Any help is much appreciated. Thanks,

I am using Nifi 1.12.1 version and configuring Prometheus for monitoring. I have multiple nifi instances running on the same machine. But I am not able to get metrics for all on the same port 9092 and getting below error in nifi log. From 3 nifi instances, only one nifi is able to start Prometheus reporting task properly others are saying Address already in use as displayed in error below. Can't we change the Prometheus metrics endpoint from nifi? For example for each nifi instance, we should be able to use a different metric endpoint in below way. nifi-1 - 9092/metrics (default metric end point)nifi-2 >>> 9092/nifi_2_metricsnifi-3 >>> 9092/nifi_3_metrics 2021-05-10 09:55:39,957 ERROR [Timer-Driven Process Thread-4] o.a.n.c.s.StandardProcessScheduler Failed to invoke the On-Scheduled Lifecycle methods of [PrometheusReportingTask[id=47b5aa61-0179-1000-ce9d-d59160596cdb], java.lang.reflect.InvocationTargetException, 30 sec] due to {}; administra

Hi All,I am doing this course on Cloudera which requires installation of anaconda, spark. After many attempts, i have successfully installed all packages but i still get this error in my notebook. import pyspark import pyspark---------------------------------------------------------------------------ImportError Traceback (most recent call last)<ipython-input-7-c15ae3402d12> in <module>()----> 1 import pyspark/usr/lib/spark/python/pyspark/__init__.py in <module>()3940 from pyspark.conf import SparkConf---> 41 from pyspark.context import SparkContext42 from pyspark.rdd import RDD43 from pyspark.files import SparkFiles/usr/lib/spark/python/pyspark/context.py in <module>()22 from tempfile import NamedTemporaryFile23---> 24 from pyspark import accumulators25 from pyspark.accumulators import Accumulator26 from pyspark.broadcast import Broadcast/usr/lib/spark/python/pyspark/accumulators.py in <module>()89 import select90 import struct---> 91

Hi,when I try to upgrade from HDP 3.0.1.0 to HDP 3.1.0 I get the following error when using the Version Definition File http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/HDP-3.1.0.0-78.xml. "Version Definition read errorAn internal system exception occurred: Stack data, Stack HDP 3.1 is not found in Ambari metainfo"What am I doing wrong ? What is the meaning behindern this internal system error ?Br,Dennis

Hi, I am using apache nifi's consumeKafka processor to consume IOT data.It is working fine when explicitly specifying the topics. However, I am looking for something more dynamic so I don't have to add topics manually in nifi each time we connect a new device.Is it possible to have wildcards in the kafka topic name field?My topics name look like domain.iot_device_name so I need something like domain.*  

I've been looking for information for a long time and I can't get it. I'm starting to think that it can't be done if the .parquet is in Azure DataLake Storage. I have a folder with subfolders in Azure DataLake Storage. In these subfolders there are many .parquet. I want to dump the data of these files in a table of a DB. If I am not mistaken when interpreting the code that generates them, the .parquet have several columns:Some whose names correspond exactly to the names of the columns in the table.Others whose names in the .parquet are in lowercase while in the table they are in uppercase.Others whose names are not in the table. I do not need these, I do not want to dump their data in the table. I manage to get the .parquet out of Azure DataLake Storage using ListAzureDataLakeStorage + FetchAzureDataLakeStorage combination. Then I try to pass them directly through a PutDatabaseRecord to dump them in the DB table. As the "Record Reader" property of the PutDatabaseRec

our files:path/sas/a.textpath/sas/b.textpath/sas//bas/a.textpath/sas/bas/b.text how use File Filter Regex and Path Filter Regex to find path/sas/a.text and path/sas/bas/b.text ?there are my filter, I always got four files.File Filter Regex:  a.text|b.textPath Filter Regex: path/sas|path/sas/bas

NiFi 1.13.2OpenJDK 1.8.0_292 (64-bit)  <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> <totalSizeCap>50GB</totalSizeCap> <!-- archive r