Member since
07-29-2019
640
Posts
111
Kudos Received
48
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
754 | 12-01-2022 05:40 PM | |
196 | 11-24-2022 08:44 AM | |
302 | 11-12-2022 12:38 PM | |
161 | 10-10-2022 06:58 AM | |
222 | 09-11-2022 05:43 PM |
12-01-2022
05:40 PM
1 Kudo
@Girish007 Sorry, no.
I can't suggest any other fix that you can implement in your production environment to mitigate the Log4shell vulnerability. And I would go further and say that I seriously doubt that anyone in a responsible position is going to tell you that you don't "really need to upgrade [y]our current nifi version 1.16.2 to a newer version" in spite of the fact that this specific situation is a great example of how a supposed "important" vulnerability brought to light by certain security scan applications really isn't actually relevant due to the specific way the library is used in a delivered system (which was the point I was trying to make earlier in this thread).
I have no doubt at all as to the accuracy of the assessment in the aforementioned Jira, and I also would strongly recommend that you plan to upgrade your current nifi version. My current understanding is that NiFi 1.19.0 included the library upgrade, and the forthcoming CFM 2.1.5.0 version will also include Apache Commons Text 1.10.0.
... View more
11-24-2022
08:44 AM
Hi @Arsh43 ,
You didn't include the Sqoop command you were running or which "form factor" of Cloudera Data Platform (CDP) you were using when you encountered this error message, but you most likely are seeing it because Sqoop needs a Java Development Kit (or JDK), and the OS you are running it from does not have one installed or it isn't accessible.
The Sqoop command you were running matters because Sqoop builds Java code for imports and exports. If you had a full JDK installed, at the command line the following commands should produce output
BBROOKS-SRVR14150:~ bbrooks$ type javac
javac is /usr/bin/javac
BBROOKS-SRVR14150:~ bbrooks$ javac -version
…that will indicate what version of the JDK's compiler is installed.
... View more
11-23-2022
08:20 AM
Hi @chintuk
While we welcome your question and a member of the Cloudera Community might well be able to answer it, I believe you would be much more likely to obtain a timely solution if you posted it to the Atlassian Community.
... View more
11-21-2022
02:22 PM
@dag
While you are waiting, can you describe in a bit more detail what you mean by "support NiFi in a commercial environment"?
As part of a CDP Subscription, Cloudera supports it's own products, which while they might be based on Apache's releases, are not freely interchangeable with them. One reason for that is there'a always certain differences between the versions released by Cloudera in products such as Cloudera DataFlow (CDF) or Cloudera DataFlow for the Public Cloud (CDF-PC) and the release of "upstream" component projects such as Apache NiFi. This is analogous to how there are differences between what mainline kernel is "current" in the open source Linux world and what RedHat, for example, releases as part of Red Hat Enterprise Linux.
In general, Cloudera doesn't provide support for the distributions of NiFi directly downloaded from Apache.
... View more
11-21-2022
10:39 AM
1 Kudo
@Girish007
The reason I was asking is because what kind of "security scan" you performed matters a great deal. Certain security scan applications are pretty basic and only look at a list of vulnerabilities and the libraries exposed to them and then compare that to a list of libraries using specific version numbers found on/retrieved from the deployed system in question. Many times, this will lead to "false positives" because the system is not actually vulnerable due to the specific way the library is used. Other times, the system vendor will have addressed the vulnerability by changing the code included in the library, without changing the name of major release number denoted in the filename.
In the specific case you're interested in, Apache Commons Text is a very popular library, so there are just going to be widespread references to vulnerable versions across anything that uses Apache code. For Apache NiFi, authoritative sources tell me that on initial investigation, NiFi had no direct uses of the vulnerable class, but had many transitive references to the library and that this was patched last month in the open source repository, so updates will be making their way to new versions.
You can view the upstream Jira issue here: NIFI-10648
... View more
11-16-2022
08:24 AM
@YogeshKumar
I'm curious as to exactly how you have determined that, because you have identified that there are previously identified vulnerabilities of critical, high and moderate severity in Log4j1 and Log4j2, that CDH 6.3.4 is exposed to those same vulnerabilities?
... View more
11-15-2022
07:31 AM
@hanumanth, Exactly how are you determining that the file path you've identified is exposed to the CVE-2022-42889 vulnerability?
... View more
11-14-2022
08:20 AM
@hanumanth I'm just curious as to exactly how you or @Girish007 have determined that a specific release of NiFi is vulnerable to the aforementioned CVE?
... View more
11-12-2022
12:38 PM
@yacine_, Authoritative sources tell me that the TEZ UI has not been a supported interface on/in CDP since the first release, so I don't think your efforts to use it will be fruitful. You can view a listing of unsupported interfaces for Hive on CDP Private Cloud Base release 7.1.7 here:
Hive unsupported interfaces and features
As of this writing, Data Analytics Studio is the only way to get the functionality you're looking for, as the status of TEZ UI hasn't changed for the 7.1.8 release.
... View more
11-07-2022
07:39 AM
@yacine_,
You didn't say what you are looking to do with the Tez UI, but there is documentation available on the equivalent functionality available in Data Analytics Studio (DAS), which is currently available on CDP, here: Difference between Tez UI and DAS
... View more
10-17-2022
11:58 PM
@yagoaparecidoti I think it's unlikely. By consulting this page:
Configuring and Using Zeppelin Interpreters
You can see a list of Zeppelin interpreters currently supported on Cloudera Data Platform. The python interpreter you are attempting to install is not one of them. This may be resolved in a subsequent release of CDP, but the version you are running now isn't able to fetch dependencies for that interpreter because they aren't available.
... View more
10-10-2022
06:58 AM
Hi @Mehdikhak ,
A very similar question to yours above was previously asked and answered in this thread:
Free Version of CDH → 6.3.2
... View more
10-04-2022
04:38 PM
Hi @Sunilkumarks
The single best way to get a timely answer to your questions would be to reach out to your Cloudera Account Team which is handling the license upgrade from HDP to CDP and ask them.
... View more
09-21-2022
09:35 AM
Hi @cxk
While the link in the document that @Chaos offered apparently does work, unless you are going to try to duplicate the issue they are experiencing in order to offer assistance, I'd like to discourage you from installing the Quickstart VM.
The last publicly available version of the Cloudera Quickstart VM — the same version referred to in that PDF document within the Google drive they linked to — was based on CDH 5.13, which was old, outdated and in fact that version of CDH went out of support in the Fall of 2020. The Cloudera Quickstart VM hasn't been readily available for download from Cloudera's website for at least a couple of years now just that reason.
Cloudera's current distribution, since the Fall of 2020, is Cloudera Data Platform (or CDP); a Trial Version of CDP Private Cloud Base Edition of Cloudera Data Platform can easily be downloaded from the "downloads" section of Cloudera's website.
... View more
09-11-2022
05:43 PM
1 Kudo
Hi @Poul
For gaining a better understanding of the issues surrounding this architectural question, I would begin by watching the recording of the "Demo Jam" from January before last during which Pierre Villard, Sr. Product Manager for NiFi at Cloudera, touched on Cloudera's recommendations in this area.
You can also read his much earlier blog post: Automate workflow deployment in Apache NiFi with the NiFi Registry …to gain a bit more background.
... View more
09-08-2022
12:43 PM
Hi @MaxineLLL ,
It would help members of the community in offering possible answers to your questions if you included a link here to the specific instructions you are following "to deploy HDP 2.6.5 via docker". Are you trying to install the HDP Sandbox or the full HDP 2.6.5 distribution?
... View more
09-05-2022
04:52 PM
1 Kudo
Hi @ravi_tadepally
Thanks for your interest in NiFi. Your question seems to indicate you misunderstand some of the finer details around how open source projects such as Apache NiFi operate, or the difference between Apache's software projects and the products and services that Cloudera offers, so I'm going to attempt to shed some light on that.
As a a 501(c)3 non-profit public charity organization incorporated in the USA, The Apache Software Foundation (ASF) exists to provide a vendor-neutral foundation for open, collaborative software development projects. Many companies that have employees who happen to contribute to Apache projects, such as NiFi, take the software developed from the project or projects and package them up to sell them as part of a larger software offering, provide support to enterprises, offer professional services or some combination of all three. Cloudera is one such company.
There are many professionals who are actively involved with the development and maintenance of Apache NiFi who also happen to work for Cloudera and develop software for it. There are also many Cloudera employees who provide professional services, including providing support and/or troubleshooting for new deployments of Cloudera's products in production to other companies. That does not mean, however, that Cloudera provides either professional services or support for any given version of NiFi that you can download from the Apache site. Instead, Apache NiFi underpins software products that you can purchase a subscription for from Cloudera.
Those products come in various "form factors" which vary in appropriateness depending on where you want to deploy them. For use with public cloud providers, NiFi is now available as part of Cloudera DataFlow for the Public Cloud (CDF-PC). For use in on premises clusters on top of the Private Cloud Base Edition of Cloudera Data Platform, NiFi powers Cloudera Flow Management (CFM).
As I alluded to above, there'a always some differences between the versions released by Cloudera in products such as CFM and the release of "upstream" component projects, such as NiFi, from Apache. This is analogous to how there are differences between what mainline kernel is "current" in the open source Linux world and what RedHat, for example, releases as part of Red Hat Enterprise Linux. For this reason, I think it's highly unlikely that you are going to be able to secure a subscription for a "vanilla" version of NiFi or obtain professional services around an Apache release, from Cloudera.
You can find out more about the current packages that Cloudera is offering and the pricing for them by contacting the Cloudera Sales Team to find out more about subscription options.
You can read about the professional services offerings from Cloudera by visiting the Cloudera Professional Services page.
You might get results closer to what you're after by trying to find support or professional services from a third party, such as one of Cloudera's partners.
... View more
09-05-2022
06:50 AM
Hi @ahlicoding
It would help members of the community in offering possible answers to your questions if you were a little more specific about what you or your client mean by "install with docker". That can mean more than one thing.
I'm doing a bit of guessing here, but one thing that could mean is deploying into a set of docker containers, where each container takes the place of what would ordinarily be a "bare metal" node in the target cluster. Or you could be referring to the common practice of collecting all the software dependencies necessary for completing the installation (e.g., a specific version of Python and version-appropriate libraries) into a Docker Container so that the process is faster and more convenient.
The redacted screen shot you've included above indicates that you're licensed for Ambari 2.7.4 and HDP 3.x (when most community members refer to Cloudera Enterprise, they usually mean the legacy releases of CDH and CDH is not interchangeable with HDP.).
I should mention that the current Enterprise Data Platform offered by Cloudera is Cloudera Data Platform (CDP), which in it's on-premises "form factor" is now called CDP Private Cloud. CDP superseded CDH as of 2021, which is important if your client is expecting support going forward.
You can read about how to access the Ambari 2.7.4.x binaries from Cloudera's private repositories as well as obtaining authentication credentials for customers and partners (implicitly verifying your license key) here: Accessing Ambari Repositories.
As far as your last question:
And which is the best for me, install on my local machine or install on Cloud VPS?
Assuming that you are talking about where to install Docker and not where to install Ambari, I think the answer depends on what OS your local machine is running and what the underlying hardware is vs what is available on your virtual private servers in the cloud.
One example of a situation where I would avoid installing Docker on my local machine and use the cloud instead is if I only had a Mac with an Apple silicon CPU. Docker has made great strides in supporting ARM architectures recently, but there's still a lot of hoops to jump through to use Docker on Apple silicon machines and some very common operations flat out still don't work. In the cloud you can spin up Linux-based instances running on commodity hardware and have the flexibility to use them as both a host for Docker and for the target nodes in the cluster you are building/installing on.
We welcome your questions, and this thread will remain visible here in the hope that some other member of the Cloudera Community will reply with more specific assistance.
... View more
08-02-2022
02:01 AM
HI @Christ
If you'd like more details about data model design and implementing them using Cloudera Data Warehouse over and above the documentation @shehbazk provided links to, I'd recommend you watch the recording of a webinar Cloudera held on the topic quite some time ago, Data Modeling for Hadoop and the subsequent blog post Common Questions on Data Modeling for Big Data.
... View more
07-29-2022
12:34 AM
Your best first step would be to contact the Cloudera Sales Team to find out more about subscription options.
... View more
07-23-2022
05:30 PM
1 Kudo
@Ploeplse For some reason that is a mystery to me, they were being miscategorized as spam. If I were to guess, it's the partial page shots, but again I don't know. I would try cutting and pasting the text of the error messages into your posts as opposed to the page shots as images.
One of the other moderators might weigh in here and tell me why I am wrong. In any case, I took action on it; your reply should now be visible in the thread.
... View more
07-22-2022
03:26 PM
Since you're using an environment only deployed for a 60-Day trial, I don't think it will hurt anything to try manually initiating the user sync as a first attempt at a resolution. You can review the procedure for doing that here:
Performing user sync
... View more
07-21-2022
10:42 AM
Hi @Ploeplse
Your assumption that the 60-day trial of Public Cloud is a great way to get familiar with CDP is correct. You should be able to open a case with via the Cloudera Support portal and request the required administrative privileges granted to you. If for some reason you don't have the ability to open a support case, you'll have to reach out to the appropriate contact person in your organization who can open a support case and have them do it for you.
... View more
07-18-2022
12:35 PM
1 Kudo
Hi @rafy
You didn't say where you looked, or if the NiFi 1.16.3 installation you are running is in the public cloud, on premises or private cloud. What @DigitalPlumber wrote earlier is correct; one of my colleagues provided the following page shot from Cloudera DataFlow (CDF) on CDP Public Cloud:
So the processors are clearly available. If for some reason you need to use Pulsar with some other distribution of NiFi, you can try checking the GiitHub repo here:
Awesome Apache NiFi + Apache Pulsar
... View more
07-16-2022
02:59 PM
Hi @Gobind
You didn't indicate where you downloaded that, or whose instructions you followed to "successfully [install] Cloudera in Oracle VM VirtualBox Manager". But I can tell you that the screen shot you've posted is not an error.
At this point in time or later, you (or anyone else reading this) should not be attempting to run the QuickStart VM on VirtualBox. The Cloudera Quickstart VM has been discontinued and has been obsolete for some time now, as it was based on CDH 5.13 which reached end of support in the Fall of 2020. Note that Cloudera never supported the QuickStart VM officially and again, it's been obsolete for years now.
Cloudera's current Enterprise Data Platform is Cloudera Data Platform (CDP), which in it's on-premises "form factor" was released back on November 22, 2019, over two and a half years ago. CDP supersedes CDH as Cloudera's data platform offering. The on-premises "form factor" of CDP is now called CDP Private Cloud. You can download a trial version of CDP Private Cloud to install on-premises here: CDP Private Cloud 60-day free trial.
... View more
07-11-2022
09:46 PM
Hi @Hanni
You didn't mention what distribution or version of Hadoop you're using, but in general you have a few widely available options. If you prefer using a low-to-no-code data ingestion engine with a rich graphical user interface and support for processors that provide connectivity, transformation, and content routing, including for JSON, you should investigate Apache NiFi, which powers Cloudera Flow Management (CFM). I would say that's your all-around best alternative, because NiFi has a very complete set of readers and writers for dealing with JSON.
If you have programming ability, another alternative would be to write a script using Spark SQL to ingest the data to HDFS. This will require the use of JDBC connectivity to the PostgreSQL DBMS just as Sqoop does. If you don't already know Spark, there's going to be a bit of a learning curve.
Lastly, if the source PostgreSQL table allows it, you could try importing the data using the hcatalog import function, which is an approach that was discussed here quite a while ago, using Oracle as the source DBMS in this thread: Sqoop to write JSON in HDFS . I am not aware of anyone getting that approach to actually work in practice using PostgreSQL.
... View more
06-21-2022
05:12 PM
p>Hi @FediMannoubi I want to add a bit here and add on to what @ckumar wrote earlier in the thread.
What tool did you capture the first screen shot from? Does that tool connect to the Postgresql database server via JDBC and using this Database Connection URL?
jdbc:postgresql://127.0.0.1:5432/ATL2
I ask because if you had, that would validate that the connection string you are using is indeed valid. I strongly recommend that you carefully read the documentation for Connecting to the Database, as it has detailed information on how a valid JDBC connection string should be formed. Comparing the available forms described on that page, and comparing the options to the string that you have captured in your page shot, the relevant option for the PostgreSQL JDBC string in this case appears to me at least to be the following form:
jdbc:postgresql://host:port/database
If the string you've already entered into the Database Connection URL property works, then you can proceed assuming that the final characters are the "database" and so in your case the database would be ATL2 The first screen shot appears to indicate that the database is named rei. I don't know what to make of that difference.
... View more
06-21-2022
10:55 AM
Hi @Techie123
Hopefully after reading the link that @SAMSAL shared earlier, you've reached the understanding that no, you don't have to install the database on the same server as the server that NiFi is installed on, but of course the server hosting the database must be accessible to the NiFi server via the JDBC protocol, regardless of where it is located. There are any number of things that could be causing the error message that you are getting.
Given what you have shared in this thread, I would recommend that you begin by talking to your DBA or perhaps the person that installed the database on the server. The string that you have entered on the properties tab, as shown in your page shot:
jdbc:oracle:thin@hostname:portnumber:sname
…needs to be replaced with the values appropriate for the server where your actual database is installed. To be a bit more specific, you have to get the actual literal values for hostname, portnumber and sname necessary to access your database and enter all of them correctly. In my personal experience, you'd also be much better off if you can verify that the database server is accessible via JDBC using some other tool so you have a way to validate the connection string before you get involved with NiFi-specific details.
... View more
06-17-2022
05:15 AM
@Uday_Singh2022 There won't be any documentation available from Cloudera regarding Flume because that lack of availability is part of what "not a supported component" means. If you must use Flume, then the documentation on the Apache site that @mszurap pointed you to above is your only option.
... View more
06-15-2022
01:31 AM
@Techie123
I strongly recommend that you read over this thread:
Problem login for first time in Nifi
…and then if you try all the troubleshooting steps described there, post again here, in this thread, and provide a link to the instructions you followed in order to perform your installation and for which version of NiFi.
... View more