Member since
06-30-2019
41
Posts
0
Kudos Received
0
Solutions
02-04-2021
11:54 PM
Hi @GangWar I did do policy "validate" and did not received any error. but replication still fails not sure where can I look for error.
... View more
02-04-2021
11:50 PM
Hi All, I'm trying to remove registered classic cluster from CDP console I do not see any option to do that, can someone please help ?
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
02-02-2021
07:43 PM
Hi Everyone,
I'm trying to replicate data from on-premise CDH 6.3.2 to CDP public cloud on Azure.
below are steps I followed.
1. Registered on premise CDH cluster onto CDP console under "Classic Clusters"
2. Added Azure credential in Replication Manager ( also created Azure datalake for storing replicated data )
3. Created replication policy.
Every time I try to run the policy it fails and I can not see any error log. Please help to advise.
attached are screenshots for error and settings done so far, not sure if I'm missing something here.
Any help would be appreciated.
... View more
Labels:
01-05-2021
11:50 PM
@GangWar thank you so much for your help I assigned myself as "Power User" and it worked like charm. However I'm bit surprised as my user is admin user still I had to assign a power user role.
... View more
01-04-2021
08:02 PM
@GangWar any suggestion how to go about resolving this issue ?
... View more
01-03-2021
06:16 PM
Hi @GangWar thank you for your reply. I tried Developer tool and I'm getting error as attached, could you suggest a possible fix ?
... View more
01-03-2021
06:08 PM
@Shelton thank you for your response I did go through prerequisite and support matrix, it looks ok. My on premise cluster is running on CDH&CM 6.3.0. Do let me know if you have more inputs
... View more
12-23-2020
01:35 AM
Hi Everyone. I already have a Data Lake and Data Hub running in my CDP environment but I’m having issues with registering the ‘Classic Clusters’. After clicking ‘Get Started’ the next screen is basically a blank page? Attached are screenshots for reference. Please help to provide your inputs. Appreciate your help
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
02-16-2020
11:29 PM
Hi @gzigldrum I indeed followed documentation and used below commands to grant required access; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman2019'; GRANT ALL ON rman.* TO 'rman'@'localhost' IDENTIFIED BY 'rman2019';
... View more
02-14-2020
02:33 AM
Hi All, I'm trying to install cloudera 6.2.0 , however am stuck with below error. Provided username, dbname and password correct, I tried same credentials to connect to DB from backend and it works. [root@tdicsadn01 cloudera-scm-server]# cat db.properties
# Auto-generated by scm_prepare_database.sh on Fri Feb 14 17:51:09 +08 2020
#
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."
#
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=scm2019
[root@tdicsadn01 cloudera-scm-server]# No firewall or Selinux [root@tdicsadn02 jdk1.8.0_181-cloudera]# getenforce
Disabled
[root@tdicsadn02 jdk1.8.0_181-cloudera]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@tdicsadn02 jdk1.8.0_181-cloudera]#
... View more
Labels:
- Labels:
-
Cloudera Manager
02-11-2020
07:53 PM
@MattWho I deleted local NIFI state directory and it worked like magic :), I finally have my first 7 node secured NIFI cluster accessible using client cert. Thanks a ton for your valuable inputs and detailed explanations without which I might have been still stuck, highly appreciated.
... View more
02-11-2020
06:34 PM
@MattWho from here does nifi picks up node Information for forming cluster ? Is this information stored in some directory or file?
... View more
02-11-2020
06:31 PM
@MattWho ill try to delete local state directory and restart nodes , is there anything else I should look into ?
... View more
02-10-2020
08:52 PM
@MattWho thanks for the inputs I was able to resolve user certificate issue, but my UI is stuck with below issue. I' m not sure why its still referring to http port. It's kind of weird , can you please advise ? java.net.ConnectException: Failed to connect to hostname/ip:8080
... View more
02-09-2020
07:24 PM
@MattWho I added client certificate in my browser . Below are steps i followed I setup initial admin indentity in nifi conf as "Admin" Got client cert for "Admin" from IT team added cert in my browser. Below is logline from my nifi-user.log. "2020-02-10 11:18:54,677 INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: Admin 2020-02-10 11:18:54,693 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Mon Feb 10 11:18:54 SGT 2020 2020-02-10 11:19:12,609 INFO [NiFi Web Server-48] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2020-02-10 11:19:12,641 INFO [NiFi Web Server-36] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response." 2020-02-10 11:19:12,666 INFO [NiFi Web Server-48] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=nifi-host1, OU=nifi, O=Liongate, L=nifi, ST=SG, C=SG) GET https://localhost:8443/nifi-api/flow/current-user (source ip: <ip>) 2020-02-10 11:19:12,673 INFO [NiFi Web Server-48] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG 2020-02-10 11:19:12,738 INFO [NiFi Web Server-48] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. 2020-02-10 11:22:11,245 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Mon Feb 10 11:22:11 SGT 2020
... View more
02-06-2020
08:30 PM
I'm stuck user certificate step if someone can help to understand on how to get these certs my nifi UI give "Insufficient Permissions"
... View more
02-06-2020
06:57 PM
Hi @justin_brock Believe you were able to fix certificate issue ? I'm have enables ssl for NIFI in CDF but facing "ERR_BAD_SSL_CLIENT_AUTH_CERT" here is link to my question of community https://community.cloudera.com/t5/Support-Questions/Unable-to-open-NIFI-web-UI-after-TLS/m-p/289190#M214098 could you please help me with steps you followed to resolve issue ?
... View more
02-06-2020
06:46 PM
Hi @MattWho I have checked configurations as mentioned by you but still I'm stuck at same issue, can you advise ?
... View more
02-05-2020
01:55 AM
Hi All,
I enabled TLS for NIFI web UI ( CDF ) , while services are running fine on cluster I'm unable to access NIFI web UI from my browser. Below are the steps I followed please suggest what might be causing issue ?
I repeated below steps for all the machines in my nifi cluster
1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem)
2. Copy the JDK cacerts file to jssecacerts as follows:
sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts
3. import rootca cert into JKS store
sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem
4. Created JKS and imported host certificate in keystore.
$JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048 -dname "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Singapore,ST=Singapore,C=Singapore" -ext san=dns:$(hostname -f) sudo $JAVA_HOME/bin/keytool -importcert -alias $(hostname -f) -file /opt/cloudera/security/pki/$(hostname -f).pem -keystore /opt/cloudera/security/pki/$(hostname -f).jks
5. creating symlinks
sudo ln -s /opt/cloudera/security/pki/$(hostname -f).pem /opt/cloudera/security/pki/agent.pem
sudo ln -s /opt/cloudera/security/pki/$(hostname -f).jks /opt/cloudera/security/pki/server.jks
6. Enabled TLS from Cloudera Manager for NIFI
7. Restarted services from Cloudera manager
8. Unable to access from Browser
... View more
Labels:
02-05-2020
01:44 AM
thanks @paras
... View more
01-30-2020
01:20 AM
Hi All,
I'm trying to enable TLS for Cloudera Manager admin console , below are the commands I followed.
1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem)
2. Copy the JDK cacerts file to jssecacerts as follows:
sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts
3. import rootca cert into JKS store
sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem
4. Created JKS and imported host certificate in keystore.
$JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048 -dname "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Singapore,ST=Singapore,C=Singapore" -ext san=dns:$(hostname -f) sudo $JAVA_HOME/bin/keytool -importcert -alias $(hostname -f) -file /opt/cloudera/security/pki/$(hostname -f).pem -keystore /opt/cloudera/security/pki/$(hostname -f).jks
5. creating symlinks
sudo ln -s /opt/cloudera/security/pki/$(hostname -f).pem /opt/cloudera/security/pki/agent.pem
sudo ln -s /opt/cloudera/security/pki/$(hostname -f).jks /opt/cloudera/security/pki/server.jks
6. Enabled TLS from Cloudera Manager admin console
Property Description
Cloudera Manager TLS/SSL Server JKS Keystore File Location
The complete path to the keystore file. For example:
/opt/cloudera/security/pki/server.jks
Cloudera Manager TLS/SSL Server JKS Keystore File Password
The password for the /opt/cloudera/security/jks/server.jks keystore.
Use TLS Encryption for Admin Console
Check this box to enable TLS encryption for Cloudera Manager.
7. Restart Cloudera Manager server
Cloudera manager starting successfully however I'm not able to open admin console from browser
sudo netstat -tulpn | grep 7183
tcp 0 0 0.0.0.0:7183 0.0.0.0:* LISTEN 4664/java
I'm getting below error on browser page
This site can’t provide a secure connection 1.1.1.1 uses an unsupported protocol.
ERR_SSL_VERSION_OR_CIPHER_MISMATCH
Unsupported protocol
The client and server don't support a common SSL protocol version or cipher suite.
Can someone please suggest what am I missing here ?
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Security
09-28-2019
07:33 AM
Hi I created the DB and recreated it later once error started coming, also this error goes away after a while but CM Server still not starting
... View more
09-27-2019
01:40 AM
Hi All, I'm trying to install Cloudera manager 6.3.0 on Centos 7.6. But when I start cloudera-scm-server its not starting, there is nothing much in cloudera-scm-server.log. Below is tail of log file. main:com.cloudera.cmon.components.CsdMetricSchemaGenerator: Loading monitoring definitions from /mdls/cdh5/oozie.mdl
2019-09-27 16:24:20,493 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 0 new metrics
2019-09-27 16:24:20,493 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 0 updated metrics
2019-09-27 16:24:20,497 INFO main:com.cloudera.cmon.components.MetricSchemaManager: CSD metrics processed.
2019-09-27 16:24:20,497 INFO main:com.cloudera.cmon.components.MetricSchemaManager: Registering cross entity aggregates...
2019-09-27 16:24:22,267 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 0 new metrics
2019-09-27 16:24:22,268 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 0 updated metrics
2019-09-27 16:24:22,269 INFO main:com.cloudera.cmon.components.MetricSchemaManager: Cross entity aggregates processed.
2019-09-27 16:24:22,990 INFO main:com.cloudera.cmon.MetricSchema: Setting current metrics schema to IN3dTiKfJ0y_QA4FqV_ZXPx7ZHg
2019-09-27 16:24:23,253 INFO main:com.cloudera.cmon.components.MetricSchemaManager: Metric schema initialized in PT5.046S.
2019-09-27 16:24:23,637 INFO main:com.cloudera.parcel.components.PeriodicParcelTasks: Set up periodic parcel tasks every 60 minutes.
2019-09-27 16:24:23,663 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files under /opt/cloudera/parcel-repo
2019-09-27 16:24:23,706 INFO main:com.cloudera.server.cmf.components.ScmActive: ScmActive started.
2019-09-27 16:24:23,728 INFO main:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: ClouderaManagerMetricsForwarder started.
2019-09-27 16:24:26,694 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 404 URI: <a href="https://archive.cloudera.com/sqoop-teradata-connector1/1.7.1c6/manifest.json" target="_blank">https://archive.cloudera.com/sqoop-teradata-connector1/1.7.1c6/manifest.json</a>
2019-09-27 16:24:38,720 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully.
Attached is screenshot can someone please advise what might be wrong ? At start i also get below error but it goes away after a while Java version - Oracle jdk1.8u181 mariadb - 5.5.64
... View more
Labels:
- Labels:
-
Cloudera Manager
09-18-2019
08:10 PM
How can the change the location of content_repository in CDF (Ni-Fi)?
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Cloudera DataFlow (CDF)
09-02-2019
07:07 PM
Hi All, I'm looking for a way to trigger Oozie from NIFI, however there is no much help available on community on this. I found details on InvokeHTTP processor from NIFI to call Oozie rest API but I do not have an idea to configure this NIFI processor Can someone help ?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Oozie
09-02-2019
06:50 PM
@adelgacem can you share sample for how you invoked Oozie using InvokHttp processor? I'm trying to run Oozie workflow from NIFI it will be a great help if you can share NIFI processor configuration details.
... View more
07-22-2019
08:10 PM
@Jay Kumar SenSharma can you please help on this ?
... View more
07-22-2019
08:10 PM
Hi All, I have designed a Nifi flow to push JSON events serialized in Avro format into Kafka topic, then I am trying to consume it in Spark Structured streaming. While Kafka part works fine, Spark Structured streaming is not able to read Avro events. It fails with below error. [Stage 0:> (0 + 1) / 1]2019-07-19 16:56:57 ERROR Utils:91 - Aborting task
org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -62
at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336)
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263)
at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414) Spark code import org.apache.spark.sql.types.{ StructField, StructType }
import org.apache.spark.sql.types.{ DecimalType, LongType, ByteType, StringType }
import org.apache.spark.sql.types.DataType._
import scala.collection.Seq
import org.apache.spark._
import spark.implicits._
import org.apache.spark.streaming._
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql._
import org.apache.spark.sql.avro._
import java.nio.file.{Files, Path, Paths}
val spark = SparkSession.builder.appName("Spark-Kafka-Integration").master("local").getOrCreate()
val jsonFormatSchema = new String(Files.readAllBytes(Paths.get("schema.avsc")))
val df = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "host:port").option("subscribe", "topic_name").load()
val df1 = df.select(from_avro(col("value"),jsonFormatSchema).as("data")).select("data.*")
df1.writeStream.format("console").option("truncate","false").start()
)) Schema used in Spark {
"type": "record",
"name": "kafka_demo_new",
"fields": [
{
"name": "host",
"type": "string"
},
{
"name": "event",
"type": "string"
},
{
"name": "connectiontype",
"type": "string"
},
{
"name": "user",
"type": "string"
},
{
"name": "eventtimestamp",
"type": "string"
}
]
} Sample topic data in Kafka {"host":"localhost","event":"Qradar_Demo","connectiontype":"tcp/ip","user":"user","eventtimestamp":"2018-05-24 23:15:07"} HDP - 3.1.0 Kafka - 2.0.0 Spark - 2.4.0 Any help is appreciated.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
-
Apache Spark
07-08-2019
06:46 AM
Can someone please advice on mentioned issue, I just can't see data in HDFS FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer; err=java.lang.NoSuchMethodException: org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
... View more
07-05-2019
06:37 PM
Hi All, I am using HDF 3.4.0, I have designed NIFI flow to read events using NETCAT and dump into HDFS . 1. I'm using FlumeSource and FlumeSink processors in NIFI. When I execute the flow, events are being read by source but the sink is not able to write event into HDFS. Below is error in nifi-app.log Here is source and sink configurations Source Sink 2. Also nifi-app.log file shows multiple start and stop for NETCAT source can someone suggest on cause ? Ideally it should be once. 2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source stopping
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source starting
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/host:44444]
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source stopping
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source starting
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/host:44444]
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source stopping
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source starting
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/host:44444]
... View more
Labels:
- Labels:
-
Apache Flume
-
Apache Hadoop
-
Apache NiFi