Re: scientific data in hadoop

Daming Xue · ‎02-09-2022

Hello If you check PI System documentation, they have a few options to extract data, like PI JDBC, Web Services etc The performance is usually limited by the PI system side, i.e. how fast can it provides the data export

Daming Xue · ‎02-08-2022

Hello Have you tried to export the metadata first? Once you export the existing metadata, you can have a working copy of the template for your version of Atlas, you can fit your bulk metadata into that and import back to Atlas

Daming Xue · ‎02-08-2022

Hello Firstly, identify the issues with the node, replace any faulty hardware if necessary You can follow the below post for the procedure https://community.cloudera.com/t5/Community-Articles/Adding-nodes-to-an-HDP-cluster/ta-p/245386

Daming Xue · ‎07-30-2021

Hello Perform the following in CM "Load Balancer Advanced Configuration Snippet (Safety Valve) for httpd.conf" Add following into above text box: ProxyTimeout 3600

Daming Xue · ‎07-29-2021

Hello This is not a NiFi problem, you may try to use some online tool to help https://jolt-demo.appspot.com/#inception

Daming Xue · ‎07-29-2021

Hello This may be related to the way you use Hue For example, if you uploading a large file using Import to table option, you may hit the proxy timeout issue, (by default, the timeout is 600 seconds), you can try to increase the proxy timeout option to a larger value, or you can optimise how you get large data files into the cluster by using other tools like NiFi etc

Daming Xue · ‎07-29-2021

Hello You can refer here: https://stackoverflow.com/questions/55070778/extract-only-the-columns-header-from-the-csv-file-using-nifi

Daming Xue · ‎07-11-2021

Hello You are welcomed to trial Cloudera CDP platform, you can find more details here: https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-trial-download-information.html

Daming Xue · ‎07-11-2021

Hello The max_row_size parameter is Impala specific parameter

Daming Xue · ‎06-04-2021

Hello The table column type must match the columnar file format of the Parquet file, else you will result in the error you got One way if you prefer to use string instead of double for that data attribute, you can recreate the parquet file with the column type of string

Daming Xue · ‎06-03-2021

Hello You can take a look of the Cloudera Manager Agent logs, and get the details about the issue Reference: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/managing-clusters/topics/cm-manage-agent-logs.html

Daming Xue · ‎06-03-2021

Hello It is common for customer to run preferred 3rd party component along side with the CDP cluster, and for your case, you can deploy Flume on the edge nodes of the cluster

Daming Xue · ‎05-16-2021

Hello Since it is related to Load Balancer, you can take a look of your configurations # cluster load balancing properties # nifi.cluster.load.balance.host=192.170.108.140 nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=50 nifi.cluster.load.balance.max.thread.count=600 nifi.cluster.load.balance.comms.timeout=45 sec For example nifi.cluster.load.balance.connections.per.node=50 The maximum number of connections to create between this node and each other node in the cluster. For example, if there are 5 nodes in the cluster and this value is set to 4, there will be up to 20 socket connections established for load-balancing purposes (5 x 4 = 20). The default value is 4 . Now you set it to 50, not sure how many nodes you got, you can do the math The rest of the configuration details you can refer here: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-system-properties/content/cluster_node_properties.html

Daming Xue · ‎05-13-2021

Hello Can you review this post and check the whether Hive namespace is created in ZK? https://community.cloudera.com/t5/Support-Questions/How-to-resolve-Unable-to-read-HiveServer2-configs-from/td-p/233701

Daming Xue · ‎05-13-2021

Hello Please check the below instructions (HDP platform) and make sure you are following the right steps https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/fault-tolerance/content/update_the_hive_metastore.html If you are using pure open source, you can seek helps from Hive community forum

Daming Xue · ‎05-13-2021

Hello Kafka is "pub and sub" If you plan you to use Spark, you can use Spark to subscribe to the topics and get the message Here is one example using Spark Streaming https://docs.cloudera.com/runtime/7.2.8/developing-spark-applications/topics/spark-streaming-example.html

Daming Xue · ‎05-02-2021

Hello It is self-explained in the documentation https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/security-ranger-authorization/topics/security-ranger-resource-policies-importing-exporting.html

Daming Xue · ‎05-02-2021

Hello Extracted from your log " Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity."

Daming Xue · ‎05-02-2021

Hello if the cluster is critical to your business, you should consider to get the subscription from Cloudera, and for your facing issues, Cloudera can create a patch by backporting the fix to an earlier CM version

Daming Xue · ‎05-02-2021

Hello Once example is to use Kafka Streams https://blog.cloudera.com/building-secure-and-governed-microservices-with-kafka-streams/

Daming Xue · ‎05-02-2021

Hello Make sure you configure in Windows hosts file for " sandbox-hdp.hortonworks.com" with a proper IP address

Daming Xue · ‎05-02-2021

Hello Please look out for Cloudera Manager 7.4.1 update, which comes with the fix to the issue you are facing The issue is because producer client ids that are not configured correctly and are generating huge metrics from anonymous producers The solution proposed in CM 7.4.1 is to make the Kafka producer metric whitelist configurable in the CM UI

Daming Xue · ‎05-02-2021

Hello As CCP is based on Metron, since Metron is retired, CCP will be retired too.

Daming Xue · ‎04-28-2021

Hello You need to have an active subscription with Cloudera, your username/password will be provided upon activation of your license subscription

Daming Xue · ‎04-26-2021

Hello This post may give you some ideas https://community.cloudera.com/t5/Support-Questions/How-to-hide-INFO-WARN-while-hive-execution/td-p/230775

Daming Xue · ‎04-26-2021

Hello Depends on the cluster size and monitored entities, certain resources are recommended Please refer to the below link and check whether your resource allocation is in-line with the recommendations https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-service-monitor-requirements.html

Daming Xue · ‎04-21-2021

Hello An example: https://stackoverflow.com/questions/38086684/hadoop-fs-du-h-sorting-by-size-for-m-g-t-p-e-z-y

Daming Xue · ‎04-21-2021

Hello Check your client library files are in the CLASSPATH Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.janusgraph.diskstorage.es.rest.RestElasticSearchClient

Daming Xue · ‎04-21-2021

Hello I am not sure the data in your Hive table got duplicates? Maybe try SELECT DISTINCT?

Daming Xue · ‎04-21-2021

Hello What's your CM version

Online	Offline
Last Visited	‎03-30-2022 01:55 AM

Date Registered	‎08-16-2015 06:33 AM
Last Visited	‎03-30-2022 01:55 AM
Total Messages Posted	96
Total Kudos Received	16

Re: scientific data in hadoop

Re: Increase max row size in HIVE

Re: Why do I have "incompatible Parquet schema" er...

Re: Flume re-engineering in CDP 7.x

Re: Service monitor keeps crashing

Re: Data ingestion from Osisoft PI System

Re: Apache Atlas Bulk import

Re: One Datanode has the status DEAD

Re: Hue proxy error

Re: How do I use Jolt to json array to nested json...

Re: Hue proxy error

Re: How to Extract only header of a csv file in ...

Re: scientific data in hadoop

Re: Increase max row size in HIVE

Re: Why do I have "incompatible Parquet schema" er...

Re: Command failed to run because service MapReduc...

Re: Flume re-engineering in CDP 7.x

Re: NIFI Cluster Node Restarts

Re: Cloudera Oozie workflow spark JDBC access to ...

Re: ambari hive metastore error

Re: input data into spark from kafka

Re: Ranger policy migration between two kerberised...

Re: Error fetching twiiter data while using flume

Re: Service monitor keeps crashing

Re: Microservices for cloudera hadoop

Re: SCP file transfer between Hortonworks Sandbox ...

Re: Service monitor keeps crashing

Re: Cloudera Cybersecurity Platform (CCP)

Re: Starting with the Ambari 2.7.5 release, access...

Re: Suppress / Hide all processing and Warning mes...

Re: Service monitor keeps crashing

Re: How to sort hdfs directories by size?

Re: How to connect ElasticSearch into Apache Atlas

Re: How to drop duplicates using nifi selectHIVEQL...

Re: Upgrade Cloudera express from 5.16 to 6.3.2