Member since
08-24-2017
32
Posts
7
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1300 | 06-11-2018 11:24 AM | |
883 | 03-23-2018 03:07 PM | |
668 | 03-07-2018 10:56 AM | |
720 | 03-07-2018 10:35 AM | |
635 | 03-06-2018 01:20 PM |
09-18-2018
08:00 AM
Also, in the log you have "Unable to locate KDC for realm XYZ.COM" make sure that REALM exists and if the mappings are correct inside the file.
... View more
09-18-2018
07:56 AM
Hi @Rohit Sharma, The "krb5.conf" file shouldn't be changed depending on the REALM, you can configure more than one REALM inside. https://web.mit.edu/kerberos/krb5-devel/doc/admin/realm_config.html Check if the keytab is pointing to the correct REALM. Hope it helps! Gonçalo
... View more
09-17-2018
07:56 AM
Hi @PJ, Atscale probably is using a JDBC connection. "it is connecting and disconnecting from hive metastore every few seconds.... is this normal?" Yes, can be normal as a JDBC connection is only established when a call is made to the target (Hive) to avoid narrowing network bandwidth for other calls, among other things to increase performance. Also, "Atscale" probably is doing some intermediate calls of some internal routines it may have. "when a user submits a query when is the connection made to hive metastore and when does it ends? " The connection should be established instantly but once again that depends of the caller app. The same for ending the connection, however for the closing of a connection the settings of the JDBC are also taken in account (Timeout property etc.). "and how often can this happen?" Depends of the number of calls the app does to Hive, Hive is just waiting to be called by third-party tools, in other words, Hive doesn't initiate connections to third-party tools. Hope it helps! Gonçalo
... View more
08-29-2018
08:24 AM
3 Kudos
Short Description:
Using JOLT transformation to modify a JSON object. Article:
In NIFI, JOLT Transformation can be used to help modifying a JSON object, the below code snippets are an example of how JOLT can be used to shift elements inside a JSON Object, in this case, the intention is to move and repeat "productionID" and "processID" inside of the array.
Original JSON Object:
{
"productionID": "BHS0928MDB8391",
"productionLine": [
{
"model": "Cooper",
"year": 2018,
"type": "Hatchback",
"motorization": "Electric",
"colour": "Midnight Black",
"stageID": "MGOP94810482042"
},
{
"model": "Cooper",
"year": 2018,
"type": "Hatchback",
"motorization": "Diesel",
"colour": "Silver",
"stageID": "MGOP9183740194"
}
],
"processID": "884910421743POR4183UK"
}
Target JSON Object:
{
"productionLine" : [ {
"model" : "Cooper",
"year" : 2018,
"type" : "Hatchback",
"motorization" : "Electric",
"colour" : "Midnight Black",
"stageID" : "MGOP94810482042",
"productionID" : "BHS0928MDB8391",
"processID" : "884910421743POR4183UK"
}, {
"model" : "Cooper",
"year" : 2018,
"type" : "Hatchback",
"motorization" : "Diesel",
"colour" : "Silver",
"stageID" : "MGOP9183740194",
"productionID" : "BHS0928MDB8391",
"processID" : "884910421743POR4183UK"
} ]
}
JOLT transformation code used: [
{
"operation": "shift",
"spec": {
"productionLine": {
"*": {
"@": "[&1]",
"@(2,productionID)": "[&1].productionID",
"@(2,processID)": "[&1].processID"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"@": "productionLine"
}
}
}
]
... View more
- Find more articles tagged with:
- Data Ingestion & Streaming
- How-ToTutorial
- jolt
- joltransformjson
- jolttransformjson
- NiFi
- nifi-processor
Labels:
07-06-2018
08:03 AM
Hi @Misbah Rehman, You should follow this doc Installing HDF service on a existing HDP cluster: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_installing-hdf-on-hdp/content/ch_install-mpack.html Gonçalo
... View more
07-04-2018
08:21 AM
Hi @Anjali Shevadkar, Check Ranger Audits, probably you don't have permissions. Gonçalo
... View more
06-11-2018
11:24 AM
1 Kudo
Hi @Bhagabati Das, RHEL 7.5 is not supported for HDP 2.6.4: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_support-matrices/content/ch01.html However, your error seems to be related to this: https://access.redhat.com/articles/2039753 Gonçalo
... View more
06-07-2018
01:14 PM
Hi @Alan B, You have examples in GitHub: https://github.com/apache/metron/tree/master/use-cases Also, you can download HDP sandbox and install HCP: https://hortonworks.com/downloads/ https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.5.0/bk_installation/content/install_using_ambari.html Gonçalo
... View more
06-07-2018
07:50 AM
Hi @Roberto Sancho, By adding HDF Mpack to a HDP cluster only enables you to add some HDF services to your existing HDP cluster. HDF will not replace your current HDP installation. Check the following documentation: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_installing-hdf-on-hdp/content/ch_add-hdf-to-hdp.html Hope it helps! Gonçalo
... View more
06-06-2018
03:07 PM
Metron has several stages where the event is processed, for example, after you parsed your event you can enrich it, that can be done with Stellar using "GEO_GET" function you can get, based on a GEO list, the origin of an IP. You can also add your own custom functions or you can do logic, arithmetic operations, etc. Using Metron UI you can see your alerts in real-time, check the page below for more detail about the UI: https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.5.0/bk_user-guide/content/using_alerts_table.html I would recommend you to install Metron and try it. Gonçalo
... View more
06-06-2018
02:38 PM
Hi @dedwards, Were you able to connect to Hive via SBX CLI? Because if you are able to connect with Kerberos to Hive via CLI then Kerberos is potentially set-up correctly in SBX and the problem is on the Windows machine connection to the SBX. Also, are you not missing the ZK information in the JDBC connection: Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/foodmart;principal=hive/sandbox-hdp.hortonworks.com@HWX.COM"); Check this Article: https://community.hortonworks.com/articles/156873/hiveserver2-kerberos-authentication-jdbc-connectio.html Hope it helps! Gonçalo
... View more
06-06-2018
02:28 PM
Hi @Anpan K, SNMP is one of the methods available. In your case you can simply choose "SMTP". Check the following link which have a step-by-step of how to enable alerts using email: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.0/bk_ambari-operations/content/creating_or_editing_notifications.html You will need to set-up a SMTP server so that Ambari is aware to where it will send the notification to. Hope it helps! Gonçalo
... View more
06-06-2018
02:23 PM
1 Kudo
Hi @Alan B,
You can perform logic operations, profiling and create scores and alerts based on that results. You can also use MaaS (Model-as-a-Service) to deploy models and create alerts based on results, etc.
Basically Metron has available a wide number of options to perform a variety of cases, in that way, is quite open.
Visit the following documentation for more information:
https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.5.0/bk_analytics/content/overview.html
Visit also the Stellar Language doc:
https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.5.0/bk_stellar-quick-ref/content/intro.html
Metron lives atop of Hadoop Stack, in that way, you will have the following requirements:
Apache Hadoop;
Apache Storm;
Apache Kafka;
Apache HBase;
Apache ZooKeeper.
Visit the following documentation for more information about requirements of HCP/Metron:
https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.5.0/bk_installation/content/software_req.html
Hope it helps! Gonçalo
... View more
04-03-2018
07:12 AM
Hi @Josh Nicholson, You can use tcpdump to see the comms between. Gonçalo
... View more
04-03-2018
07:07 AM
It seems that you are having an error accessing the repos. Do you have internet on that machine? If not, then you need to download all the dependencies of the package so that Maven is able to build the package.
... View more
04-03-2018
06:54 AM
Hi @Sankaru Thumuluru, It seems that Ambari doesn't have the AD SSL's in the Trust Store. Did you configured the Trust Store for Ambari with the AD SSL's? https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/set_up_truststore_for_ambari_server.html Gonçalo
... View more
04-03-2018
06:44 AM
Hi @Anurag Mishra, Knox doesn't sync users or groups, it queries the LDAP every time a user logs in to Knox. If I understood correctly your question, you can achieve user access control via Ranger to the different services that Knox supports. Nevertheless, you can also limit the LDAP query that Knox does via OU, DC, etc. check the example below: https://knox.apache.org/books/knox-0-12-0/user-guide.html#Advanced+LDAP+Authentication Also, you can limit the access to certain services via "proxyuser". Gonçalo
... View more
04-03-2018
06:26 AM
Hi @Gaurav Bapat, Did you followed the instructions below "Building Metron" chapter: https://github.com/apache/metron Current release: http://metron.apache.org/documentation/#releases You can also use HCP following the instructions below: https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.4.1/bk_installation/content/installing_ambari_management_pack.html Gonçalo
... View more
03-23-2018
03:26 PM
Hi @Hammad Ali, That error seems to be because you are not using the correct SSL certificates for your AD/LDAP. Assuming that you have a separate AD/LDAP instance, that AD/LDAP has is own SSL certificates and you should be using that specific SSL certificates and not the SSL certificates that you used to enable HTTPS. Create a new key store and import all the public key certificates of the AD/LDAP including the CA and Intermediate of that SSL's. Then update the following properties in "Advanced ranger-ugsync-site" in Ranger service: ranger.usersync.truststore.file=/etc/security/clientKeys/all.jks
ranger.usersync.truststore.password=<PASSWORD> Restart Ranger service. Gonçalo
... View more
03-23-2018
03:07 PM
Hi @Bramantya Anggriawan, Check if in Ambari UI the Metron repo URL is pointing to the correct version, go to "Admin -> Stack and Versions -> Versions tab -> Manage Versions -> Click your current HDP version". If not, uninstall the mpack and then go to Ambari database and remove the HCP references in the XML tag in the "repo_version" table. Before doing the change perform a backup of the Ambari database. Then install again the HCP mpack. Gonçalo
... View more
03-13-2018
09:29 AM
1 Kudo
Hi @GN_Exp, If you are using Oracle JDK, check if you have JCE installed for your Java version. You can follow this Support KB regarding on how to check if JCE is installed and how to install: https://community.hortonworks.com/content/supportkb/48974/how-to-check-if-jce-is-unlimited.html Hope it helps. Gonçalo
... View more
03-13-2018
09:16 AM
Hi @Deepu Nagesh, Test if your details are correct via ldapsearch, for example: ldapsearch -h <YOUR AD URL WITHOUT PROTOCOL> -p 389 -x -D 'CN=hdpsrv,OU=service_accounts,DC=mylab,DC=com' -W -b 'OU=ambari_roles,DC=mylab,DC=com' If you connect successfully then keep testing the other parameters that you have set-up in the Ambari AD configuration, to confirm that the Ambari AD configuration was set-up properly accordingly with your AD settings. As the error that the AD is throwing is "52e" meaning that is Invalid AD Credentials. Hope it helps. Gonçalo
... View more
03-13-2018
09:04 AM
Hi @Hassan Mahmood, Try to access via the FQDN or the IP that you have set in the hosts file. Also, check if the IP range that you have set in Virtual Box is not already being used by another device in your network. Check this tutorial to see if you missed any set-up in your Virtual Box network adapters: https://www.thomas-krenn.com/en/wiki/Network_Configuration_in_VirtualBox Hope it helps. Gonçalo
... View more
03-07-2018
11:27 AM
@abu ameen I think there's the problem. Zookeeper doesn't have any paths for Metron like -> /metron/topology/global. You can create this specific path but you can face further issues such as ACLs, etc. or you can reinstall Metron. Gonçalo
... View more
03-07-2018
10:56 AM
Hi @Nicolas Velasquez, Yes, there is. You can access Ambari UI and go to "Admin -> Stack and Versions -> Versions tab -> Manage Versions -> Click your current HDP version", in that page you will be able to update all the repo URL's. The second option would be to uninstall the mpack and then go to Ambari database and remove the HCP references in the XML tag in the "repo_version" table. Before doing the change backup the Ambari database. Then install again the HCP mpack. Gonçalo
... View more
03-07-2018
10:48 AM
Hi @Vinoth Krish, To deploy your model to Metron you should use the following: /<METRON HOME>/bin/maas_deploy.sh -zq <ZOOKEEPER>:2181-lmp <LOCAL FS MODEL PATH>-hmp <HDFS MODEL PATH>-mo ADD -m <MEMORY FOR THE CONTAINER>-n <MODEL NAME>-v <MODEL VERSION>-ni <NUMBER OF MODEL INSTANCES> If you are trying first to start MaaS Service you should use: /<METRON HOME>/bin/maas_service.sh -zq <ZOOKEEPER>:2181 If you don't have at all MaaS components installed try this (Change accordingly with your OS): yum list metron* Find the Metron MaaS package and install (Change accordingly with your OS): yum install <METRON MAAS PACKAGE NAME> The same as replied here -> https://community.hortonworks.com/questions/177048/how-to-deploy-my-own-model-in-to-metron.html For more information, you can also check the following documentation: https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.4.1/bk_analytics/content/models.html Gonçalo
... View more
03-07-2018
10:46 AM
Hi @Vinoth Krish, If you don't have at all MaaS components installed try this (Change accordingly with your OS): yum list metron* Find the Metron MaaS package and install (Change accordingly with your OS): yum install <METRON MAAS PACKAGE NAME> The same as replied here -> https://community.hortonworks.com/questions/177048/how-to-deploy-my-own-model-in-to-metron.html Gonçalo
... View more
03-07-2018
10:43 AM
Hi @abu ameen, Are you able to access to Zookeeper CLI and run a ls? What do you mean about "reseting Ambari Server"? Gonçalo
... View more
03-07-2018
10:35 AM
Hi
@Vinoth Krish,
To deploy your model to Metron you should use the following:
/<METRON HOME>/bin/maas_deploy.sh -zq <ZOOKEEPER>:2181 -lmp <LOCAL FS MODEL PATH> -hmp <HDFS MODEL PATH> -mo ADD -m <MEMORY FOR THE CONTAINER> -n <MODEL NAME> -v <MODEL VERSION> -ni <NUMBER OF MODEL INSTANCES>
If you are trying first to start MaaS Service you should use:
/<METRON HOME>/bin/maas_service.sh -zq <ZOOKEEPER>:2181
If you don't have at all MaaS components installed try this (Change accordingly with your OS):
yum list metron*
Find the Metron MaaS package and install (Change accordingly with your OS):
yum install <METRON MAAS PACKAGE NAME>
Hope it helps! Gonçalo
... View more