Member since
08-12-2014
34
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1213 | 09-22-2014 03:47 PM | |
9861 | 09-15-2014 04:04 PM |
10-01-2018
08:35 AM
I found what's going on. The cluster for which I could not access the API is using V17, while the others are V19, even though all CDH versions are all 5.12.x. After changing V19 to V17 in the URL, I can now access the API. I wonder why the version is different. Michael Reynolds
... View more
10-01-2018
07:17 AM
Greetings, We have several clusters, and there is one where I cannot access the API. For example, if I go to: https://<hostname>:7183/api/v19/hosts I get this: No webpage was found for the web address: https://<hostname>:7183/api/v19/hosts HTTP ERROR 404 Is there an option to enable the API? I have not found documentation about it. Michael Reynolds
... View more
Labels:
08-30-2018
01:31 PM
@bgooley, I found what appears to be the reason why the INVALIDATE METADATA command is not working. ERROR: AuthorizationException: User 'impala/<server>@<domain>' does not have privileges to access: <DB>.<TABLE>
Could not execute command: invalidate metadata <DB>.<TABLE> It is failing to run the command. However, after manually issuing the INVALIDATE METADATA command, Impala has no trouble accessing the table. Michael Reynolds
... View more
08-30-2018
08:10 AM
@bgooley, Thank you for your reply! If I run "invalidate metadata" on the destination, the databases appear after doing a refresh. I tested a new replication, ensuring that " Invalidate Impala Metadata on Destination" was selected, and the data does not appear on the destination until I manually run "invalidate metadata". Is there a way to ensure the invalidate command will work automatically via the replication job? Michael Reynolds
... View more
08-28-2018
09:31 AM
As a follow-up, we have CDH 5.12.2 on both clusters. Under the replication "Advanced" tab, instead of seeing "Replicate Impala Metadata" as an option, I have " Invalidate Impala Metadata on Destination". Checking this option appears to have no effect in making the data visible to Impala on the destination cluster. Michael Reynolds
... View more
08-28-2018
09:10 AM
I am testing Hive replications to a new DR cluster, and the databases are not visible to Impala. What must be done to make the data accessible via Impala?
Thank you,
Michael Reynolds
... View more
Labels:
07-13-2018
12:21 PM
Great! Thank you very much for the detailed information. Michael Reynolds
... View more
07-13-2018
11:37 AM
I need to develop some metrics and KPIs, from multiple clusters, similar to statistics shown within Cloudera Manager, but not as extensively.. Is there documentation available for the backend database structure? We are uisng Oracle and CDH 5.12. Michael Reynolds
... View more
Labels:
11-21-2016
06:13 PM
Thank you for your reply! Perhaps I am mistaken, but the informaiton in your link appears to be regarding how inactive sessions can be timed out, such as with the idle_session_timeout property. I am seeking a method to disable a user account which has not been used for a period of time, such as 90 days. Michael
... View more
11-16-2016
12:01 PM
Can user access be revoked if there has been no login for a set period of time? For example, if a user has not accessed Hue or other applications after 90 days, can their access be disabled? Michael Reynolds
... View more
Labels:
01-10-2016
08:49 AM
Hi Harsh, Thank you very much for your assistance. We are using CM 5.3.3, and plan to upgrade later. I will explore the API docs, about using the CLI. Michael
... View more
01-08-2016
03:02 PM
We have numerous backup replications defined and running, causing access via a web browser to be very slow. Is there a way to access, create, and modify schedules via the Linux shell? Michael Reynolds
... View more
- Tags:
- Backup
- replication
Labels:
12-21-2015
12:35 PM
Thank you very much for your reply. According to the agent log, the service account responsible for running the Impala service does not exist on the host. I appreciate your pointing me in the right direction.
... View more
12-21-2015
09:02 AM
1 Kudo
One of our Impala daemons will not start. There is no directory structure being created at: /var/run/cloudera-scm-agent/process/nnnnnn-impala-IMPALAD After the daemon fails to start, the stderr and stdout logs cannot be accessed because the directory structure does not exist. Instead, there is an HTTP 403 error, referencing the expected path, indicating "The server declined access to the page or resource." In addition, nothing is being written to the error, warning, or info logs at /hadoop/log/impalad. What can be done to enable the creation of the directory structure? Deleting and reinstalling the Impala role does not resolve it. There are no issues with other roles on the host. I am wondering whether it is necessary to re-initialize the host and re-install the roles. Michael Reynolds
... View more
11-10-2015
12:27 PM
The cloudera-scm-server.log only shows the commands being run for the deployment, and no errors. I also did not see errors in logs for the related services for which I am deploying configurations. Using the "All Recent Commands" interface, and changing the filter in the command line to limit=10000, I can see that there were a few successful deployments, a couple of weeks ago, and no still-active deployment commands since then; only failures.
... View more
11-03-2015
12:29 PM
Thank you for the reply. Using the maximum of 250 past commands, the only "deploy client configuration" commands listed are the failed attempts. This is a DR cluster, which is doing HDFS replications. Therefore, the command list is filled with the replication commands. Michael Reynolds
... View more
11-03-2015
11:47 AM
I am attempting to deploy the client configuration for HDFS and Hive on CDH 5.3.3, and it fails with this message: There is already a pending command on this entity. The "Running Commands" does not list any deployments. Where would I find these commands listed, so they can be killed? Thank you, Michael Reynolds
... View more
09-21-2015
11:28 AM
This has apparently been resolved, because I am no longer having issues with closed connections during parcel downloads.
... View more
09-20-2015
09:50 PM
I am experiencing a similar problem, which is preventing the parcels from downloading while installing CDH 5.4.5. The percentage of the download process continues incrementing past 100%, and eventually starts over again at 0%, and loops like this indefinitely. I checked /opt/cloudera/parcel-repo and found that the file "CDH-5.4.5-1.cdh5.4.5.p0.7-el6.parcel.part" grows in size and then starts over. I tried downloading "CDH-5.4.5-1.cdh5.4.5.p0.7-el6.parcel" manually from http://archive.cloudera.com/cdh5/parcels/5/ using wget, and it finally succeeds after a few attempts: --2015-09-20 15:37:14-- (try: 4) http://archive.cloudera.com/cdh5/parcels/5/CDH-5.4.5-1.cdh5.4.5.p0.7-el6.parcel
Connecting to archive.cloudera.com|23.235.47.68|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1259852705 (1.2G), 521097747 (497M) remaining
Saving to: âCDH-5.4.5-1.cdh5.4.5.p0.7-el6.parcelâ
81% [++++++++++++++++++++++++++++++++++++++++++===============> ] 1,020,619,926 3.65M/s in 96s
2015-09-20 15:38:50 (2.79 MB/s) - Connection closed at byte 1020619926. Retrying.
--2015-09-20 15:38:54-- (try: 5) http://archive.cloudera.com/cdh5/parcels/5/CDH-5.4.5-1.cdh5.4.5.p0.7-el6.parcel
Connecting to archive.cloudera.com|23.235.47.68|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1259852705 (1.2G), 239232779 (228M) remaining
Saving to: âCDH-5.4.5-1.cdh5.4.5.p0.7-el6.parcelâ
100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=============>] 1,259,852,705 3.92M/s in 83s
Although it eventually downloads manually, the CDH installer does not recognize it, and continues trying to do the download. Apparently, the CDH download process opens a new session on each attempt, instead of appending to the same, incomplete file.
... View more
08-21-2015
02:00 PM
Hi Darren, Thank you!! I overlooked that. I'm using parcels. Michael Reynolds
... View more
08-21-2015
12:57 PM
Greetings, I want to migrate Hue database from SQLite to PostgreSQL, and am following these instructions: "Using an External Database for Hue Using Cloudera Manager" http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_hue_service.html Under "Configuring the Hue Server to Store Data in PostgreSQL", in step 7, after installing the parcel, it says to install the package. The parcel appeared to install fine, however the package directory for CDH 5 is missing: /usr/lib/hue/build/env/bin. Hue does not have a BUILD directory. Perhaps something went wrong with the parcel install. Is there a way to obtain the missing directory structure? Thank you, Michael Reynolds
... View more
07-15-2015
02:07 PM
Hi Alex, Thank you for your message. I apologize for the delay. I finally had a chance to try your suggestion. The INSERT OVERWRITE with aggregations DOES work from the Impala shell. The trouble appears to be caused by running it via Oozie. I will need to do some experiments, because it needs to run in a workflow. Michael
... View more
07-10-2015
03:13 PM
Many of our Impala Parquet tables have aggregated data, in which we group on a number of columns. Sometimes, the data requires a reload, and it can take a while for the entire table. Instead of reloading the entire table, I would like to update a single partition. We routinely compact single partitions, to consolodate files, by doing an INSERT OVERWRITE, uisng the table being updated as the source. Because the data is already aggregated, it's a simple row-by-row udpate. We have no trouble compacting an entire month of data. However, if I attempt to do an INSERT OVERWRITE on a partition, using an aggregation from the source, I recieve this error: org.apache.oozie.action.hadoop.LauncherException: Output data exceeds its limit [2048] I updated the value for oozie.action.max.output.data to 8192, and still get the error: org.apache.oozie.action.hadoop.LauncherException: Output data exceeds its limit [8192] In fact, if I attempt to insert a minimal amount of aggregated data, the error still occurs. Therefore, does Impala support INSERT OVERWRITE uisng aggregated data? If it does support it, what changes must be made? Thank you, Michael Reynolds
... View more
06-03-2015
02:43 PM
We upgraded to CDH 5.4.2 and are now encountering occasional "Database is Locked" errors while attempting to run Impala queries via Hue 3.7.0. We are using sqlite3 as the database engine. According to another message thread, a fix can be applied by adding options='{"timeout": 30}' to the database section of the hue.ini file. re: https://community.cloudera.com/t5/Web-UI-Hue-Beeswax/DatabaseError-database-is-locked/td-p/14070 However, after restarting Hue, configuration files are created in a new directory, such as /var/run/cloudera-scm-agent/process/6086-hue-HUE_SERVER, and our changes to the hue.ini files are no longer present. While I know that the ideal solution is to use mysql as the Hue database engine, what is the correct method to apply these changes for sqllite3, to make the changes persistent? Thank you, Michael Reynolds
... View more
02-07-2015
07:39 PM
Yes. Sqoop appears to stall when an external process it's waiting for fails to complete. Thank you very much for the tip. I plan to develop a procedure to automatically kill workflows that have exceeded a specified length of processing time. Michael Reynolds
... View more
02-02-2015
04:40 PM
We are using CDH 5.2 with YARN, and sometimes encounter stalled Oozie workflows, possibly due to network issues when waiting for a Sqoop query to complete. Killing the coordinator allows the workflow to be rescheduled, however I would like to know whether this can be automated. I have checked the SLA configuration, and it appears that the only action to be taken for a late job is an emailed alert. Is there a way to force a coordinator to be killed after a specified interval? Thank you, Michael Reynolds
... View more
Labels:
01-28-2015
03:38 PM
Thank you for your reply! Here is the command: ===================== import --connect jdbc:mysql://irbasedw-${shard}.db.xxxxxxxxxxx.net:3417/irbasedw_${shard}?dontTrackOpenResources=true&useCursorFetch=true --username iretl --password-file /irdw/${env}/lib/.passwordBaseDw --table base_action_fact2 --target-dir /irdw/${env}/legacy/fact/action_fact2/text/merge_delta/shard_${shard} --as-textfile --escaped-by \\ --hive-delims-replacement \\\\n --null-non-string \\N --null-string \\N --incremental lastmodified --check-column dlu --last-value ${wf:actionData('ObtainMaxDates')['nextDluText']} --where "doe_ts<'${wf:actionData('ObtainMaxDates')['nextDoeText']}'" --fetch-size -2147483648 --columns [ comma-separated column list ] -m 1 ===================== Michael Reynolds
... View more
01-28-2015
03:19 PM
I am using CDH 5.2 with YARN, and am importing from MySQL, which has data sharded across 10 identically-defined tables. I am getting this error when importing updated records for only one of the 10 shards, using a common Sqoop query: 15/01/28 14:48:50 ERROR tool.ImportTool: Imported Failed: The value of property sqoop.merge.key.col must not be null I have been unable to locate documentation for the "sqoop.merge.key.col" property, other than usages in various Java programs. The fact that only one shard is affected implies a problem with the source. However, the error implies an issue with Sqoop. I would appreciate any information or insights about this property. Thank you! Michael Reynolds
... View more
09-22-2014
03:47 PM
I found the running coordinator in the Oozie Web Console. By referencing the job ID in the console, I was able to kill the coodinator via the Oozie -kill command. Michael Reynolds
... View more
09-19-2014
09:54 AM
I want to change the schedule for a running Oozie Coordinator, however it is missing from the list of running coordinators in Hue. Further, it is not listed if I run the following shell command: sudo -u oozie oozie jobs -oozie http://<server-url>:11000/oozie/ -jobtype coordinator | grep RUNNING How is it possible to locate the coodinator, so I can kill it? Thank you, Michael Reynolds
... View more
Labels: