database settings are correct , even i removed the oozie and added it back with new database /username/password  and it was able to connect with mysql but its failing at time of creation of tables in mysql  with error    Wed May 23 14:11:22 EDT 2018 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using 5 as CDH_VERSION using /var/lib/oozie/tomcat-deployment as CATALINA_BASE CONF_DIR=/run/cloudera-scm-agent/process/196-oozie-OOZIE_SERVER CMF_CONF_DIR=/etc/cloudera-scm-agent Copying JDBC jar from /usr/share/java/mysql-connector-java.jar to /var/lib/oozie ERROR: Oozie could not be started REASON: org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Could not load password for [oozie.service.JPAService.jdbc.password] Stacktrace: ----------------------------------------------------------------- org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Could not load password for [oozie.service.JPAService.jdbc.password] at org.apache.oozie.service.Services.loadServices(Services.java:309) at org.apache.oozie.service.Services.init(Services.java:213) at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.IllegalArgumentException: Could not load password for [oozie.service.JPAService.jdbc.password] at org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:598) at org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:585) at org.apache.oozie.service.JPAService.init(JPAService.java:160) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.setService(Services.java:372) at org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:591) ... 31 more Caused by: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2118) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2037) ... 36 more Caused by: java.io.IOException: Bad configuration of hadoop.security.credential.provider.path at localjceks://file/{{CMF_CONF_DIR}}/creds.localjceks at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:86) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2098) ... 37 more Caused by: java.net.URISyntaxException: Illegal character in path at index 18: localjceks://file/{{CMF_CONF_DIR}}/creds.localjceks at java.net.URI$Parser.fail(URI.java:2829) at java.net.URI$Parser.checkChars(URI.java:3002) at java.net.URI$Parser.parseHierarchical(URI.java:3086) at java.net.URI$Parser.parse(URI.java:3034) at java.net.URI.<init>(URI.java:595) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:67) ... 38 more ----------------------------------------------------------------- ... View more

user mapred:hadoop for logs in /tmp ... View more

Thanks alex. yeah i used distcp for hbase files as well and copied to s3. i was also able to restore that backup another cluster but you need to repair meta offline . my requirement was to copy everything to s3 and spin off ec2 instance and than later restore data from s3 backup if require. Export hbase table is also a good option but than you need to have a enough space on /tmp since export copy tables locally before copying it to s3. we ran put of space with this method so decided to copy files. ... View more

ok , this is because of jar files which was deployed in aux path . these jar was complied with old hadoop version and forcing beeline to load configuration from there. you can remove these jars or complied them against new hadoop version ... View more

did you find the solution for this ?   anyone know about this issue  ... View more

Looks like this is resloved in hadoop 2.8.0 ( not sure though)  check this ==.> https://github.com/minio/minio/issues/2965   only workaround i found is first load data without  encryption and then enable encryption  on file copied in S3 (manually) .    btw i have a general question . this SSE is to portect data in S3 only ,what about if someone with aws admin role download data to local disk ,its not more encrypted data    ... View more

Let me know what info you need , i can share . ... View more

guys any thought /suggestion on this ... View more

let me explain with example    top - 10:57:50 up 58 days, 37 min, 7 users, load average: 257.91, 256.55, 253.75 Tasks: 1322 total, 1 running, 1302 sleeping, 0 stopped, 19 zombie %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 82.3 id, 17.5 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 26370188+total, 12234492 free, 51030360 used, 20043704+buff/cache KiB Swap: 8388604 total, 5778116 free, 2610488 used. 21161840+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 39762 root 39 19 1411748 35056 3712 S 6.0 0.0 95:15.54 receptor 16806 impala 20 0 46.612g 5.321g 28840 S 1.0 2.1 732:22.32 impalad 22047 root 20 0 158908 3472 1540 R 1.0 0.0 0:00.08 top 12285 hbase 20 0 26.438g 0.018t 13132 S 0.7 7.5 1775:40 java 65 root 20 0 0 0 0 S 0.3 0.0 58:05.24 rcuos/14 11525 yarn 20 0 4991204 1.977g 14540 S 0.3 0.8 35206:02 java 12416 hbase 20 0 114580 2156 692 S 0.3 0.0 218:57.29 hbase.sh 17528 complia+ 20 0 3182376 843428 27068 S 0.3 0.3 8:00.97 java 17888 complia+ 20 0 3181380 842192 26796 S 0.3 0.3 8:35.77 java 20861 complia+ 20 0 3195300 882332 26860 S 0.3 0.3 8:51.63 java 32800 hive 20 0 3448532 1.047g 25244 S 0.3 0.4 6:42.25 java   this server hot hung with more load . since this one of the impalad server ,now imapad is not working on this server which cause whole impala service to hang .    i am thinking why  issue in one impalad server causing whole impala service to hang .    after restarting this hung server impala works fine . ... View more

Thanks Tim for the update ... View more

  at first login it should ask username/password to create new account  but i am not getting that option .  i am gettinig login page . ... View more

ok after coiple of troubleshooting it moved and completed Cloudera Data Science Workbench is not ready yet: some system pods are not ready Master node configuration successful. The application may take up to 10 minutes to initially startup. To check application status use: $ watch cdsw status but Every 2.0s: cdsw status Mon Jul 10 23:49:23 2017 Cloudera Data Science Workbench Status Service Status docker: active kubelet: active nfs: active Checking kernel parameters... Node Status Cloudera Data Science Workbench is not ready yet: kubectl command failed [root@docker ~]# kubectl get pods --show-all NAME READY STATUS RESTARTS AGE cron-2934152315-mqsei 1/1 Running 0 20m db-39862959-2gt3u 1/1 Running 0 20m db-migrate-052787a-qhp19 0/1 Completed 0 20m engine-deps-xfrsh 1/1 Running 0 20m ingress-controller-3138093376-eacx8 1/1 Running 0 20m livelog-1900214889-e8oys 1/1 Running 0 20m reconciler-459456250-t057g 1/1 Running 0 20m spark-port-forwarder-guxvb 1/1 Running 0 20m web-3826671331-1fp2r 1/1 Running 0 20m web-3826671331-66myb 1/1 Running 0 20m web-3826671331-z8ark 1/1 Running 0 20m is it still downloading the images ? ... View more

ok , i have reset the cdsw and started again but this time it stuck with <master/pki> created keys and certificates in "/etc/kubernetes/pki" <util/kubeconfig> created "/etc/kubernetes/kubelet.conf" <util/kubeconfig> created "/etc/kubernetes/admin.conf" <master/apiclient> created API client configuration <master/apiclient> created API client, waiting for the control plane to become ready <master/apiclient> all control plane components are healthy after 18.555501 seconds <master/apiclient> waiting for at least one node to register and become ready. proxy i checked and looks ok. kubectl get pods does not return anything ... View more

It moved after changing proxy setting and now waiting in infinite loop <master/addons> created essential addon: kube-proxy <master/addons> created essential addon: kube-dns Kubernetes master initialised successfully! You can now join any number of machines by running the following on each node: kubeadm join --token=2a582a.63ec0427495ec31c 10.17.160.64 Added bootstrap token KUBE_TOKEN to /etc/cdsw/config/cdsw.conf node "docker.test.com" tainted daemonset "weave-net" created Waiting for kube-system cluster to come up. This could take a few minutes... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... Some pods in kube-system have not yet started. This may take a few minutes. Waiting for 10 seconds before checking again... ... View more

did you find any solution to this error . i am also getting same . ... View more

No [root@docker ~]# docker pull gcr.io/google_containers/pause-amd64:3.0 Error response from daemon: Get https://gcr.io/v1/_ping: authenticationrequired 5:21:54.693338443-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:21:55.693601137-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:21:55.955985334-04:00" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: authenticationrequired" 15:21:55.956065010-04:00" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: authenticationrequired" 15:21:56.000584332-04:00" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: authenticationrequired" 15:21:56.000652326-04:00" level=error msg="Handler for POST /images/create returned error: Get https://gcr.io/v1/_ping: authenticationrequired" 15:21:58.396828751-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:21:58.696481059-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:06.695327527-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:07.692817530-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:13.393313446-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:14.693889069-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:19.693560635-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:22.693515008-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: 15:22:22.787636243-04:00" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: authenticationrequired" 15:22:22.787671995-04:00" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: authenticationrequired" 15:22:22.830792991-04:00" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: authenticationrequired" 15:22:22.830856352-04:00" level=error msg="Handler for POST /images/create returned error: Get https://gcr.io/v1/_ping: authenticationrequired" but if it is able to pull other images then why failing for this one REPOSITORY TAG IMAGE ID CREATED SIZE docker.repository.cloudera.com/cdsw/1.0.1/third-party/weaveexec 1.9.0 300f92429697 5 months ago 90.4 MB ... View more

14:15:54.742150498-04:00" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: authenticationrequired" 14:15:54.742216009-04:00" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: authenticationrequired" what authentication it is looking for ? ... View more

after putting proxy setting it moved and now stuck at <util/kubeconfig> created "/etc/kubernetes/admin.conf" <master/apiclient> created API client configuration <master/apiclient> created API client, waiting for the control plane to become ready [root@docker ~]# systemctl status docker ● docker.service - docker Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2017-07-10 14:04:12 EDT; 7min ago Docs: https://docs.docker.com Main PID: 20705 (dockerd) Memory: 51.1M CGroup: /system.slice/docker.service ├─20705 dockerd --log-driver=journald -s devicemapper --storage-opt dm.basesize=100G --storage-opt dm.thinpooldev=/dev/mapper... └─20720 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout... journalctl -u docker 10T14:12:06.695585728-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:07.695941688-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:12.395203820-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:12.695987744-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:17.694514892-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:20.696049524-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:25.393880714-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:27.695855879-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:30.695721595-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:32.695268088-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google 10T14:12:37.392576223-04:00" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google [root@docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.repository.cloudera.com/cdsw/1.0.1/third-party/weaveexec 1.9.0 300f92429697 5 months ago 90.4 MB ... View more

it could be kerberos configuration issue on informatica server side. can you share more detail ? ... View more

Are you getting any error while connecting ? what kind of security is enabled for LDAP ? ... View more