Member since
02-14-2017
15
Posts
7
Kudos Received
0
Solutions
12-28-2017
09:36 PM
Hi All, I have some confusion about the p hysical storage structure of HBase . Is the HBase "Column Database" or "Column Family Database" ? For example, I have a table with tow column family in HBase .Columnfaily1 has three cols and columnfamily2 has two cols, and the data stored in the hbase table are a1 ~ f2. Which of the following answers can represent physical storage structure of HBase , the way of how data store in physical storeage ? And Why ? Your prompt reply will be greatly appreciated ! Thanks Pengcheng
... View more
Labels:
- Labels:
-
Apache HBase
04-19-2017
11:13 PM
@mbigelow, thanks again. 1. Yes exactly, I just want to access to all of the features, especially the Cloudera Navigator feature, and I don't need any technical support so far. As I know there are roughly two Cloudera editions with respectively Express Edition(free) and Enterprise Edition( charged) , so what is the the community version you metioned ? Is the community version free with all features accessible? 2. Just curious for the license issue. When the license is expired, the sales guy can reach me, but they can not "reach" my cluster since my cluster is totally off-line, so there is no way the sales guy can control the accessibility of the feature once the feature is actived. The only possibility I think is that the cluster itself terminates the feature when license is expired. If so, then there are many ways to "reuse" the license in the off-line environment. Let's say, my good friend buy a 3 years license, I am also able to use his license in my cluster without letting the sales guy knows. I know this is illeagle, but just for curious how the cloudera guarantee the "one license one cluster per person" if the cluster is in a totally off-line environment. Thanks
... View more
04-19-2017
08:36 PM
@mbigelow, thanks for your reply. Let's say I upload a license to the CDH cluster, but the CDH cluster is in a totally off-line environment, what if I don't contact them for support, how can they know the license is valid or not in the off-line cluster ?
... View more
04-19-2017
08:12 PM
Hi All, I am planning to upgrade the CDH clusters from Express Edition to Enterprise Edition, but I have a few questions on Cloudera Enterprise Edition License as follows. 1. Where and How to buy the Cloudera Enterprise License, do I have to call the Cloudera custom service to get the license ? 2. Where can I find the subscription price of the Cloudera Enterprise License ? 3. My production environment is in a totally off-line environment, how can Cloudera know when I start to use the License ? Thanks
... View more
Labels:
04-11-2017
08:32 PM
@saranvisa, Yes, either source and target are MySQL. I will try some basic functions , It's really kind of you ! But during my investigation, I found there is an extra argument called --ctrl-chars , but it seems it is only applicable for Netezza , do you have any ideas if this extra argument can also be applied to MySQL ? http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_extra_arguments_3 thanks again for your help.
... View more
04-11-2017
07:09 PM
@csguna, Thanks for your reply. 1. the column with special character is needed in the production environment, and only some of rows in the column contain the special character. So the whole cloumn cannot be skipped. 2. I also thought about that, but the tables are imported into HDFS frequently, so every time when the tables get updated, I need to run the shell script to remove the special character. And the 'special character' may not be able to be predicted, for example, this time it is ^@ , next time, it may become ^A or ^H or something else, as a result, it's hard to use the shell script to remove all special character
... View more
04-11-2017
06:32 PM
@saranvisa, Thanks for your reply, and it's really a good idea to use the 'string_function' to filter the special character. but from my understanding, the applicable string_functions you mentioned are MySQL string functions, right? Is it possible that we write a customize function for specific case?
... View more
04-10-2017
08:05 PM
Hi All, I am using sqoop 1.4.6-chd5.10.0 to transfer data between hive and MySQL. The sqoop-import job was succeed, but the sqoop-export job was failed with the error "Can't parse input data". The main reason I think is that there are some special characters(i.e. ^@) in the HDFS plain file which was imported by sqoop-import as shown in below. The sqoop import command I used is : sqoop import --connect jdbc:mysql://******:3306/smart_voice_data --table sqoop_import_test --username ****** --password ****** --verbose --hive-database hive_smart_voice_data --hive-table hive_sqoop_import_test --lines-terminated-by "\n" --hive-drop-import-delims --split-by id --null-non-string NULL --null-string NULL --fields-terminated-by "|" --delete-target-dir --hive-overwrite The sqoop export command I used is : sqoop export --connect "jdbc:mysql://******:3306/smart_voice_data?useUnicode=true&characterEncoding=gbk" --username ****** --password ****** --table sqoop_hi_filog_goi_vidaa4_normalized --export-dir hdfs://cdh5node1:8020/user/hive/warehouse/hive_smart_voice_data.db/hive_sqoop_import_test/ --input-null-string "\\\\N" --input-null-non-string "\\\\N" --fields-terminated-by '|' --input-lines-terminated-by '\n' Questions: 1 、Is there any way to escape the special character ^@ (\000) when using sqoop to import data into HDFS? 2、 Is there any way to skip the special character ^@(\000) when using sqoop to export data to mysql , or is there any solution to make sqoop-export can parse the special character ^@ ? 3、Is there any way to skip the error and continue the remaining job when doing sqoop-export ? Any help is really appreciated , thanks a lot ! Pengcheng
... View more
Labels:
- Labels:
-
Apache Sqoop
03-30-2017
08:15 PM
Hi @Pythor, I had the same problem, did you find any solutions ? Thanks.
... View more
02-21-2017
10:03 PM
2 Kudos
Hi All, I am new to cloudera, and I just wonder if it is possible to add a custom service(such as Apache Zeppelin or Apache Falcon ) into cloudera manager. Then the health status of the custom service can be monitored by cloudera manager just like the HDFS service, Hive service shown in picture below. Thanks Pengcheng
... View more
Labels:
- Labels:
-
Cloudera Manager
02-16-2017
09:51 PM
Hello @bgooley, thank you very much for your help. I am willing to send you the deployment descriptor, could you please send me your email address through private letter so that I can reply you the output of http://cm_host:7180/api/v14/cm/deployment?view=export_redacted Thanks Pengcheng
... View more
02-15-2017
05:06 PM
Hello @GustavoVarisco, thanks for your reply and it's really a convincing analysis. Could I ask where you get the source code of cloudera manager 5.10 ?
... View more
02-14-2017
11:18 PM
Hello @bgooley @comtefprobtp , I had the same problem as shown in below screenshot. the exception in cloudera-scm-server.log is : 2017-02-15 13:56:35,794 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2017-02-15T05:56:35.794Z
2017-02-15 13:56:36,272 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully.
2017-02-15 13:56:38,584 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2017-02-15T05:56:38.584Z
2017-02-15 13:56:45,251 INFO 1762913326@agentServer-0:com.cloudera.cmf.command.components.StalenessChecker: No staleness check scheduled, scheduling one in 30 seconds
2017-02-15 13:56:54,846 INFO 250577740@scm-web-4:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 192.168.122.22
2017-02-15 13:57:13,249 INFO 250577740@scm-web-4:com.cloudera.server.web.common.ExceptionReport: Exception report generated accessing http://192.168.122.22:7180/cmf/alerts/config
java.lang.NullPointerException
at com.cloudera.server.web.cmf.AlertData.isParamSpecEnabled(AlertData.java:127)
at com.cloudera.server.web.cmf.AlertData.writeBooleanConfigDescription(AlertData.java:105)
at com.cloudera.server.web.cmf.ServiceAlertData.writeHealthDescriptions(ServiceAlertData.java:161)
at com.cloudera.server.web.cmf.ServiceAlertData.<init>(ServiceAlertData.java:64)
at com.cloudera.server.web.cmf.AlertController.alertConfigView(AlertController.java:65)
at com.cloudera.server.web.cmf.AlertController$$FastClassByCGLIB$$46f71363.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.cloudera.server.web.cmf.AlertController$$EnhancerByCGLIB$$4cbeb2e7.alertConfigView(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2017-02-15 13:57:15,253 INFO ScheduledStalenessChecker:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: Client config (id=10, host id=1) has a brand new heartbeat]}.
2017-02-15 13:57:15,289 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Total staleness check tasks: 34
2017-02-15 13:57:15,728 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.458S
2017-02-15 13:57:15,729 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=98ms, min=0ms, max=726ms. Any help is really appreciated ! Btw, I would also like to attach a deployment descriptor , but I don't know how to export it. Thanks Pengcheng
... View more