Web服务中的java OutOfMemoryError（无法创建新的本机线程）

Question

在我们公司，我们有一个来自AmazonWebServices的EC2实例m1。中型和运行CentOS

我们有一个Tomcat 8（从os存储库安装的Tomcat），运行一个web服务

Tomcat和web服务似乎运行正常，但当抛出OutOfMemoryError并且WS的方法无法执行时，就会出现这种情况。唯一可以做的就是重新启动Tomcat。这是堆栈跟踪示例：

组织。玻璃鱼。运动衫服务器ContainerException:java。lang.OutOfMemoryError:无法创建新的本机线程 根本原因

java.lang.OutOfMemoryError: unable to create new native thread java.lang.Thread.start0(Native Method) java.lang.Thread.start(Thread.java:714) com.amazonaws.http.IdleConnectionReaper.registerConnectionManager(IdleConnectionReaper.java:80) com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:49) com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:113) com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:213) com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:136) com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:481) com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:460) com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:442) com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:413) com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:395) com.ingartek.ws.zendesk_bizkaibus.service.AWSGateway.(AWSGateway.java:56) com.ingartek.ws.zendesk_bizkaibus.service.PlantillasService.(PlantillasService.java:39) com.ingartek.ws.zendesk_bizkaibus.service.GeneralService.getDocumentoRespuestaQueja(GeneralService.java:894) com.ingartek.ws.zendesk_bizkaibus.rest.DocumentosResource.generarDocumentoRespuestaQueja(DocumentosResource.java:113) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) org.glassfish.jersey.internal.Errors.process(Errors.java:315) org.glassfish.jersey.internal.Errors.process(Errors.java:297) org.glassfish.jersey.internal.Errors.process(Errors.java:267) org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.apache.catalina.filters.CorsFilter.handleSimpleCORS(CorsFilter.java:301) org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:169)

我一直在寻找这个问题的解决方案，找到了几个有趣的页面，比如this。在我看来，这个问题与webservice和tomcat无关，而是与操作系统有关。为什么？当抛出异常时，我访问Tomcat服务器状态页面，没有看到jvm内存耗尽。例如，现在看起来是这样的：

Free memory: 563.22 MB Total memory: 989.93 MB Max memory: 3959.50 MB

那么，我能做什么？谢谢你的建议

注意：我运行VisualVM是为了监视JVM并查找内存、cpu和线程

编辑

我在日志中发现以下内容：

23-Jan-2017 09:50:49.450 ADVERTENCIA [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ws_zendesk_bizkaibus] appears to have started a thread named [New I/O boss #3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)