java Solr 7：当某些请求命中Solr时，QueuedThreadPool线程数很高

Question

我们正在测试环境中运行一个包含3个节点的solr 7集群。 我们有一个定制的请求处理程序/响应编写器，用于在solr中添加一个端点，以支持我们的业务逻辑

硬件配置：32 GB RAM，8核

当服务器上的负载适度增加（比如每秒1-3个请求）时，我们会看到很多线程在等待锁

我们的所有方面字段都是docValues=true。 我们还使用大约20-25个方面字段和大约30-40个方面查询来进行大量的方面处理

我的solr索引中的文档数量约为：550万

 qtp1597462040-382 thread obtained org.apache.solr.uninverting.FieldCache$CreationPlaceholder's lock & did not release it. Due to that 125 threads are BLOCKED as shown in the below graph. If threads are BLOCKED for prolonged period, application will become unresponsive. Examine 'qtp1597462040-382' stacktrace to see why lock is not released.

The stacktrace of the thread that is blocking all the other similar threads is as follows:


  qtp1597462040-382 - threadId:382 - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.codecs.lucene50.ForUtil.readBlock(ForUtil.java:203)
at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.refillDocs(Lucene50PostingsReader.java:357)
at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.nextDoc(Lucene50PostingsReader.java:382)
at org.apache.solr.uninverting.DocTermOrds.uninvert(DocTermOrds.java:371)
at org.apache.solr.uninverting.DocTermOrds.<init>(DocTermOrds.java:222)
at org.apache.solr.uninverting.DocTermOrds.<init>(DocTermOrds.java:213)
at org.apache.solr.uninverting.DocTermOrds.<init>(DocTermOrds.java:206)
at org.apache.solr.uninverting.FieldCacheImpl$DocTermOrdsCache.createValue(FieldCacheImpl.java:1249)
at org.apache.solr.uninverting.FieldCacheImpl$Cache.get(FieldCacheImpl.java:199)
- locked <3dad1e71> (a org.apache.solr.uninverting.FieldCache$CreationPlaceholder)
at org.apache.solr.uninverting.FieldCacheImpl.getDocTermOrds(FieldCacheImpl.java:1236)
at org.apache.solr.uninverting.UninvertingReader.getSortedSetDocValues(UninvertingReader.java:369)
at org.apache.lucene.index.FilterLeafReader.getSortedSetDocValues(FilterLeafReader.java:390)
at org.apache.lucene.index.DocValues.getSortedSet(DocValues.java:429)
at org.apache.lucene.queries.function.valuesource.SortedSetFieldSource.getValues(SortedSetFieldSource.java:59)
at org.apache.lucene.queries.function.valuesource.SimpleBoolFunction.getValues(SimpleBoolFunction.java:48)
at org.apache.lucene.queries.function.valuesource.SimpleBoolFunction.getValues(SimpleBoolFunction.java:35)
at org.apache.lucene.queries.function.valuesource.SimpleBoolFunction.getValues(SimpleBoolFunction.java:48)
at org.apache.lucene.queries.function.valuesource.SimpleBoolFunction.getValues(SimpleBoolFunction.java:35)
at org.apache.lucene.queries.function.valuesource.MultiBoolFunction.getValues(MultiBoolFunction.java:49)
at org.apache.lucene.queries.function.valuesource.MultiBoolFunction.getValues(MultiBoolFunction.java:33)
at org.apache.lucene.queries.function.valuesource.IfFunction.getValues(IfFunction.java:47)
at org.apache.lucene.queries.function.valuesource.MultiFloatFunction.getValues(MultiFloatFunction.java:76)
at org.apache.lucene.queries.function.valuesource.MultiFloatFunction.getValues(MultiFloatFunction.java:76)
at org.apache.lucene.queries.function.valuesource.DualFloatFunction.getValues(DualFloatFunction.java:56)
at org.apache.lucene.queries.function.ValueSource$ValueSourceComparator.doSetNextReader(ValueSource.java:429)
at org.apache.lucene.search.SimpleFieldComparator.getLeafComparator(SimpleFieldComparator.java:36)
at org.apache.lucene.search.FieldValueHitQueue.getComparators(FieldValueHitQueue.java:180)
at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector.getLeafCollector(TopFieldCollector.java:122)
at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:121)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:651)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1698)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1415)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)
at org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
at com.my.match.index.solr.MySearchRequestHandler.handleRequestBody(MySearchRequestHandler.java:472)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEnd

Point$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:

除此之外，我在下面的stacktrace中看到其他几个线程非常繁忙

    at org.apache.solr.request.DocValuesFacets.accumMultiSeg(DocValuesFacets.java:366)
at org.apache.solr.request.DocValuesFacets.accumMulti(DocValuesFacets.java:326)
at org.apache.solr.request.DocValuesFacets.getCounts(DocValuesFacets.java:152)
 at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:595)
at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:426)
at org.apache.solr.request.SimpleFacets.lambda$getFacetFieldCounts$0(SimpleFacets.java:834)
at org.apache.solr.request.SimpleFacets$$Lambda$365/532253544.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.request.SimpleFacets$3.execute(SimpleFacets.java:773)
 at com.my.match.index.solr.MySearchRequestHandler.handleRequestBody(MySearchRequestHandler.java:472)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)

有几个字段是“text_general”，经过分析后仍在facets中使用。这会有问题吗？我知道在这些领域中，刻面可能很困难

我们还能分享什么信息