StormCrawler和Hortonworks 1.1.0.2.6.4.091之间的java Commons日志记录版本冲突

Question

我按照http://stormcrawler.net/getting-started/指南为拓扑生成jar文件。当我在readme文件中使用storm命令运行拓扑时，在执行FetcherBolt时出现以下错误。我在Hortonworks集群中安装了storm 1.1.0.2.6.4.0-91。无论是本地模式还是分布式模式，我都会得到相同的异常。 我有以下例外

java.lang.NoSuchMethodError: org.apache.commons.logging.impl.LogFactoryImpl.handleThrowable(Ljava/lang/Throwable;)V at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:568) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at org.apache.http.conn.ssl.AbstractVerifier.(AbstractVerifier.java:61) at org.apache.http.conn.ssl.AllowAllHostnameVerifier.(AllowAllHostnameVerifier.java:44) at org.apache.http.conn.ssl.AllowAllHostnameVerifier.(AllowAllHostnameVerifier.java:46) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.(SSLConnectionSocketFactory.java:146) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:115) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:122) at com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol.(HttpProtocol.java:76) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.digitalpebble.stormcrawler.protocol.ProtocolFactory.(ProtocolFactory.java:60) at com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:738) at org.apache.storm.daemon.executor$fn__9635$fn__9648.invoke(executor.clj:794) at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)

Answer 1

这可能是因为从httpclient库继承的commons日志版本与ApacheStorm的Hortonworks版本放在类路径上的版本之间存在冲突

[INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.9:compile

您可以通过设置

  http.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"
  https.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"

在crawler-conf.yaml文件中。注意，这并不能保证对commons日志的调用不会发生在其他地方。理想情况下，您希望解决依赖性问题，例如确保Hortonworks使用与StormCrawler所需版本相同的版本。我们只支持Storm的Apache发行版