java Spark ClassNotFoundException:org。日食码头。服务器生物插座连接器
我正在尝试将本地文件读入Spark RDD对象。 我正在用Java的API为Gradle开发Spark。 我的操作系统是Win8。164位
代码：
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
public class Resampling {
public static void main(String[] args) {
String folderName ="C:/Users/data/";
String inputfilename = "file://" + folderName + "trainingset.csv";
SparkConf conf = new SparkConf().setAppName("Simple Application").setMaster("local[2]").set("spark.executor.memory","1g");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> dataSet = sc.textFile(inputfilename).cache();
}
}
stacktrace：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.5/6edffc576ce104ec769d954618764f39f0f0f10d/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/Users/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.7/8095d0b9f7e0a9cd79a663c740e0f8fb31d0e2c8/slf4j-simple-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/02/07 18:01:03 INFO spark.SecurityManager: Changing view acls to: usern
16/02/07 18:01:03 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(usern)
16/02/07 18:01:04 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/02/07 18:01:04 INFO Remoting: Starting remoting
16/02/07 18:01:04 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@bla.corp:65518]
16/02/07 18:01:04 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@cla.corp:65518]
16/02/07 18:01:04 INFO spark.SparkEnv: Registering MapOutputTracker
16/02/07 18:01:04 INFO spark.SparkEnv: Registering BlockManagerMaster
16/02/07 18:01:04 INFO storage.DiskBlockManager: Created local directory at C:\Users\AppData\Local\Temp\spark-local-20160207180104-3802
16/02/07 18:01:04 INFO storage.MemoryStore: MemoryStore started with capacity 2.1 GB.
16/02/07 18:01:04 INFO network.ConnectionManager: Bound socket to port 65521 with id = ConnectionManagerId(bla.corp,65521)
16/02/07 18:01:04 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/02/07 18:01:04 INFO storage.BlockManagerInfo: Registering block manager bla.corp:65521 with 2.1 GB RAM
16/02/07 18:01:04 INFO storage.BlockManagerMaster: Registered BlockManager
16/02/07 18:01:04 INFO spark.HttpServer: Starting HTTP Server
Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/jetty/server/bio/SocketConnector
at org.apache.spark.HttpServer.start(HttpServer.scala:55)
at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:53)
at org.gradle.Resampling.main(Resampling.java:26)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.server.bio.SocketConnector
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 10 more
我最初使用的是jetty9.0.0，但后来我尝试使用类似问题中建议的jetty的旧版本，但仍然很成功
我的身材。gradle依赖项包括：
dependencies {
compile 'com.sparkjava:spark-core:2.2'
compile 'org.apache.spark:spark-mllib_2.10:1.0.0'
compile 'org.eclipse.jetty:jetty-server:7.0.0.RC0'
compile 'org.eclipse.jetty:jetty-servlet:7.0.0.RC0'
testCompile group: 'junit', name: 'junit', version: '4.+'
}
# 1 楼答案
你试过这个吗： Unable to run Apache Spark related unit test in SBT - NoClassDefFoundError