我想执行与机器学习相关的python文件,如您所知,有两个文件作为输入(训练和测试),这对学习过程非常重要。我也没有减少文件。在
我有三个疑问:
-input file1 -input file2
-D mapred.reduce.tasks=0
sys.stdout.flush()
。在这是我在命令窗口上的命令:
D:\hadoop\bin\hadoop jar D:\hadoop\share\hadoop\tools\lib\hadoop-streaming-2.3.0.jar
-D mapred.reduce.tasks=0
-file /in/Deeplearning.py -mapper "python Deeplearning.py"
-input /in/train.csv -input /in/test.csv
-output /output
我使用:Windows10 64位Python3.6,我的IDE是Spyder3.2.6、Hadoop 2.7.2、JavaJDK1.8.0_
这是我的python代码:
^{pr2}$我的python代码工作正常,我只想在Hadoop生态系统上运行它 当我运行命令时,我得到了:
D:\hadoop\sbin>D:\hadoop\bin\hadoop jar D:\hadoop\share\hadoop\tools\lib\hadoop-streaming-2.3.0.jar -D mapred.reduce.tasks=0 -file /in/Deeplearning.py -mapper "python Deeplearning.py" -input /in/train.csv -input /in/test.csv -output /output12
18/02/24 22:09:37 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
packageJobJar: [/in/Deeplearning.py, /D:/tmp/hadoop-Mahsa/hadoop-unjar6766109988740119098/] [] C:\Users\Mahsa\AppData\Local\Temp\streamjob6727979790632634187.jar tmpDir=null
18/02/24 22:09:39 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/02/24 22:09:39 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/02/24 22:09:41 INFO mapred.FileInputFormat: Total input paths to process : 2
18/02/24 22:09:41 INFO mapreduce.JobSubmitter: number of splits:3
18/02/24 22:09:41 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
18/02/24 22:09:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1519494353792_0004
18/02/24 22:09:42 INFO impl.YarnClientImpl: Submitted application application_1519494353792_0004
18/02/24 22:09:42 INFO mapreduce.Job: The url to track the job: http://Mahsa:8088/proxy/application_1519494353792_0004/
18/02/24 22:09:42 INFO mapreduce.Job: Running job: job_1519494353792_0004
18/02/24 22:09:55 INFO mapreduce.Job: Job job_1519494353792_0004 running in uber mode : false
18/02/24 22:09:55 INFO mapreduce.Job: map 0% reduce 0%
18/02/24 22:11:09 INFO mapreduce.Job: map 75% reduce 0%
18/02/24 22:11:09 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000002_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:11:10 INFO mapreduce.Job: map 42% reduce 0%
18/02/24 22:11:12 INFO mapreduce.Job: map 64% reduce 0%
18/02/24 22:11:15 INFO mapreduce.Job: map 67% reduce 0%
18/02/24 22:11:24 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:11:25 INFO mapreduce.Job: map 33% reduce 0%
18/02/24 22:11:28 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:11:29 INFO mapreduce.Job: map 0% reduce 0%
18/02/24 22:12:38 INFO mapreduce.Job: map 21% reduce 0%
18/02/24 22:12:39 INFO mapreduce.Job: map 74% reduce 0%
18/02/24 22:12:39 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000002_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:12:40 INFO mapreduce.Job: map 41% reduce 0%
18/02/24 22:12:41 INFO mapreduce.Job: map 53% reduce 0%
18/02/24 22:12:42 INFO mapreduce.Job: map 65% reduce 0%
18/02/24 22:12:47 INFO mapreduce.Job: map 67% reduce 0%
18/02/24 22:12:54 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000001_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:12:55 INFO mapreduce.Job: map 33% reduce 0%
18/02/24 22:12:56 INFO mapreduce.Job: map 34% reduce 0%
18/02/24 22:12:57 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:12:58 INFO mapreduce.Job: map 0% reduce 0%
18/02/24 22:14:11 INFO mapreduce.Job: map 29% reduce 0%
18/02/24 22:14:12 INFO mapreduce.Job: map 62% reduce 0%
18/02/24 22:14:12 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000002_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:14:13 INFO mapreduce.Job: map 29% reduce 0%
18/02/24 22:14:14 INFO mapreduce.Job: map 56% reduce 0%
18/02/24 22:14:18 INFO mapreduce.Job: map 67% reduce 0%
18/02/24 22:14:25 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:14:26 INFO mapreduce.Job: map 33% reduce 0%
18/02/24 22:14:31 INFO mapreduce.Job: map 34% reduce 0%
18/02/24 22:14:31 INFO mapreduce.Job: Task Id : attempt_1519494353792_0004_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
18/02/24 22:14:32 INFO mapreduce.Job: map 0% reduce 0%
18/02/24 22:14:46 INFO mapreduce.Job: map 33% reduce 0%
18/02/24 22:14:47 INFO mapreduce.Job: map 100% reduce 0%
18/02/24 22:15:02 INFO mapreduce.Job: Job job_1519494353792_0004 failed with state FAILED due to: Task failed task_1519494353792_0004_m_000002
Job failed as tasks failed. failedMaps:1 failedReduces:0
18/02/24 22:15:02 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=10
Killed map tasks=2
Launched map tasks=12
Other local map tasks=9
Data-local map tasks=3
Total time spent by all maps in occupied slots (ms)=857100
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=857100
Total vcore-seconds taken by all map tasks=857100
Total megabyte-seconds taken by all map tasks=877670400
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
18/02/24 22:15:02 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
提前谢谢你的帮助或任何想法?在
目前没有回答
相关问题 更多 >
编程相关推荐