Python中文网

一个关于 编程问题的解答网站.

有 Java 编程相关的问题?

你可以在下面搜索框中键入要查询的问题!

java MapReduce编程错误

3 日，9 小时 Questions & Answers

我的输入是许多文本文件。我希望我的map reduce程序将所有文件名和与文件名相关联的句子写入一个输出文件中，其中我只希望从映射器发出文件名（键）和相关句子（值）。reducer将收集键和所有值，并在输出中写入文件名及其相关语句

以下是我的mapper和reducer的代码：

public class WordCount {
    public static class Map extends MapReduceBase implements Mapper<LongWritable,
    Text, Text, Text> {
        public void map(LongWritable key, Text value, OutputCollector<Text,Text>
        output, Reporter reporter) throws IOException {
            String filename = new String();
            FileSplit filesplit = (FileSplit)reporter.getInputSplit();
            filename=filesplit.getPath().getName();
            output.collect(new Text(filename), value);
        }
    }
    public static class Reduce extends MapReduceBase implements Reducer<Text, Text,
    Text, Text> {
        public void reduce(Text key, Iterable<Text> values, OutputCollector<Text,
        Text> output, Reporter reporter) throws IOException {
            StringBuilder builder = new StringBuilder();
            for(Text value : values) {
                String str = value.toString();
                builder.append(str);
            }
            String valueToWrite=builder.toString();
            output.collect(key, new Text(valueToWrite));
        }
        @Override
        public void reduce(Text arg0, Iterator<Text> arg1,
        OutputCollector<Text, Text> arg2, Reporter arg3)
        throws IOException {
        }
    }
    public static void main(String[] args) throws Exception {
        JobConf conf = new JobConf(WordCount.class);
        conf.setJobName("wordcount");
        conf.setMapperClass(Map.class);
        conf.setReducerClass(Reduce.class);
        conf.setJarByClass(WordCount.class);
        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(Text.class);
        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);
        conf.setNumReduceTasks(1);
        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));
        JobClient.runJob(conf);
    }
}

结果如下：

14/03/21 00:38:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library   
for your platform... using builtin-java classes where applicable
14/03/21 00:38:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the 
arguments. Applications should implement Tool for the same.
14/03/21 00:38:27 WARN mapred.JobClient: No job jar file set.  User classes may not  
be found. See JobConf(Class) or JobConf#setJar(String).
14/03/21 00:38:27 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/21 00:38:27 INFO mapred.FileInputFormat: Total input paths to process : 2
14/03/21 00:38:27 INFO mapred.JobClient: Running job: job_local_0001
14/03/21 00:38:27 INFO util.ProcessTree: setsid exited with exit code 0
14/03/21 00:38:27 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4911b910
14/03/21 00:38:27 INFO mapred.MapTask: numReduceTasks: 1
14/03/21 00:38:27 INFO mapred.MapTask: io.sort.mb = 100
14/03/21 00:38:27 INFO mapred.MapTask: data buffer = 79691776/99614720
14/03/21 00:38:27 INFO mapred.MapTask: record buffer = 262144/327680
14/03/21 00:38:27 INFO mapred.MapTask: Starting flush of map output
14/03/21 00:38:27 INFO mapred.MapTask: Finished spill 0
14/03/21 00:38:27 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And  
is in the process of commiting
14/03/21 00:38:28 INFO mapred.JobClient:  map 0% reduce 0%
14/03/21 00:38:30 INFO mapred.LocalJobRunner:  
file:/root/Desktop/wordcount/sample.txt:0+5371
14/03/21 00:38:30 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
14/03/21 00:38:30 INFO mapred.Task:  Using ResourceCalculatorPlugin :  
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1f8166e5
14/03/21 00:38:30 INFO mapred.MapTask: numReduceTasks: 1
14/03/21 00:38:30 INFO mapred.MapTask: io.sort.mb = 100
14/03/21 00:38:30 INFO mapred.MapTask: data buffer = 79691776/99614720
14/03/21 00:38:30 INFO mapred.MapTask: record buffer = 262144/327680
14/03/21 00:38:30 INFO mapred.MapTask: Starting flush of map output
14/03/21 00:38:30 INFO mapred.MapTask: Finished spill 0
14/03/21 00:38:30 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And      
is in the process of commiting
14/03/21 00:38:31 INFO mapred.JobClient:  map 100% reduce 0%
14/03/21 00:38:33 INFO mapred.LocalJobRunner:  
file:/root/Desktop/wordcount/sample.txt~:0+587
14/03/21 00:38:33 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
14/03/21 00:38:33 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3963b3e
14/03/21 00:38:33 INFO mapred.LocalJobRunner: 
14/03/21 00:38:33 INFO mapred.Merger: Merging 2 sorted segments
14/03/21 00:38:33 INFO mapred.Merger: Down to the last merge-pass, with 2 segments  
left of total size: 7549 bytes
14/03/21 00:38:33 INFO mapred.LocalJobRunner: 
14/03/21 00:38:33 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And  
is in the process of commiting
14/03/21 00:38:33 INFO mapred.LocalJobRunner: 
14/03/21 00:38:33 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to 
commit now
14/03/21 00:38:33 INFO mapred.FileOutputCommitter: Saved output of task  
'attempt_local_0001_r_000000_0' to file:/root/Desktop/wordcount/output
14/03/21 00:38:36 INFO mapred.LocalJobRunner: reduce > reduce
14/03/21 00:38:36 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
14/03/21 00:38:37 INFO mapred.JobClient:  map 100% reduce 100%
14/03/21 00:38:37 INFO mapred.JobClient: Job complete: job_local_0001
14/03/21 00:38:37 INFO mapred.JobClient: Counters: 21
14/03/21 00:38:37 INFO mapred.JobClient:   File Input Format Counters 
14/03/21 00:38:37 INFO mapred.JobClient:     Bytes Read=5958
14/03/21 00:38:37 INFO mapred.JobClient:   File Output Format Counters 
14/03/21 00:38:37 INFO mapred.JobClient:     Bytes Written=8
14/03/21 00:38:37 INFO mapred.JobClient:   FileSystemCounters
14/03/21 00:38:37 INFO mapred.JobClient:     FILE_BYTES_READ=26020
14/03/21 00:38:37 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=117337
14/03/21 00:38:37 INFO mapred.JobClient:   Map-Reduce Framework
14/03/21 00:38:37 INFO mapred.JobClient:     Map output materialized bytes=7557
14/03/21 00:38:37 INFO mapred.JobClient:     Map input records=122
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce shuffle bytes=0
14/03/21 00:38:37 INFO mapred.JobClient:     Spilled Records=244
14/03/21 00:38:37 INFO mapred.JobClient:     Map output bytes=7301
14/03/21 00:38:37 INFO mapred.JobClient:     Total committed heap usage  
(bytes)=954925056
14/03/21 00:38:37 INFO mapred.JobClient:     CPU time spent (ms)=0
14/03/21 00:38:37 INFO mapred.JobClient:     Map input bytes=5958
14/03/21 00:38:37 INFO mapred.JobClient:     SPLIT_RAW_BYTES=185
14/03/21 00:38:37 INFO mapred.JobClient:     Combine input records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce input records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce input groups=2
14/03/21 00:38:37 INFO mapred.JobClient:     Combine output records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce output records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
14/03/21 00:38:37 INFO mapred.JobClient:     Map output records=122

当我使用相同的inputformat（keyvaluetextinputformat.class）配置运行上述映射器和reducer时，它不会在输出中写入任何内容

我应该改变什么来实现我的目标


共 (1) 个答案

  1. # 1 楼答案

    KeyValueTextInputFormat不是您案例的正确输入格式。如果要使用此输入格式，输入中的每一行都应该包含一个键、值对，默认情况下由用户指定的分隔符或制表符分隔。但在您的例子中，输入是“文件集”，您希望作业的输出是“文件名，文件内容”

    实现这一点的方法之一是使用TextInputFormat作为输入格式。我在下面进行了测试，效果良好

    在map函数中获取文件名和文件内容

       public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException 
    {
          String filename = new String();
          FileSplit filesplit = (FileSplit)context.getInputSplit();
          filename=filesplit.getPath().getName();

          context.write(new Text(filename), new Text(value));

    }

    在reduce函数中，我们构建了将成为文件内容的所有值的字符串

    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException 
    {
    StringBuilder builder= new StringBuilder();
        for (Text value : values) 
        {
            String str = value.toString();
            builder.append(str);            
        }
        String valueToWrite= builder.toString();
     context.write(key, new Text(valueToWrite));   
    }    
}

    最后，在job driver类中，将inputformat设置为自定义格式，并将还原数设置为1

            job.setInputFormatClass(TextInputFormat.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        job.setMapperClass(myMapper.class); 
        job.setReducerClass(myReducer.class);
        job.setNumReduceTasks(1);