私はhadoop.imの初心者で、二次ソートの実行中にヌルポインタ例外を取得しています
これは私のマッパークラスです
public void map(LongWritable key, Text value,
OutputCollector<Text, Employee> outputCollector, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
String employeeId = value.toString().split(",")[0];
String employeeName= value.toString().split(",")[1];
String employeeDept= value.toString().split(",")[2];
String employeejoinDate= value.toString().split(",")[3];
String employeSalary= value.toString().split(",")[4];
//System.out.println(employeSalary);
Employee employee=new Employee(Integer.parseInt(employeeId),employeeName,employeeDept,employeejoinDate,Integer.parseInt(employeSalary));
outputCollector.collect(new Text(employeeName),employee);
}
これは私の減速機です
public void reduce(Text arg0, Iterator<Employee> arg1,
OutputCollector<NullWritable,IntWritable> arg2, Reporter arg3)
throws IOException {
// TODO Auto-generated method stub
System.out.println("inside reducer");
while(arg1.hasNext()){
arg2.collect(NullWritable.get(),new IntWritable(arg1.next().getEmployeeSalary()));
}
これは私の従業員クラスです
public class Employee implements WritableComparable<Employee>{
private int employeeId;
private String employeeName;
private String employeeDept;
private String employeeJoinDt;
private int employeeSalary;
public Employee(int employeeId,String employeeName,String employeeDept,String employeeJoinDt,int employeeSalary){
this.employeeId=employeeId;
this.employeeName=employeeName;
this.employeeDept=employeeDept;
this.employeeJoinDt=employeeJoinDt;
this.employeeSalary=employeeSalary;
}
public int getEmployeeId() {
return employeeId;
}
public void setEmployeeId(int employeeId) {
this.employeeId = employeeId;
}
public String getEmployeeName() {
return employeeName;
}
public void setEmployeeName(String employeeName) {
this.employeeName = employeeName;
}
public String getEmployeeDept() {
return employeeDept;
}
public void setEmployeeDept(String employeeDept) {
this.employeeDept = employeeDept;
}
public String getEmployeeJoinDt() {
return employeeJoinDt;
}
public void setEmployeeJoinDt(String employeeJoinDt) {
this.employeeJoinDt = employeeJoinDt;
}
public int getEmployeeSalary() {
return employeeSalary;
}
public void setEmployeeSalary(int employeeSalary) {
this.employeeSalary = employeeSalary;
}
@Override
public void readFields(DataInput input) throws IOException {
// TODO Auto-generated method stubt
this.employeeId=input.readInt();
this.employeeName=input.readUTF();
this.employeeDept=input.readUTF();
this.employeeJoinDt=input.readUTF();
this.employeeSalary=input.readInt();
}
@Override
public void write(DataOutput output) throws IOException {
// TODO Auto-generated method stub
output.writeInt(this.employeeId);
output.writeUTF(this.employeeName);
output.writeUTF(this.employeeDept);
output.writeUTF(this.employeeJoinDt);
output.writeInt(this.employeeSalary);
}
public int compareTo(Employee employee) {
// TODO Auto-generated method stub
if(this.employeeSalary>employee.getEmployeeSalary())
return 1;
else if(this.employeeSalary<employee.getEmployeeSalary())
return -1;
else
return 0;
}
}
これは私のソートコンパレータクラスです
public class SecondarySortComparator extends WritableComparator {
public SecondarySortComparator(){
super(Employee.class);
System.out.println("sort");
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
// TODO Auto-generated method stub
Employee employee1 = (Employee)a;
Employee employee2 = (Employee)b;
int i = employee1.getEmployeeSalary()>employee2.getEmployeeSalary()?1:-1;
return i;
}
これは私のgroupoコンパレータクラスです
public class SecondarySortGroupingComparator extends WritableComparator{
public SecondarySortGroupingComparator(){
super(Employee.class,true);
System.out.println("group");
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
// TODO Auto-generated method stub
Employee employee1 = (Employee)a;
Employee employee2 = (Employee)b;
return employee1.getEmployeeName().compareTo(employee2.getEmployeeName());
}
}
これは私が得ているエラーです
13/09/01 19:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/01 19:13:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/01 19:13:47 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/09/01 19:13:47 INFO mapred.FileInputFormat: Total input paths to process : 1
13/09/01 19:13:47 INFO mapred.JobClient: Running job: job_local_0001
13/09/01 19:13:47 INFO util.ProcessTree: setsid exited with exit code 0
13/09/01 19:13:47 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b3f8f6
13/09/01 19:13:47 INFO mapred.MapTask: numReduceTasks: 1
13/09/01 19:13:47 INFO mapred.MapTask: io.sort.mb = 100
13/09/01 19:13:48 INFO mapred.JobClient: map 0% reduce 0%
13/09/01 19:13:48 INFO mapred.MapTask: data buffer = 79691776/99614720
sort13/09/01 19:13:48 INFO mapred.MapTask: record buffer = 262144/327680
1
1
1
1
13/09/01 19:13:49 INFO mapred.MapTask: Starting flush of map output
13/09/01 19:13:49 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
13/09/01 19:13:49 INFO mapred.JobClient: Job complete: job_local_0001
13/09/01 19:13:49 INFO mapred.JobClient: Counters: 0
13/09/01 19:13:49 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at secondarysort.JobRunner.main(JobRunner.java:31)
この問題を解決する方法について何か提案はありますか?
前もって感謝します