edu.umn.cs.spatialHadoop.mapred

Class FileSplitUtil

java.lang.Object

edu.umn.cs.spatialHadoop.mapred.FileSplitUtil

public class FileSplitUtil
extends java.lang.Object

A set of method and algorithms used to support management of file splits.

Author:

eldawy

Method Summary

Methods

Modifier and Type	Method and Description
static InputSplit[]	autoCombineSplits(JobConf conf, java.util.Vector<FileSplit> inputSplits, int numSplits) Combines a number of input splits into the given numSplits.
static InputSplit	combineFileSplits(JobConf conf, FileSplit split1, FileSplit split2) Combines two file splits into a CombineFileSplit.
static InputSplit	combineFileSplits(JobConf conf, java.util.List<FileSplit> splits, int startIndex, int count) Combines a number of file splits into one CombineFileSplit.
static java.lang.String[]	prioritizeLocations(java.util.Vector<java.lang.String> vlocations) Takes a list of locations as a vector, and returns a unique array of locations where locations on the head are more frequent in the original vector than the ones on the tail.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

combineFileSplits

public static InputSplit combineFileSplits(JobConf conf,
                           java.util.List<FileSplit> splits,
                           int startIndex,
                           int count)
                                    throws java.io.IOException

Combines a number of file splits into one CombineFileSplit. If number of splits to be combined is one, it returns this split as is without creating a CombineFileSplit.

Parameters:

splits -

startIndex -

count -

Returns:

Throws:

java.io.IOException

combineFileSplits

public static InputSplit combineFileSplits(JobConf conf,
                           FileSplit split1,
                           FileSplit split2)
                                    throws java.io.IOException

Combines two file splits into a CombineFileSplit.

Parameters:

conf -

split1 -

split2 -

Returns:

Throws:

java.io.IOException

prioritizeLocations

public static java.lang.String[] prioritizeLocations(java.util.Vector<java.lang.String> vlocations)

Takes a list of locations as a vector, and returns a unique array of locations where locations on the head are more frequent in the original vector than the ones on the tail.

Parameters:

vlocations - - A vector of locations with possible duplicates

Returns:

- A unique array of locations.

autoCombineSplits

public static InputSplit[] autoCombineSplits(JobConf conf,
                             java.util.Vector<FileSplit> inputSplits,
                             int numSplits)
                                      throws java.io.IOException

Combines a number of input splits into the given numSplits.

Parameters:

conf -

inputSplits -

numSplits -

Returns:

Throws:

java.io.IOException