edu.umn.cs.spatialHadoop.operations

Class Repartition

java.lang.Object

edu.umn.cs.spatialHadoop.operations.Repartition

public class Repartition
extends java.lang.Object

Creates an index on an input file

Author:

Ahmed Eldawy

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Repartition.RepartitionMap<T extends Shape> The map class maps each object to every cell it overlaps with.
static class	Repartition.RepartitionMapNoReplication<T extends Shape> The map class maps each object to the cell with maximum overlap.
static class	Repartition.RepartitionOutputCommitter
static class	Repartition.RepartitionReduce<T extends Shape>

Constructor Summary

Constructors

Constructor and Description
Repartition()

Method Summary

Methods

Modifier and Type	Method and Description
static int	calculateNumberOfPartitions(Configuration conf, long inFileSize, FileSystem outFs, Path outFile, long blockSize) Calculates number of partitions required to index the given file
static void	main(java.lang.String[] args) Entry point to the operation.
static CellInfo[]	packInRectangles(Path[] files, Path outFile, OperationsParams params, Rectangle fileMBR)
static <S extends Shape> CellInfo[]	packInRectangles(Path inFile, Path outFile, OperationsParams params)
static <S extends Shape> CellInfo[]	packInRectangles(Path inFile, Path outFile, OperationsParams params, Rectangle fileMBR)
static void	repartition(Path inFile, Path outputPath, OperationsParams params)
static <S extends Shape> void	repartitionLocal(Path inFile, Path outFile, CellInfo[] cells, OperationsParams params)
static <S extends Shape> void	repartitionLocal(Path inFile, Path outFile, OperationsParams params)
static <S extends Shape> void	repartitionLocal(Path in, Path out, S stockShape, long blockSize, CellInfo[] cells, java.lang.String sindex, boolean overwrite) Deprecated.
static <S extends Shape> void	repartitionLocal(Path inFile, Path outFile, S stockShape, long blockSize, java.lang.String sindex, boolean overwrite) Deprecated. this method is replaced with repartitionLocal(Path, Path, OperationsParams)
static void	repartitionMapReduce(Path inFile, Path outPath, CellInfo[] cellInfos, OperationsParams params)
static void	repartitionMapReduce(Path inFile, Path outPath, Shape stockShape, long blockSize, CellInfo[] cellInfos, java.lang.String sindex, boolean overwrite) Repartitions an input file according to the given list of cells.
static void	repartitionMapReduce(Path inFile, Path outPath, Shape stockShape, long blockSize, java.lang.String sindex, boolean overwrite) Deprecated. this method is replaced with #repartitionMapReduce(Path, Path, OperationsParams)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Repartition

public Repartition()

Method Detail

calculateNumberOfPartitions

public static int calculateNumberOfPartitions(Configuration conf,
                              long inFileSize,
                              FileSystem outFs,
                              Path outFile,
                              long blockSize)
                                       throws java.io.IOException

Calculates number of partitions required to index the given file

Parameters:

inFs -

inFile -

rtree -

Returns:

Throws:

java.io.IOException

repartitionMapReduce

@Deprecated
public static void repartitionMapReduce(Path inFile,
                                   Path outPath,
                                   Shape stockShape,
                                   long blockSize,
                                   java.lang.String sindex,
                                   boolean overwrite)
                                 throws java.io.IOException

Deprecated. this method is replaced with #repartitionMapReduce(Path, Path, OperationsParams)

Repartitions a file that is already in HDFS. It runs a MapReduce job that partitions the file into cells, and writes each cell separately.

Parameters:

conf -

inFile -

outPath -

gridInfo -

stockShape -

pack -

rtree -

overwrite -

Throws:

java.io.IOException

repartitionMapReduce

public static void repartitionMapReduce(Path inFile,
                        Path outPath,
                        CellInfo[] cellInfos,
                        OperationsParams params)
                                 throws java.io.IOException

Throws:

java.io.IOException

repartitionMapReduce

public static void repartitionMapReduce(Path inFile,
                        Path outPath,
                        Shape stockShape,
                        long blockSize,
                        CellInfo[] cellInfos,
                        java.lang.String sindex,
                        boolean overwrite)
                                 throws java.io.IOException

Repartitions an input file according to the given list of cells.

Parameters:

inFile -

outPath -

cellInfos -

pack -

rtree -

overwrite -

Throws:

java.io.IOException - #repartitionMapReduce(Path, Path, OperationsParams) instead.

packInRectangles

public static <S extends Shape> CellInfo[] packInRectangles(Path inFile,
                                            Path outFile,
                                            OperationsParams params)
                                   throws java.io.IOException

Throws:

java.io.IOException

packInRectangles

public static <S extends Shape> CellInfo[] packInRectangles(Path inFile,
                                            Path outFile,
                                            OperationsParams params,
                                            Rectangle fileMBR)
                                   throws java.io.IOException

Throws:

java.io.IOException

packInRectangles

public static CellInfo[] packInRectangles(Path[] files,
                          Path outFile,
                          OperationsParams params,
                          Rectangle fileMBR)
                                   throws java.io.IOException

Throws:

java.io.IOException

repartitionLocal

@Deprecated
public static <S extends Shape> void repartitionLocal(Path inFile,
                                                 Path outFile,
                                                 S stockShape,
                                                 long blockSize,
                                                 java.lang.String sindex,
                                                 boolean overwrite)
                             throws java.io.IOException

Deprecated. this method is replaced with repartitionLocal(Path, Path, OperationsParams)

Parameters:

inFile -

outFile -

stockShape -

blockSize -

sindex -

overwrite -

Throws:

java.io.IOException

repartitionLocal

public static <S extends Shape> void repartitionLocal(Path inFile,
                                      Path outFile,
                                      OperationsParams params)
                             throws java.io.IOException

Throws:

java.io.IOException

repartitionLocal

@Deprecated
public static <S extends Shape> void repartitionLocal(Path in,
                                                 Path out,
                                                 S stockShape,
                                                 long blockSize,
                                                 CellInfo[] cells,
                                                 java.lang.String sindex,
                                                 boolean overwrite)
                             throws java.io.IOException

Deprecated.

Repartitions a file on local machine without MapReduce jobs.

Parameters:

inFs -

in -

outFs -

out -

cells -

stockShape -

rtree -

overwrite -

Throws:

java.io.IOException - This method is @deprecated. Please use the method repartitionLocal(Path, Path, CellInfo[], OperationsParams)

repartitionLocal

public static <S extends Shape> void repartitionLocal(Path inFile,
                                      Path outFile,
                                      CellInfo[] cells,
                                      OperationsParams params)
                             throws java.io.IOException

Throws:

java.io.IOException

repartition

public static void repartition(Path inFile,
               Path outputPath,
               OperationsParams params)
                        throws java.io.IOException

Parameters:

inFile -

outputPath -

params -

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Entry point to the operation. shape: the shape to use. Automatically inferred from input file if not set. sindex Type of spatial index to build cells-of: Use the cells of the given file for the global index. blocksize: Size of each block in indexed file in bytes. -local: If set, the index is built on the local machine. input filename: Input file in HDFS output filename: Output file in HDFS

Parameters:

args -

Throws:

java.lang.Exception