edu.umn.cs.spatialHadoop.operations

Class DistributedJoin

java.lang.Object

edu.umn.cs.spatialHadoop.operations.DistributedJoin

public class DistributedJoin
extends java.lang.Object

Performs a spatial join between two or more files using the redistribute-join algorithm.

Author:

Ahmed Eldawy

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DistributedJoin.DJInputFormatArray Input format that returns a record reader that reads a pair of arrays of shapes
static class	DistributedJoin.DJInputFormatRTree<S extends Shape> Input format that returns a record reader that reads a pair of arrays of shapes
static class	DistributedJoin.RedistributeJoinMap
static class	DistributedJoin.RedistributeJoinMapNoDupAvoidance
static class	DistributedJoin.RepartitionJoinReduce<T extends Shape>
static class	DistributedJoin.SpatialJoinFilter

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	isFilterOnly
static boolean	isGeneralRepartitionMode
static boolean	isOneShotReadMode
static boolean	isReduceInactive
static int	joiningThresholdPerOnce
static RunningJob	lastRunningJob
static int	maxBytesInOneRead
static int	maxShapesInOneRead

Constructor Summary

Constructors

Constructor and Description
DistributedJoin()

Method Summary

Methods

Modifier and Type	Method and Description
static long	distributedJoinSmart(Path[] inputFiles, Path userOutputPath, OperationsParams params) Spatially joins two files.
static <S extends Shape> long	joinStep(Path[] inFiles, Path userOutputPath, OperationsParams params) Performs a redistribute join between the given files using the redistribute join algorithm.
static void	main(java.lang.String[] args)
protected static long	repartitionJoinStep(Path[] inputFiles, int fileToRepartition, Path outputFile, OperationsParams params) Spatially joins two datasets by repartitioning the smaller dataset based on the larger one, then apply one-to-one joining for each partition
protected static void	repartitionStep(Path[] files, int file_to_repartition, OperationsParams params) Repartition a file to match the partitioning of the other file.
protected static int	selectRepartition(Path[] files, OperationsParams params)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

lastRunningJob

public static RunningJob lastRunningJob

maxBytesInOneRead

public static int maxBytesInOneRead

maxShapesInOneRead

public static int maxShapesInOneRead

isOneShotReadMode

public static boolean isOneShotReadMode

isGeneralRepartitionMode

public static boolean isGeneralRepartitionMode

isReduceInactive

public static boolean isReduceInactive

isFilterOnly

public static boolean isFilterOnly

joiningThresholdPerOnce

public static int joiningThresholdPerOnce

Constructor Detail

DistributedJoin

public DistributedJoin()

Method Detail

selectRepartition

protected static int selectRepartition(Path[] files,
                    OperationsParams params)
                                throws java.io.IOException

Returns:

the index in the given array of the file to be repartitioned. -1 if all files are non-indexed

Throws:

java.io.IOException

repartitionStep

protected static void repartitionStep(Path[] files,
                   int file_to_repartition,
                   OperationsParams params)
                               throws java.io.IOException,
                                      java.lang.InterruptedException

Repartition a file to match the partitioning of the other file.

Parameters:

fs -

files -

stockShape -

fStatus -

gIndexes -

Throws:

java.io.IOException

java.lang.InterruptedException

joinStep

public static <S extends Shape> long joinStep(Path[] inFiles,
                              Path userOutputPath,
                              OperationsParams params)
                     throws java.io.IOException

Performs a redistribute join between the given files using the redistribute join algorithm. Currently, we only support a pair of files.

Parameters:

fs -

inFiles -

output -

Returns:

Throws:

java.io.IOException

repartitionJoinStep

protected static long repartitionJoinStep(Path[] inputFiles,
                       int fileToRepartition,
                       Path outputFile,
                       OperationsParams params)
                                   throws java.io.IOException

Spatially joins two datasets by repartitioning the smaller dataset based on the larger one, then apply one-to-one joining for each partition

Parameters:

inputFiles - Input datasets to be spatially joined

fileToRepartition - Index of which file will be repartitioned

outputFile - Output file contains the joining results

params - Job configurations

Returns:

Throws:

java.io.IOException

distributedJoinSmart

public static long distributedJoinSmart(Path[] inputFiles,
                        Path userOutputPath,
                        OperationsParams params)
                                 throws java.io.IOException,
                                        java.lang.InterruptedException

Spatially joins two files.

Parameters:

fs -

inputFiles -

stockShape -

output -

Returns:

Throws:

java.io.IOException

java.lang.InterruptedException

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException,
                        java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException