edu.umn.cs.spatialHadoop.core

Class SpatialSite

java.lang.Object

edu.umn.cs.spatialHadoop.core.SpatialSite

public class SpatialSite
extends java.lang.Object

Combines all the configuration needed for SpatialHadoop.

Author:

Ahmed Eldawy

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	FilterClass The class used to filter blocks before starting map tasks
static java.lang.String	INDEXING_OVERHEAD Configuration line name for replication overhead
static java.lang.String	MaxBytesInOneRead Maximum size in bytes that can be read in one read operation
static java.lang.String	MaxShapesInOneRead Maximum number of shapes to read in one read operation and return when reading a file as array
static PathFilter	NonHiddenFileFilter A filter that selects visible files and filters out hidden files.
static java.lang.String	OUTPUT_CELLS
static java.lang.String	OVERWRITE
static java.lang.String	RTREE_BUILD_MODE Whether to build the RTree in fast mode or slow (memory saving) mode.
static long	RTreeFileMarker A marker put in the beginning of each block to indicate that this block is stored as an RTree.
static byte[]	RTreeFileMarkerB
static java.lang.String	SAMPLE_RATIO Ratio of the sample to read from files to build a global R-tree
static java.lang.String	SAMPLE_SIZE Ratio of the sample to read from files to build a global R-tree

Method Summary

Methods

Modifier and Type	Method and Description
static void	addClassToPath(Configuration conf, java.lang.Class<?> klass) Ensures that the given class is in the class path of running jobs.
static CellInfo[]	cellsOf(FileSystem fs, Path path) Returns the cells (partitions) of a file.
static CellInfo[]	cellsOf(GlobalIndex<Partition> gIndex) Finds the partitioning info used in the given global index.
static Shape	createStockShape(Configuration job) Creates a stock shape according to the given configuration.
static CellInfo	getCellInfo(GlobalIndex<Partition> gIndex, int cellID)
static CellInfo[]	getCells(Configuration conf) Retrieves cells that were stored earlier using #setCells(JobConf, CellInfo[]).
static RandomShapeGenerator.DistributionType	getDistributionType(Configuration job, java.lang.String key, RandomShapeGenerator.DistributionType defaultValue) Retrieves the distribution type used for generating synthetic data
static GlobalIndex<Partition>	getGlobalIndex(FileSystem fs, Path dir) Returns the global index (partitions) of a file that is indexed using the index command.
static Rectangle	getRectangle(Configuration conf, java.lang.String name) Retrieves a rectangle from configuration parameter.
static Shape	getShape(Configuration conf, java.lang.String param) Deprecated. - Use OperationsParams.getShape(Configuration, String)
static boolean	isRTree(FileSystem fs, Path path) Checks whether a file is indexed using an R-tree or not.
static <S extends Shape> RTree<S>	loadRTree(FileSystem fs, Path file, S shape)
static void	setCells(Configuration conf, CellInfo[] cellsInfo) Set an array of cells in the job configuration.
static void	setClass(Configuration conf, java.lang.String key, java.lang.Class<?> klass, java.lang.Class<?> xface) Similar to Configuration#setClass(String, Class, Class), this method sets the given key to the provided class in the configuration.
static void	setRectangle(Configuration conf, java.lang.String name, Rectangle rect) Sets a rectangle in a job configuration.
static void	setShape(Configuration conf, java.lang.String param, Shape shape) Deprecated. Use OperationsParams.setShape(Configuration,String,Shape) instead

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NonHiddenFileFilter

public static final PathFilter NonHiddenFileFilter

A filter that selects visible files and filters out hidden files. Hidden files are the ones with a names starting in '.' or '_'

FilterClass

public static final java.lang.String FilterClass

The class used to filter blocks before starting map tasks

See Also:

Constant Field Values

RTREE_BUILD_MODE

public static final java.lang.String RTREE_BUILD_MODE

Whether to build the RTree in fast mode or slow (memory saving) mode.

See Also:

Constant Field Values

INDEXING_OVERHEAD

public static final java.lang.String INDEXING_OVERHEAD

Configuration line name for replication overhead

See Also:

Constant Field Values

SAMPLE_RATIO

public static final java.lang.String SAMPLE_RATIO

Ratio of the sample to read from files to build a global R-tree

See Also:

Constant Field Values

SAMPLE_SIZE

public static final java.lang.String SAMPLE_SIZE

Ratio of the sample to read from files to build a global R-tree

See Also:

Constant Field Values

RTreeFileMarker

public static final long RTreeFileMarker

A marker put in the beginning of each block to indicate that this block is stored as an RTree. It might be better to store this in the BlockInfo in a field (e.g. localIndexType).

See Also:

Constant Field Values

OUTPUT_CELLS

public static final java.lang.String OUTPUT_CELLS

See Also:

Constant Field Values

OVERWRITE

public static final java.lang.String OVERWRITE

See Also:

Constant Field Values

MaxShapesInOneRead

public static final java.lang.String MaxShapesInOneRead

Maximum number of shapes to read in one read operation and return when reading a file as array

See Also:

Constant Field Values

MaxBytesInOneRead

public static final java.lang.String MaxBytesInOneRead

Maximum size in bytes that can be read in one read operation

See Also:

Constant Field Values

RTreeFileMarkerB

public static byte[] RTreeFileMarkerB

Method Detail

setClass

public static void setClass(Configuration conf,
            java.lang.String key,
            java.lang.Class<?> klass,
            java.lang.Class<?> xface)

Similar to Configuration#setClass(String, Class, Class), this method sets the given key to the provided class in the configuration. In addition, it ensures that the jar file that contains that class is in the class path of the provided configuration. This is ensured by copying the jar file and adding it to the distributed cache using the method DistributedCache#addArchiveToClassPath(Path, Configuration, Path).

Parameters:

conf - - Configuration to set the key

key - - the key to set

klass - - the class to use as a value

xface - - the interface that the provided class should implement

addClassToPath

public static void addClassToPath(Configuration conf,
                  java.lang.Class<?> klass)

Ensures that the given class is in the class path of running jobs. If the jar is not already in the class path, it is added to the DisributedCache of the given job to ensure the associated job will work fine.

Parameters:

conf -

klass -

createStockShape

public static Shape createStockShape(Configuration job)

Creates a stock shape according to the given configuration. It retrieves the shape class using #getShapeClass(Configuration) then creates an instance of this shape using Class.newInstance(). This requires the shape class to have a default empty constructor.

Parameters:

job -

Returns:

setShape

public static void setShape(Configuration conf,
            java.lang.String param,
            Shape shape)

Deprecated. Use OperationsParams.setShape(Configuration,String,Shape) instead

Sets the specified configuration parameter to the current value of the shape. Both class name and shape values are encoded in one string and set as the value of the configuration parameter. The shape can be retrieved later using getShape(Configuration, String).

Parameters:

conf -

param -

shape -

getShape

@Deprecated
public static Shape getShape(Configuration conf,
                        java.lang.String param)

Deprecated. - Use OperationsParams.getShape(Configuration, String)

Retrieves a value of a shape set earlier using OperationsParams.setShape(Configuration, String, Shape). It reads the corresponding parameter and parses it to find the class name and shape value. First, a default object is created using Class.newInstance() then the value is parsed using TextSerializable.fromText(Text).

Parameters:

conf -

param -

Returns:

getGlobalIndex

public static GlobalIndex<Partition> getGlobalIndex(FileSystem fs,
                                    Path dir)

Returns the global index (partitions) of a file that is indexed using the index command. If the file is not indexed, it returns null. The return value is of type GlobalIndex where the generic parameter is specified as Partition.

Parameters:

fs -

dir -

Returns:

Throws:

java.io.IOException

isRTree

public static boolean isRTree(FileSystem fs,
              Path path)
                       throws java.io.IOException

Checks whether a file is indexed using an R-tree or not. This allows an operation to use the R-tree to speedup the processing if it exists. This function opens the specified file and reads the first eight bytes which include the R-tree signature. If the signatures matches with the R-tree signature, true is returned. Otherwise, false is returned. If the parameter is a path to a directory, only the first data file in that directory is tested.

Parameters:

fs -

path -

Returns:

Throws:

java.io.IOException

cellsOf

public static CellInfo[] cellsOf(FileSystem fs,
                 Path path)
                          throws java.io.IOException

Returns the cells (partitions) of a file. This functionality can be useful to repartition another file using the same partitioning or to draw these partitions as a high level index. This function reads the master file and returns all rectangles in it.

Parameters:

fs -

path -

Returns:

Throws:

java.io.IOException

setCells

public static void setCells(Configuration conf,
            CellInfo[] cellsInfo)
                     throws java.io.IOException

Set an array of cells in the job configuration. As the array might be very large to store as one value, an alternative approach is used. The cells are all written to a temporary file, and that file is added to the DistributedCache of the job. Later on, a call to #getCells(JobConf) will open the corresponding file from DistributedCache and parse cells from that file.

Parameters:

conf -

cellsInfo -

Throws:

java.io.IOException

getCells

public static CellInfo[] getCells(Configuration conf)
                           throws java.io.IOException

Retrieves cells that were stored earlier using #setCells(JobConf, CellInfo[]). This function opens the corresponding file from DistributedCache and parses jobs from it.

Parameters:

conf -

Returns:

Throws:

java.io.IOException

setRectangle

public static void setRectangle(Configuration conf,
                java.lang.String name,
                Rectangle rect)

Sets a rectangle in a job configuration. The Rectangle is serialized to text using Rectangle.toText(Text).

Parameters:

conf -

name -

rect -

getRectangle

public static Rectangle getRectangle(Configuration conf,
                     java.lang.String name)

Retrieves a rectangle from configuration parameter. The value is assumed to be in text format that can be parsed using Rectangle.fromText(Text)

Parameters:

conf -

name -

Returns:

getDistributionType

public static RandomShapeGenerator.DistributionType getDistributionType(Configuration job,
                                                        java.lang.String key,
                                                        RandomShapeGenerator.DistributionType defaultValue)

Retrieves the distribution type used for generating synthetic data

Parameters:

job -

key -

defaultValue -

Returns:

cellsOf

public static CellInfo[] cellsOf(GlobalIndex<Partition> gIndex)

Finds the partitioning info used in the given global index. If each cell is represented as one partition, the MBRs of these partitions are returned. If one cell is stored in multiple partitions (i.e., multiple files), their MBRs are combined to produce one MBR for this cell.

Parameters:

gIndex -

Returns:

loadRTree

public static <S extends Shape> RTree<S> loadRTree(FileSystem fs,
                                   Path file,
                                   S shape)
                                        throws java.io.IOException

Throws:

java.io.IOException

getCellInfo

public static CellInfo getCellInfo(GlobalIndex<Partition> gIndex,
                   int cellID)