edu.umn.cs.spatialHadoop.mapred

Class SpatialRecordReader<K,V>

java.lang.Object

edu.umn.cs.spatialHadoop.mapred.SpatialRecordReader<K,V>

Direct Known Subclasses:

RTreeRecordReader, ShapeArrayRecordReader, ShapeIterRecordReader, ShapeLineRecordReader, ShapeRecordReader

public abstract class SpatialRecordReader<K,V>
extends java.lang.Object

A base class to read shapes from files. It reads either single shapes, list of shapes, or R-trees. It automatically detects the format of the underlying block and parses it accordingly. The class implement the RecordReader interface allowing it to be used in MapReduce programs with an appropriate InputFormat. The key is always a Rectangle that indicates the MBR of the corresponding partition. In case of a non-indexed file, the key is an invalid rectangle. See Rectangle.isValid().

Author:

Ahmed Eldawy

See Also:

ShapeLineRecordReader, ShapeRecordReader, ShapeArrayRecordReader, RTreeRecordReader

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SpatialRecordReader.ShapeIterator An iterator that iterates over all shapes in the input file

Field Summary

Fields

Modifier and Type	Field and Description
protected long	blockSize Block size for the read file.
protected edu.umn.cs.spatialHadoop.mapred.SpatialRecordReader.BlockType	blockType The type of the currently parsed block
protected byte[]	buffer Some bytes that were read from the stream but not parsed yet
protected Rectangle	cellMbr The boundary of the partition currently being read
protected long	end Last offset to stop at
protected LineReader	lineReader Reads lines from text files
protected long	pos Position of the next byte to read/prase from the input
protected long	start First offset that is read from the input
protected Text	tempLine A temporary text to read lines from lineReader

Constructor Summary

Constructors

Constructor and Description
SpatialRecordReader(CombineFileSplit split, Configuration conf, Reporter reporter, java.lang.Integer index) Initialize from an input split
SpatialRecordReader(Configuration job, FileSplit split) Initialize from a FileSplit
SpatialRecordReader(Configuration job, long s, long l, Path p) Initialize from a path and file range
SpatialRecordReader(java.io.InputStream in, long offset, long endOffset) Construct from an input stream already set to the first byte to read.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
long	getPos() Returns the current position in the data file.
float	getProgress()
protected boolean	initializeReader() Initializes the reader to read from the input stream or file.
protected boolean	nextLine(Text value) Reads the next line from input and return true if a line was read.
protected boolean	nextRTree(RTree<? extends Shape> rtree) Reads the next RTree from file.
protected boolean	nextShape(Shape s) Reads next shape from input and returns true.
protected boolean	nextShapeIter(SpatialRecordReader.ShapeIterator iter) Returns an iterator that iterates over all remaining shapes in the file.
protected boolean	nextShapes(ArrayWritable shapes) Reads all shapes left in the current block in one shot.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

start

protected long start

First offset that is read from the input

end

protected long end

Last offset to stop at

pos

protected long pos

Position of the next byte to read/prase from the input

lineReader

protected LineReader lineReader

Reads lines from text files

tempLine

protected Text tempLine

A temporary text to read lines from lineReader

buffer

protected byte[] buffer

Some bytes that were read from the stream but not parsed yet

blockSize

protected long blockSize

Block size for the read file. Used with RTrees

cellMbr

protected Rectangle cellMbr

The boundary of the partition currently being read

blockType

protected edu.umn.cs.spatialHadoop.mapred.SpatialRecordReader.BlockType blockType

The type of the currently parsed block

Constructor Detail

SpatialRecordReader

public SpatialRecordReader(CombineFileSplit split,
                   Configuration conf,
                   Reporter reporter,
                   java.lang.Integer index)
                    throws java.io.IOException

Initialize from an input split

Parameters:

split -

conf -

reporter -

index -

Throws:

java.io.IOException

SpatialRecordReader

public SpatialRecordReader(Configuration job,
                   FileSplit split)
                    throws java.io.IOException

Initialize from a FileSplit

Parameters:

job -

split -

Throws:

java.io.IOException

SpatialRecordReader

public SpatialRecordReader(Configuration job,
                   long s,
                   long l,
                   Path p)
                    throws java.io.IOException

Initialize from a path and file range

Parameters:

job -

s -

l -

p -

Throws:

java.io.IOException

SpatialRecordReader

public SpatialRecordReader(java.io.InputStream in,
                   long offset,
                   long endOffset)
                    throws java.io.IOException

Construct from an input stream already set to the first byte to read.

Parameters:

in -

offset -

endOffset -

Throws:

java.io.IOException

Method Detail

getPos

public long getPos()
            throws java.io.IOException

Returns the current position in the data file. If the file is not compressed, this is equal to the value returned by getFilePosition(). However, if the file is compressed, this value indicates the position in the decompressed stream.

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Throws:

java.io.IOException

getProgress

public float getProgress()
                  throws java.io.IOException

Throws:

java.io.IOException

initializeReader

protected boolean initializeReader()
                            throws java.io.IOException

Initializes the reader to read from the input stream or file. First, it initializes the MBR of the partition being read if the file is globally indexed. It also detects whether the file is R-tree indexed or not which allows it to skip the R-tree if not needed to be read.

Throws:

java.io.IOException

nextLine

protected boolean nextLine(Text value)
                    throws java.io.IOException

Reads the next line from input and return true if a line was read. If no more lines are available in this split, a false is returned.

Parameters:

value -

Returns:

Throws:

java.io.IOException

nextShape

protected boolean nextShape(Shape s)
                     throws java.io.IOException

Reads next shape from input and returns true. If no more shapes are left in the split, a false is returned. This function first reads a line by calling the method nextLine(Text) then parses the returned line by calling TextSerializable.fromText(Text) on that line. If no stock shape is set, a NullPointerException is thrown.

Parameters:

s -

Returns:

Throws:

java.io.IOException

nextShapes

protected boolean nextShapes(ArrayWritable shapes)
                      throws java.io.IOException

Reads all shapes left in the current block in one shot. This function runs a loop where it keeps reading shapes by calling the method nextShape(Shape) until one of the following conditions happen. 1. The whole file is read. No more records to read. 2. Number of parsed records reaches the threshold defined by the configuration parameter spatialHadoop.mapred.MaxShapesPerRead. To disable this check, set the configuration parameter to -1 3. Total size of parsed data from file reaches the threshold defined by the configuration parameter spatialHadoop.mapred.MaxBytesPerRead. To disable this check, set the configuration parameter to -1.

Parameters:

shapes -

Returns:

Throws:

java.io.IOException

nextShapeIter

protected boolean nextShapeIter(SpatialRecordReader.ShapeIterator iter)
                         throws java.io.IOException

Returns an iterator that iterates over all remaining shapes in the file.

Parameters:

iter -

Returns:

Throws:

java.io.IOException

nextRTree

protected boolean nextRTree(RTree<? extends Shape> rtree)
                     throws java.io.IOException

Reads the next RTree from file. The file must be part of an R-tree index. If the file is not locally indexed using an R-tree, a runtime exception is thrown. If the file is locally indexed using an R-tree, the R-tree is consumed from the file and parsed by calling RTree.readFields(DataInput) on the input stream.

Parameters:

rtree -

Returns:

Throws:

java.io.IOException