edu.umn.cs.spatialHadoop.core

Class GridRecordWriter<S extends Shape>

java.lang.Object

edu.umn.cs.spatialHadoop.core.GridRecordWriter<S>

Type Parameters:

S - - type of shape written as value.

All Implemented Interfaces:

ShapeRecordWriter<S>

Direct Known Subclasses:

GridRecordWriter, GridRecordWriter2, GridRecordWriter3, RTreeGridRecordWriter

public class GridRecordWriter<S extends Shape>
extends java.lang.Object
implements ShapeRecordWriter<S>

Writes a spatial file where objects are of type S. This class is used as a base class for all spatial record writers. Since a record writer must implement a RecordWriter with specific keys and values, we do all the implementation in this class and extend it with an implementation of RecordWriter with specific key and value. This class is not made abstract because it is still used outside of MapReduce programs.

Author:

Ahmed Eldawy

See Also:

GridRecordWriter, GridRecordWriter2, GridRecordWriter3

Field Summary

Fields

Modifier and Type	Field and Description
protected long	blockSize Block size for grid file written
protected CellInfo[]	cells The spatial boundaries for each cell
protected Rectangle[]	cellsMbr MBR of the records written so far to each cell
protected java.util.ArrayList<java.lang.Thread>	closingThreads A list of threads closing cells in background
static java.util.Set<java.lang.String>	ExpandedIndexes A list of indexes in which each partition has to be expanded to fully contain all the records inside it
protected FileSystem	fileSystem File system for output path
protected Path[]	intermediateCellPath Paths of intermediate files
protected int[]	intermediateCellRecordCount Keeps the number of elements written to each cell so far.
protected int[]	intermediateCellSize Size in bytes of intermediate files written so far
protected java.io.OutputStream[]	intermediateCellStreams An output stream for each grid cell
protected JobConf	jobConf Job configuration if part of a MapReduce job
static Log	LOG
protected java.io.OutputStream	masterFile An output stream to the master file
protected static byte[]	NEW_LINE New line marker to separate records
protected Path	outDir Path of the output directory if not part of a MapReduce job
protected boolean	pack Pack MBR of each cell around its content after it's written to disk
static java.util.Set<java.lang.String>	PackedIndexes A list of indexes the can be optimized by packing each partition to remove empty space
protected java.lang.String	prefix A unique prefix to all files written by this writer
static java.util.Set<java.lang.String>	ReplicatedIndexes Indexes where an object might be replicated to multiple partitions.
protected S	stockObject A stock object used for serialization/deserialization
protected Text	text Temporary text to serialize one object

Constructor Summary

Constructors

Constructor and Description
GridRecordWriter(Path outDir, JobConf job, java.lang.String prefix, CellInfo[] cells) Creates a new GridRecordWriter that will write all data files to the given directory

Method Summary

Methods

Modifier and Type	Method and Description
void	close(Progressable progressable) Close the whole writer.
protected void	closeCell(int cellIndex) Closes (or initiates a close command) for the cell with the given index.
protected void	closeCellBackground(Path intermediateCellPath, Path finalCellPath, java.io.OutputStream intermediateCellStream, java.io.OutputStream masterFile, CellInfo cellMbr, long recordCount, long cellSize) Close the given cell freeing all memory reserved by it.
protected java.io.OutputStream	createFinalCellStream(Path cellFilePath) Creates an output stream that will be used to write the final cell file
protected Path	flushAllEntries(Path intermediateCellPath, java.io.OutputStream intermediateCellStream, Path finalCellPath) Flushes all shapes that were written to one cell to the final file.
protected Path	getFilePath(java.lang.String filename) Returns a path to a file with the given name in the output directory of the record writer.
protected Path	getFinalCellPath(int cellIndex) Returns path to a file in which the final cell will be written.
protected java.io.OutputStream	getIntermediateCellStream(int cellIndex) Returns an output stream in which records are written as they come before they are finally flushed to the cell file.
protected Path	getMasterFilePath()
void	setStockObject(S stockObject) Sets a stock object used to serialize/deserialize objects when written to disk.
void	write(NullWritable dummy, S shape) Writes the given shape to the file to all cells it overlaps with
protected void	writeInternal(int cellIndex, S shape) Write the given shape to the cellIndex indicated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final Log LOG

cells

protected CellInfo[] cells

The spatial boundaries for each cell

intermediateCellPath

protected Path[] intermediateCellPath

Paths of intermediate files

intermediateCellStreams

protected java.io.OutputStream[] intermediateCellStreams

An output stream for each grid cell

cellsMbr

protected Rectangle[] cellsMbr

MBR of the records written so far to each cell

jobConf

protected JobConf jobConf

Job configuration if part of a MapReduce job

outDir

protected Path outDir

Path of the output directory if not part of a MapReduce job

fileSystem

protected final FileSystem fileSystem

File system for output path

text

protected Text text

Temporary text to serialize one object

blockSize

protected long blockSize

Block size for grid file written

stockObject

protected S extends Shape stockObject

A stock object used for serialization/deserialization

masterFile

protected java.io.OutputStream masterFile

An output stream to the master file

closingThreads

protected java.util.ArrayList<java.lang.Thread> closingThreads

A list of threads closing cells in background

intermediateCellRecordCount

protected int[] intermediateCellRecordCount

Keeps the number of elements written to each cell so far. Helps calculating the overhead of RTree indexing

intermediateCellSize

protected int[] intermediateCellSize

Size in bytes of intermediate files written so far

NEW_LINE

protected static byte[] NEW_LINE

New line marker to separate records

prefix

protected java.lang.String prefix

A unique prefix to all files written by this writer

pack

protected boolean pack

Pack MBR of each cell around its content after it's written to disk

PackedIndexes

public static final java.util.Set<java.lang.String> PackedIndexes

A list of indexes the can be optimized by packing each partition to remove empty space

ReplicatedIndexes

public static final java.util.Set<java.lang.String> ReplicatedIndexes

Indexes where an object might be replicated to multiple partitions.

ExpandedIndexes

public static final java.util.Set<java.lang.String> ExpandedIndexes

A list of indexes in which each partition has to be expanded to fully contain all the records inside it

Constructor Detail

GridRecordWriter

public GridRecordWriter(Path outDir,
                JobConf job,
                java.lang.String prefix,
                CellInfo[] cells)
                 throws java.io.IOException

Creates a new GridRecordWriter that will write all data files to the given directory

Parameters:

outDir - - The directory in which all files will be stored

job - - The MapReduce job associated with this output

prefix - - A unique prefix to be associated with files of this writer

cells - - Cells to partition the file

pack - - After writing each cell, pack its MBR around contents

Throws:

java.io.IOException

Method Detail

getMasterFilePath

protected Path getMasterFilePath()
                          throws java.io.IOException

Throws:

java.io.IOException

getFilePath

protected Path getFilePath(java.lang.String filename)
                    throws java.io.IOException

Returns a path to a file with the given name in the output directory of the record writer.

Parameters:

filename -

Returns:

Throws:

java.io.IOException

setStockObject

public void setStockObject(S stockObject)

Description copied from interface: ShapeRecordWriter

Sets a stock object used to serialize/deserialize objects when written to disk.

Specified by:

setStockObject in interface ShapeRecordWriter<S extends Shape>

write

public void write(NullWritable dummy,
         S shape)
           throws java.io.IOException

Description copied from interface: ShapeRecordWriter

Writes the given shape to the file to all cells it overlaps with

Specified by:

write in interface ShapeRecordWriter<S extends Shape>

Throws:

java.io.IOException

writeInternal

protected void writeInternal(int cellIndex,
                 S shape)
                      throws java.io.IOException

Write the given shape to the cellIndex indicated.

Parameters:

cellIndex -

shape -

Throws:

java.io.IOException

getIntermediateCellStream

protected java.io.OutputStream getIntermediateCellStream(int cellIndex)
                                                  throws java.io.IOException

Returns an output stream in which records are written as they come before they are finally flushed to the cell file.

Parameters:

cellIndex -

Returns:

Throws:

java.io.IOException

createFinalCellStream

protected java.io.OutputStream createFinalCellStream(Path cellFilePath)
                                              throws java.io.IOException

Creates an output stream that will be used to write the final cell file

Parameters:

cellFilePath -

Returns:

Throws:

java.io.IOException

closeCell

protected void closeCell(int cellIndex)
                  throws java.io.IOException

Closes (or initiates a close command) for the cell with the given index. Once this method returns, it should be safe to reuse the same cell index to write more data in a new file.

Parameters:

cellIndex -

Throws:

java.io.IOException

closeCellBackground

protected void closeCellBackground(Path intermediateCellPath,
                       Path finalCellPath,
                       java.io.OutputStream intermediateCellStream,
                       java.io.OutputStream masterFile,
                       CellInfo cellMbr,
                       long recordCount,
                       long cellSize)
                            throws java.io.IOException

Close the given cell freeing all memory reserved by it. Once a cell is closed, we should not write more data to it.

Parameters:

cellInfo -

Throws:

java.io.IOException

flushAllEntries

protected Path flushAllEntries(Path intermediateCellPath,
                   java.io.OutputStream intermediateCellStream,
                   Path finalCellPath)
                        throws java.io.IOException

Flushes all shapes that were written to one cell to the final file. It returns a path to a (closed) file that contains all entries written.

Parameters:

cellIndex -

Returns:

Throws:

java.io.IOException

close

public void close(Progressable progressable)
           throws java.io.IOException

Close the whole writer. Finalize all cell files and concatenate them into the output file.

Specified by:

close in interface ShapeRecordWriter<S extends Shape>

Throws:

java.io.IOException

getFinalCellPath

protected Path getFinalCellPath(int cellIndex)
                         throws java.io.IOException

Returns path to a file in which the final cell will be written.

Parameters:

column -

row -

Returns:

Throws:

java.io.IOException