edu.umn.cs.spatialHadoop.util

Class FileUtil

java.lang.Object

edu.umn.cs.spatialHadoop.util.FileUtil

public final class FileUtil
extends java.lang.Object

A bunch of helper functions used with files

Author:

Ahmed Eldawy

Constructor Summary

Constructors

Constructor and Description
FileUtil()

Method Summary

Methods

Modifier and Type	Method and Description
static java.lang.String	copyFile(Configuration job, FileStatus fileStatus)
static java.lang.String	copyFile(Configuration conf, Path inFile) Copies a file to the local file system given its path.
static java.lang.String	copyFileSplit(Configuration conf, FileSplit split) Copies a part of a file from a remote file system (e.g., HDFS) to a local file.
static CompressionCodec	getCodec(Path file)
static java.lang.String	getExtensionWithoutCompression(Path path) Returns the extension of the file after removing any possible suffixes for compression
static Path[]	getFilesListInPath(Path path) function to list files in a certain directory
static long	getPathSize(FileSystem fs, Path path) Get the actual size of all data in the given directory.
static Path	writePathsToFile(OperationsParams params, Path[] paths) Writes paths to a file where each path is a line.
static Path	writePathsToHDFSFile(OperationsParams params, Path[] paths) Writes paths to a HDFS file where each path is a line.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileUtil

public FileUtil()

Method Detail

copyFile

public static java.lang.String copyFile(Configuration job,
                        FileStatus fileStatus)
                                 throws java.io.IOException

Throws:

java.io.IOException

copyFileSplit

public static java.lang.String copyFileSplit(Configuration conf,
                             FileSplit split)
                                      throws java.io.IOException

Copies a part of a file from a remote file system (e.g., HDFS) to a local file. Returns a path to a local temporary file.

Parameters:

conf -

split -

Returns:

Throws:

java.io.IOException

copyFile

public static java.lang.String copyFile(Configuration conf,
                        Path inFile)
                                 throws java.io.IOException

Copies a file to the local file system given its path.

Parameters:

conf -

inFile -

Returns:

Throws:

java.io.IOException

writePathsToHDFSFile

public static Path writePathsToHDFSFile(OperationsParams params,
                        Path[] paths)

Writes paths to a HDFS file where each path is a line.

Parameters:

paths -

writePathsToFile

public static Path writePathsToFile(OperationsParams params,
                    Path[] paths)

Writes paths to a file where each path is a line.

Parameters:

paths -

getFilesListInPath

public static Path[] getFilesListInPath(Path path)
                                 throws java.io.IOException

function to list files in a certain directory

Parameters:

path -

Returns:

Throws:

java.io.IOException

getPathSize

public static long getPathSize(FileSystem fs,
               Path path)
                        throws java.io.IOException

Get the actual size of all data in the given directory. If the input is a single file, its size is returned immediately. If the input is a directory, we returns the total size of all data in that directory. If there is a global index, the size is retrieved from that global index. Otherwise, we add up all the sizes of single files.

Parameters:

fs - - the file system that contains the path

path - - the path that contains the data

Returns:

Throws:

java.io.IOException

getExtensionWithoutCompression

public static java.lang.String getExtensionWithoutCompression(Path path)

Returns the extension of the file after removing any possible suffixes for compression

Parameters:

path -

Returns:

getCodec

public static CompressionCodec getCodec(Path file)