edu.umn.cs.spatialHadoop.nasa

Class HTTPFileSystem

java.lang.Object

FileSystem

edu.umn.cs.spatialHadoop.nasa.HTTPFileSystem

public class HTTPFileSystem
extends FileSystem

A FileSystem that is able to open HTTP files. Note that certain features are not supported by the HTTP scheme by design. First, it is read-only which makes all write functionality not supported. Second, it cannot list contents of a directory. For listing, some web browsers provide a default behavior that lists contents of a directory as HTTP links. We try to use this feature as much as possible to simulate the list functionality. The browsing feature in this class is designed specifically to browse LP DAAC NASA archives. Although most of the implementation is generic, there are some features that are hard-coded to work with LP DAAC and might not work with other web sites.

Author:

Ahmed Eldawy

Field Summary

Fields

Modifier and Type	Field and Description
static Log	LOG

Constructor Summary

Constructors

Constructor and Description
HTTPFileSystem()

Method Summary

Methods

Modifier and Type	Method and Description
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress)
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)
boolean	delete(Path f)
boolean	delete(Path f, boolean recursive)
FileStatus	getFileStatus(Path f) Returns the status of a file.
java.net.URI	getUri()
Path	getWorkingDirectory()
void	initialize(java.net.URI uri, Configuration conf)
FileStatus[]	listStatus(Path f) Lists all files and directories in a given Path that points to a directory.
static void	main(java.lang.String[] args)
boolean	mkdirs(Path f, FsPermission permission)
FSDataInputStream	open(Path f, int bufferSize)
boolean	rename(Path src, Path dst)
void	setWorkingDirectory(Path new_dir)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final Log LOG

Constructor Detail

HTTPFileSystem

public HTTPFileSystem()

Method Detail

initialize

public void initialize(java.net.URI uri,
              Configuration conf)
                throws java.io.IOException

Throws:

java.io.IOException

getUri

public java.net.URI getUri()

open

public FSDataInputStream open(Path f,
                     int bufferSize)
                       throws java.io.IOException

Throws:

java.io.IOException

create

public FSDataOutputStream create(Path f,
                        FsPermission permission,
                        boolean overwrite,
                        int bufferSize,
                        short replication,
                        long blockSize,
                        Progressable progress)
                          throws java.io.IOException

Throws:

java.io.IOException

append

public FSDataOutputStream append(Path f,
                        int bufferSize,
                        Progressable progress)
                          throws java.io.IOException

Throws:

java.io.IOException

rename

public boolean rename(Path src,
             Path dst)
               throws java.io.IOException

Throws:

java.io.IOException

delete

public boolean delete(Path f)
               throws java.io.IOException

Throws:

java.io.IOException

delete

public boolean delete(Path f,
             boolean recursive)
               throws java.io.IOException

Throws:

java.io.IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws java.io.IOException

Lists all files and directories in a given Path that points to a directory. While this function is written in a generic way, it was designed and tested only with LP DAAC archives.

Throws:

java.io.IOException

setWorkingDirectory

public void setWorkingDirectory(Path new_dir)

getWorkingDirectory

public Path getWorkingDirectory()

mkdirs

public boolean mkdirs(Path f,
             FsPermission permission)
               throws java.io.IOException

Throws:

java.io.IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws java.io.IOException

Returns the status of a file. This method is designed specifically to work with LP DAAC archive and will not work correctly with other web sites. Since HTTP does not tell whether a URL points to a file or directory, we assume that URLs ending with HDF, XML and JPG are files while anything else is considered a directory.

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Throws:

java.io.IOException