Virtual File System (pyobs.vfs)

A virtual file system (VFS) is a convenient way for allowing access to different resources with a common interface. It also makes it easy to change the actual place of storage without any changes to the code. In pyobs, a path into the VFS looks like a normal path on a Unix-like system:

/path/to/file.ext

To be more precise, a path is built like this:

/<root>/<path>/<filename>

where <root> indicates, which VFS to use, <path> is the path within that VFS and <filename> is the actual filename.

The available roots in the pyobs VFS are usually defined in the configuration file like this:

vfs:
    root1:
        class: VfsClass1
    root2:
        class: VfsClass2

With this configuration, all filenames that start with /root1/ (e.g. /root1/path/filename.ext) are handled by VfsClass1, while all filenames starting with /root2/ use VfsClass2. Both defined classes can have additional parameters, which should also be given in the configuration.

With this example, one can easily see the advantages of using a VFS:

  1. File access to all roots look similar, we always open a filename like /root/path/file.ext.

  2. Changing the handling class for one of the roots changes the way we access a file. For instance, if a file was always stored locally, but now we want to change that to a location using a SSH location, this can easily be accomplished by simply changing the configuration.

  3. Another advantage that we have not mentioned before is, that the same roots can use different handling classes on different machines. See below for details.

Imagine a file /storage/file.ext that maps to a local file on machine A, i.e. storage uses a handling class that simply changes the filename to a local filename. On machine B we can now define the same root storage, but use a different handling class that, e.g., accesses the file via SSH. Still, the filename would be the same on both machines. So A could store a file as /storage/file.ext, send the filename to B, which then can access the file via the same filename.

Currently supported are these types of file access:
  • LocalFile: Local file on the machine the module is running on.

  • HttpFile: File on a HTTP server.

  • MemoryFile: File in memory.

  • SSHFile: File on different machine that is accessible via SSH.

  • TarFile: Wrapper for a dynamically created TAR file. Can only be read from.

  • TempFile: Temporary file that will be deleted after being closed.

  • ArchiveFile: Wrapper for a file in the pyobs-archive image archive.

The base class for all of these classes is VFSFile.

VirtualFileSystem

class VirtualFileSystem(roots: Optional[Dict[str, Any]] = None, **kwargs: Any)

Base for a virtual file system.

Create a new VFS.

Parameters:

roots – Dictionary containing roots, see vfs for examples.

File Access Classes

ArchiveFile

class ArchiveFile(name: str, url: str, mode: str = 'w', token: Optional[str] = None)

Wraps a file in an archive. To be used in combination with pyobs-archive.

Creates a new archive file.

Parameters:
  • name – Name of file.

  • mode – Open mode (r/w).

  • url – Archive url url.

  • token – Authorization token.

HttpFile

class HttpFile(name: str, mode: str = 'r', download: Optional[str] = None, upload: Optional[str] = None, username: Optional[str] = None, password: Optional[str] = None, verify_tls: bool = False, timeout: int = 30, **kwargs: Any)

Wraps a file on a HTTP server that can be accessed via GET/POST. Especially useful in combination with HttpFileCache.

Creates a new HTTP file.

Parameters:
  • name – Name of file.

  • mode – Open mode (r/w).

  • download – Base URL for downloading files. If None, no read access possible.

  • upload – Base URL for uploading files. If None, no write access possible.

  • username – Username for accessing the HTTP server.

  • password – Password for accessing the HTTP server.

  • verify_tls – Whether to verify TLS certificates.

  • timeout – Timeout in seconds for uploading/downloading files.

LocalFile

class LocalFile(name: str, mode: str = 'r', root: Optional[str] = None, mkdir: bool = True, **kwargs: Any)

Wraps a local file with the virtual file system.

Open a local file.

Parameters:
  • name – Name of file.

  • mode – Open mode.

  • root – Root to prefix name with for absolute path in filesystem.

  • mkdir – Whether or not to create non-existing paths automatically.

MemoryFile

class MemoryFile(name: str, mode: str = 'r', **kwargs: Any)

A file stored in memory.

Open/create a file in memory.

Parameters:
  • name – Name of file.

  • mode – Open mode.

SSHFile

class SSHFile(name: str, mode: str = 'r', bufsize: int = - 1, hostname: Optional[str] = None, port: int = 22, username: Optional[str] = None, password: Optional[str] = None, keyfile: Optional[str] = None, root: Optional[str] = None, mkdir: bool = True, **kwargs: Any)

VFS wrapper for a file that can be accessed over a SFTP connection.

Open/create a file over a SSH connection.

Parameters:
  • name – Name of file.

  • mode – Open mode.

  • bufsize – Size of buffer size for SFTP connection.

  • hostname – Name of host to connect to.

  • port – Port on host to connect to.

  • username – Username to log in on host.

  • password – Password for username.

  • keyfile – Path to SSH key on local machine.

  • root – Root directory on host.

  • mkdir – Whether or not to automatically create directories.

TempFile

class TempFile(name: Optional[str] = None, mode: str = 'r', prefix: Optional[str] = None, suffix: Optional[str] = None, root: str = '/tmp/pyobs/', mkdir: bool = True, **kwargs: Any)

A temporary file.

Open/create a temp file.

Parameters:
  • name – Name of file.

  • mode – Open mode.

  • prefix – Prefix for automatic filename creation in write mode.

  • suffix – Suffix for automatic filename creation in write mode.

  • root – Temp directory.

  • mkdir – Whether to automatically create directories.

VFSFile

class VFSFile

Base class for all VFS file classes.