concrete.util.comm_container module

Communication Containers - mapping Communication IDs to Communications

Classes that behave like a read-only dictionary (implementing Python’s collections.Mapping interface) and map Communication ID strings to Communications.

The classes abstract away the storage backend. If you need to optimize for performance, you may not want to use a dictionary abstraction that retrieves one Communication at a time.

class concrete.util.comm_container.DirectoryBackedCommunicationContainer(directory_path, comm_extensions=[u'.comm', u'.concrete', u'.gz'])

Bases: _abcoll.Mapping

Maps Comm IDs to Comms, retrieving Comms from the filesystem

DirectoryBackedCommunicationContainer instances behave as dict-like data structures that map Communication IDs to Communications. Communications are lazily-retrieved from the filesystem.

Upon initialization, a DirectoryBackedCommunicationContainer instance will (recursively) search directory_path for any files that end with the specified comm_extensions. Files with matching extensions are assumed to be Communication files whose filename (sans extension) is the file’s Communication ID. So, for example, a file named ‘XIN_ENG_20101212.0120.concrete’ is assumed to be a Communication file with a Communication ID of ‘XIN_ENG_20101212.0120’.

Files with the extension .gz will be decompressed using gzip.

A DirectoryBackedCommunicationsContainer will not be able to find any files that are added to directory_path after the container was initialized.

Parameters:
  • directory_path (str) – Path to directory containing Communications files
  • comm_extensions (str[]) – List of strings specifying filename extensions to be associated with Communications
class concrete.util.comm_container.FetchBackedCommunicationContainer(host, port)

Bases: _abcoll.Mapping

Maps Comm IDs to Comms, retrieving Comms from a FetchCommunicationService server

FetchBackedCommunicationContainer instances behave as dict-like data structures that map Communication IDs to Communications. Communications are lazily-retrieved from a FetchCommunicationService.

If you need to retrieve large amounts of data from a FetchCommunicationService, then you SHOULD NOT USE THIS CLASS. This class retrieves one Communication at a time using FetchCommunicationService.

Parameters:
class concrete.util.comm_container.MemoryBackedCommunicationContainer(communications_file, max_file_size=1073741824)

Bases: _abcoll.Mapping

Maps Comm IDs to Comms by loading all Comms in file into memory

FetchBackedCommunicationContainer instances behave as dict-like data structures that map Communication IDs to Communications. All Communications in communications_file will be read into memory using a CommunicationReader instance.

Parameters:
  • communications_file (str) – String specifying name of Communications file
  • max_file_size (int) – Maximum file size, in bytes
class concrete.util.comm_container.RedisHashBackedCommunicationContainer(redis_db, key)

Bases: _abcoll.Mapping

Maps Comm IDs to Comms, retrieving Comms from a Redis hash

RedisHashBackedCommunicationContainer instances behave as dict-like data structures that map Communication IDs to Communications. Communications are lazily-retrieved from a Redis hash.

Parameters:
  • redis_db (redis.Redis) – redis database connection
  • key (str) – Key in redis database where hash is located
class concrete.util.comm_container.ZipFileBackedCommunicationContainer(zipfile_path, comm_extensions=[u'.comm', u'.concrete'])

Bases: _abcoll.Mapping

Maps Comm IDs to Comms, retrieving Comms from a Zip file

ZipFileBackedCommunicationContainer instances behave as dict-like data structures that map Communication IDs to Communications. Communications are lazily-retrieved from a Zip file.

Parameters:
  • zipfile_path (str) – Path to Zip file containing Communications
  • comm_extensions (str[]) – List of strings specifying filename extensions associated with Communications