concrete.util.file_io module¶

Code for reading and writing Concrete Communications

class concrete.util.file_io.CommunicationReader(filename, add_references=True, filetype=0)¶

Bases: concrete.util.file_io.ThriftReader

Iterator/generator class for reading one or more Communications from a file

The iterator returns a (Communication, filename) tuple

Supported filetypes are:

a file with a single Communication
a file with multiple Communications concatenated together
a gzipped file with a single Communication
a gzipped file with multiple Communications concatenated together
a .tar.gz file with one or more Communications
a .zip file with one or more Communications

Sample usage:

for (comm, filename) in CommunicationReader('multiple_comms.tar.gz'):
    do_something(comm)

Parameters:	filename (str) – path of file to read from add_references (bool) – If True, calls `concrete.util.references.add_references_to_communication()` on all `Communication` objects read from file filetype (FileType) – Expected type of file. Default value is FileType.AUTO, where function will try to automatically determine file type.

class concrete.util.file_io.CommunicationWriter(filename=None, gzip=False)¶

Bases: object

Class for writing one or more Communications to a file

Sample usage:

with CommunicationWriter('foo.concrete') as writer:
    writer.write(existing_comm_object)

Parameters:	filename (str) – if specified, open file at this path during construction (a file can alternatively be opened after construction using the open method) gzip (bool) – Flag indicating if file should be compressed with gzip

close()¶: Close file.

open(filename)¶

Open specified file for writing. File will be compressed if the gzip flag of the constructor was set to True.

Parameters:	filename (str) – path to file to open for writing

write(comm)¶

Parameters:	comm (Communication) – communication to write to file

class concrete.util.file_io.CommunicationWriterTGZ(tar_filename=None)¶

Bases: concrete.util.file_io.CommunicationWriterTar

Class for writing one or more Communications to a .tar.gz (.tgz) archive

Sample usage:

with CommunicationWriterTGZ('multiple_comms.tar.gz') as writer:
    writer.write(comm_object_one, 'comm_one.concrete')
    writer.write(comm_object_two, 'comm_two.concrete')
    writer.write(comm_object_three, 'comm_three.concrete')

Parameters:	tar_filename (str) – if specified, open file at this path during construction (a file can alternatively be opened after construction using the open method)

class concrete.util.file_io.CommunicationWriterTar(tar_filename=None, gzip=False)¶

Bases: object

Class for writing one or more Communications to a .tar archive

Sample usage:

with CommunicationWriterTar('multiple_comms.tar') as writer:
    writer.write(comm_object_one, 'comm_one.concrete')
    writer.write(comm_object_two, 'comm_two.concrete')
    writer.write(comm_object_three, 'comm_three.concrete')

Initialize

Parameters:	tar_filename (str) – if specified, open file at this path during construction (a file can alternatively be opened after construction using the open method) gzip (bool) – Flag indicating if .tar file should be compressed with gzip

close()¶: Close tar file.

open(tar_filename)¶

Open specified tar file for writing. File will be compressed if the gzip flag of the constructor was set to True.

Parameters:	tar_filename (str) – path to file to open for writing

write(comm, comm_filename=None)¶

Parameters:	comm (Communication) – communication to write to tar file comm_filename (str) – desired filename of communication within tar file (by default the filename will be the communication id appended with a .concrete extension)

class concrete.util.file_io.CommunicationWriterZip(zip_filename=None)¶

Bases: object

Class for writing one or more Communications to a .zip archive

Sample usage:

with CommunicationWriterZip('multiple_comms.zip') as writer:
    writer.write(comm_object_one, 'comm_one.concrete')
    writer.write(comm_object_two, 'comm_two.concrete')
    writer.write(comm_object_three, 'comm_three.concrete')

Parameters:	zip_filename (str) – if specified, open file at this path during construction (a file can alternatively be opened after construction using the open method)

close()¶: Close zip file.

open(zip_filename=None)¶

Open specified zip file for writing.

Parameters:	zip_filename (str) – path to file to open for writing

write(comm, comm_filename=None)¶

Write communication to zip file.

Parameters:	comm (Communication) – communication to write to zip file comm_filename (str) – desired filename of communication within zip file (by default the filename will be the communication id appended with a .concrete extension)

class concrete.util.file_io.ThriftReader(thrift_type, filename, postprocess=None, filetype=0)¶

Bases: object

Iterator/generator class for reading one or more Thrift structures from a file

The iterator returns a (obj, filename) tuple where obj is an object of type thrift_type.

Supported filetypes are:

a file with a single Thrift structure
a file with multiple Thrift structures concatenated together
a gzipped file with a single Thrift structure
a gzipped file with multiple Thrift structures concatenated together
a .tar.gz file with one or more Thrift structures
a .zip file with one or more Thrift structures

Sample usage:

for (comm, filename) in ThriftReader(Communication,
                                     'multiple_comms.tar.gz'):
    do_something(comm)

Parameters:

thrift_type – Class for Thrift type, e.g. Communication, TokenLattice
filename (str) –
postprocess (function) – A post-processing function that is called with the Thrift object as argument each time a Thrift object is read from the file
filetype (FileType) – Expected type of file. Default value is FileType.AUTO, where function will try to automatically determine file type.

Raises:

ValueError – if filetype is not a known filetype name or id

next()¶

Return tuple containing next communication (and filename) in the sequence.

Raises:	`EOFError` – unexpected EOF, probably caused by deserializing an invalid Thrift object `StopIteration` – if there are no more communications
Returns:	tuple containing Communication object and its filename

concrete.util.file_io.read_communication_from_file(communication_filename, add_references=True)¶

Read a Communication from the file specified by filename

Parameters:	communication_filename (str) – String with filename add_references (bool) – If True, calls `concrete.util.references.add_references_to_communication()` on `Communication` read from file
Returns:	Communication read from file
Return type:	Communication

concrete.util.file_io.read_thrift_from_file(thrift_obj, filename)¶

Instantiate Thrift object from contents of named file

The Thrift file is assumed to be encoded using TCompactProtocol

WARNING - Thrift deserialization tends to fail silently. For example, the Thrift libraries will not complain if you try to deserialize data from the file /dev/urandom.

Parameters:	thrift_obj – A Thrift object (e.g. a Communication object) filename (str) – A filename string
Returns:	The Thrift object that was passed in as an argument

concrete.util.file_io.read_tokenlattice_from_file(tokenlattice_filename)¶

Read a TokenLattice from a file

Parameters:	tokenlattice_filename (str) – Name of file containing serialized `TokenLattice`
Returns:	TokenLattice read from file
Return type:	TokenLattice

concrete.util.file_io.write_communication_to_file(communication, communication_filename)¶

Write a Communication to a file

Parameters:	communication (Communication) – communication to write communication_filename (str) – path of file to write to

concrete.util.file_io.write_thrift_to_file(thrift_obj, filename)¶

Write a Thrift object to a file

Parameters:	thrift_obj – Thrift object to write filename (str) – path of file to write to