concrete.util.file_io module¶
Code for reading and writing Concrete Communications
-
class
concrete.util.file_io.
CommunicationReader
(filename, add_references=True, filetype=0)¶ Bases:
concrete.util.file_io.ThriftReader
Iterator/generator class for reading one or more Communications from a file
The iterator returns a (Communication, filename) tuple
Supported filetypes are:
- a file with a single Communication
- a file with multiple Communications concatenated together
- a gzipped file with a single Communication
- a gzipped file with multiple Communications concatenated together
- a .tar.gz file with one or more Communications
- a .zip file with one or more Communications
Sample usage:
for (comm, filename) in CommunicationReader('multiple_comms.tar.gz'): do_something(comm)
Parameters: - filename (str) – path of file to read from
- add_references (bool) – If True, calls
concrete.util.references.add_references_to_communication()
on allCommunication
objects read from file - filetype (FileType) – Expected type of file. Default value is FileType.AUTO, where function will try to automatically determine file type.
-
class
concrete.util.file_io.
CommunicationWriter
(filename=None, gzip=False)¶ Bases:
object
Class for writing one or more Communications to a file
Sample usage:
writer = CommunicationWriter('foo.concrete') writer.write(existing_comm_object) writer.close()
Parameters: - filename (str) – if specified, open file at this path during construction (a file can alternatively be opened after construction using the open method)
- gzip (bool) – Flag indicating if file should be compressed with gzip
-
close
()¶ Close file.
-
open
(filename)¶ Open specified file for writing. File will be compressed if the gzip flag of the constructor was set to True.
Parameters: filename (str) – path to file to open for writing
-
write
(comm)¶ Parameters: comm (Communication) – communication to write to file
-
class
concrete.util.file_io.
CommunicationWriterTGZ
(tar_filename=None)¶ Bases:
concrete.util.file_io.CommunicationWriterTar
Class for writing one or more Communications to a .TAR.GZ archive
Sample usage:
writer = CommunicationWriterTGZ('multiple_comms.tgz') writer.write(comm_object_one, 'comm_one.concrete') writer.write(comm_object_two, 'comm_two.concrete') writer.write(comm_object_three, 'comm_three.concrete') writer.close()
-
class
concrete.util.file_io.
CommunicationWriterTar
(tar_filename=None, gzip=False)¶ Bases:
object
Class for writing one or more Communications to a .TAR archive
Sample usage:
writer = CommunicationWriterTar('multiple_comms.tar') writer.write(comm_object_one, 'comm_one.concrete') writer.write(comm_object_two, 'comm_two.concrete') writer.write(comm_object_three, 'comm_three.concrete') writer.close()
Parameters: - tar_filename (str) – if specified, open file at this path during construction (a file can alternatively be opened after construction using the open method)
- gzip (bool) – Flag indicating if .TAR file should be compressed with gzip
-
close
()¶ Close tar file.
-
open
(tar_filename)¶ Open specified tar file for writing. File will be compressed if the gzip flag of the constructor was set to True.
Parameters: tar_filename (str) – path to file to open for writing
-
write
(comm, comm_filename=None)¶ Parameters: - comm (Communication) – communication to write to tar file
- comm_filename (str) – desired filename of communication within tar file (by default the filename will be the communication id appended with a .concrete extension)
-
class
concrete.util.file_io.
ThriftReader
(thrift_type, filename, postprocess=None, filetype=0)¶ Bases:
object
Iterator/generator class for reading one or more Thrift structures from a file
The iterator returns a (obj, filename) tuple where obj is an object of type thrift_type.
Supported filetypes are:
- a file with a single Thrift structure
- a file with multiple Thrift structures concatenated together
- a gzipped file with a single Thrift structure
- a gzipped file with multiple Thrift structures concatenated together
- a .tar.gz file with one or more Thrift structures
- a .zip file with one or more Thrift structures
Sample usage:
for (comm, filename) in ThriftReader(Communication, 'multiple_comms.tar.gz'): do_something(comm)
Parameters: - thrift_type – Class for Thrift type, e.g. Communication, TokenLattice
- filename (str) –
- postprocess (function) – A post-processing function that is called with the Thrift object as argument each time a Thrift object is read from the file
- filetype (FileType) – Expected type of file. Default value is FileType.AUTO, where function will try to automatically determine file type.
Raises: ValueError
– if filetype is not a known filetype name or id-
next
()¶ Return tuple containing next communication (and filename) in the sequence.
Raises: StopIteration
– if there are no more communicationsReturns: tuple containing Communication object and its filename
-
concrete.util.file_io.
read_communication_from_file
(communication_filename, add_references=True)¶ Read a Communication from the file specified by filename
Parameters: - communication_filename (str) – String with filename
- add_references (bool) – If True, calls
concrete.util.references.add_references_to_communication()
onCommunication
read from file
Returns: Communication read from file
Return type:
-
concrete.util.file_io.
read_thrift_from_file
(thrift_obj, filename)¶ Instantiate Thrift object from contents of named file
The Thrift file is assumed to be encoded using TCompactProtocol
WARNING - Thrift deserialization tends to fail silently. For example, the Thrift libraries will not complain if you try to deserialize data from the file /dev/urandom.
Parameters: - thrift_obj – A Thrift object (e.g. a Communication object)
- filename (str) – A filename string
Returns: The Thrift object that was passed in as an argument
-
concrete.util.file_io.
read_tokenlattice_from_file
(tokenlattice_filename)¶ Read a
TokenLattice
from a fileParameters: tokenlattice_filename (str) – Name of file containing serialized TokenLattice
Returns: TokenLattice read from file Return type: TokenLattice
-
concrete.util.file_io.
write_communication_to_file
(communication, communication_filename)¶ Write a
Communication
to a fileParameters: - communication (Communication) – communication to write
- communication_filename (str) – path of file to write to
-
concrete.util.file_io.
write_thrift_to_file
(thrift_obj, filename)¶ Write a Thrift object to a file
Parameters: - thrift_obj – Thrift object to write
- filename (str) – path of file to write to