Pickle objects to a file with an end-of-file index (Python)
Write several pickled Python objects to one file, with an index and a header that points to it, for direct access later.
This pattern writes several pickled objects to one binary file and stores their positions in an index written at the end. A fixed-size header records where the index starts, so a reader can read the index first and then jump straight to any object.
Writing the file
import os, pickle, struct
myfile = open('/path/to/my/file', 'w+b')
myfile.write(struct.pack('L', 0)) # placeholder for the index offset
index = []
for o in objects:
index.append(myfile.tell())
pickle.dump(o, myfile)
index_loc = myfile.tell()
pickle.dump(index, myfile)
myfile.seek(0, os.SEEK_SET)
myfile.write(struct.pack('L', index_loc))
Notes
- The file is opened with
'w+b'(binary, read and write) because the header is rewritten after the objects are written. - The size of the
'L'format depends on the platform (4 bytes on Windows, usually 8 bytes on 64-bit Linux). The reader must use the same format. For a stable layout, use an explicit size such as'<Q'(8 bytes, little-endian). - Only load pickle data from a trusted source:
pickle.loadcan run arbitrary code. - The source used
seek(0, 0, os.SEEK_SET), which is invalid becauseseektakes two arguments. It is written here asseek(0, os.SEEK_SET). objectsstands for any iterable of picklable objects.