LLI Wiki — My Digital Knowledge Base

Pickle objects to a file with an end-of-file index (Python)

Write several pickled Python objects to one file, with an index and a header that points to it, for direct access later.

This pattern writes several pickled objects to one binary file and stores their positions in an index written at the end. A fixed-size header records where the index starts, so a reader can read the index first and then jump straight to any object.

Writing the file

import os, pickle, struct
myfile = open('/path/to/my/file', 'w+b')
myfile.write(struct.pack('L', 0))  # placeholder for the index offset
index = []
for o in objects:
    index.append(myfile.tell())
    pickle.dump(o, myfile)
index_loc = myfile.tell()
pickle.dump(index, myfile)
myfile.seek(0, os.SEEK_SET)
myfile.write(struct.pack('L', index_loc))

Notes

Mastodon LLIMastodon SocialFosstodon