I was playing around with putting a filesystem over MongoDB’s GridFS. GridFUSE was a prototype, one Python file, and I was surprised when somebody else used it.
I had tried enough ordinary operations to run my development stack on the mount. The README recorded that, along with informal timings and a fairly direct list of things I had not implemented. It was an experiment in how far the mapping would go. Some operations fit neatly; others exposed what a file means that a stored stream does not.
uWSGI’s new GridFS plugin prompted the experiment. I wanted a mount so I could fill a collection with ordinary file copies and work with it.
The larger idea was to load Python code, possibly even the standard library, through an import hook making loopback calls to uWSGI, with GridFS supplying the files. Switching the collection and reloading might then become a code rollout or recovery operation. I built the mount first; the import and rollout path remained a design.
In hindsight, this was an early version of the same impulse that became Zippy: package Python with the application, make it deliverable as one environment, and push toward the self-contained artifacts that more static languages made easier. GridFUSE tested the remote filesystem edge of that idea. Zippy began later and took a different route.
A directory was a query
The implementation stored a file’s mount-relative path in GridFS’s filename field, after stripping the surrounding slashes. It also stored the parent path as dirname, a visible flag, and a dictionary of stat fields.
Directory listing then became:
for sub in self.fs.files.find({
'dirname': path,
'visible': True,
}).distinct('filename'):
yield pth.basename(sub)There was no separate inode tree to traverse. Directory entries were selected from file metadata. A symlink was another GridFS entry whose mode marked it as a link and whose small body contained the target path.
This made the first operations straightforward. It also made rename a larger promise than the code fulfilled. The implementation updated a matching document’s filename and dirname; it did not walk descendants to rewrite their stored paths when a directory moved.
Overwrite was another file
GridFS did not give this implementation an in-place file overwrite. A new body became a new GridFS file with the same filename. Reads selected the latest version. Deletion changed visible entries to visible: False rather than removing their data.
Old bodies remained in GridFS, without a restore interface or retention policy.
Truncation exposed the mismatch especially clearly. GridFUSE accepted only length zero; any other requested length returned ENOTSUP. An initial unflushed chunk could be rewound in memory. Otherwise the code had to replace the stream and reconnect the handle to a fresh writer.
The write method accepted an offset argument but never used it. It appended to the current GridIn stream. Arbitrary positioned writes were unimplemented.
The placeholder that came back
A newly opened GridIn object was not yet a completed GridFS file. That made create() followed by getattr() awkward: metadata lookup needed something visible before the real writer closed. GridFUSE created a zero-byte entry first, then opened the actual writer from its metadata.
Issue 1 records what happened next. The placeholder was not hidden when the real file closed. Renaming could expose that older entry at the original path, making the rename appear to have failed.
The issue considered tagging the placeholder for cleanup, bypassing part of the GridFS library, or fixing lookup to account for open files in the cache. That last possibility names the larger problem. Some file state existed in the database and some existed in the open-handle table, but operations did not consistently consult the same combined view.
A later Filebench report described bad file descriptors in multithreaded mode and a passing run in single-threaded mode. My reply said I suspected GridFUSE, not the FUSE binding, and had moved on to another approach. The issue remained unresolved.
Where it stopped
The README recorded informal timings for copying my development environment, a large SQL file, and listing the tree. The prototype did not create database indexes yet; those timings described my workload, not a measured concurrency diagnosis.
The other caveats were substantial: permissions were not enforced, hard links and extended attributes were unsupported, and deletion only hid data.