- better physical file delete:
- both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
- the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
- on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
- when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
- some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
- thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file
- .
- misc:
- fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
- fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
- fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)
- .
- boring image stuff:
- wrote some methods to check for and pull ICC profile bytes from an image with PIL
- wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
- cleaned up some PIL/OpenCV image load and normalisation code
- the decompression bomb check is now quicker for images with rotation
- dequantization is now applied to PIL on all image load by default, it doesn't have to be invoked separately
- some metadata parsing like 'get duration of gif frames' is now faster for images not in RGB or RGBA color
- .
- boring delete code cleanup:
- wrote a heap of new 'is an orphan' filtering logic for client and server
- wrote a daemon job for physical file deletion and plugged it into a new database queue for pending deferred file deletes
- client physical file delete now works off the normal lightweight job scheduler, previously it had its own mainloop thread
- optimised complex file domain file filtering a little
- the 'clear orphan files' job in the client now uses the same updated orphan logic as the new physical delete code. it now won't clear out files in upload limbo
- fixed an issue with re-storing a file in a server after one of its file repositories had previously deleted it. this never mattered previously, when files were never physically deleted, but now the code is brushed up to work properly
- cleaned up some server db code, including the read command method lookup
- moved client 'hash exists?' test down to the master definitions module

next week

Now we have basic ICC support, I want to charge ahead on 'has ICC' and 'pixel dupe' search for the database. I'd also like to find the time to work on some more tag search tech for multiple local file services, but we'll see.