/hydrus/ - Hydrus Network

 >>/1227/
meant for  >>/1221/

 >>/1227/
Great, thanks for letting me know. For sound on your video, it may be that you have mpv now but because you once did not, hydrus reverted you back to the old renderer in the options.

Please hit help->about and see if you have mpv (it will give you a popup error if you don't). If you do, then hit up file->options->media, and for video and animation, change the 'media show action', if it is set to 'show with native hydrus viewer' to 'show using mpv'.

If you don't have mpv in help->about, let me know.

 >>/1229/
The about says "mpv: : is not currently available on macOS" and I didn't have the option to switch to mpv in the media options. I then brew installed mpv and it gives the same message in the about, but now it lets me choose mpv in the media options. However, when I choose it and try to play a video file the program hangs. Also thanks for the help with updating, I was worried my db was boned

 >>/1230/
Damn, I had forgotten that was true of the App build. I know some macOS guys who have mpv working, but they are running from source.

I'm changing tomorrow's build so it doesn't say that anymore if mpv is actually detected and loaded. Can you hit help->about in 474 sometime and let me know what 'api version' it says you have? Getting the correct versions so python side and mpv side can talk happily to each other has been tricky on Linux, I'll bet it is the same for macOS. The Windows build currently uses mpv 1.109.

I had a great week working on a whole bunch of little fixes and improvements.

The release should be as normal tomorrow.

 >>/1219/
I've narrowed down the error to the update from 438 to 439

 >>/1218/
Thank you, I am glad you like it!

 >>/1219/
 >>/1220/
Thank you for this report. I am sorry for the trouble. I know that around that time (441), we moved to builds on Github. This was because my old macOS laptop (I think it was 10.13 High Sierra) was so old the things it made were incompatible with the new Big Sur. We now build on a Big Sur machine on the cloud. 439 isn't exactly when that happened, but I know I started work on newer version requirements for libraries around that time, so I am afraid I think this might just be a version incompatibility problem. Searching "LSOpenURLsWithRole 10810" shows a variety of similar versioning problems.

However there are also some things like this: https://www.mathworks.com/matlabcentral/answers/460920-why-do-i-get-the-error-lsopenurlswithrole-failed-with-error-10810-when-trying-to-install-matlab

Maybe this is related to security? Is your macOS set up with default security rules. I know you normally have to enable 'Allow apps downloaded from anywhere' as the Readme in the App dmg says.

I am not a macOS expert however. I thought that Mojave (10.14?) was compatible with Big-Sur-built Apps. I may be wrong, or it may just be more complicated than I understand. Do you happen to have a special version of python installed on your machine? Some of the searching I did on that error suggested a conflicting python install might cause it somehow.

I assume if you just run the App by double-clicking its icon (which would normally make a database in ~/Library I think), that doesn't work either?

If it seems like this is just a versioning problem--that Mojave (or your Mojave) is too old to run the Big Sur build, I think your best shot is to see about updating your machine to a later macOS. When we move to Qt 6 over this year, I am afraid I will have to say the same about Windows 7. Unfortunately, macOS seems to retire older OSes faster than Windows. Another potential answer here, if you have some python experience, although it sucks, is to run from source. It may be possible to keep using an older version of Qt, I imagine around 5.12 (check your help->about in v438 to see what Qt seems ok for you), if you are willing to go through the work of setting up a virtual environment and testing out different library versions. Here is my page on that, if you are interested: https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

I think it would probably be easier to see about updating your OS though. And I am not convinced this is a versioning thing, I think it might be security related, maybe in combination with the new Big Sur executable.

Let me know how you get on.

I had a mixed week--unfortunately, some IRL stuff reduced my work time. I did however clean some code, fix a couple bugs, and integrate a cool new quick-navigation widget that a user wrote.

The release should be as normal tomorrow.

Screenshot_4361 png
(2.97 KB, 383x64)

How do I give it the cookie?
Why does it need the cookie to log in when I gave it the login info already?
v464, if that matters

 >>/1223/
Hmm, this error basically means that as it tried to log in, the site did not give a cookie that the login script expected to be given. This suggests the site has changed its login process since that script was written, or maybe an unusual error like your login password was wrong or something or the login page now has a captcha, so the login just failed.

For all complicated login situations, I recommend Hydrus Companion now. It lets you copy your cookies right from your logged-in browser to Hydrus using the Client API:

https://gitgud.io/prkc/hydrus-companion

full list

- times:
- if you have file viewing stats turned on (by default it is), the client will now track the 'last viewed time' of your files, both in preview and media viewers. a record is only made assuming they pass the viewtime checks under _options->file viewing statistics_ (so if you scroll through really quick but have it set to only record after five seconds of viewing, it will not save that as the last viewed time). this last viewed time is shown on the right-click menu with the normal file viewing statistics
- sorting by 'import time' and 'modified time' are moved to a new 'time' subgroup in the sort button menu
- also added to 'time' is 'last viewed time'. note that this has not been tracked until now, so you will have to look at a bunch of things for a few seconds each to get some data to sort with
- to go with 'x time' pattern, 'time imported' is renamed to 'import time' across the program. both should work for system predicate parsing
- system:'import time' and 'modified time' are now bundled into a new 'system:time' stub in the system predicates list. the window launched from here is an experimental new paged panel. I am not sure I really like it, but let's see how it works IRL
- 'system:last view time' is added to search the new field! give it a go once you have some data
- also note that the search and sort of last viewed time works on the 'media viewer' number. those users who use preview or combined numbers for stuff, let me know if and how you would like that to work here--sort/search for both media and preview, try to combine based on the logic in the options, or something else?
- .
- loading serialised pngs:
- the client can now load serialised downloader-pngs if they are a perfect RGB conversion of an original greyscale export.
- the pngs don't technically have to be pngs anymore! if you drag and drop an image from firefox, the temporary bitmap exported and attached to the DnD _should_ work!
- the lain easy downloader import now has a clipboard paste button. it can take regular json text, and now, bitmap data!
- the 'import->from clipboard' button action in many multiple column lists across the program (e.g. manage parsers) (but not every list, a couple are working on older code) also now accepts bitmap data on the clipboard
- the various load errors here are also improved
- .
- custom widget colors:
- (advanced users only for now)
- after banging my head against it, I finally figured out an ok way to send colors from a QSS style file to python code. this means I can convert my custom widgets to inherit colours from the current QSS. I expect to migrate pretty much everything currently fixed over to this, except tag colours and maybe some thumbnail border stuff, and retire the old darkmode
- if you are a QSS lad, please check out the new entries at the bottom of default_hydrus.qss and play around with them in your own QSSes. please do not send me any updates to be folded in to the install yet as I still have a bunch of other colours to add. this week is just a test--please let me know how it works for you

- misc:
- mouse release events no longer trigger a command in the shortcuts system if the release happens more than about 20 pixels from the original mouse down. this is tricky, so if you are into clever shortcuts, let me know how it works for you
- the file maintenance manager (which has been getting a lot of work recently with icc profiles, pixel dupes, some thumb regen, and new audio channel checks), now saves its work and publishes updates faster to the UI, at least once every ten seconds
- the sort entries in the page sort control are now always sorted according to their full (type, name) string, and the mouse-wheel-to-navigate is now fixed to always mirror this
- improved some 'delete file reason' handling. currently, a file deletion reason should only be applied when a file is entering trash. there was a bug that force-physical-deleting files from trash would overwrite their original deletion reason. this is now fixed. the advanced delete files dialog now disables the whole reason panel better when needed, never sends a file reason down to the database when there should be no reason, disables the panel if all the files are in the trash, and at the database level when file deletion reasons are being set, all files are filtered for status beforehand to ensure none are accidentally set by other means. I am about to make trash more intelligent as part of multiple local file services, so I expect to revisit this soon
- the new ICC Profile conversion no longer occurs on I or F mode files. there are weird 32/64 bit monochrome files, and mode/ICC conversion goes whack with current PIL code
- replaced the critical hamming test in the duplicate files system with a different bit-counting strategy that works about 9% faster. hamming test is used in all duplicate file searching, so this should help out a tiny bit in a lot of places
- .
- boring cleanup:
- cleaned up how media viewer canvas type is stored and tested in many places
- all across the program, file viewing statistics are now tracked by type rather than a hardcoded double of preview & media viewer. it will take a moment to update the database to reflect this this week
- cleaned up a ton of file viewing stats code
- cleared out the last twenty or so uses of the old 'execute many select' database access routine in favour of the new lower-overhead and more query-optimisable temporary integer tables method

next week

Back to multiple local file services. A widget to select multiple services in file search, and I'll start on a trash that can handle deletes from and undeletes back to multiple locations.

I had a good week. I fixed a heap of bugs, added a couple new ways to scale thumbnails, and finished the basics of multiple domain file search.

The release should be as normal tomorrow.

full list

- multiple file services:
- I finished the conversion of all UI search to the new multiple location object. everything from back- to frontend now supports cleverer search. since searching deleted files is simple to add, users in advanced mode will now see 'deleted from...' in a new list in the tag autocomplete dropdown file domain button
- the next step is writing a widget that allows multiple selection, and then all this should work right out of the box, and we'll be an important step closer to allowing multiple local file services
- .
- misc:
- the video parsing routine is better at detecting when a present audio track is actually silent (and hence when it should mark a video as 'no audio'). all video with audio will be requeued for a metadata reparse in the files maintenance system on update
- fixed an error from last week when trying to create a new page from the tags (e.g. middle-clicking them) in the active search list
- added 'audio mkv' format to the client, to represent mkvs without a video track. I think most of the time this is going to be audio track webms from youtube-dl and similar
- added 'file relationships: set files as potential duplicates' command to the 'media actions' shortcut set
- I expanded the 'backing up' section in 'installing and updating' help
- I wrote an 'anti-virus' section for 'installing and updating' help, since I kept writing the same basic spiel about false positives. please feel free to point people there in future to relieve their concerns
- improved some shutdown tests, the client and server should exit faster in some cases (e.g. when a hydrus repository network job is hanging on reconnection attempts, holding up the 'synchronise_repository' daemon shutdown)
- the 'file was xxx at (y timestamp), which was (z time units) before this check' line in file import notes now always puts 'z time units' as that, ignoring the 'always show ISO time' setting, which was just substituting it with 'y timestamp' again. let me know if you spot other bad grammar with this setting on, I'll fix it!
- fingers crossed, images in the LAB colourspace _should_ now normalise to sRGB with the correct whitepoint. thanks to the user who provided example test tiff images here. this now uses the new PIL-based colourspace conversion I used to make ICC profiles work, just on LAB->sRGB. as far as I understand, OpenCV uses a fixed whitepoint of D65, resulting in yellow/warm conversions for some formats, but PIL may be able to figure out if D50 is needed??? if you have some crazy LUV or YPbPr or YIQ image that shows up wrong, please send it in and I'll see what I can do!

- boring rewrites and cleanup while doing file service work:
- many more UI objects now store and do file service logic using a more complicated 'location context', which can store a mix of multiple services and 'deleted from service' data. all the search code that works on this can now propagate to display:
- the management objects behind every page now store a multiple location object, not a single file service id
- all media panels (the thumbnail grid on a page) are now instantiated by a multiple location object, and when they serve a highlighted downloader, they now inherit that from the file import options, which in future will dictate import destinations
- all canvases are now the same, inheriting their new location context from their parents
- all tag lists are the same. mostly they don't care much about file domain, but when you middle-click to create new pages from the autocomplete dropdown list or active search list, it can matter, so they now propagate it along
- the underlying medialist objects are now the same, and various delete logic (e.g. 'should we remove this thumb we just deleted?') is updated to work on complex domains
- some duplicate lookup code now works on location context
- renamed 'location search context' object to 'location context' since it is used all over now and put it in its own file. also wrote it some neater initialisation and meta object code
- mr bones now gives duplicate data based on the union of all non-trash local services sans update files (another case of now supporting n services but n is fixed for the moment at 1, 'my files')
- a bunch of places across the program that used to default to 'my files' or 'all local files' (which is everything on disk, including trash and repository update files) now default to this new union of all non-trash local media services
- when doing page-to-page file drag and drops, the location context is now preserved (previously, the new page would always be 'my files')
- whole heap of other cleanup in these systems
- when a thumbnail cannot be provided (for deleted files or many 'all known files' situations), the thumbnail cache now provides the hydrus icon stand-in instantly, no delayed waterfall
- fixed an unusual situation where the file search could not provide a file in a tagless search when that file had no detailed file info row in the database. this seems to effect a legacy borked row or two in the new deleted file domain searches
- removed some ancient dumper status code from thumbnail objects

next week

I have been concentrating on multiple local file services, so I want to take a step back and do some normal work for a week. I have bugs to catch up on and I think I'd like to do something fun, maybe 'file last viewed time', if I can fit it in.

I had a great week doing some fun stuff along with catching up on older issues. I managed to add tracking for 'last viewed time' for files, along with search and sort capability just like for import and modified times. There's also the start of custom colours for the video scanbar using the new style system.

The release should be as normal tomorrow.

I had a mixed week. The tag autocomplete rewrite was more work than I expected. While I am happy with what I did, most of my time was spent on boring background and cleanup, so there is little on the front end to show for it. Beyond that, I fixed the Linux/macOS write permissions checking problem and did a couple small things in the Client API and downloader search log. I will recommend the release just to people interested in those issues.

The release should be as normal tomorrow.

 >>/1199/
 >>/1200/
Thank you, I am sorry for the trouble! I think I have fixed this for tomorrow's release, please try it out and let me know if you have any more problems.

Hey, sorry, Endchan was not working for me at post time last week, so we missed 469 here. I will continue with 470 tomorrow, fingers crossed.

I had a great week. I finished the multiple file service rewrites and can finally start extending search. First off, for tomorrow's release, all advanced users will be able to quickly search their deleted files. I also fixed some bugs and improved some file handling, including the test that detects whether a video has an audio stream or not.

The release should be as normal tomorrow.

 >>/1202/
I just wanted to say thank you for your work.

 >>/1201/
Same anon from  >>/1199/, everything seems to be working fine now on 469. Thanks.

 >>/1203/
 >>/1204/
Thanks lads, keep on pushing.

- misc:
- the 'file log' and 'search log' buttons are now a new widget class that puts an arrow on the side that opens a menu. the secret right-click menus of these buttons is now exposed for all
- fixed a bug affecting some greyscale pngs with ICC profiles--they were coming out pure white due to a colourspace conversion problem
- fixed an import problem when PIL could not load a file (due to file malformation) but OpenCV could. this was causing a failed import from the new ICC profile detection code
- when the downloader hits a broken image file that cannot be imported due to malformation, the status is now 'error' instead of the incorrect 'ignored'
- fixed the duplicate file filesize comparison statement sometimes showing > in one direction and ≈ in the other. it happened when the larger file was between 20/19 and 21/20 times the size of the smaller, just a logic typo (issue #1028)
- the trash maintenance daemon is moved from the old threaded daemon system to the new repeating job worker pool. this is the last daemon cleaned up, so I am retiring the old and mostly defunct 'no_daemons' launch argument. a variety of other daemon infrastructure for things like shutdown checks is similarly removed. the program also now waits for the newer daemon jobs to finish working on shutdown
- moved most client daemon jobs like repository sync and dirty object save down so they start after the first session is loaded rather than right after boot
- if a file is called to regen its thumbnail but currently has no dimension, this is now a no-op rather than an error. in the situation where users force thumb regen before metadata regen and encounter this, it is sorted out later when the metadata regen recognises new dimensions and reschedules the thumb regen
- added an extensive user-written guide to the --db_synchronous_override launch argument to the launch arguments help page. it is possible and safe to run the program with synchronous=0 as long as certain caveats are obeyed. thanks to the user who figured this out and wrote it up
- the downloader engine now discards source urls in an import job if they have the same domain as any existing primary url. this will ensure that if a booru has a link back to itself as a source url, when the 'source' is really an alternate rather than a dupe, it won't be added in hydrus as a known url for that imported file
- misc cleanup in downloader system and file/search log UI
- fixed a type bug in the file and search log 'import from png' action. if you have existing pngs previously exported from here, they will import ok now
- refactored the various hydrus compression code to a new HydrusCompression file
- exported serialisable data pngs such as from file or search log that hold simple Strings now always compress the data before embedding it in the png. existing pngs that hold uncompressed strings should still load ok
- the payload in an exported png is now always compressed, and the payload description always states the uncompressed size
- sped up client shutdown when network traffic has been paused the whole time and a repo sync job might have wanted to run. these jobs also do not hang on a thread worker if network traffic is paused, but they should wake immediately when it is unpaused
- the hydrus login system is now resistant to connection failures; previously it was getting hung up and jamming the whole hydrus sync system when a server was down
- .
- client api:
- added GET /manage_database/mr_bones to the Client API. it returns a JSON Object with the same numbers used in the _help->how boned am I?_ dialog
- incremented Client API version to 23

next week

I'm on holiday for a week, so I'll be back working on Saturday the 1st. I want to grind back into getting multiple local file services done. Thanks everyone!

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had a good week. I fixed bugs--including the recent annoying scanbar border clicking issue--rewrote some bad code, and added a couple of advanced file operations to search and the client api.

The release should be as normal tomorrow.

full list

- video scanbar autohide:
- the scanbar that shows below audio and video is now embedded inside the video frame, and it show/hides based on how close your mouse is to it
- I've wanted to do this for a long time, since it will allow you to watch 16:9 videos at true 100% in borderless fullscreen, but the hackery of how the media viewer works behind the scenes means this took more work than you'd think and is still a little jank. there's a small amount of flicker when it pops in and out, which I will work on in future. in any case, please have a play with it and let me know what you think. I expect to add some more options, like for the activation padding area around it, and I will be tidying up more layout stuff throughout the media viewer
- if you are a mostly keyboard user, please check out the new 'global' shortcut to flip on/off a 'force the animation scanbar to show' mode
- I don't really want to bring back the always-on hanging-below scanbar that just takes up space, but if you try this new embedded scanbar and really hate it, we'll see what we can do
- .
- more duplicate filter search options:
- the duplicates page now has a dropdown on the search for 'must be/can be/excludes pixel dupes'!
- the duplicates page now has a number control on the search for what distance the pair was found at! I am not sure how accurate this thing is in all cases, but it seems I started tracking this data some time ago and forgot I even had it
- these new options are remembered in your session and _should_ remain fast in most normal cases. I put time into some complicated database work this week to get this going, please let me know if you have any trouble with it
- .
- misc:
- when the export filename pattern in the export files dialog means many of the files share the same base and hence need to do 'filename (5)'-style suffixes to be unique, the number here is now calculated much more efficiently. opening this dialog on 10,000 files with an oft-duplicate pattern should now be a jolt of lag but not many minutes
- when you choose to 'separate' a subscription with more than 100 queries, you are no longer forced to break it into half
- when you do break a subscription in half, it now makes sure to sort the query texts before separating
- if you are in advanced mode, the 'selection tags' list on the left of every page can now switch its tag display type between 'multiple media views', 'display', and 'storage'. this is experimental and a bunch of stuff like 'select files with this tag' won't work yet
- janitors' petition pages now start with their tag list in 'storage' mode, so you can see the actual tags being changed rather than with siblings and parents calculated
- rebalanced some janitor mapping petition weights. jannies _should_ see a smoother balance of 'lots of small petitions' vs 'a few larger petitions' amongst petitions all with the same reason and creator

- boring cleanup and little fixes:
- when you set the checker options in the edit subscription dialog, the queries now recalculate their file velocity better. previously, they would just set 'unknown' and recalc on the next run, but now they will actually recalculate if the query container is loaded into memory or otherwise put a status that says 'will calculate on next run'
- removed the 'should be namespaced' reason from the manage tags quick petition reasons. this is now all handled by siblings, tidying up storage tags manually is busywork
- when you click 'copy traceback' on an error popup, it also copies the software version, your platform, and if you are on a frozen build or running from source
- the logger now prints version number for every block, just before the timestamp
- cleaned up a variety of media viewer UI code while working on the scanbar, fixing some misc display bugs
- moved pixel hash storage responsibility from 'file metadata' to 'similar files' module
- the similar files system now searches pixel hashes when it is called to do any similar files search. they count as 'exact match' distance
- when a file gets a new pixel hash, it now sees if any other files have that same hash. if so, it now gets queued up again in the similar files search system, ensuring this match is not missed
- misc nomenclature cleanup--since we now have both 'pixel hashes' and 'phashes', phashes are now referred to as 'perceptual hashes' everywhere
- massively refactored the primary table join that drives potential duplicates search. it should work a bit faster now and it is much easier to work with
- I added pixel dupe and distance search to the standard search results version of the join and the 'system:everything' version, which has several optimisations
- silenced some shutdown handling in file maintenance that was being printed to log as an error
- fixed some 'broken object load' error handling to print the timestamp of the specific bad object, not whatever timestamp was requested. this error handling now also prints the full dump name and version to the log, and version to the exported filename. I was working with a user who had broken subs this week, and lacking this full info made things just a little trickier to put back together
- fixed some drag and drop handling where it was possible to drop thumbnails on a certain location of a page of pages that held an empty page of pages but it would not create a new child media page to hold them
- misc serverside db code cleanup
- fixed python 3.10 type bugs in window coordinate saving and Qt image generation from buffer (issue #1027)

next week

With Christmas coming up, this will be the last full work week of my year. I want to have just a simple cleanup and small-fixes week so I have a fairly unambitious and 'clean' release before I go on holiday.

 >>/1188/
not sure where you will see this first, so ill post in both places. 

as far as the media scrollbar is concerned, I GREATLY prefer to have it below and always on, I am not using hydrus to watch shows or long videos, for me the videos are essentially just replacing gifs at this point. I love being able to see how much longer they have, and being able to scrub through them without covering a good portion of the video. 

If there is a way to have a toggle between visible and hidden, on video or below video, I think that would be best, because as it stands now, it's annoying to use to scrub things because it's not always there, and at least for me it's also annoying to have to keep my mouse there to see how long the video will last.

I should also mention that the way I use hydrus is I have a 4k screen, so I have the left half as media viewer area, and the right half as either the hydrus thumbnails, or a youtube video, I never have it just fullscreen.

I had a good week. I fixed some small bugs and brushed up a little UI. Nothing too ambitious, but it should make for a nice clean release before I break for my holiday.

The release should be as normal tomorrow.

 >>/1191/
Thank you. I untangled all this UI code this week and I feel better about adding options like 'hang below' in future. I have saved your thoughts and will try to enable this again soon into the new year.

better file delete

tl;dr: Files delete from your disk better now.

For a long time, actually physically deleting a file that leaves the client's trash has been a bit hacky. To reduce UI lag, the recycle or delete action has to be deferred to the future, but since there was no permanent record of what to delete, the deferred job was always a bit rushed and in a couple of instances (e.g. if you cleared trash and closed the client soon after) it could leave file orphans behind. The system has always been fail safe, never deleting too much, but it could lead to surplus cruft in file storage. Secondly, it turns out the server never got physical file delete tech added! The file repository has never been used much, and it just never came up.

So, this week I overhauled the whole way it works. Now both client and server keep records of the files they should delete, print delete summaries to the log, use neater delete daemons that smooth out the work to be non-interrupting, have sophisticated 'is this an orphan now?' logic to ensure that edge cases are kept in storage until they are truly no longer needed, and correct themselves in odd cases (like if you re-import a file just a couple of seconds after deleting it from the trash). They are also more aware of repository update files, which is another thing neither have been good at clearing out in the past.

So, you don't have to do anything here, but with luck doing things like 'clear trash' and other large forced physical deletes should just work a bit nicer now. Users who run a server may see it clear out a couple of files after booting, and in a few weeks I'd like to roll out a 'clear orphan files' server database action for the admin menu just like the client's so you can clear out legacy files from deleted services.

full list

- image icc:
- images with embedded icc colour metadata are now normalised (to sRGB) like the rest of media rendering in hydrus. ICC can often mean photos, where a nice camera will apply ICC data to compensate for camera defects or general lighting information, or it can mean normal digital images where the software attached extra colour data when it was saved
- the image will now be rendered with 'fixed' colours in the media viewer, and new thumbnails should be good too. it applies early in image load and should work in all cases hereon, on both client and server
- images with an ICC will take a little longer to initially load. I'd estimate 10-50ms extra for most. one user with many ICC images discovered 10% of their collection had an ICC. I don't think the delay will be terrible IRL, but see how you get on and let me know! maybe giganto patreon pngs will have a fresh surprise for us
- future expansions here will be a database cache of ICC images and system:has icc, perhaps a button to click the ICC application on and off live in the media viewer, and then maybe options to load up and switch an ICC for your display

- better physical file delete:
- both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
- the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
- on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
- when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
- some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
- thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file
- .
- misc:
- fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
- fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
- fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)
- .
- boring image stuff:
- wrote some methods to check for and pull ICC profile bytes from an image with PIL
- wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
- cleaned up some PIL/OpenCV image load and normalisation code
- the decompression bomb check is now quicker for images with rotation
- dequantization is now applied to PIL on all image load by default, it doesn't have to be invoked separately
- some metadata parsing like 'get duration of gif frames' is now faster for images not in RGB or RGBA color
- .
- boring delete code cleanup:
- wrote a heap of new 'is an orphan' filtering logic for client and server
- wrote a daemon job for physical file deletion and plugged it into a new database queue for pending deferred file deletes
- client physical file delete now works off the normal lightweight job scheduler, previously it had its own mainloop thread
- optimised complex file domain file filtering a little
- the 'clear orphan files' job in the client now uses the same updated orphan logic as the new physical delete code. it now won't clear out files in upload limbo
- fixed an issue with re-storing a file in a server after one of its file repositories had previously deleted it. this never mattered previously, when files were never physically deleted, but now the code is brushed up to work properly
- cleaned up some server db code, including the read command method lookup
- moved client 'hash exists?' test down to the master definitions module

next week

Now we have basic ICC support, I want to charge ahead on 'has ICC' and 'pixel dupe' search for the database. I'd also like to find the time to work on some more tag search tech for multiple local file services, but we'll see.

I had a good week. I fixed a variety of small bugs, added some quality of life options, and made several behind the scenes database improvements that will improve search now and in the future.

The release should be as normal tomorrow.