/hydrus/ - Hydrus Network

full list

- times:
- if you have file viewing stats turned on (by default it is), the client will now track the 'last viewed time' of your files, both in preview and media viewers. a record is only made assuming they pass the viewtime checks under _options->file viewing statistics_ (so if you scroll through really quick but have it set to only record after five seconds of viewing, it will not save that as the last viewed time). this last viewed time is shown on the right-click menu with the normal file viewing statistics
- sorting by 'import time' and 'modified time' are moved to a new 'time' subgroup in the sort button menu
- also added to 'time' is 'last viewed time'. note that this has not been tracked until now, so you will have to look at a bunch of things for a few seconds each to get some data to sort with
- to go with 'x time' pattern, 'time imported' is renamed to 'import time' across the program. both should work for system predicate parsing
- system:'import time' and 'modified time' are now bundled into a new 'system:time' stub in the system predicates list. the window launched from here is an experimental new paged panel. I am not sure I really like it, but let's see how it works IRL
- 'system:last view time' is added to search the new field! give it a go once you have some data
- also note that the search and sort of last viewed time works on the 'media viewer' number. those users who use preview or combined numbers for stuff, let me know if and how you would like that to work here--sort/search for both media and preview, try to combine based on the logic in the options, or something else?
- .
- loading serialised pngs:
- the client can now load serialised downloader-pngs if they are a perfect RGB conversion of an original greyscale export.
- the pngs don't technically have to be pngs anymore! if you drag and drop an image from firefox, the temporary bitmap exported and attached to the DnD _should_ work!
- the lain easy downloader import now has a clipboard paste button. it can take regular json text, and now, bitmap data!
- the 'import->from clipboard' button action in many multiple column lists across the program (e.g. manage parsers) (but not every list, a couple are working on older code) also now accepts bitmap data on the clipboard
- the various load errors here are also improved
- .
- custom widget colors:
- (advanced users only for now)
- after banging my head against it, I finally figured out an ok way to send colors from a QSS style file to python code. this means I can convert my custom widgets to inherit colours from the current QSS. I expect to migrate pretty much everything currently fixed over to this, except tag colours and maybe some thumbnail border stuff, and retire the old darkmode
- if you are a QSS lad, please check out the new entries at the bottom of default_hydrus.qss and play around with them in your own QSSes. please do not send me any updates to be folded in to the install yet as I still have a bunch of other colours to add. this week is just a test--please let me know how it works for you

- misc:
- mouse release events no longer trigger a command in the shortcuts system if the release happens more than about 20 pixels from the original mouse down. this is tricky, so if you are into clever shortcuts, let me know how it works for you
- the file maintenance manager (which has been getting a lot of work recently with icc profiles, pixel dupes, some thumb regen, and new audio channel checks), now saves its work and publishes updates faster to the UI, at least once every ten seconds
- the sort entries in the page sort control are now always sorted according to their full (type, name) string, and the mouse-wheel-to-navigate is now fixed to always mirror this
- improved some 'delete file reason' handling. currently, a file deletion reason should only be applied when a file is entering trash. there was a bug that force-physical-deleting files from trash would overwrite their original deletion reason. this is now fixed. the advanced delete files dialog now disables the whole reason panel better when needed, never sends a file reason down to the database when there should be no reason, disables the panel if all the files are in the trash, and at the database level when file deletion reasons are being set, all files are filtered for status beforehand to ensure none are accidentally set by other means. I am about to make trash more intelligent as part of multiple local file services, so I expect to revisit this soon
- the new ICC Profile conversion no longer occurs on I or F mode files. there are weird 32/64 bit monochrome files, and mode/ICC conversion goes whack with current PIL code
- replaced the critical hamming test in the duplicate files system with a different bit-counting strategy that works about 9% faster. hamming test is used in all duplicate file searching, so this should help out a tiny bit in a lot of places
- .
- boring cleanup:
- cleaned up how media viewer canvas type is stored and tested in many places
- all across the program, file viewing statistics are now tracked by type rather than a hardcoded double of preview & media viewer. it will take a moment to update the database to reflect this this week
- cleaned up a ton of file viewing stats code
- cleared out the last twenty or so uses of the old 'execute many select' database access routine in favour of the new lower-overhead and more query-optimisable temporary integer tables method

next week

Back to multiple local file services. A widget to select multiple services in file search, and I'll start on a trash that can handle deletes from and undeletes back to multiple locations.

I had a good week. I fixed a heap of bugs, added a couple new ways to scale thumbnails, and finished the basics of multiple domain file search.

The release should be as normal tomorrow.

full list

- multiple file services:
- I finished the conversion of all UI search to the new multiple location object. everything from back- to frontend now supports cleverer search. since searching deleted files is simple to add, users in advanced mode will now see 'deleted from...' in a new list in the tag autocomplete dropdown file domain button
- the next step is writing a widget that allows multiple selection, and then all this should work right out of the box, and we'll be an important step closer to allowing multiple local file services
- .
- misc:
- the video parsing routine is better at detecting when a present audio track is actually silent (and hence when it should mark a video as 'no audio'). all video with audio will be requeued for a metadata reparse in the files maintenance system on update
- fixed an error from last week when trying to create a new page from the tags (e.g. middle-clicking them) in the active search list
- added 'audio mkv' format to the client, to represent mkvs without a video track. I think most of the time this is going to be audio track webms from youtube-dl and similar
- added 'file relationships: set files as potential duplicates' command to the 'media actions' shortcut set
- I expanded the 'backing up' section in 'installing and updating' help
- I wrote an 'anti-virus' section for 'installing and updating' help, since I kept writing the same basic spiel about false positives. please feel free to point people there in future to relieve their concerns
- improved some shutdown tests, the client and server should exit faster in some cases (e.g. when a hydrus repository network job is hanging on reconnection attempts, holding up the 'synchronise_repository' daemon shutdown)
- the 'file was xxx at (y timestamp), which was (z time units) before this check' line in file import notes now always puts 'z time units' as that, ignoring the 'always show ISO time' setting, which was just substituting it with 'y timestamp' again. let me know if you spot other bad grammar with this setting on, I'll fix it!
- fingers crossed, images in the LAB colourspace _should_ now normalise to sRGB with the correct whitepoint. thanks to the user who provided example test tiff images here. this now uses the new PIL-based colourspace conversion I used to make ICC profiles work, just on LAB->sRGB. as far as I understand, OpenCV uses a fixed whitepoint of D65, resulting in yellow/warm conversions for some formats, but PIL may be able to figure out if D50 is needed??? if you have some crazy LUV or YPbPr or YIQ image that shows up wrong, please send it in and I'll see what I can do!

- boring rewrites and cleanup while doing file service work:
- many more UI objects now store and do file service logic using a more complicated 'location context', which can store a mix of multiple services and 'deleted from service' data. all the search code that works on this can now propagate to display:
- the management objects behind every page now store a multiple location object, not a single file service id
- all media panels (the thumbnail grid on a page) are now instantiated by a multiple location object, and when they serve a highlighted downloader, they now inherit that from the file import options, which in future will dictate import destinations
- all canvases are now the same, inheriting their new location context from their parents
- all tag lists are the same. mostly they don't care much about file domain, but when you middle-click to create new pages from the autocomplete dropdown list or active search list, it can matter, so they now propagate it along
- the underlying medialist objects are now the same, and various delete logic (e.g. 'should we remove this thumb we just deleted?') is updated to work on complex domains
- some duplicate lookup code now works on location context
- renamed 'location search context' object to 'location context' since it is used all over now and put it in its own file. also wrote it some neater initialisation and meta object code
- mr bones now gives duplicate data based on the union of all non-trash local services sans update files (another case of now supporting n services but n is fixed for the moment at 1, 'my files')
- a bunch of places across the program that used to default to 'my files' or 'all local files' (which is everything on disk, including trash and repository update files) now default to this new union of all non-trash local media services
- when doing page-to-page file drag and drops, the location context is now preserved (previously, the new page would always be 'my files')
- whole heap of other cleanup in these systems
- when a thumbnail cannot be provided (for deleted files or many 'all known files' situations), the thumbnail cache now provides the hydrus icon stand-in instantly, no delayed waterfall
- fixed an unusual situation where the file search could not provide a file in a tagless search when that file had no detailed file info row in the database. this seems to effect a legacy borked row or two in the new deleted file domain searches
- removed some ancient dumper status code from thumbnail objects

next week

I have been concentrating on multiple local file services, so I want to take a step back and do some normal work for a week. I have bugs to catch up on and I think I'd like to do something fun, maybe 'file last viewed time', if I can fit it in.

I had a great week doing some fun stuff along with catching up on older issues. I managed to add tracking for 'last viewed time' for files, along with search and sort capability just like for import and modified times. There's also the start of custom colours for the video scanbar using the new style system.

The release should be as normal tomorrow.

I had a mixed week. The tag autocomplete rewrite was more work than I expected. While I am happy with what I did, most of my time was spent on boring background and cleanup, so there is little on the front end to show for it. Beyond that, I fixed the Linux/macOS write permissions checking problem and did a couple small things in the Client API and downloader search log. I will recommend the release just to people interested in those issues.

The release should be as normal tomorrow.

 >>/1199/
 >>/1200/
Thank you, I am sorry for the trouble! I think I have fixed this for tomorrow's release, please try it out and let me know if you have any more problems.

Hey, sorry, Endchan was not working for me at post time last week, so we missed 469 here. I will continue with 470 tomorrow, fingers crossed.

I had a great week. I finished the multiple file service rewrites and can finally start extending search. First off, for tomorrow's release, all advanced users will be able to quickly search their deleted files. I also fixed some bugs and improved some file handling, including the test that detects whether a video has an audio stream or not.

The release should be as normal tomorrow.

 >>/1202/
I just wanted to say thank you for your work.

 >>/1201/
Same anon from  >>/1199/, everything seems to be working fine now on 469. Thanks.

 >>/1203/
 >>/1204/
Thanks lads, keep on pushing.

- misc:
- the 'file log' and 'search log' buttons are now a new widget class that puts an arrow on the side that opens a menu. the secret right-click menus of these buttons is now exposed for all
- fixed a bug affecting some greyscale pngs with ICC profiles--they were coming out pure white due to a colourspace conversion problem
- fixed an import problem when PIL could not load a file (due to file malformation) but OpenCV could. this was causing a failed import from the new ICC profile detection code
- when the downloader hits a broken image file that cannot be imported due to malformation, the status is now 'error' instead of the incorrect 'ignored'
- fixed the duplicate file filesize comparison statement sometimes showing > in one direction and ≈ in the other. it happened when the larger file was between 20/19 and 21/20 times the size of the smaller, just a logic typo (issue #1028)
- the trash maintenance daemon is moved from the old threaded daemon system to the new repeating job worker pool. this is the last daemon cleaned up, so I am retiring the old and mostly defunct 'no_daemons' launch argument. a variety of other daemon infrastructure for things like shutdown checks is similarly removed. the program also now waits for the newer daemon jobs to finish working on shutdown
- moved most client daemon jobs like repository sync and dirty object save down so they start after the first session is loaded rather than right after boot
- if a file is called to regen its thumbnail but currently has no dimension, this is now a no-op rather than an error. in the situation where users force thumb regen before metadata regen and encounter this, it is sorted out later when the metadata regen recognises new dimensions and reschedules the thumb regen
- added an extensive user-written guide to the --db_synchronous_override launch argument to the launch arguments help page. it is possible and safe to run the program with synchronous=0 as long as certain caveats are obeyed. thanks to the user who figured this out and wrote it up
- the downloader engine now discards source urls in an import job if they have the same domain as any existing primary url. this will ensure that if a booru has a link back to itself as a source url, when the 'source' is really an alternate rather than a dupe, it won't be added in hydrus as a known url for that imported file
- misc cleanup in downloader system and file/search log UI
- fixed a type bug in the file and search log 'import from png' action. if you have existing pngs previously exported from here, they will import ok now
- refactored the various hydrus compression code to a new HydrusCompression file
- exported serialisable data pngs such as from file or search log that hold simple Strings now always compress the data before embedding it in the png. existing pngs that hold uncompressed strings should still load ok
- the payload in an exported png is now always compressed, and the payload description always states the uncompressed size
- sped up client shutdown when network traffic has been paused the whole time and a repo sync job might have wanted to run. these jobs also do not hang on a thread worker if network traffic is paused, but they should wake immediately when it is unpaused
- the hydrus login system is now resistant to connection failures; previously it was getting hung up and jamming the whole hydrus sync system when a server was down
- .
- client api:
- added GET /manage_database/mr_bones to the Client API. it returns a JSON Object with the same numbers used in the _help->how boned am I?_ dialog
- incremented Client API version to 23

next week

I'm on holiday for a week, so I'll be back working on Saturday the 1st. I want to grind back into getting multiple local file services done. Thanks everyone!

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had a good week. I fixed bugs--including the recent annoying scanbar border clicking issue--rewrote some bad code, and added a couple of advanced file operations to search and the client api.

The release should be as normal tomorrow.

full list

- video scanbar autohide:
- the scanbar that shows below audio and video is now embedded inside the video frame, and it show/hides based on how close your mouse is to it
- I've wanted to do this for a long time, since it will allow you to watch 16:9 videos at true 100% in borderless fullscreen, but the hackery of how the media viewer works behind the scenes means this took more work than you'd think and is still a little jank. there's a small amount of flicker when it pops in and out, which I will work on in future. in any case, please have a play with it and let me know what you think. I expect to add some more options, like for the activation padding area around it, and I will be tidying up more layout stuff throughout the media viewer
- if you are a mostly keyboard user, please check out the new 'global' shortcut to flip on/off a 'force the animation scanbar to show' mode
- I don't really want to bring back the always-on hanging-below scanbar that just takes up space, but if you try this new embedded scanbar and really hate it, we'll see what we can do
- .
- more duplicate filter search options:
- the duplicates page now has a dropdown on the search for 'must be/can be/excludes pixel dupes'!
- the duplicates page now has a number control on the search for what distance the pair was found at! I am not sure how accurate this thing is in all cases, but it seems I started tracking this data some time ago and forgot I even had it
- these new options are remembered in your session and _should_ remain fast in most normal cases. I put time into some complicated database work this week to get this going, please let me know if you have any trouble with it
- .
- misc:
- when the export filename pattern in the export files dialog means many of the files share the same base and hence need to do 'filename (5)'-style suffixes to be unique, the number here is now calculated much more efficiently. opening this dialog on 10,000 files with an oft-duplicate pattern should now be a jolt of lag but not many minutes
- when you choose to 'separate' a subscription with more than 100 queries, you are no longer forced to break it into half
- when you do break a subscription in half, it now makes sure to sort the query texts before separating
- if you are in advanced mode, the 'selection tags' list on the left of every page can now switch its tag display type between 'multiple media views', 'display', and 'storage'. this is experimental and a bunch of stuff like 'select files with this tag' won't work yet
- janitors' petition pages now start with their tag list in 'storage' mode, so you can see the actual tags being changed rather than with siblings and parents calculated
- rebalanced some janitor mapping petition weights. jannies _should_ see a smoother balance of 'lots of small petitions' vs 'a few larger petitions' amongst petitions all with the same reason and creator

- boring cleanup and little fixes:
- when you set the checker options in the edit subscription dialog, the queries now recalculate their file velocity better. previously, they would just set 'unknown' and recalc on the next run, but now they will actually recalculate if the query container is loaded into memory or otherwise put a status that says 'will calculate on next run'
- removed the 'should be namespaced' reason from the manage tags quick petition reasons. this is now all handled by siblings, tidying up storage tags manually is busywork
- when you click 'copy traceback' on an error popup, it also copies the software version, your platform, and if you are on a frozen build or running from source
- the logger now prints version number for every block, just before the timestamp
- cleaned up a variety of media viewer UI code while working on the scanbar, fixing some misc display bugs
- moved pixel hash storage responsibility from 'file metadata' to 'similar files' module
- the similar files system now searches pixel hashes when it is called to do any similar files search. they count as 'exact match' distance
- when a file gets a new pixel hash, it now sees if any other files have that same hash. if so, it now gets queued up again in the similar files search system, ensuring this match is not missed
- misc nomenclature cleanup--since we now have both 'pixel hashes' and 'phashes', phashes are now referred to as 'perceptual hashes' everywhere
- massively refactored the primary table join that drives potential duplicates search. it should work a bit faster now and it is much easier to work with
- I added pixel dupe and distance search to the standard search results version of the join and the 'system:everything' version, which has several optimisations
- silenced some shutdown handling in file maintenance that was being printed to log as an error
- fixed some 'broken object load' error handling to print the timestamp of the specific bad object, not whatever timestamp was requested. this error handling now also prints the full dump name and version to the log, and version to the exported filename. I was working with a user who had broken subs this week, and lacking this full info made things just a little trickier to put back together
- fixed some drag and drop handling where it was possible to drop thumbnails on a certain location of a page of pages that held an empty page of pages but it would not create a new child media page to hold them
- misc serverside db code cleanup
- fixed python 3.10 type bugs in window coordinate saving and Qt image generation from buffer (issue #1027)

next week

With Christmas coming up, this will be the last full work week of my year. I want to have just a simple cleanup and small-fixes week so I have a fairly unambitious and 'clean' release before I go on holiday.

 >>/1188/
not sure where you will see this first, so ill post in both places. 

as far as the media scrollbar is concerned, I GREATLY prefer to have it below and always on, I am not using hydrus to watch shows or long videos, for me the videos are essentially just replacing gifs at this point. I love being able to see how much longer they have, and being able to scrub through them without covering a good portion of the video. 

If there is a way to have a toggle between visible and hidden, on video or below video, I think that would be best, because as it stands now, it's annoying to use to scrub things because it's not always there, and at least for me it's also annoying to have to keep my mouse there to see how long the video will last.

I should also mention that the way I use hydrus is I have a 4k screen, so I have the left half as media viewer area, and the right half as either the hydrus thumbnails, or a youtube video, I never have it just fullscreen.

I had a good week. I fixed some small bugs and brushed up a little UI. Nothing too ambitious, but it should make for a nice clean release before I break for my holiday.

The release should be as normal tomorrow.

 >>/1191/
Thank you. I untangled all this UI code this week and I feel better about adding options like 'hang below' in future. I have saved your thoughts and will try to enable this again soon into the new year.

better file delete

tl;dr: Files delete from your disk better now.

For a long time, actually physically deleting a file that leaves the client's trash has been a bit hacky. To reduce UI lag, the recycle or delete action has to be deferred to the future, but since there was no permanent record of what to delete, the deferred job was always a bit rushed and in a couple of instances (e.g. if you cleared trash and closed the client soon after) it could leave file orphans behind. The system has always been fail safe, never deleting too much, but it could lead to surplus cruft in file storage. Secondly, it turns out the server never got physical file delete tech added! The file repository has never been used much, and it just never came up.

So, this week I overhauled the whole way it works. Now both client and server keep records of the files they should delete, print delete summaries to the log, use neater delete daemons that smooth out the work to be non-interrupting, have sophisticated 'is this an orphan now?' logic to ensure that edge cases are kept in storage until they are truly no longer needed, and correct themselves in odd cases (like if you re-import a file just a couple of seconds after deleting it from the trash). They are also more aware of repository update files, which is another thing neither have been good at clearing out in the past.

So, you don't have to do anything here, but with luck doing things like 'clear trash' and other large forced physical deletes should just work a bit nicer now. Users who run a server may see it clear out a couple of files after booting, and in a few weeks I'd like to roll out a 'clear orphan files' server database action for the admin menu just like the client's so you can clear out legacy files from deleted services.

full list

- image icc:
- images with embedded icc colour metadata are now normalised (to sRGB) like the rest of media rendering in hydrus. ICC can often mean photos, where a nice camera will apply ICC data to compensate for camera defects or general lighting information, or it can mean normal digital images where the software attached extra colour data when it was saved
- the image will now be rendered with 'fixed' colours in the media viewer, and new thumbnails should be good too. it applies early in image load and should work in all cases hereon, on both client and server
- images with an ICC will take a little longer to initially load. I'd estimate 10-50ms extra for most. one user with many ICC images discovered 10% of their collection had an ICC. I don't think the delay will be terrible IRL, but see how you get on and let me know! maybe giganto patreon pngs will have a fresh surprise for us
- future expansions here will be a database cache of ICC images and system:has icc, perhaps a button to click the ICC application on and off live in the media viewer, and then maybe options to load up and switch an ICC for your display

- better physical file delete:
- both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
- the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
- on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
- when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
- some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
- thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file
- .
- misc:
- fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
- fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
- fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)
- .
- boring image stuff:
- wrote some methods to check for and pull ICC profile bytes from an image with PIL
- wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
- cleaned up some PIL/OpenCV image load and normalisation code
- the decompression bomb check is now quicker for images with rotation
- dequantization is now applied to PIL on all image load by default, it doesn't have to be invoked separately
- some metadata parsing like 'get duration of gif frames' is now faster for images not in RGB or RGBA color
- .
- boring delete code cleanup:
- wrote a heap of new 'is an orphan' filtering logic for client and server
- wrote a daemon job for physical file deletion and plugged it into a new database queue for pending deferred file deletes
- client physical file delete now works off the normal lightweight job scheduler, previously it had its own mainloop thread
- optimised complex file domain file filtering a little
- the 'clear orphan files' job in the client now uses the same updated orphan logic as the new physical delete code. it now won't clear out files in upload limbo
- fixed an issue with re-storing a file in a server after one of its file repositories had previously deleted it. this never mattered previously, when files were never physically deleted, but now the code is brushed up to work properly
- cleaned up some server db code, including the read command method lookup
- moved client 'hash exists?' test down to the master definitions module

next week

Now we have basic ICC support, I want to charge ahead on 'has ICC' and 'pixel dupe' search for the database. I'd also like to find the time to work on some more tag search tech for multiple local file services, but we'll see.

I had a good week. I fixed a variety of small bugs, added some quality of life options, and made several behind the scenes database improvements that will improve search now and in the future.

The release should be as normal tomorrow.

full list

- misc:
- ogv files (ogg with a video stream) are now recognised by the client! they will get resolution, duration, num frames and now show in the media viewer correctly as resizeable videos. all your existing ogg files will be scheduled for a rescan on update
- wrote new downloader objects to fix deviant art tag search. all clients will automatically update and should with luck just be working again with the same old 'deviant art tag search' downloader
- added prototype copy/paste buttons to the manage ratings dialog. the copy button also grabs 'null' ratings, let me know how you get on here and we'll tweak as needed
- file searches with namespaced and unnamespaced tags should now run just a little faster
- most file searches with multiple search predicates that include normal tags should now run just a little faster
- the file log right-click menu now shows 'delete x yyy files from the queue' for deleted, ignored, failed, and skipped states separately
- the tag siblings + parents display sync manager now forces more wait time before it does work. it now waits for the database and gui to be free of pending or current background work. this _should_ stop slower clients getting into hangs when the afterwork updates pile up and overwhelm the main work
- the option 'warn at this many pages' under _options->gui pages_ now has a max value of 65535, up from 500. if you are a madman or you have very page-spammy subscriptions, feel free to try boosting this super high. be warned this may lead to resource limit crashes
- the 'max pages' value that triggers a full yes/no dialog on page open is now set as the maximum value of 500 and 2 x the 'warn at this many pages' value
- the 'max pages' dialog trigger now only fires if there are no dialogs currently open (this should fix a nested dialog crash when page-publishing subscriptions goes bananas)
- improved error reporting for unsolveable cloudflare captcha errors
- added clarification text to the edit subscription query dialog regarding the tag import options there
- added/updated a bunch of file import options tooltips
- .
- new presentation import options:
- the 'presentation' section of 'file import options' has been completely overhauled. it can do more, and is more human-friendly
- rather than the old three checkboxes of new/already-in-inbox/already-in-archive, you now choose from three dropdowns--do you want all/new/none, do you want all/only-inbox/inbox-too, and do you want to my-files/and-trash-too. it is now possible to show 'in inbox' exclusively, at the time of publish (e.g. when you highlight)
- added a little help UI text around the places presentation is used
- the downloader and watcher page's list right-click menu entries for 'show xxx files' is now a submenu, uses the new presentation import options, shows 'show inbox files', and if you click on one row it says what the default is and excludes other entries if they are duplicates

- boring presentation import options stuff:
- presentation options are now in their own object and will be easier to update in future
- the 'should I present' code is radically cleaned up and refactored to a single central object
- presentation filtering in general has more sophisticated logic and is faster when used on a list (e.g. when you highlight a decent sized downloader and it has to figure out which thumbs to show). it is now careful about only checking for inbox status on outstanding files
- presentation now always checks file domain, whereas before this was ad-hoc and scattered around (and in some buggy cases lead to long-deleted files appearing in import views)
- added a full suite of unit tests to ensure the presentation import options object is making the right decisions and filtering efficiently at each stage
- .
- boring multiple local file services work:
- I basically moved a bunch of file search code from 1 file services to n file services:
- the file storage module can now filter file ids to a complex location search context
- namespace:anything searches of various sorts now use complex location search contexts
- wildcard tag searches now use complex location search contexts
- counted tag searches now use complex location search contexts
- search code that uses complex location search contexts now cross-references its file results in all cases
- I now have a great plan to add deleted files search and keep it working quick. this will be the next step, and then I can do efficient complex-location regular tag search and hopefully just switch over the autocomplete control to allow deleted files search

next week

I managed to fit in some good work on multiple local file services this week as well. Most of the tag search code now works on n file services, with n currently locked to 1, ha ha ha. The next step here is to add a simple cache for all deleted files so I can search their tags quickly. I will move this forward when I next have some time.

Beyond that, I want to get around to adding proper ICC support. This is colour-correction data that some files ship with. It mostly appears in camera photos, but some software puts it in normal static images too. I have been talking with a user about this for a while, and I think there is now a path to do it fast and accurately. I'd love to just have natural ICC support, showing it as a bitmap flag or something and just altering colours as the file specifies.

It is Thanksgiving in the US tomorrow. Thanks everyone!

I had a good week. I managed to add support for embedded ICCs, which will improve some images' colours, and overhauled how files are deleted from the client and serevr so the system is smoother and more reliable.

The release should be as normal tomorrow.

- network redirect improvements:
- gallery page jobs now give their child 'next gallery page' jobs themselves as a referrer
- when the gallery downloader gets a 3XX redirect, the file import objects and next gallery pages it makes now get the redirected URL as referral url (previously, it used the original gallery url)
- when the post downloader gets a 3XX redirect, the redirected url is now added as a primary source url
- when the post downloader gets a 3XX redirect, child import objects and direct file downloads now get the redirect URL as referral url (previously, it used the original post url)
- when the raw file downloader gets a 3XX redirect, the redirected url is now added as a primary source url
- when the raw file downloader gets a 3XX redirect to a Post URL, it now tries to queue that URL back up in the file log, just like when a gallery fetch comes back with a Post URL. some safety code stops potential loops here as well
- .
- new services:
- a new client now starts with a second local tag service called 'downloader tags'. default tag import options are now initialised in a fresh client to pull all file post tags to this service. this should relieve new users' confusion about setting up default tag import options
- similarly, a new client now starts with a like/dislike rating service called 'favourites'. existing users who have no rating services will be asked if they want to get it on update. many users are unaware of the rating system, so this is a chance to play with it
- the 'getting started with downloading' 'and '...with ratings' help has some updated screenshots and talk about the new default services and parsing
- .
- database fixes:
- fixed a very slow database query that was involved with file search using unnamespaced tags when other search predicates had already limited the scope of search
- fixed a similar slow query when the 'bad' search predicate was 'namespace:anything', particularly when the namespace is a wildcard
- fixed the 'clear orphan tables' database maintenance routine. it had decayed due to bit rot and was deleting new repository processing tracking tables. the routine is now plugged directly into the new database modules system, and any module now can be upgraded to discover surplus service tables. the system has been updated to permit the detection and removal of duplicate tables in the wrong database file, and it also makes a statement if no orphan tables were found
- the 'metadata resync' repository maintenance task now removes surplus file rows from the processing tracking tables
- the 'metadata resync' repository maintenance task now corrects content type rows in the main processing tracking table
- the process of registering updates to a repository is now a little faster, more reliable, repairs basic damage, and keeps more good work when damage needs to be repaired
- I _think_ the users who were still getting PTR processing errors related to a database confusion about content/definitions update files should now be fixed after another full metadata resync! please let me know if there are still problems

next week

This week ended up being packed, so I did not have time for larger stuff like multiple local file services. I'll try again to move it forward.

I had a great week. As well as some little fixes and quality of life work, I improved support for ogv files (oggs with video), and completely reworked the 'presentation' options of importers, making the whole system faster and more user-friendly.

The release should be as normal tomorrow.