/hydrus/ - Hydrus Network

everything is broke again
venv won't start because there is no activate.bat
why do you hate me so much, why do you make me go through all this pain and torture? just because i won't submit to windows10?

I had a great week fixing some bugs, adding a hash lookup to the Client API, and attaching a Tag Filter to the PTR, which will make it easier to clear out some less useful and misparsed tags right as they are uploaded, no petition needed.

The release should be as normal tomorrow.

 >>/1420/
Sorry to hear your trouble. Can you talk more about what happened? Did you change the venv, or could an anti-virus or other program have interfered with it? If the activate.bat has disappeared, try running the setup-venv.bat again--that'll clear out your old venv folder and reinstall everything.

Since you have had problems with this, could you tell me what has been most difficult to get going? Has my 'running from source' help been confusing at any point? Did the setup bat fail? How did things break before, and what was the solution? How could I make it easier? I have been adjusting the help and the scripts in recent weeks to be more clear on each step, so I regret it hasn't worked for you.

 >>/1421/
I managed to make it work by building from scratch in a new folder.
Before I had database failure, due to my mistake of not backing it up properly.
Venv failed before until I found working mix of qt5, old python, new mpv.
It's painful because I don't even know the difference and simply seeking blindly for something that will run somehow.

 >>/1423/
Sorry again for the problems. I am glad you are working now. Which old version of python worked for you? 3.8 or earlier? To make it less of a 'blind' situation in future, how could I improve my help? Are there specific version recommendations I could make for people on Windows 7?

You can back up the venv folder, btw, so now you have something that works, you can save it all--it is completely portable.

I will try and ensure that hydrus keeps working for Qt5 and older python as long as I reasonably can, so let me know if a future update breaks something.

I had an ok week fixing some bugs, expanding the Client API a little, and improving some server admin tools.

The release should be as normal tomorrow.

icc profile and pixel hash

This is mostly database prep for future duplicate system expansions.

The client database now records whether still images have an ICC profile, and it also saves data for 'these images are exact pixel duplicates' decisions. On update, all your existing files will be queued for scans to fill in this data in the background. Anything with an ICC profile will also regenerate its thumbnail. You don't have to do anything, this will all happen automatically over the coming week(s).

In time, you'll be able to search for images with ICC profiles with the new 'system:has icc profile' search predicate. This predicate is weird and advanced, so I think I'll hide it away soon under an umbrella for advanced stuff.

The 'exact pixel duplicate' data will be useful in the near future, when I expand the duplicate system to find (and optionally automatically merge) certain pairs that are perfect visual dupes.

full list

- misc:
- fixed a recent bug in wildcard search where 't*g' unnamespaced wildcards were not returning namespace results
- sped up multi-predicate wildcard searches in some unusual file domains
- the file maintenance manager is now more polite about how it works. no matter its speed settings, it now pauses regularly to check on and wait until the main UI is feeling good to start new work. this should relieve heavier clients on older machines who will get a bunch of new work to do this week
- 'all local files' in _review services_ now states how many files are awaiting physical deletion in the new deferred delete queue. this live updates when the values change but should be 0 pretty much all the time
- 'all known files' in _review services_ also gets a second safety yes/no dialog on its clear deleted files record button
- updated the gelbooru 0.2.x gallery page parser, which at some point had been pulling 'delete from favourites' links when running the login-required 'gelbooru favorites by user id' downloader!!! it was deleting favourites, which I presume and hope was a recent change in gelbooru's html. in any case, the parser now skips over any deletion url (issue #1023)
- fixed a bad index to label conversion in a common database progress method. it was commonly saying 22/21 progress instead of 21/21
- fixed an error when manage tags dialog posts tags from the autocomplete during dialog shutdown
- fixed a layout issue with the new presentation import options where the dropdowns could grow a little tall and make a sub-panel scrollbar
- added handling for an error raised on loading an image with what seems to be a borked ICC profile
- increased the default per-db-file cache size from 200MB up to 256MB
- .
- some new options:
- the default tag service in the manage tags dialog (and some similar 'tag services in a row of tabs' dialogs) is reset this week. from now on, the last used service is remembered for the next time the dialog is opened. let's see how that works out. if you don't like it, you can go back to the old fixed default setting under the 'tags' options panel
- added a checkbox to the 'search' options panel that controls whether new search pages are in 'searching immediately' or 'searching paused' state (issue #761)
- moved default tag sort from 'tags' options panel to 'sort/collect'
- .
- deleted files and ipfs searchability:
- wrote a new virtual file service to hold all previously deleted files of all real file services. this provides a mapping cache and tag lookup cache allowing for fast search of any deleted file domain in the future
- ipfs services also now have mapping caches and tag search caches
- ipfs services are now searchable in the main file search view! just select them from the autocomplete dropdown file domain button. they have tag counts and everything
- it will take some time to populate the new ipfs and deleted files caches. if you don't have much deleted files history and no ipfs, it will be a few seconds. if you have a massive client with many deleted/ipfs files and many tags, it could be twenty minutes or more

- 'has icc profile' now cached in database:
- the client database now keeps track of which image files have an icc profile. this data is added on file import
- a new file maintenance task can generate it retroactively, and if a file is discovered to have an icc profile, it will be scheduled for a thumbnail regeneration too
- a new system predicate, 'system:has icc profile', can now search this data. this system pred is weird, so I expect in future it will get tucked into an umbrella system pred for advanced/rare stuff
- on update, all your existing image files are scheduled for the maintenance task. your 'has icc profile' will populate over time, and thumbnails will correct themselves
- .
- pixel hash now cached in database:
- the client database now keeps track of image 'pixel hashes', which are fast unique identifiers that aggregate all that image's pixels. if two images have the same pixel hash, they are pixel duplicates. this data is added on file import
- a new file maintenance task can generate it retroactively
- on update, all your existing image files are scheduled for the maintenance task. it'll work lightly in the background in prep for future duplicate file system improvements
- .
- boring search code cleanup:
- fixed a bug where the tag search cache could lose sibling/parent-chained values when their count was zeroed, even though they should always exist in a domain's lookup
- fixed up some repository reset code that was regenerating surplus tag search data
- with the new deleted files domain, simplified the new complex domain search pattern
- converted basic tag search code to support multiple location domains
- cleaned up some search cache and general table creation code to handle legacy orphan tables without error
- misc tag and mapping cache code and combined local files code refactoring and cleanup

next week

I'll take Christmas week off, so I only have two more proper weeks in the year. I would like to have basic pixel duplicate search working before then. Just a dropdown on the duplicates page for 'pair must/must not be pixel dupes' or similar. So, I will work on that and see if we can aim for a 'clean' release for end of year.

birthday

My todo list reminded me yesterday that I put out the first non-experimental beta of hydrus on December 14th, 2011. This is the rough 'start date' of the project and its birthday now. It will be ten years this week, which is pretty crazy.

Like a lot of people, 2021 was an odd year for me. I changed some lifestyle stuff, dropping some unhealthy habits, and also had some unexpected stress. After looking back though, I am overall happy with my work. Although I completed fewer big new projects than I hoped, and at times I felt bogged down in rewrites and fixes, the general performance of the client grew significantly this year. As well as a variety of new tag search and display options, the sibling and parent system was completely overhauled on several fronts, with the improved virtualised storage in the database and asynchronous real-time application calculation, and with that the autocomplete search finally supported 'perfect' sibling+parent adjusted tag counts in very fast time. Years-old sibling and parent application bugs were finally drilled down to and fixed. The tag lists across the program gained better sibling and parent display and commands. Wildcard searches became much faster too, and all sorts of tag and file search improved on smaller domains, sometimes by a factor of a thousand, even when a client had the whole PTR lurking in the background. We also moved to automatic repository account creation and improved serverside privacy, tags became easier to sort, file and database maintenance gained multiple new commands that saved a ton of time and inconvenience, the database learned to repair much of itself, system predicates became parseable in the Client API and editable in main UI, we moved from my duct-taped dev machines to github-built releases, the image renderer moved to a tiled system that allowed very fast zoom, sessions could grow much larger without CPU death and could save to disk with a fraction of their old write I/O, and a whole ton of other little fixes and quality of life improvements to every system.

I get a lot out of working on hydrus, and I hope to continue just like this. I appreciate everyone's feedback and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

pixel dupe and... mp4
(1.37 MB, 618x366 h264)

I had a great week. I managed to improve the duplicate filter search more than I thought, adding the ability to filter based on pixel duplicates and also pair similarity, and then I was able to rework the video scanbar so it sits inside the video frame and autohides based on mouse position.

The release should be as normal tomorrow.

dfcgvbhjnk

misc highlights

If you are interested in changing page drag and drop behaviour or regularly have overfull page tab bars, check the new checkboxes in options->gui pages.

If you are on Windows and have the default options->style, booting the client with your Windows 'app darkmode' turned on should magically draw most of the client stuff in the correct dark colours! Switching between light and dark while the program is running seems buggy, but this is a step forward. My fingers are crossed that future Qt versions improve this feature, including for multiplatform.

Thanks to a user, the twitter downloader is fixed. The twitter team changed a tiny thing a few days ago. Not sure if it is to do with Elon or not; we'll see if they make more significant changes in future.

I fixed a crazy bug in the options when you edit a colour but find simply moving the mouse over the 'colour gradient' rectangle would act like a drag, constantly selecting. This is due to a Qt bug, but I patched it on our side. It happens if you have certain styles set under options->style, and the price of fixing the bug is I have to add a couple seconds of lag to booting and exiting a colour picker dialog. If you need to change a lot of colours, then set your style to default for a bit, where there is no lag.

next week

I pushed it hard recently, and I am due a cleanup week, so I am going to take it easy and just do some refactoring and simple fixes.

I had a great week working on a variety of small cleanup, fixes, client api, sidecars, and UI quality of life.

The release should be as normal tomorrow.

- importer/exporter sidecar expansion (boring stuff):
- split the importer/exporter objects into separate importers and exporters. existing router objects will update and split their internal objects safely
- all objects in this system can now describe themselves
- all import/export nodes now produce appropriate example texts for string processing and parsing UI test panels
- Filename Tagging Options objects no longer track neighbouring .txt file importing, and their UI removes it too. Import Folders will suck their old data on update and convert to metadata routers
- wrote a json sidecar importer that takes a parsing formula
- wrote a json sidecar exporter that takes a list of dictionary names to export to. it will edit an existing file
- wrote some ui panels to edit single file metadata migration routers
- wrote some ui panels to edit single file metadata migration importers
- wrote some ui panels to edit single file metadata migration exporters
- updated edit export folder panel to use the new UI. it was already using a full static version of the system behind the scenes; now this is exposed and editable
- updated the manual file export panel to use the new UI. it was using a half version of the system before--now the default options are updated to the new router object and you can create multiple exports
- updated import folders to use the new UI. the filename tagging options no longer handles .txt, it is now on a separate button on the import folder
- updated manual file imports to use the new UI. the 'add tags before import' window now has a 'sidecars' page tab, which lets you edit metadata routers. it updates a path preview list live with what it expects to parse
- a full suite of new unit tests now checks the router, the four import nodes, and the four export nodes thoroughly
- renamed ClientExportingMetadata to ClientMetadataMigration and moved to the metadata module. refactored the importers, exporters, and shared methods to their own files in the same module
- created a gui.metadata module for the new router and metadata import/export widgets and panels
- created a gui.exporting module for the existing export folder and manual export gui code
- reworked some of the core importer/exporter objects and inheritance in clientmetadatamigration
- updated the HDDImport object and creation pipeline to handle metadata routers (as piped from the new sidecars tab)
- when the hdd import or import folder is set to delete original files, now all defined sidecars are deleted along with the media file
- cleaned up a bunch of related metadata importer/exporter code
- cleaned import folder code
- cleaned hdd importer code

next week

I focused on sidecars this week almost exclusively, so I want to catch up on small work that piled up in the meantime. It would be nice to have some more Client API commands too, but we'll see. I can feel myself going down with something, so it might be a slim week.

I was ill this week and am short on work time. I will spend tomorrow doing some more normal work instead of the release.

505 should be on the 9th of November. Thanks everyone!

----

I did just commit changes for users who run from source. I updated the easy-setup scripts and wrote variants for Linux and macOS. My test options are limited, so if you are an advanced Linux or macOS user, please do test these new scripts and let me know where things break or are otherwise confusing. The updated help walks through it all: https://hydrusnetwork.github.io/hydrus/running_from_source.html

I have thumbnails hidden by default on new pages, I have to do "pages - management & previevs - show" every time. Any way to keep them shown?

Are you on Qt5? You just see a widely stretched left-side panel, and the thumbnail panel is 0 pixels wide on the right? I am going to investigate a bug where in various situations, the thumbnail panel is disappearing in Qt5. Please let me know if I fix your issue in v505.

I had a great couple of weeks working on a variety of features. I fixed a heap of bugs, made it possible to search for files that have EXIF and other human-readable file metadata, and updated and translated my easy setup scripts, so it is now easy for any user on any platform to run the program from source.

The release should be as normal tomorrow.

I had a great week focusing on a full-scale update of the 'neighbouring .txt' import/export system. When you import or export files, you can now send URLs as well as tags; you can use .json instead of .txt; you can transform and filter text as it passes through; and you can set up workflows to multiple files at once. Everything is presented and edited in new UI.

The release should be as normal tomorrow.

 >>/1398/
I was not familiar with what 'proxy pac' was, but it is a kind of javascript proxy configuration file? I don't support this, but if you run from source there may be a way to hack it in. I did a little search, and it looks like the main network library I use, 'requests', does not support this format easily:

https://stackoverflow.com/a/31534229

Some custom libraries referred in that same question seem able to do it. They may work well, but I do not know.

Whenever someone has a complicated proxying request, my general answer is to first see if your program supplies an OS-level or application-level wrapper, since that is always going to be easier and faster and more reliable than anything I can hack together. Do you have anything like that, where you can say 'any traffic from 'hydrus client', please wrap in this proxy'?

 >>/1399/
Thank you for this report. It looks like the media panel (the thumbnails area) is not being swapped correctly. If and when this happens again, can you please hit pages->management and preview panels->show/hide? That normally hides the panel on the left. Maybe it fixes your layout, maybe it reveals more information. It might be the media panel is there, but it is 0 pixels wide on the right. There's a 'restore' command on that same menu that might be helpful in fixing it, too. Let me know what you see. And if you can figure out what is more likely to trigger this bug, so I can try reproducing it reliably on my end, that would be helpful.

321 png
(76.41 KB, 1404x468)

 >>/1400/
 >>/1400/
> 'any traffic from 'hydrus client', please wrap in this proxy
I want any traffic from 'these sites' through these proxies, according to these rules, for all apps. I imagined solution like this.

 >>/1397/
Right-clicks in the main screen are broken (thumbnails and tags). The menu goes on top of the screen. Tags in file / tag viewer seem to work.
Testing is bit difficult because everything works on boot but breaks after using Hydrus for a while, I don't know what triggers it. If I find anything I'll let you know.

Thanks again!

 >>/1401/
Sorry to say my network engine isn't clever enough for this yet. I hope to have individual domain proxy settings in the next time I update that system. It might be we are in a position then to suck up your configuration file here and auto-populate the settings, but this may be a long time in the future.

 >>/1402/
Thanks. Sorry about the breaking after a while, keep me updated!

 >>/1400/
> Thank you for this report. It looks like the media panel (the thumbnails area) is not being swapped correctly. If and when this happens again, can you please hit pages->management and preview panels->show/hide? That normally hides the panel on the left. Maybe it fixes your layout, maybe it reveals more information. It might be the media panel is there, but it is 0 pixels wide on the right. There's a 'restore' command on that same menu that might be helpful in fixing it, too. Let me know what you see. And if you can figure out what is more likely to trigger this bug, so I can try reproducing it reliably on my end, that would be helpful.
It took awhile but I waited long enough for it to happen again, its still pretty rare. I used the show/hide option in the pages menu and it fixed it or at least made it go back to its normal page. Nothing else out of the ordinary. Searches and images load fine.

full list

- autocomplete dropdown:
- the floating version of the autocomplete dropdown gets the same backend treatment the media hovers and the popup toaster recently received--it is no longer its own window, but now a normal widget floating inside its parent. it should look pretty much the same, but a variety of bugs are eliminated. clients with many search pages open now only have one top level window, rather than potentially hundreds of hidden ones
- if you have turned off floating a/c windows because of graphical bugs, please try turning them back on today. the checkbox is under _options->search_.
- as an additional consequence, I have decided to no longer allow 'floating' autocomplete windows in dialogs. I never liked how this worked or looked, overlapping the apply/cancel buttons, and it is not technically possible to make this work with the new tech, so they are always embedded in dialogs now. the related checkbox in _options->search_ is gone as a result
- if you ok or cancel on the 'OR' buttons, focus is now preserved back to the dropdown
- a bunch of weird interwindow-focus-juggling and 'what happens if the user's window manager allows them to close a floating a/c dropdown'-style code is cleared out. with simpler logic, some flicker jank is simply eliminated
- if you move the window around, any displaying floating a/c dropdowns now glide along with them; previously it updated at 10fps
- the way the client swaps a new thumbnail grid in when results are loaded or dismissed is faster and more atomic. there is less focus-cludge, and as a result the autocomplete is better at retaining focus and staying displayed as changes to the search state occur
- the way scroll events are caught is also improved, so the floating dropdown should fix its position on scroll more smoothly and capably
- .
- date system predicates:
- _this affects system:import time; :modified time; and :last viewed_
- updated the system:time UI for time delta so you are choosing 'before', 'since', and '+/- 15% of'
- updated the system:time UI for calendar date so you are choosing 'before', 'since', 'the day of', and '+/- a month of' rather than the ugly and awkward '<' stuff
- updated the calendar calculations with calendar time-based system predicates, so '~=' operator now does plus or minus one month to the same calendar day, no matter how many days were in that month (previously it did +/- 30 days)
- the system predicate parser now reassigns the '=' in a given 'system:time_type = time_delta' to '~='

- misc:
- 'sort files by import time' now sorts files correctly even when two files were imported in the same second. thanks to the user who thought of the solution here!
- the 'recent' system predicates you see listed in the 'flesh out system pred' dialogs now have a 'X' button that lets you remove them from the recent/favourites
- fixed the crash that I disabled some code for last week and reactivated the code. the collect-by dropdown is back to refreshing itself whenever you change the settings in _options->sort/collect_. furthermore, this guy now spams less behind the scenes, only reinitialising if there are actual changes to the sort/collect settings
- brushed up some network content-range checking logic. this data is tracked better, and now any time a given 206 range response has insufficient data for what its header said, this is noted in the log. it doesn't raise an error, and the network job will still try to resume from the truncated point, but let's see how widespread this is. if a server delivers _more_ data than specified, this now does raise an error
- fixed a tiny bit of logic in how the server calculates changes in sibling and parent petition counts. I am not sure if I fixed the miscount the janitors have seen
- if a janitor asks for a petition and the current petition count for that type is miscounted, leading to a 404, the server now quickly recalculates that number for the next request
- updated the system predicate parser to replace all underscores with whitespace, so it can accept system predicates that use_underscores_instead_of_whilespace. I don't _think_ this messes up any of the parsing except in an odd case where a file service might have an underscore'd name, but we'll cross that bridge if and when we get to it
- added information about 'PRAGMA quick_check;' to 'help my db is broke.txt'
- patched a unit test that would rarely fail because of random data (issue #1217)
- .
- client api:
- /get_files/search_files:
- fixed the recent bug where an empty tag input with 'search all' permission would raise an error. entering no search predicates now returns an empty list in all cases, no matter your permissions (issue #1250)
- entering invalid tags now raises a 400 error
- improved the tag permissions check. only non-wildcard tags are now tested against the filter
- updated my unit tests to catch these cases
- /add_tags/search_tags:
- a unit test now explicitly tests that empty autocomplete input results in no tags
- the Client API now responds with Access-Control-Max-Age=86400 on OPTIONS checks, which should reduce some CORS pre-flight spam
- client api version is now 34
- .
- misc cleanup:
- cleaned up the signalling code in the 'recent system predicate' buttons
- shuffled some page widget and layout code to make the embedded a/c dropdown work
- deleted a bunch of a/c event handling and forced layout and other garbage code
- worked on some linter warnings

next week

I think I have caught up on the biggest Qt6 crashes and other emergencies. I regret the bumps that happened over the past couple of months, but we got there in the end. Next week will probably be the last with a Qt5 release, so there will be some updates to the help for that.

I am due a 'medium size' job week, so I also want to get my teeth into something. I recently overhauled how tags in .txt files were imported/exported, and I would like to do more work there. It would be nice to have URLs and/or more than .txt file formats.

I had a good week fixing some bugs and improving how the program works as a build and from source. It is now easy to run the program from source on Windows.

The release should be as normal tomorrow.

full list

- misc:
- the Linux build gets the same 'cannot boot' setuptools version hotfix as last week's Windows build. sorry if you could not boot v500 on Linux! macOS never got the problem, I think because it uses pyoxidizer instead of pyinstaller
- fixed the error/crash when clients running with PyQt6 (rather than the default Qt6, PySide6) tried to open file or directory selection dialogs. there was a slight method name discrepancy between the two libraries in Qt6 that we had missed, and it was sufficiently core that it was causing errors and best, crashes at worst
- fixed a common crash caused after several options-saving events such as pausing/resuming subscriptions, repositories, import/export folders. thank you very much to the users who reported this, I was finally able to reproduce it an hour before the release was due. the collect control was causing the crash--its ability to update itself without a client restart is disabled for now
- unfortunately, it seems Deviant Art have locked off the API we were using to get nice data, so I am reverting the DA downloader this week to the old html parser, which nonetheless still sems to work well. I expect we'll have to revisit this when we rediscover bad nsfw support or similar--let me know how things go, and you might like to hit your DA subs and 'retry ignored'
- fixed a bad bug where manage rating dialogs that were launched on multiple files with disagreeing numerical ratings (where it shows the stars in dark grey), if okayed on that 'mixed' rating, rather than leaving them untouched, were resetting all those files back to the minimum allowed star value. I do not know when this bug came in, it is unusual, but I did do some rating state work a few weeks ago, so I am hoping it was then. I regret this and the inconvenience it has caused
- if you manually navigate while the media viewer slideshow is running, the slideshow timer now resets (e.g. if you go 'back' on an image 7 seconds into a 10 second slideshow, it will show the previous image for 10 seconds, not 3, before moving on again)
- fixed a type bug in PyQt hydrus when you tried to seek an mpv video when no file was loaded (usually happens when a seek event arrives late)
- when you drop a hydrus serialised png of assorted objects onto a multi-column list, the little error where it says 'this list does not take objects of type x' now only shows once! previously, if your png was a list of objects, it could make a separate type error for each in turn. it should now all be merged properly
- this import function also now presents a summary of how many objects were successfully imported
- updated all ui-level ipfs multihash fetching across the program. this is now a little less laggy and uses no extra db in most cases
- misc code and linter warning cleanup
- .
- tag right-click:
- the 'edit x' entry in the tag right-click menu is now moved to the 'search' submenu with the other search-changing 'exclude'/'remove' etc.. actions
- the 'edit x' entry no longer appears when you only select invertible, non-editable predicates
- if you right-click on a -negated tag, the 'search' menu's action label now says 'require samus aran' instead of the awkward 'exclude -samus aran'. it will also say the neutral 'invert selection' if things get complicated
- .
- notes logic improvements:
- if you set notes to append on conflict and the existing note already contains the new note, now no changes will be made (repeatedly parsing the same conflcting note now won't append it multiple times)
- if you set notes to rename on conflict and the note already exists on another name, now no changes will be made (i.e. repeatedly parsing the same conflicting note won't create (1), (2), (3)... rename dupes)

- client api:
- /add_tags/search_tags gets a new parameter, 'tag_display_type', which lets you either keep searching the raw 'storage' tags (as you see in edit contexts like the 'manage tags' dialog), or the prettier sibling-processed 'display' tags (as you see in read contexts like a normal file search page)
- /get_files/file_metadata now returns 'ipfs_multihashes' structure, which gives ipfs service key(s) and multihashes
- if you run /get_files/search_files with no search predicates, or with only tags that do not parse correctly so you end up with no tags, the search now returns nothing, rather than system:everything. I will likely make this call raise errors on bad tags in future
- the client api help is updated to talk about these
- there's also unit tests for them
- client api version is now 33
- .
- popup messages:
- the background workings of the popup toaster are rewritten. it looks the same, but instead of technically being its own window, it is now embedded into the main gui as a raised widget. this should clear up a whole heap of jank this window has caused over the years. for instance, in some OSes/Window Managers, when a new subscription popup appeared, the main window would activate and steal focus. this annoying thing should, fingers crossed, no longer happen
- I have significantly rewritten the layout routine of the popup toaster. beyond a general iteration of code cleanup, popup messages should size their width more sensibly, expand to available space, and retract better after needing to grow wide
- unfortunately, some layout jank does remain, mostly in popup messages that change height significantly, like error tracebacks. they can sometimes take two frames to resize correctly, which can look flickery. I am still doing something 'bad' here, in Qt terms, and have to hack part of the layout update routine. let me know what else breaks for you, and I will revisit this in future
- the 'BUGFIX: Hide the popup toaster when the main gui is minimised/loses focus' checkboxes under _options->popups_ are retired. since the toaster is now embedded into the main gui just like any search page, these issues no longer apply. I am leaving the two 'freeze the popup toaster' checkboxes in place, just so we can play around with some virtual desktop issues I know some users are having, but they may soon go too
- the popup toaster components are updated to use Qt signals rather than borked object callables
- as a side thing, the popup toaster can no longer grow taller than the main window size

next week

More small job catchup, and I'll try and clear some github bug reports. Let me know how the crashes are going, but my fingers crossed.

hydrus png
(61.15 KB, 527x885)

The popups now work better on Wayland but there are still some issues. The tag roster is misaligned and the right click menu spawns way above the cursor.

Anyway keep up the good work and don't burn yourself out!

Thanks. I am glad we are moving forward. I am now planning to do to the tag dropdown the same thing I did last week to the popup toaster. Having it as a separate window is too buggy in edge cases, and it doesn't necessarily need to be one, so I will rewrite its backend and embed it. Not sure I can fit it into this week, but I want to get this done.

I had a good week working on a variety of bug fixes and UI quality of life. The tag autocomplete dropdown gets an important de-janking overhaul, and the system:time search predicates are easier to work with.

The release should be as normal tomorrow.

full list

- crashes:
- I messed the mpv update up in v499. my golden rule is never to put out bleeding-edge library updates, but without thinking I gave everyone a dll from late august. it turns out this thing was pretty crashy, and many users were getting other unusual behaviour as well. it seems like people on very new versions of Windows were mostly ok, but a little instability, whereas some older-Windows users were unable to start the client or could boot but couldn't load mpv at all. these latter cases were plagued with other problems. thanks to user help, we discovered it was the newer mpv dll causing all the problems, and an older one, from early May, seems to be fine
- so, I am rolling back the mpv in the windows releases. the 'v3' 2022-08-29 I bundled in 499 was causing several users serious problems, possibly because of the advanced 'v3' chipset instructions or related advanced compiler tech. for the Qt6 release, we are going back to 2022-05-01, which several users report as stable, and for the Qt5 we are rolling back to the 498 version, 2021-02-28, which is back to mpv-1.dll. Since Qt5 users are increasingly going to be Win 7, we'll go super safe. THEREFORE, Qt5 extract users will want to perform a clean install this week: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
- (you can alternately just delete the now-surplus mpv-2.dll in your install directory, but a full clean install is good to do from time to time, so may as well)
- updated the sqlite dll in the windows release to 2022-05, and the exe in the db directory to 2022-09
- rewrote how some internal MPV events are signalled to Qt. they now have their own clean custom event types rather than piggy-backing on some bad old hydrus pubsub code
- I either fixed a rare boot crash related to the popup messaging system, maybe exclusively on macOS, or I improved it and we'll get a richer error now
- .
- tag sibling search:
- if you search explicitly for a tag that has a better sibling (one way this can happen is when loading up an old favourite search), the client will now auto-convert that tag to the ideal in the search code and give you results for the siblinged tag
- this started off as a predicted five minute thing and spilled out into a multi-hour saga of me realising some tag sibling search code was A) wrong in edge cases and B) slow in edge cases. I have subtly reshaped how core file-tag search works in the client so that it consults each tag service in turn based on its siblings and its mappings, rather than mixing them together. this does not matter for 99.98% of cases, but if you have some weird overlapping siblings across different services, you should now get the correct results. also, some optimisations are more effective, so any instance of searching for tags on small tag services on 'all known tags' is now a bit quicker
- big brain: please note the logic here is complex, and I have not yet updated autocomplete counting to handle this situation. if you type 'cat' and get 'cat (3)' from the three 'cat' tags on 'my tags', but 'cat' is siblinged to 'species:feline' on a big service like the PTR, it will still say (3), rather than (403) or whatever from the auto-corrected PTR results. I have a plan to fix this in a future cleanup round

- tag subtags and namespace wildcards:
- searching for 'samus aran' no longer delivers files that have 'character:samus aran'. the subtag->namespace logic no longer applies. this was a fun idea from the very start of the program, but it was never all that useful as default behaviour and added several headaches, now eliminated. if you wish to perform this search going forward, please enter '*:samus aran', which is now an acceptable wildcard input
- tag lookup is unaffected. typing 'samus aran' will still provide 'character:samus aran' as a tag to choose from
- a heap of rinky-dink counting logic went along with this, such as providing tag search results like ('character:samus aran (100)', 'samus aran (100-105)'), where it tried to predict how many results would come with the unnamespaced search. this no longer exists, and a decent bit of CPU is now saved in any large tag search
- wildcard searching works on similar rules now, so if you enter 'sa*s ar', you will see 'character:samus aran' as a result in the tag list, but searching for it will not give results with 'character:samus aran'. again, enter '*:sa*s ar*' to search for all namespaces (which is now provided as a quick suggestion any time you enter an unnamespaced wildcard), or enter 'character:sa*s ar*' explicitly
- 'system:tag as number' also now follows similar rules, so if you leave the namespace field blank, it will search unnamespaced numbers. it now supports namespace wildcards, so you can enter '*' to get the old behaviour. the placeholder text on the namespace input now states this
- 'system:number of tags' now uses the same UI as 'system:tag as number', where you enter '*' as the namespace to mean all namespaces, rather than checking a box
- .
- misc:
- all tag, namespace, and wildcard search predicates are now properly editable from the active search box. shift+double-click or select from the right-click menu, and you now get a simple text input alongside any system predicate panels. previously, this would only offer you a button to invert the tag to -tag and _vice versa_. now, you can add or remove the '-' and '*' characters yourself info to freely convert between tags, namespace:anything, and wildcard search predicates (issue #1235)
- thanks to a user, you can now add '{#}' to an export filename pattern to get the '#' column in your filename (useful if you want to export files in the order they are currently in on the page)
- furthermore, if you delete items from the manual file export window, the '#' column now recalculates itself to stay contiguous and in order (previously, it left gaps)
- fixed a bug when deleting siblings on a local tags service. sorry for the trouble!
- on manage siblings, when you remove, add, or replace a pair on a local tags service, you will now get a simple 'note' reason informing you more on what is going on. the 'REPLACEMENT:' thing recently added to tag repositories should now work for you too
- when a downloader or similar adds files to a page, and you have at least one existing file selected, the status bar now updates correctly

- fixed a critical issue that was affecting some users with damaged similar file search trees. when starting similar file search tree rebalancing maintenence, their client would go into an infinite loop and spool the cyclic branch into an ever-growing journal file in their temp directory until their system drive briefly ran out of space. sorry for the trouble, and thank you for the excellent reports that helped to figure this out (issue #1239)
- the similar files search tree rebalance maintenance now detects more sorts of damaged trees and handles them gracefully, and the full tree regeneration clears out any damaged maintenance information too
- fixed another problem with the tree branch maintenance system when the root was accidentally queued for branch rebalance
- when you right-click->copy a wildcard search tag, it now copies the actual wildcard text, not the display text with (wildcard search) over the top
- I added ',' to the list of non-decodable characters in the hacky URL Class encoding/decoding routine. sites that use an encoded comma (%_2C) for regular path components or query parameters should now work
- a user has fixed a regex parsing problem in the predicate parser for system:hash
- OR search predicates now sort their sub-predicates on construction/editing, meaning the label is always of set order, and they can now compare with and hence reliably nullify each other
- the manage logins dialog now boots a little taller
- the main gui tab bar may look a bit nicer/more appropriate in macOS
- updated the help text on gui pages where it talks about overflowing rows of tabs, which auto-scroll even worse in Qt6, hooray
- .
- client api:
- the  client api now handles request disconnects better. the hydrus server code benefits from the same engine improvements
- the 'twisted.internet.defer.CancelledError' logspam is cleaned up!
- if a client disconnects before a client api autocomplete tag search or a file search is complete, that database job is now cancelled quickly just like when you type new characters in the client UI or stop a slow search
- if you are a client api dev, please let me know how this works out IRL. I'm not 100% sure what a 'disconnect' means in this context, but if you want to develope autocomplete quick lookup as the user types, and you have a way clientside to cancel/kill an ongoing request before it is complete, please give it a go and let me know if this all works. cancelled requests don't make a log record right now, but you should see the client's db lock free up instantly. at the very least, I have the proper infrastructure for this now, so I can add more/better 'cancel' hooks as we need them
- .
- uninteresting code cleanup:
- refactored the file note mapping db code to a new module
- refactored the file service pathing db code (this does directory structures and multihashes for ipfs) to a new module
- refactored some tag display, tag filtering, and tag autocomplete calls down to appropriate db modules
- refactored and extended some tag sibling database methods and names to clarify whether they were working with ids or strings

next week

I was not able to get to many things I wanted to this week. Things have piled up, so I'm just a bit buried right now. I will just continue working on urgent issues and smaller issues and see how we are on the crashing.

I'm stressed about my Dad, more than anything because there has just been a ton of energy-draining stuff to do, but not as upset as I thought. As I said before, we had a great relationship, so there are no huge regrets. I'm sure it will kick in more in a couple months. Since the hydrus userbase trends young and my Dad died a bit early, I don't expect many of you have organised a funeral. My serious advice is A) talk to your parents now about what they want in a funeral, and B) make sure they have a will. We were good on both fronts, and it has made the whole thing so much easier. Almost all children bury their parents, so get it done now, while it is easy.

As for hydrus, getting out 500 versions is pretty cool. I've been at this for ten years, and the codebase is now 10 MB over almost 300 files. I still enjoy working on it, and I want to keep at it as long as there is interesting stuff to do. The hydrus userbase has grown significantly this year, and my todo list is overflowing more than ever, so running short on work is not a worry. Handling stress and burnout has been tricky at times, but assuming I stay healthy on that front, I can comfortably see 750 in the distance. Let's see what machine learning does to us all over the next five years.

Thanks everyone!

I had an ok week. I cleared out some bugs, upgraded the popup toaster, and improved some UI quality of life.

The release should be as normal tomorrow.

 >>/1361/
Yeah, Cloudflare 403s are a pain. I am not sure about Tor and Cloudflare, that may just be a step too far in complexity, but the 'normal' way to get through a 403 Cloudflare block is to visit the site in your browser (which does the clever problem solving) and then use Hydrus Companion to copy the solved cookies over to Hydrus.

https://gitgud.io/prkc/hydrus-companion

However, this may not work with Tor. Tor probably sets its own User-Agent and I guess is using different IP addresses regularly, and those are the two things that need to be the same as your browser for the Hydrus Companion trick to work. Maybe if you did the work in the Tor browser, but then copying cookies across may be even more difficult.

That said, some sites are just strict when it comes to Cloudflare rules, so if your VPN or whatever network solution is frowned upon by Cloudflare this week, then you'll just have trouble with that site. My own answer is mostly just to navigate away from those sites and stick to what I know works for me. I'm already overloaded with stuff to process, so no point spending 50% effort trying to fetch 5% more stuff that I missed.

For updating, no worries, the normal method is to extract the release on top of your old install folder. The extract straight from the zip file has no 'database' files, so it extracts safely over what you already have without overwriting anything user-created. Please check the help here:

https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#updating

I personally make a backup, extract the fresh release on top, and boot. My update code will investigate your database, notice it is an older version, and apply several rules to update it in place. You will keep the same session and everything as when you last closed the client. The idea of the backup at that time is that if the update fails for some reason (rare, but it happens to a handful of users a few times a year), or more likely I messed something up, I can roll back immediately to where I was, no danger.

If you update using the normal routine and keep your old database, it will remember all its old known urls and be able to skip repeated downloads.

Let me know if you have any trouble with it!

 >>/1367/
 >>/1368/
Sorry for the trouble with the new release. Some users are getting several crashes, and I think the new mpv is the culprit. Since you are on Windows 7, I suspect the new mpv is incompatible with it in the same way things like Qt6 are too. Can you try going in to your install directory and deleting the 'mpv-2.dll' file? You should still have mpv-1.dll from a previous install, and that should work on its own.

If that still gives you trouble, can you try using the client as normal but avoid:

A) mpv (so don't load up any movies into the preview or media viewer)
B) the duplicate filter (I have some reports this is giving trouble too)

If the program does not crash until one of these is loaded, that is good evidence that it is mpv after all. Please avoid them for now, or if you have a backup, roll back to v498, and I will work on these problems and see if I can fix some things for v500.

If you go to "install_dir/db/client (date).log", you might like to scroll down to the bottom of the one for September. Are there any meaty-looking errors in there? They might be called 'traceback'. Most crashes do not leave any log information (they break too quick), but if there are some lead-up errors in there, that would also be useful to know.

 >>/1370/
Thanks. Yeah, I am going to be careful and not try to do three things at once. I'm actually feeling better emotionally than I thought I would. It sucks, a lot, but everyone has been wonderful and neighbours and so on are stepping up to help out. Of all the ways my Dad could go, he didn't want it drawn out, so there are more than a few silver linings here. A lot of people have to see their parents waste away from cancer over years, or to Alzheimers, so we've been saved that. I expect I will start missing him more in a few months, when his absence has a proper shape to it, but a friend of mine wisely said not to make any big decisions, so I am keeping my head down and pushing on with practical help for the situation. The good news is the most urgent funeral preparation was finished last night, so now everything is lined up. Getting everything sorted needed focus and energy, but now we just have to do picking family up from airports and stuff, and the actual day.

Since most hydrus users are going to be on the young side, and most will not have lost their parents yet, the advice I am going to give from now on is:

- Talk to your parents about what they would want in a funeral
- Make sure you have a will

Having both questions sorted ahead of time has been deeply useful during this.

 >>/1371/

Thanks for the detailed reply. I'll make sure to check out Hydrus Companion and whether it works with Tor. Yes, generally I'd also just move on to a more available source, but for some things that's difficult.

I'm glad to hear that updating is that simple and the db is cleanly divided from the actual program. I set out updates since I messed up once and read about the big update situation. But I'll try that again, one version at a time to be sure and see how it goes.

 >>/1372/

True. It helps a lot to have things worked out and off your mind. I can only recommend to take some dedicated time to face the emotional trouble inside you and go through with it, no distraction or pushing it away. You'll cry like a bitch, but you will feel a lot better afterwards. Unresolved trauma will eat you up from the inside. It's like getting a tooth pulled at the dentist, painful but so much relief. Don't worry, you're built to endure that stuff.

 >>/1372/
> Sorry for the trouble with the new release. Some users are getting several crashes, and I think the new mpv is the culprit. Since you are on Windows 7, I suspect the new mpv is incompatible with it in the same way things like Qt6 are too. Can you try going in to your install directory and deleting the 'mpv-2.dll' file? You should still have mpv-1.dll from a previous install, and that should work on its own.

Ok I removed the mpv-2.dll and it was indeed causing the problem of not being able to play animated files(gifs, webms, mp4, etc.). They play normally now and I don't get the "MPV is not available" errors anymore. 

As for the crashes I can't remember what tags I used to cause them now lol however, I feel like it was more related to mpv-2 but I'll continue to be on the lookout. The crash reports doesn't really tell you much. Just normal boots and exists and doesn't even mention it crashed other than after booting Hydrus and saying
 
> "Found and deleted the durable temporary database on boot. The last exit was probably not clean."

And I don't see any "traceback" either in the logs nor in the db folder.

I had a good week-and-a-bit getting back to schedule. I have fixed some critical bugs, including the crashy mpv problem on Windows, cleaned a bunch of code, and improved regular tag searching with new tools, cleaner logic, and automatic sibling substitution.

The release should be as normal tomorrow, probably a little early. Windows Qt5 users will have special install instructions.