/hydrus/ - Hydrus Network

tag search logic

I have updated the tag search logic in two important ways:

- First, if you give a file search a tag that currently has a better sibling (loading up old favourite searches can do this), let's say you enter 'shinji', which would now normally display as 'character:shinji', the database now recognises that this tag has a better sibling and will give you the results as if you had entered 'character:shinji'. Previously, it would give you no results since that tag 'didn't exist' any more. This sounds like a small easy thing to change, but as I peeled this system apart this week, I recognised there were several logical problems and inefficiencies with edge-case sibling search, so I rebuilt the bad parts and cleared several issues up. In any case, sibling tag search = better.

- Second, searching files for the tag 'asuka' will no longer give files that have 'character:asuka'. This 'unnamespaced search tags give all namespaced variants too' rule has been in place since the start of the program, but it has always been awkward to implement, sometimes confusing, and it makes it difficult to search for an unnamespaced tag explicitly. If you would like to search for all namespaced variants of a subtag, please enter '*:asuka', which is now acceptable input.

Note that this does not affect tag lookups. If you type 'rei', you will see 'character:rei' in the result list to choose from. But entering 'rei' will not find files with any namespaced version of the tag.

Wildcards follow the new unnamespaced rule too, now. If you enter 'm*na', you will not get files with 'character:mana'. '*:m*na' will, though, and these 'any namespace' wildcard rules are now supplied as suggestions whenever you enter an unnamespaced wildcard.

other highlights

You can now fully edit tag, namespace, and wildcard search predicates. Either shift+double-click some active tag search predicates, or right-click and select 'edit', and you can now change their text. You can also convert between one or another just by typing.

Thanks to a user, the 'pattern' you use when declaring export filenames now supports '{#}', which will give you the same as the incrementing number '#' column in the manual export dialog. You can now export files and give them a filename based on their current thumbnail order.

full list

- crashes:
- I messed the mpv update up in v499. my golden rule is never to put out bleeding-edge library updates, but without thinking I gave everyone a dll from late august. it turns out this thing was pretty crashy, and many users were getting other unusual behaviour as well. it seems like people on very new versions of Windows were mostly ok, but a little instability, whereas some older-Windows users were unable to start the client or could boot but couldn't load mpv at all. these latter cases were plagued with other problems. thanks to user help, we discovered it was the newer mpv dll causing all the problems, and an older one, from early May, seems to be fine
- so, I am rolling back the mpv in the windows releases. the 'v3' 2022-08-29 I bundled in 499 was causing several users serious problems, possibly because of the advanced 'v3' chipset instructions or related advanced compiler tech. for the Qt6 release, we are going back to 2022-05-01, which several users report as stable, and for the Qt5 we are rolling back to the 498 version, 2021-02-28, which is back to mpv-1.dll. Since Qt5 users are increasingly going to be Win 7, we'll go super safe. THEREFORE, Qt5 extract users will want to perform a clean install this week: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
- (you can alternately just delete the now-surplus mpv-2.dll in your install directory, but a full clean install is good to do from time to time, so may as well)
- updated the sqlite dll in the windows release to 2022-05, and the exe in the db directory to 2022-09
- rewrote how some internal MPV events are signalled to Qt. they now have their own clean custom event types rather than piggy-backing on some bad old hydrus pubsub code
- I either fixed a rare boot crash related to the popup messaging system, maybe exclusively on macOS, or I improved it and we'll get a richer error now
- .
- tag sibling search:
- if you search explicitly for a tag that has a better sibling (one way this can happen is when loading up an old favourite search), the client will now auto-convert that tag to the ideal in the search code and give you results for the siblinged tag
- this started off as a predicted five minute thing and spilled out into a multi-hour saga of me realising some tag sibling search code was A) wrong in edge cases and B) slow in edge cases. I have subtly reshaped how core file-tag search works in the client so that it consults each tag service in turn based on its siblings and its mappings, rather than mixing them together. this does not matter for 99.98% of cases, but if you have some weird overlapping siblings across different services, you should now get the correct results. also, some optimisations are more effective, so any instance of searching for tags on small tag services on 'all known tags' is now a bit quicker
- big brain: please note the logic here is complex, and I have not yet updated autocomplete counting to handle this situation. if you type 'cat' and get 'cat (3)' from the three 'cat' tags on 'my tags', but 'cat' is siblinged to 'species:feline' on a big service like the PTR, it will still say (3), rather than (403) or whatever from the auto-corrected PTR results. I have a plan to fix this in a future cleanup round

- tag subtags and namespace wildcards:
- searching for 'samus aran' no longer delivers files that have 'character:samus aran'. the subtag->namespace logic no longer applies. this was a fun idea from the very start of the program, but it was never all that useful as default behaviour and added several headaches, now eliminated. if you wish to perform this search going forward, please enter '*:samus aran', which is now an acceptable wildcard input
- tag lookup is unaffected. typing 'samus aran' will still provide 'character:samus aran' as a tag to choose from
- a heap of rinky-dink counting logic went along with this, such as providing tag search results like ('character:samus aran (100)', 'samus aran (100-105)'), where it tried to predict how many results would come with the unnamespaced search. this no longer exists, and a decent bit of CPU is now saved in any large tag search
- wildcard searching works on similar rules now, so if you enter 'sa*s ar', you will see 'character:samus aran' as a result in the tag list, but searching for it will not give results with 'character:samus aran'. again, enter '*:sa*s ar*' to search for all namespaces (which is now provided as a quick suggestion any time you enter an unnamespaced wildcard), or enter 'character:sa*s ar*' explicitly
- 'system:tag as number' also now follows similar rules, so if you leave the namespace field blank, it will search unnamespaced numbers. it now supports namespace wildcards, so you can enter '*' to get the old behaviour. the placeholder text on the namespace input now states this
- 'system:number of tags' now uses the same UI as 'system:tag as number', where you enter '*' as the namespace to mean all namespaces, rather than checking a box
- .
- misc:
- all tag, namespace, and wildcard search predicates are now properly editable from the active search box. shift+double-click or select from the right-click menu, and you now get a simple text input alongside any system predicate panels. previously, this would only offer you a button to invert the tag to -tag and _vice versa_. now, you can add or remove the '-' and '*' characters yourself info to freely convert between tags, namespace:anything, and wildcard search predicates (issue #1235)
- thanks to a user, you can now add '{#}' to an export filename pattern to get the '#' column in your filename (useful if you want to export files in the order they are currently in on the page)
- furthermore, if you delete items from the manual file export window, the '#' column now recalculates itself to stay contiguous and in order (previously, it left gaps)
- fixed a bug when deleting siblings on a local tags service. sorry for the trouble!
- on manage siblings, when you remove, add, or replace a pair on a local tags service, you will now get a simple 'note' reason informing you more on what is going on. the 'REPLACEMENT:' thing recently added to tag repositories should now work for you too
- when a downloader or similar adds files to a page, and you have at least one existing file selected, the status bar now updates correctly

- fixed a critical issue that was affecting some users with damaged similar file search trees. when starting similar file search tree rebalancing maintenence, their client would go into an infinite loop and spool the cyclic branch into an ever-growing journal file in their temp directory until their system drive briefly ran out of space. sorry for the trouble, and thank you for the excellent reports that helped to figure this out (issue #1239)
- the similar files search tree rebalance maintenance now detects more sorts of damaged trees and handles them gracefully, and the full tree regeneration clears out any damaged maintenance information too
- fixed another problem with the tree branch maintenance system when the root was accidentally queued for branch rebalance
- when you right-click->copy a wildcard search tag, it now copies the actual wildcard text, not the display text with (wildcard search) over the top
- I added ',' to the list of non-decodable characters in the hacky URL Class encoding/decoding routine. sites that use an encoded comma (%_2C) for regular path components or query parameters should now work
- a user has fixed a regex parsing problem in the predicate parser for system:hash
- OR search predicates now sort their sub-predicates on construction/editing, meaning the label is always of set order, and they can now compare with and hence reliably nullify each other
- the manage logins dialog now boots a little taller
- the main gui tab bar may look a bit nicer/more appropriate in macOS
- updated the help text on gui pages where it talks about overflowing rows of tabs, which auto-scroll even worse in Qt6, hooray
- .
- client api:
- the  client api now handles request disconnects better. the hydrus server code benefits from the same engine improvements
- the 'twisted.internet.defer.CancelledError' logspam is cleaned up!
- if a client disconnects before a client api autocomplete tag search or a file search is complete, that database job is now cancelled quickly just like when you type new characters in the client UI or stop a slow search
- if you are a client api dev, please let me know how this works out IRL. I'm not 100% sure what a 'disconnect' means in this context, but if you want to develope autocomplete quick lookup as the user types, and you have a way clientside to cancel/kill an ongoing request before it is complete, please give it a go and let me know if this all works. cancelled requests don't make a log record right now, but you should see the client's db lock free up instantly. at the very least, I have the proper infrastructure for this now, so I can add more/better 'cancel' hooks as we need them
- .
- uninteresting code cleanup:
- refactored the file note mapping db code to a new module
- refactored the file service pathing db code (this does directory structures and multihashes for ipfs) to a new module
- refactored some tag display, tag filtering, and tag autocomplete calls down to appropriate db modules
- refactored and extended some tag sibling database methods and names to clarify whether they were working with ids or strings

next week

I was not able to get to many things I wanted to this week. Things have piled up, so I'm just a bit buried right now. I will just continue working on urgent issues and smaller issues and see how we are on the crashing.

I'm stressed about my Dad, more than anything because there has just been a ton of energy-draining stuff to do, but not as upset as I thought. As I said before, we had a great relationship, so there are no huge regrets. I'm sure it will kick in more in a couple months. Since the hydrus userbase trends young and my Dad died a bit early, I don't expect many of you have organised a funeral. My serious advice is A) talk to your parents now about what they want in a funeral, and B) make sure they have a will. We were good on both fronts, and it has made the whole thing so much easier. Almost all children bury their parents, so get it done now, while it is easy.

As for hydrus, getting out 500 versions is pretty cool. I've been at this for ten years, and the codebase is now 10 MB over almost 300 files. I still enjoy working on it, and I want to keep at it as long as there is interesting stuff to do. The hydrus userbase has grown significantly this year, and my todo list is overflowing more than ever, so running short on work is not a worry. Handling stress and burnout has been tricky at times, but assuming I stay healthy on that front, I can comfortably see 750 in the distance. Let's see what machine learning does to us all over the next five years.

Thanks everyone!

I had an ok week. I cleared out some bugs, upgraded the popup toaster, and improved some UI quality of life.

The release should be as normal tomorrow.