/hydrus/ - Hydrus Network

https://youtube.com/watch?v=eYN9WivpQ6M
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v590/Hydrus.Network.590.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v590/Hydrus.Network.590.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v590/Hydrus.Network.590.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v590/Hydrus.Network.590.-.Linux.-.Executable.tar.zst

I had a pretty good week and have a bunch of quality of life improvements to roll out.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The 'check now' button in 'edit subscriptions' and 'edit subscription' is now more intelligent and handles pause status. It'll ask you about paused subscriptions and queries, and it better examines and navigates what you can and will be resurrecting.

If you shrink the search page's 'preview' window down to 0 pixel size, it now recognises it is hidden and will no longer load videos and stuff in the background!

In the parsing system, the 'character set' choice for a 'String Match' now includes the quick-select options for hexadecimal and base64 chars.

When hitting share->copy hashes on a thumbnail, the program now loads up the md5, sha1, and sha512 hashes into the menu labels, so you can now verify md5s or whatever at a glance.

I wrote a thing for API devs and any other advanced users interested in hydrus content updates about how the Current Deleted Pending Petitioned model works in hydrus: https://hydrusnetwork.github.io/hydrus/developer_api.html#CDPP . I went into very specific detail because when talking about this with a user the other day, I couldn't remember it perfectly myself.

Lastly, I am happy to say that I succeeded in doing, as I planned, some meaty background work. The 'Metadata Conditional' object, which does 'does this file have x property, yes/no?', and which I have been thinking about for a couple years, got its first version, and it went a lot better and simpler than I expected. The whole thing plugs into the existing system predicate system and will share much edit UI with it. The MC will be a key lego brick in the development of the duplicate auto-resolution system, and most of the comparison logic for the first version is essentially done now. I pretty much just have to write the database tables and maintenance daemon, and then some UI, so fingers crossed it won't be too long before I am rolling it out for advanced users to play with.

next week

I want to add local media file path fetching to the API and do a bit of API permissions cleanup.

https://youtube.com/watch?v=Ka4pfP2z8iA
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v589/Hydrus.Network.589.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v589/Hydrus.Network.589.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v589/Hydrus.Network.589.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v589/Hydrus.Network.589.-.Linux.-.Executable.tar.zst

I had a good couple of weeks mostly cleaning code and optimising things.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

If you have a lot of files, your database update may take a couple minutes this week. Some users have had unoptimised similar-files search for a while, and this fixes it.

If you have a lot of tag siblings and parents (e.g. if you sync with the PTR), I have reduced the lag around sibling/parent processing significantly. There's a cache that previously had to be regenerated on every sibling/parent change, and for the PTR that could take 1.3 seconds, now it just updates in a couple milliseconds. If I have been working with you on lag issues, and we discovered that shutting off 'normal time' sibling/parent work fixed it, try turning it back on this week and let me know how it goes. If you do prefer manual control of this, there's also a new 'sync now' menu entry under tags->sibling/parent sync that basically slams all the 'work hard' buttons and functions as a catch-all 'do all outstanding work now'.

A bunch of big lists like the 'file log' should now initialise and sort a little bit faster, particularly when you are pushing 10,000+ items.

I fixed the 'page chooser' dialog's 3x3 layout, which was accidentally un-done in a recent layout overhaul. This thing is stupid, but I'm still fond of it. There's also some new checkboxes under options>pages that govern whether 'my files' and the more advanced 'local files' will show up in the 'files' choice.

The safebooru parser should stop grabbing '?' tags, and catbox collection URLs are now parseable.

next week

I want to get some meaty work done, either chipping away at dynamic file storage or duplicate auto-resolution.

https://youtube.com/watch?v=X7OpjB_8sHQ
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v588/Hydrus.Network.588.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v588/Hydrus.Network.588.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v588/Hydrus.Network.588.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v588/Hydrus.Network.588.-.Linux.-.Executable.tar.zst

I had an ok week. Multi-column lists work faster across the program.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I finished my list rewrite. Multi-column lists look and work exactly as they did before, but they initialise and sort faster. I still have some optimisation to do, but my test list of 170,000 items now sorts in about four seconds. More generally, many normal delete and insert events should have just a little less lag. I hope this makes dealing with large file logs and so on a bit less of a hassle!

Otherwise, I fixed some visual bugs and cleaned up some similar files maintenance code.

next week

I want to optimise some db maintenance code and otherwise just do some simple cleanup.

https://youtube.com/watch?v=czrmBIHANV4
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v587/Hydrus.Network.587.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v587/Hydrus.Network.587.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v587/Hydrus.Network.587.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v587/Hydrus.Network.587.-.Linux.-.Executable.tar.zst

I had an ok week. I didn't have time to finish my big list rewrite, so I'm just rolling out some little jobs today.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I made another stupid typo last week, breaking the tags->manage tag display and search dialog! Fixed now, sorry for the trouble.

The top-right media viewer hover now shows all the local file services a file is in. For most users this will just be 'my files', but if you use multiple file services, I hope this will be a bit cleaner than the spammy labels I removed from the top hover the other week. I think I'm ultimately going to make these into buttons or add checkboxes or something so we can have one-click local service migrations in future.

I cleaned up the code that handles the two resizable splitters/sashes that separate the normal media page sidebar from the preview canvas and the main thumb grid. There was some ugly stuff in there, and I think I have fixed some odd layout problems certain window managers had. That said, this stuff can be temperamental, so if you are on a weird OS and your pages suddenly lay out crazy, please roll back to your v586 backup and let me know.

next week

I will finish this list rewrite. I have all the code pretty much done, and I feel good about it, but I need to do a ton of testing and polish. It should let us view huge lists with far less UI lag.

https://youtube.com/watch?v=CVAaLlOUD00
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v586a/Hydrus.Network.586a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v586a/Hydrus.Network.586a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v586a/Hydrus.Network.586a.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v586a/Hydrus.Network.586a.-.Linux.-.Executable.tar.zst

Hey, I did a hotfix to fix a stupid bug when moving from videos to images. If you got the release within twenty minutes of this post going live, get the updated v586a above!

I had a great week getting siblings and parents lookups running faster and finishing some long-planned Client API work. The update may take a minute this week!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I am happy with the new siblings and parents dialogs, but unfortunately the fetch jobs were frequently running super slow on the PTR. This has been a long time problem in other sibling/parent places, and I could never really figure it out. This exposed the problem better and I simply put a bunch of time into the database sibling/parent storage structure and search code this week, and I think/hope I have fixed the worst of it. I also fixed the crazy-long lag spikes we were seeing, which was, unfortunately, just me being stupid last week. If you sync with the PTR (or not!) and have had slow sibling/parent lookups (including in places like the tag autocomplete results list), let me know how it goes!

If you have the media scanbar set to hide completely when the mouse is not over it, I think I fixed the issue where it would come up blank if the media was paused while the scanbar was hidden!

The options widgets that are an editable number with a checkbox beside saying 'no limit' are now initialised with a nicer default number when they start with 'no limit' checked. Previously, this stuff was all initialising to 1 every time, which wasn't always helpful if you actually wanted to go edit it. I'm pretty sure all the 'noneable' integer widgets in the options dialog now soft-initialise to the actual defaults those numbers are on a fresh client.

If you use multiple local file services, then when you middle-click a tag in the media viewer, the new search page now correctly retains the original file domain of the media viewer. Although I use multiple local file services myself on my IRL client, I do not browse around in them all that much, so let me know where else this sort of stuff defaults to 'all local files' or 'all my files'.

I have removed a hard limit that said 'don't run an import folder for more than an hour'. If you have a mega import folder with hundreds of thousands of files, let's see how it goes.

If you have done Client API or tag migration work and ended up with some bizarre tags that are both pending (+1) and petitioned (-1) to the same service, check the changelog this week!

client api

Thanks to a user, the Client API call that renders images can now output in jpeg and webp, can change the quality of the output, and will render to a target resolution!

I also think I finally finished off the first full version of 'multiple local file services' Client API support. You can now set a custom import destination for the 'add file' and 'add URL' commands, and you can now copy files from one local file service to another using a new 'migrate_files' call.

next week

I've been working on several things recently that can populate a multi-column list with a hundred thousand or more rows, and it has reminded me that my core list code relies on an old hack in it that makes initialising and sorting such big lists super laggy. I have researched how to improve it and hope to do so next week!

Unfortunately, I do feel myself going down with something, so it might be delayed.

https://youtube.com/watch?v=LREOmHLII70
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v585/Hydrus.Network.585.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v585/Hydrus.Network.585.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v585/Hydrus.Network.585.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v585/Hydrus.Network.585.-.Linux.-.Executable.tar.zst

I had a great couple of weeks getting the tag siblings and parents dialogs to load quickly.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

fast siblings and parents

The PTR has been a painful success. It is great, and I am grateful for how it keeps growing, but every time we add ten thousand or a hundred thousand new things somewhere, it lags out some bit of UI where I never thought it would be a problem. Anyone who has tried to work with PTR siblings or parents knows what I am talking about--it can take five or ten seconds, every single time, to load the manage tag siblings/parents dialogs. The same is true for anyone who has programmatically imported siblings from a booru--adding 100,000 pairs can be neat, but editing them manually is then a constant frustration.

So! I have rewritten both dialogs, and the long and short of it is they now load instantly. If they need to review some pairs to do logic (like 'hey, does this new pair the user wants to add conflict with the existing pair structure?'), it all happens in the background, usually so quickly you never notice it. We'll see how this actually works on IRL situations, but I do feel good about all this work. There was a lot to do, but it ultimately seemed to go well, enough that I had time for some bells and whistles.

Beyond some optimisations and loop-detection fixes, there's a workflow change in that these two dialogs now have a 'stickier' workspace. The list of pairs has typically shown anything related to what you have waiting to be added, and I have now extended that to say 'and they now stay in view after you add'. Whenever you type in a tag, everything related to that tag is loaded up and stays in view while you work on other things. If you want to clear the workspace, you can just click a button to reset. I hope this makes it easier to edit and even merge larger sibling groups.

This is all a big change, and I'm sure I've messed up somewhere. If you do siblings or parents a lot, give it all a go and let me know how it works out. The PTR really is huge, and some larger groups may still take a second or two to load--we'll see.

other highlights

Hitting escape now deselects any taglist!

options->media viewer gets a new 'Do not allow mouse media drag-panning when the media has duration' checkbox. If you often misclick when scrubbing, try it out.

I pared down the spammy 'added to x service 3 days ago' lines in the media viewer's top hover. It now pretty much just says 'imported, modified'. If you need to see archived time or something, note that the timestamps are still available on the normal media right-click menu, on the flyout submenu off the top row.

next week

I have lots of small things to be getting on with, so I'll just catch up on my normal queue.

https://youtube.com/watch?v=L5dcODmclFU
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v584/Hydrus.Network.584.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v584/Hydrus.Network.584.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v584/Hydrus.Network.584.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v584/Hydrus.Network.584.-.Linux.-.Executable.tar.zst

I had an ok week working on some small jobs and new Client API commands.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I fixed a bug that was allowing wasteful file re-downloads from Pixiv and Twitter. I accidentally left a hole in the recent changes to the URL 'neighbour-testing' logic, where it tries to determine if a 'already in db/previously deleted' URL determination is trustworthy, and sites where posts can have multiple files were not able to return 'already in db' or 'previously deleted' until the file itself was redownloaded. I have filled the hole in--thank you for the reports, sorry for the trouble, and let me know if you notice anything else weird going on.

This is a weird thing, but in the same way that if you double-click a tag in the normal search page, it adds that tag to the search, if you ctrl+double-click a tag, you enter '-tag'. It was difficult to pull this off due to ctrl+single-click doing a deselect (usually it was best done with ctrl+enter on keyboard), but I've smoothed out the click selection logic and it does, a bit more, what you think it should. Give it a go!

If you ever use 'regex' 'system:url' predicates, they may run significantly faster this week if you mix them with other search predicates. regex URL preds now run absolutely last in the master file search, so they benefit if tags or other system predicates have already reduced the search space. Regex URL search is pretty much the worst-performing part of all hydrus file search, and there isn't much I can do about it, but I did a little research this week and I have a couple of ideas for the future--it might even allow for sometimes-fast regex tag search, but we'll see.

client api

Everything in the hydrus 'pending' menu (e.g. for sending mappings to the PTR) is now available on the Client API, with a new 'Commit Pending' permission needed to do it.

A new command in the file relationships set also lets you remove all the potential pairs for files.

next week

I will make the tag siblings/parents dialogs load instantly. This job keeps on getting put off, but I am determined to finally clear it, even if it takes a couple weeks.

https://youtube.com/watch?v=-KWxaOmVNBE
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v583/Hydrus.Network.583.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v583/Hydrus.Network.583.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v583/Hydrus.Network.583.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v583/Hydrus.Network.583.-.Linux.-.Executable.tar.zst

I had an ok week mostly cleaning code.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The command palette (ctrl+p by default) gets some new settings under the new options->command palette panel. You can now search for page of pages, and the menubar/media actions, which were previously hidden behind advanced mode, can now be turned on or off separately. I haven't touched this thing much since it was first submitted, but let me know what you would like it to do in future. I know it currently needs some filtering to get rid of unhelpful menu actions, so let me know what is most annoying and I'll see what I can do.

The 'manage times' dialog now lets you set timestamps to multiple domains at once, either the website domains or file services. You can just select multiple rows and hit edit, and you'll set the same time for everything you selected. You can even do a 'cascade'.

If you have weirdly rotated PNGs, try hitting manage->maintenance->regen file metadata on them. I was applying EXIF rotation incorrectly to PNGs. Also, it looks like the 'determine if the file has EXIF metadata' job may have been firing false negatives in the recent past, so if you have been using this in the past few weeks, you might want to run those jobs again.

next week

I want to do some Client API stuff, and probably just some small fixes and quality of life otherwise.

https://youtube.com/watch?v=eN3UHq6hjBc
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v582/Hydrus.Network.582.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v582/Hydrus.Network.582.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v582/Hydrus.Network.582.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v582/Hydrus.Network.582.-.Linux.-.Executable.tar.zst

Hey, if you use the macOS App and use the 'hydrus default darkmode' stylesheet, skip the release this week! I removed it by accident.

I had a great week mostly working on UI fixes and quality of life.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

sidecar UI

The sidecar UI now has a testing panel with a live preview of what is going to be sourced. If you are doing a manual import or managing an import folder, it shows a sample of the files actually about to be imported and what their sidecars will be loading, and if you are doing a manual export or managing an export folder, it shows a sample of the files in hydrus you will be exporting and what tags/urls/whatever you have selected to be exported. If you set up string processing rules, the test panel also shows these.

I hope this helps you see better that you have the sidecar metadata migration set up how you want, but it is only one step. I want to do more: more and better testing panels, and fewer ridiculously nested dialogs, and also in the general parsing and string processing UI, which suffers from the same problems. Let me know how this works for you!

stylesheet colours

The colours in options->colours are now all settable in a stylesheet! The 'colours' options panel remains, but it is now wrapped in a 'override what is set in the stylesheet?' checkbox, which is default True for existing users and False for new users. Selecting a darkmode stylesheet with the override set to False now gives you some good dark colours, instead of the ugly mix of dark and white we had before. Thanks for bearing with me as we figured this out.

If you are a stylesheet maker, please check the hydrus_default.qss to see how this works. I have pasted the internal hydrus darkmode default colours into all the default stylesheets in hydrus, so feel free to go in there and adjust these to something better for your particular stylesheet, and if you send it in I'll be happy to update the defaults.

I am not totally sure what the future of the 'colours' panel is. I suspect it will be migrated into a more dynamic system that has odd custom rules like 'if a file has >4 star rating, give it a gold border'. We'll see how this goes, and we'll see how Qt 7, which I understand has some better darkmode tech, proves itself.

next week

I'd like to take a week to focus on boring code cleanup.

https://youtube.com/watch?v=Sy5BFkcqqF8
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v581/Hydrus.Network.581.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v581/Hydrus.Network.581.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v581/Hydrus.Network.581.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v581/Hydrus.Network.581.-.Linux.-.Executable.tar.zst

I had a great week mostly working on a new tool for tag repository janitors.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

normal stuff

If you use the 'Dark Blue' stylesheet, check out the new '1.1' version this week, created by the original author.

There's a new 'shimmie' downloader, thanks to a user. It has better tag, file, and hash parsing tech. By default, for our purposes, this means r34h and r34@paheal downloaders work better and more efficiently.

I wrote some new tech this week to scan the database for tag filters more efficiently. This now works in the 'migrate tags' system. If you try to do a 'migrate series: tags' or similar, the setup time for that job should now just be a couple of seconds, whereas previously for something like the PTR it could be several minutes.

janitor stuff

There's a new 'admin' menu on all tag right-clicks.

If you have 'change options' permission, you will see which files are blocked or not blocked and can directly edit the repository's tag filter right from the menu.

If you have 'resolve mappings petition' permission, you will see a new 'purge tags' command. This launches a new window that lets you select certain tags to be completely deleted from the repository. No more searching in 'all known files' and then doing a laggy ctrl+a->f3 thing--you can just search up the tags and then fire off the job and it is all done in the background with a custom 'migrate tags' job that queues up the petitions for you, usually in a few seconds. Take care, and let me know how it works out!

Also, there's a new command under services->administrate services, 'purge tag filter', which lets you do this 'purge tags' command on the current tag filter, so there's now an easy way to retroactively sync a tag repository to whatever you have blocked right now.

All this tech works clientside, no need to update the server. I expect to add some bells and whistles in future like showing the current tag count in the 'purge tags' list and offering some sort of 'replace what you delete with tag y' hard-replace tech, and I think I'll provide a similar version of 'purge tags' for normal users so they can queue up big deletes/petitions (i.e. local or repository) all the easier.

next week

I want to make the sidecar UI nicer to work with.