/hydrus/ - Hydrus Network

https://youtube.com/watch?v=EtVjGWpbE7k
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Linux.-.Executable.tar.gz

I had a mixed week, but I fixed some things, and there is more filetype support. There are a couple of database fixes this week, so anyone with a large database, including anyone who syncs or has synced with the PTR, can expect the update to take several minutes.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user, the program now supports gzip (a type of archive) and xcf (GIMP savefiles)! SVG thumbs also look correct now if you have 'fill' thumbnails. Since the 'filetype selector' widget is getting pretty stacked, I'm going to rewrite it very soon, and I think we are going to have one or more groups, maybe spitting raster/vector and grouping 'image' application types like psd, krita, and xcf, together.

I fixed an issue where even if you deleted a tag from all the files somewhere (i.e. reduced its autocomplete count to 0), it wasn't disappearing from autocomplete results. It shouldn't happen again, and one of the updates this week will be scanning through your autocomplete data to remove all existing orphans.

Otherwise it is mostly background work this week! One thing is more responsive, another is more efficient, and some UI protects against mistakes, but none of it is too interesting. There's a clever delete-lock update to the Client API if you handle a lot of advanced file deletions.

future build

There's another test build out today, here: https://github.com/hydrusnetwork/hydrus/releases/tag/v536-future-2

It is the same 536 but uses newer libraries. If you have trouble with the current version of Qt (e.g. borked high UI scale text sizes) or just want to test out newer tech, feel free to have a go. We've had some real problems with the recent crop of Qt updates, but this one looks tentatively good, so I'm very hopeful this jump will put the issues to bed. Let me know if there are any problems!

next week

I'm behind on my github issues, so I'll give them a good look, and otherwise continue pushing at my immediate backlog of smaller jobs.

https://youtube.com/watch?v=qv6yDjtIl8k
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Linux.-.Executable.tar.gz

I had a good couple weeks working on better petition-processing workflow for repository janitors. Most users won't see this, so you can ignore most of this release!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

krita and SVG

Thanks to two different users, hydrus gets Krita (.kra, .krz) and SVG (.svg) support today. Both formats also get thumbnails!

These formats can get quite complicated, so if you have one that won't import right (for Krita, it'll look like a zip; for SVG, probably HTML), please send it in and I'll have a look. Same for if the thumbnails are broken, although there's a limit to how clever we can go there.

Adding SVG also exposed that I don't have a good category/pipeline for handling vector vs raster graphics. Several things like thumbnail size calculations are different with an SVG (which has no true fixed resolution), so I will be doing several waves of background updates in future to handle this and other vectors better. Who knows, we might have beautifully zooming SVG media viewer support with one day.

petitions

Ok, this was almost all of my work this past two weeks. It kind of kicked my ass, but I'm happy it is done. It changes UI that only a handful of users see, so most users can skip over it completely.

For server admins, this new tech needs a server update, and it is not compatible with older clients (nor can newer clients talk to older servers), so please plan to update your server and janitors' clients at roughly the same time. It isn't going to break anything, but the client won't be able to do janitor work and will produce some harmless errors either way.

For janitors, I've made the petition processing page handle multiple petitions at once. There's a new panel, a multi-column list, and when you click the 'fetch petitions' button, it populates with as many petitions as you like. You can see and sort by the action (ADD, DELETE, REPLACE), account id, reason, and total size (e.g. '60 mappings in 12 petitions'), and then double-click any to highlight it in the normal petition UI you are used to.

This is about speed of processing. When you process a petition, it starts uploading in the background and the page moves immediately on to the next one. No more waiting for the server to summon another petition. Also, it gives you a better overview of your queue and even lets you mass-approve/deny a set of petitions when one account or reason keeps making good or bad stuff. You can fire off twenty petitions in one go and quickly move on to the next.

https://youtube.com/watch?v=x33quKIyVu0
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Linux.-.Executable.tar.gz

I had an ok week. There's a mix of different work and a fix to the recent macOS crashing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

all misc this week

Subscription popup windows should update far more smoothly and frequently this week. Some of the 'should I check for new text?' logic was broken when I embedded the popup toaster into the main UI a few months ago--it is fixed.

If you right-click an image that is larger than 1,024 pixels on a side, the share->copy menu now has a second 'copy bitmap' that crops the image down for easier and faster pasting in lookup services like SauceNAO.

I added some info about the not-well-documented tag/predicate mouse shortcuts (e.g. shift+double-click to edit system predicates) to the help here: https://hydrusnetwork.github.io/hydrus/getting_started_searching.html#editing_predicates

I added a link to the Client API help for a neat new tool that finds video duplicates and queues them up in the duplicate filter. Here: https://github.com/appleappleapplenanner/hydrus-video-deduplicator

Since v529, at least some users of the macOS app have had terrible crashing. Last week's attempt didn't do it, but I think we've fixed it now! It was something I never even knew about updating in the background to an incompatible version. Thank you for your patience and help testing, and let me know if you still have any trouble!

If you are an advanced downloader maker, check out the changelog regarding sub-gallery urls. Nothing huge has changed, but they'll inherit and parse tags and some other stuff from their parent parsing step now.

The experimental QtMediaPlayer now initialises with and updates according to the volume and mute settings. Previously it was stuck on 100% always. The volume bar is still hidden on the actual player, but we are moving forward.

next week

I failed to do as I planned this week. I'd hoped to set aside a good whack of time to figure out a PTR tag filtering system, but it didn't happen. I'll see if I can get the time and energy together again this week, and if not, I'll try and kick out some ratings in the Client API instead.

https://youtube.com/watch?v=eclMFa0mD1c
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Linux.-.Executable.tar.gz

The 532a is a hotfix, I messed something up with the new pixiv parser first time around.

I had a good week. I've improved stability and added some little features.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

stability

Some damaged videos were causing mpv to dump out in a way Qt was unhappy with, and it lead to black video screens that would crash after a few clicks. I have written a new error-catching system that should recognise these specific files and immediately unload them along with a warning to the user. Also, all other general errors (like missing metadata or chapter marks, which are common for our sort of files) are now printed to your log for future investigation. As we learn more, I expect to add more 'unload this immediately' and 'don't print to log, this error is nbd' hooks here in future.

The macOS release has unfortunately been unstable for some/all users since v530. Clicking on lists can simply not work and then suddenly cause a crashe. With some user help, I have learned an important potential crash in Qt and tightened up all the respective areas in hydrus code. If you are a macOS user and have had trouble, let me know how you get on with today's release. If you still get crashes from simple list interaction (e.g. in manage subscriptions), please revert to v529, which does not appear to have the problem. There have been no big database changes since then, so all you have to do is re-install v529 and you should be good to go.

The above macOS error handling improvements may have fixed some other unexplained crashes. What once caused a crash should now appear as a normal popup error. Let me know what you see.

other highlights

When you hit one of the 'open these files in a new page' commands, the new page is now   search-enabled and starts with a 'system:hashes' of the files it opened with. This lets you quickly add more search predicates to narrow down the selection!

When you right-click a tag, the 'search' menu now lets you open the tag(s) in a new duplicate filter page!

I updated the pixiv downloader again. A couple weeks ago, Pixiv decided that if you aren't logged in, you can't get the 'original' quality of an image. I updated the downloader before my vacation by telling it to get the resized version, but as this was importing low quality files silently, it was not a great idea. I have given it another go, and now, if you aren't logged in to pixiv, it sets those files as 'ignored' and the 'note' on the file log mentions you need to be logged in to get the original quality. If you want the previous 'get the medium quality' version, it is now selectable as a different parser for 'pixiv file page api' under network->downloader components->manage url class links.

next week

I'd like to sink my teeth into a medium size job. It'd be nice to finally catch up on some serverside tech for more PTR updates.

https://youtube.com/watch?v=7ACn4nCY4lQ
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Linux.-.Executable.tar.gz

NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 

I had a good week working on simple fixes and cleanup before my break.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

I also made another future build for v531 here: https://github.com/hydrusnetwork/hydrus/releases/tag/v531-future-1

all misc this week

I fixed editing favourite searches, which I accidentally broke last week!

If you right-click a tag or a file and it has a billion siblings/parents/children/known URLs, the menus now truncate the giga list. Previously, child-heavy tags like pokemon would make a menu that would laggily take up your entire screen. Now it lists about ten items and says '438 more...'.

I fixed the Pixiv parser for guest logins, but it no longer gets the 'original' quality. It seems like they hide this for non-logged-in users now. If you do a lot of Pixiv downloading, figure out a login solution with Hydrus Companion or cookies.txt imports.

If you have some downloaders that were inexplicably halted on 'waiting for a work slot', even after client restart, check them again this week--I cleaned up some bad status reporting.

The Client API gets an overhaul to service info this week. '/get_services' and '/get_files/file_metadata' are affected. If you do Client API work, please check the changelog and the updated help here https://hydrusnetwork.github.io/hydrus/developer_api.html#services_object . The old method of giving service names and types still works, so nothing breaks today, but it is now deprecated and will be deleted in early 2024. Please plan to move over!

next week

I am now on vacation! I'm off for a week to shitpost not-E3. I'll be back at it in a week, and I want to start pushing on my larger projects again. v532 should be out on the 21st. Thanks for the support everyone!

https://youtube.com/watch?v=Gp1dzdFmwmc
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Linux.-.Executable.tar.gz

NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 

I had a good week. You can now type system predicates in a normal search, and highlighting big downloaders is nicer.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The tag autocomplete text inputs on search pages get a couple of neat improvements this week:

First off, you can now type most system predicates, and they'll appear in the dropdown. You type the whole complete predicate, like 'system:imported before 2021-03-09'. This is the same system predicate parser that has been available in some advanced places, now for everyone. Note that it can't do every predicate yet, and while I try to cover several grammatical variants, some it is still quite inflexible. Let me know how you get on, and I'll keep working on it!

Second, the autocomplete now has a paste button. It accepts multiple newline-separated tags and will understand -negation, wildc*rds, namespace:* wildcards, and the new system predicates.

For downloaders, I made the 'highlighting' when you double-click a downloader or thread watcher asynchronous--it now loads the files in the background, only presenting them when the results are ready. This means if you highlight a big 6,000 file downloader, it won't hang the UI for several seconds. You can cancel a load by un-highlighting or highlighting something else, and a file-load that takes more than a few seconds will make a popup with its progress.

next week

I've just got one more week before my vacation, so I'll just do some code cleaning and little fixes again. Thanks everyone!

https://youtube.com/watch?v=zIY17xqTzmk
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Linux.-.Executable.tar.gz

NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 

I had a good week. There's a new quick lookup system that lets you search for a file's source without importing it.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

sauce

Every now and then, I am looking at a file outside of hydrus and I can't remember who the artist/character is, or I want to know if I have the artist subbed. When it isn't something the common online source platforms support, I usually download the file, import it to my client, and then do a 'open->similar looking files' on it to see everything I have that looks like it to get more info. I'm basically doing SauceNAO but on my own client. I wanted a way to do this faster.

So, this week, check out the renamed 'system:similar files' predicate on a fresh search page. It now has two tabs. The latter is the normal 'system:similar to' that takes file hashes, if you need to do some manual lookup between imported files, but the first panel is now essentially a paste button. If you copy image data or a file path to your clipboard and paste it there, it'll calculate the similar files info and embed it into the search predicate, letting you search for anything that looks similar to what you have in your clipboard. Give it a go!

I also added a search cache to the main 'similar files' search system. Repeat searches should be massively faster, and searches you re-run with increased distance, like 0 to 4 to 8, and the background similar files search should also accelerate naturally as the cache populates. My 10k file test client's similar files search sped up about 3-4x! I'm not sure what a million+ file client will do, but let me know what you see.

https://youtube.com/watch?v=I2lgy4_Quko
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Linux.-.Executable.tar.gz

NOTICE! Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 

I had an ok week. A bit short on work time, but there's some simple quality of life and bug fixes. The macOS App is also fixed.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

If you regularly use a custom slideshow time, you can now edit the quick-select times shown in the media viewer menu under options->media. It is hacky comma-separated numbers, like the zoom intervals, but have a fiddle and you'll get it.

I improved a bunch of the file search cancel tech. If you put in a search with 'system:imported since 6 months ago', and you then update it before it finishes, it should cancel that ongoing search and start the new one very quickly. This _should_ also vastly improve Client API browsing latency for these situations, since the cancel tech is supposed to fire when a search request is dropped, but let me know how it works IRL.

The time-based system predicate parsing is also more flexible. You can type 'since' or 'before' rather than having to figure out which '' you mean, and putting a colon in or 'ago' on the end should parse more reliably. My ideal is to get the text that a system predicate copies to clipboard perfectly parseable. I'd also, fairly soon, like to plug system predicate parsing into the normal file search autocomplete so more users can play with this.

The macOS App had a couple problems last week, with the client->hydrus_client change, and it wasn't booting correctly. Thanks to the users who helped figure this out. Please just install as normal, you still have no special update instructions for the v527 step (it'll just work now!).

next week

Mostly more simple work like this. I'm toying with the idea of a 'paste file/bitmap onto the client, it then does a similar-files search without importing the file' process, basically a 'sauce?' command from your clipboard, which might actually be real easy to add, but we'll see.

https://www.youtube.com/shorts/xK5SJ6CuezY
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Linux.-.Executable.tar.gz

I had a good week working on important updates to the build and a new media player prototype.

This build has special update instructions! All users who update from 526 or earlier to 527 or later will have to do this once!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

build update

A couple of weeks ago, I tried updating OpenCV (our image library) for everyone, but it caused a dll conflict and the program wouldn't boot. I am trying again this week, and since we have to do this change, I am also rolling in a long-planned change to the main exectuable names. 'client' is now 'hydrus_client' and 'server' is now 'hydrus_server'. Some users will have to do a 'clean install', and most will have to update their shortcuts or launch scripts.

As always--but particularly this week--please make a backup before you update. The following instructions are simple, but if anything goes wrong, you can always roll back to a safe backup.

Ok, the instructions, by situation, are:
- If you use the Windows installer, update as normal. It will do the clean install for you. Your start menu 'hydrus client' shortcut should also be overwritten with one to the new executable, but if you use a custom shortcut, you will need to update that yourself.
- If you use one of the normal extract builds, do a clean install. You also need to update any program shortcuts.
- If you use the macOS app, there are no special instructions. Update as normal.
- If you run from source, git pull as normal. If you haven't already, feel free to run setup_venv again to get the new OpenCV. Update any custom launch scripts to point at the new hydrus_client.py file.

A 'clean install' is as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
You are basically going to be deleting everything except your database (to clear out old dll files) and then updating as normal.

To update your shortcut or launch script, simply edit it, and where it says 'install_dir\client.exe', just rename it to 'install_dir\hydrus_client.exe'. For Linux users, that'll be 'install_dir/client'.

If you have a custom firewall rule for the Client API, you'll need to update it for the new executable name too.

And that's it! Let me know if you have any trouble.

QMediaPlayer

This is for advanced users only. Also only users who run from source and use Qt6!

I integrated Qt's built-in media player this week. I hope that in future it may be a fallback for users who cannot run mpv. Unfortunately, while it worked surprisingly well when it worked, it was also very crashy. I've got it stable when running from source, but it seems to insta-crash when in a build, so that's something to figure out later.

If you are an advanced user who runs Qt6 from source, you will now see the new experimental Qt Media Player as a selectable viewer under options->media for audio, animations, and video. It has a number of limitations (check the changelog for the full list), but it does work in most cases. Have a play with it and let me know how you get on.

next week

This week was a lot of work for what felt like little outcome. I was disappointed that QMediaPlayer crashed in the build. In any case, I'm short on getting small things done, so I'll return to that. Just some little fixes and quality of life.

I am taking my vacation week on the 8-14th of June, so I have four releases left before then. I won't go for anything huge, but try, if I can, to tie up some loose ends.

https://youtube.com/watch?v=2_fxNRtmyAE
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Linux.-.Executable.tar.gz

I had a mixed week. I was a little short on time, so this is a simple bugfix and quality of life release.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

When you petition in the manage tags/siblings/parents panel on a repository, it typically asks you for a reason for the change. There are some fixed reasons, but now the system will remember the last five custom reasons you enter too. I hope this saves some typing/pasting work for people with clever petitions to make. You can change the number of recent reasons saved (including setting to 0 to disable) under options->tags.

Every multi-column list in the program now has a 'reset the column widths for this type of list' entry in their header right-click menus. It also resizes immediately (an old debug routine that did this for all lists needed a restart). If your thread watcher list goes crazy wide, it should be fixable now! Let me know how it goes.

I added the basic media keys you find on many keyboards to the shortcut system--previous, next, play/pause, volume up, volume down, and mute. On Windows, these keys seem to have a global hook that fires regardless of how I capture it too, so they may not be so useful yet, but feel free to play around with them.

I fixed a stupid logical typo in my new date calculating code that was breaking system time predicates with the same number of months as the current month (like "system:modified time before 2 years 5 months ago" in May). Sorry for the trouble!

next week

There is going to be an important update next week!

Last week, I tried to update OpenCV for everyone, but it caused a dll conflict in the updated build and I quickly rolled it back in a hotfix. We are going to try again next week, which will require 'extracting' users to perform a clean install. Since we are making that change, I have decided to also roll in a long-planned renaming of the program executables. 'client' will become 'hydrus client' and 'server' 'hydrus server'. This part will require updating of custom shortcuts and launch scripts. I will reiterate all this with specific instructions for all platforms in next week's release, and I'll drop reminders for several weeks after.

Next week is also a 'medium size' job week. If I have time and energy, I'd really like to see if I can get the Qt media player working as an mpv fallback.