/hydrus/ - Hydrus Network

https://youtube.com/watch?v=wpUEdGDdaTQ
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Linux.-.Executable.tar.gz

I had a good week. There's lots of smaller fixes and improvements, and ratings are added to the Client API.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user submission, hydrus now recognises and supports importing .sai2 files! Basic metadata for now, but I'll see about resolution and thumbnail support in future.

It turns out PNG and WEBP files can have EXIF data, and our EXIF scanner works on them! All newly importing PNGs and WEBPs will be scanned, and on update, you'll have a yes/no dialog asking if you want to scan your existing PNGs. If you have a gigantic database, or your files are on a remote cloud storage, you might want to defer it, but it isn't really a big deal.

I banged my head against the media viewer 'notes hover' layout code, and I was actually really successful this time. Many of the borked layout issues are fixed, and I think I've completely fixed the overlapping hover problems in the duplicate filter.

I fixed a bunch of weird bitmap paste problems in the new 'similar files' paste button. Let me know if you still have any issues!

Just a small thing, but the 'loading' page in hydrus, when you refresh a search, is no longer grey. It is the same colour as the normal thumbgrid, so the loading flicker is gone.

Twitter is gone from the downloader defaults for all new users. All existing users should look to migrate away, too. I hope to roll out some downloader engine improvements in the near future that'll make nitter (on multiple domains) easier to set up, but there aren't a lot of excellent solutions.

I also added ratings to the Client API. The service object now talks about min/max stars and star shape, the file metadata call gives every file's ratings, and you can set and clear ratings too with a new command and permission. It is all fairly simple, but there's several possible data types depending on the rating type, so check out the Client API help for details and examples.

next week

I regret that I again did not find time for PTR janitor workflow improvements this week. Now that ratings on the Client API is clear, this is top priority. Depending on how things go, I may take a two-week release, since I definitely want some solid work done here.

Screenshot from... png
(259.22 KB, 492x392)

Locked out of discord because of their shitty mobile verification, so doing my part tagging paywalled furry art ¯\_(ツ)_/¯

>  https://youtube.com/watch?v=SMTz9nIUkGc

Screenshot 2021-05-26... png
(92.45 KB, 1242x682)

whattup,

I've been fiddling with hydrus for a bit but cannot figure out how to keep my tags visible and usable at all times. I have two tags so far, but expect to use dozens, if not hundreds in total in the future.

The whole thing isn't easy for me introductory tutorial that I've gone over so far has basically been "select image, press f3, do ur tag stuff".

Can I see all my own created tags without selecting an image and pressing f3 and be able to edit/delete them? Also, how do I keep tags in the "selection tags" section at all times?

Thanks in advance.

https://youtube.com/watch?v=Cv87NJ2xX0k
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Linux.-.Executable.tar.gz

I had a great week cleaning code and fixing bugs. If you have a big database, it will take a minute to update this week.

all misc this week, most bug fixes

I fixed a critical bug in tag siblings. It was causing some tag siblings to be forgotten, particularly on local tag services, and particularly when a tag had a sibling removed and added (e.g. 'delete A->B, then add A->C') in the same transaction. I will keep working here and will trigger a new sibling reprocess in the future just as 450 did so we can fix more PTR issues.

The new content-based processing tracking had a couple more issues. Some Linux users, in particular, it just broke for, due to a SQLite version issue. I have fixed that, and I have fixed some issues it caused for IPFS. There are new unit tests to make sure this won't happen again.

I fixed an issue with sessions recently not saving thumbnail order correctly!

I fixed issues with some of the new Client API file search parameters not working right!

Big video files should import a bit faster, and will show more status updates as they do their work.

We have had more anti-virus problems in recent weeks. We'd hoped building on github would eliminate them, but it hasn't completely. A user helped me trace one issue to the Windows installer. We have 'corrected' it (believe it or not, it was removing the 'open help' checkbox on the final page of the install wizard, and yes the reasons for why this was causing problems are ridiculous), so with luck there will be fewer problems. Thank you for the reports, and let me know if you have more trouble!

full list

- stupid anti-virus thing:
- we have had several more anti-virus false positives just recently. we discovered that at least one testbed used by these companies was testing the 'open html help' checkbox in the installer, which then launched Edge on the testbed, and then launched the Windows Update process for Edge and Skype, which was somehow interacting with UAC and thus considered suspicious activity owned by the hydrus installer process, lmao. thereafter, it seems the installer exe's DNS requests were somehow being cross-connected with the client.exe scan as that was identified as connected with the installer. taking that checkbox out as a test produced a much cleaner scan. there is a limit to how much of this nonsense I will accomodate, but this week we are trying a release without that 'open help' link in the installer, let's see how it goes
- semi-related, I brushed up the install path message in the installer and clarified help->help will open the help in the first-start welcome popup message

https://youtube.com/watch?v=vDPoR5PHw7Y
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.Linux.-.Executable.tar.gz

I had a good week. There are improvements to PDFs and system predicate parsing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user, PDFs now have thumbnails! I'm happy and surprised with how this turned out. The module it uses also offers raw text access, so I have rejuvenated a I-think-good-enough 'num words' count for PDFs. Also, PDFs can have a little embedded metadata like Author and Title, and this is viewable on the special button at the top of the media viewer. On update, all your existing PDFs will be scheduled for some file maintenance work to figure all this out, so give it a little while to catch everything up. As always, please send in any PDFs that get weird results. I expect we'll do a little more here, particularly pulling the embedded data into hydrus proper as tags etc...

System predicate parsing (e.g. by typing 'system:inbox' into the tag autocomplete) has several improvements. All comma-grouped numbers are now parseable, so you can paste 'system:width = 1,920' and it'll work. 'system:tag as number' has some special fixes, and several tags have presentation updates within the client to make sure that the text they are producing for a label is actually something that parses back to them.

Tag export sidecars now have a 'tag display type' selector to choose between 'storage' and 'display' tags. Until now, all the sidecars were all 'storage', meaning they did not include parent or sibling tags (like the list you edit in manage tags), whereas 'display' includes sibling replacements and implied parents (like the normal file search view shows). Existing tag export sidecars will stay on 'storage', but new ones will default to 'display'.

next week

I did some more file storage updates this week, but it remained pretty boring behind the scenes stuff. I'm going to keep bashing away, and I'd love to have a bit of other fun too, maybe custom searches for Mr Bones and the file history chart, which I have been meaning to do for ages.

https://youtube.com/watch?v=xJRAubWCkcc
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.Linux.-.Executable.tar.gz

I had a good week. There's a mix of all sorts of work.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user, we are adding support for two more filetypes today: QOI, which is a png alternative, and Procreate, an Apple image project file. QOIs have full support, Procreate files just have thumbnails.

If you use one of the darkmode styles on Windows and your tooltips suddenly got too dark to read a couple weeks ago, please check out the new 'alternate-tooltip-colour' variants of the styles in the options. It should fix you up for now. Try switching back when we move up to a new version of Qt again--it is a bug on their end.

There's two new checkboxes in options->media to ignore 'uninteresting' import and modified times. By default, if the modified time is very close to when a file was added to 'my files', I hide it in the media viewer. If you would prefer to always see it, you can now set this.

Importing massive apngs is now significantly faster.

The parsing system has a new 'String Joiner', which does string concatenation. It can glue the strings together with any custom text, including the empty string for pure concatenation, and it can join groups in 1-2 1-2 1-2 or 1-2-3 1-2-3 style patterns too.

The Client API can now fetch the siblings and parents of tags. If you want to get into siblings and parents but have never done it before, it can get complicated, so brace for impact.

file storage upgrades

I started on this work this week, but it is all boring behind the scenes stuff so far. I'm happy with the progress though, and I'd like, in the nearish future, to have:

- lower file access latency when you have millions of files
- the ability to say 'store no more than 200GB in this folder'
- background migration of files rather than the current block-the-whole-program-for-ages 'move files now' system

I'll keep chipping away at this over the next few weeks. I feel good about it.

next week

More like this. Catchup on older issues and pushing on file storage.

https://youtube.com/watch?v=ANDQoEl2StM
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.Linux.-.Executable.tar.gz

I had an ok week. There's a mix of smaller work.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

system:filetype parsing now supports more human-friendly words. You can type 'system:filetype is image' or 'system:filetype is static gif, png' and it should just work. I think that any existing filetype predicate string (e.g. if you right-click->copy it) should now parse ok.

I fixed a bunch of broken 'system:similar files' image bitmap pastes that were not producing the correct pixel hash. Also, I've started a background job today to fix some lingering bad duplicate pixel data for pngs, so if you have recently done a duplicate filter with 'must not be pixel dupes' but got some anyway, I hope this will fix itself soon.

I added links to https://github.com/Garbevoir/wd-e621-hydrus-tagger (and https://github.com/abtalerico/wd-hydrus-tagger, which it is based on) to the Client API help. These use the recent 'I give text, it draws an image' AI tech everyone has been messing around with to do the reverse--you give it an image, and it automatically tags it. This stuff is just getting ready to seriously play with, so if you are interested, give it a go!

next week

I didn't get to the file storage upgrades I wanted this week, so I'll give it another go.

https://youtube.com/watch?v=htZEYN1TkOE
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Linux.-.Executable.tar.gz

I had an ok week. There's full PSD support and a variety of quality of life improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

PSD files

Just like last week, if you run from source, you might like to rebuild your venv again this week, and you'll get the PSD support. Just run 'setup_venv' again.

Thanks to a user, PSD files should now show in the program just like any other image! Since it takes a little extra CPU to render them, we're starting with conservative view settings--PSD files will show in full in the normal media viewer, but you'll get the cheaper 'open externally' button in the preview panel. If you want to try showing them everywhere, you can change the view settings under options->media.

PSD thumbs are also much nicer now, and your existing PSD thumbnails should regenerate very soon in the background.

As always, if you run across a PSD that has crazy colours or bad transparency--or if it straight up won't load at all, which we think some will--please send them in and I'll see what I can do.

other highlights

I'm splitting 'gif' files into 'static gif' (under images) and 'animated gif' (under animations) this week. It isn't a huge thing, and there aren't all that many static (i.e. non-animated) gifs going around anyway, but it simplifies some view settings as well as some code, and it aligns with how the other formats like png/apng work.

Rating system predicates are now parseable. Type 'system:has rating (service_name)' or 'system:rating for (service name) > 4/5' into the tag autocomplete, and it should prefill the result. It should work for all like/disklike, numerical, and inc/dec rating services, and like the other parseable system predicates, this extends to the Client API. Try it out!

I fixed the duplicate filter's right-hand hover window back in place. It was a mistake to have it 'avoid' the notes hover, since the buttons bounced around too much for easy clicking, so now they can just overlap, with the duplicates hover taking precedence.

next week

I'm due some larger work, and I think I'm going to tackle some file storage improvements. The migrate database dialog sucks, and the file storage system in general groans when it has millions of files, so I'd like to have the first steps towards user friendly storage settings and cleverer storage overall. We'll see!

https://youtube.com/watch?v=8yXk3_o1Hoo
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Linux.-.Executable.tar.gz

I had a great week. There's some new libraries and new file support.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

notes

If you get a popup on update about missing indices, don't worry. Something I forgot last week is being fixed. Just let it work.

If you run from source, I recommend you rebuild your venv this week. Just run the 'setup_venv' script again with the same settings you chose before. You'll get new Qt, OpenCV, and the HEIF support. If you are on an older OS and run into trouble with Qt, the setup now has a 'Qt (o)lder' option that gives you something we know is very stable and compatible.

=highlights

The main UI and image libraries that hydrus uses are updated today. We've been in an awkward place with Qt for a while, stuck on a version I regret updating to, so I hope this jump relieves this. It should speed some image load, reduce some flicker, and fix some UI scaling issues. There are no special install instructions, and we've been testing for months on different computers and haven't seen any specific update problems, so I'm not expecting any significant issues. If you do run into anything, then please roll back to your backup and let me know.

Thanks to a user, we now have full support for HEIF, HEIC, and AVIF files! They import and work like any other image. We also support the 'sequence' version of these files, which are essentially like (currently super rare) high quality animated gifs. I'm very new to all these formats, and they are obviously less common, so if you have a weird version or your colours are all messed up, I'll be interested in examples. I don't even have a 'HEIC sequence' to test with!

My deferred delete system is complete. Most users don't have to ever think about it, but it is under database->database maintenance. There's an UI panel to track what still needs to be deleted and some basic settings to control when it does its work. If you delete the PTR or do a large regeneration job, the (very slow) actual delete of the existing data now happens in pieces in the background in the future. I'm really pleased with the performance, and even deleting 30-million-row tables seems to work very well, but if you delete the PTR (2 billiow row table!) and get unacceptable lag every few seconds, check that menu and disable the work during idle/normal time and let me know, and we'll figure it out.

tags->migrate tags has cleverer file filtering. You can now select the tags for files 'deleted from x' or 'currently in y or z' real easy.

next week

I'm generally happy with my current productivity, but I'm falling behind on many things at once. I would like to focus on system predicate parsing and Client API for a week, because it keeps slipping.

https://youtube.com/watch?v=pg-yeXEvVwA
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Linux.-.Executable.tar.gz

I had an ok week. There's a mix of bug fixes and a couple of neat improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

First off, the filetype selector, which you'll see in 'system:filetype' and any File Import Options, is completely rewritten to a checkbox tree. It is easier to work with than the old expanding panels system and scrolls smoothly when it gets internally tall. I've been worried about adding more filetype support for a while because of how busy this widget was getting, but this will relieve things.

Also, the 'applications' group is now split up. All the archives like zip and rar are now in 'archives', and all the xcf, psd, and krita formats are in 'image project files'. It doesn't matter much yet, but these groups also exist as separate rows in the viewing settings under _options->media_.

Secondly, I have figured out a 'fast delete' method for very large database objects. Deleting the PTR is now an instant operation! Unfortunately, under the hood this is really just a deferred delete, so the object still uses its hard drive space until it is deleted later in the background in pieces. Unfortunately x2 is that I haven't written the maintenance job that does the delete yet, lol, so if you need to delete some service or do a large database regenerate job, please bear with the bloat for a week.

If you updated last week, jumping up several versions at once, and got a popup that some EXIF scanning update did not work, don't worry. I fixed the problem, and nothing important was actually broken anyway. Thanks for the reports!

I fixed the grid layout on tags->migrate tags, which exploded under my nose. Sorry if you were recently confused by this!

next week

I'll finish this background deletion maintenance system and keep on trucking on smaller fixes. I've got a lot of immediate work like this to do and github issues to catch up on. I'd also love to get some more system predicate parsing, for ratings and more, working.