/hydrus/ - Hydrus Network

Hey,

I was recently trying to import some files into hydrus that I had downloaded using gallery-dl. Thanks to a config-file mistake, all the tag files were saved as html pages. Is there a way to use regex to extract the tags? I attached one of the html files, if that helps.

Thanks!

https://youtube.com/watch?v=GuVnZZ1sFIc
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Linux.-.Executable.tar.zst

I had a mixed week. Thumbnail rearranging is added, and some bugs and quality of life issues are cleared.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

thumbnail rearranging

So, if you right-click any thumbnail, you'll now see a 'move' menu. This makes it simple to rearrange your thumbnails. You can move your current selection to the start, the end, left one, right one, or 'to here' if you have multiple selected. If your multiple selection is non-contiguous, it will be made so on move, with the move focusing around the position of the first selected item.

You can also map these commands to keyboard shortcuts under the 'thumbnails' shortcuts set. I haven't added any default shortcuts for this yet, but let me know if and what you would prefer--I've been playing around with ctrl+numpad numbers on my dev machine, and it feels nice to me.

In future I'll try and figure out mouse drag-and-drop rearranging. It might have to wait for a larger pending rewrite of the thumbnail grid though--we'll see.

other highlights

Unfortunately, there were a couple of stupid typos in the content processing changes last week. One in the duplicate filter (which the v560a hotfix addressed), and another fixed today that was causing 'already in db' results to not get metadata updates correctly. Sorry for the trouble, and thank you to the users who reported these.

Ctrl+C/Ctrl+Insert is now hardcoded to copy tags from the taglist.

The thumbnail and media viewer menus should now be much thinner. I hate how wide they can get, how annoying it is to hit their many nested submenus when they get like that, so let me know if they still go crazy in some situations.

There was a bitrot issue in v559, the millisecond timestamps conversion, that made it impossible/bad to update from a much older version. This has always been a tricky technical issue to talk about and communicate to the user, so I've now written a better in-client error reporting process that stops the user before the update is even attempted. The upshot this week is that if your client is v551 or older and you try to update to v561 or later, you will be told to update to v558 first.

In lieu of a proper rewrite, I've made some semi-hacky newline processing improvements to the parsing system. If you make downloaders and hate having to deal with extra whitespace in multiline content, notes or otherwise, check out the full changelog and let me know how you get on with it all.

next week

I want to work on github bug reports, which I haven't put proper time into for too long!

https://youtube.com/watch?v=UBYThAP_C5A
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Linux.-.Executable.tar.zst

I was suddenly without internet for a while, so there was no release last week, but everything is back to normal now, and I had a great couple weeks' work. The 'edit times' dialog can now handle multiple files.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

editing times for multiple files at once

If you launch 'edit times' on a single file, everything is the same. If you open it on a larger selection, it will now show, in a summarised way, the range of each particular time type, and how many files in the selection have or do not have a time set there. The controls all work as before, but in general, when you set a time, all files are now locked to that new time. Give it a play and you'll see how it all works.

SInce we don't always want to set exactly the same time to a set of files, but rather ideally a similar, staggered time, there is also a new 'cascade step' system in the multiple file version of the dialog. Every time-edit you open up has the ability to enter a little step, say 100ms, which will cause the dialog to set each successive file in the selection to be that much later (or earlier, negative values are allowed!) than the last. This way, if you have a bunch of files of something contiguous, like a comic chapter, that all have different, merged, or otherwise awkward import times, you can now manually sort them in the UI using tags or hackery and apply a new import time based on the first file plus a few milliseconds each time. Thereafter, any time you sort those files by import time, they'll also be nicely in page order time, just as if you had imported them all together from one directory.

I am quite happy with the new dialog, and I plan to copy some of the new techniques I figured out to other dialogs for similar better multi-file handling. I'd also really like, as we have discussed before, to tackle a 'cascading' tagger in 'manage tags', so we can quickly set 'page:n' tags to many files at once.

https://youtube.com/watch?v=zBuJD-ugT_g
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Linux.-.Executable.tar.zst

I had a great week working mostly on one thing: converting the timestamps in the program from seconds to milliseconds.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

milliseconds

tl;dr: time numbers better, you don't have to do anything

Since the program started, it has tracked the various file 'times', like import and archive time, using a system that stores time with second precision. It knows that file x was imported at 2023-12-02 14:04:51, but nothing finer. This is common in many computing applications, but sometimes you want more. Particularly, in hydrus, whenever you have a fast importer that brings in two files within the same second, the program does not know, later, which was actually imported first (e.g. when you sort by 'imported time'). This can be annoying when you import a whole chapter of a comic and don't have good page tags set up yet--random pairs of pages will be flipped whenever you next try to sort those files by 'import time'.

So, this week I converted the database to use a time system that has millisecond resolution. Any files imported from now on, or deleted, archived, modified, or 'last viewed', will now get a millisecond timestamp. You don't have to do anything, and nothing outside of the 'edit file timestamps' dialog will appear any different, but it may take a minute to update.

This work was quite simple, but there was a ton of it, years and years of system build-up to go through. Rather than make an already messy system more complicated with a rough injection, I decided to invest the time and clean everything as I moved to the new format. I've tested it back and forth, but given the number of changes, I wouldn't be surprised there is a typo in there somewhere--if you see a file saying it was last viewed '54 years ago' or similar, let me know!

Also, the Client API can now edit timestamps, including with the new millisecond precision. If you are an API dev, check out the new call (and 'edit file times' permission) here: https://hydrusnetwork.github.io/hydrus/developer_api.html#edit_times_set_time	

misc

Just a side note, I have removed the sankaku downloader defaults from the program, so new users won't see them. None of them were working well/at all, especially in recent weeks. If you want to grab from complicated sites, and there isn't a clean solution on the shared downloader repo here, https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts , I recommend going with a more robust solution like gallery-dl/hydownloader or just finding the content elsewhere.

If you ever run something like 'system:known url: regex = blah.com/(regex stuff)' and are annoyed at how slow it is, try adding a 'system:known url: domain = blah.com' pred in addition. This combination will now be explicitly much faster than just the bare regex (previously, it could be faster, by accident).

next week

Now we have ms timestamps and nicer code, I'd like to try tackling an 'edit timestamps' dialog that works for multiple files, including a 'cascade' command for force some clever sorts.

https://youtube.com/watch?v=crJst7Yfzj4
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Linux.-.Executable.tar.zst

I had an ok week. I figured out 'system:number of urls', and you can now import rtf files!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

So, you can now search for 'num URLs'. You will find it under a new stub, 'system:urls', at the bottom of the normal system predicate list (where 'system:known urls' has also moved). This first version is simple--it counts all URLs, regardless of how important, but I can see a future version having the ability to scan by 'post URLs' or specific URL classes. Give it a go and let me know if you run into any trouble!

Also, thanks to a user, you can now import rtf files! There's no word count yet, but I think it should be doable in future.

If you select multiple tags, you can now hide or favourite them all at once. There's a bit of workflow and presentation improvement here, too.

The old 'tag suggestions' box called 'favourites' is now called 'most used'. It was stupid to have two tag systems called 'favourites', so this is now fixed. You still edit them under options->tag suggestions; they just have a more precise and less confusing name now.

If you are an advanced macOS user (i.e. you know how to build something with xcode), you might like to check out Hydra Vista, a new user-made macOS App that presents your client (via the Client API, e.g., perhaps on another computer) in a booru-like wrapper. https://github.com/konkrotte/hydravista

next week

I would like to add timestamp editing to the Client API.

https://youtube.com/watch?v=xNKXyr6YN8Y
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Linux.-.Executable.tar.zst

I had a good week back after the holiday. There are some bug fixes and improvements to system:hash parsing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Tag filters (which operate the various tag whitelists/blacklists across the program) now edit much much faster when they are full up with stuff. You can paste 5,000 items into an empty tag filter in less than a second now, and removing one or a handful of items from a filter with thousands of items already in is now instant. Previously, these things could take many seconds or even minutes due to inefficient overhead.

CBZs that have four or fewer pages should now be recognised correctly.

When you copy 'system:hash' and the 'system:similar to' predicates to clipboard, you now get a much longer string that includes all the hashes in the predicate. These copied strings are all parsable, meaning you can paste them into the same client elsewhere or a different client entirely and it should all just work in a complete loop. The human labels are updated to give you more information, too.

There may be other predicates that are currently parsable if you type carefully but do not themselves copy a parsable string, but I am not sure which they are, so if you run into one, let me know and I'll see what I can do.

If you run from source on Windows--or you'd like to--but you haven't installed the extremely convenient but bulky-to-install 'Git for Windows', I had to go through the installer again to set up my new dev machine, and I wrote out a guide for the full 12-page wizard here: https://hydrusnetwork.github.io/hydrus/running_from_source.html#core It is actually easy to do, and almost everything can be left as default and things will be fine. In any case, Git is great and lets you update in about three seconds, so if you run from source on Windows, give it a go!

next week

My old dev machine died just before Christmas, but I bought a new one and am back to normal work. For next week, I'd like to get some sort of system:num_urls figured out.

https://youtube.com/watch?v=e3mkeK6pRaY
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.Linux.-.Executable.tar.zst

I had an ok week. I fixed some bugs and added a system to force-set filetypes.

You will be asked on update if you want to regenerate some animation thumbnails. The popup explains the decision, I recommend 'yes'.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

forced filetype

The difference between a zip and an Ugoira and a cbz is not perfectly clear cut. I am happy with the current filetype scanner--and there are a couple more improvements this week--but I'm sure there will always be some fuzziness in the difficult cases. This also applies to some clever other situations, like files that secretly store a zip concatenated after a jpeg. You might want that file to be considered something other than hydrus thinks it technically is.

So, on any file selection, you can now hit right-click->manage->force filetype. You can set any file to be seen as any other file. The changes take affect immediately, are reflected in presentation and system:filetype searches, and the files themselves will be renamed on disk (to aid 'open externally'). The original filetype is remembered, and everything is easily undoable through the same dialog.

Also added is 'system:has forced filetype', under the 'system:file properties' entry, if you'd like to find what you have set one way or the other.

This is experimental, and I don't recommend it for the most casual users, but if you are comfortable, have a play with it. I still need to write better error handling for complete nonsense cases (e.g. calling a webm a krita is probably going to raise an error somewhere), but let me know how you get on.

other highlights

I fixed some dodgy numbers in Mr. Bones (deleted file count) and the file history chart (inbox/archive count). If you have had some whack results here, let me know if things are any better! If they aren't, does changing the search to something more specific than 'all my files'/'system:everything' improve things?

Some new boot errors, mostly related to missing database components, are now handled with nicer dialog prompts, and even interactive console prompts serverside.

I _may_ have fixed/relieved the 'program is hung when restored from minimise to system tray' issue, but I am not confident. If you still have this, let me know how things are now. If you still get a hang, more info on what your client was doing during the minimise would help--I just cannot reproduce this problem reliably.

Thanks to a user who figured out all the build script stuff, the Docker package is now Alpine 3.19. The Docker package should have newer libraries and broader file support.

https://youtube.com/watch?v=BE4ptZRUJZo
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.Linux.-.Executable.tar.zst

GET

I had a good week. There's some cbz/Ugoira follow-up, nicer system:time parsing, and much better boot error-handling.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The CBZ/Ugoira stuff last week went ok! We had a few too many false positive Ugoiras, so that test is tightened up, and those files should soon become CBZs. For false positive CBZs (i.e. just a zip of CG images or similar), we don't have a good automatic solution, but I still plan to roll out 'force this file to be this filetype' tech in future so we can, amongst other things, assign these fixes manually.

The various 'system:archived since xxx' predicates' parsing is now plugged into the same excellent date parser we are using in the downloader system. If you type them in, they'll now take all sorts of date phrasing. Try 'since 01/05/2016' or 'before june 2022' into the autocomplete--you may even be able to use your own language!

There is still a little work to do with the 'since/before x time units ago' variants, though. In some unclear cases (including foreign languages), the before/since may be flipped to what you type. Also, I have decided to soon migrate these predicates to just store days and hours (no more year/month). You can still enter '1 year ago' with this new parser, but on my end, trying to calculate leap years and weird month durations has caused too many problems, so I am going to simply pull back over the near future and let you put 365 or 30 in yourself! In any case, give this stuff a go and let me know how you get on.

When the hydrus client fails to boot really early on, before the main UI system is live, it should now nonetheless pop up a dialog saying what happened! The only way this will fail is if the problem with the boot is the Qt UI library, lol. In either case, the 'hydrus_crash.log' file is still made on your desktop.

After talking it out with users, I have decided to move towards dropping the image library OpenCV from the program. It has served us well, but it is often difficult to install and a bloat, and our flexible alternative, Pillow, works extremely well these days. I'm not ready to flick the switch yet, but we have done work here and there, and if you would like to help me test this out, please hit the 'IN TESTING: Load images with PIL' checkbox under options->media and let me know if you have any images that suddenly load incorrectly.

The String Splitter and Joiner objects in the parsing system now accept \n and \t for newline and tab. If you need to split or join by \, use \\. To not break any existing parsers, existing objects that have a \ have been updated to have \\.

next week

I only have two more weeks in the year, so I'll just do some cleanup and little jobs.

https://youtube.com/watch?v=HUVtoWIeyp8
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.Linux.-.Executable.tar.zst

I had an excellent week. Some important bugs are fixed, and we have some basic support for CBZ and Ugoira files.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

bugs

I screwed up two important calls last week, in fixed-period checker timers and job statuses. If you had watchers not start, subscriptions with incorrect check times, and/or popup messages that threw display errors or wouldn't auto-dismiss, I am sorry! I didn't plan the changes here properly, and several things slipped through my tests.

All the affected systems have been given proper rework this week and have some unit tests to make sure this doesn't happen again. Please let me know if you still have any problems.

animations

I fixed up more of the 'while checking for transparency, this file produced x error!' issues. Checking GIFs for transparency should be a bit faster and more fault tolerant too, now.

Also, the native GIF renderer has much improved transparency support. The multilayered 'noclip' artifacts should be gone, and damaged GIFs will recover better. GIFs also now get thumbnails that are x% in with proper transparency.

Also, APNGs now get transparency: when rendering in the native renderer; in their thumbnails (which are also now x% in); and for 'has transparency' checks.

CBZ and Ugoira

I am adding basic recognition and thumbnails for these filetypes today. Behind the scenes, both formats are essentially just zips with a list of images, so all your zips will be scanned, and if they look like a CBZ or Ugoira, their filetype will change and they will get a thumbnail. Ugoira thumbnails will be x% in, like for other video.

Also, a user is working on true Ugoira rendering now, so I hope we will be able to finally roll this out in the medium term.

Unfortunately, neither format has a particularly definitive/unique specification, so while I have tried to be careful, my tests here are imperfect. We can expect a few incorrect determinations one way or the other. If you get an outrageous false positive or false negative here (e.g. something you know is a Ugoira that stays as a ZIP, or a ZIP of misc files that detects as a CBZ), please send in the details, and I'll see if I can tweak my tests.

next week

I put in extra time this week to figure out CBZ, so I'll let things breathe and just catch up on simple, small work. I have a bunch of interesting quality of life UI items in my immediate todo, so I'll probably focus on that!

https://youtube.com/watch?v=FhGziRJY730
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.Linux.-.Executable.tar.zst

I had a great week. The issues with gifs are fixed, and slideshows with videos are smoother.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I was happy with the 'has transparency' work last week, but unfortunately some damaged animated gifs (usually where one frame was borked) were raising a serious error when then were inspected. This error, which was interpreted as a hard drive fault, was a false positive! If you got it (should have been a popup talking about an 'I/O Error', and that maintenance work would halt until the next boot), do not worry--it wasn't a big deal. I have fixed up the error handling and a bunch of other gif issues. Sorry for the trouble!

There are several new slideshow timing options under options->media. You don't need to tweak them unless you really care, but now, when you do a slideshow that involves video or audio, the transitions should happen earlier or a little later in order to line up with where the video naturally ends. Check the changelog and tooltips of the new widgets for full details.

The various 'import options' on downloaders now say '(all default)', '(some set)', and '(all set)' on their labels, so you can quickly, at a glance, see what you have set where.

The thumbnail manage->regenerate menu is now called maintenance, and it now lists all the possible file maintenance commands. Should make it easier to fix weird problems.

next week

I have some server work to do. Hopefully clear out some github bug reports too.

I also just realised we only have about four more releases before the year is done--it feels to me like the second half of this year has flown by. I'll try and fit in one more medium-sized job before then.