/hydrus/ - Hydrus Network

- search change for potential duplicates:
- this is subtle and complicated, so if you are a casual user of duplicates, don't worry about it. duplicates page = better now
- for those who are more invested in dupes, I have altered the main potential duplicate search query. when the filter prepares some potential dupes to compare, or you load up some random thumbs in the page, or simply when the duplicates processing page presents counts, this all now only tests kings. previously, it could compare any member of a duplicate group to any other, and it would nominate kings as group representatives, but this lead to some odd situations where if you said 'must be pixel dupes', you could get two low quality pixel dupes offering their better king(s) up for actual comparison, giving you a comparison that was not a pixel dupe. same for the general searching of potentials, where if you search for 'bad quality', any bad quality file you set as a dupe but didn't delete could get matched (including in 'both match' mode), and offer a 'nicer' king as tribute that didn't have the tag. now, it only searches kings. kings match searches, and it is those kings that must match pixel dupe rules. this also means that kings will always be available on the current file domain, and no fallback king-nomination-from-filtered-members routine is needed any more
- the knock-on effect here is minimal, but in general all database work in the duplicate filter should be a little faster, and some of your numbers may be a few counts smaller, typically after discounting weird edge case split-up duplicate groups that aren't real/common enough to really worry about. if you use a waterfall of multiple local file services to process your files, you might see significantly smaller counts due to kings not always being in the same file domain as their bad members, so you may want to try 'all my files' or just see how it goes--might be far less confusing, now you are only given unambiguous kings. anyway, in general, I think no big differences here for most users except better precision in searching!
- but let me know how you get on IRL!
- .
- misc:
- thank's to a user's hard work, the default twitter downloader gets some upgrades this week: you can now download from twitter lists, a twitter user's likes, and twitter collections (which are curated lists of tweets). the downloaders still get a lot of 'ignored' results for text-only tweets, but this adds some neat tools to the toolbox
- thanks to a user, the Client API now reports brief caching information and should boost Hydrus Companion performance (issue #605)
- the simple shortcut list in the edit shortcut action dialog now no longer shows any duplicates (such as 'close media viewer' in the dupes window)
- added a new default reason for tag petitions, 'clearing mass-pasted junk'. 'not applicable' is now 'not applicable/incorrect'
- in the petition processing page, the content boxes now specifically say ADD or DELETE to reinforce what you are doing and to differentiate the two boxes when you have a pixel petition
- in the petition processing page, the content boxes now grow and shrink in height, up to a max of 20 rows, depending on how much stuff is in them. I _think_ I have pixel perfect heights here, so let me know if yours are wrong!
- the 'service info' rows in review services are now presented in nicer order
- updated the header/title formatting across the help documentation. when you search for a page title, it should now show up in results (e.g. you type 'running from source', you get that nicely at the top, not a confusing sub-header of that article). the section links are also all now capitalised
- misc refactoring

- bunch of fixes:
- fixed a weird and possible crash-inducing scrolling bug in the tag list some users had in Qt6
- fixed a typo error in file lookup scripts from when I added multi-line support to the parsing system (issue #1221)
- fixed some bad labels in 'speed and memory' that talked about 'MB' when the widget allowed setting different units. also, I updated the 'video buffer' option on that page to a full 'bytes value' widget too (issue #1223)
- the 'bytes value' widget, where you can set '100 MB' and similar, now gives the 'unit' dropdown a little more minimum width. it was getting a little thin on some styles and not showing the full text in the dropdown menu (issue #1222)
- fixed a bug in similar-shape-search-tree-rebalancing maintenance in the rare case that the queue of branches in need of regeneration become out of sync with the main tree (issue #1219)
- fixed a bug in archive/delete filter where clicks that were making actions would start borked drag-and-drop panning states if you dragged before releasing the click. it would cause warped media movement if you then clicked on hover window greyspace
- fixed the 'this was a cloudflare problem' scanner for the new 1.2.64 version of cloudscraper
- updated the popupmanager's positioning update code to use a nicer event filter and gave its position calculation code a quick pass. it might fix some popup toaster position bugs, not sure
- fixed a weird menu creation bug involving a QStandardItem appearing in the menu actions
- fixed a similar weird QStandardItem bug in the media viewer canvas code
- fixed an error that could appear on force-emptied pages that receive sort signals

next week

We are due a cleanup week, and I think I need one. Just some simple background work for a bit so I can rest, and I'll fit in some other small work like this week. As usual, I don't have anything exciting planned for the anniversary of v500. I was thinking about making that the week for the exclusive switch to Qt6, but with some users still getting crashes, I am less keen.

Thanks everyone!

 >>/1357/

 >>/1359/

No, thanks to you. Your work on this program is nothing short of amazing. There's more updates than I can keep up with and it's one of the programs actually worth reading the change notes regularly. Keep up the good work!

If you don't mind me asking, how do we work around 403 errors when sending requests via Tor? I guess there is not really much of an option, besides running some proxy behind the exit node, or if available logging into websites or playing around with the downloader/header details? Cloudflare does not like Tor traffic. I really love the subscription function of your program, even if it is primarily oriented for art it can be a very powerful tool for OSINT.

Is there an easy answer to how to regularly update that I haven't found yet? So far I know that updates can break things and backups are a must. My method of updating from an older version to a newer one (a lot of versions actually) is to simply export everything and re-import it. A lot of work and probably stupid, but it works. I want to avoid breakage. I simply sync the whole folder with rsync -vacu when having made changes to back them up.

I'd also be interested in knowing how to manage adding new uploads, without having to cross reference the old ones all the time. Since the backups are not available on the working machine, I'd like to just add available uploads based on a parameter like time (all uploads from date X on i.e.). I'm almost sure I have overlooked something in the options. Having this set up right would avoid redownloading old uploads over and over again, saving on bandwith and time.

Again, thanks for your amazing work and enjoy your time off, you really earned it!

 >>/1357/
I have an old PC running Windows 7 that I decided to keep Hydrus on for other things. For it I usually used the exe to install and update Hydrus. Since the latest windows installer is exclusively for Qt6. Is it fine to run the Ot5 zip over my exe installed Hydrus ie, copy and paste the zip contents to the Hydrus Network folder?

Hey, everyone. My dad died last night. I don't know how any of the timetable is going to work out yet, but I am not going to be able to keep up with my hydrus schedule for the time being. Please expect v500 and my general responses to be a week or two late. I will post to all my normal channels when I am back to regular schedule.

 >>/1361/
I'll catch up with you later and give a proper reply. I'm glad you like hydrus.

 >>/1362/
Yeah just drag and drop the extract over the installer. Make sure you make a backup beforehand, just so you are super safe in case anything goes wrong.

 >>/1363/
My condolences.
I think all agrees that you should take whatever time you need.

Forgot the role signature.

 >>/1363/

I'm sorry to hear that. Take your time off, real life always comes first. I'll be waiting for you when you return, don't worry.

Yes, Hydrus is a superb tool for organizing and downloading media. I switched several collections over to it completely. Once you got a system established it makes it very easy to organize as broad or as specific as one wants. But of course you of all people know that best.

My mid term goal is to automate subscriptions, tags, etc. and connect them to twint or maltego to create relationship maps. The automated acquisition and organization within Hydrus of twitter media solves a lot, I'm sure it could also handle other media and platforms, OCR the contents and auto-tag, but that's long term.

 >>/1363/
Sorry about your dad man, sounds rough.

> Yeah just drag and drop the extract over the installer. Make sure you make a backup beforehand, just so you are super safe in case anything goes wrong.
Alright thanks, I updated and everything went ok except that whenever I try to play a video file like a webm it says "MPV is not available" and tells me to open the webm externally.

 >>/1367/
Also I seem to be getting crashes after certain searches are completed. Hydrus would crash partly through when loading several images from a tag search, like it'll load several images then stop and give me the crash error. I was able to repeat this multiple times using the same tag search and getting the same results, while a different tag search won't have any problems. Also the crash error comes from Hydrus and not windows, which is rare.

And again sorry for your lost, its not a big issue for me so take as much time off as you need.

Thanks everyone for the kind comments. There have been dozens of things to do, but the bulk of the immediate work is now done. My Dad's death was completely unexpected, so this has been an awful shock, far earlier than I ever thought, but we had a great relationship, and I have no big regrets hanging over me.

For hydrus, I am missing work and keen to get back to it when my schedule is free again. More than anything right now, I am simply busy and tired. I won't rush things, but I'll have some empty days coming up over the next week and would like to catch up with my messages at least. Rather than put out an anemic release on the 21st, I'll work that day instead, let it all roll over, and be back to normal schedule on Saturday the 24th, with a proper full-changelog v500 out on the 28th.

Thanks again!

 >>/1369/

I feel you, both with loss and work. I'm in no position to tell you what to do, just going off experience take your time to deal with things and take time to regenerate. Every ounce of energy you spend must be filled up again, with sleep, food, relaxation, meditation, stretching, fresh air, etc. If you strain yourself for too long without rest, you'll get a breakdown. I just don't wish that to happen to anybody.

I'm glad to hear you're back on track though.

 >>/1361/
Yeah, Cloudflare 403s are a pain. I am not sure about Tor and Cloudflare, that may just be a step too far in complexity, but the 'normal' way to get through a 403 Cloudflare block is to visit the site in your browser (which does the clever problem solving) and then use Hydrus Companion to copy the solved cookies over to Hydrus.

https://gitgud.io/prkc/hydrus-companion

However, this may not work with Tor. Tor probably sets its own User-Agent and I guess is using different IP addresses regularly, and those are the two things that need to be the same as your browser for the Hydrus Companion trick to work. Maybe if you did the work in the Tor browser, but then copying cookies across may be even more difficult.

That said, some sites are just strict when it comes to Cloudflare rules, so if your VPN or whatever network solution is frowned upon by Cloudflare this week, then you'll just have trouble with that site. My own answer is mostly just to navigate away from those sites and stick to what I know works for me. I'm already overloaded with stuff to process, so no point spending 50% effort trying to fetch 5% more stuff that I missed.

For updating, no worries, the normal method is to extract the release on top of your old install folder. The extract straight from the zip file has no 'database' files, so it extracts safely over what you already have without overwriting anything user-created. Please check the help here:

https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#updating

I personally make a backup, extract the fresh release on top, and boot. My update code will investigate your database, notice it is an older version, and apply several rules to update it in place. You will keep the same session and everything as when you last closed the client. The idea of the backup at that time is that if the update fails for some reason (rare, but it happens to a handful of users a few times a year), or more likely I messed something up, I can roll back immediately to where I was, no danger.

If you update using the normal routine and keep your old database, it will remember all its old known urls and be able to skip repeated downloads.

Let me know if you have any trouble with it!

 >>/1367/
 >>/1368/
Sorry for the trouble with the new release. Some users are getting several crashes, and I think the new mpv is the culprit. Since you are on Windows 7, I suspect the new mpv is incompatible with it in the same way things like Qt6 are too. Can you try going in to your install directory and deleting the 'mpv-2.dll' file? You should still have mpv-1.dll from a previous install, and that should work on its own.

If that still gives you trouble, can you try using the client as normal but avoid:

A) mpv (so don't load up any movies into the preview or media viewer)
B) the duplicate filter (I have some reports this is giving trouble too)

If the program does not crash until one of these is loaded, that is good evidence that it is mpv after all. Please avoid them for now, or if you have a backup, roll back to v498, and I will work on these problems and see if I can fix some things for v500.

If you go to "install_dir/db/client (date).log", you might like to scroll down to the bottom of the one for September. Are there any meaty-looking errors in there? They might be called 'traceback'. Most crashes do not leave any log information (they break too quick), but if there are some lead-up errors in there, that would also be useful to know.

 >>/1370/
Thanks. Yeah, I am going to be careful and not try to do three things at once. I'm actually feeling better emotionally than I thought I would. It sucks, a lot, but everyone has been wonderful and neighbours and so on are stepping up to help out. Of all the ways my Dad could go, he didn't want it drawn out, so there are more than a few silver linings here. A lot of people have to see their parents waste away from cancer over years, or to Alzheimers, so we've been saved that. I expect I will start missing him more in a few months, when his absence has a proper shape to it, but a friend of mine wisely said not to make any big decisions, so I am keeping my head down and pushing on with practical help for the situation. The good news is the most urgent funeral preparation was finished last night, so now everything is lined up. Getting everything sorted needed focus and energy, but now we just have to do picking family up from airports and stuff, and the actual day.

Since most hydrus users are going to be on the young side, and most will not have lost their parents yet, the advice I am going to give from now on is:

- Talk to your parents about what they would want in a funeral
- Make sure you have a will

Having both questions sorted ahead of time has been deeply useful during this.

 >>/1371/

Thanks for the detailed reply. I'll make sure to check out Hydrus Companion and whether it works with Tor. Yes, generally I'd also just move on to a more available source, but for some things that's difficult.

I'm glad to hear that updating is that simple and the db is cleanly divided from the actual program. I set out updates since I messed up once and read about the big update situation. But I'll try that again, one version at a time to be sure and see how it goes.

 >>/1372/

True. It helps a lot to have things worked out and off your mind. I can only recommend to take some dedicated time to face the emotional trouble inside you and go through with it, no distraction or pushing it away. You'll cry like a bitch, but you will feel a lot better afterwards. Unresolved trauma will eat you up from the inside. It's like getting a tooth pulled at the dentist, painful but so much relief. Don't worry, you're built to endure that stuff.

 >>/1372/
> Sorry for the trouble with the new release. Some users are getting several crashes, and I think the new mpv is the culprit. Since you are on Windows 7, I suspect the new mpv is incompatible with it in the same way things like Qt6 are too. Can you try going in to your install directory and deleting the 'mpv-2.dll' file? You should still have mpv-1.dll from a previous install, and that should work on its own.

Ok I removed the mpv-2.dll and it was indeed causing the problem of not being able to play animated files(gifs, webms, mp4, etc.). They play normally now and I don't get the "MPV is not available" errors anymore. 

As for the crashes I can't remember what tags I used to cause them now lol however, I feel like it was more related to mpv-2 but I'll continue to be on the lookout. The crash reports doesn't really tell you much. Just normal boots and exists and doesn't even mention it crashed other than after booting Hydrus and saying
 
> "Found and deleted the durable temporary database on boot. The last exit was probably not clean."

And I don't see any "traceback" either in the logs nor in the db folder.

I had a good week-and-a-bit getting back to schedule. I have fixed some critical bugs, including the crashy mpv problem on Windows, cleaned a bunch of code, and improved regular tag searching with new tools, cleaner logic, and automatic sibling substitution.

The release should be as normal tomorrow, probably a little early. Windows Qt5 users will have special install instructions.