/hydrus/ - Hydrus Network

Screenshot 2024-02-24... png
(46.14 KB, 1454x987)

Yeah, this is tricky. I still think your best answer is to wait for me to implement a proper xml/html parser for sidecars, or do the parsing yourself in an external script (e.g. with python) and then convert your html into nicer .txt files that hydrus can suck up easier. You could also filter your undesired tags better there.

If the 'description' string is always in the same location, and that location is always the first or last index of your list of strings, then the 'string selector' might help. It does list slicing like in programming, if you are familiar, like "my_list[4:6]". If you aren't familiar with that, or the description is in the middle of the list here, you are correct in trying a string match, which is basically a filter.

If the description line has a classname in the html, you might be able to exclude it with a string match before you Split/Convert all the html garbage away.

I am not totally sure what you mean by being unable to import by pngs, but if I click 'import->from pngs' (pic related is from the 'sidecars' tab of the 'add tags/urls with the import' dialog after you drop some files on the client), I get a file dialog that allows me to import a png like this. Do you get different?

highdiskusage png
(202.89 KB, 1600x900)

pngparserbug png
(128.77 KB, 1600x900)

Hey,

Sorry that it took me a while to respond. I figured out a solution to my html parsing issue. Apparently, the parser first splits sidecars by the specified string, then passes the split strings onto the postprocessor, breaking the regex splitting I used in the process, before then passing it back to the parser, where it is then split by the string again and saved. I figured out a solution to this problem, but thanks for your help anyways. To answer your questions about the parser import, I am using Linux Mint 21.3 Cinnamon with qt6.5.2, rather than windows, and the pngs do not appear in the window. Maybe the apis are different? I attached a screenshot of this, if that helps.

Thanks for your help!

Ps: why is this client writing TBs of data to my ssd while processing the PTR? I understand if it writes a lot, but multiple TBs (according to some math I did, I haven't gotten that far yet, just ~750gb) still seems a bit excessive. I attached a screenshot of this too.

hydrusdiskwtf png
(199.57 KB, 1600x900)

Update:

(i am using a crucial mx500 500gb with luks and ext4, if that helps)

I was looking at launch options to fix the storage issue, do you think that --no_db_temp_files would help? I have 16gb ram, is that enough?

 >>/1610/
 >>/1611/
 >>/1612/
Yeah, the write is most likely temp storage. I've heard that in some situations, the actual amount written to disk is less since some temp-file magic means some of the shortest-lived data is purged from the write cache before it can actually be committed, but I don't have good numbers on the topic.

The PTR does a lot of database work. The no_db_temp_files parameter will reduce it by a good amount. Not sure how much, but it would reduce it.

Increasing db_transaction_commit_period may also reduce it, I think, since it will reduce the commit frequency (at the cost of increasing per-transaction size, which will strain your temp folder and slow processing time).

A user also wrote this document with a lot of related info, if it helps: https://hydrusnetwork.github.io/hydrus/Fixing_Hydrus_Random_Crashes_Under_Linux.html

I'm not a Linux expert myself, so I can't talk super confidently here. Let me know how you get on!