/hydrus/ - Hydrus Network

- code cleanup:
- about 80KB of code moved out of the main ClientDB.py file:
- refactored all combined files display mappings cache code from the code database to a new database module
- refactored all combined files storage mappings cache code from the code database to a new database module
- refactored all specific storage mappings cache code from the code database to a new database module
- more misc refactoring of tag count estimate, tag search, and other code down to modules
- hooked up specific display mappings cache to the repair system correctly--it had been left unregistered by accident
- some misc duplicate action options code cleanup
- migrated some ancient pause states--repository, subscriptions, import&export folders--to the newer options structure
- migrated the image and thumbnail cache sizes to the newer options structure
- removed some ancient db and dialog code from the retired dumper system

next week

I want to catch up on some github issues and do a little more multiple local file services work.

I was ill for the start of the week and am short on work time. Rather than put out a slim release, I will spend tomorrow doing some more normal work and put the release off a week.

483 should be on the 4th of May. Thanks everyone!

I had a good couple of weeks. There are a variety of small fixes and quality of life improvements and the first version of 'multiple local file services' is ready for advanced users to test.

The release should be as normal tomorrow.

- cleaned up some old focus test logic that weas used when hovers were separate windows
- rewrote how each note panel in the new hover is stored. a bunch of sizing and event handling code is less hacked
- significantly improved the accuracy of the 'how high should the note window be?' calculation, so notes shouldn't spill over so much or have a bunch of greyspace below
- right- or middle-clicking a note now hides its text. repeat on its name to restore. this should persist through an edit, although it won't be reflected in the background atm. let's see how it works as a simple way to quickly browse a whole stack of big notes
- a new 'notes' option panel lets you choose if you want the text caret to start at the beginning or end of the document when editing
- you can now double-click a note tab in 'edit notes' to rename the note. some styles may let you double-click in note greyspace to create a new note, but not all will handle this (yet)
- as an experiment, all the buttons on the media viewer hover windows now do not take focus when you click them. this should let you, for instance, click a duplicate filter processing button and then use the arrow keys and space to continue to navigate. previously, clicking a button would focus it, and navigation keys would be intercepted to navigate the 'form' of the buttons on the hover window. you can still focus buttons with tab. if this affects you, let me know how this goes!
- .
- misc:
- added checkboxes to _options->gui pages_ to control whether ctrl- and shift- selects will highlight media in the preview viewer. you can choose to only do it for files with no duration if you prefer
- the 'advanced mode' tag autocomplete dropdown now has 'OR' and 'OR*' buttons. the former opens a new empty OR search predicate in the edit dialog, the latter opens the advanced text parser as before
- the edit OR predicate panel now starts wider and with the text box having focus
- hydrus is now more careful about deciding whether to make a png or a jpeg thumbnail. now, only thumbnails that have an alpha channel with interesting data in it are saved to png. everything else is jpeg
- when uploading to a repository, the client will now slow down or speed up depending on how fast things are going. previously it would work on 100 mappings at a time with a forced 0.1s wait, now it can vary between 1-1,000 weight
- just to be clean, the current files line on the file history chart now initialises at 0 on your first file import time
- fixed a bug in 'if file is missing, remove record' file maintenance job. if none of the files yet scanned had any urls, it could error out since the 'missing and invalid files' directory was yet to be created
- linux users who seem to have mpv support yet are set to use the native viewer will get a one-time popup note on update this week just to let them know that mpv is stable on linux now and how to give it a go
- the macOS App now spits out any mpv import errors when you hit _help->about_, albeit with some different text around it
- I maybe fixed the 'hold shift to not follow a dragged page' tech for some users for whom it did not work, but maybe not
- thanks to a user, the new website now has a darkmode-compatible hydrus favicon
- all file import options now expose their new 'destination locations' object in a new button in the UI. you can only set one destination for now ('my files', obviously), but when we have multiple local file services, you will be able to set other/multiple destinations here. if you set 'nothing', the dialog will moan at you and stop you from ok-ing it.
- I have updated all import queues and other importing objects in the program to pause their file work with appropriate error messages if their file import options ever has a 'nothing' destination (this could potentially happen if future after a service deletion). there are multiple layers of checks here, including at the final database level
- misc code cleanup

- client api:
- added 'create_new_file_ids' parameter to the 'file_metadata' call. this governs whether the client should make a new database entry and file_id when you ask about hashes it has never seen before. it defaults to false, which is a change on previous behaviour
- added help talking about this
- added a unit test to test this
- added archive timestamp and hash hex sort enum definitions to the 'search_files' client api help
- client api version is now 31

next week

Next week is cleanup. Nothing too exciting, but I'd like to break the database code up a bit more.

I had a great week of simple work. I fixed some bugs--including the scrolled taglist selection issue--and improved some quality of life.

The release should be as normal tomorrow.

full list

- file notes and media viewer hover windows:
- file notes are now shown on the media viewer! this is a first version, pretty ugly, and may have font layout bugs for some systems, but it works. they hang just below the top-right hover, both in the canvas background and with their own hover if you mouseover. clicking on any note will open 'edit notes' on that note
- the duplicate filter's always-on hover _should_ slide out of the way when there are many notes
- furthermore, I rewrote the backend of hover windows. they are now embedded into the media viewer rather than being separate frameless toolbar windows. this should relieve several problems different users had--for instance, if you click a hover, you now no longer lose focus on the main media viewer window. I hacked some of this to get it to work, but along the way I undid three other hacks, so overall it should be better. please let me know how this works for you!
- fixed a long time hover window positioning bug where the top-right window would sometimes pop in for a frame the first time you moved the mouse to the top middle before repositioning and hiding itself again
- removed the 'notes' icon from the top right hover window
- refactored a bunch of canvas background code
- .
- client api:
- search_files/get_thumbnail now returns image/jpeg or image/png Content-Type. it _should_ be super fast, but let me know if it lags after 3k thumbs or something
- you can now ask for CBOR or JSON specifically by using the 'Accept' request header, regardless of your own request Content-Type (issue #1110)
- if you send or ask for CBOR but it is not available for that client, you now get a new 'Not Acceptable' 406 response (previously it would 500 or 200 but in JSON)
- updated the help regarding the above and wrote some unit tests to check CBOR/JSON requests and responses
- client api version is now 30
- .
- misc:
- added a link to 'Hyshare', at https://github.com/floogulinc/hyshare, to the Client API help. it is a neat way to share galleries with friends, just like the the old 'local booru'
- building on last week's shift-select improvement, I tweaked it and shift-select and ctrl-select are back to not setting the preview focus. you can ctrl-click a bunch of vids in quick silence again
- the menu on the 'file log' button is now attached to the downloader page lists and the menu when you right-click on the file log panel. you can now access these actions without having to highlight a big query
- the same is also true of the search/check log!
- when you select a new downloader in the gallery download page, the keyboard focus now moves immediately to the query text input box
- tweaked the zoom locking code in the duplicate filter again. the 'don't lock that way if there is spillover' test, which is meant to stop garbage site banners from being hidden just offscreen, is much more strict. it now only cares about 10% or so spillover, assuming that with a large 'B' the spillover will be obvious. this should improve some odd zoom locking situations where the first pair change was ok and the rest were weird
- if you exit the client before the first session loads (either it is really huge or a problem breaks/delays your boot) the client will not save any 'last/exit session' (previously, it was saving empty here, requiring inconvenient load from a backup)
- if you have a really really huge session, the client is now more careful about not booting delayed background tasks like subscriptions until the session is in place
- on 'migrate database', the thumbnail size estimate now has a min-max range and a tooltip to clarify that it is an estimate
- fixed a bug in the new 'sort by file hash' pre-sort when applying system:limit

next week

I would like to push multiple local file services some more. Probably some more infrastructure work in delete and import UI.

I had a great week. I fixed some problems with last week's hover window rewrite, improved the notes display and editing, and did a mix of other small work.

The release should be as normal tomorrow.

- when you hit 'copy->hash' on a file right-click, it now shows the sha256 hash for quick review
- in the duplicate filter, the zoom locking tech now works better™ when one of the pair is portrait and the other landscape. it now tries to select either width or height to lock both when going AB and BA. it also chooses the 'better' of width or height by choosing the zoom that'll change the size less radically. previously, it could do width on AB and height on BA, which lead to a variety of odd situations. there are probably still some issues here, most likely when one of the files almost exactly fills the whole canvas, so let me know how you get on
- webps with transparency should now load correct! previously they were going crazy in the transparent area. all webps are scheduled a thumbnail regen this week
- when import folders run, the count on their progress bar now ignores previous failed and ignored entries. it should always start 0, like 0/100, rather than 20/120 etc...
- when import folders run, any imports where the status type is set to 'leave the file alone' is now still scanned at the end of a job. if the path does not exist any more, it is removed from the import list
- fixed a typo bug in the recent delete code cleanup that meant 'delete files after export' after a manual export was only working on the last file in the selection. sorry for the trouble!
- the delete files dialog now starts with keyboard focus on the action radiobox (it was defaulting to ok button since I added the recent panel disable tech)
- if a network job has a connection error or serverside bandwidth block and then waits before retrying, it now checks if all network jobs have just been paused and will not reattempt the connection if so (issue #1095)
- fixed a bug in thumbnail fallback rendering
- fixed another problem with cloudscraper's new method names. it should work for users still on an old version
- wrote a little 'extract version' sql and bat file for the db folder that simply pull the version from the client.db file in the same directory. I removed the extract options/subscriptions sql scripts since they are super old and out of date, but this general system may return in future
- .
- file history chart:
- added 'archive' line to the file history chart. this isn't exactly (current_count - inbox_count), but it pretty much is
- added a 'show deleted' checkbox to the file history chart. it will recalculate the y axis range on click, so if you have loads of deleted files, you can now hide them to see current better
- improved the way data is aggregated in the file history chart. diagonal lines should be reduced during any periods of client import-inactivity, and spikes should show better
- also bumped the number of steps up to 8,000, so it should look nice maximised on a 4k
- the file history chart now remembers its last size and position--it has an entry under options->gui
- .
- client api:
- thanks to a user, the Client API now accepts any file_id, file_ids, hash, or hashes as arguments in any place where you need to specify a file or files
- like 'return_hashes', the 'search_files' command in the Client API now takes an optional 'return_file_ids' parameter, default true, to turn off the file ids if you only want hashes
- added 'only_return_basic_information' parameter, default false, to 'get_metadata' call, which is fast for first-time requests (it is slim but not well cached) and just delivers the basics like resolution and file size
- added unit tests and updated the help to reflect the above
- client api version is now 29
- .
- help:
- split up the 'more files' help section into 'powerful searching' and 'exporting files', both still under the 'next steps' section
- moved the semi-advanced 'OR' section from 'tags' to 'searching'
- brushed up misc help
- a couple of users added some misc help updates too, thank you!

- misc boring cleanup:
- cleaned up an old wx label patch
- cleaned up an old wx system colour patch
- cleaned up some misc initialisation code

next week

Next week is a medium sized job week. I would like move the 'notes' system forward. Top priority is to get some preview of notes on the media viewer, next to think about is duplicate file note merging and parsing notes from sites.

I had a good week. There's a simple first version of showing notes in the media viewer and several quality of life UI improvements.

The release should be as normal tomorrow.

- the 'set_notes' command now checks the type of the notes Object. it obviously has to be string-to-string
- the 'get_thumbnail' command should now never 404. if you ask for a pdf thumb, it gives the pdf default thumb, and if there is no thumb for whatever reason, you get the hydrus fallback thumbnail. just like in the client itself
- updated client api help to talk about these
- updated the unit tests to handle them too
- did a pass over the client api help to unify indent style and fix other small formatting issues
- client api version is now 28

next week

I am feeling good about multiple local file services. Most of the cleanup this week was for that, and now there are only about three things left to do before we can start playing with it for real--UI and some importer code to handle imports to multiple locations, UI to present deletes and undeletes for multiple locations, and UI and db code to do move/copy across locations. I'll push on these in the coming weeks.

Next week will be a 'small jobs' week, and I would like to catch up on github issues in particular.

I had an ok week with a variety of work. I fixed some bugs, tweaked some UI (including a neat change to shift+select of thumbnails), improved the new file history chart, and added to the Client API.

The release should be as normal tomorrow.

full list

- misc:
- the network engine now parses the 'last-modified' response header for raw files. if this time is earlier than any parsed source time, it is used as the source time and saved to the new 'domain modified time' system. this provides decent post time parsing for a bunch of sites by default, which will also help for subscription timing and similar
- to get better apng duration, updated the apng parser to count up every frame duration separately. previously, if ffmpeg couldn't figure it out, I was just defaulting to 24 fps and estimating. now it is calculated properly, and for variable framerate apngs too. all apngs are scheduled for a metadata regen this week. thanks to the user who submitted some long apngs where this problem was apparent
- fixed a bug in the network engine filter that figures out url class precedence. url classes with more parameters were being accidentally sorted above those with more path components, which was messing with some url class matching and automatic parser linking
- improved the message when an url class fails to match because the given url has too few path components
- fixed a time delta display bug where it could say '2 years, 12 months' and similar, which was due to a rounding issue on 30 day months and the, for example, 362nd day of the year
- fixed a little bug where if you forced an archive action on an already archived file, that file would appear to get a fake newer archived timestamp in UI until you restarted
- updated the default nitter parsers to pull a creator tag. this seemed to not have been actually done when previously thought
- the image renderer now handles certain broken files better, including files truncated to 0 size by disk problem. a proper error popup is made, and file integrity and rescan jobs are scheduled
- .
- file history chart:
- for a long time, a user has been generating some cool charts on file history (how many files you've had in your db over time, how many were deleted, etc...) in matplotlib. you may have run his script before on your own database. we've been talking a while about integrating it into the client, and this week I finally got around to it and implemented it in QtCharts. please check out the new 'view file history' underneath Mr Bones's entry in the help menu. I would like to do more in this area, and now I have learned a little more about QtCharts I'd like to revisit and polish up my old bandwidth charts and think more about drawing some normal curves and so on of other interesting data. let me know what you think!
- I did brush up a couple things with the bandwidth bar chart already, improving date display and the y axis label format
- .
- client api:
- a user has written several expansions for the client api. I really appreciate the work
- the client api now has note support! there is a new 'add notes' permission, 'include_notes' parameter in 'file_metadata' to fetch notes, and 'set_notes' and 'delete_notes' POST commands
- the system predicate parser now supports note system preds
- hydrus now supports bigger GET requests, up to 2 megabytes total length (which will help if you are sending a big json search object via GET)
- and the client api now supports CBOR as an alternate to JSON, if requested (via content-type header for POST, 'cbor' arg for GET). CBOR is basically a compressed byte-friendly version of JSON that works a bit faster and is more accessible in some lower level languages
- cbor2 is now in the requirements.txt(s), and about->help shows it too
- I added a little api help on CBOR
- I integrated the guy's unit tests for the new notes support into the main hydrus test suite
- the client api version is now 27
- I added links to the client api help to this new list of hydrus-related projects on github, which was helpfully compiled by another user: https://github.com/stars/hydrusnetwork/lists/hydrus-related-projects

next week

Next week is cleanup. I will focus on clearing out old code, particularly in file handling for multiple local file services.

hydev file history,... png
(21.66 KB, 1033x762)

Here's my chart! Two notes:

1) My inbox is a little higher than it should be because I rescheduled everything for a second scan some years ago.
2) Guess when 'many queries on one subscription' was added. :^)

I had a good week. I cleaned a heap of code, fixed some bugs, brushed up the new file history graph, and wrote some small extensions to the Client API.

The release should be as normal tomorrow.

full list

- domain modified times
- the downloader now saves the 'source time' (or, if none was parsed, 'creation time') for each file import object to the database when a file import is completed. separate timestamps are tracked for every domain you download from, and a file's number can update to an earlier time if a new one comes in for that domain
- I overhauled how hydrus stores timestamps in each media object and added these domain timestamps to it. now, when you see 'modified time', it is the minimum of the file modified time and all recorded domain modified times. this aggregated modfified time works for sort in UI and when sorting before applying system:limit, and it also works for system:modified time search. the search may be slow in some situations--let me know
- I also added the very recent 'archived' timestamps into this new object and added sort for archived time too. 'archived 3 minutes ago' style text will appear in thumbnail right-click menus and the media viewer top status text
- in future, I will add search for archive time; more display, search, and sort for modified time (for specific domains); and also figure out a dialog so you can manually edit these timestamps in case of problems
- I also expect to write an optional 'fill in dummy data' routine for the archived timestamps for files archived before I started tracking these timestamps. something like 'for all archived files, put in an archive time 20% between import time and now', but maybe there is a better way of doing it, let me know if you have any ideas. we'll only get one shot at this, so maybe we can do a better estimate with closer analysis
- in the longer future, I expect import/export support for this data and maintenance routines to retroactively populate the domain data based on hitting up known urls again, so all us long-time users can backfill in nicer post times for all our downloaded files
- .
- searching tags on client api:
- a user has helped me out by writing autocomplete tag search for the client api, under /add_tags/search_tags. I normally do not accept pull requests like this, but the guy did a great job and I have not been able to fit this in myself despite wanting it a lot
- I added some bells and whistles--py 3.8 support, tag sorting, filtering results according to any api permissions, and some unit tests
- at the moment, it searches the 'storage' domain that you see in a manage tags dialog, i.e. without siblings collapsed. I can and will expand it to support more options in future. please give it a go and let me know what you think
- client api version is now 26
- .
- misc
- when you edit something in a multi-column list, I think I have updated every single one so the selection is preserved through the edit. annoyingly and confusingly on most of the old lists, for instance subscriptions, the 'ghost' of the selection focus would bump up one position after an edit. now it should stay the same even if you rename etc... and if you have multiple selected/edited
- I _think_ I fixed a bug in the selected files taglist where, in some combination of changing the tag service of the page and then loading up a favourite search, the taglist could get stuck on the previous tag domain. typically this would look as if the page's taglist had nothing in it no matter what files were selected

- if you set some files as 'alternates' when they are already 'duplicates', this now works (previously it did nothing). the non-kings of the group will be extracted from the duplicate group and applied as new alts
- added a 'BUGFIX' checkbox to 'gui pages' options page that forces a 'hide page' signal to the current page when creating a new page. we'll see if this patches a weird error or if more work is needed
- added some protections against viewing files when the image/video file has (incorrectly) 0 width or height
- added support for viewing non-image/video files in the duplicate filter. there are advanced ways to get unusual files in here, and until now a pdf or something would throw an error about having 0 width

next week

Back to multiple local file services, which is in endgame. I have a ton of ancient file handling code to simply clean to newer standards.

I had a good week. As well as some general fixes and more file modified time work, there is also a user-written expansion to the Client API that adds 'notes' editing support and a bit of fun.

The release should be as normal tomorrow.

- I added a list of some quick links back in to the main index page of the help
- I wrote an unlinked 'after_distaster' page for the help that collects my 'ok we finished recovering your broken database, now use your pain to maintain a backup in future' spiel, which I will point people to in future
- .
- misc:
- fixed a bug where changes to the search space in a duplicate filter page were not sticking after the first time they were changed. this was related to a recent 'does page have changes?' optimisation--it was giving a false negative for this page type (issue #1079)
- fixed a bug when searching for both 'media' and 'preview' view count/viewtime simultaneously (issue #1089, issue #1090)
- added support for audio-only mp4 files. these would previously generally fail, sometimes be read as m4a. all m4as are scheduled for a metadata regen scan
- improved some mpeg-4 container parsing to better differentiate these types
- now we have great apng detection, all pngs with apparent 'bitrate' over 0.85 bits/pixel will be scheduled for an 'is this actually an apng?' scan. this 0.85 isn't a perfect number and won't find extremely well-compressed pixel apngs, but it covers a good amount without causing a metadata regen for every png we own
- system:hash now supports 'is' and 'is not', if you want to, say, exclude a list of hashes from a search
- fixed some 'is not' parsing in the system predicate parser
- when you drag and drop a thumbnail to export it from the program, the preview media viewer now pauses that file (just as the full media viewer does) rather than clears it
- when you change the page away while previewing media with duration, the client now remembers if you were paused or playing and restores that state when you return to that page
- folded in a new and improved Deviant Art page parser written by a user. it should be better about getting the highest quality image in unusual situations
- running a search with a large file pool and multiple negated tags, negated namespaces, and/or negated wildcards should be significantly faster. an optimisation that was previously repeated for each negated tag search is now performed for all of them as a group with a little inter-job overhead added. should make '(big) system:inbox -character x, -character y, -character z' like lightning compared to before
- added a 'unless namespace is a number' to 'tag presentation' options, which will show the full tag for tags like '16:9' when you have 'show namespaces' unticked
- altered a path normalisation check when you add a file or thumbnail location in 'migrate database'--if it fails to normalise symlinks, it now just gives a warning and lets you continue. fingers crossed, this permits rclone mounts for file storage (issue #1084)
- when a 'check for missing/invalid file' maintenance job runs, it now prints all the hashes of missing or invalid files to a nice simple newline-separated list .txt in the error directory. this is an easy to work with hash record, useful for later recovery
- fixed numerous instances where logs and texts I was writing could create too many newline characters on Windows. it was confusing some reader software and showing as double-spaced taglists and similar for exported sidecar files and profile logs
- I think I fixed a bug, when crawling for file paths, where on Windows some network file paths were being detected incorrectly as directories and causing parse errors
- fixed a broken command in the release build so the windows installer executable should correctly get 'v475' as its version metadata (previously this was blank), which should help some software managers that use this info to decide to do updates (issue #1071)

- some cleanup:
- replaced last instances of EVT_CLOSE wx wrapper with proper Qt code
- did a heap of very minor code cleanup jobs all across the program, mostly just to get into pycharm
- clarified the help text in _options->external programs_ regarding %path% variable
- .
- pycharm:
- as a side note, I finally moved from my jank old WingIDE IDE to PyCharm in this release. I am overall happy with it--it is clearly very powerful and customisable--but adjusting after about ten or twelve years of Wing was a bit awkward. I am very much a person of habit, and it will take me a little while to get fully used to the new shortcuts and UI and so on, but PyCharm does everything that is critical for me, supports many modern coding concepts, and will work well as we move to python 3.9 and beyond

next week

The past few months have been messy in scheduling as I have dealt with some IRL things. That's thankfully mostly done now, so I am now returning to my old schedule of cleanup/small/medium/small week rotation.

Next week will be a 'medium size' job week. I'm going to lay the groundwork for 'post time' parsing in the downloader and folding that cleverly into 'modified date' for searching and sorting purposes. I am not sure I can 'finish' it, but we'll see.

I had a good week bringing two neat new features: a user has implemented tag autocomplete search for the Client API, and I managed to get 'post time' saving from the downloader system to the database and augmenting 'modified time'.

The release should be as normal tomorrow.

Anyone here know what's wrong with my Hydrus Companion?
Sending tabs from Gelbooru it says it sends them successfully, but Hydrus says 400 Unknown Error.

Hey, I finally moved to new programming software this week (WingIDE to PyCharm). It was a little jarring dealing with new UI and shortcuts and a billion new settings, so I mostly stuck to simple code cleanup to get to grips with it. I don't have much exciting in my changelog beyond a tweak to system:hash, so rather than put out a thin build tomorrow, I will do some more work instead and move the release on a week.

If you haven't seen it, though, please check out the new help a user put together: https://hydrusnetwork.github.io/hydrus/

It has nice features like search and tables of contents and will be easier to edit in future. 475 will have it too for the local copy and should be out on the 2nd of March.

 >>/1235/
If you check your 'client - date.log' file in the install_dir/db folder, and go to the bottom and look for the timestamp when this happened (and maybe ctrl+f for "400" or "unknown error", whatever text you have), can you see any 'traceback' information for the errors here? Can you clip and pastebin them or just post here? That will show more information on what is actually failing here.

If you are getting popups in the client UI, it should let you just copy the traceback there, too.

 >>/1236/
I go to the hydrus page by typing hy and then enter. going there it's nicer than it was before. however it took quite a while for me to find the links to the various places hydrus is, would it be possible to copy at least the downloads and links to the places you make update posts to the homepage?

 >>/1237/
Thank you, yes, I will. When we collapsed it down a level, the old index with the links go removed. I'll add something back in.

I had a good couple of weeks. I updated my behind the scenes environment and cleared out a wide variety of misc work--bug fixes and little improvements. The built release will also get a local copy of the nice new help a user put together.

The release should be as normal tomorrow.