/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.


New Reply on thread #1354
X
Max 20 files0 B total
[New Reply]

[Index] [Catalog] [Banners] [Logs]
Posting mode: Reply [Return]


https://youtube.com/watch?v=D5T9RByz9i8
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Linux.Qt6.-.Executable.tar.gz

I had a good week. I got started on some long-delayed serverside janitor improvements, and I got so stuck it in was all I worked on! This release has nothing important for regular users, so if you are not involved in running a hydrus server, you can skip it entirely!

server stuff

Only for server admins and janitors!

Improving the janitor workflow is now my 'big job' for the rest of the year. This is the first step of what I'd like to fit in to spare days over the coming weeks and months.

This week I update the repositories so they cache counts of their various metadata. They can now quickly say 'I have 2,124,543 mappings' and so on. If you have a large server, it will take a few minutes to update as it counts everything up for the first time.

Once you are booted, make sure you are in help->advanced mode, and on review services you will have a new 'service info' button. Click it and you can see all the numbers, including the current lengths of the petition queues. Anyone with an account can see these for now. If you want more privacy, I can figure something out, but tbh I think it is probably good for users to be able to see everything here.

The numbers in the petition processing page are fed by this too. No longer will it manually count up and max out at 1,000 petitions--it will deliver the actual number real fast.

Also, a sibling petition can now have both ADD and DELETE rows. This happens if the same account gives the same reason (like 'replacing a->b with a->c') to a sibling petition and a sibling pend. You now see those together, with that shared reason, and action it as one item.

I suspect we'll need some more UI clientside to encourage using the same reason, but for now I have updated manage tag siblings to give the same 'reason' when you replace an existing sibling. Previously, this is where it would give a 'AUTO-CONFLICT...' style reason. Now, those things should be bundled into the same thing you see.

This stuff changes some of the hydrus network protocol. Normally, I would update the network version number, but that requires everyone to update. Since this only affects advanced users, and I expect I'll be doing more in coming weeks, I am not updating the version number. An old client will run into errors if it tries to pull petitions from a new server, but I think a new client will be able to work with an old server. In any case, if you are a server admin or janitor, please update your clients and servers at roughly the same time this week, or you'll get some harmless but annoying parse/404 errors.

As a side thing, as a server admin, if the service info numbers ever get borked, please hit 'regen service info' in your 'administrate services' menu. I've added extensive testing this week to ensure the update routines are mostly good, but I'm sure there are some complicated situations where the counting logic is dodgy. Let me know how you get on with it!
full list

almost all the changes this week are only important to server admins and janitors. regular users can skip updating this week
- overview:
- the server has important database and network updates this week. if your server has a lot of content, it has to count it all up, so it will take a short while to update. the petition protocol has also changed, so older clients will not be able to fetch new servers' petitions without an error. I think newer clients will be able to fetch older servers' ones, but it may be iffy
- I considered whether I should update the network protocol version number, which would (politely) force all users to update, but as this causes inconvenience every time I do it, and I expect to do more incremental updates here in coming weeks, and since this only affects admins and janitors, I decided to not. we are going to be in awkward flux for a little bit, so please make sure you update privileged clients and servers at roughly the same time
- .
- server petition workflow:
- the server now maintains an ongoing fast count of its various repository metadata, such as 'number of mappings' and 'number of petitions of type x'. when you fetch petition counts, no longer will it count live and max out at 1,000, it'll give you good full numbers every time, and real fast
- you can see the current numbers from the new 'service info' button on review services, which only appears in advanced mode. any user with an account key can see these numbers, which include number of petitions in the queue. I can make this more private if you like, but for now I think it is good if advanced users can see them all
- in the petition processing page, sibling and parent petitions will now include both delete and add rows if the account and reason are the same. I'm aiming to get better 'full' coverage of a replace petition, so you can see and approve/deny both the add and the remove parts in one go. for fetching, these combined petitions count as 'delete' petitions, and won't appear in the 'add' petition queue
- when users encounter an automatic conflict resolution in the manage siblings dialog, those auto-petitioned pairs are now assigned the same reason as the original conflicting pended pairs. they _should_ show up together in the new petition processing UI
- as part of this, sibling and parent petitions are no longer filtered by namespace. you will see everything with that same account and reason in one go. let's try it out, and if it is too much, I will add filters clientside or something. since we are now starting to see add and remove together, we'll want to at least have the option to see everything

- boring server stuff:
- the petition object is updated to handle multiple actions per petition, and the clientside petition UI is updated appropriately
- the server tracks 'actionable' petition counts as separate to the number of raw petition rows. some of this was happening before, but the logic is improved, including clever counting of the new petitions that include both add and delete rows
- for when my count-update logic inevitably fails, there is now a 'regen service info' entry in the 'administrate services' menu for all repositories. numbers generated will be printed to server log
- some unusual repo upload logic is cleaned up, e.g. if a user with 'create permission' uploads a sibling or parent, any pending rows for that content will now be properly cleared)
- fixed a stupid swap logical bug where janitors who could only moderate siblings (and not parents) were only being given parent numbers and vice versa
- all server services now respond to /busy check. it requires no authentication and just returns 1 or 0 depending on the current lock state
- fixed a bug where tag siblings or parents that were denied would still make a new definition record for the child/bad tag
- with all the fine number changes, fleshed out the server unit tests with more examples of submitting and altering content and then checking for numbers afterwards. now checked are: file add, file admin delete, mapping add, mapping admin delete, mapping petition, mapping petition approve+deny, parent add, parent admin delete, parent pend, parent pend approve+deny, parent petition, parent petition approve+deny
- significant refactoring of the tail end of server content update pipeline. more things now go through logic-harmonised update methods that ensure count is reliable
- did some misc server db and constant enum code cleanup
- .
- misc:
- to match the new change in the server, in the client, tag and rating services now store their 'num_files' service info count as the new 'num_file_hashes'. existing numbers will be converted over during update
- fixed a probably ten year old bug where 'num pending/petitioned files' had the same enum as 'num pending/petitioned mappings'. never noticed, since no service has done both those things
- if the upload pending process fails due to an unusual permission error or similar, the pending menu should now recover and update itself (previously it stayed greyed out)

next week

Back to small jobs. I had planned to do a little janitor stuff this week and then do regular work, but these number updates killed me and I ended up in a rabbit hole of unit tests making sure everything was good enough. The goods news is I fixed some other long-time server issues in doing that, but the bad is I did nothing else. So, I'll do some small work and try to get to some github issues too.

It would be nice if I could hammer out some final Qt6 problems so we can move to it fully in a couple weeks.


Post(s) action:


Moderation Help
Scope:
Duration: Days

Ban Type:


2 replies | 0 file
New Reply on thread #1354
Max 20 files0 B total