/hydrus/ - Hydrus Network

I had a fantastic week figuring out tech to make the 'manage tag siblings' dialog boot and work faster, even if you have hundreds of thousands of pairs. I now have a 98% working prototype that does everything--and a few workflow improvements--without needing to load up all the pairs on every boot. Rather than do a release tomorrow, I will copy this work to the manage parents dialog, polish the whole thing, and catch up on other work besides for next week.

v585 should be out on the 7th. Thanks everyone!

Screenshot... png
(31.34 KB, 535x671)

I'm getting the following error on Windows 11, and as far as I can tell it's because https://ptr.hydrus.network:45871/ responds with an expired HTTPS certificate (but maybe that's just when the requests come from a web browser - missing header or something?)

I've just upgraded according to instructions from an old version to v584, if it's relevant.  Everything appears to work besides new connections to PTR, and my browser reports invalid certs for that too. 

Traceback:

v584, win32, frozen
CancelledException
Download cancelled: This hydrus service "hydrus service: public tag repository" could not do work because: Service has had a recent error or is otherwise not functional! You might like to try refreshing its account in 'review services'. Specific error was: Problem with SSL: SSLError(MaxRetryError("HTTPSConnectionPool(host='ptr.hydrus.network', port=45871): Max retries exceeded with url: /session_key (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))")) - next request in 13 hours 53 minutes
Traceback (most recent call last):
  File "hydrus\client\gui\ClientGUIAsync.py", line 61, in _doWork
  File "hydrus\client\gui\services\ClientGUIClientsideServices.py", line 2651, in work_callable
  File "hydrus\client\ClientServices.py", line 1416, in SyncAccount
  File "hydrus\client\ClientServices.py", line 1296, in Request
  File "hydrus\client\networking\ClientNetworkingJobs.py", line 2061, in WaitUntilDone
hydrus.core.HydrusExceptions.CancelledException: Download cancelled: This hydrus service "hydrus service: public tag repository" could not do work because: Service has had a recent error or is otherwise not functional! You might like to try refreshing its account in 'review services'. Specific error was: Problem with SSL: SSLError(MaxRetryError("HTTPSConnectionPool(host='ptr.hydrus.network', port=45871): Max retries exceeded with url: /session_key (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))")) - next request in 13 hours 53 minutes


This error appears regardless of which specific HTTP request is being issued. GET accounts, session_key, etc. In the browser, I get the same certificate even when connecting with a VPN. The browser receives the following cert:
Certificate	a0805523c271615bf581779a9830a1c3eab0993136edeb532b866cae61e88616
Public Key	28da14961bff00aca499c34dd7eacb108bb2473231e536a67db69bcdbcff7506
(see attached image)

Then issue persists even when I build from source instead of use most recent build.

 >>/1671/
Next step I guess is to poke around in the code and see what / where it is getting the root CA list, what cert it is actually receiving from ptr.hydrus.network (same expired cert as browser or different one). 

Would appreciate any tips / if you have something I could easily run. I see certifi in the dependency chain, but I haven't checked what actually happens in the application.

 >>/1671/
 >>/1672/
Thank you, this is an interesting report. Since most hydrus repositories are run with a self-signed http cert (since they are hosted off of local networks and such), hydrus is set to not actually verify hydrus traffic. i.e. In the 'requests' Session, I explicitly set 'verify=False'.

https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/networking/ClientNetworkingSessions.py#L228

It is normal to have to add an exception if you look at a repo's landing page html with your browser. The PTR has usually been an exception since it had a real cert.

So this is odd. Even if the PTR Let's Encrypt cert has expired, hydrus itself shouldn't care since I tell it to use it anyway. You are the only report of this, and when I extract a fresh extract, I can get it to talk to the PTR with no problems, and since you get it with source too, that suggests there is something else in your environment going on here.

This part, 'unable to get local issuer certificate', I wonder what that really means. I think that 'requests' might in some situations ask your OS for some root cert stuff, so is there any chance you have an unusual setup for Windows itself? I assume you can download from normal https sites fine with hydrus, and it is just the PTR that has trouble? So I wonder if somehow requests is going 'hey OS, please give me your root cert, I want to see if this hydrus cert is ok', and the OS is going 'uh, that hydrus cert violates my policy, I refuse to give you my root cert', or something like that, so OS policy is overriding hydrus policy.

The requests root cert is under 'install_dir/certifi' folder I think. It is just this: https://pypi.org/project/certifi/ I wouldn't have thought it would need to poke your OS given it has that, but perhaps there's something else going on. Do you have that 'cacert.pem' file in that folder, or is it possible some anti-virus or something has quarantined it? I would have thought that setting 'verify = False' to the requests Session would skip all this shit, but perhaps not.

Screenshot... png
(55.24 KB, 342x435)

 >>/1673/
I've excluded the Hydrus Network directory from Windows Defender, to no effect. The cacert.pem file is in place, haven't checked what's actually being read yet but it seems normal.

I don't have any special environment that's relevant to this as far as I know, but I am a software engineer so there could feasibly be something. Only reason I'm not 100% sure there's an environment issue is because it worked fine with the old version, even though it was years old, and the rest of the application worked fine after update. So presumably in that time there was some dependency change that now interacts with my environment, that was isolated in the past.

It can fetch HTTPS URLs, such as https://endchan.net/hydrus/res/1669.html , fine via debug->network actions->fetch a url, and it can download and import files fine via downloaders, from https urls such as https://danbooru.donmai.us/posts/3579168 

Only PTR seems to be the problem. I've tried removing the PTR service entirely and re-adding it via the standard help menu, too, which had no impact (other than leaving me semi-tagless).

 >>/1674/
Or perhaps there's some really old cached data somewhere that's still being respected? Something outside the Hydrus Network root directory, that I'm not wiping with each new clean install. It's not something in my old database - the problem persists with a clean install on a different drive with no DB.

 >>/1674/
I also can't guess what in my environment would only impact PTR requests. Some weird firewall rule? Something I did to fix Hydrus years ago that is now breaking it? I'm not even 100% sure I'm on the same windows install, and I would  The issue persists on a VPN, so it shouldn't be about my local network.

I'll look into it.

I had an excellent couple of weeks. The manage tag siblings and parents dialogs now load and operate quickly, even when the underlying service has hundreds of thousands of pairs. I also cleared a bunch of normal small work.

The release should be as normal tomorrow.

 >>/1674/
 >>/1675/
 >>/1676/
Yeah, I think you are right that this is probably something outside of hydrus, and/or something related to some borked dll somewhere.

Since it specifically cites '_ssl.c:1006' in the error, I'm pretty sure that is a .pyd file (probably _ssl.pyd either in the install dir or your python environment), or _perhaps_ some Windows dll it is calling. Since you can fetch normal https with valid certificates fine, I'm guessing this is happening because of some obscure security policy that is set in your Windows or something that requests (or the ssl pyd somehow) is force-applying despite me saying verify=false, or indeed if there is some init that happens despite the verify=false and that's triggering something else. I'm afraid I am not expert enough in this to talk too cleverly though.

Let me know what you find out! Fingers crossed, this fixes itself anyway when the guy who hosts the PTR refreshes his cert.

P.S. This is a bit in the weeds, but this is as best as I can immediately see where it is working with the 'verify=False' at the coalface: https://github.com/psf/requests/blob/79b74ef704dc0d804937c0d015c5e9c3b02b79bd/src/requests/adapters.py#L111
There's some other stuff in that same file, including line 409, where perhaps there is a route for a non-nothing cert policy to be set despite verify=False. I remembered that 'requests' is build on urllib3, so perhaps that can set a different cert security policy here.

Of course the correct answer, truly, here, is for me to write a proper management system that saves self-signed certs and lets users approve them manually, and then I'd be setting the cert to verify itself or whatever here, instead of False, although maybe that would still fail on an expired.

Sorry for the trouble!

 >>/1677/
As a follow-up, I was just looking through their docs--I don't suppose you have a cert file set in your environment, do you? I wonder if requests or urllib is pulling that as a fallback and since that value now isn't initialising as None, it leads to the cert policy being 'use the one from the environment'? I think it would be REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE, or perhaps something similar? It would presumably have a value that pointed to a local cert file in your system dir somewhere.

 >>/1678/
You got it, CURL_CA_BUNDLE was set by Postgres. Seems to work now. Thanks!

 >>/1679/
Hell yeah! Glad we could sort it--interesting problem!