9qj6dZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z dZdZhdZhdZGdd eZd/dZd0dZd1dZd2dZd3dZd4dZ d5d6d!Zd7d"Zd8d#Zd9d%Zd9d&Zd:d;d*Zdd.Z"dS)?a Download of images (and of any other file) from hosts that don't like being scripted. This module deliberately imports nothing from the project (no constants, no Environment): the bot uses it through download_utils.py, and scripts/download_character_images.py uses it standing on its own, outside the bot's environment. A download goes through these ways, in order, until one of them works: 1. urllib, sending the headers of a browser that is loading an image (see get_browser_headers) 2. curl_cffi, if it is installed: the same request, but with the TLS and HTTP/2 handshake of a real Chrome. A CDN can tell a script from a browser by that handshake alone, whatever the User-Agent header says, and answer 403 to the first one 3. An image proxy, if the IMAGE_PROXY_URL environment variable is set: another server downloads the image instead of this one, which is the way around a CDN that blocks the IP address of this one Why a failure happened is never lost: every way that failed is logged, with its reason. ) annotationsNzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36IMAGE_PROXY_URL>>ceZdZdZdS)NotAnImageErrorzYThe host answered successfully, but with something that isn't an image (e.g. a web page).N)__name__ __module__ __qualname____doc__/app/src/utils/image_fetch.pyr r 4sccccrr urlstrreturndictc 2tj|}|jd|jd}|jdr;|jddd}|rd|d}tdd|d d d d d dd S)a[ Get the headers of a browser loading an image embedded in a web page: not only the User-Agent but everything a real browser sends along with it, since some hosts tell a script from a browser by that combination (and, for hotlink protection, by the Referer) :param url: The url of the file to download :return: The headers z:///zwikia.nocookie.netrzhttps://z .fandom.com/zBimage/webp,image/apng,image/png,image/jpeg,image/*;q=0.8,*/*;q=0.5zen-US,en;q=0.9zA"Google Chrome";v="125", "Chromium";v="125", "Not.A/Brand";v="24"z?0z "Windows"imagezno-corsz cross-site) User-AgentAcceptzAccept-LanguageReferer sec-ch-uasec-ch-ua-mobilesec-ch-ua-platformzSec-Fetch-DestzSec-Fetch-ModezSec-Fetch-Site) urllibparseurlparseschemenetlocendswithpathstripsplitBROWSER_USER_AGENT)r parsed_urlrefererwikis rget_browser_headersr/8s&&s++J";;z'8;;;G !!"6774$$S))//44Q7  43333G)V+X )!#&   rheadbytesboolc|dph|dpS|dp>|dddko |ddd kp"|ddd kp|d S) a0 Check if the first bytes of a file are those of an image format (by its signature, not by the file name or the Content-Type, which a host that is blocking you can set to anything) :param head: The first bytes of the file, at least 12 of them :return: True if it starts like an image ssPNG  )sGIF87asGIF89aNsRIFF sWEBPsftypsBM) startswith)r0s rlooks_like_imager8[s (( " ??/ 0 0 " ??1 2 2 " !H  9D2J'$9 " !9   " ??5 ! ! rcdtj|jdD}d|vr|d|d}|r%tj|dnd}d|vr|ddd n|}tj d d | d }||ks|sitj | d p| d }|r|d|ddn |dd}|dS)a Get the name the file of an image gets when it is saved next to the bot (bundled with it) instead of being downloaded each time. It is made from the url alone, and not from the revision of the file the url may point to, so the same image always gets the same name (https://static.wikia.nocookie.net/onepiece/images/f/f1/Abdullah_Anime_Infobox.png/revision/latest?cb=1 -> Abdullah_Anime_Infobox.jpg) :param url: The url of the image :return: The file name, safe on any file system cg|]}||Srr).0parts r z/get_bundled_image_file_name..xs Q Q QdD QT Q Q QrrrevisionN.rz[^A-Za-z0-9._-]+_z._zutf-8-r5z.jpg)r"r#r$r(r*indexunquotersplitresubr)hashlibsha1encode hexdigest)rparts original_name original_stemstemdigests rget_bundled_image_file_namerTmsT R Qfl33C88=CCCHH Q Q QEU/ J///07<DFL((r333"M7:m7K7KM((a0033Q^M 6%sM : : @ @ F FD }Dm227;;Rszz'?R?RSS]]__)->$%%%%%6#2#;===r str | Nonectjtpd}|sdSd|vrt jdtddS|S)z~ Get the image proxy url template that is set up, if there is one and it is usable :return: The template, or None r@N{url}z Ignoring zg: it has to contain {url}, where the url of the image goes (e.g. https://wsrv.nl/?url={url}&output=jpg))osenvirongetIMAGE_PROXY_ENV_NAMEr)loggingwarning)templates rget_image_proxy_urlr_s|  344:AACCH th I, I I I   t Orc2 ddl}dS#t$rYdSwxYw)zi Check if curl_cffi, which downloads like a real Chrome, is installed :return: True if it is rNTF) curl_cffi Exception)ras r"is_browser_impersonation_availablercs: t uus  error BaseExceptionct|tjjrt |St |jd|S)z\ Describe an error in a line :param error: The error :return: What happened : ) isinstancer"rd HTTPErrorrtyperrds rdescribe_errorrlsD%/005zz5kk" - -e - --rF file_pathtimeoutfloat expect_image proxy_urlc 4|t}d}g}t||||D]\}} ||rt||r0tjd|d|dd||cS#t $rY} t|||dt| || }t| rYd} ~ n Yd} ~ d} ~ wwxYwtj d|dd|||_ |) a? Download a file the way a browser would, trying every way there is (see the top of this file) until one works :param url: The url of the file :param file_path: Where to save it :param timeout: Seconds to wait for the host in every request :param expect_image: If the file has to be an image. A host that blocks you often answers with a web page instead of an error, which is then not taken as a download that worked :param proxy_url: Url template of the image proxy to use as the last way. Default: the one of the IMAGE_PROXY_URL environment variable, if any :return: The name of the way that worked :raises: The error of the first way (urllib) if none of them worked, so that whoever calls this can tell a dead link (HTTPError 404) from a blocked one (HTTPError 403) as always Nz Downloaded z through z after: z; rgzCould not download z - ) r_ _iter_ways_check_file_is_imager\infojoinrb_remove_quietlyappendrl _is_dead_linkr]download_failures) rrnrorqrr first_errorfailuresway_namewayes r fetch_to_filers,')) KH#CGYGG #  CEEE 0$Y/// b `3````499U]K^K^``aaaOOO    I & & & OOx>>>!+<+<>> ? ? ?" Q        OG#GG$))H2E2EGGHHH%-K! sAA>> C!A CC!c#zKdfdfVtr dfdfVrdfdfVdSdS)a Get the ways to download a file, in the order they are tried. It is a generator on purpose: the ones after urllib (curl_cffi isn't even loaded) are only looked at if urllib didn't work :param url: The url of the file :param file_path: Where to save it :param timeout: Seconds to wait for the host in every request :param proxy_url: The url template of the image proxy, if there is one :return: (name of the way, function that downloads the file that way) r"c&tSN)_fetch_with_urllibrnrorsrz_iter_ways..s.sIwGGrz!browser impersonation (curl_cffi)c&tSr)!_fetch_with_browser_impersonationrsrrz_iter_ways..s5c9gNNrz image proxyc(tSr)_fetch_through_image_proxy)rnrrrorsrrz_iter_ways..s.sIw RRrN)rc)rrnrorrs````rrtrts GGGGGG GGGG)++ / N N N N N N      R R R R R R R        rc\t|tjjo |jt vS)zr Check if an error says that the file is not there :param error: The error :return: True if it is )rhr"rdricodeDEAD_LINK_STATUS_CODESrks rrzrz s& eV\3 4 4 ]G]9]]rNonecR tj|dS#t$rYdSwxYw)zo Remove a (partially) downloaded file, if it is there :param file_path: The file :return: None N)rXremoveOSError)rns rrxrxs?  )      s  &&ct|d5}|d}dddn #1swxYwYt|std|dddS)z Check that a downloaded file is an image :param file_path: The file :return: None :raises NotAnImageError: If it is not rb Nz.the host sent something that is not an image: )openreadr8r )rnfiler0s rruru s i  $yy}} D ! !^\tTWUWTWy\\]]]^^s 377attemptsintcxtj|t|}t |D]} tj||5}t |d5}tj||dddn #1swxYwYdddn #1swxYwYdS#tj j $r6}||dz kr'|j tvrtjdYd}~d}~wtj j$r!||dz krtjdYwxYwdS)a2 Download a file with urllib, sending the headers of a browser :param url: The url of the file :param file_path: Where to save it :param timeout: Seconds to wait for the host in every request :param attempts: How many times to ask when the problem looks temporary :return: None )headers)rowbNrB)r"requestRequestr/rangeurlopenrshutil copyfileobjrdrirRETRYABLE_STATUS_CODEStimesleepURLError) rrnrorrattemptresponseout_filers rrr/sn$$S2Ec2J2J$KKG?? '''AA ;X)T**;h&x:::;;;;;;;;;;;;;;; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; FF|%   A%%!&4J*J*J 1  |$   A%% 1    sl!B:"B,3B B,B B,B B, B:,B0 0B:3B0 4B::D7+C?>C??4D75D7cddlm}t|}dD]}|||||d|d}|jdkr:t j||jt|jddt|d 5}| |j ddddS#1swxYwYdS) a= Download a file with curl_cffi, which does the TLS and HTTP/2 handshake of a real Chrome (and sends the headers that go with it, so the User-Agent is left to it) :param url: The url of the file :param file_path: Where to save it :param timeout: Seconds to wait for the host :return: None r)requests)rrr r!chromeT)r impersonateroallow_redirectsNr) rarr/poprZ status_coder"rdri_get_status_phraserwritecontent)rrnro curl_requestsrheaderrrs rrrPs;433333!#&&GW F   W(GUY!Hs""l$$ %'9(:N'O'OQUW[    i  )(x'((())))))))))))))))))sCC C c|dtj|d}t |||ddS)aO Download a file through an image proxy: the proxy asks the host for it, not this server :param url: The url of the file :param file_path: Where to save it :param timeout: Seconds to wait for the proxy :param proxy_url: The url template of the proxy, with {url} where the url of the file goes :return: None rWr@)safe)rN)replacer"r#quoter)rrnrorr proxied_urls rrrmsK##GV\-?-?"-?-M-MNNK{IwCCCCCCrrcX tj|jS#t$rYdSwxYw)z Get the standard phrase of an HTTP status code :param status_code: The status code :return: The phrase (e.g. Forbidden) zUnknown status)http HTTPStatusphrase ValueError)rs rrr{s> {++22     s  )))rrrr)r0r1rr2)rrrr)rrU)rr2)rdrerr)rmFN) rrrnrrorprqr2rrrUrr)rrrnrrorprrrU)rdrerr2)rnrrr)r) rrrnrrorprrrr)rrrnrrorprr) rrrnrrorprrrrr)rrrr)#r __future__rrKrr\rXrIrr urllib.errorr" urllib.parseurllib.requestr+r[rrrbr r/r8rTr_rcrlrrtrzrxrurrrrrrrrs>$#"""""   %)).--dddddiddd    F$8(     . . . .  44444n    4^^^^      ^ ^ ^ ^B)))): D D D D       r