feat: add FlareSolverr to bypass cloudflare #1124

kaiserbh · 2024-03-18T14:45:37Z

Added https://github.com/FlareSolverr/FlareSolverr to bypass cloudflare anti-bot protection.

A brief summary of FlareSolverr:

FlareSolverr is a proxy server designed to bypass Cloudflare and DDoS-GUARD protection. It works by starting a proxy server that uses Selenium with undetected-chromedriver to open a web browser (Chrome), navigate to a URL with user parameters, and wait until the Cloudflare challenge is solved before sending back the HTML code and cookies. These cookies can then be used to bypass Cloudflare with other HTTP clients.

Before and after

With FlareSolverr	Without


Hello	OKay

Settings in the advance section.

Video:

Without flareSolverr

Screen_recording_not_working.mp4

With FlareSolverr

flareSolverr_works_compressed_700k.mp4

Note

I think everything works fine but I am not sure if the cookie will be persisted at the start, for example in the flaresolverr video it shows no result found even though it managed to get the cookie. Not sure exactly but unless we restart

Ignore the above pushed a fix for it, I should have used the cookie in the Header asap instead of waiting for it to populate cookie jar. The stuff like no result or click on chapter and clicking retry as well it's fixed. Essentially it should load on first visit. Of course that is if the source is working.

Signed-off-by: KaiserBh <kaiserbh@proton.me>

…o feat/add-flare-solverr-bypass-cf

Signed-off-by: KaiserBh <kaiserbh@proton.me>

I don't think reverse proxy going to work maybe it does but no luck on my end not that I need it tbh, I think using tailscale or wireguard should suffice and safer as there is no security in flaresolverr. Signed-off-by: KaiserBh <kaiserbh@proton.me>

We should make sure the cookie is attached when the request proceeds that way we don't have to relay on the app cookie jar state, for example we get the result immediately instead of going out of the extension and going back in, same withe the chapters, they use subdomain and it's random, so we need to make sure the cf_clearance is added to the subdomains dynamically. Signed-off-by: KaiserBh <kaiserbh@proton.me>

Signed-off-by: KaiserBh <kaiserbh@proton.me>

kaiserbh · 2024-03-19T05:32:18Z

Do we still need to enable DoH when using this?🤔

Don't think so, but I tested it with both and works with and without.

NeKoOuO · 2024-04-05T04:03:44Z

Does this PR include the functionality of adding Captcha Solvers in addition ?🤔

kaiserbh · 2024-04-05T05:01:49Z

Does this PR include the functionality of adding Captcha Solvers in addition ?🤔

It's possible, but that depends on you, FlareSolverr already has the capability.

https://github.com/FlareSolverr/FlareSolverr?tab=readme-ov-file#environment-variables

You can read more on it.

jobobby04

Here is a Suwayomi implementation of FlareSolverr that uses some of my comments. Feel free to take things from it.
https://github.com/Suwayomi/Suwayomi-Server/pull/844/files

app/src/main/java/eu/kanade/presentation/more/settings/screen/SettingsAdvancedScreen.kt

…o feat/add-flare-solverr-bypass-cf

Signed-off-by: KaiserBh <kaiserbh@proton.me>

kaiserbh · 2024-04-13T23:38:34Z

Refactored.
But flareSolverr is not working on my linux for some reason getting hit into timeout loop (FlareSolverr/FlareSolverr#1036), so I will need to test this on windows, don't merge it yet.

jobobby04 · 2024-04-13T23:45:34Z

Try changing the timeout on the network okhttpclient

…/add-flare-solverr-bypass-cf

So this kinda works, it sets the cookie in the jar but for some reason fails to retrieve it. But if I directly use CookieManager.getInstance() It works fine??

So this is the working version using CookieManager Directly.

kaiserbh · 2024-06-06T14:43:21Z

Okay so @jobobby04, this works well kinda, I can't get it to work when I use the networkhelper cookie manager... But If I use the cookieManager: CookieManager = CookieManager.getInstance() It works fine, well it fails to load the browse page (refer to the video basically that) the first time it fetches but it works afterwards. I don't know how the cookie manager works and It should technically work as we are building new request with the cookies. If you can have a look at this and see what's the issue, I would appreciate it.

I made two commit, one with the network helper cookie jar (8902355) and the other one accessing it directly (edf35a0).

Also I am using this version of FlareSolverr:

---
version: "2.1"
services:
  flaresolverr:
    # DockerHub mirror flaresolverr/flaresolverr:latest
    image: 21hsmw/flaresolverr:nodriver
    container_name: flaresolverr2
    environment:
      - LOG_LEVEL=${LOG_LEVEL:-info}
      - LOG_HTML=${LOG_HTML:-false}
      - CAPTCHA_SOLVER=${CAPTCHA_SOLVER:-none}
      - TZ=UTC
    ports:
      - "${PORT:-8191}:8191"
    restart: unless-stopped

Since the other ones doesn't seem to work.

cuong-tran · 2024-08-02T06:21:08Z

is this working yet?

NeKoOuO · 2024-08-02T06:34:04Z

@kaiserbh Do u wanna try merge on Komikku…?

jobobby04 · 2024-08-11T22:33:00Z

app/src/main/java/eu/kanade/presentation/more/settings/screen/SettingsAdvancedScreen.kt

+    ) {
+        val json: Json by injectLazy()
+        val jsonMediaType = "application/json".toMediaType()
+        val client = OkHttpClient.Builder().build()


Can we use the networkManager http client here?

jobobby04 · 2024-08-11T22:33:54Z

core/common/src/main/kotlin/eu/kanade/tachiyomi/network/AndroidCookieJar.kt

@@ -51,4 +52,40 @@ class AndroidCookieJar : CookieJar {
    fun removeAll() {
        manager.removeAllCookies {}
    }
+
+    fun addAll(url: HttpUrl, cookies: List<Cookie>) {


Remove debug logs if you are done with them, cookies shouldn't be logged if possible

jobobby04 · 2024-08-11T22:36:21Z

i18n/src/commonMain/resources/MR/base/strings.xml

+    <string name="pref_flare_solverr_url_summary">URL of the FlareSolverr instance to use for example http://192.168.1.202:8191/v1</string>
+    <string name="pref_test_flare_solverr_and_update_user_agent">Test FlareSolverr and update user agent</string>
+    <string name="pref_test_flare_solverr_and_update_user_agent_summary">Test FlareSolverr to update the user agent. Only required once; then toggle as needed.</string>
+


SY strings should go in i18n-sy

jobobby04 · 2024-08-11T22:37:35Z

core/common/src/main/kotlin/eu/kanade/tachiyomi/network/interceptor/FlareSolverrInterceptor.kt

+            return chain.proceed(originalRequest)
+        }
+
+        Log.d("FlareSolverrInterceptor", "Intercepting request: ${originalRequest.url}")


All logs should use logcat()

pixelroll · 2024-09-11T21:08:57Z

it's abandoning ? the idea was great!

kaiserbh added 7 commits March 18, 2024 23:24

feat: added FlareSolverr to bypass CF anti-bot.

d72d9ca

Signed-off-by: KaiserBh <kaiserbh@proton.me>

Merge branch 'master' of https://github.com/jobobby04/TachiyomiSY int…

dbaedcd

…o feat/add-flare-solverr-bypass-cf

chore: add summary to Test FlareSolverr button.

dd7c874

Signed-off-by: KaiserBh <kaiserbh@proton.me>

refactor: move the check into the ShouldIntercept.

aab8b6e

Signed-off-by: KaiserBh <kaiserbh@proton.me>

refactor: add the option to only add cf_clearance cookie.

15db7c5

Signed-off-by: KaiserBh <kaiserbh@proton.me>

refactor: update summary.

51896bf

I don't think reverse proxy going to work maybe it does but no luck on my end not that I need it tbh, I think using tailscale or wireguard should suffice and safer as there is no security in flaresolverr. Signed-off-by: KaiserBh <kaiserbh@proton.me>

kaiserbh changed the title ~~feat: add flare solverr bypass cloudflare~~ feat: add FlareSolverr bypass cloudflare Mar 18, 2024

kaiserbh changed the title ~~feat: add FlareSolverr bypass cloudflare~~ feat: add FlareSolverr to bypass cloudflare Mar 18, 2024

chore: detekt stuff.

ead9079

Signed-off-by: KaiserBh <kaiserbh@proton.me>

This comment has been minimized.

Sign in to view

NeKoOuO mentioned this pull request Apr 5, 2024

Mikan 数据源用索引搜索, 处理多季度区分问题 open-ani/ani#103

Closed

This comment has been minimized.

Sign in to view

jobobby04 reviewed Apr 13, 2024

View reviewed changes

kaiserbh added 2 commits April 14, 2024 06:30

Merge branch 'master' of https://github.com/jobobby04/TachiyomiSY int…

8305100

…o feat/add-flare-solverr-bypass-cf

refactor: use kotlinx, and avoid using org.JSON

9552544

Signed-off-by: KaiserBh <kaiserbh@proton.me>

Mylloon mentioned this pull request Apr 14, 2024

FlareSolverr implementation mihonapp/mihon#668

Closed

4 tasks

kaiserbh added 4 commits June 6, 2024 20:10

Merge remote-tracking branch 'refs/remotes/upstream/master' into feat…

15126f0

…/add-flare-solverr-bypass-cf

chore: fix?

8902355

So this kinda works, it sets the cookie in the jar but for some reason fails to retrieve it. But if I directly use CookieManager.getInstance() It works fine??

fix: working version.

edf35a0

So this is the working version using CookieManager Directly.

chore: optimize imports.

022f2a4

kaiserbh requested a review from jobobby04 June 6, 2024 14:44

soredake mentioned this pull request Jun 10, 2024

Feature request: add FlareSolverr solver to bypass CloudFlare protection lycheeverse/lychee#1439

Open

This comment has been minimized.

Sign in to view

NeKoOuO mentioned this pull request Jul 26, 2024

增加幾個常見的BT首發站 open-ani/ani#647

Closed

3 tasks

jobobby04 requested changes Aug 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add FlareSolverr to bypass cloudflare #1124

feat: add FlareSolverr to bypass cloudflare #1124

kaiserbh commented Mar 18, 2024

This comment has been minimized.

kaiserbh commented Mar 19, 2024

NeKoOuO commented Apr 5, 2024

kaiserbh commented Apr 5, 2024

This comment has been minimized.

jobobby04 left a comment

kaiserbh commented Apr 13, 2024 •

edited

Loading

jobobby04 commented Apr 13, 2024

kaiserbh commented Jun 6, 2024

This comment has been minimized.

cuong-tran commented Aug 2, 2024

NeKoOuO commented Aug 2, 2024 •

edited

Loading

jobobby04 Aug 11, 2024

jobobby04 Aug 11, 2024

jobobby04 Aug 11, 2024

jobobby04 Aug 11, 2024

pixelroll commented Sep 11, 2024

feat: add FlareSolverr to bypass cloudflare #1124

Are you sure you want to change the base?

feat: add FlareSolverr to bypass cloudflare #1124

Conversation

kaiserbh commented Mar 18, 2024

Before and after

Settings in the advance section.

Video:

Without flareSolverr

With FlareSolverr

Note

This comment has been minimized.

kaiserbh commented Mar 19, 2024

NeKoOuO commented Apr 5, 2024

kaiserbh commented Apr 5, 2024

This comment has been minimized.

jobobby04 left a comment

Choose a reason for hiding this comment

kaiserbh commented Apr 13, 2024 • edited Loading

jobobby04 commented Apr 13, 2024

kaiserbh commented Jun 6, 2024

This comment has been minimized.

cuong-tran commented Aug 2, 2024

NeKoOuO commented Aug 2, 2024 • edited Loading

jobobby04 Aug 11, 2024

Choose a reason for hiding this comment

jobobby04 Aug 11, 2024

Choose a reason for hiding this comment

jobobby04 Aug 11, 2024

Choose a reason for hiding this comment

jobobby04 Aug 11, 2024

Choose a reason for hiding this comment

pixelroll commented Sep 11, 2024

kaiserbh commented Apr 13, 2024 •

edited

Loading

NeKoOuO commented Aug 2, 2024 •

edited

Loading