Python script for finding blocked sites on Indian ISPs

[JB701]

its been almost a month and the script is still running. i've set a bash script to commit updated list every 6 hours.

https://github.com/sudozid/blockeddomaincheck/blob/main/blockedsiteslist/tlsblockedsites-oracle-combined-nosubdomains-duperemoved.txt

~21400 domains found so far (though there are some small number of false positives which i'll remove by running the script again).

 

[imdarkray]

@JB701 not a single link in this thread is working now. Did you delete everything?

 

[JB701]

ah yea i made forgot to make it public after setting it private a few months ago. it's up now.

the last search on half a billion sites took months to complete btw (and certainly missed blocked sites). i tried running the script which just sends tcp handshake and a tls client hello but that ended up creating a ton of false positives (maybe something wrong with the way i coded it).

 

[JB701]

so, i got myself a RB750gr3 as a backup router and used it for testing.

i found out that you can unblock all the sites (atleast on airtel transit) with a simple firewall rule

/ip firewall raw add chain=prerouting protocol=tcp tcp-flags=rst action=drop

this drops all tcp rst packets so connections aren't killed by the isp.

is this a good idea? i don't think so as rst flags does have legit uses (maybe one can use this with specific ips or tls hosts? i haven't tried that myself).

pfsense has a similar tcp flag based blocking thing but i haven't been able to figure it out.


other types of blocking seem more advanced, i've found that certain sites on TATA transit return End Of File error instead of TCP reset. They do this by ending connection using FIN,ACK TCP flags. Even when i drop packets with these two flags, the page won't load (timeout).

 

[wheatbread]

There'a a new list from Cloudflare https://radar.cloudflare.com/charts/LargerTopDomainsTable/attachment?id=24&n=1000000&d=2022-10-03

 

[Sushubh]

lol

 

---

[JB701]

on airtel atleast cloudflare sites seem to return 525 error now instead of tcp reset :/ so my old script won't work.

i suspect what is happening is that, CloudFlare's Delhi proxy server is routed through an provider where the sites are blocked, preventing it from connecting with the actual site.

Shouldn't be too difficult for CloudFlare to fix, just prevent all these sites from going through Indian nodes.

 

[wheatbread]

This is Cloudflare's upstream blocking leetx, airtel doesn't block anything on http/3 yet.