-
Amazon Deals - ToS - Like us?
Python script for finding blocked sites on Indian ISPs
- Thread starter JB701
- Start date
- Replies 37
- Views 8,458
I have added a new method of changing SNI on SSL to check blocked sites. It doesn't require the site to have SSL/TLS by using a different server IP. It appears to be faster and uses less CPU.
The new script is way faster, completed in an hour here are the sites on Namecheap VPN. Almost
github.com
~806 sites out of 1 Million blocked
On Oracle ~920 sites out of 1 Million blocked
now I'm doing 10 million
blockeddomaincheck/blockedsiteslist at main · sudozid/blockeddomaincheck
Check blocked domains on Indian ISPs. Contribute to sudozid/blockeddomaincheck development by creating an account on GitHub.
github.com
~806 sites out of 1 Million blocked
On Oracle ~920 sites out of 1 Million blocked
now I'm doing 10 million
Last edited:
~4899/10 million sites blocked https://raw.githubusercontent.com/s...iteslist/tlsblockedsites-oracle-10million.txt
Many are subdomains though, Airtel/TATA are likely blocking by regex or wildcards
I have combined multiple potentially blocked sites list and the blocked sites among top 1million and top 10 million sites, removed duplicates and sorted. ~11896 sites. Though there are www subdomains which need to be removed
After removing subdomains and duplicates, its https://raw.githubusercontent.com/s...-oracle-combined-nosubdomains-duperemoved.txt 6651 sites.
This should be a significant chunk of all the blocked sites but clearly not even close to how much there is.
www.medianama.com
NOW ITS TIME FOR 260 MILLION
Download list of top 10 million domains based on Open data from Common Crawl & Common Search
View or download the entire list of the top 10 million domains based on Open PageRank, an initiative that use Open data from Common Crawl & Common Search
www.domcop.com
Many are subdomains though, Airtel/TATA are likely blocking by regex or wildcards
I have combined multiple potentially blocked sites list and the blocked sites among top 1million and top 10 million sites, removed duplicates and sorted. ~11896 sites. Though there are www subdomains which need to be removed
After removing subdomains and duplicates, its https://raw.githubusercontent.com/s...-oracle-combined-nosubdomains-duperemoved.txt 6651 sites.
This should be a significant chunk of all the blocked sites but clearly not even close to how much there is.
Govt blocked more than 16,000 websites between 2018-2020, but figures show inconsistency. Again.
The Indian government issued directions to ban as many as 16,283 “website/webpages/accounts” between 2018 and 2020, Electronics and IT Minister Ravi Shankar Prasad revealed in Lok Sabha on Wednesday.
www.medianama.com
NOW ITS TIME FOR 260 MILLION
Last edited:
Now I'm running a the search on 260 million domains. The csv file was ~4GB so I had to split it to ~130 smaller files with 2million domains each.
After ~14hrs of running it has already found ~620 domains which was kinda surprising considering the large amount to be checked. I'm running the same on another Oracle VPS to make sure no domain is missed.
After ~14hrs of running it has already found ~620 domains which was kinda surprising considering the large amount to be checked. I'm running the same on another Oracle VPS to make sure no domain is missed.
260 million search is only partly done but i've combined all the blocked domains from both vps+all the blocked domains from earlier search (1mill, 10mill, adult gambling sites etc) and removed subdomains and duplicates. Right now its at 12896 blocked domains.
This is not completely related to the original topic but I thought it was worth mentioning.
Apparently Mikrotik RouterOS has a feature to match packets by TLS Host. This can be used to easily policy route blocked sites over a VPN without relying on IP Based policy routing (by resolving blocked sites using DNS) which can have issues because of subdomains, ip changes etc. Its a shame that this feature isn't available on PFSense.
systemzone.net
Apparently Mikrotik RouterOS has a feature to match packets by TLS Host. This can be used to easily policy route blocked sites over a VPN without relying on IP Based policy routing (by resolving blocked sites using DNS) which can have issues because of subdomains, ip changes etc. Its a shame that this feature isn't available on PFSense.
MikroTik Blocking Websites with TLS Host Firewall Matcher
The TLS Host matcher is a new MikroTik Firewall property that can be used to block any https website such as Facebook, Youtube, Twitter and so on.
Tl;dr "TLS Host" = SNI
Update tlsblockedsites-oracle-combined-nosubdomains-duperemoved.txt · sudozid/blockeddomaincheck@8120242
Check blocked domains on Indian ISPs. Contribute to sudozid/blockeddomaincheck development by creating an account on GitHub.
github.com
6779 more domains added.