-
Amazon Deals - ToS - Like us?
Sudden Slowness - BSNL Bharat Fiber Connection
- Thread starter Gaurav_G
- Start date
- Replies 53
- Views 4,476
@Anurag Bhatia As you said previously BSNL (AS9829) uses the Tata Communications (VSNL)(AS4755) as the main upstream for outbound traffic.
My dumb doubt are :
1. If there any issues or outages happens in the AS4755 for BSNL will they switch the upstream to any other or they continue with the same ?
2. I have seen your blog about the instability of BSNL ASN. Is it still the same ? I have also seen AS9829 maintaining the top 10 ranking in the BGP Instability Report by Potaroo
Why they can’t overcome this instability issue
3.There is a lot of latency fluctuations in the AS9829 IPv4 address. Some IP address have decent latency and some have worst cause when I test the traceroute the gateway to the Tata Communications Kochi (VSNL Kochi gateway ) is more than 10-50ms (Latency to kochi-kochi ) and also their IPv6 is Broken
My dumb doubt are :
1. If there any issues or outages happens in the AS4755 for BSNL will they switch the upstream to any other or they continue with the same ?
2. I have seen your blog about the instability of BSNL ASN. Is it still the same ? I have also seen AS9829 maintaining the top 10 ranking in the BGP Instability Report by Potaroo
Why they can’t overcome this instability issue
3.There is a lot of latency fluctuations in the AS9829 IPv4 address. Some IP address have decent latency and some have worst cause when I test the traceroute the gateway to the Tata Communications Kochi (VSNL Kochi gateway ) is more than 10-50ms (Latency to kochi-kochi ) and also their IPv6 is Broken
Hi
1. There is always redundancy. They likely have high local preference set on Tata Comm routes. If they disapear, other ones would come up.
2. This report on Potaroo for BSNL gives some idea. Since graph on that page takes a few seconds to generate, here's the graph for "BGP updates" sent by BSNL:
Compare this for compariable other Indian network with massive IP address holding - Airtel AS24560. BSNL is sending massive number of BGP updates which hints towards lot of flapping happening within the network.
3. Most of their latency problems are return path related. Keep in mind latency is round trip i.e time taken by packets from you to destination and from destination to you. So when you traceroute to the Tata Communications Kochi packets you will see forward path on trace (likely BSNL > Tata Comm) but you won't be able to see return path easily. It can be Tata Comm > BSNL, or Tata Comm > Other BSNL transits > BSNL.
In general most of their global routing issues are because of design where 1) They use different transit players around the world, 2) Do not peer at IX and 3) Have lot of far off circuits. Having "transit 1" in Singapore and "transit 2" in Europe will cause issues in routing. Fixing this design is hard, someone needs to take initiative and change what has been working from long time.
1. There is always redundancy. They likely have high local preference set on Tata Comm routes. If they disapear, other ones would come up.
2. This report on Potaroo for BSNL gives some idea. Since graph on that page takes a few seconds to generate, here's the graph for "BGP updates" sent by BSNL:
Compare this for compariable other Indian network with massive IP address holding - Airtel AS24560. BSNL is sending massive number of BGP updates which hints towards lot of flapping happening within the network.
3. Most of their latency problems are return path related. Keep in mind latency is round trip i.e time taken by packets from you to destination and from destination to you. So when you traceroute to the Tata Communications Kochi packets you will see forward path on trace (likely BSNL > Tata Comm) but you won't be able to see return path easily. It can be Tata Comm > BSNL, or Tata Comm > Other BSNL transits > BSNL.
In general most of their global routing issues are because of design where 1) They use different transit players around the world, 2) Do not peer at IX and 3) Have lot of far off circuits. Having "transit 1" in Singapore and "transit 2" in Europe will cause issues in routing. Fixing this design is hard, someone needs to take initiative and change what has been working from long time.
Update:
I got one email and one Twitter DM asking to elaborate. I avoided that in last reply as it would mean long reply and ultimately one has to make up mind whether it's worth typing a super long message (at risk of people not reading it) OR relatively smaller message with summarized info.
Announcement only in one location
For BSNL's un-optimized routing from Internet > BSNL, what happens is that they taken certain transit based on certain locations. Take e.g Cogent AS174. BSNL is announcing 59.90.158.0/24 to Cogent AS174 and Sprint Busiuness AS1239 (which is also now part of Cogent). Outside of India BSNL does not peer publically at an IXP, and Cogent port in this specific case is in Singapore. So for outside world BSNL is saying "entry point for this prefix is ONLY in Singapore". This is fine when traffic is coming from Singapore itself or Hong Kong or Taiwan but is an issue for traffic from Europe.
hop 5 - Frankfurt
hop 6 - Singapore
Again it's important to understand that this is not a Cogent routing issue. It's BSNL's design issue. I can take a IPLC from my home in Haryana to US, buy IP transit only in US and do not announce routes anywhere else. How would Indian ISPs reach me in that case...ofcourse from the US!
Routing challenge / BGP behaviour
Globally networks often set three layers of localpref (local preference = attribute in BGP). Routes with high localpref take precedence over routes with lower localpref. So networks often have highest localpref towards customer routes, second highest towards peering routes and lowest towards transit routes. Broad idea here is that networks try to send traffic out from there customer facing ports whereever they can (because people are paying money for those ports and ideally more traffic = faster chance of next upgrade), if they cannot send traffic from customer ports and learning route only from a settlement free peer, they would do that and prefer to do that instead of last i.e to end via a transit and pay for it.
Take case of prefix: 59.88.176.0/20 - 59.88.176.0/20 - bgp.he.net announced to: Tata Comm AS4755 (India), Cogent (AS174) US, NTT AS2914 (Singapore), GTT AS3257 (New York), Telecom Italia AS6762 (Singapore).
Frankfurt, Paris, New York >> BSNL handover >>>> IPLC to India (likely via EU again). So it's EU > US > EU > India flow and ofcourse much higher latency. Cogent here did not handover traffic in EU because 1) There is NO announcement/enty point for BSNL for this prefix in EU and 2) even if there was one, Cogent would prefer it's customer port. Same issue with traffic from say Telecom Italia in EU. It will go from EU > Singapore > India path adding to the latency.
Fix <- Purely from routing angle
Fix <- With commercial angle:
I got one email and one Twitter DM asking to elaborate. I avoided that in last reply as it would mean long reply and ultimately one has to make up mind whether it's worth typing a super long message (at risk of people not reading it) OR relatively smaller message with summarized info.
Announcement only in one location
For BSNL's un-optimized routing from Internet > BSNL, what happens is that they taken certain transit based on certain locations. Take e.g Cogent AS174. BSNL is announcing 59.90.158.0/24 to Cogent AS174 and Sprint Busiuness AS1239 (which is also now part of Cogent). Outside of India BSNL does not peer publically at an IXP, and Cogent port in this specific case is in Singapore. So for outside world BSNL is saying "entry point for this prefix is ONLY in Singapore". This is fine when traffic is coming from Singapore itself or Hong Kong or Taiwan but is an issue for traffic from Europe.
hop 5 - Frankfurt
hop 6 - Singapore
Again it's important to understand that this is not a Cogent routing issue. It's BSNL's design issue. I can take a IPLC from my home in Haryana to US, buy IP transit only in US and do not announce routes anywhere else. How would Indian ISPs reach me in that case...ofcourse from the US!
Routing challenge / BGP behaviour
Globally networks often set three layers of localpref (local preference = attribute in BGP). Routes with high localpref take precedence over routes with lower localpref. So networks often have highest localpref towards customer routes, second highest towards peering routes and lowest towards transit routes. Broad idea here is that networks try to send traffic out from there customer facing ports whereever they can (because people are paying money for those ports and ideally more traffic = faster chance of next upgrade), if they cannot send traffic from customer ports and learning route only from a settlement free peer, they would do that and prefer to do that instead of last i.e to end via a transit and pay for it.
Take case of prefix: 59.88.176.0/20 - 59.88.176.0/20 - bgp.he.net announced to: Tata Comm AS4755 (India), Cogent (AS174) US, NTT AS2914 (Singapore), GTT AS3257 (New York), Telecom Italia AS6762 (Singapore).
- Tata Comm drop in this specific prefix case is local within India and hence they can pick traffic globally and drop nearest to BSNL and in majority of cases routing would be ok.
- Cogent drop here is in US. Cogent (and their peers) in US -> 59.88.176.0/20 will be OK, but what about Cogent in Europe? Here's trace from their looking glass:
Frankfurt, Paris, New York >> BSNL handover >>>> IPLC to India (likely via EU again). So it's EU > US > EU > India flow and ofcourse much higher latency. Cogent here did not handover traffic in EU because 1) There is NO announcement/enty point for BSNL for this prefix in EU and 2) even if there was one, Cogent would prefer it's customer port. Same issue with traffic from say Telecom Italia in EU. It will go from EU > Singapore > India path adding to the latency.
Fix <- Purely from routing angle
- BSNL should buy from same set of players on both sides. Whatever providers they fix, they should take port from those in EU as well as Singapore. US can be optional because US via EU > India and US > Singapore > India both is mostly OK. One ideally expects US East > EU > India and US West > Asia > India but that is less of issue. If they want to have IPLC in US, they need to use same set of transits there as well.
- They can announce aggregate of their own pools everywhere but do more specific announcements limiting them by BGP community. So BSNL's announcement to Cogent in Singapore can go to Cogent's peers and customer in Singapore + US + South America but not EU. Same with GTT New York, it should stay within EU peers and customers, not export to EU or Singapore. Not perfact but solves lot of cases.
- What I put in #1 sounds easy in writing but is very hard to implement in govt. machinery going from tenders to tenders. What you do with existing capacity, 100G in US, EU and Singapore may not actually have same usage and hence it has to be caliberated effort. They add 100G and suddenly traffic hits 90G, they would have to add 100G more. But on other side 100G may go little un-utilized. But ideally it should be similar set of upstreams along with logic to announce their full routes (own originated + downstream) everywhere. This would fix 99% of the routing issues.
Fix <- With commercial angle:
- Networks like Airtel, Jio usually peer outside of India extensively besides transit. They have fair spread of transit but because peering is so extensive, usually you won't see routing issue specially kind BSNL has. Tata Comm has different routing design as they are one of few transit free networks and broad idea there is that they peer with other large networks and mostly have good routing with that. BSNL ultimately if runs this longhual network outside of India - it would have to add peering in the mix and that would involve putting gear in some participating DC, joining IX, aggressively peering etc.
- Over time a large part of content has come to India. Likely for inbound heavy network like BSNL they could potentially offload 80-85% traffic within India on peerings. Actual number might be lower because they lack bilateral BGP sessions across IXPs like DECIX Mumbai with som content players. If 80-85% is offload, is remaining 15% worth all above effort? Ultimately similar to compute & storage market, bandwidth is commodity and it makes sense to do things only at scale. Either they have massive traffic of their own or transit downstreams. In absence of both the effort may just not be worth it. I don't know their numbers and cannot say for sure but with loss of more and more customers over time it's unclear. As stated in some other reply in past, bandwidth pricing is like 1:3:10. $1 at core, $3 on middle mile and $10 on last mile. Whether BSNL buys at 3 or 2.9 or 3.1 (not actual price but example ratio), it won't make or break them (as 15% of their traffic would barely hit this) if their 10 paying customer are unhappy due to bad routing and leaving as soon as they have an alternate.