@Anurag Bhatia
I have read your blog earlier and checks it out for new posts.
I had few questions regarding the working of BSNL.
1. If they have bought bandwidth, it is definitely possible for them to peer outside of India with various networks and cloud services having straight control ove routing. What exactly is the resistance point because AFAIK it's just a little configuration to be completed on any of their end
router which announces routes? Moreover, the termination of most circuits is an exchange (my assumption).
2. What exactly is the issue with buying circuits? I assume they still have control over how they route their traffic based of the target IP location. So, It should still work and work even better since they basically have dedicated capacity from one end to another. Is it just stupid personnel who are lazying around instead of creating better routing rules or this "buying bandwidth" has some effect on it?
This would need a long answer thus excuse a longer reply.
1. Peering outside India is surely doable but practically it's not that straightforward for a Govt. operator. Do not under estimate logistics as well as Govt. policies around purchase of colocation, hardware, services etc. BSNL has circuits from India to Los Angeles and they just buy transit there. If today they decide to go ahead with peering at say any large IX over there, say
Equinix Los Angeles then they have to do a few things: Buy colocation. Can they legally just pick and decide a datacenter colo, I doubt that. Most of things would require to call for a tender/invitation for bidding. Will Equinix and other datacenters will pick that text heavy Indian paper work, may be yes, may be no. So you see it's hard for the decision maker to justify specific datacenter as well as specific IX. I think they have some workaround but it remains hard bureaucracy wise. Assume that part is navigated, next you hit issue: American datacenter would typically expect money in USD and not INR to avoid currency exposure. They would typically not sell/do contract in INR because if INR losses value over short periods, it can cause issues for them. Can BSNL negotiate in USD? Would be hard as would need approval of dept. outside of BSNL like finance ministry and even RBI to issue payments. Assume this part is navigated, next they have to deploy hardware - essentially a router & atleast a switch etc. How you do that? Can you ship hardware from India? That would be expensive and even counterproductive. Can you buy or lease locally - well again how you decide vendor? At each step where purchase is required they will not have the free hand. Assume this part is navigated. Next comes - how do you install and maintain that gear which is far away from India? Again you need local vendor/RHS support etc. So do not underestimate the fact that lot of logistics is involved here. Plus BSNL would typically not have free hand in making purchases as the private players. Lack of free hand introduces inefficiency and giving a fully free hand may likely lead to corruption. So it's a hard problem to solve.
2. It's not an issue if BSNL peers with networks at those locations instead of buying IP transit/wholesale bandwidth. Peering is hard as I explained in above due to fact that they are a Govt. organisation. If they pick IP transit outside of India BGP will not always pick best path. Have a look at slide 27
here to read BGP path selection algorithm. So looking at routing right now I see BSNL has IP transit from Airtel,
Tata Comm (domestic), Tata Comm (outside of India), Sri Lanka Telecom, Cogent, GTT, Telia, Century Link etc. Let's pick any one specific transit and find any specific prefix announced there say from GTT AS3257.
route-server> sh ip bgp regexp 3257 9829
BGP table version is 0, local router ID is 64.62.142.154
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
- i59.88.208.0/20 62.115.181.197 48 70 0 1299 3257 9829 i
- i 62.115.181.197 48 70 0 1299 3257 9829 i
- i 216.218.252.173 48 70 0 1299 3257 9829 i
- i 216.218.252.176 48 70 0 1299 3257 9829 i
- i 216.218.252.154 48 70 0 1299 3257 9829 i
- i 216.218.252.82 48 70 0 1299 3257 9829 i
- i 216.218.252.184 48 70 0 1299 3257 9829 i
- i 216.218.252.189 48 70 0 1299 3257 9829 i
- i 62.115.181.197 48 70 0 1299 3257 9829 i
(and more)
GTT is American player & from looking at routes, I can confirm BSNL is taking this transit in the US. Let's now trace from a few locations in the Europe. Ideally it should be Europe > India as direct submarine cables exist. Let's read a few traces:
DigitalOcean UK > BSNL India
traceroute to 59.88.208.1 (59.88.208.1), 30 hops max, 60 byte packets
1 * * *
2 10.80.4.42 (10.80.4.42) [*] 0.882 ms 0.883 ms 0.881 ms
3 138.197.249.98 (138.197.249.98) [AS14061] 1.138 ms 1.218 ms 1.219 ms
4 138.197.251.128 (138.197.251.128) [AS14061] 0.802 ms 0.813 ms 0.811 ms
5 212.187.195.85 (212.187.195.85) [AS3356/AS9057] 1.984 ms 1.997 ms 1.997 ms
6 ae1.3115.edge7.London1.level3.net (4.69.166.2) [AS3356] 1.996 ms 1.782 ms 1.729 ms
7 GTT-level3-London1.Level3.net (4.68.72.202) [AS3356] 5.948 ms 5.949 ms 5.948 ms
8 ae3.cr6-nyc6.ip4.gtt.net (213.200.121.34) [AS3257] 95.635 ms 95.395 ms 95.336 ms
9 * * *
10 * * *
11 218.248.113.66 (218.248.113.66) [AS9829] 243.614 ms 244.164 ms 244.152 ms
12 218.248.113.65 (218.248.113.65) [AS9829] 243.523 ms 243.408 ms 243.388 ms
13 59.88.208.1 (59.88.208.1) [AS9829] 243.635 ms 243.784 ms 244.050 ms
Custodian datacenter AS50300 > BSNL India
traceroute to 59.88.208.1 (59.88.208.1), 30 hops max, 60 byte packets
1 rtr-151.cdc.custdc.net (109.74.240.241) [AS50300] 0.684 ms 0.769 ms 0.764 ms
2 rtr-121.thn.custdc.net (109.74.255.251) [AS50300] 1.793 ms 2.082 ms 2.161 ms
3 xe-8-2-1-1141.cr1-lon1.ip4.gtt.net (77.67.122.149) [AS26769] 1.049 ms 1.052 ms 1.049 ms
4 ae3.cr6-nyc6.ip4.gtt.net (213.200.121.34) [AS3257] 70.177 ms 70.181 ms 70.200 ms
5 * * *
6 * * *
7 218.248.113.66 (218.248.113.66) [AS9829] 243.023 ms 243.058 ms 243.110 ms
8 218.248.113.65 (218.248.113.65) [AS9829] 242.990 ms 243.037 ms 243.105 ms
9 59.88.208.1 (59.88.208.1) [AS9829] 241.716 ms 241.697 ms 241.811 m
Let's pick somewhere else in Europe. Say Germany & Netherlands
AS62874 Germany > BSNL India
traceroute to 59.88.208.1 (59.88.208.1), 30 hops max, 60 byte packets
1 167.160.40.64 (167.160.40.64) [AS62874] 1.309 ms 1.357 ms 1.331 ms
2 ae0-463.fra30.core-backbone.com (5.56.17.169) [AS33891] 0.628 ms 0.623 ms 0.614 ms
3 ae12.cr2-fra6.ip4.gtt.net (87.119.97.185) [AS34991] 13.022 ms 12.987 ms 12.978 ms
4 ae3.cr6-nyc6.ip4.gtt.net (213.200.121.34) [AS3257] 84.396 ms 84.391 ms 84.383 ms
5 * * *
6 * * *
7 218.248.113.66 (218.248.113.66) [AS9829] 263.942 ms 264.031 ms 263.972 ms
8 218.248.113.65 (218.248.113.65) [AS9829] 263.652 ms 263.516 ms 263.500 ms
9 59.88.208.1 (59.88.208.1) [AS9829] 263.882 ms 263.863 ms 263.919 ms
AS12859 Netherlands > BSNL India
traceroute to 59.88.208.1 (59.88.208.1), 30 hops max, 60 byte packets
1 cust-cloud01-gw.jun2.galilei.network.bit.nl (212.114.113.3) [AS12859] 0.361 ms 0.352 ms 0.352 ms
2 xe-1-0-1.jun1.bit-1.network.bit.nl (213.136.1.73) [AS12859] 5.154 ms 5.157 ms 5.157 ms
3 83.231.213.57 (83.231.213.57) [AS2914] 1.775 ms 1.775 ms 1.875 ms
4 ae-3.r20.amstnl07.nl.bb.gin.ntt.net (129.250.7.86) [AS2914] 1.751 ms 1.754 ms 1.752 ms
5 ae-0.a00.amstnl07.nl.bb.gin.ntt.net (129.250.7.65) [AS2914] 1.752 ms 1.752 ms 1.752 ms
6 ae15.cr4-ams1.ip4.gtt.net (46.33.83.249) [AS3257] 3.302 ms 3.152 ms 3.136 ms
7 ae3.cr6-nyc6.ip4.gtt.net (213.200.121.34) [AS3257] 77.283 ms 88.641 ms 88.637 ms
8 * * *
9 * * *
10 218.248.113.66 (218.248.113.66) [AS9829] 245.653 ms 245.656 ms 245.656 ms
11 218.248.113.65 (218.248.113.65) [AS9829] 243.647 ms 243.652 ms 243.637 ms
12 59.88.208.1 (59.88.208.1) [AS9829] 242.699 ms 242.743 ms 242.550 ms
So it's just a endless list. In all these cases traffic is going from the respective country in Europe (from where we initiate trace) to BSNL India via the US. ae3.cr6-nyc6.ip4.gtt.net - tells that it's New York router of GTT and hence most of traffic here is routing via New York. So instead of latency being 130-170ms (depending on where the destination is in India) it's as high as 250ms.
Why BGP is taking this unoptimised path. That's due to the fact that BSNL is announcing routes only in the US via these large networks and not in Europe. Hence for anyone to send traffic entry point is only the US. Besides GTT, for the same pool BSNL is also using Tata Comm AS6453 but again outside of India. Here are a couple of traces to show the same:
(multi part reply to stay in word limit)
)
, I know it's not enough)