There are lots of stories like this. Many of them are worse; at least you can imagine that the UK is holding these addresses in reserve. There are companies with giant allocations that are holding them so that they can give every desktop and every printer in their enterprise a routable address; others are doing the same thing, but also operating flat, unrouted networks.
More evidence for the core problem: fiat allocation doesn't work. If there was a functioning, liquid, accessible market for advertisable IP prefixes, you wouldn't have to convince anyone that a /8 was worth $1bn; it just would be.
Sort of, and sort of not right? Fiat allocation of a fixed resource, sure, but ISO does fiat allocation of OID space and it works because folks are free to grow as much as they want below it. As I recall one of the original 'next gen' IP proposals was like a huge address translation cloud, IPs were encapsulated in yet another layer which provided 'context' for the IP address inside. Basically every entity got their own 32 bit address space and ingress/egress to the Internet resulted in encapsulation and some additional router magic. One of the arguments against was the huge additional bandwidth cost. Of course we didn't realize at the time that few protocol changes could waste as much bandwidth as SPAM does.
The other challenge is that a functioning liquid market for IP addresses would most likely lead to speculation in IP address blocks. Worse is buying an address block from someone in Germany so that packets have to go to Germany first (top level static routing) to the router where they are re-allocated and then to their 'real' destination router. All very inefficient.
I, like other folks at Sun, was a big fan of a more modest 64 bit address proposal for V6, but alas it was not to be. That the conversion has taken as long as it has (and still isn't "here" nearly enough for a lot of users) really illustrates the dangers of the argument to 'permanently' fix things[1]. But innumeracy aside, the confounding issue is that IP addresses are 'structural' like telephone numbers and managing mobility of structural identifiers is always challenging (the cellular market deals with this all the time) so creating a truly liquid marketplace for these identifiers would ideally include fixing the structural issues which would allow the constraints to be fixed and thus eliminate the market.
Bottom line, it would have been great if folks had thought of that first, but they didn't and while future protocol developers can (and should) benefit from that experience, we're stuck with V4 address blocks that are stuck in various regions of the world.
[1] The most persuasive argument against 64 bit addresses was that this would only push the problem down the road, whereas 128 bit addresses fixed things once and for all.
If we're talking about advertisable blocks, the Germany scenario doesn't happen. For the time being, there can't be a liquid market for /32's, at least not one that works acros multiple ISPs.
I agree with you strongly that the 128 bit IP address is a huge design mistake, one that has unnecessarily retarded the deployment of IPv6 (I personally believe fatally so). Even in the '90s, a 64 bit number was still a scalar on most platforms you'd care about. And even in 2012, 128 bit numbers aren't. I think people drastically underestimate the amount of work it's going to take to upgrade the huge amount of software built on the assumption that you can store an IP address in a scalar.
> huge amount of software built on the assumption that you can store an IP address in a scalar
The truly huge universe of code is at the application level. Sane applications use strings for remote hosts, or work with URL's, and let library- or OS-level code deal with the details of turning that into an address.
OS'es -- even Windows -- have supported IPv6 for nearly a decade. Most networking libraries I've come across also have IPv6 support.
The main problem isn't software, it's configuration. Namely the configuration of the interface between the customer network and the ISP. The ISP doesn't want to turn on IPv6 because it might break some customers, and the customer can't test IPv6 functionality since their ISP doesn't support it.
These problems are compounded for residential ISP's, since most of their customers can't be bothered to learn anything technical.
That is nice to hear. My biggest problem with IPv6 has always been the size of the addresses, which I think is too big, but I always thought I was just being old-fashioned. It's interesting to see other people think the same, although probably for different or more justified reasons.
At the time it was really frustrating to hear people say "64 bits is just twice as big as it is now, that will never last" and I would say emphatically "No, that is FOUR BILLION times bigger!" and get hit with "Well what if every light switch in the world had its own IP address huh? huh?" and I would say "Yup, if that happened we MIGHT use up may 10 or 16 BILLIONTHS of the address space just for lights, the equivalent impact of dedicating a class B in the current address space to lights, which nobody would really think twice about."
What an embarrassing failure of economic reasoning.
IPv4 addresses are a scare resource. Acknowledging that, and allowing them to be traded at a price that reflects their value, is the surest way to get them out of the hands of those who are wasting them in the way tptacek and the article describe and into the hands of those who will do something more productive with them. Pricing IPv4 addresses would hasten adoption of IPv6 by providing a financial incentive to switch from IPv4 to IPv6 to those who can do so most easily (i.e., cheaply).
In other words, if you want to "get around this mess", stop allocating scarce and valuable IPv4 addresses as if they have no value.
But (and someone correct me if I'm wrong), but individual IP addresses are not a commodity. If you broke apart that /8 and auctioned them off, if would be a routing nightmare.
Smallest prefix that can be meaningfully used is /24 (you cannot advertise smaller prefix), but I think that everyone advertising random /24s would lead to even larger costs for router upgrades than IPv6.
I think there are still bigger than /24s where /24 announcements aren't allowed. People filtered for a while (at least around 2000-2002) in those ranges, so you couldn't just take a random swamp /16 and turn it into a bunch of /24s and expect them to be visible to everyone, although it did work to most. I haven't paid much attention to routing politics for the past 5 years or so though.
> What an embarrassing failure of economic reasoning.
1. IPv4 addresses are overdemanded and undersupplied.
2. IPv6 addresses are practically limitless and will serve the same purpose once we switch over.
3. We should switch to IPv6 ASAP.
Where's the economic flaw?
(Also, it should be pointed out that allowing trades on IPv4 would not decrease availability, it would only increase it: Right now, you can't switch the owner of /x blocks without IANNA approval, so right now you have no choice but to push for switching to v6, whereas with trading you could waste money on v4 instead. Moreover, this would encourage the companies with the most money to buy IPv4 blocks rather than invest in switching over to v6, which would slow down the process for everyone since they're the biggest companies. TLDR -- force investment in IPv6 not IPv4. There's also a fairness issue since these IPv4 blocks were originally allocated for free -- should MIT be able to make billions on their block?)
Not any more. Within the ARIN region it's legal to straight-up sell addresses as long as the buyer is actually going to use them (people argue about this point a lot but it doesn't seem like a big deal IMO).
Actually no, IP addresses are not a resource. They're not limited at all, except by convenience which was chosen at a time when they had to chose a limit and it was not practical to set it any higher. All that is needed is to make a new standard which allows for a higher practical limit -- and that's what we're doing with IPv6.
So I?v4 addresses only seem valuable in this moment, because so many of them have been distributed -- bvut something that can be added in unlimited quantities (like a number) is inherently not valuable.
Do you remember when DOS and Windows file names could only contain eight plus three letters? Was the solution to proclaim file names valuable and charge extra for adding new combinations? Or simply to develop a new platform which allowed for longer names?
What you've done here is generalized the concept out so far that we can't discuss it. We're not talking about a fuzzy concept of "addresses"; we're talking specifically about IPv4 RIB entries.
The opposite is true. The current situation incentivizes entities with billion dollar budgets to hold address space forever, because there is no cost to them for doing so.
Realistically, anyone with an /8 to sell would probably break it up. There is some speculation that legacy holders are selling off their unused addresses slowly because a glut would reduce prices.
>More evidence for the core problem: fiat allocation doesn't work. If there was a functioning, liquid, accessible market for advertisable IP prefixes, you wouldn't have to convince anyone that a /8 was worth $1bn; it just would be.
you are missing a big point. This isn't like real-estate. This is plumbing.
Go do a search on "routing table growth"
This is actually a much larger problem than IPv4 runout. I mean, it's a problem that is further out, but it is going to hit well within my expected lifetime, and it's going to be a dramatically more difficult problem to solve.
The thing is, every time you break up a larger block into a smaller block, every router on the internet needs an entry in their routing tables[1] Usually, in content-addressable memory; Expensive stuff.
The thing is, the size of the global routing table? it's growing faster than moore's law.[2] I mean, it's not a huge deal now; for the price of a new compact car, you can get a router that has enough content-addressable memory to handle full tables and two 10g uplinks with 48 1g downlinks. reasonable cost of entry, if you ask me, and if you use dram (good luck with line rate 10G. don't think about faster line-rate.) well, then the cost is trivial. The routing table is under 500,000 entries in IPv4, so it's a trivial amount of dram.
The problem is that if this continues? (and all indications are that it's going to /massively accelerate/ if IPv6 catches on. At a minimum, you're gonna see the same number of ipv6 routes as ipv4 routes... and because IPv6 addresses are larger, ipv6 routes are larger) those routers are going to get more and more expensive.
So yeah, really? we need a way of charging, not for each IP, but for each announced route. Of course, that comes down hard on the little guys, whereas a per-IP charge is much more progressive. (a /8 takes up as many routing table entries as a /24.)
Getting by with a reduced number of IPs would be way easier for me than getting by without announcing my own block. Without BGP announcing my own block, my ISP has my balls in a vice. If they go down? I go down. More importantly, I need to change IPs when I switch to a new ISP.
I know what this is like, because I started out on my ISPs IP addresses, like almost everyone does. My provider found out that it's hard for me to switch off those IPs, so now I'm paying them, in total, for all the services they provide me, about twice market rate.
(It's actually really interesting, watching the dynamics of this problem play out on PPML or the like. Small players, like me, want to keep it easy to announce smaller blocks. We are fucked if you can only announce large blocks. Large players want tight restrictions on what size of a block you can announce, because they have the address space and they would rather not pay for more super-expensive CAM every time a Luke Crawford gets it in his head to start an ISP.)
[1] This is not strictly true. "every router that matters" would be closer to the truth. If you don't have full tables from multiple upstreams, well, you will be effected by rather more partial outages than if you do.
A Xeon E3-1220 has an 8MB L3 cache and costs about $210. If you deaggregate the entire IPv4 routing table to /24s to support fast lookups and store one nibble of destination data per route (probably just a destination interface number, if you can live with only 15 or 16 possible destinations, but you probably don't have that many upstreams), doesn't that mean that the entire routing table can fit in the L3 cache on the CPU (with a bit of the L3 not used by class E space, which will leave a bit of space for code, assuming you aren't doing anything but routing on this CPU)?
Now, it's quite possible that the Linux kernel's routing isn't dimensioned to provide this sort of performance, but if you want to avoid CAM in the IPv4 world, I don't see why you'd really need CAM...
(Also, I haven't looked carefully at how the L3 cache actually works on the E3s to make sure that it's shared between code and data, etc etc etc. Also, if you're actually implementing this, you presumably do want to include class E in the table, and then hope/assume that when class E doesn't get used, the CPU will be smart enough to cache the code instead.)
>A Xeon E3-1220 has an 8MB L3 cache and costs about $210. If you deaggregate the entire IPv4 routing table to /24s to support fast lookups and store one nibble of destination data per route (probably just a destination interface number, if you can live with only 15 or 16 possible destinations, but you probably don't have that many upstreams), doesn't that mean that the entire routing table can fit in the L3 cache on the CPU (with a bit of the L3 not used by class E space, which will leave a bit of space for code, assuming you aren't doing anything but routing on this CPU)?
Hm. interesting. so, uh, for simplicity, we have 256^3 routes, right? each one is, uh, what,33 bits of data? I mean, you need 3 bytes for the network and then what, uh, 5 bits for the mask? then okay 4 bits for the dest so 33 bits per route, no? so (33*256^3)/8 is 69206016 bits, or what,66 megabytes? sweet jesus, you are right. I mean, you are off by an order of magnitude, but at this scale, who gives a shit about an order of magnitude.
Interesting. 'cause none of the commercial routers do this. which is fucking weird. With this optimization (only store the first 3 octets, as you aren't routing anything smaller, have a lookup table for dest. addresses.) you could take full tables in puny amounts of CAM that come on, say, l3 switches. I will ask around as to why this isn't done.
(of course, with IPv6, this does not come close to solving the problem. /32 is the 'standard' handout for ISPs, and usually the smallest prefix you accept is a /48. There are a lot of fucking /32s.)
Also, you'd have some complications, as /32s are commonly used to blackhole DDoS targets, but you only have a handful of those, so that would add complexity but it wouldn't kill the idea.
I bet I'm missing something- I mean, every two bit ISP would be really happy to give you ten grand for a 10G full-tables bgp router, and you can get l3 switches with enough cam for that and a few 10G ports for little more than half that.
So, I /really/ made myself look like an idiot here:
>you can get l3 switches with enough cam for that and a few 10G ports for little more than half that.
Because I was confusing TCAM with DRAM; rather different sorts of things. (and that, I'd classify as a mistake of inattention. I made other mistakes in that message due to lack of knowledge, but, uh, yeah; I'm not usually /that/ dumb.)
Anyhow, uh, yeah. Looks like I also misunderstood what the parent comment was trying to do (/increase/ the size of the lookup table by breaking larger blocks into /24s, in order to reduce the number of cycles the CPU spends on the lookup.)
So yeah, in full? I'd delete my comment that I'm responding to here if I could. As I can't, I'd like to acknowledge my ignorance, for the record.
If I could rewrite that, I'd say something to the effect that huge amounts of effort have gone into making TCAM obsolete, and so far? well, progress is being made, but the progress isn't moving any faster than line-rates are going up, as far as I can tell. There are very smart people working on the problem, and they say it's a big problem. (my understanding and experience, as a sysadmin that also deals with networks, is that your linux-based PC routers can route something between 1G and 10G of line-rate small packets. Above that, PC routers can't cope with the PPS.)
Note, uh, I do have two spare E3 xeons laying about the office, and 10G interface cards for both, as well as (at least for a short while longer) two nice arista brand 10G switches, so if you are in the sunnyvale area, and you do want to test an idea or bench the current state of the art of software routers, well, it's something I've gotta do anyhow. I still don't have anything even attempting to route my new 10GbE Cogent port. (I've split off a 1Gbe connection using a switch, and plugged the 1Gbe connection into my existing quagga router.)
Given your constraints, I wonder if what you actually want is one switch for each upstream, and a gigabit link from each of your physical Xen servers to each switch that goes to an upstream. If your switches have to route at all, they can each have a default route to the one respective upstream connected to that switch. Then, have a VM on each physical Xen server with full routing tables to control how the outbound packets from all the VMs on that physical host get routed. And hopefully your switches can deal with a /32 route for each VM or something, if they just have a route per VM plus a default route to the one ISP they're connected to.
There's also the variant where you have a /24 block (or something) which has one IP address which is your ISP's router, and then one IP address per physical Xen server, and if your ISP is willing to take individual /32 announcements to your servers (I think this is technically feasible, but may require more mental flexibility than some ISPs have; on the other hand, I think you're talking Cogent, and mentioned that Cogent is willing to peer directly with your customers, which may imply that you're dealing with a flexible ISP), you can have the ISP do the layer 3 routing to the right physical host for traffic going to each VM. (The idea is that the ISP would not propagate the /32s to their peers; you'd separately send them a broader announcement that they could pass along to their peers.)
One other issue in this scheme is that if a single gigabit link between one physical Xen host and the switch going to one ISP breaks, inbound traffic that happens to come to the ISP attached to the switch with the broken link ought to have some mechanism to find its way over to the other switch for the one physical Xen host that has the broken link. I think there are ways that OSPF and or iBGP with the non-default next-hop-self setting can be made to do this.
More evidence for the core problem: fiat allocation doesn't work. If there was a functioning, liquid, accessible market for advertisable IP prefixes, you wouldn't have to convince anyone that a /8 was worth $1bn; it just would be.