These are unedited transcripts and may contain errors.

Plenary session, 2:00:

3rd May, 2011:

CHAIR: Hello. We're past 2:00 and we're going to continue for the day. So if you get a receipt, take your place. There's more IPv6 to come. Short one. House keeping again. Switch your phones to mobile when you are you're in the room. The sessions are recorded. If you have any questions, please make use of the microphones and state your name and affiliation. First speaker of today is ?? I think quite interesting because he actually is going to show a real live solution to a real live problem that a lot of people are coping with.

SPEAKER: Thank you. Hello. I work for Altibox. It's home in Norway. I will talk about our company quickly so you understand where our problem is coming from and some of the history that we have.

We're ?? our ISP started in 2002. Currently we have 215,000 customers home and business users. We do triple play or multiplay. Quick run through of our services so you can look more on the slide sets if you want that. But we have done this since 2002 and are quite successful on that one. Customer growth, you can see we're growing quite fast. 54 depletion will be a problem for us but I'll come to that more in a bit. Altibox is a kind of special company. We're only a product house. We have a lot of partners around, so we actually only supply the service. The partner, other utility companies, typically, mainly utility companies do the actual fiber, they do the only routers, the only CPs, but my company, we actually manage it, we run it, we support it, we do all the stuff around it. So it's quite a special business model. We're located all around Norway, also in Denmark. So we're kind of Nordic as well. The business model is more or less like this, the partner sells, develops the areas, they install the equipment, we do the producting, the billing and also the operation. And the partner do the invoicing, the first line customer service for all the customers. This is a great way of doing things because we can correlate the things we're good at and also the partner have an ownership, they have their investments, they have their stuff up there. So it's working great for us. We have a national backbone, looks like this. We're upgrading for the time being. You can see we have all over Norway. So the special thing here is that we have the Altibox Nordic backbone, that's Altibox's network and the partner model is based on that. The partner model owns this part, typically that part and a lot of other things. We do the transits for the partner and also have the data center for the supporting services for the customer, DSB service, DNS service and all that stuff. And the special thing with the way we've done is is that we actually only in 99% of the cases supply the fiber through the curve. The residents themselves dig the fiber, put it into the house and then our installers come and do the end of the installation. This is an image series where they took the fiber themselves, three, four, five kilometres and installed everything themselves and our engineers came and installed the fiber and the end equipment and all that stuff. So in that way we're able to keep the service very cheap, which is good, because it creates a demand on our service. So happy customers. All kind of happy customers.

Just a quick disclaimer, I'm only covering what we call Internet play, because that is the issue at hand. We have all garden services, we're not actually working on that right now because it's not a necessity for our current problem.

The reason why I talked about the partner model and showed you the drawing is that we actually do some partner aggregation. We typically give them a /20, we produce the customer Internet services from them, so we make aggregation very easy in our own core network. It makes the growth and we also are very eager to conserve IP addresses, obviously. And we also do the access router model, I will show you that in a bit, but we do layer 3 switches at the edge, both layer 2 and layer 3 to the edge. And we have the router instance there, have a prefix size similar to the connected customer. The amount of collected customers. So if we get more customers in an area, we add a second IP address. This was actually a problem before. Before a few years ago we did v4 addressing compared to the number of ports on the switch. But, of course, if you have a 348 switch port unit there and you only have 90 customers, that's quite a waste. As you can see, we have 43% utilisation and that stopped me from getting more IP addresses, and now we're up to 75%.

We have two different access models, one for corporate customers, standard VLAN model, but we also have shared VLAN, N1. That's been done since 2004. It works perfect for our services and we look into the challenges of doing this in v6 soon. We also own and manage the CP. We have a whole infrastructure from the core network access router and also the CP, our partner own the CP but I'm talking Altibox has the whole partnership.

How we do shared VLAN today, v4 only, so you can see how we do stuff. This is trying to illustrate the layer 3 switch. We have the private isolated VLANs there. We have the router instance, the VLAN instance in this part. Of course, it's a router to the network. So the number switch port varies of what kind of switch it is, typically 324 to 384, and might have up to four or six access router in one port. As mentioned the subnet size depends on the number of connected customers. If you have 24 customers, we do /6; if you have 324, we do maybe two /25s and maybe three /25s, depending on how many customers we have connected up there.

We only do one IP per CP on the CP side. And that's restricted by the DHCP server. We also DCP relay forward packet to the server. And we use option 92 as a binding for isolating or identifying the customer's port and slot so we know which customer gets which IP at which time. We do the delegation in the DHCP server, so the customer will get the same one each time. The reason we did that is that the customer are also able to set the CP in bridge mode. When a customer hooks up a PC, if we use the same ?? if you reuse the Mac address for the host, they wouldn't get the new IP address when they had connected another PC. So instead we translate option 82 into a Mac address then we're able to give the same IP to the same customer in the same port every time, independent of how we actually do ?? which equipment he actually hooks up. So we have full control of that.

Current platform today is Cisco. Works perfectly. Very happy. Good support.

A bit into config, jumping right down to it. The trick of doing VLAN, snooping, also specialist part local proxy, because you need to control and that's done by local proxy command. We also have the private VLAN mapping. We do VLAN 92 and we map that to 102 and typically switch port and track service. Standard out of the book.

So why did we start doing v6? Simple: v4 depletion, of course. Everybody which grows, specially us who grows, have this problem. We are increasing the customer base with approximately 50,000 customers a year, which, in Norweigian scale, is healthy, if you might say. All the ISP growth, we'll hit IPv4 depletion. Of course I know when but I won't tell you. So we're very persistent in giving our customer the best Internet service they can get. And if we can't do that with v4, we need v6. So that's why we need to do this now, and hopefully, also, the rest of the world will see this point, as well. And, of course, we need to use transition methods of course. We will deplete v4 but we'll look at that later.

How did we start? Started summer 2010, last year, actually not a year ago at all. When we looked at the estimates, all our CXOs, they actually understood the problem, which is quite nice for us engineers who actually work with this. So they started giving us funding, started giving us product money and we were able to start the project in September. And we did the standard approach. First of all, we identified which equipment needed to be upgraded according to a specific plan and that's to give IPv6 to all Internet facing customers, all garden stuff, backing stuff, we don't care about yet. We did, then, an evaluation and planned an implementation and we are at the testing phase right now. And hopefully we'll be able to implement over the summer, start the real implementation and give our customers and web server pages, web services v6 connect activity.

So we have divided the project into three parts. First part we're working on is getting v6 into the core network, enable it for all new residential services with an underscore on the new. Enable it for all corporate server customers. They want one VLAN model, very easy. Turn it out. We'll do that on demand for all our corporate customers. And also enable it for all our Internet facing services, like web servers, mail servers, all that stuff. The target is Q411. We are quite aggressive on our planning. We need to be that because of v4 depletion.

Next phase is how do we handle v4 depletion? We've set our self one more quarter to solve that problem, because we need to be ready and I don't want to stop doing v4 when I'm out. I want to stop before so I have someone safe in case of bad weather some day. And last is giving v6 for all legacy customers, hooked up right now. Giving ourselves a year to do that, because it's a lot of planning, money involved, access routers to be changed, features that we need and stuff like that.

So phase one has given us an overview of the cost, which is approximately 350K euro, which is not that bad, actually. So we have a very kind of easy upgrade. Most of the Internet services phasing platform is linear based, so open stores and our new platform, both CP and the access routers, are IPv6 capable, we just need the software. We've done lab testing. We tested it in London, works perfectly. But we have a risk of this schedule being changed due to lack of features and stuff. But I'll come back to that.

So we have a few designed principles, do it as simple as possible. IPv6 designed to be as similar to IPv4 as possible. Same security, same redundancy, same traffic pattern, ensure co?existence, and I will not do any transition technology without also doing IPv6. That's my message.

So v6 address plan. Same principles as before. We have an established system, we have an established existing system and procedure and everything is done the way we do today. So we want to do this as similar as v4 as possible. We need /39 equivalent to /20 to the partner. Give more /39s ?? or shift the bit one and give it /38 in case there's a bigger partner. Partners vary between 2,000 customers to 50,000 customers today, so it varies quite a bit. And we allocate the /48 for the partner, we do link local on all links except for the links between core network and the metro core network and all pairing. On those links we use public links with /64 allocated but only use 127, that's for ease of management.

We will do ?? since we're doing [end to one shed] VLAN, I'm not going to talk about the core, you all know about that, but for the anti?VLAN share model we're going to do /64 on the access router itself but the CP will get a /28 from that. All the residential customers will get a /56. Small corporate customers will initially get the /56 but if they need more they'll get more without many questions and medium large will get the /48. We also allocate /39 for our own data center to have somebody to play with there.

So the backbone, tested in the POC lab. Already starting implementing 6 PE, do the 6 VPE, up and running, global routing table, all pairings are in place. We actually did the gathering a few weeks back with v6 native and it works perfectly.

This is how the partner network looked like. We have PEs, the partner have their core, they have have core routers and they have these access router or layer 3 switches, we have the customers around here. The routing protocol between access router and the core is OSPF and, instead, is BGP and also and dual routed, depending on the size of the core and stuff like that. We have different models to handle there.

So how do we do dual stack on shared VLAN? Much on the same principle as before. As I mentioned, the /64 ?? and we assign addresses. We're not using slack. That doesn't work, as far as I know, due to some mechanisms.

We do a /56 for prefix. We have the DHCP forwarding. The trick here that we do is we do not advertise the prefix to the CP. And that's one of the clues here. Because if the CP is compliant to our C 5942, it will assign itself a /128 based on the DCP address it gets. In that manner, we don't need an enabler discovery proxy. So we come around that problem. We don't need that because the CP doesn't know if anybody else ?? all the traffic will be sent up to the central VLAN.

So this is running in my lab, working perfectly. So I just need production code to actually deploy it. We're working closely to get that up and running.

This is an example from the vendor code, which is depending code, we have the ND inspection, command called glean, destination guard. These commands are implemented in the 6500 platform from Cisco. You'll find more information there. These are the security features that one would need for doing shared VLAN and security, these will inhibit or do the equivalent to the CP snooping table, arc inspection and stuff like that. Works perfectly. You also see the no advertisement from the prefix in the discovery, which is the clue here for keeping it as simple as possible.

So these are the features that we need which are not in the production code yet. The layer 3, layer 2 mapping equals the C ?? the prefix address space, we have extra address space delegated to the customer. We need to secure that, as well, because it's very easy without those security measurements to steal the prefix. You can easily spoof the Mac address and get the prefix and stuff like that. The vendor needs to implement features to get those security measurements in place. In the code I have it is in place. We know it works; it's a matter of getting it into the production tray.

So the final commitment dates, we're still waiting for those, but they're indicated Q2 2012, which is fairly late but we're working on that. Close relationship with Cisco to get this running. I got a message yesterday evening, option 32 on the platform, not the new feature, just a bug. They'll fix that and we'll get it in July. So I'll be able to do dual stack IPv6 without the security features. I can live with that. It will be a small customer base, not many customers know this. If they do stuff they will only ruin for themselves and the customer on the access router, not on the whole network. I can live with that for now. So, Mark, please, continue doing a great job.

AUDIENCE SPEAKER: Thanks for the help.

SPEAKER: Thank you.
Plan B, I need a plan B, it will be six RD, I don't say it's bad, but it's tunnelling, and two great people talked about this yesterday who normally don't agree, tunnelling sucks. I agree totally. But, in case, we have a plan for that, as well. As I mentioned in this point here, in the legacy partner for network, we probably need to do some kind of six RD, anyway, since my partner owns the equipment, we can't demand him to change that to newer access router without normal standard change up. So we'll have this as plan B as well, so they can get dual stack as well.

But, of course, depletion is coming. We need to have a plan for that as well. So, for now, our best choice is Nat 444. I'm sorry to say it. I don't have any other running code option. Luckily for me, it's no change to my provisioning system because my provisioning system doesn't care. But I need to implement VRF light to separate the Nat 444 traffic from the global traffic I'll have. I need to do stuff in my core network. So we'll ?? we don't have a thorough plan for this yet. This is phase 2 and we haven't started looking clearly into that.

There are other ways, too, but I need running code and native IPv6. I mentioned a few of them here. Hopefully, we can resolve this, all these features.

So CPE. We have ?? we're using [Telcy], it's an Italian company located in Venice. We've been working closely with them for many many years. This new CP will have ?? it's fairly new. It does support v6 but it does not support v6 in hardware yet because of the broad care chips in it. However, thanks to all the guys, they made these RFC and TR 124 I two, when we approached Telcy with this, they said finally some specifications and they brought it to dot com and they said just the same. And they've been working closely to get this code up and running, with full TR 124 I2 must features in place. And my better code is coming on Thursday. Hopefully, I will start testing stuff real soon, on these units. But, as you see, it's a fairly aggressive unit, but because we have 3030 megabit and up to 400 megabit for those who want to pay the money, I need a CP to handle it, but because it doesn't do v6 in hardware, they tested it in a 100 megabit. I can live with that. And we also tested 6 RD to 60 megabit, which is also fairly good to be a pure CPU in terms of case. So we're very happy and I'm really looking forward to coming home from this meeting to start testing stuff and see how things are. And of course I need to have it world 56 day as well, with our main website. I mention this, I've seen on the mailing lists, do not get the customer sites up and running, we will not do that, it will only be the informational web server with the sales information and stuff. The customer service sites will not be able to use that.

The back end systems, minor adjusts, fairly easy, no worries, we use diamond IP and also installing a new NMS system for support.

So we're very focused on v6, working very hard to get it done, we need to get it done. My personal opinion, we'll probably be the first large ISP in Norway that will support IPv6 for residential customers. If anybody disagrees, throw something at me. We have some challenges with the vendors, but we're working it out as we go. We're very happy for the support we get and hopefully we'll have running a code and doing this in real life quite soon. But to get traffic for ?? to get traffic on v6 we need to not only do stuff ourselves, we need to do stuff outside our own company. We're also very eager and involved in the national groups to encourage ISP service providers, company providers, all of them to start doing v6, so we're very ?? working very closely with those affiliations as well because that's very important for us.

So any questions?

AUDIENCE SPEAKER: Hi. This is Chris Buckridge from the RIPE NCC and we have a comment and question from Jabber, from [] [Tore Anderson Red Pill Linpro]. One of the claimed advantage with IPv6 is it will restore the entwined transparency that was lost when IPv4 became common place. You wrote in your slides that it will be RFC 6204 compliant. It states that a compliant device should support RFC 6092, which, in turn, specifies might go about blocking unsolicited inbound traffic just like that device does out of necessity. However, RFC makes no recommendation whether or not this firewall function alternate should be default on or off. So you have a choice: either you provider users with security and no entwined transparency or the other. I'm not trying to start a discussion on whether or not this security advantage is real or not. While I'm sure your customer will be able to change the defaults easily, as Lorenzo pointed out earlier, almost nobody does, therefore the default setting that gets chosen by Altibox is important as to whether developers can go back to writing applications. So my question is this: What are Altibox's thoughts on this dilemma; have you made a decision already?

SPEAKER: Thank you for your question, but the short answer is: we don't know yet.

(Applause.)

But we're following close and seeing what happens. Security is always, as you know, a difficult thing, because, from the customer's perspective, they're kind of used to security through the false impression of the Nat because normally a standard CP default set?up doesn't have a firewall; it has a Nat, which gives you some kind of security. But there is also many mechanisms for opening that firewall but on v6 we don't have much of that yet. So, on a personal note, I would probably have the CPE firewall on v6 turned off, but that's my personal opinion. So, yes. Lorenzo?

LORENZO COLITTI: I hope you can pull it off. Lorenzo, Google. Couple of questions: Number one, what I hear all the time from ISPs is everything else compared to replacing the CP is trivial, cost?wise. Basically, the CP dominates all costs if you replace it. With that in mind, I was looking at the slide of capabilities and you said v6 doesn't do hardware. Are you concerned that you're pushing hardware that can't do full scale v6?

SPEAKER: Yes, of course we are. But the problem is broad com chips that do support v6 in hardware is not bound until this autumn and I need to deliver customers and this is my only platform. Yes, it's a problem, and, yes, we'll bite ourselves in the whatever at the later stage, but we can't stop projection just because the vendors have not.

LORENZO COLITTI: There are vendors that do wire v6.

SPEAKER: Yes.

LORENZO COLITTI: At least two I know of.

SPEAKER: And they also bid on the CP.

LORENZO COLITTI: Second question was: So it's ?? it's a discussion between shared and one to one VLAN and this discussion keeps going on: What really came out of your talk, I think, is that you have a v4?based service, you want to bring v6 to it therefore you want to do it with the minimum amount of change and you want to ?? so you're using all these complex v4 features to make it work and you have to make all these complex v6 features otherwise you can't deploy it. The question is, if you did one to one VLAN you could turn it on today. So the question is: If you look at the long term in a future where v4 will be either not there or tunnelled over v6, why do you try to take the v4 design and turn into v6 instead of changing the design now? Out of curiosity. You could look at it, I have to change the design anyway, instead of doing all this complex stuff and stuff that's likely not to work, all the v6 security stuff, you could say, well, I'm going to change it now, people who get v6 will get more complex v4 but that's going to work because the vendors support it, I'm going to make v4 complicated but keep v6 simple. My experience is the less v6 you try to get boxes to do, the more reliable they are. You have to be very careful what you ask them to do. In general, have you thought with that approach?

SPEAKER: Yes. The biggest issue we have there is we have systems today that were built back in 2004, for parishoning this, and it's a matter of, for the short term, it should either ?? should we do stuff to change things to one to one or should we ask our vendor to change stuff so shared VLAN actually works? For us it's easier to ask the vendor than doing it ourselves, obviously. In the long term, we have been discussing different models, but we haven't looked thoroughly into it because we're more concerned with getting v6 up and running and not being able to support our customers, probable, when we deplete the v4, so it's actually not ?? hasn't been an issue. We've been thinking about it very thoroughly.

SPEAKER: One comment to that is: You might want to keep that in the back of your mind as you go on and actually progress through the testing phase because all my experience with this stuff is that it kind of sort of works but not quite. So just, as, you know, note of advice as we've seen lots of bugs, keep the option open to do one to one in case you hit a bug that you just can't.

SPEAKER: Thank you.

SPEAKER: Short one. We're slowly running out of time. These are the last three questions I'll take and we'll go on with the next presenter.

MARK TOWNSLEY: Mark Townsley, Cisco. I'm going to summon my friends from Free Telecom to speak a little bit about what they've done with residential employment in response to the question about the firewalls. He mentioned what choices are made now might lead the industry. Free has four million customers, over 500,000 and, as of now, all new subscribers have v6 by default. The 500,000 chose v6, and none of them, since day one, starting 2007, have had a firewall function available unless they went on bought it and added it. None.

SPEAKER: That's interesting.

SPEAKER: Take that. Run with it. Copy what they did. I think it's the right way to do.

SPEAKER: It might work.

SPEAKER: One of the points was most of those devices that have IPv6 have a firewall turned on by default on the host and this is one of the points, okay, well, if the hosts are doing it for me, why would I do it myself. And you also get the in bank.

SPEAKER: Thank you.

PATRICK ARKLEY: Patrick Arkley, Verizon Europe. Just a clarification on your IP address management. When you say "customer," do you mean customer or customer location?

SPEAKER: Customer.

SPEAKER: So you give a small customer /56 and when he has his second location going live, what do you do; do you rely on the customer to provide you a network or would you assign it?

SPEAKER: I guess you're talking about the corporate side of our business.

SPEAKER: Yes, not residential.

SPEAKER: Residential is easy, of course, they only have one location. It's a good question. We haven't dug into that far, how to do the corporate stuff because we feel that that might be easy, but obviously it might not be that easy. But address?plan?wise we're looking at at least having one prefix per customer and then we have to look into how we can split that up into different parts if it's necessary. We do purely a free service.

SPEAKER: Do consider the fact that you don't want the customer to give you networks for your additional locations from their assignments. You control the assignments.

SPEAKER: For most part of the customers. We also have corporate customers who have their own assignments which we do transit for.

SPEAKER: Of course. Okay, thank you.

CHRIS BUCKRIDGE: Chris Buckridge, RIPE NCC, with another, much briefer question: Why was /64 assigned to back instead of /28 which Cisco recommends?

SPEAKER: That must be a lack of information on the slides. Of course we do /128 through back, so sorry about that. It's a /124 for each throwback.

(Applause)


CHAIR: Next one up is Ondrej Sury from CZ.NIC.

ONDREJ SURY: Hello. While I'm presenting work done by a student at the University of Brno. Too much. Okay. This was done as a master thesis. We have a corporation with difficulty of information technology in Brno, while we do supervise some of the places there. The code is to be released yet but I already spoke to the student that we will release it under some free licence. If you're really bored and want to watch me, you can go directly to the web page and try it yourself. I have some demonstration ready. Please be easy on the server while it's still a toy ?? I mean, tests.

So there are some parts of the project, there are data mining robot, pearl script, which harvested the database. And it's on best effort. So there be bugs and there probable are bugs. The next part is the script trance forms those addresses to coordinates using Google geocoding API. And while ?? it is if he had into the database and there is ?? where there's a utility to look up data from the database. And the last part which I will show you here is the visualization using Google maps and IPI.

There's command line interface so you can look up the specific IPv6 address. You can do full text search in addresses and a few pictures before I show you the demo.

This is the Czech Republic IPv6 map. It's not very accurate because the grouping is done by the Google map so it's not always accurate but it's close to real picture. This is the Netherlands IPv6 map. As you can see, there is 34,000 prefixes somewhere. It turns out it's six SS prefixes. While I removed the feature from the web interface because it will eat your browser if you try to show up so many points on the map. And the last picture, this is the US IPv6 map. And I will just show you ?? this is the web interface, and I tagged a feature that you can type in, for example after nickname, and it will show up the IPv6al indications. As you can see, sometimes Google has it wrong. For example, this is a South African address, but, for some reason, it's located in India. You can click here and it will zoom. For some reason, the Google API doesn't know Bermuda which may be why the IP addresses are lost because it's in the Bermuda Triangle. These 19 and these 19 are in Bermuda. I don't know what happened. But ?? you can try it yourself, it's already on line, so I will not bore you more with my ideas.

So this is just quick summary of sources of errors you may find there. The who is information is generally better than IPv4. Sometimes this is disagreement on the region versus registrant. For example, I found a blog registered for 2007 and it was allocated for Australia but it's a US?based registrant so it shows up in the US. Sometimes, there's an error because the US States have similar codes to country codes and the last part is the errors in the script. So this is the end of my short talk about the IPv6 allocation.

Any questions? If you have implementation questions, you'd better send it to the other guy listed on the front page of the presentation. But I may try to answer for them.

CHAIR: Nobody? Really? Not? Eric?

ERIK KLINE: I might as well. Erik Kline from Google. The sources, it was just data scraping from Whois and from ??

SPEAKER: From delegations, we have delegations list on the IFTP.

SPEAKER: It's not actually looking at what's actually deployed? Multinationals that can register things in dozen of countries could show up of being in the country of their corporation?

SPEAKER: We're just looking at the Whois corporation. I saw Google, as well, they have block in Australia showing up in US.

SPEAKER: And stuff in Bermuda, for example, are there really 38 IPv6 prefixes actually being served in Bermuda or is that where the company is incorporated? I don't know if there would be that much IPv6 in Bermuda. Of course that would be awesome.

SPEAKER: There's two IPv6 in Myanmar, which surprised me.

SPEAKER: So you're not actually looking at ?? there's no measurement of what's deployed or comparing it with what.

SPEAKER: I guess the student wouldn't be able to make it to finish his school to take on a project that big.

SPEAKER: He'd need to be on the web server side to be able to do some of that kind of work.

GEOFF HOUSTON: Geoff Houston. APNIC. It just appeared that the IETF needed an IPv6 prefix and when I looked around in 2007, APNIC had a policy that allowed them to do that easily and I think they are they were having trouble with that IRI policies at the time. These things happy. A number of RIRs, including APNIC, have been very much behind making v6 incredibly easy and v6 is really a one?click service so you can get a slash 32 straight away. In some of these cases what you find is there's an awful lot of effort to see the entire world with v6 addresses and that's probably a very good thing.

SPEAKER: Thank you for clarification. I was trying not to pick on APNIC. There was 2007 in use delegation.


ERIK KLINE: Erik Kline from Google again. Follow?up question. At IPv6 I saw a conference in Paris in 2003 the topic of geolocation came up as being a barrier on the server side because you do have to comply with legal requirements for things you can and cannot do and show in certain countries. And one of the ISPs said wait a minute you guys want this information in you want our geo information? Why don't we just give it to you? I said, I don't know, why don't you? He said every time we move a block from Germany to France, I have to go out and actively ping all the geo people and is ask them to update their databases because the rules change. There was an attempt at the moment to say, well, if you as an ISP really want to publish where these blocks are located we'd love to consume that as an additional source of information. Do ISPs really want to publish this information? Is there a benefit? Would they be interest? Formats and mechanics can be discussed later. Is it possible? Is it desirable, that sort of thing? Maybe that's just sort of an open question for the floor.

SPEAKER: I'm not ISP so I cannot answer that. The Google still shows up the Italian version here at RIPE.

WILIFRIED WOEBER: Wilifried Woeber. There is an item for the working group meeting on Thursday, which is or was ?? it was taken off but we we can reinstate it, which is a follow on to the presentation we had in Rome about the gentleman who was arguing in favour of extending the registry database to include some sort of geolocation information. If you're interested we can have a chat afterwards and put something together. Taking the working group chair's hat off and speaking as a person who travels a little bit around the world and who gets bitten again and again by the use of geolocation information, the question is who and what should actually be in control of the data that is in the registry? Because there are lots and lots of different things that consume that information and react to it one way or another. And this reaction might be in favour of different parties in the game. It might be in favour of media rights management, it might be in favour of free information offer like to give the news information in Dutch if the thing believes I'm physically a Dutch?capable person. It might be against the interests of someone what I want to ?? for example, when I happen to be in a hotel in Amsterdam, I'm believed to be a Dutch person but I'm still hooking up to my system and home base in Vienna and I would like to be geolocated as an Austrian system. This is from my personal point of view. I would be interested to have this discussion started somewhere, whether it should be or needs to be in the database working group or somewhere else, I'm hope.

CHAIR: No further questions then. Thank you.

SPEAKER: Thank you.

(Applause.)



CHAIR: And last speaker up for this session is Arien Vijn from AMS?IX about some behavior they've seen on the peering LAN.

ARIEN VIJN: Okay. I'd first like to start with something actually very nice, something pleasant. For example years we have on our websites this graph here that shows the percentage of IPv4 and IPv6 traffic. For many many years this was all yellow, all IPv4 but IPv6 got visible. We can see it.

(Applause.)
So this is actually what it looks like in more absolute terms, almost 3 gigabits of traffic peak. Surprisingly right after the IPv4 address blocks were handed out to the IRIRs. We saw an uptake in IPv6, more people asked for IPv6 address and we got more issues. That's actually the agenda for this talk. The recent issues we saw with regard to IPv6. Of course we are a layer 2 operator. We only look from a site. We don't do IPv6 ourselves. We do it for our own systems and do IPv6 peering ?? the peerings between our members is not our business. So what we mostly look at is the multi [] /KOS /ET traffic on the peering LAN. We have machines looked up, we have the multicast there, we see the broadcast there, we can actually see something on that.

If you look at the number of neighbor solicitation messages that are multicasted, in February it reached about 18 million messages a day. Is that much? Well, if you compare it to its IPv4 counterpart, Arp, we can see Arp is pretty stable, around 5 million messages a day. So IPv6 is bigger than IPv4 in this respect. We don't know if this is a good thing. IPv6 is designed a little bit better than IPv4 in this respect. Hosts, routers in our case, don't have to look at all these 18 million messages and don't have to persist them. They have chopped up in multicast groups. Most routers can ignore this in hardware. Most are able to ignore multicast groups except the ones the router is interested in. This is actually according to the book. But we got actually a complaint from a Cisco GSR user stating that all messages were going to its route processer and 18 million messages a day and this is peaks comes and goes, actually the route processer needs to maintain BGP, maintain the messages and got bombarded with all these discovery messages for which most of it weren't even meant for this router.

In the end, first they asked us can you put a filter on this. We couldn't. He implemented a IPv6 filter only allowing the multicast and this we solved the problem. But yeah, if you have a Cisco GSR, you might want to have a look at this.

When we looked further into this problem, we noticed that most of the neighbor discovery messages were only for a smaller set of addresses. And when we investigated more of these addresses we noticed that these addresses, these routers don't react on the neighbor discovery messages we send them. We multicasted them. But if you set a static entry in the neighbor cache and first send unicast?only traffic, then it suddenly became reachable. So this posed a problem. Contact being the members affected, so contacting the members for which there are lots of messages, turns out fairly often that people put way too restrictive ICMP v6 filtering in. It looks to me that people apply ICMP filtering from IPv4. So it's kind of scarery, you want to ?? you want to firewall that from the outside and that is being applied to IPv6 too. And, therefore, blocking the neighbor discovery messages that are coming in, you can still send them out, still work them because you send them out we got unicast reply back which we rerouted and expect and things start to work. But if you get unsolicited message then it doesn't work.

So this was ?? this is actually a problem that is still common today. Today, actually, we got already that was doing the same thing.

Listener discovery turns out to be a big problem too. Multicast listener discovery is IGMP for IPv6. It turns out most routers you cannot switch it off. You can type that you don't want it but then it only stops to ask as a courier. If there's one courier in the LAN, most routers do reply on it, send a listener report to which multicast groups they are interested in. The newer Linux kernels, you cannot switch that off. I've been looking for the command, it is not there. If you want to block it, you have to use 56 stables to block it from your own system. It makes Linux operating system that is ?? that is the case. Why is this a problem? It used not to be a problem. We had some MLD messages, household, sent an email to query, can you switch it off? It wasn't a big deal. It became a big deal when a popular Cisco platform apparently switched on MLD snooping by default. And it looks like, first of all, most MLD messages don't have the multicast groups in them, I'm interested in the multicast group for neighbor discovery. This Cisco platform, they get an MLD report message and it starts to block off all multicast that the router is not interested in, including neighbor discovery. The funny thing is that it seems that this also blocks messages from the router itself, but not always. How it actually works I must show you. So sessions tend to go up in the beginning and they tend to stay up because of unicast. The multicast is nothing for those routers. So people say, hey, this is no problem, why not contact them, all my sessions are up. And I say, well, you get a lot of queries for your router and we can't resolve your router's address but all BGP is up. So where does all this neighbor discovery come from then? It turns out these are mostly users that also peer with the route server. So they expect traffic from other sources than they have a BGP session with. That traffic is actually the route server get, next up the router sends the traffic needs to resolve the address, that doesn't get resolved because of MLD snooping this message never receives the router so there will never be an answer. The effect is quite stunning. This is ?? these are the messages for a certain router. And here people switched MLD snooping off and immediately you see that the number of queries for this router ?? and these are the queries per power ?? dropped dramatically till almost zero. It also creates suddenly quite a peak of neighbor discovery messages from that router that weren't there before and that suddenly we self next and traffic can start flowing. Because everybody looks at BGP and not all the traffic sent unless a customer complaints, nobody noticed it.

So these are more sort of book like operational errors. For years with IPv6 we have seen books, many many many book bug. It's actually our frames because we're very vigilant on filtering ?? our only reliable way of lip resolving. We only allow one Mac address. If you see something else, then this gets lost. We now know if you see a certain Mac address we recognise IPv6 in those addresses. Cisco GSRs are doing that. We worked with a certain member and next we got this got this bug report where it turned out certain Cisco GSRs sent corrupted NS messages. This is not really a big issue. It doesn't really harm your traffic. Junos, Junos is notorious in sending encapsulated frames for many many years. Sometimes we worked with some members to get it fixed but it's a question that comes back and back and back. And we now can recognise these kind of addresses. This is just the start of an IPv6 address at the place of a destination of eitherer net header. It turns out to be unencapsulated BGP. Well the bug department of mostly harmless, because I presented this before, well, it's mostly harmless. Well, let's do something harmful. Junos 10.4, all releases out and also Junos 11, starts to answer on neighbor discovery queries for addresses that are in there neighbor discovery cash. Of course they block all the multicast groups that are not in there, there are not interesting. So if this query is in the same multicast group it hits the processor of the Juniper and sends the answer back and makes the Juniper a neighbor discovery spoofer. It starts to answer with its own Mac address for another IP address and can take away traffic from there. This is actually a harmful bug because this causes other people traffic to be dropped or to be moved to another router. This is a bug that you don't necessarily have to experience because in many many platforms the multicast groups are basically the same as the addresses so you won't notice this problem. But on the numbering scheme that we made years ago on IXPs, the multicast groups actually shared amongst a number of routers. So this is actually a harmful but common Internet exchange.

So, to conclude my short talk, in conclusion, what we see a lot is that multicast or neighbor discovery messages are being blocked and that protocol abuseness makes people think that everything works and that's what I think was the most interesting thing that we noticed lately. More, many years there are still bugs with IPv6 neighbor discovery. They used to be harmless, but recent Juniper bug doesn't make it that harmless.

So does this mean that you should not do IPv6? Does it mean that there are many many bugs, all problems? No, it does not. My experience and IPv6 is just something that you must do and then it turns out not to be that hard. It's not much different from IPv4. Of course, I'm preaching to the choir here. It's not that much different, there are bugs there as well, there are issues there as well. I would say just do it, IPv6. That concludes my talk.

SPEAKER: Hi, this is Wolfgang from DE?CIX. I would just like to add some numbers to the harm for the Juniper bug. We've seen that to a cure of about six customers, you actually don't notice it as an exchange unless somebody else complains about that and you start looking around who announces more than one IP address in the neighbor table, then you contact them. If you are lucky they resolve that in a short time or they shut down IPv6. Or if you're unlucky, we had to shut down their IPv6. We had that on six customers, four of them replied. Two of them we had to basically block IPv6 on their port until they did a software upgrade on their Junipers.

SPEAKER: We only had it with two people. One switched off IPv6 and one down graded to 10.3.

LORENZO COLITTI: Lorenzo. That's a bug and that's been fixed?

SPEAKER: It's not been fixed yet.

SPEAKER: We should put that version on our router so we can explain about it.

SPEAKER: I looked this morning on the release notes and it's still there.

SPEAKER: Okay. Good to know. We can ask some questions. So the thing I want to say about is to your earlier point about IPv6: I think that's an important point because the people that are usually accustomed to doing IPv4 ackles, they don't think about Arp because it flies under their radar, when you type it in Juniper, they don't see those packets in before. They adapt the ackles to be six and we don't need this, don't need this, it took us several times to get it right. Because depending on whether the remote peer used link local or depending on whether, you know, which direction it's in, it could be that you establish the peering in one direction, the other one doesn't work. What I'm saying is that if you've done some work already to write up this problem, it would be great if you could come up with a web page that says when you set up the IPv6 filter do it this way, allow BGP, these five different types, you need incoming solicited, all those different types, put them somewhere and it would help reduce your support calls and also would help operators that need to do this for the first time.

SPEAKER: That's a good idea. We have a conflict guide. We might want to put it in there. This is mostly what we found out proper actively. Most people that experienced this, that filter for IPv6, don't notice themselves. I had to inform them, I have some neighbor discovery messages for you guys. You don't answer. What is wrong with you?

SPEAKER: But the thing is it's a hard problem. We notice because our parent sessions wouldn't come up in some cases with some peers in some directions.

SPEAKER: It's ICMP v6 that's what we get from the outside.

SPEAKER: That's what we get for designing it differently, right? In general, the one thing that's really the saving grace of v6 is it's so similar to v4 that we almost know how to operate it. When these strange little things come around then we stumble.

SPEAKER: That was basically my message.

SPEAKER: So the people who come we should have done this, should have fixed the other problem, that's too hard.

SPEAKER: We were one of the six of the DE?CIX customers who run into this problem. We first didn't see the problem because we were not off line we just took others off line. So for us it was comfortable. There is a fix from Juniper. You can get fixed images but they're not in the main versions of the operating systems yet. You have to ask your Juniper representatives. We were the third or fourth customer who experienced this and this was no idea so the others didn't get to Juniper, they just made a down grade, but they didn't complain about it. So Juniper was very slow to fix it because they had no pressure. So please if you run bugs like this, go to the window and bash him, bash him bash him until he fixes these issues.

SPEAKER: I can not do that. We are not a Juniper customer. The customers of these routers have to do that.

SPEAKER: I hope your customers go, the router operators have to go speak one of members knot back with a case number or something, that's all we get. But we are not able to foul these bugs unless we have Juniper ourselves. So we're always working with members.

SPEAKER: I was following up on this. So if you have this sort of issues which you observe in your fabric and it has something to do with vendor Juniper, please include this bug number or in this case like a PR number, because other people tuning into the session can use this number to put pressure on vendor to be sure that it gets addressed.

SPEAKER: Yes, this is also the problem that Internet exchange, we're always look being at the sites. So people can decide to make this a problem with Juniper. Basically the unencapsulated BGP messages that are coming and going and coming and going for almost ten years or so, for several times we pushed members ?? I raised this with Juniper, raised this with Juniper. In the end Juniper contact us and I say I think it happens when a peer falls away, then your router stuck to do this, sometimes sometimes not. Then they could reintroduce it. But in the end the customer vendor relationship is with our members, not with us. That's always difficult. So we can flag it as a problem, but we don't have any power to make it ?? we can only say to our customers or our members, can you please raise this with Juniper. If they say, we're just going to down grade, they do that.

SPEAKER: Freelance IPv6 guy. I don't want to play down the problem of Juniper here, but unless I'm totally mistaken, we have about 6.7 million of those multicast addresses and I suppose you have less customers connecting directly to you, so could this problem have been avoided by you if you had taken the computation of the multicast address into consideration when you devised your addressing plans?

SPEAKER: The Juniper problem, yes. But we have have just.

SPEAKER: It's not just because it's a Juniper problem, but if multiple machines listen to the same multicast?enable discovery, that's not as efficient as it could be. So, of course, it's always easy to come up afterwards, well, you could have done so and so. But that might have ?? might be considered a lesson that could be learned, try to assign new addresses in such a way that you won't have multiple machines on the same solicited multicast.

SPEAKER: There are multi machines on the same multicast group, these are the case for almost ten years or even longer. That has never been any issue, as far as I could see. What I showed you was the total number of neighbor discovery messages. If you play down per multicast group, there are roughly 400 multicast groups. Some are really quiet, some are really active, mostly depending on the querier, the one that wants to get the address. And it's more to do with the implementation of IPv6, how fast I'm going to query. And that seems to be more doing how much of a super skill CPU does it have to do that. You see that with the types that have fast power PC processors or general scale purpose users, there are really really past queries and there's nothing we can do about that.

SPEAKER: As I understand it, the RFCs leave a lot of slack for the implementers to choose whatever timing permit they consider reasonable. And it sort of makes sense, but, basically, my point is, as soon as I have multiple machines listening to the same multicast address, I'm putting an extra burden on them, especially if they have low?power CPU which is an unnecessary burden that could have been avoided in hindsight.

SPEAKER: I think that our scheme has other benefits and it's very easy to assign ?? it's very easy to see who you are peering, it's ?? but, yeah, to be honest when we came with this scheme, I wasn't really aware of these multicast groups.

SPEAKER: I suppose very few people were.

CHAIR: Thank you.

(Applause.)

CHAIR: With that we're at the end of this part of the session and there's supposed to be coffee outside. We're a bit early so they might still be setting it up. Back in this room for 4:00 for more IPv6. Thanks.