Transcript: Community Broadband Bits Episode 172

This is Episode 172 of the Community Broadband Bits Podcast. Chris interviews Tim Pozar, Director of Operations for San Francisco-based Fandor, on how open access networks improve competition. Listen to this episode here.

Tim: What we'd like to see is that competition that we have at the data center extended out to the street address so that you just don't have one, two, or maybe even three internet service providers you can pick from. You have 40 internet service providers you can pick from.

Lisa: Hello. This is the Community Broadband Bits Podcast from the Institute for Local Self-Reliance. I'm Lisa Gonzalez. Tim Pozar joins us for Episode 172. In addition to his role as Director of Operations for San Francisco-based Fandor, he's also a gentleman with decades of experience in the telecommunications industry and a strong advocate for network neutrality.

Chris and Tim get into the nuts and bolts of open access this week. In addition to details on how open access can be delivered, the conversation turns to various ways open access can improve the competitive environment for consumers. Learn more about Tim's company at Fandor.com and be sure to check out our many stories on open access at MuniNetworks.org.

Now, here are Chris and Tim Pozar discussing open access.

Chris: Welcome to another edition of the Community Broadband Bits Podcast. I'm Chris Mitchell. Today I'm speaking with Tim Pozar, the Director of Operations for Fandor and I think more importantly for today's conversation a concerned citizen, community broadband advocate and generally interesting guy to talk to about open access.

Welcome to the show.

Tim: Thank you. Appreciate it.

Chris: Tim, you and I, I met with you one time when I was out in that area and I've known of you from reputation, but can you tell our listeners how is it that you've come to know so much about the internet and why should people listen to you as an expert?

Tim: Well, let's see. I've been playing in this industry for quite a number of years. I was one of the early advocates of bulletin board systems. I was a developer with a bulletin board system called FidoNet way back when with a guy, Tom Jennings. Tom built up and developed FidoNet. During those days I co-wrote software that would hook FidoNet up to this thing called Usenet and UUCP, which was also eventually a gateway to the internet.

Then eventually Tom and I co-founded an early ISP back in 1990 or so, which we called Little Garden. I've been doing internet stuff at least since the '80's, mid-80's or so and then somewhat of a serial entrepreneur. After we sold Little Garden I helped with a company called Internet Archive, Brewster Kahle. I was one of the first Director of Operations there. Then I co-founded another company called Brightmail, which was an anti-spam company. Part of that, because we were very concerned about how spam was influencing the internet and how we can manage it for the future. We eventually sold that company. 

Since then I've been doing other work, things like I was a co-owner of a data center, an ISP in San Francisco, and also, as you point out, I'm with a company called Fandor, which does streaming videos, so we are definitely interested in things like, for instance, network neutrality, make sure that we can stream our videos as well as Netflix can. It doesn't take a whole pile of cash to be able to compete with them.

Chris: One of the things that I've long associate you with is this idea of open access networks. I'd like to ask you to get into this discussion by talking about the end. If you're successful, and I think if many of the people who are interested in this model are successful, what will the end result be?

Tim: It's really based on the problem and the fact that we have incumbents, which are just one or two players, that are serving, are the internet access providers for a community. In some cases you have none. In some cases you have one. In many cases you'll have two. What that does is that means that there's an artificial cartel or monopoly in delivering internet access to street addresses. I'd like to see more competition there, because that will help reduce the cost. We can see this time and time again looking at communities that have started their own broadband and how the incumbents react to that as well as hopefully this addresses, again, things like network neutrality, because if one player is behaving badly, you have a choice to going to another player.

Chris: Yes. However, I think your solution, you would not be as enthusiastic with Google Fiber, for instance, which in some ways accomplishes some of those goals of giving people another choice but doesn't really unlock what I think we might hope would be a true market of opportunities for entrepreneurs and others to provide ISP services; right?

Tim: Let me sort of roll back. I mentioned that I was a co-founder of an ISP back in 1990. The model that I give people so that they can understand where we're trying to go on this is back then we were paying our upstream provider around 5- to $6,000 for a MG and a half worth of bandwidth to the internet. 

Of course we weren't trying to push a lot of multimedia at that point. It was mainly very small email, things like that, so we could actually run an ISP off a T1's worth of bandwidth, but the cost in getting that bandwidth from the provider to us was around $500, so about 90% of the cost of connecting the internet was getting the broadband access.

Now, data centers, things have gone on. Twenty years have elapsed or so, and we have competition at the data center bringing that price down from 5- to $6,000 per MG and a half down to 50 cents, so it's gone down about 10,000 times, because at a data center you have 30, 40 different ISPs out there all trying to compete to sell you bandwidth. We don't have the competition at the last mile.

What we'd like to see is that competition that we have at the data center extended out to the street address so that you just don't have one, two or maybe even three internet service providers you can pick from. You have 40 internet service providers you can pick from. 

Just having Google or some other ISP building out fiber and going to the street address just means that instead of two people or two companies competing, you have three. You don't have a real true competition again that you have at a data center.

Chris: I would actually add that I would be afraid that over a certain period of time, five, ten years, you'd actually probably go back to two or maybe even one because we see this consolidation. I wouldn't just take it for granted that we would still have those three for a long period of time.

Tim: Yes. There is an aggregation of ISPs. We saw this early on in the 2000's or so or actually the late 1990's. A company called Varial went up and tried to buy every little mom and pop ISP out there, but you still see some ISPs out there, and they're still trying to compete. They've gotten smart enough to become Clux, so I think we still have some diversity. For instance, at a data center at this point, if I go to a data center like 200 Paul in San Francisco, I can still go to 20 or 30 different companies and get bandwidth at that location.

Chris: Let's talk about how to extend that out to the public. I think a lot of your examples I suspect will focus on San Francisco, but we'll be talking in terms of general concepts that would apply to any community, because I think a lot of communities would like to figure this out. One of the first questions is, who's going to pay for it. In your ideal world, who pays for extending this infrastructure?

Tim: One of the reasons that you're not going to see a third or fourth typically ISP or company that's going to come out there and try to compete with the telephone company and the cable company means that there's going to be quite a bit of infrastructure that has to be deployed to layer on top of the existing two, so in the case of ... I've heard numbers anywhere from 700 to $2,000 per street address for deployment. At one point Verizon Fios was quoting $710, and I'm hearing up to 2,000, so it depends on the density of the city and the type of architecture that you're going to have to do. In other words, can you underground it? Can you fly it as an aerial cable? Obviously aerial cables can go much cheaper.

What happens is there's a chunk of change that has to happen, and that has to get paid back somehow, so even if you have a third or fourth company come in there that tries to come in, typically you're going to have an investor throwing money at this company and expecting some sort of return within, say, five or ten years so that they know that what they invested they can get their money back on.

You have to keep in mind that if you're going to be with the third company that's going to be competing in this, that means you're diluting the number of customers that are going to be connecting to, so best case, maybe you'll get a third of the customers or maybe get a half, and the other two companies will get the rest, plus you're going to have to compete in price. That means your profit margin has to be or your end user price may have to go down and undercut the other two, so your profit margin may likely be even less than what their existing profit margin is.

All of this adds up to a return on investment that may not be five or ten years. It may be 20 or 30 years, so many investors are likely going to back away from that and say, "Eh, we're going to throw our money in something else that we can see the money back much more quicker." 

Who are the other companies or organizations that have deeper pockets? Typically that usually falls onto government. Government usually has deep enough pockets and they usually have enough insight to be able to do a 20- or 30-year, even 100-year project and understand how they're going to get their money back on that.

Many governments usually run businesses like sewer and water and maybe sometimes even run their own electricity and resell that. In that case what they'll do is they'll go out, raise bonds so that the money doesn't necessarily come out of the general fund, so it's not out of taxes and such, but they'll raise bonds. They'll go out and deploy the infrastructure and charge appropriately for it.

What I'm hoping to see in organizations or towns like City of San Francisco, and this is something we've been trying to work with them on is for them to create a fiber network. In other words, raise enough funds to possibly even cherry-pick to some degree, to experiment to see how this works for them and create a fiber network that goes back to a data center. Basically, they'll layer two or the transport provider to extend out the competition at the data center back out to the street address.

Chris: When you say, "cherry-pick," I think it's important, this is something that I think about a lot and it's important to note that when one cherry-picks, intentions matter. When the term "cherry-pick" could be used for an entity that might only build out to one area, the highest margin customers and then refuse to go further. However, a different model that I think you'd be more supportive of would be one in which one merely does it as the first stage, recognizing that that will generate the revenue needed to push deeper and deeper and out into the higher cost areas of the areas where there's less demand.

Tim: Exactly. What I'd like to see this done as is as sustainable as possible. Many times government will go out and create an infrastructure on good intentions but find that the infrastructure is just not sustainable and that they have to prop it up with additional funds from the general fund or taxes or some other way to be able to subsidize effectively this infrastructure. I'd like to see this infrastructure as a whole run as at least for break-even if not a somewhat profitable organization.

Again, we see this in cities where they usually have a business unit such as the water. In the case of the City of San Francisco, the airport is run as an independent business unit. The Port of San Francisco is run as an independent business unit. 

Admittedly, this is new territory, a new field, for governments. They know how to run things like water pipes and sewers and things like that, but extending this to putting glass in the ground and lighting them up, that's kind of a new thing. They did this on a small scale for looking at the government buildings, but trying to do this to every single street address is a little new.

Chris: We actually have the numbers on it in terms of number of cities that have done it. I just updated it, and we're tracking 90 communities across a somewhat smaller number of networks, probably on the order of 60, 65 networks that are doing municipal fiber to the home and 180 that I believe are doing fiber, just a few areas, not unlike San Francisco might be doing, where there's some limited availability for the private sector to lease it. Then I estimate in the high hundreds to low thousands communities that are doing fiber tanker institutions. It's something that has been growing rapidly, but it's still I think, and for a lot of elected officials even to think along this line.

Tim: Well, right. I should say, and those numbers are impressive, by the way. Those are much better than I remember about a year ago. 

Most of this is rather foreign. The people that are going to make the decisions and pull the triggers, this concept is extremely alien to them, and they're also very cautious too because the main argument that I hear about this is, "Well, can't another company ...?" We don't necessarily want to prevent or step in where competition can take over along lines of putting in a last mile, but they don't get the fact that competition hasn't happened. It hasn't happened for 20, 30 years and partially because the only reason that we have the current incumbents in there is because we have this artificial monopoly and a guaranteed profit margin for the telephone company and for the cable company through franchises and the fact that the telephone company was basically a government-sanctioned monopoly.

Chris: Right. I think a lot of people don't appreciate that there's very few examples one can point to of a large-scale telecommunication network being built without some form of monopoly. The Google Fiber Network might be among the closest in terms of going into large cities in such high scale, but even Verizon's Fios was basically built on the back of Ma Bell.

Tim: Right. It was certainly subsidized and such. Then they finally said, "Well, I don't think we can make enough money on it" and stopped it.

Chris: We're at a point now where we accept and we'd like to see local governments taking a more active role building the fiber out. One of the questions that I have is something that I've seen a number of people that have thought about this deeply have raised, which is, if you have fiber to every home and it connects back to a data center, you may not be in that great of a position, because almost every provider in that data center does not want to deal with individual users. They'd prefer to aggregate it. They don't want to deal with people that are dropping and adding and changing their arrangements and moving and stuff like that. How does it work practically in terms of making this work in the data center with all these potential residential customers coming in?

Tim: When Google was pitching a wireless network for the City of San Francisco, which was based on a sort of archaic and ... It's not archaic but older technology that we fought against, and this was the point in time where we discovered that there was quite a bit of abandon as well as fiber that the City of San Francisco had, and we said, "Well, gosh, instead of just doing this wireless deployment where you're running up against Shannon's Law on how much bandwidth you can actually push through, why not use fiber?"

Our original model was pretty much based on the OSI layers and the fact that we were thinking about an active Ethernet deployment to individual houses. In that case it was basically all software switch. It was a giant SDN in effect.

Chris: That's Software Defined Networking?

Tim: Right. Sorry. Thank you. 

In that case you would basically have an Ethernet drop at your house, and we would, or the City or whoever, would basically assign a VLAN or an MPLS point-to-point circuit or whatever else that could be done to create a virtual circuit between the street address and the ISP at the data center. 

In that case the ISP, and some ISPs are very happy to work with individual customers. They're designed to work with individual customers as opposed to wholesale, so they may have much better customer support and maybe even a much larger customer support so that they may even have to charge a little bit more, but you're happy to go with them because you know you're going to be in better hands.

Chris: For that, are you thinking about a company like Sonic, for instance, in the Bay Area or another kind of company?

Tim: Actually, Dane's a friend so I didn't want to necessarily sound like I'm plugging him, but Sonic is, and it's not just me. A number of other folks will point to Sonic as being a great example of a good customer support whereas when you call there you're not necessarily hitting somebody in Bangkok or Bangalore or something else like that, who has no idea who you are or even the network that they're trying to maintain. You get somebody who can fix the problem 90% of the time when you call them.

Chris: Right. I think there's a few others. I think Ting is going in that model. XMission in the Salt Lake City area certainly has that reputation. There's a few like that, but frankly, not a ton.

Tim: Right. I think Dane points it out, that because he has this great customer support and service, that makes him relatively unique in the field and therefore people are going to come to him. If we can have enough competition out there, maybe somebody else will say, "Hey, you know, I like Dane's model. I'm going to do the same thing."

Chris: You don't think there's a technical challenge? There might be a little of a challenge in terms of seeing more businesses that are going to recognize what a winning formula Dane Jasper has, but there's no real technical difficulty in terms of taking a whole bunch of residential fibers and plugging them into different ports in the data center to be able to pick and choose among different companies?

Tim: The original model was possibly a little naive. I think it still could work. Dane points out that the way that he is set up to manage, and I had a conversation, a couple conversations with him about this. The way that he set up to manage end users at this point is he really would like to be able to have control all the way to the prem. His model he'd like to see is where he's going to be putting an Adtran 5000 or something else like that that's going to actually do a PON ...

Chris: I think you're heading into dangerous territory with the jargon with some of our listeners, but basically you're talking about gear that would allow him to use the line basically, and there's different ways of doing that, and he has this preference for the gear, and he wouldn't want to be operating on network using gear that was unfamiliar or inferior to what he was used to.

Tim: Or out of his control. His model would be that he may release bandwidth to a neighborhood. He wouldn't put his own gear in and then light the glass that goes into the prem within the neighborhood.

Chris: Right. For instance, you wouldn't have a fiber from my house, a single fiber that runs all the way back to the central. You'd run to a neighborhood aggregation point and that's where Dane would have his equipment.

Tim: Right. The concern at that point is that, well, if you only have one piece of glass going in and that just hooks up to Dane, well, how do you have the competition again? Dane and I talked about this for quite a while. He said, "Well, what happens if we put three pieces of glass into a prem?" I said, "You only need one strand to light a PON."

Chris: PON is just Passive Optical Network. It just basically allows you to take a strand and split it among multiple parties.

Tim: Right. Right, multiple end users, the customers; right. 

Say, for instance, we had two extra strands and those go back to the neighborhood aggregation point. I could see having a scenario where you have Dane having equipment at that aggregation point, maybe another ISP. We'll just call it B having the same thing, so if a customer says they really want to be able to switch off of Dane onto ISP B, they would or contact ISP B and then basically move the equipment from or get other CPE equipment, Customer Prem Equipment, and plug it into this second strand that's coming into the prem.

Chris: That works because the cost of the fiber itself, doing three fibers to every home rather than one is basically the exact same cost?

Tim: Probably your customers and your listeners know 99% of the cost is basically digging it up and penetrating into the building and pulling it and doing all that other stuff. The fiber is relatively insignificant in cost. 

There's one other scenario to that. If you don't want to be spending a bunch of money to light up a neighborhood aggregation point, I was thinking that you could have something similar to, kind of like a hybrid of what we talked about, which was our original idea, which was an active Ethernet. What would happen if, say, for instance, that third strand was lit by, say, a company that all they do is light up the last mile or last 100 feet or whatever it takes to get from the aggregation point to the prem and what they do is they do the cross-connect back at the data center. That one piece of glass that goes into the prem gets somehow attached to again an MPLS circuit or a VLAN or whatever that goes back to where the bandwidth is being picked up at the data center.

A small ISP that doesn't want to go out and spend a chunk of change for neighborhood aggregation gear could use that other scenario as kind of another transport mechanism.

Chris: Oh, okay. You have multiple options basically throughout the network?

Tim: Right. You could either be like Dane, where he's going to put his own gear at the neighborhood aggregation point, or you could be leasing that neighborhood aggregation point bandwidth and gear from somebody else who is in service of reselling that to multiple ISPs.

Chris: Would you see this as something in terms of how the homeowner might pay for it? Would they be maybe making one infrastructure payment to the city and another payment to an ISP, or would it all go to the ISP and the ISP would remit back to the city? Do you have any thoughts on different arrangements for paying for it?

Tim: Since this infrastructure is hopefully wholly owned by the City of San Francisco or whatever municipality this goes into, and so at that point I'd rather probably not mix the money up with paying back another ISP or having that ISP try to subsidize the cost, because I'd be concerned about conditions placed on the fiber and the equipment.

Typically, a lot of communities are in the effort of trying to underground a facility. I think this is probably a great time to build up that infrastructure. The City of San Francisco at this point has a major effort to underground aerial facilities like in the Sunset and various other locations around San Francisco, and also they have a hundred-year-old sewer and water infrastructure that they know that they have to replace very quickly.

In the case of San Francisco this is a grand time to exploit that. That may not necessarily be the case for a facility or a neighborhood that was built out, say, in the 50's or in the 60's. In that case you may already have undergrounded utilities and putting this kind of thing in, you'd probably have to do another dig, and that could, that's going to be a little bit more costly and you're not going to be able to aggregate that with other projects.

Chris: One of the things that I often think about is that communities I think are looking at solutions, they're trying to figure out how to do this in the next three to four years. I think that's nice in the sense that it is a high priority and I'd love to have it in my house today, not in three or five years or certainly not ten years. Also there's a sense of if we just get started doing something, that will be better than this just waiting forever and then trying to figure out, can we do a very high-cost project? Probably not.

I've been encouraging cities to take incremental approaches, saying, "Our goal will be to connect everyone. However, we're going to start with Phase One, and Phase One is going to connect X, Y and Z, and Phase Two will go further." I'm curious what you think about that kind of approach, whether that's just basically setting us up for a potential failure if in three years they decide they're done with it and they're not going to build anymore.

Tim: I think that actually makes much more sense. There's a couple different scenarios. You make your commitment to light up every single street address, and that's going to be a large amount of money. In the case again for the City of San Francisco you're talking about a quarter billion if not a half a billion dollars to do that kind of deployment. That's going to be hard to finance. It's going to be hard to execute it, plus by the time you're done with it, you'll probably say, "You know, I really wish we did this this way when we started doing our deployment." 

I really like the idea of saying, and I used the word "cherry-picker" earlier, and maybe what I should say is "low-hanging fruit." In order to be able to get the expertise under the belt of the Department of Technology or whatever organization that's going to be doing that within the City, plus it will develop or make room or give a runway to getting whatever regulatory and organization issues out of the way. 

We talked about the fact that politicians are really nervous about putting political cred on line, and if you say you need about a half a billion dollars to do this project, they're going to get scared off. If you could come up with and say, "Listen, you know what? For a couple mill, we could probably put fiber within these several blocks, and we can see if this works. Then if it does, then we can go back," and likely a politician or the mayor, whoever else, is likely going to be able to get behind it better.

Chris: I like that. I've been advocating that, so I'm glad to hear that. Thank you so much for coming on the show and sharing your thoughts. You've been thinking about this for a long time in terms of how to solve our competition problem and built the infrastructure of the future.

Tim: Thank you, Chris.

Lisa: That was Tim Pozar, Director of Operations for Fandor.com talking with Chris about open access. 

Send us your ideas for the show. Email us at Podcast@MuniNetworks.org. You can follow us on Twitter. Our handle is @CommunityNet. If you use Facebook, you can find us by searching for Community Broadband Networks. 

Once again, we want to thank bkfm-b-side for their song, Raise Your Hands, licensed through Creative Comments, and thank you for listening.