Fast, affordable Internet access for all.
Perspectives From An Established Muni Network - Episode 491 of the Community Broadband Bits Podcast
This week on the podcast, Christopher is joined by PJ Armstrong, Interim General Manager at Monmouth Independence Networks (MINET) operating in Oregon’s Willamette Valley. During the conversation, the two discuss how MINET came into existence over fifteen years ago, unique perspectives from an older municipal network, progress on MINET’s recent investor-backed expansion into Dallas, Oregon, and how the pandemic has affected the operations and marketing of municipal networks. Christopher and PJ also geek out about MINET’s custom-built operational support system (OSS) and the technology that powers their networks.
This show is 24 minutes long and can be played on this page or via Apple Podcasts or the tool of your choice using this feed.
We want your feedback and suggestions for the show-please e-mail us or leave a comment below.
Listen to other episodes here or view all episodes in our index. See other podcasts from the Institute for Local Self-Reliance here.
Thanks to Arne Huseby for the music. The song is Warm Duck Shuffle and is licensed under a Creative Commons Attribution (3.0) license.
PJ Armstrong: I proposed a couple of different things that we expand into this small town of Rickreall next door, and then a very rural area south of the city of Dallas. And it's not a lot in terms of dollars for construction, but it's pretty significant for my head.
Christopher Mitchell: Welcome to another episode of the Community Broadband Bits podcast. I'm Christopher Mitchell at the Institute for Local Self-Reliance in St. Paul, Minnesota. I almost forgot where I was there for a second. I'm not gonna lie. . Today we're speaking with a new person from a network we've talked to multiple times. Welcome to the show, PJ Armstrong the general manager. Interim at my
PJ Armstrong: Thanks, Chris.
Christopher Mitchell: It's great to have you. Don Patton has been running this show out there for a while and I knew him when he was here in Minnesota and seemed like he did good work, but if he didn't, you could expose him now because he can't take any revenge on you now.
PJ Armstrong: , I won't do that. And I'll be honest, he did awesome work. I mean, he really came in, I think it was 2013 and turned mine around to what it is today and really put us in a position to be able to do some of the things that we're probably gonna talk about today.
Christopher Mitchell: Sure, nice. I think it's a good place to start with some of that background. I mean, I find interesting is a rare case of two municipal networks working together under one name. So tell us a little bit about how that started.
PJ Armstrong: The simple story is the cities a month and independence went out to the incumbents and talk to them about upgrading the infrastructure because the quality of service is not good. That same old story and we're told Go pound sand, and it didn't happen. And so they put together an intergovernmental agreement and came together and formed mine, which is a fiber to the home company that serves the city limits of Monmouth and independence in the Willamette Valley here in Oregon. And we've expanded into Dallas. That's kind of the high level.
Christopher Mitchell: And the Willamette Valley is known for being just insanely fertile. A lot of farms,
PJ Armstrong: Lot of farms, lot of farmland. And we'll probably talk about that with some of the expansion discussion because it's going into those rural areas. But yeah, lot of farmland where we're at tons of wineries.
Christopher Mitchell: Now, when you say the city limits, do you have customers aside from Dallas, we'll come back to that in a second, but did you have customers where you expanded outside of city limits? And I mean the city limits, do you have farms within the city limits? What's the situation like there?
PJ Armstrong: No, I mean it's surrounded by farmland but not within the city limits. And that was the original construction was the city limits a month and independence. So we really haven't had an ability to go outside of that
Christopher Mitchell: Yet. And you're old school, I mean this is not one of the first waves , right? , but you're not one of the first wave, It's sort of second wave municipal fiber, I think. I wanna say like 2006, 2007 or so. Is that right?
PJ Armstrong: Yeah, thousand six is when we turned up customer number
Christopher Mitchell: One. And so now you're looking at all these young ones that are being born in Colorado and these whipper snappers just coming along
PJ Armstrong: . Yeah, that's right. And the type of infrastructure that we built here, and this is a little technical and geeky probably, but the type of infrastructure we built here at Independence is old school FOS enclosures, and two count drop and manual splice. And then when we get into the Dallas market and pre connector drops and installations that are a lot more simple than what we started out with here at mine.
Christopher Mitchell: So do your texts then, and I mean we don't have to worry about being a little too technical. There's some folks who listen to this show just hoping that we get into the technical details. So
PJ Armstrong: That's where I'm probably more comfortable. . I love
Christopher Mitchell: It. Do you still have texts then that have to have the older technology skills of maintaining those networks then for the original ones that you built?
PJ Armstrong: Well, yeah, I mean if you're talking about just with SPLA mint splicing, I mean we have good splicers, but yeah, they absolutely do. They're in the FOS enclosures and they're buckling down the strength members and breaking out two count drop and all of that stuff. Managing enclosures that have been there for 16 years and some people that might not have taken care of them as well as they should have, and they're a rats nest and all that kind of good stuff. Yeah, Yeah.
Christopher Mitchell: So that's where I feel like when people are thinking, Oh, we'll build a fiber optic network, you don't think about that and how the technology changes and you have to deal with the Furman
PJ Armstrong: . Yeah, yeah, for sure. Yeah, , every year we get a couple squirrel shoes. Yeah.
Christopher Mitchell: Oh yeah. Well I think that's somewhat inevitable probably. I remember when the plan for Dallas came along. I also remember that just before that it seemed like there was an effort to try to recruit people to come do more farming in the area because of advanced agriculture type opportunities and things like that. So I don't know if did anything really come of that you're familiar with?
PJ Armstrong: A little bit from on the City of Independence side and not direct related to mine it, but I can tell you I'm not gonna have a lot of detail for you today. But I can tell you I just actually talked to the economic development gentleman with the City of Independence relative to expansion work that we're gonna be doing outside of the city limits. And we've very briefly touched on agriculture and how this is going to this expansion work will bring our services out to some of that farmland that surrounds
Christopher Mitchell: Us. Excellent. So the kind of big expansion that you did several years ago I think is actually once again a little bit ahead of the curve because just this week we saw in Montana there's another large bond in which Utopia working with some local folks in Montana have borrowed a significant amount of money at a really reasonable interest rate without anyone backing it. And I feel like that's kind of similar to the Dallas approach where you're working with a small community and you didn't wanna take the risk by issuing debt that would be on in your balance sheet. And so there's kind of a vehicle, but really it was the strength of your operational history that gave investors the faith to support it. They paid for that project and now you're finished building it. So how's Dallas going?
PJ Armstrong: And Don Patton was obviously instrumental in bringing that to the table. And what we do really well here, and I've probably mentioned this, again, we talk about the team and whatnot, but top notch level of service, it's fiber to the home that kind of takes care of itself. But we are really great in our support. So trouble calls that come in on a daily basis, we resolve those almost 99, 90 9% of the time they're resolved in less than four hours. So anyway, about the Dallas market? Yeah, so were outside investors that came in. My understanding is typically they were building schools and prisons and that type of thing invested in the fiber network. And I think it went really well. It was originally slated to be a one year construction timeline and it ended up being a year and a half and there were a lot of contractors involved. So there was a bit of a hering cat kind of thing. It was a design build project. So sometimes it was build and design, it was a little bit bumpy, but I think it went really well overall.
Christopher Mitchell: And you were just mentioning the resolution that you would resolve so many of those problems immediately. And I think that is probably why you have such a high take rate in your areas, which is what gave the investors I think confidence that you would be a good bet to build out this. And how big is Dallas? How many people are in there?
PJ Armstrong: There's roughly 5,500 passings available to us at this point in time.
Christopher Mitchell: So I mean that's probably a good 12, 15,000 people or so. I'm guessing that's a sizeable community. Yeah.
PJ Armstrong: Yeah, it is.
Christopher Mitchell: One of the things I wanted to ask now that we're two years into the pandemic is how has that reacted? I mean, thinking more about specifically M method independence, I think they've had this fiber available for a long time. You already had a ton of people signed up. Did you see any significant changes with the pandemic?
PJ Armstrong: That's kind of the thing about the minute market. We were so well established that there were some periods of time that we shut down installs. Oregon is pretty militants in their mandates, so we had two or three months where we shut down installs. But the customer base of mine is pretty well established. It did affect the Willamette Valley fiber deployment because in Dallas, because we got that one and a half year build had gotten completed and it was a month after that we had this huge bubble of interest and we were about to go out and install a lot of people. And then we got this shutdown that occurred and it kind of took a lot of momentum away from that. And we've been working on that since. But yeah, in my net market, we're pretty well established, so it didn't have as much of an impact.
Christopher Mitchell: And to the extent that you saw other human beings in the real world did your and your crews, your employees, they were kind of treated like heroes in terms of I can imagine a lot of other folks have cable DSL around not nearly as good a service,
PJ Armstrong: Correct? Yeah. What we saw then they were, You're right. That's a really good point. What we saw a lot of, which I'm sure a lot of providers did, was first of all, our overall data usage during peak time didn't change a whole lot. Our 24-hour average that's what changed. So a lot of people during home during the day, and we saw that change quite a bit. We initially, when this whole thing started happening, the pandemic, we got a lot of those calls that were, well, I'm connecting the word for my VPN and it connects, but then it drops out. Or I'm trying to do a video call and I'm not getting the best kind of service. And a lot of times what that equates to is my internet sucks and that's not the case. So we did a lot in the beginning, we did a lot of troubleshooting with our customers and helping them understand that it's really probably your wireless network that you're having the trouble with. And
Christopher Mitchell: I thought you might be going there.
PJ Armstrong: . Yeah, wireless is not as robust as you thought it at it was. So that's kind of a lot of what we experienced in the beginning.
Christopher Mitchell: Do you have a managed wireless solution now to help?
PJ Armstrong: We've had one for quite a while. We started with Smart RG and during the pandemic we started transitioning and we are still currently doing that to Plume, which we're a year into it. And I have to say, I mean it's a really great solution. I'm really happy with it. So far the only trouble, and that's what we did in those types of cases was offer that to the customer base and it's really been taken very well. But of course we're dealing with chain of supply issues. That's really the rug right now.
Christopher Mitchell: The different equipment I know can do different things. Do you have the ability to just do a speed test on the customer's side without touching their home network? You can. Okay.
PJ Armstrong: That's one of the great things about the Plum environment and the acs, the management system with Plum and there's a lot of overly detailed insight that you can get into if you really wanted. But yeah, a lot of troubleshooting tools that are there for our customer service reps that are really helping us avoid truck roles.
Christopher Mitchell: Excellent. Bouncing around a little bit in Dallas, how did you go about marketing it? I mean, you're here, you have probably a hundred years of history in my net with the utilities. How did you go about marketing yourself in Dallas?
PJ Armstrong: Yeah, good question. I'm gonna do the best I can here cause I'm not the sales marketing guy, but I have sense of what he did. And initially we tried to start out with a system where we market it by sector as we go into different sectors. That was as it was broken out and we have champions for each sector. And that was kind of starting to happen. And then the pandemic happened and that went away and for quite a long time, and in fact it's still in place, the door to door got shut down in Dallas. So we can't go door to door and we can't market to people. And that really has thrown a wrench in the gears, but he does the obvious stuff. We do the obvious stuff every time we're going out to get ready for a conduit job for an installation that's gonna happen in the future. We tag the doors next door We don't door knock, but we tag them, let them know we're coming and that we're gonna be there doing work. The sales marketing guy has ads on the shopping carts in the local shopping center, that kind of stuff. He's very involved in the community there, the downtown association, Business Association. We do a lot of support in the community. And I think the biggest thing I would say is I have been telling myself word of mouth after pandemic is going to start rolling. And I hope that by the summer of 2022, it's kind of at a point where we're hearing all kinds of stuff in social media about how good we're doing and what a great service that it is in our support. And that's actually already happening. It's early. So I'm pretty excited about that. And I think word of mouth is the best marketing tool that we have right now. It's going very well.
Christopher Mitchell: Have you been hitting your marks then even with that slowdown and whatnot in terms of the business plan?
PJ Armstrong: Yeah, we are. So that's another pretty great thing that since the pandemic, it took us probably three months to ramp back up two, three months. But we've had a very consistent level of install on a month to month to month basis.
Christopher Mitchell: Excellent. So let's talk about the rural expansion then. You just got some good news which is that you don't have to pay a hundred percent of the cost out of your own bank
PJ Armstrong: outside from mine outside of the Dallas expansion, which is kind of a lightning in a bottle sort of situation. Minnet has never had the opportunity to expand outside of our city limits. So the history of it is the county put a survey out to residents in the county to ask them about their broadband, the level of broadband service that they have and their experience. And they identified some areas of need little pockets via gis. And they literally came up with some ellipses. Here's the areas that we want to be able to bring service to. And they have some ARPA money to do that. So they called in all the service providers and explained that to them. And I just started taking a look at those areas and going, Well, how could my net fit into that? And they're very open to any proposal. So I proposed a couple of different things that we expand into this small town of Rickreall next door and then a very rural area south of the city of Dallas. And it's not a lot in terms of dollars for construction, it's there's two projects and it's one point about 3 million, but it's pretty significant for my.
Christopher Mitchell: And do you think that there's going to be more of that obviously the state of Oregon, it's gonna have to distribute a lot of money. I'm presuming that there are more areas close to you that also will qualify for that infrastructure dollars because they don't have reasonable service right now.
PJ Armstrong: Yeah, absolutely. So there's a couple, I actually made another proposal that was equally as big north of Dallas. It's just that the funds that they had weren't there at the time. One, a small chunk of this is a feasibility study for a city that is nearby to this. We're surrounded by this rural area. And in my mind, and actually one of the things that I told our staff in relaying this information to them about expansion that we're gonna be doing is there's all this federal money coming down for exactly this type of thing. And I'm viewing this as a stepping stone for minded. And we're gonna go out there and we're gonna be successful in these rural areas and we're gonna prove ourselves and it's gonna be an example of what we can do. And there's a lot of potential in the future.
Christopher Mitchell: So I have a quick question that no, let me put that differently. I have a last question, which may not be quick which I didn't prepare you for, but I just came to me since we have a few extra minutes. So as operations director, I'm curious are there things that advice that you could give to other communities who are building networks in terms of things that, you know, came in and saw processes or approaches that weren't working well and you figured out how to improve them?
PJ Armstrong: Yeah, I can think of a couple. And one is that I saw this organization at a time where it was very siloed. So like I said, we have 22 or 23, we're pretty small. 22 or 23 employees right now. One of those two, I can't remember. For
Christopher Mitchell: The telecom division
PJ Armstrong: That's entirely
Christopher Mitchell: For the entire utility
PJ Armstrong: Entire, Yes, our employee base, our entire team is 22 or 23 people. We just hired a technician, I can't remember what it is, but there was a time where we were siloed and at the bottom of the job description, other duties as a sign didn't exactly apply to everybody and that can't be the case. And so that's one of them. Like I said, we all wear a lot of hats and we're pretty nimble, and so we're not all stuck to exactly, this is your job and that's the only job you do. We're all here to get a job done. That's a big part of it. I will say that I don't know how much advice this is, but when I stuck took on the operations director role in 2015 and a couple years prior to that, actually I saw a gap in the operations, which was the oss, the operational support system. And I programmed, we built our own from the ground up and to the point that it is today. And we have our own internally developed programmed OSS for both markets that serves a whole lot of different functions. And I think it really allows us to communicate across the board without having to walk up and see somebody or track things on paper or on spreadsheets, that kind of a thing. So that was a huge help and we could talk real geeky about that if you want.
Christopher Mitchell: Yeah, I think we should talk about that for a few seconds cuz I've long heard people talk about how important it is to have a good system. Is that the sort of thing that allows one person to take a phone call and describe a problem and whatnot and then the technician gets those notes and whatnot, and is that the sort of thing that software does?
PJ Armstrong: And it came from, and I'm drop a couple names here, but originally back in the day, at the beginning of mine, we had an antiquated system built quest and we used eti, and ETI was good. It was a provisioning system, but there was just a lot of that communication stuff that wasn't there. And so I wrote a system and this may be of interest to, like I said, really geeky types, but based on Linux, Apache, MySQL, php, we have a web gooey front end and the lamp stack. Yeah, lamp stack. Exactly. And it's exactly what you said. And we're an ADTRAN backend, and that was when we converted from B PON to G P O N. We previously were Alcatel. That was a fairly closed system. What I loved about ADTRAN when we converted to it is that it's very open. You can get any of the data in the system you want. It's a MySQL backend. It's just a normal database backend. They have an XML interface, so it's really easy to get information out of the system. And that's what that web UI does, that lamp stack, our customer service rep, a person calls up and they enter their address in the gooey and it queries the PAW chassis and it says, All right, well I know this is the ont, what are the services that are provisioned? What's the current alarms on the nt? they enter their trouble calls in there, it gets emailed out to a technician. The technician jumps in there and they close their trouble calls on it. We use it for almost a hundred percent of our tracking.
Christopher Mitchell: It sort of staggers my imagination that on a team, I mean, you're at 22 or 23 now, probably smaller than . You had time to just do this in the evenings. I mean, where did you find time to design your own system?
Christopher Mitchell: Began. So you're able to just iterate on it over time then?
PJ Armstrong: Yeah, exactly. And that's exactly how it's been. And so if you looked at the minute version of that well, I'll start with the Dallas one, which Dallas started a couple years ago. If you look at the Dallas one, the programming on that is it's very clean. If you went back the minute one, well, it started in 2006, and it has been iterations over iterations. And if you look at the programming, it's a little bit messy, but it works.
Christopher Mitchell: excellent. Often talk with people who are a marketing person or a general manager, and we don't usually get a chance to get into the details of how these things actually run. Would you recommend that a place start building their own? Or is there enough solutions out there that are improved now that would just make sense to jump on one of the commercial ones?
PJ Armstrong: I would say yes. The answer to that question is yes, the short answer, because if you can do it with what I said, we have 22 or 23 people. If you could find somebody that can kind of take on that little bit of programming, it's not, I'm no programming genius, but if you could find somebody to support that, yeah, I think any all day long, I would take a customized solution that you can make fit to your own operation. I would do it that way all day long.
Christopher Mitchell: Okay. Excellent. Well, this has been wonderful. Really appreciate your time and yeah,
PJ Armstrong: Thank you for having me.
Christopher Mitchell: Yeah, the work out there is just remarkable and I'm looking forward to hearing more about grants coming your way so you can keep building out the Willamette Valley.
PJ Armstrong: Yeah, absolutely. That is our goal.
Christopher Mitchell: Cool. Thank you.
PJ Armstrong: All right, thanks a lot.
Ry Marcattilio-McCracken: We have transcripts for this and other podcasts firstname.lastname@example.org slash broadband bits. Email email@example.com with your ideas for the show. Follow Chris on Twitter, his handles at communitynets follow muni networks.org. Stories on Twitter that handles at muni networks. Subscribe to this and other podcasts from I lsr, including Building Local Power Local Energy Rules, and the Composting for Community Podcast. You can access them anywhere you get your podcasts. You can catch the latest important research from all of our initiatives if you subscribe to our monthly firstname.lastname@example.org. While you're there, please take a moment to donate your support in any amount. Keeps us going. Thank you to Arnie Hughes. Be for the song, Warm Duck Shuffle, licensed through Creative Commons. This was the Community Broadband Bits podcast. Thanks for listening.