Transcript: Community Broadband Bits Episode 150

Thanks to Jeff Hoel for providing the transcript for the episode 150 of the Community Broadband Bits podcastw tih Weston Hacker on security measures for community networks. Listen to this episode here.



Weston Hecker:  Some of the ISPs have the 9-1-1 system, or E-911 system.  They'll have paging systems for the actual police systems -- you know, banks, hospitals.  Those are who guys' customers are.  And that's -- those are the networks they want to go after.


Lisa Gonzalez:  Hello.  You are listening to the Community Broadband Bits Podcast, from the Institute for Local Self-Reliance.  This is Lisa Gonzalez.

This week's podcast is a departure from our usual presentations on policy or a profile of a specific municipal network.  In this episode, Chris and his guests get more technical, as they discuss security needs associated with ISPs.  Regardless of the size of the provider, security is critical.  In this interview, Chris speaks with Weston Hecker, Senior System Security Analyst and Pentester for KLJ.  You can learn more about the firm at  Weston helps us understand more about what is happening under the surface.  We feel it only fair to warn you that this conversation is pretty technical.  You will certainly walk away with a new appreciation for network security.  Now, here are Chris and Weston.


Chris Mitchell:  Welcome to another edition of the Community Broadband Bits Podcast.  I'm Chris Mitchell.  Today, I'm speaking with Weston Hecker, a Senior Systems Security Analyst and Pentester for KLJ.  Welcome to the show.


Weston Hecker:  Yeah.  Thanks for inviting me.


Chris:  You know, your name is not very lisp-friendly -- or, your title's not very lisp-friendly -- with the "Senior Systems Security Analyst."  [laughs]


Weston:  [laughs]  That's kind of a mouthful.  The particular path to email -- bottom also.  So --


Chris:  Yes.  What is pentesting.  You're a pentester.  What does that mean?


Weston:  Well, basically, "pen test" is short for penetrations tests.  Which is, basically, where somebody takes an outside look at a network, as if they were an attacker.  And they find the vulnerabilities and close them before the attackers actually do.  And it's done with hands-on weeding.  Like, we actually go through and actually try to penetrate the systems, as the hackers -- or, the bad "black-hat" hackers would.


Chris:  Yeah, I was actually going to ask you if you're required to wear a white hat when you're at work?


Weston:  [laughs]  Yeah, that would be the designation.  I do have a certified ethical hacking and CISSP.  So there are some certifications that allow people to be, you know, more professional when it comes to some of the security audits.  There's background checks that you have to go through, and things like that.  So, it does bring us up to a certain level of professionalism.  It's not just somebody with a little bit of Linux experience trying to hack into a network.


Chris:  Well, I think maybe some of our listeners are wondering what I'm up to with this episode.  So, why don't you tell us a little bit about KLJ, and who some of your clients are.


Weston:  We're an engineering company of ** people.  And we've been doing pentesting for the last two years.  I personally have been doing it for the last ten years professionally.  So -- and it's something that I was brought into the telecommunications department just because of the amount of security vulnerabilities and flaws, and the lack of expertise in some of the small areas that people are living.  So, in a smaller city, it's really hard to get a hold of security staff.  And that's something -- that's a gap that KLJ has been filling for two plus years now.  And, professionally, I've been doing it for about ten years.  And, yeah, as far as pentesting of ISPs, it's something that we've grown very, very accustomed to, especially with some of the more obscure gear -- Calix, Brocade -- that you won't find in the enterprise environments.


Chris:  One of the things that I think people might want to be reminded of is that, when it comes down to small ISPs, even the largest municipal network would be considered a small ISP.  And so, I guess a question might be, why should small ISPs be concerned with security?


Weston:  There's a lot of things that they push out to other companies, such as third-party billing, third-party patch management.  There's third-party pretty much everything, for as far as it goes, because it's too small to take care of it in-house with staff.  They usually implement, like, a Linux box that's running specific software -- specified software.  And what I've seen lately is, a lot of those systems have been up for 1300 days, 456 days, or -- you know, just some extreme range, where they're still Shellshockable, and all these vulnerabilities have come out.  And it's just something that's a full-time task, just to keep these systems patched and up-to-date.  And not a lot of people in smaller areas are able to have somebody with a lot of hand-on Linux or security experience -- like I was saying also.
So, it seems to only be, for the most part, an ISP problem.  Most of the rest of the world is comfortably dealing with Windows-based operating systems and something that they're a little bit more familiar -- a little bit more graphically based.  Some of the more obscure services that ISPs have to deal with.  They all run on Linux platforms, for the most part, and actual vendors don't keep them up-to-date as well as they should, without ridiculous service contracts in some cases.


Chris:  So, what is the actual threat that a small ISP might face?  I mean, I might think, well, I'm a small ISP in a Midwestern city.  No one's going to try and attack me here.  I mean, why would they?


Weston:  Yeah.  And that's a lot of the mentality that people in North Dakota -- where our main branch is based out of -- also thought, until the oil boom came along.  Or, you know, like, some of the ISPs have the 9-1-1 system, or E-911 system.  They'll have paging systems for the actual police systems -- you know, banks, hospitals.  Those are who guys' customers are.  And that's -- those are the networks they want to go after.  Not only to mention you guys' bandwidth.  It's been used in denial-of-service attacks.  A lot of people were doing reflection attacks.  'Cause a lot of the smaller telcos and ISPs actually had misconfigured DNS services.  So they were actually attacking their upstream and downstream people.  A lot of those gouts have been fixed.  And a lot of that stuff's been taken offline.  But it just goes to show how it's, you know, a full-time task staying on top of some of those things.


Chris:  I guess one of the things you might think about, then, as an ISP, is, am I paying for bandwidth that I don't have to?  Because someone has compromised a system of mine and they're using it to just, you know, flood an area?  You know, in some of these reflection attacks, it may not be very large, but I'm sure it adds up over time.


Weston:  Oh, yeah.  The largest ones have, you know, gotten in the 300 range, for as far as -- that's pretty much the entire backbone of the Internet.  Some of the reflection attacks, they were able to take down, you know, 300 gigabit throughput and stuff like that.  So, it's something that definitely was a problem, and it was a -- Yeah, with the nature of the reflection attacks, it -- with a little bit of configurations, or best practices, that were followed, they were able to mitigate a lot of this stuff.  And it's something that you will definitely see on your utilization, especially if you're being charged for some of that stuff.  So --

And one of the bigger things that people are stealing is PRI and SIP services.  And over one weekend, I've seen where, you know, people will get, like, $30,000 phone bills, from having hijacked SIP and PRI services.  So, basically, people are making calls to illegitimate numbers, or numbers that have been also hijacked.  And that's one of the scams that has been running out there, as far as when people use default codes and things like that that are built into systems.

And that's something that it's nice to be able to do hands-on, with pentesting, is, I have scripts that will check all that stuff.  And, you know, it isn't all about, you know, buying expensive software or anything like that.  A lot of this stuff can be manually checked by just running a couple scripts, before the bad guys run exactly the same kind of script.


Chris:  So -- as you're mentioning the scripts -- if I was an ISP, and I hired you, what are the sorts of things that you would do?  Can you get into that in a little bit more detail?


Weston:  Yeah.  These -- we have a lot of things for -- not only for optimizing the network in general.  There's best practice roll-outs.  And one of the most -- best starting points for actually getting secure, is actually rolling out an IT framework.  That is something that I recommend for every single company.  A lot of them have simple frameworks that are in place, but there's actual -- a couple standards that are going to be, most likely, enforced within the next few years -- the same way that power companies and banks, hospitals -- you know, HIPAA -- like, there's going to be some kind of compliance that's going to be pushed into place.  And that's going to be nice to have an actual security framework in place.  Not together rushing something into implementation.  Because that's going to be very, very expensive.  And somebody's -- you know, tasks and processes can take three years in a large company.  So, with some of the smaller companies, it's nice to be able to get something smaller in place.  And then, if you do have to bring a professional in to do that stuff, it keeps them more honest, ** the quote time.  Yeah, that would definitely be a good starting point for that kind of stuff.


Chris:  Can -- a lot of the things that you're doing -- can you do them remotely, or do you have to be on-site?  You know, I have to admit that whenever I think about pentesting, I think of, you know, Sneakers, with -- the movie where they're breaking into the bank, or they're doing something that involves a -- physical acts.


Weston:  Ah, we can do basically the same quality of pentesting and vulnerability assessments as if we were on-site.  By either sending a laptop.  Or we can take it from the outside and actually break in in exactly the same way that the bad people would.  Yeah, that's something that -- it's very, you know, inexpensive.  A lot of the smaller communities, especially North Dakota -- and the Midwest in general -- all the way down to the Mid-South and stuff like that.  I've see where people are still charging these people '90s prices.  And not every single ISP can afford a $12,000 audit, or a $24,000 audit.  That's why there's, like, a $4,500 option.  Or we even have to -- down, like $1,200 or $1,300 dollars for some of the vulnerability assessments.  And it's very, very reasonable.  And we've done -- for a couple of Fortune 500 companies, we've done apples-to-apples challenges on Web application scanning, and we've actually found more, because we rely less on our automated tools.  We're in there actually doing the hands-on stuff.  And that's going to be where it shines.


Chris:  And what happens -- you know, one of the things that you and I were talking previously -- and since you were just mentioning some of the products, I thought, it's really worth noting that you had said that for some of your clients that after a new vulnerability came out, you were willing to help them just understand if they were vulnerable or not.


Weston:  And that's something we're -- a lot of the bigger guys, they'll actually have to -- they'll have to get a whole new service contract, to be able to bring them back in, to be able to test for --  Like, for example, when Heartbleed and Shellshock came out, all the people we did pentests for, we went back and actually rescanned everything for them, or gave them a portal so that they could actually scan it, and -- even some of the customers they provided stuff to -- they were able to scan to see if their Linux systems, or some of the virtualization systems, or anything that was Shellshockable -- you know, they were able to test that.  Using simple scripts.  It wasn't something that they got charged more for.  It's just that little extra mile that, you know, gives it that down-to-earth, like, small-town feel that people are used to.  That's something that I think we do better than the big guys.  I'm not trying to do a sales pitch, but it's just something that -- there's so many companies right now that are just hiding behind their large names, and -- yeah -- they're scurrying to pick up as many security personnel they can, as far as network security.


Chris:  And so, what are some of the things that a small ISP can do?  You know, some of the things that I think you would just -- are prophylactic -- that don't necessarily involve spending a lot of money?


Weston:  Yeah, there's a couple of good starting points.  Like I was saying, they'd want to shoot contacts.  So, I could definitely send them some boilerplate frameworks that they could put in place with security frameworks.  I love helping people.  If they have any compliances, or questions about that stuff, especially when it comes to HIPAA, PCI, FDIC, stuff like that.  That's something I'm definitely well-versed on, and it doesn't hurt to interview people once in a while, to, you know, pick their brain about it.  And I would love talking security with people.  So, if people ever have any questions at all, that's always something I like to help people out with.  And, like I've been saying, some of that stuff is very, very daunting.  And it's, you know, several hundreds of hours of reading.  And it can be help- -- it's nice to be able to send it to somebody who can have a -- just straight forward question with.


Chris:  Yeah.  We find that, in this area as well, when we're just -- in any matter of telecommunications, I feel like, yeah, there's a world of difference between just being able to ask a couple of simple questions, and being pointed and being pointed in the right direction, versus, you know, trying to wade in and try to figure everything out on your own.


Weston:  Oh, yeah.  I would definitely agree with that.  So --


Chris:  So, one of the things that I think would impact all of our listeners is, a lot of times it seems like ISPs, they may show up at your house, they set up your service, and if you don't have your own router already, then they may give you one.  And some of those have some pretty significant vulnerabilities.  And I'm curious if you've seen any ISPs that do a really good job of making sure that their users' homes are secure, or if you have any best practices you might recommend around that?


Weston:  Um, I've seen, when it starts to degrade their service, or their actual customers, or their IP address ranges are being black-flagged, then I see a lot of people, especially the larger ones, reacting to it.  There was an actual framework that was installed on a lot of the customer premise equipment, and some of the actual on-site routers and things like that, that they were able to be accessed with their default passwords and stuff like that.  And made into huge, large botnets, that were used to attack and infect other computers.  And that's something that we'll come across in the pentests.  And be able to delegate -- And there's ways that you can check, just by --  Yeah, if you have manufacturers, as far as your on-site routers and stuff like that, that would be a concern.  There's definitely ways that people can -- they can be able to check, to see if those vendors are up-to-date, or have any vulnerabilities.  And be kept in the loops for that kind of stuff.  That's something that -- you know, there's simple email lists -- and we actually host one, for, like, ISP-specific area -- you know, if it was Shellshockable, the questions --  Or some of the vulnerabilities that come out.  If anyone ever has questions about ISP-specific stuff, that the vendors don't want to step up and tell people right away, that's something we stay on top of.


Chris:  I have to ask.  Something that's long been an interest of mine is, I think, making the world a better place.  In the sense of, a lot of spam, and a lot of these attacks, they come from places where they allow -- where ISPs allow forged TCP connections.  Right?  And I'm curious, now, is it standard practice now for ISPs to monitor their outbound traffic, to make sure that everything that's outbound has a logical TCP origination on it?


Weston:  Um, once again, once it starts to degrade their service, they will usually start to bring third parties in.  Or some of the larger companies will be able to catch it for themselves.  But as far as that goes, there's no real standardized procedure.  There is some stuff with getting certain classifications on data centers.  Like, if you're hosting emails, or if you're hosting HIPAA-compliant stuff.  That if people have data centers that they'll be kept into these -- kept in the loop and kept up-to-date with the best practices with that stuff.  But other than that, unless it degrades service, I haven't seen many ISPs actually step up with that one.  It's something where, if it's happening on a small scale, it's most likely ignored.  Until they start getting cease-and-desist letters.  Or if they start getting, you know, any financial or legal obligations from it, then it's something -- it's almost where they have to be forced to --  And there's procedures in place where people can actually stop those kind of things from happening.


Chris:  Is that an expensive solution?  To do that sort of checking?


Weston:  Well, not for the most part.  Like I was saying, a lot of the newer firewalls and things like that, you'll be able to actually do them with passive services.  'Cause when people actually buy, like, some of the -- you know, they'll throw on a few words, like "next-generation," ...


Chris:  [laughs]


Weston:  ... and things like that, into the firewall.  Sometimes they're not just marketing terms.  They are actually -- you know, they have some smarts behind them, where they're able to connect, and be able to deeply inspect the traffic without actually cutting down on the bandwidth that much.  And that's -- there -- if you want to buy a device, it's always good to do research on them.  And, you know, there's ones that you're familiar with, that sometimes you have to reach outside the box, and give them a chance -- that you do give a chance a couple of years ago, because they were in infancy.  I know, IDS and IPS Systems, when I originally got back into them, it was something that made me cringe, because I just remember all these false positives, where it would tell me that, you know, this was getting attacked, or that was getting attacked.  Or this IP address connected from this point.  You know, so many false positives.  And they worked a lot of that stuff out.  So if you guys had a bad experience or something in the past, it's always a good chance to give it a second try.


Chris:  All right.  Well, thank you so much for coming on the show, and giving us a better sense of some of these security questions.


Weston:  Right.  Perfect.


Lisa:  Send us your ideas for the show.  E-mail us at .  Remember to like us on Facebook and follow us on Twitter.  We are @communitynets .  Thanks again to Persson for the song, "Blues walk," licensed through Creative Commons.  And thank you for listening.  Have a great day.