What a week….
I think it’s time to pass along a long story of what’s gone on over the last week or so here and some of the reasons there hasn’t been anything posted. Generally, I would say that work has been busy, but something happened last week that went a bit beyond the day to day and there might be some items worth considering. The short story is my internet access was suspended and I’ve been only connected to the internet for 30 minutes or so at a time to retrieve mail and spent dozens of hours reviewing system logs…. but the long story is needed to sort out what has happened. I’m not going to break this up into multiple posts, but I may pull out some details for seperate posts at some point.
About 12 or so Thursday UPS arrived with a wireless router that I was to configure and prepare as a wide open access point for a client. They specifically wanted open access and I had sketched out a network structure that should protect their internal LAN from arp poisoning/mitm attacks and around 12:40 (according to my arpwatch log) I plugged it into the network wide open. I probably had everything configured by about 1PM. Next up was the configuration of a wired to wireless bridge. The machine in particular that this was to connect to is an unknown quantity. I assume it will have ethernet, probably not wireless, so I dig into configuring the bridge to talk to the wireless router.
The lean manual doesn’t give any clues to it’s default IP address so online I go. I find the default IP and login information (no username just password) and give it a try, defaults are not accepted. It’s at the right IP, but doesn’t like my login. So, back online. Still no joy, try again and do another google search….. the page could not be found. Huh? OK – take a look – cable modem looks ok all lights are good. So now I get sidetracked into getting back online (the wireless bridge took the same item in both the username and password fields which contradicted the manual I was able to find on the cd and online which said JUST password, but that’s another story…)
I unplugged the cable modem for the requisite few minutes and went and checked in on the firewall. Uptime 361 days everything else looks normal, it just looks as though it suddenly lost the internet connection. I check in with my Dad who said he has had some problems in the last day with his connection (we’re in the same subnet.) So, I spent a good part of the afternoon with the cable modem and firewall doing the dance of rebooting, powering down, trying to renew/release dhcp address, etc. I had an appointment to get to and so left things as they were and on coming back in the evening pulled out my old netgear router/firewall and tested with that. I tested with a linksys router and I tested with a direct connection to my laptop. After eliminating all the possibilities but the cable connection I called my ISP.
Now, given that I run linux on my systems I dread calling my ISP tech desk. Because “we only support Windows / Mac systems…” So, I took a deep breath and booted my laptop into Windows for the first time in ~10 months or so. Fortunately it’s more stable because ~10 months ago I was replacing the hard drive and had to reinstall it to get things straightened out. Anyway, I’m booted up into windows, I’ve got the cable modem in reach and I’ve got the laptop hooked up to the config page. I’ve done all the troubleshooting that I feel like I can do at my end so I’m loaded to bear to explain the situation. I get into the phone tree and am given the choice between Windows support and Mac support (how about connection support?) I choose Windows and within the next few minutes believe my eyes are going to bleed…. they’ve got an automated “let’s try a few things while you’re waiting” troubleshooting tree.
The troubleshooting tree took me through unplugging/replugging the cable modem (for the nth time now), renewing and releasing the ip address, opening a web browser, starting over…. finally the brilliant troubleshooting tree opined… “I think I’ve narrowed down where the problem could be” and left me to the sales pitch interrupted by occasional music. A voice cheerfully told me that the hold time was just 15 minutes. This was about 7:30PM, at 9:30 PM my estimated hold time was just 15 minutes and I’d just about memorized their schpeil about their new phone service and all the other great things they’re working on……
About this time my wife cut on my cell phone and dialed in, she started through the tech support and then hung up, called back and selected that she was interested in buying something. Surprise… she dropped right to an operator (can you imagine?) This person gave her a ticket number and a phone number to call. It turns out the number was wrong (some sort of “live chat” line…) So, she tried again while I continued grooving to the sales pitch on the tech support hold line…. This time I have no idea what department she got, but he was not happy to have someone calling him. He grumbled a bit, (we traded phones)…. But he put in a call to the tech support line (same one that I had only a 15 minute hold time on….) He got through to someone magically in about 5 minutes. While we were holding he told me the account was listed as suspended for abuse.
The person that came on the line gave me a ticket number and another number to call. (Just off by two digits from the prior number – I think they must have just mispoken and gave an 800 when it should have been 866. Anyway, that is their Security Response number. At this point I’m not sure what to expect…. the person that picks up starts in asking about the network setup and if I have a wireless router. Well, I explain I have an old access point that I leave off most of the time. He asks if it’s encrypted (it is.) He asks if I keep my systems up to date (I do.) Then he asks if I’ve got antispyware software, I start to explain I use linux on all the systems but he cuts me off before I get to finish the sentence describing the IDS systems I use, the paranoid security settings on the outside facing machine, the rootkit detection. He cuts me off before I get to that saying “that’s no guarantee”. I tell him “I know but…” and start to describe (again) the security settings when I’m cut off again about having problems with some wireless routers like mine (I remind him it’s an access point.) Which he says I should update the firmware for. (In fact that’s already been done.)
So I ask what happened…. “At 13:16 I think GMT I’m seeing a lot of nslookups whois searches and a net screen attack against a whole IP block, let’s see 12:28PM” I was writing quickly to take down the ip block that the claimed scan was against and at some point I interrupted to say “isn’t whois lookup just a utility to find information about a domain, if it’s registered, who it’s registered by, etc.” (Legitimate tool…) “Oh yeah, but they can use it for more stuff. This that I’m seeing here is against a whole netblock.” At which point he tells me that I’ve got something nasty on one of my machines and he’s going to cut on my connection so I can download their security package (which he hasn’t seemed to connect is incompatible with linux…)
So at this point I’m a bit overwhelmed at what I’m needing to do. I’ve got the laptop (alone) hooked up to the internet and the first thing I do is get online and (go to a webpage) to do a whois lookup on the netblock that was supposedly “net screen attacked”. I also read through the terms of use for the ISP (again). Among other things “network probing” is not allowed (from my conversation it sounds like the person I talked with viewed nslookups as network probing…. guess I’ll have to start learning the ip addresses for all the sites I want to visit?) Anyway, the first thing I do is audit all the routine scripts. I’ve got one in particular that monitored a nameserver for another provider because they suspected the providers nameserver problems were part of an issue they had. On looking, it was merely a single ping on a routine basis, but I disabled it anyway.
13:16 GMT would be 8:16AM localtime. I wasn’t even logged in at that point, so it should be obvious it wasn’t something I intentionally did. (Just a note – I do use port scanning on my INTERNAL network and LAN’s that I administer – port scanning against public networks is NOT a good idea – unfortunately I’m in the prickly situation of having a good amount of knowledge about network security and probing which could make it look even fishier… But the bottom line is that I know better than doing a broad port scan out on a public network. (Convincing someone else of this might be another matter, because for some possession of that knowledge and tools to do so is indictment enough.)) So, I start through the logs on all the systems. (and rootkit detection) (and for what it’s worth clamantivirus scanning….) Everything turns up clean. It took hours that evening to go through. I found repeated attempts to gain access to the ssh server on my server, but none appeared successful. (technically someone could have covered their tracks, but there were no other evidences of fishy activity and the rootkit scans have come up empty.)
So, the firewall get’s inspected too (and before bringing it back onto the internet I close off all but one service port – no ssh temporarily just in case.) So, I worked until maybe 3AM Thursday on that front and picked up again Friday morning. I also start a log of when I’m connected and when I’m disconnected, because at this point I’m wondering if someone could have been doing IP spoofing of some sort and made it look as though my IP was the source. I don’t know what measures my ISP has in place to detect/filter spoofed packets, but given the main topic of discussion on this site is computer security I can see how I might be targetted. Up until now my home IP has probably been easily found by looking at my domain registrations, doing reverse dns on those IP’s and finding the one that looks like a residential ISP account. So, if there’s no ingress filtering on the ISP’s routers, then someone outside the local subnet could specifically target me. We’ll call this the “paranoid” answer to the puzzle.
By afternoon Friday I was still turning up nothing in auditing the systems. All looked (and even after 7 days now looks) clean. I’d been running tcpdump at the gateway each time I connected to log the packet data and see if there was anything suspicious, nothing there, so I called my ISP again. I gave them the ticket number and explained I had called last night and was looking for more information on the incident. The person I spoke with this time struck me as a bit more reasoned in his approach. He didn’t talk about whois lookups or nslookups. He said, at 13:16 there was a “net screen” scan against port 443 across an entire subnet. I then asked about the time – he couldn’t be sure if it was GMT or local time. Hmmm. I then asked about the 12:28 PM time that I had been told was another incident. That turns out was the time stamp on the email from engineering to security. Probably their local time, so maybe CST? He wasn’t clear on that.
So, there might have been an open access point on the network when the incident happened. I got the wireless router to configure at 12. I have a log entry of it being on the network at 12:40PM, 1:16PM (13:16 GMT or local they’re not sure) appears as though it may be the time of an aggressive network scan from my IP address. 1:28 EST (could be 12:28 local or central?) was the time that the engineers emailed security and my account was off by 1:50PM EST. It’s a timeline that makes sense to me and right now this is my most likely suspect as to what caused the mess I”ve gone through the last week.
Open access points are forbidden in many ISP’s terms of use (that was news to me although I always encourage people to use encryption…) I’ve observed wireless activity (since then) in the early afternoon 1:30’ish from at least one wide open access point that would be well within range of where this AP was sitting. The road is also well within the coverage area of where this router was. So, this is yet another argument to never leave an access point wide open, EVEN if you’ve comfortably separated wireless traffic from the rest of the network. What’s truly disturbing to me is that this would have happened within 30-40 minutes of plugging the router in.
So, I’m still doing packet logging at the gateway to be more certain, I’m still logging the times I’m connected to the internet. I’ve done “IP evasive manuevers”… in case my IP was targetted (I made sure I got a different IP address and that no registered domain points to the residential IP.) Anytime I’ve had to configure an open Access point of some sort, I pull the plug on the internet. I convinced the client that this was to be configured for that it would be wise to use encryption and that way they will have a bit more control over the users (ask for the passphrase). What’s really bothersome is the following train of though….
How could I prevent someone on a wireless segment doing an aggressive network scan against a common port? I could tighten down an access point to only allow traffic to port 80, but an attacker might still do a port 80 scan and get the owner of the AP bounced from their ISP account. It would be interesting if a firewall were able to throttle “aggressively timed” attempts to connect to a lot of addresses within a few seconds. I don’t know of any such type of throttling (there is bandwidth/traffic shaping, but that’s a slightly different concept.)
So, a week on, I’m tired, I haven’t found any evidence of intrusion/rootkit/unauthorized use on my systems. I suspect that either 1)the wireless was abused very shortly after plugged in, or 2) my IP address was spoofed (either intentionally or I just happened to “luck out”.) or 3) I have some sort of intrusion/rootkit that known rootkit scanners do not detect. I feel as though my every move online is being watched now that I’ve had one suspicious incident. I’m avoiding using whois or nslookup or dig locally and I was a bit reluctant to post this article (increased paranoia).
I’m using ssh port forwarding quite a bit more than I used to. I haven’t dared to use my (encrypted) wireless yet since the event. I still plan to change the encryption key. Password are being changed all just in case. I’m going to leave this article open to comments in large part for the goal that I would LOVE to know if there are any techniques for preventing a user on a LAN from doing any kind of port scan of the outside world. The only thing I can think of is limiting access to outside ports (which would still let it happen, just limit the options for which ports could be scanned.) The idea of throttling or slowing down “aggressive” connect attempts might be a possibility. But I don’t know how that might be implemented yet.
On a related note…. at a recent appointment I found an install of nmap on a machine that was used by quite a large number of untracable users. (Old Windows 98 machine public access. Was being replaced.) It looks as though the install was crippled (no libpcap) and it didn’t seem to work against the LAN (I support their LAN, I have permission to scan it…) Outside of logging (which gives you evidence after the fact), I don’t know of any way to keep a LAN user from scanning the outside world (yes you could limit the ability to install software, but what if you’re dealing with a wireless access point? Could you see it shut down in a matter of an hour over a user scan of the outside world?)
Anyway, that’s been the story of the last week. I’m still cautious in trusting my local machines, but at this point the lack of evidence would seem to point me to the explanation that the “event” used a temporarily connected open wireless router to get me suspended. What a mess.