The Assailant Left Behind Some Clues
In my last article I told you about my system being compromised. Since then, I have been by and large offline, because every time I plug my computer into my router, which stays disconnected on the wan side, there are literally dozens of processes using very odd port numbers, and there all listening for a reply from one of the two DNS servers I have configured on the standard port 23. If I had more sophisticated software, I'd be able to see which URI they are trying to resolve. Something is in the works, and I may be able to do that soon. But that's not my story. The story is that I now believe that the spam they sent out from my system was merely a distraction.
In my offline forensic work, amateur as it is, I've discovered several things. The details will be somewhat vague because I'm not trying to divulge their methodology, in fear that it will be picked up by others. But I will give you enough so that you can inspect your own systems, and see if you also have an intruder.
First of all, I now know that the intruder had been in my system far earlier than Aug 28th, when I went offline. In fact, if I had to guess, what happened on Aug 28th was a retaliatory strike against my blog, for having taken counter measures on my side against the intruders. But why were they there to begin with?
As I write this, I have just pulled an all night'er going over something the intruder left behind. The actual logs of my system settings on Aug 3rd. In fact, each of the logs are dated and time stamped in their name, not just the last modified date. Well, obviously these were huge logs, so I did a diff on them, incrementally by dates to see what had been changed.
The first one is dated syssettings.200708030841, or Aug 3, 2007 at 8:41am. When I diff it against the next one dated 200708030900 what I get is the addition of 156 entries under the group:
> swupdate:ItemsRecordsArray:_arrary_index:0-155:Description = <hex …>
And one modified line:
< swupdatelastChecktime = "2007-08-03 07:41:36 -0400"
> swupdate:lastChecktime = "2007-08-03 08:42:16 -0400"
So you can deduce that a software update was performed in between the two logs. All the data is in hex, but since they all end with 0a, macs preferred end of line character, it would seem to be, as the item indicates, a text based description. I haven't decoded them yet, because that's not what is significant. What is significant is that there were 156 updates, in the span of less than an hour, if you believe the lastChecktime stamps. My system, which I have owned for many months was set to auto-update, which meant that it was unusual for me to even get more than one (1) or two (2) updates at time. Why would they want these updates?
Well, as my system is the Server version of Mac OS X, it has the capability of mirroring updates, that, if I were properly licensed, could be used for network installs. But I only have one Mac, and I'm only licensed for 10 clients. The other computers that make up my lab are all MS XP boxes. This was my first outing with Macintosh.
I'm not going to discuss what I found in the diffs of the other files specifically, as I believe this would be to the advantage of other intruders. But I will say that the changes were made to my default printer configuration and the mail:postfix and mail:imap areas. So, the changes that occurred on Aug 3, 2007 were two pronged.
I had already been highly suspicious of what was going on with my computer, because XXX suggestions kept popping up in my iGoogle page as suggestions based on my Internet usage. Since, as you all know, I'm a 50 something retired geek who spends his time dreaming up interesting, at least to me, things to do with the language Scheme and assorted other functional languages, I was not a little disturbed that iGoogle would be suggesting I head over to the latest `Live Nude Girls' site. When I reviewed my search history, it was exactly what you would expect from a nerd. But, when I went to Google Reader, I discovered quite a different history. You'd think I was a nineteen year old horn dog from the list of sites that were displayed in my history. I tried in vain to get this corrected by Google. Since their security reporting system is form based, and while you can report abuse quite easily, you must have a specific post or email to provide them. I even tried using a fake email, so I could get through to somebody at Google, but they aren't that easily fooled. If anyone is aware of an way to submit a security related concern or simply email their security group, I would appreciate knowing how one goes about it.
So naturally I changed my Google password to something quite a bit stronger. However, after another month or so went by I started to get the same types of tawdry suggestions from iGoogle. This time I simply put in a password so long and random that I figured the odds of someone guessing it were essentially nil. Well, as I now know, they weren't guessing at all. Since these passwords were so long, they had to be stored on disk, and I was obliged to copy and paste them in whenever I needed to access my account. But your probably saying, this happened months before Aug 3rd, and you'd be correct.
That's where another piece of forensic work comes in to play. It turns out that quite a number of executables in /sbin had a different date than the others, when you view it with `ls -lc'. For those not familiar with *nix, the -c option shows the dates of last change, instead of the creation date. What makes this unusual is that the date is Jun 25 12:43 for the lot. Where as the few not affected are dated Nov 15, 2006, well before I had purchased the computer. [Author's Note: After some more thought, there is a benign explaination for this. If a single Apple update required newer versions of the majority of the programs in /sbin, then it is possible that they could have a different date, but all be the same.] The other interesting thing I discovered was as a result of recently learning that the reason I was never getting any color coding of file types in an ls listing, was that the default terminal emulator, that I had been using, didn't support color. For those familiar with *nix, it will come as no surprise that what I saw was an assortment of executables using black text on a red background. For those not familiar, as I was not until just recently, this indicates that a file has the SUID bit set. This does not show up in any standard usage of ls; by default all they present to you is the user, group, and other permissions. [Author's Correction: After a little sleep, I see that an `s' is used in place of an`x' in the user symbolic permissions when the suid bit is set. You see what you know, as I'm known to say with some frequency.] A file with SUID set will execute with the privilege of the superuser, regardless of the privilege of the user executing the program.
So, it is now seems clear that the intruder had gained access to my system through an escalating of privilege attack, and was probably inside my intranet prior to Jun 25, 2007. People thought, including myself, that I was over-reacting to a threat I could only vaguely describe, from the strange suggestions in iGoogle, and the various error messages that were showing up in my logs. I had even gone to the extent of making my OS X box nearly FIPS compliant, only lacking in the physical security department. But the audit logs take a while to figure out, and Apple's support system for their Server product is quite expensive. The cheapest support option is some $2,500. There's always the per incident option, which will put you back $200 a call. And, in fact, I made two such calls to sort out some odd behavior with my box. The last analyst I got told me to turn off the firewall that comes with the server, which is disabled by default. He told me I'd be fine with a hardware firewall router. But, if any of you are Steve Gibson fans, you should know that an external firewall can't block a process that has no business going out to the Internet, because it only cares about the port that a packet is going to, and as long as that packet is well formed, and the port is not blocked because it's a common port for outgoing traffic, the firewall is obliged to send it on its way. And since the conversation initiated from within the intranet, it will gladly accept well formed responses, as long as they are correct for the state of the conversation. [Author's Note: There may exist security devices that can be configured to work with software on the computer to identify the process that is sending out a packet. I'm not a router guru. I just know that I've never owned a router that has this capability.]
So, I didn't turn off my firewall. In fact, I got a much better software firewall, that actually showed me where the packets were going, and from which process they came from. However, the intruder had rigged `ipfw' (Apple's not so comprehensive out of the box firewall). So, the better firewall was being foiled by the infected `ipfw'. It wasn't sufficient to merely stop ipfw from the cute little GUI interface apple has for their *nix daemons. I had to physically quarantine `ipfw'; but, of course, this was way too late for anything to stop the damage already done. It only provided some additional forensic evidence.
I have megabytes of forensic evidence, but I'm still facing a complete re-installation of all my computers. You see, at some stage, either early on to begin the attack, or later on, to continue the attack, despite my precautionary measures, they were using the XP boxes to gain access to the server. The infection completely shuts down Norton 360, whenever you reboot the computer, and even while 360 appears to be scanning, and giving a clean bill of health to your system, the infection is raging. In fact, the more you do to prevent it's operation, the more it cripples your system, even attacking DVD/CD drivers leaving you no solution but to live with the infection or rebuild from scratch.
As I promised, I'll give you a list of warning signs about this particularly nasty infection.
First of all, make sure you have audit enabled on your system. I don't have the link handy, but if you search the Apple Knowledge base for audit, you should get the appropriate pdf that gives you everything you need to know about becoming as FIPS compliant as you want to.
Second, buried in my printer log, I now know, were early signs of the infection. The messages are few and far between, but they will indicate that postfix is running with `root' privilege's. This should never be the case. Postfix has a postfix account that it should be running on.
Another clue, is to run
$ sudo ps -Ao 'pid, uid, user, command' | less
If you notice that all your daemons are being executed by either yourself, www and root, you have got a serious problem. You might even see the unknown user in the above command or in your logs listed as `[uid -2] [gid -2]'. This would be a bad thing.
And then there are countless other error messages that refer to out of range parameters to empty arrays, that typically coincide with a malloc error message. When I brought this to Apple's attention on my second service call, they told me I must have been out of memory. Yeah, on a 4GB machine, I'm out of memory. I don't think so, and neither should you.
One final bit of information. While searching for these error messages, I ran across a very strange site, forgive me, I don't have the URL at hand. It detailed an attack that exploits the postfix system in OS X, and insures it's survival by infecting the program responsible for resetting permissions on an OS X drive. The sign of infection will be that the program has a strangely recent last modified date. They use this program, because Apple, in it's wisdom, has chosen to have it's SUID bit set. And, if you've ever called Apple for support, you know that the first thing they have you do is run this program. In fact, the documentation suggest that you run it after any application installation. This is wise advice, since the install may have left inappropriate permission on directories. But, by allowing any user to run the program, it becomes a natural target for exploits, since it runs as root. Again, I'm leaving out details on purpose. Why anyone would publish such a thing is beyond me, but in their minds they feel they are ultimately doing good. So, their site will continue to provide would be intruders with well thought out exploits.
I'm not so sure I've done you much good either, since by the time your seeing these messages in your logs, it's probably too late. But at least you would know to take your machine off the Internet, where it is no doubt infecting other machines.
And the motive behind these intrusions may well be to steal Apple software. From what I see in the logs left behind, it appears that they are using the zombie machines to access the built in capability to deliver client software from an Apple server. This ought to be a top priority for Apple, since every server they infect, could conceivably be delivering countless clients, free of charge to the intruders.
I wish I could end this post with my usual Enjoy! But I'm afraid today, it's more appropriate to say,