Life Pattern » linux

Attack vectors deja vu

zimage — Mon, 11 May 2009 07:01:37 +0000

I have to keep an eye on the IT security news. You know, “security is a process not product”. Just recently, Linux kernel vulnerability CVE-2009-1337 caught my attention. This even has l33t in its name The more interesting part is, of course, not the CVE number but the attack vector used in a recent exploit. Basically, a core is dumped to the logrotate.d directory. After this, logrotate executes the malicious code included in this dump since it uses rather naive parsing to find instructions in its configuration files.

Inevitably, this reminded me of a very similar situation from few years ago. In 2006, CVE-2006-2451, which is another kernel vulnerability, allowed core to be dumped in a directory that the attacker isn’t allowed to write to. A weakness in cron.d parsing similar to that in recent versions of logrotate was used as attack vector.

Just a few weeks ago, I had another deja vu. There’s a flaw in udev versions before 1.4.1 that allows local users to gain root privileges by not checking whether a NETLINK message originates from kernel (CVE-2009-1185). It took me some time to remember why this sounded so familiar since the older case is from 2003. Back then, the zebra routing suite failed to check the NETLINK message originators (CVE-2003-0858).

Oh well, to err is human, don’t you think?

Linux kernel vmsplice root exploit

IO performance monitoring

zimage — Sat, 11 Apr 2009 12:08:21 +0000

If you can measure it, you can manage it. I’m a measurement, monitoring, analysis and statistics addict

That’s why I’ve always wanted to be able to monitor the IO load of the Linux systems I’ve worked with. While there are well established monitoring and accounting tools for the CPU usage – both system wide and per process – there were virtually none for the IO system until very recently.

Two of the more important reasons why I’d like to see better IO load monitoring are:

The mechanical drives have big latency. In general the CPU feels much better than the disks when overloaded. For example if load average 10 is caused by CPU bound processes the system feels much more responsive than the same load but caused by IO bond processes. CPU load average 10 on a server system with two processors isn’t very noticeable. At the same time IO load average of 10 on the same system with 2x 7200 rpm disk drives in RAID1 feels very sluggish.
The hard disk drives failed to keep up with the performance improvements in microprocessor technologies. Disk capacity has grown quite well, but the speed and especially access times are far behind. The IO performance is the most common bottleneck and most precious resource in today’s systems. Or at least the systems I work with

At the beginning of my Linux career, ten years ago, there was only one metric – blocks read/written. And that’s it. How busy the disk is you can guess only by looking at load average and checking how many processes are stuck in D state. I wish there are separate load average readings for CPU and IO…

At some point (linux 2.5 times?) extended statistics were added and things like queue size, utilization in % etc. became available. Much better. Still it was hard to tell who exactly is causing the load. If we speak of multi user system all you can see is multiple processes in D state. It’s unclear whether these are the ones causing the IO havoc or just victims of the already overloaded IO subsystem waiting.

In Linux 2.6.20 another step was made by adding per process IO accounting. I was very excited when I heard about this feature and eager to try it. It turned out that this per process IO accounting counts only the bytes read/written by a process. Not that better. A modern 7200 rpm SATA drive is only capable of about 90 IOPS so it could be choked with the pathetic 90 bytes per second…

Then there are the atop patches. These add per process IO occupation percentage. That sounds great but… when you have a lot of small random writes they go to the page cache first and only then are periodically flushed to the physical device. This is performance feature and is generally a (very) good thing as it allows the elevators to group writes together etc. Unfortunately, atop ends up accounting all these writes and IO utilization to pdflush and kjournald.

Ok, lets see what’s the state of the affairs in some other operating system. Everybody talks about dtrace so it’s time to check it out. Linux doesn’t have dtrace. At least yet. There is work in progress by Paul Fox. On the other hand Linux has system tap but it doesn’t look very mature to me. Anyway, there are number of operating systems that support dtrace: as it is create by Sun engineers first come Solaris and OpenSolaris. Then there is the FreeBSD port and Apple OS X. I’m familiar with FreeBSD but I wanted to check the current state of OpenSolaris kernel. On the other hand I wanted to keep the learning curve less sloppy, so I opted for Nexenta core 2 rc1. Nexenta is GNU userspace (Debian/Ubuntu) and OpenSolaris kernel.

Download, install – everything was smooth. The install defaulted to root fs on ZFS. Good! I was thinking about playing with ZFS these days anyway.

And the moment of truth:

I started dbench -S 1, run dtrace -s iotop.d and here’s the output:

  UID    PID   PPID CMD              DEVICE  MAJ MIN D   %I/O
    0      0      0 sched            cmdk0   102   0 W     17

Hm, that looks somewhat familiar. I see a pattern there. Isn’t sched the ZFS cousin of pdflush/kjournald? Oh, well it is: http://opensolaris.org/jive/thread.jspa?threadID=39545&tstart=285

No luck… dtrace’s iotop works with UFS but has problem with ZFS.

Turns out the proper IO monitoring is a very tricky business.

MySQL Usage Accounting

Virtual Private Servers

zimage — Sat, 23 Aug 2008 07:59:23 +0000

A year ago I decided it’s about time to get a Virtual Private Server for my pet projects. Prior to that I had 2 sites on a shared hosting server and some other non-web network related apps spread around workstations that are always on. But at some point it all started to get messy and moreover I needed to put a few more things online. So I decided to get a VPS and consolidate all of my projects in one place. Compared to shared hosting environment the VPS gives me more freedom and fine grained control over the exact versions of applications and libraries I use among other things. It requires more work on my part though.

The fact that I have 10 years of experience working as a (unix) systems and network administrator is very handy when it comes to installing, configuring and troubleshooting the software I use on my VPS. Unfortunately it doesn’t help that much with the selections process so I had to sit down and filter the myriad of VPS providers out there.

I dug through blogs & forums, asked friends. Web Hosting Talk forums were particularly useful. Finally I have chosen Future Hosting. It turned out to be a good choice indeed. I’m with them for almost a year already and there weren’t any major issues. I asked for PTR RRs (that’s reverse resolving from IP to hostname) and their support staff quickly added these for me.

However good a hosting provider is you must keep a backup of your own. I was rather passively looking for second VPS provider for some time. One day I saw (can’t remember where) a Comfy Host’s ad. It looked suspiciously cheap at $10/mo to me and I haven’t heard anything about them before. But this was supposed to be a backup VPS so I decided to give it a try. I placed my order with Comfy Host using my PayPal account and started waiting for a welcome email. One day later I received an email saying that due to the surplus of orders my VPS setup will be delayed a little bit. Ok it happens. Few days later I had to submit a support ticket to ask what’s going on with my account. After some more back and forth support tickets I got my VPS up and running.

I used it for three months and it was pretty stable. I used it only for my automated daily backup and monitored it with nagios. Then one day suddenly the monitoring lost connection with the VPS. Since it is a backup server and I was busy with other things I let it stay this way for a week. Finally I logged into the control panel and guess what… I was paying on a monthly basis after receiving an invoice from Comfy Host. Last month I didn’t receive an invoice. And I couldn’t pay either because according to the control panel I didn’t owe money despite being late with the payment. Hm… I submitted a ticket to ask why is that. I had to wait several days for a reply that read:

“Sorry about the late reply. The ticket seems to have been overlooked without a response. We received no payment and therefore your VPS was shut off.”

Enough. I went looking for another backup VPS. I remembered that Leo from Zen Habits once said he is satisfied with his current provider – namely slicehost.com. So I’m with slicehost.com as my backup VPS for about a month now. So far so good.

TLS SNI – almost there… or not?

Family’s IT Consultant: The Computer Display

zimage — Tue, 17 Jun 2008 10:08:44 +0000

In today’s world of technological wonders few of us wouldn’t welcome some guidance. And since my friends and relatives know I work with computers they often ask me to help solve various issues and help them choose their new electronic devices. Actually it might have been worse… not that I’m not helpful guy (I am friendly) but I’m not the most communicative person out there so they don’t bother me as much as they might probably wanted to. On the other hand quite a lot of my friends are IT experts

Anyway, my little niece Teda will be starting school this fall. She already has a room of her own, her parents are buying her textbooks and other school aids and finally a computer. We decided to use an older computer to keep the cost down and invest instead in a good display.

The computers I work with are quite specific and I had to brush up a little bit my knowledge of general purpose desktop systems. For example my personal computer is a 3 years old IBM ThinkPad X41 (12″ screen) and I’m totally satisfied with it’s performance and features. The rest are servers with 8 or more hard drives and 8 CPU cores each.

We decided to go for a 19″ or 20″ wide screen TFT LCD display but also weren’t willing to pay more than 450 leva (~$350). First things first and I went for learning more about various types of current TFT technologies on the market. I almost instantly ruled out TN displays and focused on MVA panels since IPS wouldn’t fit in our price range. Most of the MVA panels are more pricey than we liked but finally I spotted an Asus PW201 that according to the manufacturer’s specifications was built with a P-MVA panel and also its price has recently dropped from about 700 leva to 449 leva.

This price drop seemed suspicious to me so I called the shop to make sure that this particular model and batch is P-MVA and not TN. After they assured me several times that this is a P-MVA panel (they called their warehouse) I proceeded to order but still suspicious I wrote in the order’s comment field: “If you are not sure whether this monitor is of P-MVA type please cancel the order”.

Finally the display arrived. I paid to the delivery guy and opened the box. Oh boy, this thing looks splendid. Very stylish. It has integrated stereo speakers although expectedly the sound quality is mediocre. The integrated USB hub is very convenient because you don’t need to crawl under the desk in order to plug USB devices. Audio, USB and VGA cables run together so despite using DVI for video you’ll have to use the VGA cable as well. The buttons on the front are actually small touch sensitive areas and react somewhat erratically. My overall impressions about PW201 are very good. It is definitely worth its price.

I intended to install Kubuntu 8.04 (Hardy Heron) so I hooked up monitor’s DVI port to the PC, attached the keyboard and… suddenly realised I don’t have an optical drive on this PC. Okay let’s try network install I followed the instructions listed at http://wiki.koeln.ccc.de/index.php/Ubuntu_PXE_Install but since I wanted to install Kubuntu I opted for “console only” installation in the Ubuntu installer. After the installation had finished I installed the rest of the Kubuntu with:

sudo apt-get install kubuntu-desktop

So far so good. The X server detected the Nvidia GeForce4 MX 440 graphics adapter and used nv to drive it. It properly detected the monitor’s properties and started in the panel’s native resolution of 1680×1050. Unfortunately the nv opnesource driver lacks 3D acceleration (because Nvidia refuses to provide the hardware specifications needed to add 3D support) and this is where I spent 3 hours trying to get the proprietary nvidia driver to work properly with my setup. Most of the time was spent trying to set it up to use 1680×1050 but nvidia driver kept thinking that PW201 has maximum pixel clock of 135 MHz when it’s actually 146 MHz. Finally I found this set of options working for me:

Section “Device”
Identifier “GeForce4 MX 440 with AGP8X”
Driver “nvidia”
Option “ExactModeTimingsDVI” “true”
Option “NoBandWidthTest” “true”
Option “ModeValidation” “NoDFPNativeResolutionCheck, NoEdidMaxPClkCheck, NoMaxPClkCheck”
EndSection

This did the trick but after only few minutes of playing video with mplayer and Xv (Xvideo) output driver the video window started to show only colorful noise. Tried OpenGL output driver and worked better but used 95% of the CPU vs. 55% for Xv (that’s AMD Athlon XP 1800+). Moreover after half an hour video artefacts started to appear all over the screen and I reverted to the “nv” driver.

I don’t know whether this is a software/driver problem or maybe the graphics adapter is having some issues but since it works with the nv driver (and my niece doesn’t need 3D acceleration) I’ll stick with it for now.

No related posts.

Linux kernel vmsplice root exploit

zimage — Mon, 11 Feb 2008 20:47:38 +0000

Two strings walk into a bar. The first says, “Hello, I’d like a ciderO’y?kI’U`,E’*@???’?? ?!>A~Xx?(y’n?.” The second says, “Please excuse my friend, he’s not null-terminated.”

—

If you are running Linux kernel newer than 2.6.17 but older than 2.6.24.2 or 2.6.23.16 then any local user can easily become root or at least crash your system.

There are actually two different security issues related to vmsplice() system call and both of them could lead to local privileges escalation. This is especially bad for people who don’t fully control content on and access to their servers – e.g. web hosting companies. The other bad news is that vmsplice() is part of the core kernel and there is no configuration option to exclude it.

Two separate exploits have been publicly released which exploit each of the two issues respectively.

The first issue was classified as CVE-2008-0009 and CVE-2008-0010 and was fixed by linux kernels 2.6.23.15 and 2.6.24.1.

The situation with the second issue, classified as CVE-2008-0600 was much worse. It was introduced with the initial implementation of vmsplice() and affects all kernels after 2.6.17 inclusively. There was an exploit in the wild for more than 24 hours without proper fix for the problem. I’m sure that even though there are patched versions now – linux 2.6.24.2 and 2.6.23.16 – at least 2-3 more days will pass before the number of vulnerable systems is reduced enough.

IMHO this is one of the worst 0-day Linux kernel exploits in years. I hope it won’t happen again soon. But you should be careful because all this has happened before and will definitely happen again someday.

Attack vectors deja vu