Tag: it

  • IO performance monitoring

    If you can measure it, you can manage it. I’m a measurement, monitoring, analysis and statistics addict 🙂

    That’s why I’ve always wanted to be able to monitor the IO load of the Linux systems I’ve worked with. While there are well established monitoring and accounting tools for the CPU usage – both system wide and per process – there were virtually none for the IO system until very recently.

    Two of the more important reasons why I’d like to see better IO load monitoring are:

    • The mechanical drives have big latency. In general the CPU feels much better than the disks when overloaded. For example if load average 10 is caused by CPU bound processes the system feels much more responsive than the same load but caused by IO bond processes. CPU load average 10 on a server system with two processors isn’t very noticeable. At the same time IO load average of 10 on the same system with 2x 7200 rpm disk drives in RAID1 feels very sluggish.
    • The hard disk drives failed to keep up with the performance improvements in microprocessor technologies. Disk capacity has grown quite well, but the speed and especially access times are far behind. The IO performance is the most common bottleneck and most precious resource in today’s systems. Or at least the systems I work with 🙂
    • At the beginning of my Linux career, ten years ago, there was only one metric – blocks read/written. And that’s it. How busy the disk is you can guess only by looking at load average and checking how many processes are stuck in D state. I wish there are separate load average readings for CPU and IO…

      At some point (linux 2.5 times?) extended statistics were added and things like queue size, utilization in % etc. became available. Much better. Still it was hard to tell who exactly is causing the load. If we speak of multi user system all you can see is multiple processes in D state. It’s unclear whether these are the ones causing the IO havoc or just victims of the already overloaded IO subsystem waiting.

      In Linux 2.6.20 another step was made by adding per process IO accounting. I was very excited when I heard about this feature and eager to try it. It turned out that this per process IO accounting counts only the bytes read/written by a process. Not that better. A modern 7200 rpm SATA drive is only capable of about 90 IOPS so it could be choked with the pathetic 90 bytes per second…

      Then there are the atop patches. These add per process IO occupation percentage. That sounds great but… when you have a lot of small random writes they go to the page cache first and only then are periodically flushed to the physical device. This is performance feature and is generally a (very) good thing as it allows the elevators to group writes together etc. Unfortunately, atop ends up accounting all these writes and IO utilization to pdflush and kjournald.

      Ok, lets see what’s the state of the affairs in some other operating system. Everybody talks about dtrace so it’s time to check it out. Linux doesn’t have dtrace. At least yet. There is work in progress by Paul Fox. On the other hand Linux has system tap but it doesn’t look very mature to me. Anyway, there are number of operating systems that support dtrace: as it is create by Sun engineers first come Solaris and OpenSolaris. Then there is the FreeBSD port and Apple OS X. I’m familiar with FreeBSD but I wanted to check the current state of OpenSolaris kernel. On the other hand I wanted to keep the learning curve less sloppy, so I opted for Nexenta core 2 rc1. Nexenta is GNU userspace (Debian/Ubuntu) and OpenSolaris kernel.

      Download, install – everything was smooth. The install defaulted to root fs on ZFS. Good! I was thinking about playing with ZFS these days anyway.

      And the moment of truth:

      I started dbench -S 1, run dtrace -s iotop.d and here’s the output:

        UID    PID   PPID CMD              DEVICE  MAJ MIN D   %I/O
          0      0      0 sched            cmdk0   102   0 W     17
      

      Hm, that looks somewhat familiar. I see a pattern there. Isn’t sched the ZFS cousin of pdflush/kjournald? Oh, well it is: http://opensolaris.org/jive/thread.jspa?threadID=39545&tstart=285

      No luck… dtrace’s iotop works with UFS but has problem with ZFS.

      Turns out the proper IO monitoring is a very tricky business.

  • The Internet Tablet Revolution

    Since the introduction of the first Palm devices back in 1997 I was tracking the evolution of pocket computers with great interest. However, for a very long time I couldn’t find the combination of factors that would make me buy one. The first device I seriously considered was N770 from Nokia. It was breakthrough in many ways but still quite limited in it’s memory and processing speed. I kept an eye on its development and when Nokia released it’s successor N800 I was sold. I bought one and used it for about 9 months when N810 was released. I immediately upgraded. I wear my N810 wherever I go for more than a year now and I’m very satisfied.

    Some of the reasons I didn’t like any of the previous generations of PDA were their small screens, the use of closed operating systems or their high price tag. So the main advantages of N810 over it’s rivals at least for me are:

    Superior display than most of the PDAs. 4.1″ is very good and balanced size. Any bigger and it won’t fit in your pocket. Superb 800×480 screen resolution – you can’t go better for a display of this size. And due to the fact this display is transflective it’s readable even in bright daylight.

    N810 is more mobile than the netbooks, which are gaining popularity recently. It’s smaller and lighter and it fits in your pocket or in a lather sleeve like this one. Better mobility is also aided by the very good battery life. I use mine to read while commuting, some internet browsing and I might turn on the gps once in a while. That’s about 1.5 hrs daily and I keep it in off-line suspended mode (but instant on – less than a second) for the rest of the time. Its battery lasts about 5 days between recharges with this usage pattern.

    N810 runs an open source, linux based OS and application stack designed for use with touchscreens. Openness, flexibility, best of breed package management (debian’s apt). Want maemo mapper + openstreetmap, xterm + openssh, Doom or other popular application? No problem – check out http://maemo.org/downloads/OS2008/. Not strictly related but it’s browser is gecko based (that’s mozilla & firefox engine) and has flash support. Some of the greatest apps for N810 are written by the community.

    N810 has built-in gps and comes with Wayfinder pre-installed. If you want turn by turn navigation you’ll have to pay a modest subscription fee to use this option. maemo mapper is a free mapping and navigation alternative. maemo mapper’s routing capabilities are not as extensive and route recalculation needs an Internet connection but it’s very good app nevertheless. Go try it. There’s also navit – a very promising navigation app in the making. This gps led us trough most of the southern Spain few months ago.

    As expected for an Internet Tablet N810 has bluetooth and wifi. No device is useful without good connectivity options in this day and age. Hook it to the nearest WiFi hotspot or your 3g enabled bluetooth phone. And it’s WiFi is among the best I’ve used – very good range and compatibility with hot spots around the world.

    One of the great surprises was the quality of the built-in qwerty keyboard. I didn’t expected nearly as comfortable typing. Sometimes I have this feeling that I didn’t hit the right key but when I look at the screen it’s all correct. Very good job, Nokia.

    Another surprise was the amazing sound for a device this small. It actually sounds better than my ThinkPad X41. It’s last.fm player, vagalume, is killer app. Use it like a portable personalized radio.

    There are built-in camera and mic as well. Combined with the pre-loaded software like jabber client with voice and video support (and gtalk), gizmo and skype it’s the ultimate communicator.

    It’s light sensor is very handy. Based on the current light conditions the display brightness is adjusted to save power and make work with the tablet more comfortable.

    And last but not least the price: it was first released at $450 and is currently (Dec 2008) being sold for about $335 in various on-line stores.

    What’s next? There are quite a lot people that prefer their internet tablet to have some built-in long range connectivity options in addition to WiFi like GPRS, 3G or WiMAX. Nokia responded to this demand with N810 WiMAX edition. I personally am completely satisfied with 3G via bluetooth phone.

    But what’s more interesting is the upcoming fourth generation internet tablet (N900?). Rumours are that it will be OMAP3 based, will have built-in HSPA connectivity, high definition camera and hardware based graphics acceleration. Recently nokia released the Maemo 5 SDK which is targeted at this future device’s developers.

    I’m glad someone (Nokia) finally has taken this technological way. I hope they continue evolving the internet tablet series in the same way and spirit. Can’t wait to see what N9xx will look like…

  • MySQL Usage Accounting

    Those of you who use MySQL in a multi-user environment are probably disappointed by it’s lack of per user usage statistics. For example in Linux there are BSD process accounting for CPU usage and IO accounting for disk usage tracking. Most applications like apache, ftp servers and mail servers provide some sort of usage statistics as well. But MySQL is the great anonymizer.

    Not anymore, thanks to the people at Percona for their microslow patch and Google for the UserTableMonitoring patch.

    Latest version of the patches can be found here but at the time of this writing they are against MySQL 5.0.62 and the current MySQL version is 5.0.67. So I tweaked the patches to apply cleanly to MySQL 5.0.67 and placed them for download: microslow, userstats, microslow+userstats.