Posted: February 14th, 2010 | Author: charlie | Filed under: IT Management, Linux / Unix, Networking | Tags: linux, monitoring, networks, NMS | 20 Comments »
Another perfect example of open source software gone commercial is Zenoss. As a full-featured network and service monitoring solution, Zenoss is one of the best monitoring tools available.
Most importantly, Zenoss combines two functionalities. First and foremost an enterprise environment requires host and service monitoring, with notifications. Network monitoring really means checking services, checking that hosts are up (they ping), and possibly writing your own plugins to check various other aspects of a server or network device. Until now, Nagios has filled that role.
Second, once a decent monitoring solution is in place, getting time-based information becomes desirable. Memory and CPU usage is the most prevalent example: if you’re checking available swap space every so often with Nagios, you may know when you start running low. But it may be just as important to see a graph of the last week’s usage. Tools like Cacti or Munin, which collect data frequently and use RRD graphs to display it, are very useful.
Zenoss fills both roles, without the annoying shortcomings prevalent in the alternative solutions. Zenoss uses the terms Availability Monitoring and Performance Monitoring to describe these two fundamental roles.
Performance of monitoring tools is important, and often times overlooked until it becomes a debilitating problem. For example, if you want to chart pretty RRD graphs of systems statistics like available RAM or disk space, Munin is an option. Unfortunately it’s all Perl, and designed in such a way that prevents it from scaling to even moderate amounts of hosts. Cacti is a bit better, but monitoring close to 100 hosts is painful with either option. Along comes Zenoss.
Zenoss is written in Python, and uses a MySQL backend for storage, and by all accounts it appears to perform very well. The really great thing about corporate-backed open source is quality control. The community simply isn’t responsible enough to say, “No, this won’t work, re-implement it.” A company with QA is.
Speaking of features, Zenoss isn’t missing many. Flexibility seems to be top priority–it can monitor hosts with SNMP, Nagios agents, SSH, Windows WMI, and various other mechanisms. Many features they claim are a bit over-inflated, such as ZenPing (marketed as Network Topology Monitoring) but the feature set is rich nonetheless.
Zenoss’s primary functions involve four features:
- Inventory Tracking
- Availability Monitoring
- Performance Monitoring
- Event Monitoring and Management
Inventory tracking claims some sort of “configuration” reporting as well, but it seems very limited. Zenoss will discover your inventory and auto-populate a database. This is great for knowing which IP addresses are in use, for example, but means that “configuration” reporting is limited to an outside observer’s perspective. It can tell you which servers have a Web server running, but it certainly doesn’t deal with the configuration of the Web server. Of course, inventory tracking isn’t limited to automatically discovered information; there are manual input capabilities too.
Availability monitoring is basically Nagios, plus. It can ping, it can monitor Windows machines, and it can pretty much do whatever you need. Even your old Nagios plugins will work with Zenoss. It does generate reports, but much better ones than Nagios is capable of.
Host monitoring, performance monitoring, or whatever you’d like to call it, is quite robust in Zenoss. Some would think it’s light on features, but there’s a good reason that Zenoss requires you use SNMP: it’s much more scalable than SSH’ing to each server every minute. A bit of up-front configuration is required, in that all your hosts will need SNMP configured and working, but it’s completely worth it. Zenoss too uses RRD graphs, and it can generate events and alerts based on pre-defined thresholds.
Finally we come to event monitoring. Zenoss is also encroaching on Splunk‘s territory a bit. It can combine syslog, availability monitoring alerts, SNMP traps, and even Windows event log data. Much like Splunk, Zenoss correlates similar events for easier viewing and troubleshooting. This is the portion that processes all events and generates alerts to pagers or e-mail, taking into account the escalation procedure you’ve defined.
To top it all off, the Zenoss Web interface is top-notch. It includes a customizable “dashboard” for monitoring, and everything is AJAX-enabled. AJAX provides the user experience similar to Splunk and Google’s Gmail.
Marketing fluff aside, Zenoss really does provide a wonderful product. It is, of course, open source and available for free.
At last year’s LISA conference, Zenoss gave a demonstration that sadly coincided with free beer time. Stumbling in toward the end, I demanded one of their free baseball caps, and sat to listen to the last few audience questions. One thing was very obvious: everyone in the room was excited about this product. If hardcore sysadmins are excited, you know this is something worthwhile.
Zenosss is very functional and full of features. It may even be possible to replace three separate pieces of software with this one product: host inventory database, Nagios, and your performance monitoring tool of choice. Maybe even Splunk some day. We can’t wait to see what features they will be adding next.
20 Comments » Related posts:
- Managing Virtual Machine and Cloud Sprawl
- Squeeze Your Gigabit NIC for Top Performance
- Back to Basics: Unix System Stats Utilities
- The Perils of Sudo With User Passwords
- Built-in Security with Cisco IPS
Posted: February 13th, 2010 | Author: charlie | Filed under: Networking | Tags: linux, networks, NIC, tuning, windows | 1 Comment »
Many new workstations and servers are coming with integrated gigabit network cards nowadays, but quite a few people soon discover that they can’t transfer data much faster than they did with 100 Mb/s network cards. Multiple factors can affect your ability to transfer at higher speeds, and most of them revolve around operating system settings. In this article we will discuss the necessary steps to make your new gigabit enabled server obtain close to gigabit speeds in Linux, FreeBSD, and Windows.
Hardware considerations
First and foremost we must realize that there are hardware limitations to consider. Just because someone throws a gigabit network card in a server doesn’t mean the hardware can keep up. Network cards are normally connected to the PCI bus via a free PCI slot. In older workstation and non server-class motherboards the PCI slots are normally 32 bit, 33MHz. This means they can transfer at speeds of 133MB/s, but since it is a shared bus between many parts of the computer, realistically it’s limited to around 80MB/s in the best case. Gigabit network cards are 1000Mb/s, or 125MB/s. If the PCI bus is only capable of 80MB/s this is a major limiting factor for gigabit network cards. The math works out to 640Mb/s, which is really quite a bit faster than most gigabit network card installations, but remember this is probably the best-case scenario. If there are other hungry data loving PCI cards in the server, you’ll likely see much less throughput. The only solution for overcoming this bottleneck is to purchase a motherboard with a 66MHz PCI slot, which can do 266MB/s. Also, the new 64 bit PCI slots are capable of 532MB/s on a 66MHz bus. These are beginning to come standard on all server-class motherboards.
Assuming we’re using decent hardware that can keep up with the data rates necessary for gigabit, there is now another obstacle – the operating system. For testing, we used two identical servers: Intel Server motherboards, Pentium 4 3.0 GHz, 1GB RAM, integrated 10/100/1000 Intel network card. One was running Gentoo Linux with a 2.6 SMP kernel, and the other is FreeBSD 5.3 with an SMP kernel to take advantage of the Pentium 4’s HyperThreading capabilities. We were lucky to have a gigabit capable switch, but the same results could be accomplished by connecting both servers directly to each other.
Software considerations
For testing speeds between two servers, we don’t want to use FTP or anything that will require data be fetched from disk. Memory to memory transfers are a much better test, and many tools exist to do this. For our tests, we used ttcp (http://www.pcausa.com/Utilities/pcattcp.htm).
The first test between these two servers was not pretty. The maximum rate was around 230 Mb/s, about two times as fast as a 100Mb/s network card. This is an improvement, but far from optimal. In actuality, most people will see even worse performance out of the box. However, with a few minor setting changes, we quickly realized major speed improvements – more than a threefold improvement over the initial test.
Many people recommend setting the MTU of your network interface larger. This basically means telling the network card to send a larger sized Ethernet frame. While this may be useful when connecting two hosts directly together, it becomes less useful when connecting through a switch that doesn’t support larger MTUs. At any rate, this isn’t necessary. 900Mb/s can be attained at the normal 1500 byte MTU setting.
For attaining maximum throughput, the most important options involve TCP window sizes. The TCP window controls the flow of data, and is negotiated during the start of a TCP connection. Using too small of a size will result in slowness, since TCP can only use the smaller of the two end system’s capabilities. It is quite a bit more complex than this, but here’s the information you really need to know:
For both Linux and FreeBSD we’re using the sysctl utility. For all of the following options, entering the command ‘sysctl variable=number’ should do the trick. To view the current settings use: ‘sysctl ’
Maximum window size:
FreeBSD:
kern.ipc.maxsockbuf=262144
Linux:
net.core.wmem_max=8388608
Default window size:
FreeBSD, sending and receiving:
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
Linux, sending and receiving:
net.core.wmem_default = 65536
net.core.rmem_default = 65536
RFC 1323:
This enables the useful window scaling options defined in rfc1323, which allows the windows to dynamically get larger than we specified above.
FreeBSD:
net.inet.tcp.rfc1323=1
Linux:
net.ipv4.tcp_window_scaling=1
Buffers:
When sending large amounts of data, we can run the operating system out of buffers. This option should be enabled before attempting to use the above settings. To increase the amount of “mbufs” available:
FreeBSD:
kern.ipc.nmbclusters=32768
Linux:
net.ipv4.tcp_mem= 98304 131072 196608
These quick changes will skyrocket TCP performance. Afterwards we were able to run ttcp and attain around 895 Mb/s every time – quite an impressive data rate. There are other options available for adjusting the UDP datagram sizes as well, but we’re mainly focusing on TCP here.
Windows XP / 2000 Server / Server 2003
The magical location for TCP settings in the registry editor is:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
We need to add a registry DWORD named TcpWindowSize, and enter a sufficiently large size. 131400 (make sure you click on decimal) should be enough.
Tcp1323Opts should be set to 3. This enables both rfc1323 scaling and timestamps.
And similarly to Unix, we also want to increase the TCP buffer sizes:
ForwardBufferMemory 80000
NumForwardPackets 60000
One last important note for Windows XP users needs to be made. If you’re installed service pack 2, then there is another likely culprit of poor network performance. Explained in knowledge base article 842264, Microsoft says that disabling Internet Connection Sharing after an SP2 install should fix performance issues.
The above tweaks should enable your sufficiently fast server to attain much faster data rates over TCP. If your specific application makes significant use of UDP, then it will be worth looking into similar options relating to UDP datagram sizes. Remember, we obtained close to 900Mb/s with a very fast Pentium 4 machine, server-class motherboard, and quality Intel network card. Results may vary wildly, but adjusting the above settings are a necessary step toward realizing your server’s capabilities.
1 Comment » Related posts:
- Zenoss: We Can Ditch Nagios Now
- What the Heck is a TCAM?
- How Much Server do you Need?
- Understanding Linux Virtual Memory
- Back to Basics: Unix System Stats Utilities
Posted: February 1st, 2009 | Author: charlie | Filed under: IT Management | Tags: linux, ROI, web hosting | 2 Comments »
Whether you are a small business, fortune 500, or in-between, Web hosting providers may have something to offer. Web hosting is extremely competitive, and not many companies survive. Once the need for Web hosting services is established, you need to do some research to determine whether a specific hosting company is a good fit.
Small businesses will depend more heavily on their hosting provider, as their entire Web infrastructure may reside at a hosting company. As businesses get larger, and acquire their own IT staff, they tend to start running their own Web servers. At some point, maybe due to cost-cutting measures, companies may revisit the hosting option. If it’s really only $5 per month, then why not?
Hosting companies also offer virtual machines that your IT staff can run. These are a great alternative to hosting DR servers in remote data centers. VM plans are more expensive than a regular shared Web hosting plan, but not much more. With both options we need to pay close attention to the hosting company’s track record and value-add services.
What to Expect
There are literally hundreds of popular hosting companies out there. Each has their own tools for managing your domains, and each offers their own unique set of value-add services.
At a minimum, there are a few things you should demand from any Web host. Due to the brutally competitive nature of the business and interchangeability of these services, you should never be shy about demanding a new feature or picking up and moving to a new provider. A few of the basic things you should expect are:
- A Web interface to manage your domains, billing, and user accounts
- Shell, SFTP, and various other methods of remote access
- Unlimited MySQL databses, and an easy way to create and manage them
- Easy, automated installation mechanisms for common open source Web applications
- Unlimited e-mail boxes and a few options for Web-based access
- Log analysis and detailed reporting
- And lately: unlimited disk space and bandwidth
Five or ten years ago you would not find most of these features with the most popular hosting companies. These days, however, they all offer most of these items and a lot more.
Companies that offer VM hosting, or Virtual Private Servers (VPS) as some call it, will provide a basic set of tools to manage your virtual machine. You can generally select from a few operating systems and reload it at any time. Afterwards, however, you’re on your own. You get the root password and are free to install whatever you need.
Is it Worth It?
Psychologically, most people have a problem purchasing something that appears too cheap. When looking at Web hosting services, like WordPress Web Host, the list of options leaves most people drooling. Then they see the price: sometimes less than $5 per month, for unlimited disk space, bandwidth, accounts, and with a free domain registration. How can they do that?
The servers must be horribly overloaded as these companies cram more and more customers onto them. At $60 per year per customer, the hosting company can’t exactly afford to buy top of the line servers, right?
The truth is, hosting is brutally competitive. Most companies fail. The ones that survive have a high level of automation to allow them to manage their servers with very little manhours. Even at today’s sub-$5 pricing for a monthly plan, that equates to one $3000 server per 50 customers, at their yearly rate. A properly configured Apache Web server environment should be able to handle 300 Web sites on today’s hardware without a problem.
Of course, someone could write a PHP script that goes wild and consumes way too many resources, slowing down the entire server. The hosting companies monitor the servers, and this type of thing is usually dealt with quickly. However, it is worthwhile to ask your chosen hosting company whether or not they allow adult content. Adult sites are generally very high-traffic, and if you’re site is sharing the same Web server, it may suffer performance issues.
The Case for Secondary Hosting
Enterprises should not rule out hosting providers. You don’t need to outsource your entire Web team, but part of a good disaster recovery strategy will certainly include multiple off-site resources. Instead of paying co-location fees and trodding your own servers to a datacenter across the country, why not get a few virtual machines in multiple locations?
If your physical server dies, you may be sending FTE to the remote site to replace the hardware. In a hosted VM environment, you never have to worry about taking employees away from their daily work and paying travel expenses. Just click a button in the hosting provider’s Web interface to create a new VM if something goes awry.
You can host your own Web servers, secondary DNS, or just about any other service your company relies upon. If you’re worried about reliability, you can always select two or three VM providers in separate parts of the country (or world). To equal the cost of one server in a co-location environment, you’d have probably 10-20 virtual machines spread throughout the world.
Next Steps
If you’re interested in hosting Web sites with a Web hosting provider or VMs, the next step is to do some research. Read reviews of various hosting companies, and pay attention to issues people cite. There will always be a certain amount of people unhappy with a service, but pay careful attention to what they are saying went wrong. Likewise, pay attention to the few positive reviews you might find. Most happy customers remain silent, so when you find a positive review it holds more weight than a rant.
“Cheap web hosting” is a great Google search. You will find a few unbiased sites that list providers, the services they offer, reviews, and pricing. We don’t want to make any recommendations, but just remember: more expensive is not always better in a fiercely competitive market such as Web hosting.
2 Comments » Related posts:
- Managing Virtual Machine and Cloud Sprawl
- Zenoss: We Can Ditch Nagios Now
- The Perils of Sudo With User Passwords
