Back to Basics: Unix File Permissions

The most basic, yet important part of mastering Unix is to fully understand the nuances of file permissions. Tools exist to manage permissions easily, but true enlightenment and quick troubleshooting skills come to those who wholly master the concept. Remember, 80% of Unix problems are permissions issues.

The Concept
At the most basic level, there are three types of access:

• Read – the ability to open a file and read it
• Write – the ability to write the file
• Execute – the ability to execute (run) the file

Directories, though similar, are subject to special rules. Write permissions on a directory imply that you can create new files and directories within. Execute permissions are required to ‘cd’ into the directory, and read permissions are required to list the contents (‘ls’).

You will generally see permissions represented as r, w, or x; for read, write, and execute. Running ‘ls –al’ on the command line will show three sets of these strung together.

For example: -rwxr-xr-x

The dash means that the permission is not set. The first place is always reserved for special identifiers, like ‘d’ for directories or ‘c’ for character devices. The next place begins the actual permissions, for the user, group, and other categories.

Every access control in Unix is based on “who you are.” The user is identified by the uid (user ID), as defined by a person’s user account. The third field in the /etc/password file, for example, specifies what a user’s uid is. Similarly, every user belongs to a default group, as identified by the fourth field in the passwd file. Users can belong to many groups, but they’re always a member of their default group.

The above example of -rwxr-xr-x means that the owner of the file may read, write and execute it, the group members may read or execute it, and everyone else on the system may also read or execute the file.

A full example, from the output of ‘ls -l’ is:
-rw-r–r–  1 charlie root        164 2006-12-10 23:51 test.js

The file named test.js is owned by me with read and write permissions, is set to the root group who can only read it, and also allows everyone else to read it.

How it Really Works
That’s basically enough to get by, but being able to understand the more advanced modes of file permissions, your umask, and the numeric representation demands a full understanding. In reality, there are 8-bits available for each type of attribute. Take a look at Figure 1 and note that wherever you see a 1 in the binary column, a corresponding permission will exist.

As you can see, if a “bit” in a certain position of the binary representation is set, the permissions in that space are activated. The number column is the octal representation, and the “Binary” column is how it really works, from the operation system’s perspective.

Example time. Let’s say we wish to give ourselves read/write/execute permissions, the group read/execute, and everyone else read/execute permissions. The following commands both do the same thing:

• chmod u+rwx .; chmod go+rx .
• chmod 755 .

Since we know that setting ‘5’ means rx, we can simply say ‘5’ instead of ‘rx.’ The real advntage to knowing the octal representation is that we can set any arbitrary permissions with a single command. Running the chmod command using the mnemonic requires that we run it each time for each set of permissions.

Likewise, to set our umask, we must know how the permissions are numerically represented. The umask is the default mode with which files and directories will get created. It’s a mask, so if we want to create all files with permissions like 755, we must take the mask. Simply subtract 7 from each item, and 022 reveals itself as the magic setting. See the umask man page for further details.

Advanced Modes
There are, in fact, three other modes you can set on a file or directory. All Unixes support the following:

• 4000 set user id (suid) on execution
• 2000 set group id on execution
• 1000 the sticky bit

If suid is enabled, the permissions look like: -rws——
This means that when the file is executed, it will run with the permissions of the owner of the file. It’s dangerous, but some times necessary and quite useful. For example, a file suid and owned by root will always run as root.

When sgid is enabled, the permissions look like: -rwxrws—
When set on a directory, sgid means that all files created within the directory will have the gid set to the current directory’d gid. This is handy when sharing files with other people, who will often forget to give other members read or write permissions.

The sticky bit looks like: -rwx——T
When the sticky bit is enabled, only the owner of the file can change its permissions or delete it. Without the sticky bit, anyone with write permissions can change the modes (including ownership) or delete a file. This one is also handy when sharing files with a group of people.

There are other tidbits of information, once you get into the nuts and bolts of Unix file permissions too. For example, you can also set ACL attributes, which get horribly complex. Yes, you can give individual users access to your files, but it’s better not to. Creating a new group and sticking to general permissions can accomplish most things. Often the extended attributes aren’t necessary, and ACLs likely won’t work over NFS if you’re using Linux.

Spend some time with the chmod manual page to master tricky parts, if they still aren’t clear. It will also mention some implementation-specific limitations you may need to be aware of.

Feel free to share!

Hide Sites

Related posts:

1. Back to Basics: Unix System Stats Utilities
2. Back To Basics: Unix Differences in Performing Tasks
3. Working With Unix Variant Differences
4. Multi-user Security in Linux
5. The Perils of Sudo With User Passwords

What the Heck is a TCAM?

Let’s talk about TCAM hardware, Cisco SDM templates, and try to answer that elusive question: “why do I have to reboot my router to enable certain features, which in turn disables others?”

First, CAM stands for Content Addressable Memory. A CAM is a special type of memory; some would say the opposite of RAM. With normal computer memory (RAM) the operating system provides an address, and receives the data stored at the supplied address. With a CAM, the operating system supplies the data, and the CAM returns a list of addresses where the data is stored, if it finds any. Furthermore, a CAM searches the entire memory in one operation, so it is considerably faster than RAM.

CAMs are very expensive, so they aren’t normally found in PCs. Even router vendors will sometimes skimp, opting to instead implement advanced software-based searching algorithms to plod through RAM. Most commonly, CAMs and TCAMs are found in network processing devices, including Intel IXP cards and various routers or switches. The most commonly implemented CAMs are called binary CAMs. They search only for ones and zeros; a simple operation. MAC address tables in switches commonly get stored inside binary CAMs. You can bet that any

A Renesas TCAM

switch capable of forwarding Ethernet frames at line-speed gigabit is using CAMs for lookups. If they were using RAM, the operating system would have to remember the address where everything is stored. With CAMs, the operating system can find what it needs in a single operation. In this case desired data is the switchport that data should be sent out, based on the given MAC address, i.e. the essence of a MAC table. Some older Cisco switches running CatOS even opted to call this table the cam table, thereby causing great confusion across the land. Bridge table, forwarding table, mac-address table, cam table; it’s all the same.

Finally, a TCAM is a Ternary CAM. This allows the operating system to match a third state, “X.” The X state is a mask, which means you don’t care what it is. This naturally lends itself to networking, since netmasks operate this way. To calculate a subnet address we mask the bits we don’t care about, and then apply the logical AND operation to the rest. Being able to do this in hardware is a great benefit for routers. Additionally, routers can store their entire routing table in these TCAMs, allowing for very quick lookups. A router with routing tables in TCAMs can find the next-hop destination in a single operation every time instead of trying to search through a tree (or other data structure) in RAM.

Hardware can sometimes seem magic, but it isn’t always transparent. When configuring routers most people will run into a situation where enabling a new feature will require that the Cisco SDM (Switching Database Manager) template be changed. This template is actually a method Cisco uses to assign specific applications to specific TCAM resources.

Some routers will allow you to manually specify how much TCAM space you want to allocate to a specific feature. Others aren’t so nice. They make you choose from a few restrictive templates, which allocate the resources automatically based on a few predetermined settings. For example, on the Cisco 3750, we recently wanted to enable policy-based routing (PBR) to implement a layer 3 jail. The basic idea with template-only routers is that you have to choose where you want most of the optimizations, and compromise on the rest.

For this platform, there are four templates: default, routing, PBR, and VLAN. Each of these tries to allow for a bit more resources allocated to the specified task. For policy routing, we’d have to choose “routing” or “PBR,” which in turn limits the amount of unicast MAC addresses that can be held in TCAMs. Likewise, selecting a VLAN template will make PBR impossible, but allow for more VLAN database information to be held in TCAMs. There are always compromises when we need to use more advanced features. Keeping true with the spirit of router operating systems, there’s also some mysterious side-effects when a new template is chosen. On our specific router, if the PBR template is chosen, the router will become unable to support VPN routing/forwarding tables (VRF). The next unsightly gotcha is that with the IOS version that supports IPv6, you cannot even enable PBR. There is no template to allow both policy routing and IPv6.

Perhaps the main idea of TCAM allocation still isn’t clear. Just because, for example, 8K is allocated to routing tables, this doesn’t mean that you can only have a routing table of that size. There’s always the fallback of process switching. Process switching means that everything will be done by the processor instead of in hardware (TCAMs). Processor intervention is not desirable, mostly because it is much slower than hardware lookups. Also, the processor is supposed to be used for things like sending logs to a syslog server and controlling SSH sessions. If a router doing process switching gets really busy, it may be unable to service your console access attempts. Generally speaking, the more expensive the router, the less it will use the processor.

Hardware is finite, and we always need more. More expensive routers don’t always suffer from the constant struggle for TCAMs because they have enough to support most features that currently exist. Unfortunately, most companies won’t want to purchase the latest and greatest router with seemingly endless hardware resources unless they can justify the added cost by showing a current need for them. So, most of us are stuck having to adjust TCAM allocations.

Further reading: an interesting blog from Plixer.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Subnetting – Slice Up 32-bits
2. Are Cisco Flex Links the End of STP?
3. Networking 101: Layer 2, Link and Spanning Tree
4. Networking 101: More Subnets, and IPv6
5. Networking 101: Understanding Layers

Managing Virtual Machine and Cloud Sprawl

Virtualization (in the cloud or locally) is great; that much we can all agree on. Virtual machines (VMs) can tend to grow out of control, however, now that it’s so easy to create them. This should not be all that surprising, but many small to medium businesses are also dabbling in VMs, and they are suddenly overwhelmed by the VM growth.

Each VM is another server that an administrator must manage. Security updates must be applied and global configuration changes now need to be propagated to all these new machines. While it’s easy to create 3-4 (or more) servers on one physical piece of hardware, you’ll certainly struggle if you aren’t already set up to scale.

Unfettered Growth
The number of physical machines in a small company may drop dramatically; maybe 40%, when virtualization is implemented. Unfortunately, the number of OS instances will generally increase by two-fold or more at the same time. The power and cooling savings are realized, as was promised by virtualization, but taking 20 servers to 12 servers, for example, will means you may soon have 40 OS instances to manage.

Puppet, from Reductive Labs

The reasons for VM proliferation depend on your culture, but the most common reason is that delegating control of an entire OS is easier than managing an application for customers. IT customer, be they engineers, application developers, or smaller IT units within an organization, frequently need more access then cenral IT is willing to give. The easy solution: give them a server of their own. Test environments, too, are best served by virtual machines.

To keep hardware (and power and cooling) costs down, many companies implement policies about the implementation of new services. New applications and servers need to be run on VMs first, unless it’s really requires its own server. Policies such as these are good, in that they limit wastefulness, but they do tend to exacerbate VM sprawl.

Sprawl aside; it’s worth noting that higher utilization levels on your servers does not mean that they’ll use an appreciably larger amount of power. In fact, the power savings claims are really true, and can be even greater if your utilization is low and you use VirtualCenter’s power management features. VMWare can migrate VMs to fewer servers if utilization isn’t high enough, and actually power off unnecessary servers. This works best with Dell hardware, but other large vendors are supported as well. Imagine: all your VMs migrating to a few blades in a blade server during the nighttime, and then as utilization increases during the day, blades quickly boot up and take the load as needed. Granted, I don’t personally know any enterprise environments that are brave enough to try it yet, but in theory the concept is wonderful.

Dealing
Something magical happens when a company grows to around 50 operating systems. It’s too many to manage by simply logging in and running commands, so people start to write scripts. In Windows land, if it hasn’t already happened, you must implement Active Directory. For the Unix/Linux servers, configuration management becomes even more important. Writing a script that SSH’s to each server and runs a command doesn’t scale, no matter how hard people want it to. You need a real configuration management system (such as puppet or cfengine) to ensure that servers are configured exactly how you want, and that they will remain that way.

If you already operate in a large environment with good automated installations and configuration management systems, chances are scaling 100-fold won’t be a problem. Barring scaling issues with the management software itseld, that is. A good network-booting deployment system is only half the battle, because every server isn’t going to be configured identically. If you’re “doing it right,” you should be able to arbitrarily reinstall any server, walk away, and know that it’ll come back up patched and running all the services it’s supposed to. Servers, or rather the OS that runs on them, should be truly disposable.

Management of a “golden image” is promised by VMWare, probably because ITIL mentions it, but it doesn’t really help in practice. You have to create your images (somehow). There’s no mechanism to update a golden image with security patches and apply them to existing systems; you’ll generally have to reinstall the OS instances. And that’s what you should do periodically, but without some kind of configuration management system, you’ll also be manually installing and configuring the services that the VMs used to provide in order to restore service functionality.

VM growth, therefore, is no different from server growth. It may be easier and cheaper, but from the OS management viewpoint, you’re doing the same thing. Likewise, the availability of your services is also in danger. Running five VMs on a single piece of hardware means that a hardware failure takes out five servers instead of one. VMWare and Xen can both be clustered and run from shared storage, such that a hardware failure will result in the VMs immediately (instantly, even) being migrated to other servers. The problem is that VMotion requires the most expensive VMWare license, and a VirtualCenter server. Shared storage isn’t as big of an issues these days with iSCSI, but its still another aspect that must be configured. We’ll cover this issue in-depth in a future article, focusing on Xen and RHEL Clustering Services.

The point is: dealing with VM sprawl is no different than dealing with scaling up to support more physical servers. Use whatever mechanisms are available on your given platforms, and “do it right.” A VM is, and always will be, just another server.

Feel free to share!

Hide Sites

Related posts:

1. Understanding Linux Virtual Memory
2. How Much Server do you Need?
3. Zenoss: We Can Ditch Nagios Now
4. The Perils of Sudo With User Passwords
5. Is Cheap Web Hosting Worth It?

Manage Devices and Configurations with Cisco SDM

Ever wanted to make something “just work” in a secure and reliable way? We, too, have often thought that common configurations should just be selectable. The Cisco Security Device Manager(SDM) is a Java-based Web application for managing Cisco devices. It implements many management features aside from just security-related tasks, and it’s quite interesting. In this article we’ll explain what it can do, and why you might want to take it for a test drive.

Network admins can use SDM to generate Cisco TAC approved configurations with the click of a few buttons. It’s not just limited to simple configurations either. Some tricky configuration tasks such as QoS and VPNs also become easier with the SDM because it ensure that configuration errors don’t exist. In short, you can deploy new devices and services much quicker by using the SDM.

As the name implies, SDM also intently focuses on security. A feature called “one-click lockdown” will set your router up as Cisco recommends—a good starting point for new routers. Also, the security audit function of the SDM will check your configuration and offer up a surprisingly large set of recommendations for hardening security. Many are things that most administrators don’t worry about, but with the SDM you can easily click “fix it” for each item after reading a description. There’s no reason to leave any possible vulnerability open when you have a quick, easy GUI manager pointing out what should change.

Cisco SDM user interface

The SDM is also a management console that gives you a real-time look at your device. It provides a nice interface for viewing system logs, firewall logs, and even real-time performance statistics. You probably already gather performance data via SNMP for historical charting, but being able to see the real-time information while you’re logged into the device manager, where you can also make changes to the configuration, is quite convenient.

SDM is available for most IOS-based routers running 12.2 and above. It is install by downloading a zip file from Cisco and copying it to the router’s flash memory. It’s then accessed from your Web browser (Firefox or IE required, as well as certain Java versions).

Making it Work

First, we must point out that using the SDM requires that you enable the HTTP server on your device. Yes, most Cisco security holes involve the Web server, and yes, a Web spider can easily DoS your router if it starts crawling Web pages and runs it out of RAM. Fortunately, both of these are negligible if you don’t allow access to the Web server from external networks. So first things first, enable: ip http secure-server, then configure ACLs to limit access properly.

After unzipping the file downloaded from Cisco, you can browse to: https://$server/flash/sdm.shtml

Then, login with a highly privileged account (level 15 is required). Up comes the Java applet, and you’re in! It couldn’t be easier than that.

Features

At the top, you’ll see things like Wizard, Advanced, and Monitor. The left had side lists things you can do in Wizard mode, and includes things such as VPN, Firewall, and LAN configuration options.

At the top you’ll also see a “deliver” button, which is another way of saying “commit.” All changes made within the SDM are committed to flash and merged into the running configuration when deliver is clicked.

Various configuration menus exist, most of which make the task at hand slightly easier. For the advanced administrator, it means you can just select options quickly without remembering the specific syntax. More junior admins can make previously confusing concepts work with little effort as well, and then look at the configuration that was generated.

The neatest feature is the security audit. When run, it will gather information about your device and then provide a list of problems. A nice “fix it” check box next to each item can be clicked, or you can elect to choose “fix all.” Beware that Cisco’s idea of security is basically very locked down. Selecting “fix all,” for example, will disable SNMP. It’s true that exposing SNMP to the external world is unwise, but you really do need it enabled for internal access.

You can also configure ACLs and interface parameters from within the GUI. Interfaces can be configured completely via the SDM, and the really nice part is that it lists all available setting for the particular interface. You’ll see check boxes for every option, along with a nice description of each option. ACLs can also be configured, and the GUI presents a nice view of which services will be allowed, and in which direction, on each interface.

In advanced mode, you can easily change many things, including OSPF and BGP settings. It’s just a matter of a few clicks to add another OSPF process ID or add another network to an existing one. Being able to see networks each OSPF process advertises and configure passive interfaces in a single well laid out window is very exciting.

In Monitor mode, you can see which interfaces are down, how much CPU is being utilized, and how much RAM is being taken up by which processes. Very useful information, sure to put a smile on your face the first time you see it.

The SDM does not support everything you’d want to do on a router, but the majority of common tasks are covered. It’s definitely a time-saver, learning tool, and convenience crutch all in one. Don’t feel bad using the SDM; convenience always outweighs prestige, assuming you can do it via the command line too. Enable the “show changes before delivering config” option to see what commands the SDM is about to run, and you’ll avoid surprises and possibly learn something at the same time.

Feel free to share!

Hide Sites

Related posts:

1. Built-in Security with Cisco IPS
2. Cisco AutoQoS: VoIP QoS for Mere Mortals
3. Are Cisco Flex Links the End of STP?
4. Multi-user Security in Linux
5. What the Heck is a TCAM?

Understanding Linux Virtual Memory

Virtual memory is one of the most important, and accordingly confusing, piece of an operating system. Understanding the basics of virtual memory is a requisite to understanding operating system performance. Beyond the basics, a deeper understanding allows a systems administrator to interpret system profiling tools better, leading to quicker troubleshooting and better decisions.

The concept of virtual memory is generally taught as though it’s only used for extending the amount of physical RAM in a system. Indeed, paging to disk is important, but virtual memory is used by nearly  every aspect of an operating system.

In addition swapping, virtual memory is used to manage all pages of memory, which incidentally are required for file caching, process isolation, and even network communication. Anything that queues data, you can be assured, traverses the virtual memory system. Depending on a server’s role, virtual memory functionality may not be optimal. An administrator can dramatically improve overall system performance by adjusting certain virtual memory manager settings.

To optimally configure your Virtual Memory Manager (VMM), it’s necessary to understand how it does its job. We’re using Linux for example’s sake, but the concepts apply across the board, though some slight architectural differences will exist between the Unixes.

Nearly every VMM interaction involves the MMU, or Memory Management Unit, excluding the disk subsystem. The MMU allows the operating system to access memory through virtual addresses by using data structures to track these translations. Its main job is to translate these virtual addresses into physical addresses, so that the right section of RAM is accessed.

The Zoned Buddy Allocator interacts directly with the MMU, providing valid pages when the kernel asks for them. It also manages lists of pages and keeps track of different categories of memory addresses.

The Slab Allocator is another layer in front of the Buddy Allocator, and provides the ability to create cache of memory objects in memory. On x86 hardware, pages of memory must be allocated in 4KB blocks, but the Slab Allocator allows the kernel to store objects that are differently sized, and will manage and allocate real pages appropriately.

Finally, a few kernel tasks run to manage specific aspects of the VMM. The bdflush manages block device pages (disk IO), and kswapd handles swapping pages to disk. Pages of memory are either Free (available to allocate), Active (in use), or Inactive. Inactive pages of memory are either dirty or clean, depending on if it has been selected for removal yet or not. An inactive dirty page is no longer in use, but is not yet available for re-use. The operating system must scan for dirty pages, and decide to deallocate them. After they have been guaranteed sync’d to disk, an inactive page my be “clean,” or ready for re-use.

Tuning
Tunable parameters may be adjusted in real-time via the proc fils system, but to persist across a reboot, /etc/sysctl.conf is the preferred method. Parameters can be entered in real-time via the sysctl command, and then recorded  in the configuration file for reboot persistence.

You can adjust everything from the interval pages are scanned to the amount of memory to reserve for pagecache use. Let’s see a few examples.

Often we’ll want to optimize a system for IO performance. A busy database server, for example, is generally only going to run the database, and it doesn’t matter if the user experience is good or not. If the system doesn’t require much memory for user applications decreasing the available bdflush tunables is beneficial. The specific parameters being adjusted are just too lengthy to explain here, but definitely look into them if you wish to adjust the values further. They are fully explained in vm.txt, usually located at: /usr/src/linux/Documenation/sysctl/vm.txt.

In general, a IO-heavy server will benefit from the following setting these values in sysctl.conf:
vm.bdflush=”100 5000 640 2560 150 30000 5000 1884 2”


The pagecache values control how much memory is used for pagecache. The amount of pagecache allowed translates directly to how many programs and open files can be held in memory.

The three tunable parameters with pagecache are:

• Min: the minimum amount of memory reserved for pagecache
• Borrow: the percentage of pages used in the process of reclaiming pages
• Max: percentage at which kswapd will only page pagecache pages; once it falls below, it can swap out process pages again

On a file server, we’d want to increase the amount of pagecache available, so that data isn’t moved to disk as often. Using vm.pagecache=”10 50 100″ provides more caching, allowing larger and less frequent disk writes for file IO intensive work loads.

On a single-user machine, say your workstation, large number will keep pages in memory, allowing programs to execute faster. Once the upper limit is reached, however, you will start swapping constantly.

Conversely, a server with many users that frequently executes many different programs will not want high amounts of pagecache. The pagecache can easily eat up available memory if it’s too large, so something like vm.pagecache=”10 20 30” is a good compromise.

Finally, the swappiness and vm.overcommit parameters are also very powerful. The overcommit number can be used to allow more memory allocation than RAM exists, which allows you to overcommit the amount of pages. Programs that have a habit of trying to allocate many gigabytes of memory are a hassle, and frequently they don’t use nearly that much memory. Upping the overcommit factor will allow these allocations to happen, but if the application really does use all the RAM, you’ll be swapping like crazy in no time (or worse: running out of swap).

The swappiness concept is heavily debated. If you want to decrease the amount of swapping done by the system, just echo a small number of the range 0-100 into: /proc/sys/vm/swappiness. You don’t generally want to play with this, as it its more mysterious and non-deterministic than the advanced parameters described above. In general, you want applications to swap to avoid them using memory for no reason. Task-specific servers, where you know the amount of RAM and the application requirements, are best suited for swappiness tuning (using a low number to decrease swapping).

These parameters all require a bit of testing, but in the end, you can dramatically increase the performance of many types of servers. The common case of disappointing disk performance stands to gain the most: give the settings a try before going out and buying a faster disk array.

Feel free to share!

Hide Sites

Related posts:

1. Multi-user Security in Linux
2. Managing Virtual Machine and Cloud Sprawl
3. Understanding Fibre Channel
4. Networking 101: Understanding Layers
5. Back to Basics: Unix System Stats Utilities

Built-in Security with Cisco IPS

Another new feature available in IOS (12.3) is Cisco’s Intrusion Prevention System. An IDS has been part of IOS for a long time, but they recently took it a step further. As part of its Self-Defending Network campaign, Cisco realized that an IPS should be integrated into the network fabric. We’ll explain what this means, and show you how to implement it.

Actively preventing the attack makes it an IPS. The standard old IDS solution means that it can detect and alert, but blocking attacks is not normally part of an IDS’s feature set. Thus, if you want to prevent attacks rather than just receive alerts, you need an IPS. Cisco’s IPS works like any other: you get a signature file, called the Signature Definition File (SDF) by Cisco, and if the IPS finds that a packet matches a signature, it’s blocked.

There are appliances, Catalyst switch modules, and router modules, but IPS is also built-in to certain IOS images now. Since Cisco claims IPS features won’t impact router performance (since the latest release), it may be possible to skip the purchase of a dedicated module for IPS.

The catch, of course, is that an IPS is not robust without constant signature updates. Attacks are constantly evolving, and without an update you aren’t protected against the latest and greatest attacks. Something completely new could sneak in, but the idea is that after the first few attacks Cisco will update the SDF and you’ll be notified that it’s time to download a new version. That’s right, you have to manually download and install a new signature file. This requires a subscription service above and beyond what you pay for SMARTnet. Services for IPS, as it’s called, provides SDF updates and the other features (support, warranty) that SMARTnet does as well. Accordingly, your SMARTnet contract is discounted when you purchase a Cisco Services for IPS contract, according to Cisco’s Q&A documentation.

Configuring IPS for Sensor Modules

There are many different cases for configuring IPS depending on your device. First, we’ll show you how to enable it on any IPS sensor module that uses the IPS 5.1 or later, then we’ll show you how to take advantage of the IOS built-in default IPS features.

The IDS Device Manager (IDM) is a graphical interface for configuring all IDS (and IPS) functionality. If you prefer that, then refer to the Cisco documentation after reading about how it’s done via the CLI here.

The general idea we’re working with here is called the VLAN pair method. This means that we’ll configure two VLANs in a pair group, and all traffic received by a sensor will be inspected and either forwarded on to the other VLAN, or dropped. Up to 255 VLAN pairs can be configured on most sensors.

First we enter configuration mode, then the service interface, and finally select the physical interface that we wish to configure:

sensor#configure terminal

sensor(config)#service interface

sensor(config-int)#physical-interfaces GigabitEthernet0/1

Next, we must configure the VLAN pair (and give it a meaningful description):

sensor(config-int-phy)#subinterface-type inline-vlan-pair

sensor(config-int-phy-inl)#subinterface 1

sensor(config-int-phy-inl-sub)#vlan1 10

sensor(config-int-phy-inl-sub)#description vlans 10 and 11

Conceptually, the interface will now be added to a virtual sensor, and once it’s enabled it will monitor traffic. We now need to enable a virtual sensor:

sensor(config)#service analysis-engine

sensor(config-ana)#virtual-sensor vs0

Once that’s completed, we simply add the previously-defined subinterface to the sensor, and we’re done:

sensor(config-ana-vir)#physical-interface GigabitEthernet0/2 subinterface-number 1

Configuring IPS for IOS

You can enable IPS features in IOS using the default SDF. Signatures may be added manually to the SDF, or you can pay Cisco for the latest signatures.

First we need to enable what’s called Security Device Event Exchange notifications:

router(config)#ip ips notify sdee

Then we must configure an IPS rule name that will be used for associating with interfaces.

router(config)# ip ips name MYIPSRULES

The next step is to specify where the SDF file will come from. The following command specifies that the file 256MB.sdf can be found in flash memory. You can also specify tftp or any other protocol your Cisco knows how to handle, but it’s best to use flash memory to ensure no dependencies on other servers.

router(config)# ip ips sdf location flash:256MB.sdf

Finally, we simply enable IPS on the interface (in both directions). It is also a good idea to enable IP reassembly on the interface, so that the IPS rule can evaluate entire IP packets at once.

router(config)#interface fastEthernet 0

router(config-if)#ip ips MYIPSRULES in

router(config-if)#ip ips MYIPSRULES out

router(config-if)#ip virtual-reassembly

Now you have a working IPS, based on the file in your flash called 256MB.sdf. That file must be downloaded from Cisco using your CCO login linked to a valid support contract.

The Power of Community

If you don’t feel like paying Cisco for signature updates, you can update the SDF yourself. When a new attack surfaces, you’ll often find Cisco IPS XML signatures posted to various online forums. You can and should use them.

To view your current SDF version, you can run: sh ip ips signatures

To merge the IPS SDF configuration with new information, you can copy in an XML file. Just like copying in any configuration snippet, the updates will be merged, not replaced. Say we got sigs.xml from a helpful network operator. To enable these signatures, we simply run:

router#copy tftp://serer.fqdn/sigs.xml ips-sdf

That’s it! You’ll see that 256MB.sdf on the flash memory is now a bit larger. It’s a good idea (and is recommended by Cisco) to rename 256MB.sdf to avoid confusion, now that you are no longer running a Cisco-sanctioned version.

Enabling IPS on supported routers is quite easy, but can lead to some interesting troubleshooting sessions. Be sure you have a syslog server that your routers all log to: it will save hours of work. Also, search around; you may find a source for XML updates that you wish to trust, and then it’s pretty easy to automate daily merges into your local SDF.

Feel free to share!

Hide Sites

Related posts:

1. Manage Devices and Configurations with Cisco SDM
2. Are Cisco Flex Links the End of STP?
3. Cisco AutoQoS: VoIP QoS for Mere Mortals
4. Multi-user Security in Linux
5. What the Heck is a TCAM?

Zenoss: We Can Ditch Nagios Now

Another perfect example of open source software gone commercial is Zenoss. As a full-featured network and service monitoring solution, Zenoss is one of the best monitoring tools available.

Most importantly, Zenoss combines two functionalities. First and foremost an enterprise environment requires host and service monitoring, with notifications. Network monitoring really means checking services, checking that hosts are up (they ping), and possibly writing your own plugins to check various other aspects of a server or network device. Until now, Nagios has filled that role.

Second, once a decent monitoring solution is in place, getting time-based information becomes desirable. Memory and CPU usage is the most prevalent example: if you’re checking available swap space every so often with Nagios, you may know when you start running low. But it may be just as important to see a graph of the last week’s usage. Tools like Cacti or Munin, which collect data frequently and use RRD graphs to display it, are very useful.

Zenoss fills both roles, without the annoying shortcomings prevalent in the alternative solutions. Zenoss uses the terms Availability Monitoring and Performance Monitoring to describe these two fundamental roles.

Performance of monitoring tools is important, and often times overlooked until it becomes a debilitating problem. For example, if you want to chart pretty RRD graphs of systems statistics like available RAM or disk space, Munin is an option. Unfortunately it’s all Perl, and designed in such a way that prevents it from scaling to even moderate amounts of hosts. Cacti is a bit better, but monitoring close to 100 hosts is painful with either option. Along comes Zenoss.

Zenoss is written in Python, and uses a MySQL backend for storage, and by all accounts it appears to perform very well. The really great thing about corporate-backed open source is quality control. The community simply isn’t responsible enough to say, “No, this won’t work, re-implement it.” A company with QA is.

Speaking of features, Zenoss isn’t missing many. Flexibility seems to be top priority–it can monitor hosts with SNMP, Nagios agents, SSH, Windows WMI, and various other mechanisms. Many features they claim are a bit over-inflated, such as ZenPing (marketed as Network Topology Monitoring) but the feature set is rich nonetheless.

Zenoss’s primary functions involve four features:

• Inventory Tracking
• Availability Monitoring
• Performance Monitoring
• Event Monitoring and Management

Inventory tracking claims some sort of “configuration” reporting as well, but it seems very limited. Zenoss will discover your inventory and auto-populate a database. This is great for knowing which IP addresses are in use, for example, but means that “configuration” reporting is limited to an outside observer’s perspective. It can tell you which servers have a Web server running, but it certainly doesn’t deal with the configuration of the Web server. Of course, inventory tracking isn’t limited to automatically discovered information; there are manual input capabilities too.

Availability monitoring is basically Nagios, plus. It can ping, it can monitor Windows machines, and it can pretty much do whatever you need. Even your old Nagios plugins will work with Zenoss. It does generate reports, but much better ones than Nagios is capable of.

Host monitoring, performance monitoring, or whatever you’d like to call it, is quite robust in Zenoss. Some would think it’s light on features, but there’s a good reason that Zenoss requires you use SNMP: it’s much more scalable than SSH’ing to each server every minute. A bit of up-front configuration is required, in that all your hosts will need SNMP configured and working, but it’s completely worth it. Zenoss too uses RRD graphs, and it can generate events and alerts based on pre-defined thresholds.

Finally we come to event monitoring. Zenoss is also encroaching on Splunk‘s territory a bit. It can combine syslog, availability monitoring alerts, SNMP traps, and even Windows event log data. Much like Splunk, Zenoss correlates similar events for easier viewing and troubleshooting. This is the portion that processes all events and generates alerts to pagers or e-mail, taking into account the escalation procedure you’ve defined.

To top it all off, the Zenoss Web interface is top-notch. It includes a customizable “dashboard” for monitoring, and everything is AJAX-enabled. AJAX provides the user experience similar to Splunk and Google’s Gmail.

Marketing fluff aside, Zenoss really does provide a wonderful product. It is, of course, open source and available for free.

At last year’s LISA conference, Zenoss gave a demonstration that sadly coincided with free beer time. Stumbling in toward the end, I demanded one of their free baseball caps, and sat to listen to the last few audience questions. One thing was very obvious: everyone in the room was excited about this product. If hardcore sysadmins are excited, you know this is something worthwhile.

Zenosss is very functional and full of features. It may even be possible to replace three separate pieces of software with this one product: host inventory database, Nagios, and your performance monitoring tool of choice. Maybe even Splunk some day. We can’t wait to see what features they will be adding next.

Feel free to share!

Hide Sites

Related posts:

1. Managing Virtual Machine and Cloud Sprawl
2. Squeeze Your Gigabit NIC for Top Performance
3. Back to Basics: Unix System Stats Utilities
4. The Perils of Sudo With User Passwords
5. Built-in Security with Cisco IPS

Squeeze Your Gigabit NIC for Top Performance

Many new workstations and servers are coming with integrated gigabit network cards nowadays, but quite a few people soon discover that they can’t transfer data much faster than they did with 100 Mb/s network cards. Multiple factors can affect your ability to transfer at higher speeds, and most of them revolve around operating system settings. In this article we will discuss the necessary steps to make your new gigabit enabled server obtain close to gigabit speeds in Linux, FreeBSD, and Windows.

Hardware considerations

First and foremost we must realize that there are hardware limitations to consider. Just because someone throws a gigabit network card in a server doesn’t mean the hardware can keep up. Network cards are normally connected to the PCI bus via a free PCI slot. In older workstation and non server-class motherboards the PCI slots are normally 32 bit, 33MHz. This means they can transfer at speeds of 133MB/s, but since it is a shared bus between many parts of the computer, realistically it’s limited to around 80MB/s in the best case. Gigabit network cards are 1000Mb/s, or 125MB/s. If the PCI bus is only capable of 80MB/s this is a major limiting factor for gigabit network cards. The math works out to 640Mb/s, which is really quite a bit faster than most gigabit network card installations, but remember this is probably the best-case scenario. If there are other hungry data loving PCI cards in the server, you’ll likely see much less throughput. The only solution for overcoming this bottleneck is to purchase a motherboard with a 66MHz PCI slot, which can do 266MB/s. Also, the new 64 bit PCI slots are capable of 532MB/s on a 66MHz bus. These are beginning to come standard on all server-class motherboards.

Assuming we’re using decent hardware that can keep up with the data rates necessary for gigabit, there is now another obstacle – the operating system. For testing, we used two identical servers: Intel Server motherboards, Pentium 4 3.0 GHz, 1GB RAM, integrated 10/100/1000 Intel network card. One was running Gentoo Linux with a 2.6 SMP kernel, and the other is FreeBSD 5.3 with an SMP kernel to take advantage of the Pentium 4’s HyperThreading capabilities. We were lucky to have a gigabit capable switch, but the same results could be accomplished by connecting both servers directly to each other.

Software considerations

For testing speeds between two servers, we don’t want to use FTP or anything that will require data be fetched from disk. Memory to memory transfers are a much better test, and many tools exist to do this. For our tests, we used ttcp (http://www.pcausa.com/Utilities/pcattcp.htm).

The first test between these two servers was not pretty. The maximum rate was around 230 Mb/s, about two times as fast as a 100Mb/s network card. This is an improvement, but far from optimal. In actuality, most people will see even worse performance out of the box. However, with a few minor setting changes, we quickly realized major speed improvements – more than a threefold improvement over the initial test.

Many people recommend setting the MTU of your network interface larger. This basically means telling the network card to send a larger sized Ethernet frame. While this may be useful when connecting two hosts directly together, it becomes less useful when connecting through a switch that doesn’t support larger MTUs. At any rate, this isn’t necessary. 900Mb/s can be attained at the normal 1500 byte MTU setting.

For attaining maximum throughput, the most important options involve TCP window sizes. The TCP window controls the flow of data, and is negotiated during the start of a TCP connection. Using too small of a size will result in slowness, since TCP can only use the smaller of the two end system’s capabilities. It is quite a bit more complex than this, but here’s the information you really need to know:

For both Linux and FreeBSD we’re using the sysctl utility. For all of the following options, entering the command ‘sysctl variable=number’ should do the trick. To view the current settings use: ‘sysctl ’

Maximum window size:
FreeBSD:
kern.ipc.maxsockbuf=262144
Linux:
net.core.wmem_max=8388608

Default window size:

FreeBSD, sending and receiving:
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
Linux, sending and receiving:
net.core.wmem_default = 65536
net.core.rmem_default = 65536

RFC 1323:
This enables the useful window scaling options defined in rfc1323, which allows the windows to dynamically get larger than we specified above.
FreeBSD:
net.inet.tcp.rfc1323=1
Linux:
net.ipv4.tcp_window_scaling=1

Buffers:
When sending large amounts of data, we can run the operating system out of buffers. This option should be enabled before attempting to use the above settings. To increase the amount of “mbufs” available:
FreeBSD:
kern.ipc.nmbclusters=32768
Linux:
net.ipv4.tcp_mem= 98304 131072 196608

These quick changes will skyrocket TCP performance. Afterwards we were able to run ttcp and attain around 895 Mb/s every time – quite an impressive data rate. There are other options available for adjusting the UDP datagram sizes as well, but we’re mainly focusing on TCP here.

Windows XP / 2000 Server / Server 2003

The magical location for TCP settings in the registry editor is:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

We need to add a registry DWORD named TcpWindowSize, and enter a sufficiently large size. 131400 (make sure you click on decimal) should be enough.

Tcp1323Opts should be set to 3. This enables both rfc1323 scaling and timestamps.

And similarly to Unix, we also want to increase the TCP buffer sizes:
ForwardBufferMemory 80000
NumForwardPackets 60000

One last important note for Windows XP users needs to be made. If you’re installed service pack 2, then there is another likely culprit of poor network performance. Explained in knowledge base article 842264, Microsoft says that disabling Internet Connection Sharing after an SP2 install should fix performance issues.

The above tweaks should enable your sufficiently fast server to attain much faster data rates over TCP. If your specific application makes significant use of UDP, then it will be worth looking into similar options relating to UDP datagram sizes. Remember, we obtained close to 900Mb/s with a very fast Pentium 4 machine, server-class motherboard, and quality Intel network card. Results may vary wildly, but adjusting the above settings are a necessary step toward realizing your server’s capabilities.

Feel free to share!

Hide Sites

Related posts:

1. Zenoss: We Can Ditch Nagios Now
2. What the Heck is a TCAM?
3. How Much Server do you Need?
4. Understanding Linux Virtual Memory
5. Back to Basics: Unix System Stats Utilities

Networking 101: IP addresses

Networks don’t work without addresses: whenever you are sending something, you need to specify where it should go, and where it came from. To be an effective network engineer or administrator, you need to understand IP addresses backwards and forwards: you need to think be able to think on your feet. If something breaks, likely as not it’s some address assignment has been screwed up. And spotting the problem quickly is likely to be the difference between being the hero, or the guy who “takes a long time to fix the problem.” Before covering subnetting in the next Networking 101 installment, we’d like to thoroughly explore IP addresses in their primal form. This is crucial to understanding subnets.

IPv4 Addresses and 32-bit Numbers
IP addresses are just 32-bit binary numbers, but they’re important binary numbers: you need to how to work with them. When working with subnet masks, new network administrators generally get confused with the ones they haven’t memorized. All the subnet mask amounts to is moving the boundary between the part of the address that represents a “network” and the part that represents a “host.” Once you’re comfortable with this method of thinking about IP addresses and masks, you’re mastered IP addressing.

Binary is quite simple. In binary the only numbers are zeros and ones, and a 32-bit number holds 32 zeros and ones. We’re all used to base-10 numbers, where each place in a number can hold any number from 0-9. In binary each place holds either a zero or a one. Here’s the address 255.255.255.0 in binary:

11111111.11111111.11111111.00000000

For convenience, network engineers typically break IP addresses into four 8-bit blocks, or octets. In an 8-bit number, if all of the bits are set to 1, then the number is equal to 255. In the previous address, 11111111 represents 255 and 00000000 represents zero.

The way binary really works is based on powers of two. Each bit represents a different power-of-two. Starting at the left-hand side, the most significant bit, numbers form in the following manner:

The result is additive, meaning that if all bits are set, you simply add the power-of-two value up for each place. For example, if we have an 8-bit number, 11111111 , we simply add: 2⁷ + 2 ⁶ + 2 ⁵ + 2 ⁴ + 2 ³ + 2 ² + 2 ¹ + 2 ⁰ = 255

Try a non-trivial example now: 11110000

We can see that four bits are “set” in the above 8-bit number. Summing the power-of-two values in those places yields: 2⁷ + 2⁶ + 2 ⁵ + 2 ⁴ = 240

It is just that simple. If you can convert a binary number to decimal form, you can easily figure out subnet masks and network addresses, and we’ll show you how in the next issue of Networking 101.

Focusing on 32-bit IPv4 addresses themselves now, there are a few different types that need to be understood. All IP addresses can be in the range 0.0.0.0 to 255.255.255.255, but some have special uses.

Loopback:
Packets that will not leave the host–i.e. they will not traverse an external network interface. Example: 127.0.0.1

Unicast:
Packets that are destined for a single IP address. Example: 2.2.2.2

Multicast:
Packets that will be duplicated by the router, and eventually routed by multicast routing mechanisms. Example: 226.0.0.2

Limited Broadcast:
A broadcast packet, sent to every host, limited to the local subnet. Example: 255.255.255.255

Directed Broadcast:
Packets that are routed to a specific subnet, and then broadcast. Example, assuming we are not on this subnet: 1.1.1.255

There are also some special cases of IP addresses, including private and multicast addresses. Addresses in the range 224.0.0.0 – 239.255.255.255 are reserved for multicast. Everything below that range is fair game on the Internet, excluding addresses reserved by RFC 1918 and a few other special-purpose assignments. These “1918 addresses” are private addresses, meaning Internet routers will not route them. The ranges include:

• 10.0.0.0 -10.255.255.255
• 172.16.0.0 – 172.31.255.255
• 192.168.0.0 – 192.168.255.255

These IP addresses can be assigned locally to as many computers as you want, but before those computers access the Internet, the addresses must be translated to a globally routable address. This is commonly done via Network Address Translation ( NAT ). The 1918 addresses aren’t the only reserved spaces, but they are defined to be “site local.” Multicast also has a reserved range of addresses that aren’t designed to escape onto the Internet: 224.0.0.0 – 224.0.0.255 are multicast “link-local” addresses.To give the necessary background for the next issue of Networking 101, we need to make sure everyone understands the concept of a local subnet. Once we have assigned a valid IP address to a computer, it will be able to speak to the local network, assuming the subnet mask is configured properly. The subnet mask tells the operating system which IP addresses are on the local subnet and which are not. If an IP we wish to talk to is located on the local subnet, then the operating system can speak directly to it without using the router. In other words, it can ARP for the machine, and just start talking. IP address and subnet mask configuration is fairly straightforward for general /24 networks. The standard 255.255.255.0 mask means that the first three octets are the network address, and the last part is reserved for hosts. For example, a computer assigned the IP of 10.0.0.1 and a mask of 255.255.255.0 (a /24, or 24-bits if you write it out in binary) can talk to anyone inside the range 10.0.0.1-10.0.0.255.Be sure to digest everything here, because in part two we’ll get to the meat of subnetting with CIDR.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Subnetting – Slice Up 32-bits
2. Networking 101: More Subnets, and IPv6
3. Networking 101: Understanding Layers
4. Networking 101: Layer 2, Link and Spanning Tree
5. What the Heck is a TCAM?

Is Cheap Web Hosting Worth It?

Whether you are a small business, fortune 500, or in-between, Web hosting providers may have something to offer. Web hosting is extremely competitive, and not many companies survive. Once the need for Web hosting services is established, you need to do some research to determine whether a specific hosting company is a good fit.

Small businesses will depend more heavily on their hosting provider, as their entire Web infrastructure may reside at a hosting company. As businesses get larger, and acquire their own IT staff, they tend to start running their own Web servers. At some point, maybe due to cost-cutting measures, companies may revisit the hosting option. If it’s really only $5 per month, then why not?

Hosting companies also offer virtual machines that your IT staff can run. These are a great alternative to hosting DR servers in remote data centers. VM plans are more expensive than a regular shared Web hosting plan, but not much more. With both options we need to pay close attention to the hosting company’s track record and value-add services.

What to Expect

There are literally hundreds of popular hosting companies out there. Each has their own tools for managing your domains, and each offers their own unique set of value-add services.

At a minimum, there are a few things you should demand from any Web host. Due to the brutally competitive nature of the business and interchangeability of these services, you should never be shy about demanding a new feature or picking up and moving to a new provider. A few of the basic things you should expect are:

• A Web interface to manage your domains, billing, and user accounts
• Shell, SFTP, and various other methods of remote access
• Unlimited MySQL databses, and an easy way to create and manage them
• Easy, automated installation mechanisms for common open source Web applications
• Unlimited e-mail boxes and a few options for Web-based access
• Log analysis and detailed reporting
• And lately: unlimited disk space and bandwidth

Five or ten years ago you would not find most of these features with the most popular hosting companies. These days, however, they all offer most of these items and a lot more.

Companies that offer VM hosting, or Virtual Private Servers (VPS) as some call it, will provide a basic set of tools to manage your virtual machine. You can generally select from a few operating systems and reload it at any time. Afterwards, however, you’re on your own. You get the root password and are free to install whatever you need.

Is it Worth It?

Psychologically, most people have a problem purchasing something that appears too cheap. When looking at Web hosting services, like WordPress Web Host, the list of options leaves most people drooling. Then they see the price: sometimes less than $5 per month, for unlimited disk space, bandwidth, accounts, and with a free domain registration. How can they do that?

The servers must be horribly overloaded as these companies cram more and more customers onto them. At $60 per year per customer, the hosting company can’t exactly afford to buy top of the line servers, right?

The truth is, hosting is brutally competitive. Most companies fail. The ones that survive have a high level of automation to allow them to manage their servers with very little manhours. Even at today’s sub-$5 pricing for a monthly plan, that equates to one $3000 server per 50 customers, at their yearly rate. A properly configured Apache Web server environment should be able to handle 300 Web sites on today’s hardware without a problem.

Of course, someone could write a PHP script that goes wild and consumes way too many resources, slowing down the entire server. The hosting companies monitor the servers, and this type of thing is usually dealt with quickly. However, it is worthwhile to ask your chosen hosting company whether or not they allow adult content. Adult sites are generally very high-traffic, and if you’re site is sharing the same Web server, it may suffer performance issues.

The Case for Secondary Hosting

Enterprises should not rule out hosting providers. You don’t need to outsource your entire Web team, but part of a good disaster recovery strategy will certainly include multiple off-site resources. Instead of paying co-location fees and trodding your own servers to a datacenter across the country, why not get a few virtual machines in multiple locations?

If your physical server dies, you may be sending FTE to the remote site to replace the hardware. In a hosted VM environment, you never have to worry about taking employees away from their daily work and paying travel expenses. Just click a button in the hosting provider’s Web interface to create a new VM if something goes awry.

You can host your own Web servers, secondary DNS, or just about any other service your company relies upon. If you’re worried about reliability, you can always select two or three VM providers in separate parts of the country (or world). To equal the cost of one server in a co-location environment, you’d have probably 10-20 virtual machines spread throughout the world.

Next Steps

If you’re interested in hosting Web sites with a Web hosting provider or VMs, the next step is to do some research. Read reviews of various hosting companies, and pay attention to issues people cite. There will always be a certain amount of people unhappy with a service, but pay careful attention to what they are saying went wrong. Likewise, pay attention to the few positive reviews you might find. Most happy customers remain silent, so when you find a positive review it holds more weight than a rant.

“Cheap web hosting” is a great Google search. You will find a few unbiased sites that list providers, the services they offer, reviews, and pricing. We don’t want to make any recommendations, but just remember: more expensive is not always better in a fiercely competitive market such as Web hosting.

Feel free to share!

Hide Sites

Related posts:

1. Managing Virtual Machine and Cloud Sprawl
2. Zenoss: We Can Ditch Nagios Now
3. The Perils of Sudo With User Passwords