Connecting to existing buckets in S3 with boto, the right way

Here’s another interesting tidbit.

If you have scripts that connect to S3, and you run out of buckets (Amazon only allows 100 buckets per account), you might get a nasty surprise.

See, you may have been using create_bucket(name-of-bucket) to get your bucket object. It’s undocumented as far as I can see, but apparently if you use create_bucket() on an bucket that actually exists, it’ll return the Bucket object. That’s handy! Except it breaks if you’re unable to create more buckets (even though you aren’t really trying to create more). Sigh, so I refactored as such:

# old and busted: bucket = s3_conn.create_bucket(bucket_name)

# new hotness:
# iterate over Bucket objects and return the one matching string:
def find_s3_bucket(s3_conn, string):
    for i in s3_conn.get_all_buckets():
        if string in i.name:
            return i

Used as: bucket = find_s3_bucket(s3_conn, bucket_name)

There is likely a more elegant way, but hey this works.

Feel free to share!

Hide Sites

Related posts:

1. Finding instances by name with boto in python
2. LDAP: Understand the Protocol and Work With Entries

Finding instances by name with boto in python

OK, I know I need to blog more. Rather than think I don’t have anything useful to say, I’ll start adding quick entries of what-I-learned.

Random tidbits from today:
I got annoyed with EC2 failures, and having to manually terminate and redeploy instances today, so I finally worked on a script I’ve been meaning to write. One thing I had to figure out (which isn’t all that complex), is how to discover an instance by name.

If you tag an instance with the hostname you’re using in your deployment script, you don’t need to fumble in the AWS console to find an instance ID. Ever. I don’t find it acceptable to manually click around or run scripts to discover information that’s available from an API

So, to “find” the instance using python and boto (assume AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are defined in your shell environment):

import boto
ec2conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
my_fqdn = "example.com" # trailing part of my domain

Now, ‘instances’ can be iterated over to find instances with the name you desire. I wrote a little wrapper function to do this, and it returns an instance object (which I call instance.terminate() on, for this purpose). Code:

def find_instance_by_nametag(instances, name):
    # support short or full hostname usage
    if not my_fqdn in name:
        name = name + my_fqdn
    for i in instances:
        if "Name" in i.tags and name in i.tags['Name']:
            return i
    sys.exit("sorry, I couldn't find an instance with that name!")

Easy as that!

Feel free to share!

Hide Sites

Related posts:

1. Connecting to existing buckets in S3 with boto, the right way

Display current location on your web page using the SPOT GPS tracker

The SPOT GPS tracker is not only a lifesaver, but also a handy tool for motorcycle (ok, and other types) travelers. If you subscribe to the Track Progress service, you can tell the GPS device to send your location to SPOT via satellite every 10 minutes, and then export those tracks at a later time.

They even have an API from which you can fetch an XML document with all your current tracks! The unfortunate part, and the reason for this post, is that they only keep 30 days of GPS coordinates. It’s not a problem if you export your data to Spot Adventures and create an “adventure” — that will live forever. But if you wish to present “my current location” on your personal web page, for example, you’re out of luck. It will only work as long as you’ve used the track progress functionality within the last 30 days.

I guess you need to cache the last used location yourself. OK, this shouldn’t be hard. This is how to do it with a little perl script (download here, see in action in CharlieTracker here):

#!/usr/bin/perl
# Author: Charlie Schluting <charlie@schluting.com> (c) 2011
#
use XML::Simple;
#use strict.. heh, no, this barely works.
$CACHEFILE = "/home/charlie/lastspotlocation.txt";
$XML = "/home/charlie/spot.xml";
$JS = "/stash/www/charlierides.com/files/map.js";

The CACHEFILE is where I store the last known latitude and longitude. It’s mostly for reference, so that other things (aside from this script) can use it.

XML is the location I store the fetched XML from SPOT’s API.

JS is the file I write the javascript out to. OK, you probably don’t need to care about this part of the script.. but here’s the details: if you use joomla or wordpress, you might not want to enable PHP code execution or other evil things like that. So, in order to include a snippet of generated HTML, the only choice (as far as I know, aside from editing the DB where a “display html” module stores its data), is to write javascript to include another javascript file. So that’s what I do.. the script actually generates javascript that when run, will spit out html. If you don’t care about that and just want the map part, read on.

# hahaha, oh man..
`wget -q -O $XML http://share.findmespot.com/messageService/guestlinkservlet?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW`;

This part fetches the XML file from spot, using my shared page identifier.

# create object
$xml = new XML::Simple;
# read XML file
$data = $xml->XMLin("$XML");

# this is how we overcome spot's API not keeping >30 days. If they've aged out, do nothing (i.e. keep using the old data).
die("No messages found, totalCount is 0, ABORTING LIKE AN UGLY KID") unless $data->{totalCount} > 0;

# the first object is always the most recent:
$lat = $data->{message}->[0]->{latitude};
$long = $data->{message}->[0]->{longitude};

# just because (hey, what if something else wants to use this?)
open(FILE, ">$CACHEFILE");
print FILE $lat . "," . $long . "\n";
close(FILE);

And the rest (above) is pretty self explanatory. Using just this part of the script, you’ve overcome the annoying limitation that is SPOT Track Progress (losing your last known location). Your last known coordinates will be in CACHEFILE. For completeness, I will include the rest of this horrible hack I used:

# ugly shit that writes out javascript to write out html, to include a linked static google maps image
$googleoptions = "&zoom=8&size=140x152&sensor=false&maptype=hybrid";
$googlelink = "<a target=\"blank\" href=\"http://maps.google.com/maps?q=" . $lat . "," . $long . "+(charlie)&z=8&t=h\"\>";
$header = "<p>Current location (since the last GPS update):</p>";
$js = 'document.write(\''. $header .'\');' . '
document.write(\''. $googlelink .'\');' . '
document.write(\'<img src="http://maps.googleapis.com/maps/api/staticmap?center=' . $lat .
',' . $long . $googleoptions . '" />\');
document.write(\'</a>\');
document.write(\'<p>Or <a href="http://share.findmespot.com/shared/faces/viewspots.jsp?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW" 
target=blank>view all recent tracks.</a></p> \');
';
open(FILE, ">$JS");
print FILE $js;
close(FILE);

So, what this writes out (in HTML, finally), is an IMG tag embedding a static google map (because I’m using a small thumbnail), which links to the google maps page when clicked:

Map: http://maps.googleapis.com/maps/api/staticmap?center=45.42553,-122.52251&zoom=8&size=140×152&sensor=false&maptype=hybrid
Link target: http://maps.google.com/maps?q=45.42553,-122.52251+(charlie)&z=8&t=h

The parameters you can pass google maps are well documented, so I don’t need to rehash them here. You may even prefer to use the real maps API, rather than the static image one. That too, is well documented.

Run this from cron every 15 minutes, and you’ll always have your last checked-in GPS coordinates!

Feel free to share!

Hide Sites

No related posts found.

Are Cisco Flex Links the End of STP?

Cisco Flex Links gives network operators a simple, reliable, and more scalable method of layer 2 redundancy. The Spanning Tree Protocol (STP) is not destined for the scrap bin, but it will certainly fall out of favor with many enterprise networks.

Flex Links are a pair of layer 2 interfaces configured to act as a backup of each other. Configuring Flex Links is very simple, but it’s a manual process. Spanning tree can configure itself if you just enable it, albeit likely a sub-optimal configuration, but a working one nonetheless. Flex Links, on the other hand, require manual setup and layout of your layer 2 network. If you don’t want to leave anything to chance, then Flex Links are preferred over STP.

The benefits of FlexLinks include:

• simplicity, which equals stability.
• instant failover.
• rudimentary load balancing capabilities, so one link isn’t wastefully idle.
• load balancing works across switches in a stack, including port channels.

Flex Links’ primary operating mode is just like spanning tree: one on, one off. With per-VLAN spanning tree, a trunk port can have some VLANs enabled and some blocked at the same time, so on the surface it seems that STP is superior. In reality, you can configure Flex Links to load balance VLANs, and we’ll show you how shortly.

Configuration

Conceptually, you configure Flex Links by telling one link it’s the active link, and another that it’s the backup of that

Flex Links Design Map

primary (active) one. Without configuring VLAN load balancing, it will completely disable the backup, and if the active link goes down the backup will take over.

For example, to configure port gi1/0/1 as a active link, and gi1/0/2 as the backup, you’d run:

Switch# configure terminal
Switch(conf)# interface gigabitethernet1/0/1
Switch(conf-if)# switchport backup interface gigabitethernet1/0/2

That’s all there is to configuring the basic mode, which gets you failover but no load balancing. Before talking about load balancing, let’s take a look at preemption and “mac address-table move update.”

Preemption

Preemption, that is, the preferred port for forwarding traffic, is also configurable. This is most often used in combination with multiple links that have differing bandwidth capacities. If you wish to ensure that port 1, a primary port that has more bandwidth, will return to the active link when it comes back up, you would set:  interface preemption mode bandwidth andswitchport backup interface preemption delay. The delay is used to set the amount of time (in seconds) to wait before allowing port 1 to preempt port 2 and begin taking over traffic again.

MAC Address-Table Move Update

Enabling the MAC address-table move update feature allows for rapid convergence when a primary link goes down and the backup takes over traffic forwarding duties. Without this feature enabled, neighboring switches may continue to forward traffic for a short time to a dead port, since they have learned MAC addresses associated with that link.

When move update is enabled, the switch containing Flex Links will broadcast an update packet to let other switches know what happened, and they will in turn un-learn that false MAC address mapping.

On the switch with Flex Links, simply configure:

Switch(conf)# mac address-table move update transmit

All switches, including ones with Flex Links, need to receive these updates. This is not enabled by default, so you’ll need to run the following command on all of your devices:

Switch(conf)# mac address-table move update receive

To see the status and verify that “move update” is enabled, run: show mac address-table move update. Checking the status of your Flex Links is much the same: show interfaces [interface-id] switchport backup.

Load Balancing

Flex Links should be configured such that both ports are forwarding traffic at the same time. This way, you get load balancing in addition to redundancy. The limitation is that only one port can be forwarding a single VLAN at a time. If we have VLANs 1-200, we need to choose which VLANs are forwarded primarily through which port. The most simple configuration, ignoring traffic requirements, would be that VLANs 1-100 use port 1, and VLANs 101-200 use port 2.

Before we get into configuring preferred VLANs, let’s talk about multicast. Multicast, of course, becomes an issue with this type of setup. If a port passed an IGMP join, and the switch is part of a multicast group, when the port goes down the switch will no longer be able to receive multicast traffic for that group. The quick fix is to make both Flex Links always be part of learned groups, with the command: switchport backup interface gigabitEthernet 1/0/12 multicast fast-convergence.

Now, on to VLAN load balancing. It is quite easy; just specify which VLANs you prefer on which links:

Switch(config-if)#switchport backup interface gigabitEthernet1/0/2 prefer vlan 101-200.

If you have VLANs 1-200 on the switch, show interfaces switchport backup will show you:

Vlans Preferred on Active Interface: 1-100
Vlans Preferred on Backup Interface: 101-200

If a link goes down, VLANs that are preferred on that interface will be moved to the other link in the pair. Likewise, when a link returns to service, its preferred VLANs are blocked on the backup and returned to the preferred link.

Be sure to run show interfaces switchport backup detail to see the full status, including link speeds, preemption modes, the MAC address-table move update status.

In summary, the simplicity of Flex Links make it a better choice for carrier and core enterprise networks over the ubiquitous spanning tree protocol. Link-level redundancy is had via STP, but with Flex Links you have more control and better load balancing capabilities. This certainly means that it takes longer to configure since you are planning the layer 2 network manually, but when you need a stable no-surprises link-layer network, Flex Links are definitely the way to go.

Feel free to share!

Hide Sites

Related posts:

1. Built-in Security with Cisco IPS
2. Cisco AutoQoS: VoIP QoS for Mere Mortals
3. Manage Devices and Configurations with Cisco SDM
4. Networking 101: Layer 2, Link and Spanning Tree
5. What the Heck is a TCAM?

Cisco AutoQoS: VoIP QoS for Mere Mortals

WANs often need Quality of Service (QoS) configured to ensure that certain traffic is classified as “more important” than other traffic. Until now, it took a serious Cisco guru to configure a network properly for VoIP if the network was at all bandwidth constrained. AutoQoS, a new IOS feature for Cisco routers, makes deploying VoIP easy, even on busy WAN links. In this article we’ll cover the basics, what AutoQoS does, and some of its limitations.

The first whack at AutoQoS was Cisco recognizing the need to simplify VoIP traffic prioritization. VoIP is especially sensitive to any latency, jitter, or loss, and users will notice problems. To ensure the best possible VoIP call, the network must ensure that lower priority traffic does not interfere with time-sensitive VoIP. AutoQoS can be enabled on both WAN links and Ethernet switches to automatically provide a nice best-practices based template for VoIP prioritization. If you’re lucky enough to have metro Ethernet service, like AT&T ethernet service for example, you should contact your provider to find out if QoS settings on your switches can be duplicated through theirs.

How it Works

QoS allows a router to classify which types of traffic are most important, and ensure that that traffic passed as quickly as possible. If necessary, other traffic will be queued until the higher priority traffic has had a chance to pass. Before a router can know when to queue versus when to attempt to pass all traffic, it must be configured with bandwidth settings for each link.

Configuring QoS on a Cisco router normally involves a complex series of interactions, which require understanding not only the protocols, but a router’s strange way of associating policies. The basic steps are:

• Use an ACL to define which traffic gets matched
• A class-map classifies matched traffic into classes
• A policy-map assigns priorities to the classes
• The policy-map is applied to the interface, which enables the processing of all packets through the ACL, class-map, and policy-map

Each of these “maps” are quite complicated and prone to error. Most sites are going to be duplicating effort because of common problems, like VoIP, needing QoS help.

Why AutoQoS

QoS configuration is not simple. It requires understanding the protocols your network interfaces are using, as well as the type of data you’re passing. To configure QoS for VoIP, for example, you must understand how VoIP works. In short, it requires a guru. If you’re like me, you literally giggled out loud the first time you encountered the word, “AutoQoS.”

AutoQoS enables any network administrator to just “turn on” a solid solution for ensuring VoIP is happy. VoIP is the pain point for most organizations, so that’s what Cisco focused on first, and that’s what we’re focusing on here. Given the limited scope of AutoQoS, it’s believable that it works well enough. In reality, QoS configurations generally classify many types of traffic, and then place a priority on each one.

The main benefit of AutoQoS is that administrator training is much quicker. It also means that VoIP deployments often go much smoother, and upgrading WAN links isn’t usually required. Finally, AutoQoS creates templates that can be modified as needed and copied elsewhere for deployment.

Limitations

Before talking about how to enable AutoQoS, which is literally three commands, let’s talk about where this works best, and what’s required to use AutoQoS.

First and foremost, you can only configure AutoQoS on a few types of router interfaces. These interfaces include:

• PPP or HDLC serial interfaces
• ATM PVCs
• Frame Relay (point-to-point links only)

Cisco catalyst switches also support an AutoQoS command to prioritize Cisco VoIP phones, but you cannot prioritize (using AutoQoS) generic VoIP protocols.

Next, there are some limitations with ATM sub-interfaces. If you have a low-speed ATM link (less than 768Kbps), then AutoQoS will only work on point-to-point sub-interfaces. Higher speed ATM PVCs are fully supported though. For standard serial links, AutoQoS is not supported at all on sub-interfaces. A quick litmus test to see if AutoQoS will work on your desired interfaces or not is to verify that the service-policy configuration is supported. If not, you’ll probably have to reconfigure some links.

AutoQoS will not work if an existing QoS configuration exists on an interface. Likewise, when you disable the AutoQoS configuration, any changes you may have made to the template after the initial configuration will be lost.

Bandwidth statements are used by AutoQoS to determine what settings it should use, so remember that after updating bandwidth statements in the future, you have to re-run the AutoQoS commands.

Making it Work

In the most standard situation, where VoIP isn’t performing as it was promised, the network admin can quickly save the day by running the following on the WAN interface:

interface Serial0
bandwidth 256
autoqos voip

If it’s the local network that needs tuning, the following can be run on Catalyst switches (if running Enhanced Images):

auto qos voip cisco-phone
auto qos voip trust

It really couldn’t be easier than that.  For the WAN example, we told the router that interface Serial0 has 256 Kbps, and to enable VoIP QoS. The switch example is similar, for Cisco phones.

The neat part about this is that AutoQoS is actually doing more than just generating a configuration for you and forgetting about it. If you run the command show autoqos interface s0, you will see much more than just your standard old interface configuration. It will show that a Virtual Template “interface” has been created, and that a class is applied to the interface. The same output will also show you the configuration of the template and class-map, with an asterisk next to each entry that was generated by AutoQoS. It’s actually keeping track of what was done automatically so that you can learn what AutoQoS is doing. As mentioned previously, however, don’t forget that removing the AutoQoS configuration will destroy all QoS settings on an interface, not just the ones that AutoQoS configured.

Finally, remember to enable QoS on both sides of a WAN link to truly prioritize VoIP packets. Don’t forget to read through the Cisco documentation before deploying it, even though AutoQoS is simple, in comparison. It is simple, but the more prepared you are the easier it is to deploy.

Cisco will hopefully continue this trend of providing Auto features for complicated, but common tasks. AutoQoS for VoIP sure does enable a much larger audience to correctly deploy VoIP over a wide variety of networks.

Feel free to share!

Hide Sites

Related posts:

1. Built-in Security with Cisco IPS
2. Are Cisco Flex Links the End of STP?
3. Manage Devices and Configurations with Cisco SDM
4. What the Heck is a TCAM?
5. Networking 101: More Subnets, and IPv6

Networking 101: Layer 2, Link and Spanning Tree

What’s more important than IP and routing? Well, Layer 2 is much more important when it’s broken. Many people don’t have the Spanning Tree Protocol (STP) knowledge necessary to implement a Layer 2 network that’s resilient. A switch going down shouldn’t prevent anyone from having connectivity, excluding the hosts that are directly attached to it. Before we can dive into Spanning Tree, you must understand the innerworkings of layer 2.

Layer 2, the Data Link layer, is where Ethernet lives. We’ll be talking about bridges, switching, and VLANs with the goal of discovering how they interact in this part of Networking 101. You don’t really need to study the internals of Ethernet to make a production network operate, so if you’re inclined, do that on your own time.

Ethernet switches, as they’re called now, began life as a “bridge.” Traditional bridges would read all Ethernet frames, and then forward them out every port, except the one it came in on. They had the ability to allow redundancy via STP, and they also began learning which MAC addresses were on which port. At this point, a bridge then became a learning device, which means they would store a table of all MAC addresses seen on a port. When a frame needed to be sent, the bridge could look up the destination MAC address in the bridge table, and know which port is should be sent out. The ability to send data to only the correct host was a huge advancement in switching; collisions were much less likely. If the destination MAC address wasn’t found in the bridge table, the switch would simply flood it out all ports. That’s the only way to find where a host actually lives for the first time, so as you can see, flooding is an important concept in switching. It turns out to be quite necessary in routing too.

Important terminology in this layer includes:

Unicast segmentation : Bridges can limit which hosts hear unicast frames (frames sent to only one MAC address). Hubs would simply forward everything to everyone, so this alone is a huge bandwidth-saver.

Collision Domain : The segment over which collisions can occur. Collisions don’t happen any more, since switches use cut-through forwarding and NICs are full-duplex. If you see collisions on a port, that means someone negotiated half-duplex accidentally, or something else is very wrong.

Broadcast Domain : The segment over which broadcast frames are sent and can be heard.

A few years later, the old store-and-forward method of bridge operation was modified. New switches started only looking at the destination MAC address of the frame, and then sending it instantly. Dubbed cut-through forwarding, presumably because frames cut through the switch much quicker and with less processing. This implies a few important things: a switch can’t check the CRC to see if the packet was damaged, and that implies collisions needed to be made impossible.

Now, to address broadcast segmentation, VLANs were introduced. If you can’t send a broadcast frame to another machine, they’re not on your local network, and you will instead send the entire packet to a router for forwarding. That’s what a Virtual LAN (VLAN) does, in essence–it makes more networks. On a switch, you can configure VLANs, and then assign a port to a VLAN. If host A is in VLAN 1, it can’t talk to anyone in VLAN 2, just as if they lived on totally disconnected devices. Well, almost; if the bridge table is flooded and the switch is having trouble keeping up, all data will be flooded out every port. This has to happen in order for communication to continue in these situations. This needs to be pointed out because many people believe VLANs are a security mechanism. They are not even close. Anyone with half a clue about networks (or with the right cracking tool in their arsenal) can quickly overcome the VLAN broadcast segmentation. In fact, a switch will basically turn into a hub when it floods frames, spewing everyone’s data to everyone else.

If you can’t ARP for a machine, you have to use a router, as we already know. But does that mean you have to physically connect wires from a router into each VLAN? Not anymore, we have layer 3 switches now! Imagine for an instance, if you will, a switch that contains 48 ports. It also has VLAN 1 and VLAN 2, and ports 1-24 are in VLAN 1, while ports 25-48 are part of VLAN 2. To route between the two VLANs, you have basically three options. First, you can connect a port in each VLAN to a router, and assign the hosts the correct default route. In the new-fangled world of today, you can also simply bring up two virtual interfaces in each VLAN. In Cisco land, the router interfaces would be called vlan1 and vlan2. They get IP addresses, and the hosts use the router interface as their router.

The third way brings us to the final topic of the layer 2 overview. If you have multiple switches that need to contain the same VLANs, you can connect them together so that VLAN 1 on switch A is the same as VLAN 1 on switch B. This is accomplished with 802.1q, which will tag the packets as they leave the first switch with a VLAN identifier. Cisco calls these links “trunk ports,” and you can have as many VLANs on them as the switch allows (currently 4096 on most hardware). So, the third and final way to route between VLANs is to connect a trunk to a router, and bring up the appropriate interfaces for each VLAN. The hosts on VLAN 1, on both switch A and B will have access to the router interface (which happens to be on another device) since they are all “trunked” together and share a broadcast domain.

We’ve saved you from the standard “this is layer 2, memorize the Ethernet header” teaching method. To become truly guru you must know it, but to be a useful operator, (something the cert classes don’t teach you) simply understand how it all works. Join us next time for an exploration of most interesting protocol in the world, Spanning Tree.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Understanding Layers
2. Networking 101: More Subnets, and IPv6
3. Networking 101: IP addresses
4. Networking 101: Subnetting – Slice Up 32-bits
5. Are Cisco Flex Links the End of STP?

Understanding Fibre Channel

As we dive deeper into SAN technology, it’s Fibre Channel’s turn to be examined. FC is the underpinning of all SAN technologies these days, as it won the protocol war roughly 25 years ago.

FC wouldn’t be much use without something on top of it, namely SCSI. FC is the low-level transport that ships data, but hosts are normally talking SCSI as far as they’re concerned. The hubs, switches, and HBAs in a SAN all speak FC, while the applications that use SAN storage continue to use familiar protocols, like SCSI.

The idea behind FC was to create a high throughput, low latency, reliable and scalable protocol. Ethernet wouldn’t quite cut it for highly-available storage needs. FC can currently operate of speeds up to 10Gb/s (10GFC) for uplinks, and 4Gb for standard host connections. FC also provides small connectors. As silly as it sounds, SCSI cables become unruly after time, and small strands of fibre are certainly easier to manage. The equipment required to connect to a FC SAN (multiple HBAs for each host, fibre, and switches) is extremely expensive, and was the main reason SAN technologies took so long to become widely adopted.

Topologies
In reality, two different protocols, or topologies, make up the FC protocol. FC supports all topologies, but the behavior of the protocol changes depending on the topology. The following three types of topologies are supported:

• PTP (point to point): normally used for DAS configurations.
• FC-AL (FC Arbitrated Loop): Fabric Loop ports, or FL ports on a switch, and NL_Ports (node loop) on an HBA, support loop operations.
• FC-SW (FC Switched): the mode when operating on a switched SAN.

FC-AL operation is quite scary, but sometimes a device doesn’t support FC-SW operations, and there’s no choice. A hub has no choice but to operate in FC-AL mode, and therefore attached hosts must as well. When a device joins an FC-AL, or when there’s any type of error or reset, the loop must reinitialize. All communication is temporarily halted during this process, so it can cause problems for some applications. FC-AL is limited to 127 nodes due to the addressing mechanism, in theory, but in reality closer to 20. FC-AL is mostly relegated to niche uses now, including but not limited to internal disk array communications and internal storage for high-end servers.

FC switches can be connected any way you please, since the FC protocol avoids the possibility of a loop by nature. Ethernet isn’t so lucky. The addressing scheme used does impose a limit of 239 switches though. FC switches use FSPF, a link-state protocol like OSPF in the IP world, to ensure loop-free and efficient connectivity.

FC networks are generally designed in one of two ways: either one big star, or one big star with edge switches hanging off it. These are commonly known as “core-only” and “core-edge” configurations. Normally a SAN will contain two of these networks, and each host’s HBA or storage device’s controller will attach to each. Keeping these networks separate isn’t as necessary as it is with FC-AL topologies, but even with FC-SW setups it still provides complete isolation and assurance that a problem in one fabric won’t impact the other. An FSPF recalculation, for example, could cause a brief interruption in service.

Ports
As previously mentioned, there are different port types in a SAN, and it can get confusing. Let’s try to clear up some of that terminology:

• N_Port: Node Port; the node connection point; end points for FC traffic
• F_Port: Fabric Port; a switch-connected port, that is a “middle point” connection for two N_Ports
• NL_Port: Node Loop Port; connects to others via their NL_Ports, or to a switched fabric via a single FL_Port; or NL_port to F_Port to F_Port to N_Port (through a switch)
• FL_Port: Fabric Loop Port; a shared point of entry into a fabric for AL devices; example  NL_Port to FL_Port to F_Port to N_Port
• E_Port: Expansion Port; used to connect multiple switches together via ISL (inter-switch links)
• G_Port: Generic Port; can switch between F_Port and E_Port operation depending on how it’s connected
• TE_Port: Trunked Expansion Port; link aggregation of multiple E_Ports for higher throughput

You’ll generally only see F_Ports and FL_Ports when looking at a single SAN switch, and knowing the difference helps. FL means that you’re talking FC-AL, and there’s a device attached that is either a hub, something that can’t do anything but FC-AL, or something strange. Ports will automatically configure themselves as an FL_Port if the attached device is Loop-only, otherwise it will be an F_Port. It’s also worth noting that some brands of FC switches don’t allow you to have an E_Port unless you pay a higher licensing fee. It’s something to think about if you ever plan to connect multiple switches together.

FC Layers
FC has its own layers, so in fact, calling it “like Ethernet” isn’t quite accurate, even if it helps for understanding. They are:

• FC-0: The interface to the physical media; cables, etc
• FC-1: Transmission protocol or data-link layer, encodes and decodes signals
• FC-2: Network Layer; the core of FC
• FC-3: Common services, like hunt groups
• FC-4: Everything! Protocol mapping for SCSI, iSCSI, FCP, IP, and others

The bulk of FC is really in FC-2. FC-PH refers to FC-0 through FC-2, which are strangely dubbed the physical layers.

FC also supports its own naming and addressing mechanism, which sheds light on the previously mentioned limitations in FC-AL and FC-SW topologies. Next time, we’ll discuss the header format for FC-2 as well as FC address assignment and name resolution.

In a Nutshell:

• FC is the transport mechanism, and SCSI or even IP rid atop FC
• FC-AL is a loop, where all connected devices see each other, and a re-initialization takes out the entire SAN
• Port types reveal what is actually happening, and knowing what they stand for can aid in topology visualization when looking at a switch’s configuration

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: Understanding Layers
2. Understanding Linux Virtual Memory
3. SAN 101: Intro to SANs and Storage
4. Are Cisco Flex Links the End of STP?
5. Networking 101: Layer 2, Link and Spanning Tree

Multi-user Security in Linux

A wise man once said, “everyone is root if you allow them to login as a user,” in retort to a question about the security of a multi-user Linux system. There is plenty of truth in that, but just accepting eminent compromise isn’t always acceptable. Let’s take a look at how you can limit your exposure while letting unknown and untrusted users login with a shell.

There are basically two groups of people who’d want to restrict login users heavily. First, the collaborators, possibly two separate organizations that have been forced to work together. Second, people who wish to allow some shady characters access to a shell, but believe they may attempt to compromise security. If at all possible, the best policy is to simply not give access out, and if you do, make sure patches are applied daily.

To say that you simply shouldn’t give out shells to untrustworthy users may work in a few instances. Say, for example, there is a need for remote users at another site to login and run the same series of commands every day. Say, for the sake of argument, their task can be easily scripted. If this is their only purpose on the server, a shell certainly isn’t necessary. OpenSSH allows a set of restrictions to be applied to an SSH key.

At the end of an SSH key entry, you can tack on these options:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command=”~/bin/script.sh”
This effectively restricts any SSH connections using this key to only being allowed to run the mentioned script. This can even be a setuid script that restarts a web server, for example. It’s quite safe, because OpenSSH will reject any variation of the command= text. Users possessing this key will only be able to execute the command that is explicitly allowed.

Aside from that, and possibly some fancy web-based tools or cron jobs, there aren’t may options left. At times users just need to be able to login and work.

It should go without saying that you need to stay up-to-date on patches. We won’t focus too much on that, aside from saying: automate! Securing a machine is an entirely different topic all together, but here are a few points to consider.

Enabling SELinux (Security-Enhanced Linux) is your first line of defense against unknown attacks. SELinux can prevent buffer overflows, as opposed to simply taking the “updates” path, which requires that a publicly known hole be fixed before some tries to exploit it. SELinux provides a significantly improved access system to limit programs from accessing things they don’t require to be operational. That, combined with overflow prevention makes it quite difficult to compromise a Linux system.

Further, on the issue of securing a multi-user machine, there is a much-debated precept: that users shouldn’t be able to see what processes are running, unless they own them. This restriction is simple to enable in Linux and the BSD’s, but does it really buy you anything? The answer is “maybe,” and at the same time, “not really.” To satisfy the maybe camp, consider a process’s arguments. When you run a command with a given set of arguments, the command as well as the arguments will show up in a ‘ps’ listing. If you have provided a password on the command-line for some reason, it will be visible to anyone running a ‘ps’ while your process is still running. Many people think that allowing users to see running daemon processes on a server will allow them to know what to try attacking. This information is trivial to obtain via other means anyway, so “not really.”

Every time this discussion starts, someone quickly suggests a chroot jail. The chroot command stands for “change root,” which does just that. If you run the command: ‘chroot /home/charlie /bin/bash’ then chroot will look for the shell in /home/charlie/bin/bash, and then proceed to lock you into that directory. The new root of the file system, for the lifetime of the bash shell, is /home/charlie. You now have zero access to any other part of the actual file system. Any available command, and its required libraries, needs to be copied into the chroot jail. Providing a usable environment is a ton of work. It’s actually easier to give each user their own Linux Xen or Solaris Zone instance. Really.

Finally we come to the restricted shells. The most popular, rbash, is a restricted bash shell. Setting a user’s shell to rbash will provide absolutely zero security. In theory, rbash will prevent users from running anything by specifying a full path, including ‘./’ (the current directory). This implies that it’s difficult for users to run commands, including scripts they write or downloaded exploits. Since $PATH is controlled globally, users can only run things in those locations. Unfortunately, /bin/ is going to need to be in their path, so all a user needs to do is run a new shell, and rbash is no longer in the picture: ‘exec bash’

One method of alleviating this is to give users only one item in their path, a directory the administrator created. Within the directory, simply place symlinks to all the authorized commands. This is nearly as cumbersome as setting up chroot, but much more tolerable.

Security isn’t convenient, and if it is, you’re doing something wrong.

There are certainly ways to prevent users from running downloaded programs, but in the end, the multi-user security of a system will depend on security of every piece of software installed. Preventing the exploits from being successful, a la SELinux, adds the most viable method of protection. Coupled with a frequently updated system, additional restrictions such as rbash aren’t generally necessary.

Feel free to share!

Hide Sites

Related posts:

1. The Perils of Sudo With User Passwords
2. Built-in Security with Cisco IPS
3. Understanding Linux Virtual Memory
4. Back to Basics: Unix File Permissions
5. Working With Unix Variant Differences

Networking 101: Understanding Layers

Continuing our journey, it’s time to take a trip up the OSI Reference Model, and learn what this mysterious thing is all about. The network stack is of great significance, but not so much that it’s the first thing you should learn. The networking 101 series has waited to ensue the “layers” discussion for good reason. Many so-called networking classes will start by teaching you to memorize the name of every layer and every protocol contained within this model. Don’t do that. Do realize that layers 5 and 6 can be completely ignored, though.

The International Standards Organization (ISO) developed the OSI (Open Systems Interconnection) model. It divides network communication into seven layers. Layers 1-4 are considered the lower layers, and mostly concern themselves with moving data around. Layers 5-7, the upper layers, contain application-level data. Networks operate on one basic principle: “pass it on.” Each layer takes care of a very specific job, and then passes the data onto the next layer.

The physical layer, layer 1, is too often ignored in a classroom setting. It may seem simple, but there are aspects of the first layer that oftentimes demand significant attention. Layer one is simply wiring, fiber, network cards, and anything else that is used to make two network devices communicate. Even a carrier pigeon would be considered layer one gear (see RFC 1149). Network troubleshooting will often lead to a layer one issue. We can’t forget the legendary story of CAT5 strung across the floor, and an office chair periodically rolling over it leading to spotty network connectivity. Sadly, this type of problem exists quite frequently, and takes the longest to troubleshoot.

Layer two is Ethernet, among other protocols; we’re keeping this simple, remember. The most important take-away from layer 2 land is that you should understand what a bridge is. Switches, as they’re called nowadays, are bridges. They all operate at layer 2, paying attention only to MAC addresses on Ethernet networks. The common fledgling network admin always seem to mix up layers two and three. If you’re talking about MAC address, switches, or network cards and drivers, you’re in the land of layer 2. Hubs live in layer 1 land, since they are simply electronic devices with zero layer 2 knowledge. Layer two will have it’s own section in Networking 101, so don’t worry about the details for now, just know that layer 2 translates data frames into bits for layer 1 processing.

On the other hand, if you’re talking about an IP address, you’re dealing with layer 3 and “packets” instead of layer 2′s “frames.” IP is part of layer 3, along with some routing protocols, and ARP (Address Resolution Protocol). Everything about routing is handled in layer 3. Addressing and routing is the main goal of this layer.

Layer four, the transport layer, handles messaging. Layer 4 data units are also called packets, but when you’re talking about specific protocols, like TCP, they’re “segments” or “datagrams” in UDP. This layer is responsible for getting the entire message, so it must keep track of fragmentation, out-of-order packets, and other perils. Another way to think of layer 4 is that it provides end-to-end management of communication. Some protocols, like TCP, do a very good job of making sure the communication is reliable. Some don’t really care if a few packets are lost–UDP is the prime example.

And arriving at layer seven, we wonder what happened to layer 5 and 6. They’re useless. A few applications and protocols live there, but for the understanding of networking issues, talking about these provides zero benefit. Layer 7, our friend, is “everything.” Dubbed the “Application Layer,” layer 7 is simply application-specific. If your program needs a specific format for data, you will invent some format that you expect the data to arrive in, and you’ve just created a layer 7 protocol. SMTP, DNS, FTP, etc, etc are all layer 7 protocols.

The most important thing to learn about the OSI model is what it really represents. Pretend you’re an operating system on a network. Your network card, operating at layers 1 and 2, will notify you when there’s data available. The driver handles the shedding of the layer 2 frame, which reveals a bright, shiny layer 3 packet inside (hopefully). You, as the operating system, will then call your routines for handling layer 3 data. If the data has been passed to you from below, you know that it’s a packet destined for yourself, or it’s a broadcast packet (unless you’re also a router, but never mind that for now). If you decide to keep the packet, you will unwrap it, and reveal a layer 4 packet. If it’s TCP, the TCP subsystem will be called to unwrap and pass the layer 7 data onto the application that’s listening on the port it’s destined for. That’s all!

When it’s time to respond to the other computer on the network, everything happens in reverse. The layer 7 application will ship its data onto the TCP people, who will stick additional headers onto the chunk of data. In this direction, the data gets larger with each progressive step. TCP hands a valid TCP segment onto IP, who give its packet to the Ethernet people, who will hand it off to the driver as a valid Ethernet frame. And then off it goes, across the network. Routers along the way will partially disassemble the packet to get at the layer 3 headers in order to determine where the packet should be shipped. If the destination is on the local Ethernet subnet, the OS will simply ARP for the computer instead of the router, and send it directly to the host.

Grossly simplified, sure; but if you can follow this progression and understand what’s happening to every packet at each stage, you’re just conquered a huge part of understanding networking. Everything gets horribly complex when you start talking about what each protocol actually does. If you are just beginning, please ignore all that stuff until you understand what the complex stuff is trying to accomplish. It makes for a much better learning endeavor! In future Networking 101 articles we will begin the journey up the stack, examining each layer in detail by discussing the common protocols and how they work.

Feel free to share!

Hide Sites

Related posts:

1. Networking 101: More Subnets, and IPv6
2. Networking 101: Subnetting – Slice Up 32-bits
3. Networking 101: Layer 2, Link and Spanning Tree
4. Understanding Fibre Channel
5. Networking 101: IP addresses

SAN 101: Intro to SANs and Storage

Welcome! We begin outr Storage Networking 101 series with an introduction to Storage Area Networks and storage technologies. In case you missed it, be sure to read the entire Networking 101 series (link coming soon) before embarking on the Storage journey—a solid understanding of various network protocols is required.

What is a storage network?
A storage network is any network that’s designed to transport block-level storage protocols. Hosts (servers), disk arrays, tape libraries, and just about anything else can connect to a SAN. Generally, one would use a SAN switch to connect all devices, and then configure the switch to allow friendly devices to pair up. The entire concept is about flexibility: in a SAN environment you can move storage between hosts, virtualize your storage at the SAN level, and obtain a new level of redundancy than was ever possible with direct-attached storage.

A FC-SAN, or Fiber Channel SAN, is a SAN comprised of the Fiber Channel protocol. Think of Fiber Channel (FC) as an Ethernet replacement. In fact, Fiber Channel can transport other protocols, like IP, but it’s mostly used for transporting SCSI traffic. Don’t worry about the FC protocol itself for now; we’ll cover that in another article later on.

A fairly new type of SAN is the IP-SAN: an IP network that’s been designated as a storage network. Instead of using FC, an IP-SAN uses Ethernet with IP and TCP to transport iSCSI data. There’s nothing to stop you from shipping iSCSI data over your existing network, but an IP-SAN typically means that you’re using plumbing dedicated for the storage packets. Operating system support for the iSCSI protocol has been less than stellar, but the state of iSCSI is slowly improving.

Another term you’ll frequently see thrown around is NAS. Network Attached Storage doesn’t really have anything to do with SANs—they’re just file servers. A NAS device runs something like Linux, and serves files using NFS or CIFS over your existing IP network. Nothing fancy to see here; move along.

There is one important take-away from the NAS world, however. That is the difference between block-level storage protocols and file-level protocols. A block-level protocol is SCSI or ATA, where as file protocols can be anything from NFS or CIFS to HTTP. Block protocols ship an entire disk block at once, and it gets written to disk as a whole block. File-level protocols could ship one byte at a time, and depend on the lower-level block protocol to assemble the bytes into disk blocks.

Block-level protocols
A protocol always defines a method by which two devices communicate. Block storage protocols are no different: they define how storage interacts with storage controllers. There are two main block protocols used today: SCSI and ATA.

ATA operates in a bus topology, and allows for two devices on each bus. Your IDE disk drive and CD ROM are, you guessed it, using the ATA protocol. There are many different ATA standards, but we’ll cover just the important ones here. ATA-2 was also known as EIDE, or enhanced IDE. It was the first of the ATA protocol we know today. ATA-4 introduced ATAPI, or the ATA Packet Interface, which allows for CD ROM devices to speak SCSI-like on the same bus as a regular ATA device.

The neat thing about ATA is that the controllers are integrated. The only “traffic” sent over the ATA bus is plain electrical signals. The host operating system is actually responsible for implementing the ATA protocol, in software. This means that ATA devices will never, ever be as fast as SCSI, because the CPU has to do so much work to just talk to these devices. As far as SANs are concerned, ATA isn’t that important. There are some ATA-based devices that allow you to connect cheap disks, but they translate operations into SCSI before sending them out to the SAN.

SCSI, on the other hand, is very confusing. SCSI-1 and SCSI-2 devices were connected via a parallel interface to a bus that could support 8 or 16 devices, depending on the bus width. Don’t worry about the details unless you’re unfortunate enough to have some older SCSI gear lying around.

SCSI-3 separated the device-specific commands into a different category. The primary SCSI-3 command set includes the standard commands that every SCSI-3 device speaks, but the device-specific commands can be anything. This opened up a whole new world for SCSI, and it has been used to support many strange and wonderful new devices.

SCSI controllers normally contain a storage processor, and the commands are processed on-board so that the host operating system doesn’t become burdened to do so, as with ATA. Such a SCSI controller is called a Host Bus Adapter. In the SAN world, the FC card is always called an HBA.

The main thing to know about SCSI is that it operates in a producer/consumer manner. One SCSI device (the initiator) will initiate the communication with another device, which is known as the target. The roles can be reversed! Most people call this a command/response protocol, because the initiator sends a command to a target, and awaits a response, but not always. In asynchronous mode, the host (initiator) can simply blast the target with data until it’s done. The SCSI bus, parallel in nature, can only support a single communication at a time, so subsequent sessions must wait their turn. SAS, or Serial Attached SCSI, does away with this limitation by automatically switching back and forth.

SCSI is tremendously more complex, but that’s the gist of it.

We need to understand SCSI to know how our storage network is going to ship data. The SCSI protocol plays an enormous role in storage networking, so you may even want to look at it more in-depth.

Next up, we’ll begin talking about Fiber Channel itself, which, as chance would have it, is much more complex than Ethernet. This is certainly going to be a fun journey.

In A Nutshell:

- A FC-SAN is a network that uses Fiber Channel at Layer 2, instead of Ethernet, and is dedicated to moving around SCSI commands.
- SCSI initiator, generally a host’s storage controller, is called an HBA; the SCSI target is most often the storage device you’re talking to.
- iSCSI can transport SCSI over your existing network, but a network dedicated to iSCSI is called an IP-SAN.

Feel free to share!

Hide Sites

Related posts:

1. Understanding Fibre Channel
2. Networking 101: Understanding Layers
3. How Much Server do you Need?