Automating and Monitoring Your AWS Costs

Regarding costs, there are two things I find most important when managing an infrastructure in AWS. The two sections below talk about automated mechanisms to monitoring your bill, and decide if you could be saving money somewhere.

Is the bill going to be a surprise?

You can login to the EC2 console and look at your estimated bill, but that requires memory, and time, and clicking around in a horrible interface.

I recommend two things to track your costs.

1) Set a cloudwatch alarm at your expected bill level.

If you expect to spend, say, $80K per month, then simply set a cloudwatch alarm at that level. You’ll get an email hopefully at the very end of the month (not the middle) saying you’ve reached $80K. This page shows you how: http://docs.aws.amazon.com/AmazonCloudWatch...

2) Graph your estimated charges.

You use graphite, right?

There is a cloudwatch CLI that you can use to fetch metrics from cloudwatch – including estimated billing, once you’ve enabled those metrics as described in the above link. Using the mon-get-stats is really annoying, but it works. Here is the shell script I use to grab billing metrics and shove them into graphite:

#!/bin/sh
export AWS_CLOUDWATCH_HOME=/home/charlie/cloudwatch/CloudWatch-1.0.13.4
export JAVA_HOME=/usr/lib/jvm/default-java

# Get the timestamp from 5 hours ago, to avoid getting > 1440 metrics (which errors).
# also, remove the +0000 from the timestamp, because the cloudwatch cli tries to enforce
# ISO 8601, but doesn't understand it.
DATE=$(date --iso-8601=hours -d "5 hours ago" |sed s/\+.*//)

#echo $COST

SERVICES='AmazonS3 ElasticMapReduce AmazonRDS AmazonDynamoDB AWSDataTransfer AmazonEC2 AWSQueueService'

for service in $SERVICES; do

COST=$(/home/charlie/cloudwatch/CloudWatch-1.0.13.4/bin/mon-get-stats EstimatedCharges --aws-credential-file ~/.ec2_credentials --namespace "AWS/Billing" --statistics Sum --dimensions "ServiceName=${service},Currency=USD" --start-time $DATE |tail -1 |awk '{print $3}')

if [ -z $COST ]; then
 echo "failed to retrieve $service metric from CloudWatch.."
 else
 echo "stats.prod.ops.billing.ec2_${service} $COST `date +%s`" |nc graphite.example.com 2023
 fi

done

# one more time, for the sum:
COST=$(/home/charlie/cloudwatch/CloudWatch-1.0.13.4/bin/mon-get-stats EstimatedCharges --aws-credential-file ~/.ec2_credentials --namespace "AWS/Billing" --statistics Sum --dimensions "Currency=USD" --start-time $DATE |tail -1 |awk '{print $3}')

if [ -z $COST ]; then
 echo "failed to retrieve EstimatedCharges metric from CloudWatch.."
 exit 1
else
 echo "stats.prod.ops.billing.ec2_total_estimated $COST `date +%s`" |nc graphite.example.com 2023
fi

You will have to install the Java-based cloudwatch CLI, and the ec2 credentials file has to be in a specific format – refer to the docs.

I run this from cron every 5 minutes, and the data is sent straight to graphite (not via statsd). Then, I can display the graph with something like:

http://graphite.example.com/render/?from=-7days&until=now&hideLegend=false&\
title=AWS%20estimated%20bill,%20monthly,%20in%20USD&lineWidth=3&lineMode=connected\
&target=legendValue(aliasByMetric(stats.prod.ops.billing.ec2_*)%2C%22last%22)\
&target=legendValue(alias(color(stats.prod.ops.billing.monthly_unused_RI_waste%2C%22red%22)\
%2C%22monthly%20cost%20of%20unused%20RIs%22)%2C%22last%22)

The final metric, stats.prod.ops.billing.monthly_unused_RI_waste, is pulled from rize.py (mentioned below), to show how much money is being spent on reserved instances that aren’t running (i.e. waste).

After all that, you can have a graph on the wall that shows your estimated monthly billing. Here is an example with totally made up dollar amounts:

Could I be saving money by doing something differently?

Likely, you can.

1) Spot instances

Spot instances are scary, especially if you aren’t using the ELB autoscaler. In the meantime, before your re-design and AMI-building project comes to fruition so you can use the autoscaler in a sane way, there are a few non-scary ways to use spot instances.

You can run one-off compute-heavy jobs (even via Elastic MapReduce!) using spot instances. If you bid just a few cents above the current price, your instances may get terminated and you may have to start over. This rarely happens, but it may. I recommend bidding all the way up to the on-demand price for the instances, if you don’t want them to disappear. People have run spot instances for 6+ months without having them terminated.

Another strategy is to provision a cluster of instances for your workload using reserved instances, up to the number of instances you think you need to provide the appropriate response-time (i.e. serve the traffic). You will be wrong about the maximum capacity of these instance, by the way. But that’s OK – next, you provision double capacity using spot instances. In this use-case, I’d bid 20% above the 30-day historical max market price. You’ll be paying a fraction of the on-demand cost for these instances – depending on the type and number, you can often double the capacity of your cluster (1/2 with spot instances) for just the cost of 1-2 on-demand instances. I know you’re thinking “when the market price shoots up and they all die, I don’t have enough capacity!” I recommend using the autoscaler.. but barring that, another strategy is to provision 20% of the extra nodes with a spot bid price as high as the on-demand rate. Chances are, they will never be terminated. Or better yet, provision 20% above your wrongly-estimated max capacity with on-demand or reserved instances, just to be safe

2) Reserved instances

You can save ~60% by reserving your on-demand EC2 instances for 1 year.

I wrote a python script to tell you how many instances you’re running, vs. how many are reserved, and it’s quite useful! Here:
https://github.com/manos/AWS-Reserved-Instances-Optimizer 

3) S3 lifecycle policies

S3 can be another huge cost in EC2. You have a few options for making sure your buckets don’t grow out of control.

Depending on your use case, you can:

• Set an expiry policy, so e.g. all objects in a bucket are deleted after 1 year
• Set a lifecycle policy, so e.g. all objects in a bucket are archived to Amazon Glacier after 60 days

5) Dead instances still cost money

#!/usr/bin/env python
#
# print a list of 'failed' instances in ec2.
# first and only argument is the region to connect to (default: us-east-1)
# run from cron for each region to check.
#
import boto.ec2
import sys
import collections

regions = boto.ec2.regions()
names = [region.name for region in regions]

try:
    if len(sys.argv) > 1:
        region = regions[names.index(sys.argv[1])]
    else:
        region = regions[names.index('us-east-1')]
except ValueError:
    sys.stderr.write("Sorry, the region '%s' does not exist.\n" % sys.argv[1])
    sys.exit(1)  # proper return value for a script run as a command

ec2 = region.connect()
stats = ec2.get_all_instance_status(filters={"system-status.reachability": "failed"})

if len(stats) > 0:
    print "The following instances show 'failed' for the ec2 reachability check: "

for stat in stats:
    reservation = ec2.get_all_instances(filters={'instance-id': stat.id})
    dead_instance = reservation[0].instances[0]

    print dead_instance.tags.get('Name'), stat.id, stat.zone, stat.state_name

    if isinstance(stat.events, collections.Iterable):
        print "\tmost recent events: ", [(event.code, event.description) for event in stat.events]

I could go on forever – but this is getting rather long

Feel free to share!

Hide Sites

Starting out: a new approach to systems monitoring.

OK, not new to some. Circonus does it this way, and so do some very large sites like Netflix.

But new to me, and certainly new to anyone currently using nagios/zenoss/zabbix/etc. Here’s the story:

The Idea

Metrics

At work (Krux), we have graphite and tons of graphs on the wall. We can see application-level response times in the same view as cache hit/miss rates and requests per second. That’s nice. It’s also not very proactive.

Monitoring

We also have cloudkick (think: nagios with an API). We have tons of plugins checking thresholds, running locally on each box. We recently re-evaluated our monitoring solution, and ultimately decided to write our own loosely coupled monitoring infrastructure using a variety of awesome tools. We migrated from cloudkick to collectd with a bunch of plugins we wrote, using a custom python library I wrote, called monitorlib (collectd and pagerduty parts). The functionality is basically the same: run scripts on each node every 60 seconds, check if some threshold is met, and alert directly to pagerduty. meh.

Combining

What I really want is a decision engine.

I want applications to push events, when they know something a poll-based monitoring script doesn’t.
I want to suppress threshold-based alerts, based on a set of rules, and only alert some people.
I want to check the load balancer to see how many nodes are healthy, before alerting that a single node went down.
I want to check response-time graphs in graphite, by polling the holt-winters confidence bands, and then alert based on configured rules.

Basically, we are in a world where we have great graphs, and old-school threshold-based alerts. I want to alert on graphs, but also much more – I want to combine multiple bits of information before paging someone at 2am.

How to get there

Going to the next level requires processing events, accepting event data from multiple sources, and configuring rules.

This blog post has some good ideas http://www.control-alt-del.org/2012/03/28/collectd-esper-amqp-opentsdbgraphite-oh-my/ and it outlines a few options.

Basically, I want *something* to sit and listen for events. I want all my collectd scripts to send data via HTTP POST (JSON), or protobufs, along with the status (ok, warn, error) every minute. Then, the *thing* that’s receiving these events, will decide – based on state it knows or gathers by polling graphite/load balancers/etc – whether to alert, update a status board, neither, or both.

Building that *thing* is the hard part. There are Complex Event Processing (CEP) frameworks available, most notably Esper, written in Java. Using Esper requires writing a lot of Java. There is a google open source thing, which seems like a bundle of code published but not maintained, called rocksteady. Using rocksteady may help the “ugh, don’t want to Java” aspect.

Then there is Riemann - this is what I’m starting with first. After learning a bit of Clojure, it should provide immediate benefit. And it’s actively developed and the author is very responsive. We’ll see how it goes!

Final notes

I think what I’m trying to do is a bit different than most.

I don’t want to send all my data (graphite metrics – we do around 150K metrics/sec to our graphite cluster) through this decision engine. I want it to get *events* which would historically have been something to page or email about. Then, it needs to make decisions: check graphs as another source of data; check load balancers; re-check to make sure it’s still a problem; maybe even spin up new EC2 instances. I may also want to poll graphite periodically to check various things, perhaps with graphite-tattle.

At this point, I don’t know what else it can/should do. The first step is to send all alerts to the decision engine, and define rules. It shall grow from there

Feel free to share!

Hide Sites

ELBs with stickiness can get you into.. sticky situations

AWS strongly recommends against using the stickiness features in their Elastic Load Balancer. Indeed, it’s nice to evenly distribute all requests and not worry about another layer of logic around “why a request went here.”

Unfortunately, in the real world, some applications really want stickiness. For example, at work we run a geoip service, which returns the location of an IP address. All 20+ nodes running this service have a local copy of the database, to avoid having an off-node dependency on the data. Lookups via the tornado-based service are not fast, however. So we toss varnish in front to cache 4+ GB worth of responses. If clients were hopping around between all nodes in an AZ, cache hit rates would suck. Point being, there are times that you want stickiness, and it doesn’t necessarily mean you have poor application design (so stop saying that, Amazon).

Now, when you point your CNAME at a region’s ELB CNAME, you’ll find that it returns 5 or more IP addresses. Some IPs are associated with individual AZs where you have live instances. There is no load balancing to determine which AZ gets which traffic; instead that’s handled by DNS (and to some extent, ELBs forwarding traffic between themselves). When a client hits ELB(s) in an individual AZ, requests are balanced across all instances in that AZ. This means that you can will observe all your repeated requests end up in a single AZ, but rotating between all instances. Unless you enable stickiness, of course.

So you enable stickiness because you want requests from the same client to end up at the same node (for caching, in this example).

Then the inevitable happens — a few of your nodes in the same AZ have hardware failures. You notice the load getting way too high in the remaining 2 instances, and decide that you’d really prefer to balance out that traffic between three other AZs. Like any logical person, you remove the instance in the AZ from the ELB, expecting the ELB(s) to notice there are no longer any instances in that AZ, and stop sending traffic (there’s nothing to receive it!).

This is when you start hearing about failed requests.

If you only remove instances from an ELB, and don’t remove the AZ itself from the ELB, the cluster of ELBs will still try very hard to forward traffic toward the AZ associated with the shared sticky state they know about. Your traffic is forwarded to, and hits an LB with no instances, and dies.

The moral of the story is: if you use stickiness, and want to disable a zone, you must remove the ZONE from the ELB, not just all instances.

That was an annoying lesson learned

Quick side-note: if you use ELB-generated cookies for stickiness, the ELBs will remove your Cache-Control headers!

It’s best to configure varnish/apache/etc set a Served-By cookie, and tell the ELB to use that. That also makes it really easy to tell which node served a request, if you’re debugging via ‘curl -v’ or similar.

Feel free to share!

Hide Sites

I give thee, cronlib and puppet-cron-analyzer

I’ve been working on a puppet cron analyzer tool, which is coming along nicely:

https://github.com/manos/Puppet-Cron-Analyzer

Feel free to share!

Hide Sites

Connecting to existing buckets in S3 with boto, the right way

Here’s another interesting tidbit.

If you have scripts that connect to S3, and you run out of buckets (Amazon only allows 100 buckets per account), you might get a nasty surprise.

See, you may have been using create_bucket(name-of-bucket) to get your bucket object. It’s undocumented as far as I can see, but apparently if you use create_bucket() on an bucket that actually exists, it’ll return the Bucket object. That’s handy! Except it breaks if you’re unable to create more buckets (even though you aren’t really trying to create more). Sigh, so I refactored as such:

# old and busted: bucket = s3_conn.create_bucket(bucket_name)
# new hotness:
# iterate over Bucket objects and return the one matching string:
def find_s3_bucket(s3_conn, string):
    for i in s3_conn.get_all_buckets():
        if string in i.name:
            return i
Used as: bucket = find_s3_bucket(s3_conn, bucket_name)

There is likely a more elegant way, but hey this works.

Feel free to share!

Hide Sites

Finding instances by name with boto in python

OK, I know I need to blog more. Rather than think I don’t have anything useful to say, I’ll start adding quick entries of what-I-learned.

Random tidbits from today:
I got annoyed with EC2 failures, and having to manually terminate and redeploy instances today, so I finally worked on a script I’ve been meaning to write. One thing I had to figure out (which isn’t all that complex), is how to discover an instance by name.

If you tag an instance with the hostname you’re using in your deployment script, you don’t need to fumble in the AWS console to find an instance ID. Ever. I don’t find it acceptable to manually click around or run scripts to discover information that’s available from an API

So, to “find” the instance using python and boto (assume AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are defined in your shell environment):

import boto
ec2conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
my_fqdn = "example.com" # trailing part of my domain

Now, ‘instances’ can be iterated over to find instances with the name you desire. I wrote a little wrapper function to do this, and it returns an instance object (which I call instance.terminate() on, for this purpose). Code:

def find_instance_by_nametag(instances, name):
    # support short or full hostname usage
    if not my_fqdn in name:
        name = name + my_fqdn
    for i in instances:
        if "Name" in i.tags and name in i.tags['Name']:
            return i
    sys.exit("sorry, I couldn't find an instance with that name!")

Easy as that!

Feel free to share!

Hide Sites

Display current location on your web page using the SPOT GPS tracker

The SPOT GPS tracker is not only a lifesaver, but also a handy tool for motorcycle (ok, and other types) travelers. If you subscribe to the Track Progress service, you can tell the GPS device to send your location to SPOT via satellite every 10 minutes, and then export those tracks at a later time.

They even have an API from which you can fetch an XML document with all your current tracks! The unfortunate part, and the reason for this post, is that they only keep 30 days of GPS coordinates. It’s not a problem if you export your data to Spot Adventures and create an “adventure” — that will live forever. But if you wish to present “my current location” on your personal web page, for example, you’re out of luck. It will only work as long as you’ve used the track progress functionality within the last 30 days.

I guess you need to cache the last used location yourself. OK, this shouldn’t be hard. This is how to do it with a little perl script (download here, see in action in CharlieTracker here):

#!/usr/bin/perl
 # Author: Charlie Schluting <[email protected]> (c) 2011
 #
 use XML::Simple;
 #use strict.. heh, no, this barely works.</code>
 $CACHEFILE = "/home/charlie/lastspotlocation.txt";
 $XML = "/home/charlie/spot.xml";
 $JS = "/stash/www/charlierides.com/files/map.js";

The CACHEFILE is where I store the last known latitude and longitude. It’s mostly for reference, so that other things (aside from this script) can use it.

XML is the location I store the fetched XML from SPOT’s API.

JS is the file I write the javascript out to. OK, you probably don’t need to care about this part of the script.. but here’s the details: if you use joomla or wordpress, you might not want to enable PHP code execution or other evil things like that. So, in order to include a snippet of generated HTML, the only choice (as far as I know, aside from editing the DB where a “display html” module stores its data), is to write javascript to include another javascript file. So that’s what I do.. the script actually generates javascript that when run, will spit out html. If you don’t care about that and just want the map part, read on.

# hahaha, oh man..
 `wget -q -O $XML http://share.findmespot.com/messageService/guestlinkservlet?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW`;

This part fetches the XML file from spot, using my shared page identifier.

# create object
 $xml = new XML::Simple;
 # read XML file
 $data = $xml->XMLin("$XML");

# this is how we overcome spot's API not keeping >30 days. If they've aged out, do nothing (i.e. keep using the old data).
 die("No messages found, totalCount is 0, ABORTING LIKE AN UGLY KID") unless $data->{totalCount} > 0;

# the first object is always the most recent:
 $lat = $data->{message}->[0]->{latitude};
 $long = $data->{message}->[0]->{longitude};

# just because (hey, what if something else wants to use this?)
 open(FILE, ">$CACHEFILE");
 print FILE $lat . "," . $long . "\n";
 close(FILE);

And the rest (above) is pretty self explanatory. Using just this part of the script, you’ve overcome the annoying limitation that is SPOT Track Progress (losing your last known location). Your last known coordinates will be in CACHEFILE. For completeness, I will include the rest of this horrible hack I used:

# ugly shit that writes out javascript to write out html, to include a linked static google maps image
 $googleoptions = "&zoom=8&size=140x152&sensor=false&maptype=hybrid";
 $googlelink = "<a target=\"blank\" href=\"http://maps.google.com/maps?q=" . $lat . "," . $long . "+(charlie)&z=8&t=h\"\>";
 $header = "<p>Current location (since the last GPS update):</p>";
 $js = 'document.write(\''. $header .'\');' . '
 document.write(\''. $googlelink .'\');' . '
 document.write(\'<img src="http://maps.googleapis.com/maps/api/staticmap?center=' . $lat .
 ',' . $long . $googleoptions . '" />\');
 document.write(\'</a>\');
 document.write(\'<p>Or <a href="http://share.findmespot.com/shared/faces/viewspots.jsp?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW"
 target=blank>view all recent tracks.</a></p> \');
 ';
 open(FILE, ">$JS");
 print FILE $js;
 close(FILE);

So, what this writes out (in HTML, finally), is an IMG tag embedding a static google map (because I’m using a small thumbnail), which links to the google maps page when clicked:

Map: http://maps.googleapis.com/maps/api/staticmap?center=45.42553,-122.52251&zoom=8&size=140×152&sensor=false&maptype=hybrid
Link target: http://maps.google.com/maps?q=45.42553,-122.52251+(charlie)&z=8&t=h

The parameters you can pass google maps are well documented, so I don’t need to rehash them here. You may even prefer to use the real maps API, rather than the static image one. That too, is well documented.

Run this from cron every 15 minutes, and you’ll always have your last checked-in GPS coordinates!

Feel free to share!

Hide Sites

Shell programming techniques and tips

Plenty of shell programming tutorials exist already; this isn’t yet another “howto.” We’re going to spend a little time talking about some of the frequently unused or misunderstood techniques in shell programming, and also cover some neat tricks involving the shell that may not be obvious to novice users.

For the most part, we’re talking about the bourne shell, because it exists everywhere and is compatible with bash. Bash does have some interesting features, but for our purposes the bourne shell will do just fine.

1. Implementing a lockfile 
Every once in a while we’ve got a shell script that needs to run, but dangerous things can happen if we’re running two copies at the same time. The simplest method is to just bail out if a lockfile exists, but you can also implement blocking. Here’s a basic example of checking for a lock file and then bailing out:

#!/bin/sh
LOCKFILE=/tmp/mylock

if [ ! –s $LOCKFILE ]; then
      echo $$ > $LOCKFILE
      # do stuff; the bulk of the script is here
      : > $LOCKFILE 
exit 0
else
      echo “PID `cat $LOCKFILE` running” | mailx –s “$0 can’t run” root
      exit 1
fi

Of course you can get much fancier. If this was inside a while loop, you could sleep a few seconds and then retry the lock. The above example uses ‘test’ with the bracket notation to check if the lockfile exists, and that it isn’t empty (the -s option to test). If the return value of test is true, the block of code runs and puts the script’s current PID into the lockfile. At the end of the “do stuff” block of code, which will probably call another script, we truncate the lockfile by using the null command. If the lockfile is non-empty, we send mail to root and bail out with an unsuccessful return code. The subject of the message includes the name of the script ($0), and the body of the message indicates which process ID is currently running.

2. Check return values, always

If you call an external program, or another script of your own, you must always check the return value of that program. Unexpected failures of commands can cause the rest of your script to misbehave, so you need to make sure everything ran properly. The bourne shell has a built-in variable $? that holds the return value of the last command. See item number three for an example.

Most importantly, you *very often* want to stop executing your script if any command fails. Simply toss a ‘set -e’ at the top of your script, and any command that returns non-zero will result in your script exiting.

3. Using return codes

Remember, the ‘if’ statement uses the return value of the statement immediately following ‘if’ to determine whether or not it should succeed or fail. So the test command can be used (with bracket notation), and so can any other command.

if [ “$?” -eq ‘0’ ]; then echo yay ; fi

The above statement will print “yay” if the last command executed returned success, else it does nothing. Remember that ‘0’ in the shell is success; it’s the opposite of when you’re programming in C. We also collapsed everything onto a single line here. The parser doesn’t see it that way, though, since the semi-colon represents a newline.

You can also put commands inside if statements. Let’s test to see if a list of machines ping before we try to scp the known_hosts file to them (Solaris ping syntax):

#!/bin/sh
for host in `cat ./hostlist.txt`
do
  echo "doing $host.."
  if ping $host 1; then
  scp /etc/ssh/ssh_known_hosts2 $host:/etc/ssh
fi
done

4. Capturing output from multiple commands

Frequently, people who want to capture output from multiple commands in a script, will end up appending output to the same file. You can use a subshell instead, and redirect all stdout (or stderr if you need to) from the subshell:

#!/bin/sh
( 
cat /etc/motd
cat /etc/issue 
) > /tmp/motd-issue

5. Other subshell tricks

Changing directories, and then executing a command, can be very useful when you’re piping stdout over ssh. It’s useful for local commands too. Using tar to copy the present directory into /tmp/test/:

tar cf - . | (cd /tmp/test && tar xpf -)

Note that if the cd fails (directory doesn’t exist), then tar will not be executed.

Parallel execution:
In our previous scp example, it would have run very slowly with a long list of hosts. We can make each scp command execute almost in parallel, by backgrounding the subshell process. The following portion of the script will burn through the loop very quickly, because the subshell is backgrounded, and therefore the commands run and the shell doesn’t wait for them to complete before looping again.

if ping $host 1; then
(
scp /etc/ssh/ssh_known_hosts2 $host:/etc/ssh
scp /etc/ssh/ssh_known_hosts2 $host:/etc/ssh
) &
fi

7. Piping data and commands over ssh

There are many examples of how piping things over ssh is useful. You can print remotely: cat file.ps | ssh $host lpr Which works because ssh sends its stdin to the stdin of the program you tell it to execute.

You can use tar to copy a directory to a remote host, over ssh:

tar cf - ./dir | ssh $host “cd /dir/ && tar xpf –“

And finally, you can execute scripts without having to manually copy the script to the host:

cat script.sh | ssh $host /bin/sh

8. Here documents

If you want to run commands over ssh in a long argument, similar to #7’s tar copy, you must take care to escape any special characters, or the local shell will try to use them. A quick and dirty way to avoid this is with a here document:

ssh $host <<\EOF
#!/bin/sh
# Entire script goes here, on multiple lines
EOF

If you omit the backslash before EOF, variable substution is also possible. Here documents are frequently used to embed a script within a script too. As a matter of fact, some software vendors distribute their software as a shell script, and the tar file containing the binaries is actually a here document. To extract a here document to /tmp/file, just use this in the script: cat <<EOF > /tmp/file and then start the document on the next line. You can use any string you want, EOF is just the standard.

9. Live shell programming

Remember how we said that the semicolon is used to indicate a newline? This implies that scripts can be written on the same line, excluding here documents. Here’s a quick example of a command I recently ran:

ssh $box "if test \`uname -r\` = '5.10'; \
then "mv /etc/dt/config/Xresources /etc/dt/config/C/ && \
/etc/init.d/dtlogin reset" ; fi"

You can add all kinds of shell logic here, and the backslash is useful to start typing on a new line–the shell won’t execure the command until you hit return without a backslash at the end. Just remember to quote things that your local shell will try and interpret. Generally people will use && in succession with multiple commands, or if you want the second (or third, or…) command to execute regardless of the return value of the previous command, just use the semicolon.

You can also just type “for” at the command prompt, and it will bring you to a new line waiting for input. An entire shell script embedded inside a loop can be written in real-time in this manner.

Knowing all these shell tricks certainly makes one’s life easier, but you really shouldn’t be spending all your time running commands on remote hosts. Stay tuned for more posts about configuration management

Feel free to share!

Hide Sites

Are Cisco Flex Links the End of STP?

Cisco Flex Links gives network operators a simple, reliable, and more scalable method of layer 2 redundancy. The Spanning Tree Protocol (STP) is not destined for the scrap bin, but it will certainly fall out of favor with many enterprise networks.

Flex Links are a pair of layer 2 interfaces configured to act as a backup of each other. Configuring Flex Links is very simple, but it’s a manual process. Spanning tree can configure itself if you just enable it, albeit likely a sub-optimal configuration, but a working one nonetheless. Flex Links, on the other hand, require manual setup and layout of your layer 2 network. If you don’t want to leave anything to chance, then Flex Links are preferred over STP.

The benefits of FlexLinks include:

• simplicity, which equals stability.
• instant failover.
• rudimentary load balancing capabilities, so one link isn’t wastefully idle.
• load balancing works across switches in a stack, including port channels.

Flex Links’ primary operating mode is just like spanning tree: one on, one off. With per-VLAN spanning tree, a trunk port can have some VLANs enabled and some blocked at the same time, so on the surface it seems that STP is superior. In reality, you can configure Flex Links to load balance VLANs, and we’ll show you how shortly.

Configuration

Conceptually, you configure Flex Links by telling one link it’s the active link, and another that it’s the backup of that

Flex Links Design Map

primary (active) one. Without configuring VLAN load balancing, it will completely disable the backup, and if the active link goes down the backup will take over.

For example, to configure port gi1/0/1 as a active link, and gi1/0/2 as the backup, you’d run:

Switch# configure terminal
Switch(conf)# interface gigabitethernet1/0/1
Switch(conf-if)# switchport backup interface gigabitethernet1/0/2

That’s all there is to configuring the basic mode, which gets you failover but no load balancing. Before talking about load balancing, let’s take a look at preemption and “mac address-table move update.”

Preemption

Preemption, that is, the preferred port for forwarding traffic, is also configurable. This is most often used in combination with multiple links that have differing bandwidth capacities. If you wish to ensure that port 1, a primary port that has more bandwidth, will return to the active link when it comes back up, you would set:  interface preemption mode bandwidth andswitchport backup interface preemption delay. The delay is used to set the amount of time (in seconds) to wait before allowing port 1 to preempt port 2 and begin taking over traffic again.

MAC Address-Table Move Update

Enabling the MAC address-table move update feature allows for rapid convergence when a primary link goes down and the backup takes over traffic forwarding duties. Without this feature enabled, neighboring switches may continue to forward traffic for a short time to a dead port, since they have learned MAC addresses associated with that link.

When move update is enabled, the switch containing Flex Links will broadcast an update packet to let other switches know what happened, and they will in turn un-learn that false MAC address mapping.

On the switch with Flex Links, simply configure:

Switch(conf)# mac address-table move update transmit

All switches, including ones with Flex Links, need to receive these updates. This is not enabled by default, so you’ll need to run the following command on all of your devices:

Switch(conf)# mac address-table move update receive

To see the status and verify that “move update” is enabled, run: show mac address-table move update. Checking the status of your Flex Links is much the same: show interfaces [interface-id] switchport backup.

Load Balancing

Flex Links should be configured such that both ports are forwarding traffic at the same time. This way, you get load balancing in addition to redundancy. The limitation is that only one port can be forwarding a single VLAN at a time. If we have VLANs 1-200, we need to choose which VLANs are forwarded primarily through which port. The most simple configuration, ignoring traffic requirements, would be that VLANs 1-100 use port 1, and VLANs 101-200 use port 2.

Before we get into configuring preferred VLANs, let’s talk about multicast. Multicast, of course, becomes an issue with this type of setup. If a port passed an IGMP join, and the switch is part of a multicast group, when the port goes down the switch will no longer be able to receive multicast traffic for that group. The quick fix is to make both Flex Links always be part of learned groups, with the command: switchport backup interface gigabitEthernet 1/0/12 multicast fast-convergence.

Now, on to VLAN load balancing. It is quite easy; just specify which VLANs you prefer on which links:

Switch(config-if)#switchport backup interface gigabitEthernet1/0/2 prefer vlan 101-200.

If you have VLANs 1-200 on the switch, show interfaces switchport backup will show you:

Vlans Preferred on Active Interface: 1-100
Vlans Preferred on Backup Interface: 101-200

If a link goes down, VLANs that are preferred on that interface will be moved to the other link in the pair. Likewise, when a link returns to service, its preferred VLANs are blocked on the backup and returned to the preferred link.

Be sure to run show interfaces switchport backup detail to see the full status, including link speeds, preemption modes, the MAC address-table move update status.

In summary, the simplicity of Flex Links make it a better choice for carrier and core enterprise networks over the ubiquitous spanning tree protocol. Link-level redundancy is had via STP, but with Flex Links you have more control and better load balancing capabilities. This certainly means that it takes longer to configure since you are planning the layer 2 network manually, but when you need a stable no-surprises link-layer network, Flex Links are definitely the way to go.

Feel free to share!

Hide Sites

Cisco AutoQoS: VoIP QoS for Mere Mortals

WANs often need Quality of Service (QoS) configured to ensure that certain traffic is classified as “more important” than other traffic. Until now, it took a serious Cisco guru to configure a network properly for VoIP if the network was at all bandwidth constrained. AutoQoS, a new IOS feature for Cisco routers, makes deploying VoIP easy, even on busy WAN links. In this article we’ll cover the basics, what AutoQoS does, and some of its limitations.

The first whack at AutoQoS was Cisco recognizing the need to simplify VoIP traffic prioritization. VoIP is especially sensitive to any latency, jitter, or loss, and users will notice problems. To ensure the best possible VoIP call, the network must ensure that lower priority traffic does not interfere with time-sensitive VoIP. AutoQoS can be enabled on both WAN links and Ethernet switches to automatically provide a nice best-practices based template for VoIP prioritization. If you’re lucky enough to have metro Ethernet service, like AT&T ethernet service for example, you should contact your provider to find out if QoS settings on your switches can be duplicated through theirs.

How it Works

QoS allows a router to classify which types of traffic are most important, and ensure that that traffic passed as quickly as possible. If necessary, other traffic will be queued until the higher priority traffic has had a chance to pass. Before a router can know when to queue versus when to attempt to pass all traffic, it must be configured with bandwidth settings for each link.

Configuring QoS on a Cisco router normally involves a complex series of interactions, which require understanding not only the protocols, but a router’s strange way of associating policies. The basic steps are:

• Use an ACL to define which traffic gets matched
• A class-map classifies matched traffic into classes
• A policy-map assigns priorities to the classes
• The policy-map is applied to the interface, which enables the processing of all packets through the ACL, class-map, and policy-map

Each of these “maps” are quite complicated and prone to error. Most sites are going to be duplicating effort because of common problems, like VoIP, needing QoS help.

Why AutoQoS

QoS configuration is not simple. It requires understanding the protocols your network interfaces are using, as well as the type of data you’re passing. To configure QoS for VoIP, for example, you must understand how VoIP works. In short, it requires a guru. If you’re like me, you literally giggled out loud the first time you encountered the word, “AutoQoS.”

AutoQoS enables any network administrator to just “turn on” a solid solution for ensuring VoIP is happy. VoIP is the pain point for most organizations, so that’s what Cisco focused on first, and that’s what we’re focusing on here. Given the limited scope of AutoQoS, it’s believable that it works well enough. In reality, QoS configurations generally classify many types of traffic, and then place a priority on each one.

The main benefit of AutoQoS is that administrator training is much quicker. It also means that VoIP deployments often go much smoother, and upgrading WAN links isn’t usually required. Finally, AutoQoS creates templates that can be modified as needed and copied elsewhere for deployment.

Limitations

Before talking about how to enable AutoQoS, which is literally three commands, let’s talk about where this works best, and what’s required to use AutoQoS.

First and foremost, you can only configure AutoQoS on a few types of router interfaces. These interfaces include:

• PPP or HDLC serial interfaces
• ATM PVCs
• Frame Relay (point-to-point links only)

Cisco catalyst switches also support an AutoQoS command to prioritize Cisco VoIP phones, but you cannot prioritize (using AutoQoS) generic VoIP protocols.

Next, there are some limitations with ATM sub-interfaces. If you have a low-speed ATM link (less than 768Kbps), then AutoQoS will only work on point-to-point sub-interfaces. Higher speed ATM PVCs are fully supported though. For standard serial links, AutoQoS is not supported at all on sub-interfaces. A quick litmus test to see if AutoQoS will work on your desired interfaces or not is to verify that the service-policy configuration is supported. If not, you’ll probably have to reconfigure some links.

AutoQoS will not work if an existing QoS configuration exists on an interface. Likewise, when you disable the AutoQoS configuration, any changes you may have made to the template after the initial configuration will be lost.

Bandwidth statements are used by AutoQoS to determine what settings it should use, so remember that after updating bandwidth statements in the future, you have to re-run the AutoQoS commands.

Making it Work

In the most standard situation, where VoIP isn’t performing as it was promised, the network admin can quickly save the day by running the following on the WAN interface:

interface Serial0
bandwidth 256
autoqos voip

If it’s the local network that needs tuning, the following can be run on Catalyst switches (if running Enhanced Images):

auto qos voip cisco-phone
auto qos voip trust

It really couldn’t be easier than that.  For the WAN example, we told the router that interface Serial0 has 256 Kbps, and to enable VoIP QoS. The switch example is similar, for Cisco phones.

The neat part about this is that AutoQoS is actually doing more than just generating a configuration for you and forgetting about it. If you run the command show autoqos interface s0, you will see much more than just your standard old interface configuration. It will show that a Virtual Template “interface” has been created, and that a class is applied to the interface. The same output will also show you the configuration of the template and class-map, with an asterisk next to each entry that was generated by AutoQoS. It’s actually keeping track of what was done automatically so that you can learn what AutoQoS is doing. As mentioned previously, however, don’t forget that removing the AutoQoS configuration will destroy all QoS settings on an interface, not just the ones that AutoQoS configured.

Finally, remember to enable QoS on both sides of a WAN link to truly prioritize VoIP packets. Don’t forget to read through the Cisco documentation before deploying it, even though AutoQoS is simple, in comparison. It is simple, but the more prepared you are the easier it is to deploy.

Cisco will hopefully continue this trend of providing Auto features for complicated, but common tasks. AutoQoS for VoIP sure does enable a much larger audience to correctly deploy VoIP over a wide variety of networks.

Feel free to share!

Hide Sites