Follow me on Twitter:

Connecting to existing buckets in S3 with boto, the right way

Posted: January 23rd, 2012 | Author: | Filed under: DevOps | Tags: , | No Comments »

Here’s another interesting tidbit.

If you have scripts that connect to S3, and you run out of buckets (Amazon only allows 100 buckets per account), you might get a nasty surprise.

See, you may have been using create_bucket(name-of-bucket) to get your bucket object. It’s undocumented as far as I can see, but apparently if you use create_bucket() on an bucket that actually exists, it’ll return the Bucket object. That’s handy! Except it breaks if you’re unable to create more buckets (even though you aren’t really trying to create more). Sigh, so I refactored as such:

# old and busted: bucket = s3_conn.create_bucket(bucket_name)
# new hotness:
# iterate over Bucket objects and return the one matching string:
def find_s3_bucket(s3_conn, string):
    for i in s3_conn.get_all_buckets():
        if string in i.name:
            return i
Used as: bucket = find_s3_bucket(s3_conn, bucket_name)

There is likely a more elegant way, but hey this works.


No Comments yet... be the first »

Finding instances by name with boto in python

Posted: January 22nd, 2012 | Author: | Filed under: DevOps | Tags: , | No Comments »

OK, I know I need to blog more. Rather than think I don’t have anything useful to say, I’ll start adding quick entries of what-I-learned.

Random tidbits from today:
I got annoyed with EC2 failures, and having to manually terminate and redeploy instances today, so I finally worked on a script I’ve been meaning to write. One thing I had to figure out (which isn’t all that complex), is how to discover an instance by name.

If you tag an instance with the hostname you’re using in your deployment script, you don’t need to fumble in the AWS console to find an instance ID. Ever. I don’t find it acceptable to manually click around or run scripts to discover information that’s available from an API 🙂

So, to “find” the instance using python and boto (assume AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are defined in your shell environment):

import boto
ec2conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
my_fqdn = "example.com" # trailing part of my domain

Now, ‘instances’ can be iterated over to find instances with the name you desire. I wrote a little wrapper function to do this, and it returns an instance object (which I call instance.terminate() on, for this purpose). Code:

def find_instance_by_nametag(instances, name):
    # support short or full hostname usage
    if not my_fqdn in name:
        name = name + my_fqdn
    for i in instances:
        if "Name" in i.tags and name in i.tags['Name']:
            return i
    sys.exit("sorry, I couldn't find an instance with that name!")

Easy as that!


No Comments yet... be the first »

Display current location on your web page using the SPOT GPS tracker

Posted: August 3rd, 2011 | Author: | Filed under: Tricks & Tips | Tags: | 1 Comment »

 

UPDATE: Ignore all the crazy shit below. Just use this script:

https://gist.github.com/manos/8359450

It fetches your tracks and saves them in a .json file, as well as a plain text file containing the last latitude,longitude. The below stuff may still be helpful for how to get a Google map displayed on your site… there’s got to be better ways though 🙂

The SPOT GPS tracker is not only a lifesaver, but also a handy tool for motorcycle (ok, and other types) travelers. If you subscribe to the Track Progress service, you can tell the GPS device to send your location to SPOT via satellite every 10 minutes, and then export those tracks at a later time.

They even have an API from which you can fetch an XML document with all your current tracks! The unfortunate part, and the reason for this post, is that they only keep 30 days of GPS coordinates. It’s not a problem if you export your data to Spot Adventures and create an “adventure” — that will live forever. But if you wish to present “my current location” on your personal web page, for example, you’re out of luck. It will only work as long as you’ve used the track progress functionality within the last 30 days.

I guess you need to cache the last used location yourself. OK, this shouldn’t be hard. This is how to do it with a little perl script (download here, see in action in CharlieTracker here):

#!/usr/bin/perl
 # Author: Charlie Schluting <[email protected]> (c) 2011
 #
 use XML::Simple;
 #use strict.. heh, no, this barely works.</code>
 $CACHEFILE = "/home/charlie/lastspotlocation.txt";
 $XML = "/home/charlie/spot.xml";
 $JS = "/stash/www/charlierides.com/files/map.js";

The CACHEFILE is where I store the last known latitude and longitude. It’s mostly for reference, so that other things (aside from this script) can use it.

XML is the location I store the fetched XML from SPOT’s API.

JS is the file I write the javascript out to. OK, you probably don’t need to care about this part of the script.. but here’s the details: if you use joomla or wordpress, you might not want to enable PHP code execution or other evil things like that. So, in order to include a snippet of generated HTML, the only choice (as far as I know, aside from editing the DB where a “display html” module stores its data), is to write javascript to include another javascript file. So that’s what I do.. the script actually generates javascript that when run, will spit out html. If you don’t care about that and just want the map part, read on.

# hahaha, oh man..
 `wget -q -O $XML http://share.findmespot.com/messageService/guestlinkservlet?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW`;

This part fetches the XML file from spot, using my shared page identifier.

# create object
 $xml = new XML::Simple;
 # read XML file
 $data = $xml->XMLin("$XML");

# this is how we overcome spot's API not keeping >30 days. If they've aged out, do nothing (i.e. keep using the old data).
 die("No messages found, totalCount is 0, ABORTING LIKE AN UGLY KID") unless $data->{totalCount} > 0;

# the first object is always the most recent:
 $lat = $data->{message}->[0]->{latitude};
 $long = $data->{message}->[0]->{longitude};

# just because (hey, what if something else wants to use this?)
 open(FILE, ">$CACHEFILE");
 print FILE $lat . "," . $long . "\n";
 close(FILE);

And the rest (above) is pretty self explanatory. Using just this part of the script, you’ve overcome the annoying limitation that is SPOT Track Progress (losing your last known location). Your last known coordinates will be in CACHEFILE. For completeness, I will include the rest of this horrible hack I used:

# ugly shit that writes out javascript to write out html, to include a linked static google maps image
 $googleoptions = "&zoom=8&size=140x152&sensor=false&maptype=hybrid";
 $googlelink = "<a target=\"blank\" href=\"http://maps.google.com/maps?q=" . $lat . "," . $long . "+(charlie)&z=8&t=h\"\>";
 $header = "<p>Current location (since the last GPS update):</p>";
 $js = 'document.write(\''. $header .'\');' . '
 document.write(\''. $googlelink .'\');' . '
 document.write(\'<img src="http://maps.googleapis.com/maps/api/staticmap?center=' . $lat .
 ',' . $long . $googleoptions . '" />\');
 document.write(\'</a>\');
 document.write(\'<p>Or <a href="http://share.findmespot.com/shared/faces/viewspots.jsp?glId=0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW"
 target=blank>view all recent tracks.</a></p> \');
 ';
 open(FILE, ">$JS");
 print FILE $js;
 close(FILE);

So, what this writes out (in HTML, finally), is an IMG tag embedding a static google map (because I’m using a small thumbnail), which links to the google maps page when clicked:

Map: http://maps.googleapis.com/maps/api/staticmap?center=45.42553,-122.52251&zoom=8&size=140×152&sensor=false&maptype=hybrid
Link target: http://maps.google.com/maps?q=45.42553,-122.52251+(charlie)&z=8&t=h

The parameters you can pass google maps are well documented, so I don’t need to rehash them here. You may even prefer to use the real maps API, rather than the static image one. That too, is well documented.

Run this from cron every 15 minutes, and you’ll always have your last checked-in GPS coordinates!


1 Comment »

Shell programming techniques and tips

Posted: March 25th, 2011 | Author: | Filed under: DevOps, Linux / Unix, Tricks & Tips | Tags: | No Comments »

Plenty of shell programming tutorials exist already; this isn’t yet another “howto.” We’re going to spend a little time talking about some of the frequently unused or misunderstood techniques in shell programming, and also cover some neat tricks involving the shell that may not be obvious to novice users.

For the most part, we’re talking about the bourne shell, because it exists everywhere and is compatible with bash. Bash does have some interesting features, but for our purposes the bourne shell will do just fine.

1. Implementing a lockfile 
Every once in a while we’ve got a shell script that needs to run, but dangerous things can happen if we’re running two copies at the same time. The simplest method is to just bail out if a lockfile exists, but you can also implement blocking. Here’s a basic example of checking for a lock file and then bailing out:

#!/bin/sh
LOCKFILE=/tmp/mylock
if [ ! –s $LOCKFILE ]; then
      echo $$ > $LOCKFILE
      # do stuff; the bulk of the script is here
      : > $LOCKFILE 
exit 0
else
      echo “PID `cat $LOCKFILE` running” | mailx –s “$0 can’t run” root
      exit 1
fi

Of course you can get much fancier. If this was inside a while loop, you could sleep a few seconds and then retry the lock. The above example uses ‘test’ with the bracket notation to check if the lockfile exists, and that it isn’t empty (the -s option to test). If the return value of test is true, the block of code runs and puts the script’s current PID into the lockfile. At the end of the “do stuff” block of code, which will probably call another script, we truncate the lockfile by using the null command. If the lockfile is non-empty, we send mail to root and bail out with an unsuccessful return code. The subject of the message includes the name of the script ($0), and the body of the message indicates which process ID is currently running.

2. Check return values, always

If you call an external program, or another script of your own, you must always check the return value of that program. Unexpected failures of commands can cause the rest of your script to misbehave, so you need to make sure everything ran properly. The bourne shell has a built-in variable $? that holds the return value of the last command. See item number three for an example.

Most importantly, you *very often* want to stop executing your script if any command fails. Simply toss a ‘set -e’ at the top of your script, and any command that returns non-zero will result in your script exiting.

3. Using return codes

Remember, the ‘if’ statement uses the return value of the statement immediately following ‘if’ to determine whether or not it should succeed or fail. So the test command can be used (with bracket notation), and so can any other command.

if [ “$?” -eq ‘0’ ]; then echo yay ; fi

The above statement will print “yay” if the last command executed returned success, else it does nothing. Remember that ‘0’ in the shell is success; it’s the opposite of when you’re programming in C. We also collapsed everything onto a single line here. The parser doesn’t see it that way, though, since the semi-colon represents a newline.

You can also put commands inside if statements. Let’s test to see if a list of machines ping before we try to scp the known_hosts file to them (Solaris ping syntax):

#!/bin/sh
for host in `cat ./hostlist.txt`
do
  echo "doing $host.."
  if ping $host 1; then
  scp /etc/ssh/ssh_known_hosts2 $host:/etc/ssh
fi
done

4. Capturing output from multiple commands

Frequently, people who want to capture output from multiple commands in a script, will end up appending output to the same file. You can use a subshell instead, and redirect all stdout (or stderr if you need to) from the subshell:

#!/bin/sh
( 
cat /etc/motd
cat /etc/issue 
) > /tmp/motd-issue

5. Other subshell tricks

Changing directories, and then executing a command, can be very useful when you’re piping stdout over ssh. It’s useful for local commands too. Using tar to copy the present directory into /tmp/test/:

tar cf - . | (cd /tmp/test && tar xpf -)

Note that if the cd fails (directory doesn’t exist), then tar will not be executed.

Parallel execution:
In our previous scp example, it would have run very slowly with a long list of hosts. We can make each scp command execute almost in parallel, by backgrounding the subshell process. The following portion of the script will burn through the loop very quickly, because the subshell is backgrounded, and therefore the commands run and the shell doesn’t wait for them to complete before looping again.

if ping $host 1; then
(
scp /etc/ssh/ssh_known_hosts2 $host:/etc/ssh
scp /etc/ssh/ssh_known_hosts2 $host:/etc/ssh
) &
fi

 

7. Piping data and commands over ssh

There are many examples of how piping things over ssh is useful. You can print remotely: cat file.ps | ssh $host lpr Which works because ssh sends its stdin to the stdin of the program you tell it to execute.

You can use tar to copy a directory to a remote host, over ssh:

tar cf - ./dir | ssh $host “cd /dir/ && tar xpf –“

And finally, you can execute scripts without having to manually copy the script to the host:

cat script.sh | ssh $host /bin/sh

8. Here documents

If you want to run commands over ssh in a long argument, similar to #7’s tar copy, you must take care to escape any special characters, or the local shell will try to use them. A quick and dirty way to avoid this is with a here document:

ssh $host <<\EOF
#!/bin/sh
# Entire script goes here, on multiple lines
EOF

If you omit the backslash before EOF, variable substution is also possible. Here documents are frequently used to embed a script within a script too. As a matter of fact, some software vendors distribute their software as a shell script, and the tar file containing the binaries is actually a here document. To extract a here document to /tmp/file, just use this in the script: cat <<EOF > /tmp/file and then start the document on the next line. You can use any string you want, EOF is just the standard.

9. Live shell programming

Remember how we said that the semicolon is used to indicate a newline? This implies that scripts can be written on the same line, excluding here documents. Here’s a quick example of a command I recently ran:

ssh $box "if test \`uname -r\` = '5.10'; \
then "mv /etc/dt/config/Xresources /etc/dt/config/C/ && \
/etc/init.d/dtlogin reset" ; fi"

You can add all kinds of shell logic here, and the backslash is useful to start typing on a new line–the shell won’t execure the command until you hit return without a backslash at the end. Just remember to quote things that your local shell will try and interpret. Generally people will use && in succession with multiple commands, or if you want the second (or third, or…) command to execute regardless of the return value of the previous command, just use the semicolon.

You can also just type “for” at the command prompt, and it will bring you to a new line waiting for input. An entire shell script embedded inside a loop can be written in real-time in this manner.

Knowing all these shell tricks certainly makes one’s life easier, but you really shouldn’t be spending all your time running commands on remote hosts. Stay tuned for more posts about configuration management 🙂

 


No Comments yet... be the first »