Raspberry-Pi Emulation on OSX Maverick

Here's what did to emulate a Raspberry-PI on OSX Mavericks (v10.9.3)

 1. Install qemu
 2. Setup emulator (http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/)
 3. Make a golden image
 4. Copy WIP image to SD card

Install qemu

You will need homebrew installed, then simply run:

> brew update ; brew install qemu

If you have problems, maybe fallback to this blog post that I used before homebrew had a qemu formula.

Setup emulator

Take a look at this blog post from xecdesign.com. Follow the instructions for "Preparing the environment", "Check that you have everything" and "First Boot".

The key points are to create a working directory and download the kernel and an .img file to it.  The .img file should the one you want to boot/configure.

Here is the command that I used to launch my image for the first time:

> qemu-system-arm -kernel kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" -hda 2013-12-20-wheezy-raspbian.img

Make a "Golden Image"

The "First Boot" section of that blog post mentions changing some stuff.  As mentioned, I had to comment out the libcofi_rpi.so line in /etc/ld.so.preload, as well as, creating the new file /etc/udev/rules.d/90-qemu.rules, then halt the system.

Now that my .img is a good base image for the emulator and my PI,  I treat it as a "golden image" (2013-12-20-wheezy-raspbian.img in the above example).  As such, my process is to make a copy named wip.img and use this copy for subsequent boots.  Using wip.img allows me to simply delete it if I foobar something  -- then I can start over with a fresh copy from my golden image.  But this is simply a suggestion.

Now lets boot out image and start configuring it!  I do so with the same command as above, but I point it to my WIP image:

> qemu-system-arm -kernel kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" -hda wip.img

Once you are done hacking on it, then halt the system and copy the image to an SD card.

Copy image to SD Card

Once you get the image on your mac configured the way you want it.  Copy your .imf file to an SD card and see if it will boot on your PI.

   > sudo dd bs=1024 if=wip.img of=/dev/disk1
 > sync

Credits

Special thanks to http://xecdesign.com/ and http://mikelev.in/ for their tips that got this going!

Passing values from one Chef resource to another

Overview

In a Chef recipe it is sometimes useful to set the value of an attribute in one resource, and then be able to use that value in another resource.  In scripting languages, it is typical that a function will return some value which is then passed to another function.

In the Chef language doing this is not as simple as it sounds -- one cannot simply pass values from one resource to another.

Below I give an example of the typical misconception I've seen and then show you a way to achieve the desired results.

What you might expect 

Here is how one might expect things work:


However, the resource on line 15 will log "!!!!set in compile phase!!!!".

Why? All resource attributes are evaluated in the "compile" phase -- almost like a "pre-processor" step in C.   In other words, the attribute values for each resource were already set before the recipe even started to execute.

This can be a little hard to get your head around if you are just learning Chef.  For more information about the "compile" phase see Anatomy of a Chef Run.

Manipulating the Resource Collection

To achieve the desired results, we're going to rely on the inner-workings of Chef -- namely the resource_collection.  You can think of the resource_collection as a list of all the Chef resources in your run-list that are compiled and ready to be executed.

We will insert a new helper resource into our example recipe which will perform the value passing for us.  Here is our new recipe:


This time, the resource on line 23 will log "!!!!set in converge phase!!!!".  Profit!!

As you can see, the new ruby_block[update_log_message] in step #3 performs the lookup of the log[update_target] from the resource_collection and updates the message with our new attribute value.

That's all there is to it!

Make it idempotent

Now that you get the idea, let's preform our due diligence to make this idempotent -- like every good Chef should:


On line #7, we add an only_if so the ruby_block will only execute if our attribute is still the default value.

We also set the action on line #24 to :nothing, so our helper resource will only execute if the notifies clause on line #13 is triggered. 

In Closing

This idea of modifying the resource_collection is nothing new -- some of the old school (Iron?) Chefs have been doing it for years.  This idea is also the basis for Brian Berry's chef-rewind resource used in the library cookbook pattern outlined in his "How to Write Reusable Chef Cookbooks, Gangnam Style" post.

I just haven't found much documentation explaining how to do it, nor for it's use for solving this particular problem.

If there is another way to solve "passing" values or a technique that avoids the need to pass values, I would love to hear about it.

Follow up

Thanks to Noah and Kannan for the review and the feedback.  You can find some other use cases for this trick here.

I am excited to see that the delayed attribute feature will be available in Chef v10.28 and v11.6.

Also here is the code that I used in the examples -- incase you want to play with it yourself.

Cheers!

Synergy gets encryption

As of version 1.4.11, Synergy, my favorite software keyboard-mouse switch, now encrypts traffic between machines.  A much needed security feature!! See announcement for more information.  Nice work @NickBoltonUK!

The RightScale plugin for Knife

Ohai!

Here is a brief demo of how one can provision Chef Clients with RightScale using the knife command-line utility.  To do this I will use the knife-rightscale plugin.  At the time of this writing there is just the first beta release available v0.0.2.

This post is intended for chef developers with existing cookbook collateral that want to experiment with  RightScale while leveraging their existing Hosted Chef or Chef Server for client configuration management.

Motivation

More and more the knife utility is becoming an integral part of continuous integration/delivery toolchains.  It is easily invoked from your Jenkins/CI server to provision staging infrastructure on IaaS clouds before running integration/regression tests.  There is already a wide array of cloud plugins to provision servers (e.g. ec2, rackspace, etc.), but leveraging RightScale for provisioning you also get many services/features that you don't need to create or maintain yourself (like monitoring, auto-scaling, and usage reporting just to name a few) -- so you can focus on deploying your company's services.

Requirements

In this demo, I will be using Hosted Chef for my Chef Server -- because I don't want to manage one myself and would rather let the experts do that.  If you don't already have one you can sign up for a free trial of Hosted Chef from Opscode here.

Likewise, you will need a RightScale account with at least one cloud registered.  You can
sign up for a free trial account here.

If you want to following along with this demo, you might also want to take a look at the source so you can cut-and-paste the commands from the README.

Installation

On your local development machine, be sure you are running the latest version Chef 10. Versions earlier than 0.10.0 don't support plugins.   You can install the Chef that I tested with using:

    gem install chef -v 10.24.0

I have not yet tested with Chef 11, but feel free to try!  Now we need to install the plugin. To install it, simply run:

    gem install knife-rightscale

Depending on your system's configuration, you may need to run this command with
root privileges.

Provisioning a Chef Client

As mentioned in the requirements, you will need at least one cloud registered in your RightScale account.  If you only have an EC2 account please see "the fine print" section below, since you will need a feature enabled in your account.

Once the gem is installed, I can list the clouds available in my account:

Okay, most folks wont have this many to choose from, but I work at RightScale and we haz all the clouds.  For this demo, I will randomly choose the "HP Cloud".

Now I need to find my "Chef Client" ServerTemplate.  A ServerTemplate is analogous to a versioned Chef role associated with specific cloud images (among other stuff) that the role has been tested with.  To find the "Chef Client" ServerTemplate I use the following command:

The --name option finds all ServerTemplates with names that contain the string "Chef Client".  I am going to use the latest revision of the v13.4 client template, which is at Revision 4.

NOTE: If you are following along with your own account, the command above wont find anything the first time.  You will need to first import it into your RightScale account -- get it here.  After that the command should find it.

Now that we have identified our cloud and ServerTemplate we can create a new server with the following command:

Notice I pass in the ServerTemplate ID and not the name -- this is to be specific about which version of the template I want.  I also pass in the cloud name, the deployment, and the name I want for the server "CP:ChefClient".  I also specify some inputs for the ServerTemplate including the node name "MyChefClient" and comma separated list of roles (although I just specify one -- "hello_world").  The validation_name and validation_pem inputs reference I am using "cred:" type inputs.  You can just paste your key material on the command-line, but I suggest you login to your RightScale dashboard and create these credential objects to hold your validation secrets.

Here is the output I see as I wait for the server to be ready...

During this time the server is provisioned, RightScale bootstraps you chef client and performs it's first converge.  Once the cloud gives the instance it's network the info is displayed.

Now we can SSH in...

And tail the chef log...

Or view the server in the RightScale dashboard:

When you are done with the server just delete it:

The fine print

I was lazy and only added support for the RightScale API 1.5 (since API 1.0 is EC2 only and being deprecated).  As such, this plugin cannot currently provision servers on EC2.  If you need this capability and would like to take part in a private beta to enable that functionality, please contact support@rightscale.com and they will hook you up.  If you have any problems, please send me an email directly.

Cleaning up a virtual image before performing a Bundle

Below are instructions to remove ssh keys, logs, rightlink state, etc. before bundling a running VM into an image. This is typically a good idea, especially if you plan to share you image with others.  

DISCLAIMER: This code should work on CentOS, RHEL and Ubuntu.  See notes at bottom of post before running.  This post was written specifically for cleaning RightImages, but should apply to custom images as well.  This is alpha code provided with no warranty, YMMV.

That being said, feel free to post questions or issues and I will do what I can to help.

Cleaning a virtual image

Download clean_for_rebundle.tar

Step 1: configure VM the way you want it

Step 2: clean up the system before image "bundle" using attached script

vagrant@lucid32:/vagrant/data$ sudo cp clean_for_rebundle.tar /root/

vagrant@lucid32:/vagrant/data$ sudo -i

root@lucid32:~# tar xvf clean_for_rebundle.tar 

clean.sh

functions.sh

truncate.sh

root@lucid32:~# ./clean.sh

Step 3: rebundle image using cloud API or "bundle" button on rightscale dashboard.

Step 4: ??

Step 5: profit!

Notes

Just some extra info that may be pertainate to the image you are attempting to clean...

RightLink 5.7 

The tar file contains a clean.sh file for RightScale RightLink 5.8 images.  Replace this file with the contents of clean_rightlink5_7.sh if bundling a VM running on an older rightimage.  

CentOS/RHEL 

you must first have lsb_release utility installed.  To install run:

> sudo yum install -y redhat-lsb


Good luck!!

Source code project here https://github.com/caryp/clean_for_rebundle

Hacking the Promise SmartStor NS4600

I had success exploiting the MLDonkey hack to get telnet access working for the admin account, but had to repeat the commands every time I wanted telnet access which became a pain (especially since I don't leave BitTorrent running)

I spent a little time digging into how this box works and added some notes below, but for those who just want to enable telnet for good, skip down to 'The Script' section.


Disclaimer: if you change things on your NAS you have the very real chance of turning it into a bookend.  This will definitely void your warranty from Promise, so use at your own risk!  

The Details

This box had three flash memory devices:

/dev/mtdblock4

Is a flash device which contains a file named usr_sqfs.  This file contains a squashfs loopback filesystem.  This file gets mounted read-only at /usr and contains the /usr/sbin/chkhttpd shell script which rewrites /etc/telnet.user.  This script get run every minute as a cron job and is why the telnet.user keeps getting disabled.

/dev/mtdblock6

Is also a flash device which contains a file named app_sqfs.  This file gets mounted read-only at /promise.  This directory looks to contain mostly the webapp for system configuration and other apps -- not real interesting for the goal of permanently enabling telnet.

/dev/mtdblock5  

Is yet another flash device that gets mounted directly to /dev and is read-write.  Files from here overwrite files in /etc and /usr on boot. This is definitely interesting.

The Idea

Since we can't change /usr/sbin/chkhttpd directly, the plan is to utilize cron to help us undo the telnet restrictions.  We do this by placing a script somewhere on the filesystem and modifying /data/usr/crontab (which is on the writable flash drive) to run our script right after the /usr/sbin/chkhttp script is run.

The Script 

This creates an executable script to enable telnet access to the admin user.  Cut & paste this script into your telnet terminal. This will add the admin_telnet.sh script to your filesystem. Then add the following line to /data/etc/crontab:

* * * * *       root    /VOLUME1/homes/admin/etc/cron.d/admin_telnet.sh >/dev/null 2>/dev/null

After the chkhttpd line.  This will ensure the script when needed.  To test, reboot your NAS and verify that telnet is still enabled. 

Good luck!

Working with Widgets

Had some time to play around with the new RightScale dashboard widgets. Now, another group worked on these UI treats, so I am not patting myself on the back when I say that these things are super cool!

I used a combination of built-in widgets and some that others have written to make a custom landing page that gives me a great overview in a glance.

Doing R&D I've got so many servers launching and terminating throughout the days, weeks, months... sometimes it is hard to keep track.  Finally when things do slow down I find a server or two that's been running for two months -- doing nothing!!  So I decided to write my first widget.  Basically it visualizes my running servers and their up time.  It looks like this:

Yup, pretty basic. But now I won't forget to shut down that long running server again.

The expanded view gives you probably all the other info you might want to know: ServerTemplate, Deployment, CPU graph, etc.  The only trick, is that you need to add your initials to each server you launch (a good practice to get into in a multi-user account).

AFAIK, there's no real way to share these outside your account, so here's the code.  Add here are the other settings:

I hope you find these things as useful as I do.  The RightScale widgets are implemented using Liquid markup.  Find more information about what you can do with RightScale widgets on the support site.