Wednesday, December 30, 2009

Approaching the End of the Decade

Did 10 years really just went by?!? I still remember when we were gearing up to tackle Y2K and we're now approaching the end of the first decade of the millennium.

Sunday, December 27, 2009

Restoring RPM and YUM on Fedora after an accidental yum remove rpm

By default there is nothing that prevents an user from accidentally removing both yum and rpm from a Fedora system and once removed it is very painful to try to put them back. If only yum was removed it would be a little easier to restore. Yum can be reinstalled through rpm using the yum-*.rpm package downloaded from Fedora or the install disc. Without RPM, it's like a chicken-and-the-egg problem. The packages from Fedora and install discs (including the rpm package itself) are .rpm files and needs rpm to install.

Today I found myself in the situation where both YUM and RPM got removed by accident. I was following Fedora's instructions on finding and removing orphaned packages after upgrading and had yum remove some Fedora 11 packages that was left over. Being distracted by other things at the time, I noticed too late that removing those F11 orphaned packages also led yum to remove a bunch of F12 packages including rpm and yum.

So, what does one do in this situation besides "backup and reinstall? which I wanted to avoid if at all possible. It took me awhile to figure out but I was get rpm back on my system and restore it to its earlier state. Here are the steps I took.

My first inclination was to try to compile RPM. However, this was a FAIL since it required additional packages that I didn't have installed and I didn't have a package management system... Instead, I downloaded the rpm-*.rpm package from Fedora and was hoping to extract the files so I can use them to reinstall rpm and yum. To extract files out of an RPM package requires... rpm! Oh, the irony. There is a rpm2cpio program that will do that but that is available in . . . the rpm-build-*.rpm package! Fortunately, there are a lot of scripts available that will do it including some shell scripts. The Fedora RPM docs included some scripts and I downloaded the bash script version but it didn't work. After a little more digging, I found out that Fedora changed the compression scheme it uses in F12 from gzip to xz, so I modified the shell script as follows:


#!/bin/sh

pkg=$1
if [ "$pkg" = "" -o ! -e "$pkg" ]; then
echo "no package supplied" 1>&2
exit 1
fi
leadsize=96
o=`expr $leadsize + 8`
set `od -j $o -N 8 -t u1 $pkg`
il=`expr 256 \* \( 256 \* \( 256 \* $2 + $3 \) + $4 \) + $5`
dl=`expr 256 \* \( 256 \* \( 256 \* $6 + $7 \) + $8 \) + $9`
# echo "sig il: $il dl: $dl"
sigsize=`expr 8 + 16 \* $il + $dl`
o=`expr $o + $sigsize + \( 8 - \( $sigsize \% 8 \) \) \% 8 + 8`
set `od -j $o -N 8 -t u1 $pkg`
il=`expr 256 \* \( 256 \* \( 256 \* $2 + $3 \) + $4 \) + $5`
dl=`expr 256 \* \( 256 \* \( 256 \* $6 + $7 \) + $8 \) + $9`
# echo "hdr il: $il dl: $dl"
hdrsize=`expr 8 + 16 \* $il + $dl`
o=`expr $o + $hdrsize`
dd if=$pkg ibs=$o skip=1 2>/dev/null | xz -d


With this script, I extracted the rpm (rpm, rpm-build, rpm-libs, rpm-devel, rpm-python) files to a temp directory and install the files manually:


rpm2cpio.sh rpm-*.rpm | cpio -d -i
tar cf - ./usr ./etc ./bin ./var | (cd /; tar xvf -)


This installs the files onto the system but doesn't update the rpm databases, so I did a 'rpm -ivh rpm*.rpm' in addition to the dependent packages they needed.

With RPM installed, I was able to install YUM . I had kept a list of all the files that got removed earlier so I had yum reinstall all those packages and my system is now restored.

Upgrading to Fedora 12 (part 2)

It took awhile but Fedora upgraded itself and booted in to the GUI without a hitch. Everything seems to be running even with both the LCD TV and monitor hooked up (past upgrades would forget which was the primary display). Then I noticed that there was no audio. My first thought was that somehow Fedora 12 didn't configure the audio correctly (uh oh) which wouldn't be the first time that a piece of hardware worked in a previous version but not the upgraded version. Fortunately, I thought to just double check the volume setting and noticed that the upgrade set the audio-out to MUTE for some reason. Unchecking that re-enabled the sound.

Upgrading to Fedora 12 (part 1)

I decided to take the time off from the holidays to upgrade from Fedora 11 to Fedora 12. My past few upgrades of Fedora have all hit some sort of snag some of which I felt were real obstacles that the typical computer user would find it very difficult to overcome.

For this upgrade, I was ambitious and decided to try the live upgrade feature that was introduced in Fedora 10. By running "preupgrade", Fedora will download all the packages needed to upgrade to the new version in the background so the system can still be used. When everything is downloaded, the system will reboot, install the packages and then the system will be updated. That's the theory at least. This method of upgrading should be faster since it only downloads the packages needed instead of every package. Given my complaint from the last post where I downloaded the entire DVD iso then having to download nearly another DVD worth of packages on the first 'yum update', this method would address that since it will bypass the first part.

The instructions for using using Preupgrade is on the Fedora wiki. Essentially, it is suppose to just be running the command 'preupgrade' and everything else is taken care of automatically. That was kind of accurate. I ran the command and immediately ran into the most common bug with F12 upgrade. I used method 1 to work around it, but I didn't have any extra kernels so it was the tunefs that got me the extra space I needed. Seriously does the Fedora team really think that consumers would be comfortable having to do this in order to upgrade an OS? What Fedora did very well is that it warned me about the problem and none of it effected the current OS and as it promised, I was able to continue to use the system.

Okay, with that out of the way, it completed the download and it was now time to reboot. This is the stage where the system is not able to be used as it install the downloaded packages. However, at reboot, the system just hanged with the monitor showing "GRUB" and a blinking cursor... The upgrade corrupted GRUB and in order to fix it would require re-installing GRUB. The existing linux system is still there unharmed, the boot process just never made it that far. The irony of this bug is that "preupgrade" was meant to avoid having to create a bootable Fedora disc, but a bootable disc is needed to reinstall GRUB. I dug out my Fedora 11 install disc and booted into rescue mode. The following will install GRUB back to the system:


chroot /mnt/sysimage
grub-install /dev/sda # change /dev/sda to the drive you have your /boot


Rebooting after this, the system booted back into Fedora 11...?!? I guess I was able to confirm that so far my F11 instance is still completed unharmed, but this is suppose to be an automated upgrade to F12. I looked at the grub.conf file and it was suppose to give me the option to do a F12 Upgrade. So I rebooted again and made sure that I selected that option in GRUB before it booted back to F11.

Now the system is installing the package so let's see how it goes. I took a bigger chance this time by keeping both my LCD TV and USB drive connected. In the past, I already had to disconnection in order to upgrade successfully, so let's see what happen this time.

Tuesday, December 22, 2009

Upgraded to Thunderbird 3

With most of the attention on the Google Chrome browser release for OSX and Linux, the release of Thunderbird 3 got overshadowed. While some feel that stand-alone email readers are becoming obsolete, I still believe that a dedicate email client is very useful especially for power users. That said, the convenience and advancements of browser-based email such as GMail have come such a long way that we're not far from the time (if we're not already there) when email clients compliments the web version rather then the other way around.

Upgrading from TB2 to TB3 was painless. Just install the application and it handled all the upgrades. It asks whether you want to switch to using some of the new interface features or stick with the previous ones and then it will start to create the search index for all your email. Search was definitely one of the weaknesses of previous TBs especially when you compare it to Gmail's search, but that seems to be one of the areas that TB3 really tries to improve on in this release. I think TB3 also took the right steps in keeping itself relevant by integrating better with web-based mail system. The one shortcoming is that it still doesn't integrate directly with online contacts that you might have with Yahoo or Google. TB3's interface is also a little more Firefox-like. Desktop clients have some advantages of running on the OS so I still feel that they should continue to use that advantage. It isn't a radical change, though, so it might be that good balance between the bringing users some familiarity with browsers while still exposing email features.

I need to play with it more but so far I tip my hat to the team for this nice release.

Thursday, December 10, 2009

Programmatically authenticating to Google App Engine with Python

Even before joining Google, I was already a fan of two of its services: Google App Engine and Google Apps for Domains. Apps itself brings a lot of value and combined with app engine, it's especially powerful for small business and start-ups. Apps provides the most essential IT pieces for a business (email, office suite, calendaring) and Google App Engine (GAE) allows the business to build their own software on Google's infrastructure without the heavy initial investment in hardware and IT infrastructure. This post focuses on just GAE specifically how you can programmatically access app engine from a python script.

The code to authenticate against the production instance of GAE is actually fairly well documented by Google on the GAE site. The issue I ran into is that before I push my code live, I wanted to test it on a development instance. GAE makes setting up a development server easy and comes with its own web server and datastore (database), but the one area where the development environment is different from production is how authentication is handled. The development server doesn't do real authentication. It just sets the right cookie that simulate that you've already logged in. I wanted to write my code with minimal differences between how it calls production and how it calls the localhost. Authentication to your application on production means that you need to get a token from the Google servers, but this doesn't exists on the dev environment. Looking at how the developer instance handled the login screen, I was able to do the following:



#!/usr/bin/python

import cookielib
import urllib
import urllib2

# Setup to be able to get the needed cookies that GAE returns
cookiejar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
urllib2.install_opener(opener)

# Set 'prod' based on how your system determines whether to connect to prod or localhost.
if prod:
# This is the setup to construct the login URL for authentication on prod.
authreq_data = urllib.urlencode({'Email': '',
'Passwd': '',
'service': 'ah',
'source': '',
'accountType': 'HOSTED_OR_GOOGLE'})

# Authentication server
token_uri = 'https://www.google.com/accounts/ClientLogin'

# This is where you want to go after log in. Replace:
target_uri = 'http://auth_example.appspot.com/home'
## Get an AuthToken from Google Accounts
auth_req = urllib2.Request(token_uri, data=authreq_data)
auth_resp = opener.open(auth_req)
auth_resp_body = auth_resp.read()
auth_resp_dict = dict(x.split('=')
for x in auth_resp_body.split('\n') if x)
authtoken = auth_resp_dict['Auth']

authreq_data = urllib.urlencode({'continue': target_uri,
'auth': authtoken})
login_uri = ('http:///_ah/login?%s'
% authreq_data)
else: # authenticate against the local dev server
target_uri = 'http://localhost:8080/home'
authreq_data = urllib.urlencode({'email': 'test@example.com',
'continue': target_uri,
'action': 'Login'})
login_uri = ('http://localhost:8080/_ah/login?%s' % authreq_data)

# Okay, we're done at this point with the difference. From this point everything else
# should be the same for either prod or dev.

# Do the actual login and getting cookies.
serv_req = urllib2.Request(login_uri)
opener.open(serv_req)

# Rest of code here.


That all there is to it!

.vimrc

I've been trying to set up my VIM environment to be more comfortable for me so I can be more productive. It's getting better although I still miss having a shell inside the editor that EMACS provide. Here is my current .vimrc:


" Enable loading filetype and indentation plugins
filetype plugin on
filetype indent on

" Turn on syntax highlighting
syntax on

" Global settings
set autowrite

" Allow backspacing over everything
set backspace=indent,eol,start

" Insert mode completion
set completeopt=menu,longest,preview

" Use UTF-8 as the default buffer encoding
set enc=utf-8

" Remember up to 100 'colon' commmands and search patterns
set history=100

" Enable incremental search
set incsearch

" Always show status line, even for one window
set laststatus=2

" Jump to matching bracket for 2/10th of a second (works with showmatch)
set matchtime=2

" Don't highlight results of a search
set nohlsearch

" Enable CTRL-A/CTRL-X to work on octal and hex numbers, as well as characters
set nrformats=octal,hex,alpha

"Show line, column number, and relative position within a file in status line
set ruler

" Show matching brackets
set showmatch

" Show line numbers
set number
" case-insensitive search
set ignorecase

" Code folding
set foldenable
set foldmethod=indent
set foldlevel=100

" Have VIM recursively search upward for tag file
set tags=tags;/

" Toggle tag list
nmap :TlistToggle
" Builds tags for current directory
nmap :!/usr/bin/ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .

" Bufexplorer plugin toggle
nmap \be

" Change to directory of current working file
au BufEnter * lcd %:p:h



I use 2 plugins: TagList and Bufexplorer along with Exuberant-Ctags to make moving around the code easier.