Find out which app is using a port.

I always forget and have to do a search, but to find out the process using a port the following can be used:

lsof -i :<port>

or

netstat -p

The cost of software development.

I think running application over the web/cloud/internet is super useful especially now that we typical carry multiple devices and use multiple computers. Syncing data across these different physical devices can be real pain so having a centralized service somewhere where there are paid professionals handling the IT work behind-the-scenes is wonderful thing for consumers.

However, there is a cost to building web applications is sometime overlooked even by the companies that built them. The Web has often been touted as making software cheaper (no more packaging, update/patch disks and store shelf space to pay for!) and easier to build (HTML is simple!), but as any modern web application developer can tell you things are never that easy. Maybe during the first few years of the web where everything was basically static HTML and images it was true, but now the cost of software development for web applications goes way beyond what it used to cost to build shrink-wrap software.

This cost is maintenance. Shrink-wrapped software is run and managed by the consumer that buys it. Even if at some point the software is not supported, the user can still run it themselves. Online web application is not the same. There is never an end to an online application unless the plug is pulled and it stops completely. Engineers can't deploy version 1.0 and then move on to the next version because once it is available developers often become the system admins keeping the software alive. Companies often forget this and as soon as a version is done they jump on to the next "big idea", but the engineers aren't done with that version. Their work only increases following the launch.

Does this mean that cloud computing should be avoided? Absolutely not. Consumers needs to be a little smarter in what they chose to use. With cloud computing, it is the data that is more important to keep alive and not those WordPerfect install disks. So find companies and products that makes it clear what their data export policies are. Do they support data libration or are they a closed garden?

Setting up GIT for home network.

I have a few different computers that I use at home and I wanted to set up GIT that I can access my code from any of them. It took me a bit because I was too used to working with a client-server model where I designate one machine as the "server" that holds the repository and everything else was a client. Instead, GIT seems to operate more like a merge tool and every local copy is its own "master." Once I understood that, it turns out that setting up GIT is very simple and just needs GIT itself and SSH.

Let's start with 2 machines. The first one, "remote", is where you want to store the code and could be where other users will also pull copy of the code from. The second one, "local", is the machine where you want to check out the code and make your edits.

On "remote" we want to store the repository in /usr/local/src:

cd /usr/local/src
git --bare init

On "local" we want to work on the code in ~/workspace and we already have some files to populate the repository with so we first create a local git repository:

cd ~/workspace
git init
git add .
git status
git commit

Now let's push this code to the remote server:

git push ssh://@/usr/local/src

Done! Now we have both servers with the files. Files can be edited locally, commit locally and periodically synced/pushed to the remote server.

Now, on another computer, we want to "check out" the source code to work on it:

mkdir ~/workspace
cd ~/workspace
git clone ssh://@/usr/local/src

Some things to make it easier to work with:

1. Use ssh keys so that you don't have to put in your password each time you push to the remote host.

Google App Engine can further encourage Open Source.

One of the challenges of open source web applications are that they require users to have their own web hosting solution in place, but recently I've been noticing more open source web applications being released as Google App Engine applications. This is a great direction as it removes the barrier of having to setup one's own web servers, network access, etc. Users can download the the GAE project's source and either run it locally using the GAE SDK's own web server or use the free GAE instances that Google provides.

I've been using GAE a lot recently and have come to really like it despite some of the limitations.

VIM copy/paste/navigation in Insert mode.

A few months ago, I made a commitment to learn the VIM editor. I was primarily an EMACS user, but I wanted to become proficient with VIM since EMACS is not always available I found myself too slow when using VIM. To learn it, I pretty much told myself that I'm not allowed to use any other editor while programming and just immerse myself in VIM. For the most part, it worked. I've been using VIM daily for about 4 months now and I've gotten used to it... mostly.

I can see the reasoning and benefits of having a separate mode for inserting text, but for me it isn't as intuitive to use. Maybe my fingers are just to conditioned to be able to write chunks of text and still be able to copy, paste, etc. The way I work, I edit text while jumping around a lot and that mode leads to a lot of hitting ESC to the point where I'm using more key strokes then on EMACS (where the criticism is that it requires mult-combos to do an action such as 'C-x C-c'). So I found myself editing text, ESC, move around, 'i', edit text. For the most part, I was willing to live with that, but the breaking point is that I just kept making mistakes editing. I'd hit 'i' when I'm already in edit mode or be typing outside of edit mode leading to typos and jumping to another part of the screen. I figure that 4 months is enough and while I'll stay with VIM, I'll likely be in Insert mode a lot and so I want to reduce bouncing back-and-forth between modes.

I made the following changes to my .vimrc to allow me to do things like move around (without using the arrow keys) and copy/paste without leaving Insert mode:

" Key mappings in INSERT mode
" Tired of lifting hand to hit ESC or having ctrl-[
imap ;; 

" navigate without lifting hand off of keys
imap  
imap  
imap  
imap  

" paste in Insert Mode
imap  

" undo
imap  u

" Select text
imap  vgG

Android 2.1 SDK released

For those who has been asking for the 2.1 SDK since the Nexus One launch, the new SDK can be downloaded here: http://developer.android.com/sdk/android-2.1.html

Google APIs

"Eat your own dog food" is not an uncommon term in Silicon Valley. It means for a company to use the product they built themselves. If a company offers an email product to customers then they should also be using that same email product. It's not always easy to eat one's own dog food especially when that dog food provide a critical business functionality or is fairly new. Email, for example, has become vital for companies and they depend on it to be scalable, stable and reliable, but if you're eating your own dog food you might be using the pre-production version which might not be 100% stable or it's missing some functionality and which you need to build yourself. Because of this, even though many tech company talk about eating their own dog food, but don't really pull it off.

I find Google to be very serious about eating their own dog food. Everything released to the public is also used internally and this includes their APIs. Often companies release APIs but don't use them internally because they have direct access to the underlying technologies. Google does use their own APIs and during my first two months here, I've been learning a lot of the APIs in order to do my work. There are some amazing stuff out there and they are all listed at code.google.com.

Specifically, I've been learning the Google Data APIs. There is an amazing amount of access to APIs to allow developers to build sophisticated applications so I encourage everyone to try them out!

Programmatically authenticating to Google App Engine with Python

Even before joining Google, I was already a fan of two of its services: Google App Engine and Google Apps for Domains. Apps itself brings a lot of value and combined with app engine, it's especially powerful for small business and start-ups. Apps provides the most essential IT pieces for a business (email, office suite, calendaring) and Google App Engine (GAE) allows the business to build their own software on Google's infrastructure without the heavy initial investment in hardware and IT infrastructure. This post focuses on just GAE specifically how you can programmatically access app engine from a python script.

The code to authenticate against the production instance of GAE is actually fairly well documented by Google on the GAE site. The issue I ran into is that before I push my code live, I wanted to test it on a development instance. GAE makes setting up a development server easy and comes with its own web server and datastore (database), but the one area where the development environment is different from production is how authentication is handled. The development server doesn't do real authentication. It just sets the right cookie that simulate that you've already logged in. I wanted to write my code with minimal differences between how it calls production and how it calls the localhost. Authentication to your application on production means that you need to get a token from the Google servers, but this doesn't exists on the dev environment. Looking at how the developer instance handled the login screen, I was able to do the following:

#!/usr/bin/python

import cookielib
import urllib
import urllib2

# Setup to be able to get the needed cookies that GAE returns
cookiejar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
urllib2.install_opener(opener)

# Set 'prod' based on how your system determines whether to connect to prod or localhost.
if prod:
   # This is the setup to construct the login URL for authentication on prod.
   authreq_data = urllib.urlencode({'Email': '',
                                     'Passwd': '',
                                     'service': 'ah',
                                     'source': '',
                                     'accountType': 'HOSTED_OR_GOOGLE'})

    # Authentication server
    token_uri = 'https://www.google.com/accounts/ClientLogin'

    # This is where you want to go after log in.  Replace:
    target_uri = 'http://auth_example.appspot.com/home'
    ## Get an AuthToken from Google Accounts
    auth_req = urllib2.Request(token_uri, data=authreq_data)
    auth_resp = opener.open(auth_req)
    auth_resp_body = auth_resp.read()
    auth_resp_dict = dict(x.split('=')
                          for x in auth_resp_body.split('\n') if x)
    authtoken = auth_resp_dict['Auth']

    authreq_data = urllib.urlencode({'continue': target_uri,
                                     'auth': authtoken})
    login_uri = ('http:///_ah/login?%s'
                 % authreq_data)
else:  # authenticate against the local dev server
    target_uri = 'http://localhost:8080/home'
    authreq_data = urllib.urlencode({'email': 'test@example.com',
                                     'continue': target_uri,
                                     'action': 'Login'})
    login_uri = ('http://localhost:8080/_ah/login?%s' % authreq_data)

# Okay, we're done at this point with the difference.  From this point everything else
# should be the same for either prod or dev.

# Do the actual login and getting cookies.
serv_req = urllib2.Request(login_uri)
opener.open(serv_req)

# Rest of code here.

That all there is to it!

.vimrc

I've been trying to set up my VIM environment to be more comfortable for me so I can be more productive. It's getting better although I still miss having a shell inside the editor that EMACS provide. Here is my current .vimrc:

" Enable loading filetype and indentation plugins
filetype plugin on
filetype indent on

" Turn on syntax highlighting
syntax on

" Global settings
set autowrite

" Allow backspacing over everything
set backspace=indent,eol,start

" Insert mode completion
set completeopt=menu,longest,preview

" Use UTF-8 as the default buffer encoding
set enc=utf-8

" Remember up to 100 'colon' commmands and search patterns
set history=100

" Enable incremental search
set incsearch

" Always show status line, even for one window
set laststatus=2

" Jump to matching bracket for 2/10th of a second (works with showmatch)
set matchtime=2

" Don't highlight results of a search
set nohlsearch

" Enable CTRL-A/CTRL-X to work on octal and hex numbers, as well as characters
set nrformats=octal,hex,alpha

"Show line, column number, and relative position within a file in status line
set ruler

" Show matching brackets
set showmatch

" Show line numbers
set number
" case-insensitive search
set ignorecase

" Code folding
set foldenable 
set foldmethod=indent
set foldlevel=100

" Have VIM recursively search upward for tag file
set tags=tags;/

" Toggle tag list
nmap  :TlistToggle
" Builds tags for current directory
nmap  :!/usr/bin/ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .

" Bufexplorer plugin toggle
nmap  \be

" Change to directory of current working file
au BufEnter * lcd %:p:h

I use 2 plugins: TagList and Bufexplorer along with Exuberant-Ctags to make moving around the code easier.

One script with different script names.

Sometimes I want to have my bash script behave differently based on how I call it without having to set up different parameter inputs. For example, I might have a script that can sync a file from a local server to one of two remote servers. I can make my script to take in parameters so I can call it using 'syncfiles -h serverA; syncfiles -h serverB', but sometimes I forget what my params are and it's easier to remember 'syncToA' or 'syncToB'.

To do this is pretty simple, yet I always have to look it up each time so I'm writing this as note to myself.

#!/bin/sh

echo $0

if echo $0 | grep "syncToA" > /dev/null
then
   # do stuff
elif echo$0 | grep "syncToB" > /dev/null
then
   # do stuff
fi

Next, create symlinks to the original source files with the names you chose (i.e. 'symlink syncToA syncfiles').

GNU Screen and Bash

As pretty as GUIs are, sometimes it is most efficient to work in a text console which is why I find myself using GNU Screen. However, Screen's command starts with CTRL-A which is also Bash's move to start of line command. To have it work, use 'ctrl-a a'.

Also, screen sets the value of TERM to be screen so if you have any settings based on the TERM value, update it appropriately.

One thing about using screen is that you lose your terminal's scroll buffer so as line scroll past your view you can't just use the scrollbar to move up to see what you missed. Instead, screen has it's own scroll buffer that you can use. You can set the buffer size using 'defscrollback #' (where # is the number of lines) in your .screenrc or in the screen command line (ctrl-a). You can switch to the buffer with 'ctrl-a [' (and exit buffer mode with ESC). Navigation follows normal VI keys.

Finally, I noticed that hitting the delete key didn't send backspace as expected with using the OSX terminal. To correct that, add the following to your .screenrc:

termcapinfo xterm-color kD=\E[3~

This tip came from here.

My VIM cheatsheet

To help me learn VIM keys, I put together a cheat sheet which I load onto my desktop much like how I handle my calendar/todo.

I'm not sure how much this cheat sheet will help others because I left out the commands that I already know well and left the commands that I'm currently learning. I expect that I'll update the list as I get more familiar with VIM.

----  BUFFER  ----
:e          set buffer
:b          goto buffer (next|prev)
:sp         new window/file above
:vs         new window left
:q          close current window
:qa         quit all windows
ctrl-w      move to window
ctrl+w =    autosize window
:ls         list buffer
:bd         buffer delete
:sav        save-as
u           undo
ctrl-r      redo
.           repeat
ESC         ctrl-[

---- NAVIGATION -----
gg          goto start of file
G           goto end of file
:54         goto line
80|         goto column
ctrl-g      show file info
ctrl-e      scroll up
ctrl-y      scoll down
ctrl-b      page up
ctrl-f      page down
zt          scoll line to top
w           next word (ctrl+right)
b           previous word (ctrl+left)
zi          toggle folding

---- SELECTION ----
v           visual select
shft-v      line select
ctrl-v      column select
y           copy selection
p           paste selection
=           reindent 
>           indent
<           unindent
:set list!  toggle visible whitespace
xp          transpose
r           replace
s           substitue
x           delete char

Trying to switch to VIM.

I've always been an EMACS user on UNIX and it sometimes seems like it can do EVERYTHING. The one problem with EMACS is that a lot of production systems (and even many non-prod) won't install it which means that when I'd have to use vi/vim and I'm just not as fluent with it. Especially since when I'm on those systems means that I'm trying to do fast debugging and navigation, being slowed down by my lack of experience with the editor is really frustrating.

I've decided to try to switch to VI/VIM to be the primary editor for awhile to force me to learn. There is no need to tell me that EMACS kicks butt (I agree), but unless you can get emacs installed everywhere, this is really more a decision based on my current situation.

Nice font for terminal and code editing

Terminus is a very nice looking font that is well suited for programming editors and terminals.



It is an open source font and many linux distributions have it available. I haven't tried any for OSX, but you can compile for xterm on OSX/X11 from macports.

Pretty printing source code

Sometimes it is just more comfortable reading source code on paper, but printing directly from a text editor often gives an output that is ugly and wasteful. Just try opening up your source code in notepad, textmate, etc. and print. It might look very nice on your screen but on paper the fonts are big with lots of white space.

Printing source code is a different beast then printing narrative text and has some special requirements. The first is to include line numbers for easier reference. Different languages have different formatting and sometimes you want to print with color syntax to help find what you're looking for. It can also be helpful to print two pages on one sheet so you can see more of the code and reduce paper waste.

On Windows/DOS, programmer editors usually have features for handling these types of pretty source code printing. Strangely, none of the editors I've seen on OSX have it (TextMate, Eclipse, Smultron, etc.) and since I've been dealing with code more I wanted a solution.

Fortunately, OSX's UNIX underpinning allows me to use the classic enscript tool to handle this.

 enscript -2 -q -C -Ec --color -f Courier8 -r -p OUTPUTFILE SOURCEFILE

This generates a Postscript file (default), with 2 columns (2), quiet mode (-q), line numbers (-C) , using C syntax (-Ec), in color with Courier 8 pt font (-f), in landscape mode (-r) to a file called OUTPUTFILE (-p) from SOURCEFILE.

You can find different language syntax available for the -E option with "enscript --help-highlight" and you can set different output formats besides PostScript with the -W option (i.e. html).

"-p -" will tell enscript to print to stdout so you can pipe it to lpr.

Consola Font Pack

Microsoft's mono-spaced font:

http://www.microsoft.com/downloads/details.aspx?familyid=22e69ae4-7e40-4807-8a86-b3d36fab68d3&displaylang=en

Google Update open source

Couldn't remember if I wrote this, but awhile back Google open source'd its software update engine, Google Updates. The project name is Omaha.

The OSX update engine is a subset of this project.

Firefox 3.5 icon with version number

4965

Download the image here.

Firefox 3.5 is out and since those of us working on the web usually have multiple versions of the browser installed, having an icon with the version number on it makes it easier to identify the instance we're loading.

This is inspired by http://browserversionicons.com/firefox/

With the icon from Mozilla.

Instructions for running multiple instances of Firefox are here.

3 pixels vs 4 pixels

Doug Bowman's post on leaving Google had a quote that has gotten picked up by many bloggers as representative of what is wrong with the engineering culture:

"I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case."

He doesn't give much more context about this debate, but it reminded me of some of my own experiences where I got involved with almost the exact same debate. In my case, the cause of the debate was almost always because there were multiple designers. Either it was two more more designers arguing about the width or it was after a product launch that the next designer decide that the width decision should be thrown out.

What drove engineers nuts was that because decisions are so arbitrary anyone can override the design at any time. Maybe Google is on the extreme end to demand numerical metrics to back up any decision, but for most engineers it was mainly that they wanted to understand the reasoning for a change. Otherwise, engineers would spend their entire lives changing the size of borders.

The value of Google Chrome

A friend of mine recently posted his thoughts on the "Chrome's overinflated importance". If viewed only as a web browser then it is just another entry in the various browsers that exists out there. However, I believe the true value of Chrome will be if it can change the way people and developers look at web applications.

In many ways, advancement in web development is hindered because developers and businesses have not been able to really think outside of limits imposed by the "WWW" of 10 years ago. Another way to say is that the world wants to move to using the Internet as a platform but we are still stuck on using the WWW as the platform instead.

There has been many attempts to try to break out of that mentality. The unfortunate term "Web 2.0" was one attempt (I think) although it never clearly articulated what it really meant and is now just a overused marketing term. In some ways, "Cloud" seems to be another way to break out from the mindset of "web development" to building applications on the Internet.

Chrome's importance to me is whether it can open up the developer's mind to new possibilities by providing them a tool that goes beyond building web sites and web applications. I hope that someday the "web" will move beyond being viewed as an interactive magazine or at least let the "web" be the interactive magazine while we're also using applications on the Internet.