Google I/O session videos posted with slides

As I said earlier, Quite a few presentations from Google I/O have been posted for your viewing pleasure. I'll be glued to these this weekend while I'm on call.

Topics include
Ajax
KML
Sketchup
Android
OpenSocial
Appengine
Data APIs
Theory Talks
Youtube
Gears
Mashups
Maps
and more!

Blogger adds bloggroll

Blogger in draft added 2 new features today: a blogroll page element, and post scheduling. Post scheduling is pretty useful, you can write up your post, set the time (in the future) that you want it to post, and Blogger will do it automatically. The second feature, a blogroll, I would like a lot more if it was done better. The main thing I like about it is that it integrates with Google Reader so if I'm linking to my friend's blogs, I just scroll to my 'Friends' Tag and add them. The thing I don't like however is that it does not support XFN or FOAF therefore not getting picked up by the SocialGraph API. For example, if I linked to my girlfriend's blog, the blogroll just lists this as

Maybe-Not

A better link, for example from a wordpress blogroll would look something like

Maybe-Not<

...and get picked up but the SocialGraph as having a relationship to me. Blogger is great and I love using it, but the features of wordpress are blowing it out of the water.

Google Reader stats out of Wack

First, Techcrunch came up with a list of the top blogs that users subscribe to using Google Reader. The top 3 on the list were BBC with 202,463, Google News wih 192,100, and ESPN with 189,274 . Remember, these are only the stats if the subscriber is using Google Reader, obviously there are blogs with a higher subscription count, such as the Official Google Blog (They still haven't moved over to blogs.google.com eh?) with over 669,000 subscribers as reported by FeedBurner.

It was all well and good that Google Reader showed high stats for these blogs, but.... we were apparently duped...

In his article Google Reader Stats are BullSh*t (With Proof), Pete Cashmore goes into his explanation of the "default feed effect." A default feed is a feed that comes already subscribed in a news aggregator , or is part of a default group (like say Digg, Engadget, and Gizmodo being in the Technology group on Google Reader). Being in these groups is a major traffic boost.

However the system has some flaws

• There is subscription data for feeds that don't exist
• Google Reader does not check to see if a subscriber is active or not. (Someone signs up for Reader, adds a feed, and never touches Reader again, they count as a subscriber)
• Feedburner pulls some stats from Google Reader
• This problem is not limited to Google Reader. It includes many feed readers and start pages

Tune in next time after I do some research into how out wack FeedBurner statistics are.

Displaying what you read from Google Reader

I've been wanting to share what I subscribe to in Google Reader and using the functions I wrote I was able to do just that. Check out the article for the full run down on the unofficial Google Reader API. This is written in python but should be easily portable to php. If i get around to it, I want to make a WordPress plugin so bloggers can share what they read with their readers. This will be followed (or in parallel depending on my mood) with a Javascript version so Blogspot users can do the same in the sidebar. On to the code!

To start off we'll just copy the functions we need from last time. Generally this is the login and SID token functions, as well as the feed list function.

from django.shortcuts import render_to_response
from django.template import Library
from elementtree import ElementTree   
import urllib
import urllib2
import re

login = 'timothy.broder@gmail.com'
password = '***'
source = 'gPowered'

google_url = 'http://www.google.com'
reader_url = google_url + '/reader'
login_url = 'https://www.google.com/accounts/ClientLogin'
token_url = reader_url + '/api/0/token'
subscription_list_url = reader_url + '/api/0/subscription/list'

#login / get SED
def get_SID():
    header = {'User-agent' : source}
    post_data = urllib.urlencode({ 'Email': login, 'Passwd': password, 'service': 'reader', 'source': source, 'continue': google_url, })
    request = urllib2.Request(login_url, post_data, header)
   
    try :
        f = urllib2.urlopen( request )
        result = f.read()
   
    except:
        print 'Error logging in'
       
    return re.search('SID=(\S*)', result).group(1)

#get results from url
def get_results(SID, url):
    header = {'User-agent' : source}
    header['Cookie']='Name=SID;SID=%s;Domain=.google.com;Path=/;Expires=160000000000' % SID
    print url
    request = urllib2.Request(url, None, header)
   
    try :
        f = urllib2.urlopen( request )
        result = f.read()
   
    except:
        print 'Error getting data from %s' % url
   
    return result

#get a specific feed.  It works for any feed, subscribed or not
def get_feed(SID, url):
    return get_results(SID, get_feed_url + url.encode('utf-8'))
   
#get a token, this is needed for modifying to reader
def get_token(SID):
    return get_results(SID, token_url)

#get a list of the users subscribed feeds
def get_subscription_list(SID):
    return get_results(SID, subscription_list_url)

Then we'll want to get rid off all the information in the feed that we don't want and load what we do into a data dictionary. After its in the dictionary, feed names and links (and the folders they are in) are ready to be displayed. As usual, I use Django to display my pages, but everything is the same up to the final return in the Feeds method. Below is an example of what each subscription looks like in the Google Reader Feed, and below that is how to process it

class myFeed:
    def __init__(self, name, link):
        self.name = name
        self.link = link

def Feeds(request):
    SID = get_SID()
    feeds = get_subscription_list(SID)
    tree = ElementTree.fromstring(feeds)
    d = dict()   

    #loop through each feed   
    for object in tree.findall('list')[0].findall('object'):
        strings = object.findall('string')
        key = object.findall('list')[0].findall('object')[0].findall('string')[1].text
       
        #tag already exists, add to the list
        try:
            d[key].append(myFeed(strings[1].text, strings[0].text.replace('feed/', '')))
        #tag doesn't exist, create list
        except KeyError:
            d[key] = [myFeed(strings[1].text, strings[0].text.replace('feed/', ''))]

       
    return render_to_response('pages/feeds.html', {
    'feeds': d,
    })

For those of you that use django or are just curious how I end up displaying the feeds, this is what i have in my view:

My Reading

{% for item in feeds.items %}
{{ item.0 }}
    

        {% for feed in item.1 %}
        
{{ feed.name }}
        {% endfor %}   
    
{% endfor %}

Again, too see what I subscribe to, click here

Google Reader adds Search Box!!

Finally!! No more using Yahoo Pipes to search the feeds!

I'm sitting here waiting for my friend Eli to pick me up from his apartment for a nice lunch over at dreamworks, glancing over a few feeds (I have a LOT to catch up on when I get back to NY), and one jumped out at me, the Google Reader Team has finally added a search box to Google reader.



This box will search all of your feeds! Also included in this release are:

- Ability to hide side navigation (I just tried it, it's very well done)
- Unread count is now 1,000+ rather then 100+ (this is going to be good for me when I get back)
- Forward and Back buttons now have more responsiveness when moving between folders and feeds

Check out Google Blog Search for more blog searching.

HOWTO: Google Reader API Functions

I've been wanting an API for Google reader since I started using it, and especially since i started gPowered so I could display a list of the feeds I read on the site. The official word on an API for reader is "It's coming in a few weeks," but that was back in late 2005. The reason being that at the time, the URLs the API would use were going to change a lot. So, after a bit of research and coding I came up with some python functions to do the job.



The first step was authenticating against Google accounts without using the client library. The Python Gdata Library makes login very easy but Reader isn't part of the Client Library yet (maybe I'll try to add it, we'll see...) but this was the method I was using for gdata and python pre-Client Library, and the principles still hold true for working with Reader. Thankfully, most of the research for working with the 'Reader API' was done for me already by Niall Kennedy. This is an unofficial, unsupported API and the URLs for some of the queries have changed since the writing of that article. Here we go...

We're going to use urllib(2) to handle the communication with this one. I rather would have used httplib, but I was having trouble with the authentication cookie. Each retrieval has its own URL to query against

import urllib
import urllib2
import re

login = 'timothy.broder@gmail.com'
password = '****'
source = 'gPowered'

google_url = 'http://www.google.com'
reader_url = google_url + '/reader'
login_url = 'https://www.google.com/accounts/ClientLogin'
token_url = reader_url + '/api/0/token'
subscription_list_url = reader_url + '/api/0/subscription/list'
reading_url = reader_url + '/atom/user/-/state/com.google/reading-list'
read_items_url = reader_url + '/atom/user/-/state/com.google/read'
reading_tag_url = reader_url + '/atom/user/-/label/%s'
starred_url = reader_url + '/atom/user/-/state/com.google/starred'
subscription_url = reader_url + '/api/0/subscription/edit'
get_feed_url = reader_url + '/atom/feed/'

When we authenticate against Google Reader with a gmail account and password in the browser, a cookie is stored. We'll have to recreate the values in this cookie. The static values are the Domain (.google.com), the Path (/), and Expires (we'll use 160000000000). The unique value, based on the current login session, is the SID (Session ID?), which we will need to retrieve. We'll do the login and retrieval in the same function:

#login / get SED
def get_SID():
    header = {'User-agent' : source}
    post_data = urllib.urlencode({ 'Email': login, 'Passwd': password, 'service': 'reader', 'source': source, 'continue': google_url, })
    request = urllib2.Request(login_url, post_data, header)
    
    try :
        f = urllib2.urlopen( request )
        result = f.read()
    
    except:
        print 'Error logging in'
        
    return re.search('SID=(\S*)', result).group(1)

We'll also need a function that can handle any of those URLs, create the header, attach a cookie to it, and retrieve the data from Google. I left the return as a raw data string so you could use whatever XML parsing library you want. I personally like using ElementTree.

#get results from url
def get_results(SID, url):
    header = {'User-agent' : source}
    header['Cookie']='Name=SID;SID=%s;Domain=.google.com;Path=/;Expires=160000000000' % SID

    request = urllib2.Request(url, None, header)
    
    try :
        f = urllib2.urlopen( request )
        result = f.read()
    
    except:
        print 'Error getting data from %s' % url
    
    return result

The following methods are the calls that I've gotten working so far; I'm going to keep working on the 'edit' functions, like adding, removing feeds, changing tags, etc. See the comments for what they do. Note: Any edit against the API needs to send over a changing token as part of the call

#get a token, this is needed for modifying to reader
def get_token(SID):
    return get_results(SID, token_url)

#get a specific feed.  It works for any feed, subscribed or not
def get_feed(SID, url):
 return get_results(SID, get_feed_url + url.encode('utf-8'))

#get a list of the users subscribed feeds
def get_subscription_list(SID):
    return get_results(SID, subscription_list_url)

#get a feed of the users unread items    
def get_reading_list(SID):
    return get_results(SID, reading_url)

#get a feed of the users read items    
def get_read_items(SID):
    return get_results(SID, read_items_url)
    
#get a feed of the users unread items of a given tag    
def get_reading_tag_list(SID, tag):
        tagged_url = reading_tag_url % tag
        return get_results(SID, tagged_url.encode('utf-8'))
        
#get a feed of a users starred items/feeds
def get_starred(SID):
    return get_results(SID, starred_url)

#subscribe of unsubscribe to a feed    
def modify_subscription(SID, what, do):
    url = subscription_url + '?client=client:%s&ac=%s&s=%s&token=%s' % ( login, do.encode('utf-8'), 'feed%2F' + what.encode('utf-8'), get_token(SID) )
    print url
    return get_results(SID, url)
    
#subscribe to a feed
def subscribe_to(SID, url):
    return modify_subscription(SID, url, 'subscribe')

#unsubscribe to a feed
def unsubscribe_from(SID, url):
    return modify_subscription(SID, url, 'unsubscribe')

Example usage:

SID = get_SID()
print get_subscription_list(SID)
#print get_reading_list(SID)
#print get_read_items(SID)
#print get_reading_tag_list(SID, 'me')
#print get_reading_tag_list(SID, 'nada-mas')
#print get_starred(SID)
#print get_token(SID)

#test_feed = 'http://picasaweb.google.com/data/feed/base/user/timothy.broder/albumid/5101347429735335089?kind=photo&alt=rss&hl=en_US'

#print subscribe_to(SID, test_feed)
#returns ok but I don't see the feed in reader?

#print get_feed(SID, test_feed)

Like I said, I'd like to keep going with this and get the edit functionality to work better. I'm also going to take a look into the Client Library and see if I could set this up as a patch that people could use if they wanted to use the API.

View Google Groups posts in Reader

I don't know why I never noticed this before but you can subscribe to the mail sent to Google Groups through an RSS feed at the bottom of each group. I'm trying it out for a few groups, I might like it better then reading through the mail. I wonder if there is a way to subscribe to the rolled up versions of the posts...