PyCon 2010 Financial Assistance Grant for Women


EventsNewsPythonThoughts

I am happy as hell to announce that this grant exists. The deadline is Dec. 18th, and I strongly encourage you to apply.

http://us.pycon.org/2010/registration/financial-aid/

This conference gets a bit bigger each year, but the organizers make a great effort to keep the small conference feel. It also has many level-100 tutorials, and is both socially and technically welcoming for py-newbies.

Open space sessions (everything from software development to Settlers) and poster sessions happen every night, tutorials run two days prior, and code sprints run for a few days after the conference. It promises to be a great learning and social experience you should not miss.

I am reserving a room and sharing it: http://us.pycon.org/2010/registration/room_sharing/
I’ll room with as many as possible, to cut costs for everyone. Bring a sleeping bag :)

See you there,

Gloria

PyArkansas: Small town, big tech!


DjangoEventsPeoplePython

Last weekend I flew out to to Little Rock, Arkansas, took a two hour crawl through a snarl of traffic, and arrived just in time for my Friday night pre-PyArkansas tutorial in Conway. Held on the stunning campus of Hendrix College, I wended my way around buildings, a massive fountain, inspiring structures, until I found the building where my tutorial was about to happen. Standing in the foyer with the beautiful Foucault pendulum, I could not help but to stop for a moment, exclaiming “Oooo!!!!” aloud, wishing I had gotten there thirty minutes earlier.

My tutorial was intended to encourage women in computer science by serving two purposes: discussing the source code and functionality of a particular project, and openly discussing some of the issues they faced in their current programs and surroundings. I was pleasantly surprised to discover that these women needed no technical or social encouragement. They are already enthused, technically and socially well prepared, and on their way to a very bright future in engineering or computer science. I was pleasantly surprised that they were very comfortable in the bash shell, and as comfortable in their current college curriculum. Like children at play, they picked up the moderate-levelled tutorial code quickly, made great strides in such a short time, and had a lot of fun doing it.

It was profoundly encouraging to see such a small computer science program achieve diversity as well as such a high level of skill. It made me wonder why larger colleges and universities cannot accomplish the same on bigger budgets, with larger staff, and a more diverse mix of students. It touched me to hear and see the enthusiasm, eagerness, and skill of the students in this program. Their learning experience under the Department Chair, Dr. Burch, comes as close to perfect as I have ever seen.

The next day’s events at PyArkansas were held at the also-very-nice campus of University of Central Arkansas, where an entire day’s worth of tutorials took place. Two Python 101 tracks were held: one for programmers and one for non programmers (a great concept). An all-day Django Track was given, where the advanced course was taught by Jacob himself. I held an afternoon tutorial addressing advanced Python concepts, with downloadable example code, where we compared and contrasted build and deployment tools, played with regex, and showed examples of some internal Python oddities involving static variables. I unfortunately missed the Python Blender tutorial, held at the same time as mine, and I heard it went quite well.

The campus facilities were very accomodating. Everything was well organized,and up and running for us when we arrived. This is a very welcome surprise to anyone who has travelled a bit to do tutorials. I was specifically told by Dr. Chenyi Hu, the Department Chair of UCA, that he really does care about diversity, and it is something they strive to achieve. This was truly touching, quite impressive, and a pleasant surprise from such a small town.

Kudos to Greg Lindstrom, Dr. Carl Burch of Hendrix College, Dr. Chenyi Hu of UCA, and everyone else involved. You induced a big technical “tremor” through your small town, which echoed far and wide. It is yet another example of the great people drawn to the Python community, and the amount of quality effort they are willing to give back. I feel honoured to have been part of this event, and I hope to be involved in many more to come.

Gloria

I Love Python: ReSTful DB CRUD dispatching using CherryPy


BookDatabasePythonServersThoughts

CherryPy has been one of my favorite Python tools for several years. It should be mentioned here that a ReSTful dispatcher could easily be written in web.py, or pylons as well, and even comes for free in the latest TurboGears implementation.

But if you’re looking for a small, easily manageable and extremely dynamic ReST dispatching solution without the heft of an entire web framework, I’m about to show you how CherryPy can help you in three different ways, depending on your model.

Assuming this mapping:

HTTP GET or HEAD = DB Read
HTTP POST = DB update
HTTP PUT = DB insert
HTTP DELETE = DB delete

Let’s also standardize on one common method across all examples, for determining the HTTP request type, and matching it to the function of the same name. Here is the full code snippet for accomplishing this task:

methods = ('OPTIONS','GET','HEAD','POST',
'PUT','DELETE','TRACE','CONNECT')

if cherrypy.request.method not in self.methods:
    raise cherrypy.HTTPError(400,'Bad Request')

# If request method is HEAD, return the page handler
# for GET, and let CherryPy take care of dropping
# the response body
method = cherrypy.request.method

if cherrypy.request.method == "HEAD":
    method = "GET"

http_method = getattr(self,method)

#print "HTTP Method: %s" % method

result=(http_method)(args,kwargs)

In our examples, we’re going to shorten this to:

http_method = getattr(self.m,cherrypy.request.method)
return (http_method)(args,kwargs)

All of this essentially determines how HTTP was called (GET/PUT/POST/DELETE), and calls the method in a class which exactly matches this name (self.GET(), self.PUT(), etc)
When you see this code, know that it’s just the HTTP method resolving code.

Now for the fun. Let’s look at the dispatcher options we have.

Way 1: A hard-coded URL pointing to fixed resources:

CherryPy can be used in a manner similar to this to establish a fixed URL, and corresponding resources, driven from predefined classes instantiated in the ‘root’ hierarchy:

import cherrypy

class ReSTPaths1:
	@cherrypy.expose
	def index(self):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)()

	def GET(self):
		return "In GET 1.."

class ReSTPaths2:
	@cherrypy.expose
	def index(self):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)()

	def GET(self):
		return "In GET 2.."

class ReSTPaths3:
	@cherrypy.expose
	def index(self,client_id=None):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)(client_id)

	def GET(self,client_id=None):
		return "IN Get 3, your client_id is %s\n" % (client_id)


cherrypy.server.socket_port=8081

root=ReSTPaths1()
root.client = ReSTPaths2()
root.client.address = ReSTPaths3()
cherrypy.quickstart(root)

Once this is running, the URL to invoke it looks like this:

http://localhost:8081/

http://localhost:8081/client/

http://localhost:8081/client/address/

http://localhost:8081/client/address/?client_id=34567

http://localhost:8081/client/address/34567

Output looks something like this:

In GET 1..
In GET 2..
IN Get 3, your client_id is None

If you’re new to CherryPy or Python in general, I’ll reiterate for you how we are calling the GET method in our class.

When we issue this request, we’re issuing what HTTP calls a GET request:

http://localhost:8081/

The CherryPy service above, listening on port 8081, calls the index() method on the root class. The root class was set to:

root=ReSTPaths1()

at the bottom of that file. The index() method from the ReSTPaths1 Class looks like this, at the top of that file:

	def index(self):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)()

If we were to insert a print cherrypy.request.method statement before the return, we would see it set to “GET”.

getattr simply says: “get me the function name in self, matching the string “GET”.
it returns a reference to self.GET(), which is set directly below the index:

	def GET(self):
		return "In GET 1.."

Notice that the index() method has a @cherrrypy.expose decorator above it. This makes the index method callable by the public. The GET method does not have it, which means we could never invoke the GET method by typing:

http://localhost:8081/GET

If you try this, you’ll get a 404 Not Found error, because it’s not visible through the CherryPy interface.

GET() has to be invoked through index(), which means GET can only be called if an HTTP GET request is issued. If we posted form data to this same URL from, say, a form entry asking people for data input, we would need to add a POST method to this ReSTPaths1() class, to receive the POST data entered in the form fields.

Now back to our example:

In this example, no part of the URL or associated resources are dynamic, in either initialization or run time. This is fine, and suits the needs of most ReSTful CRUD interfaces.

Way 2: URL paths and associated components dynamically set once, upon dispatcher init/startup:

Now let’s say we want to determine the contents of the root, and therefore the URLs and associated resources for our ReSTful interface, dynamically during initialization/startup.

We can assign the root setting by using a Python metaclass to generate classes in our CherryPy startup code, and set the root components to each generated class. This goes beyond the average needs for CRUD access, but it’s such a nice implementation that I must show it off:

import cherrypy

class MetaCRUD(type):
	@cherrypy.expose
	def index(cls):
		http_method = getattr(cls,cherrypy.request.method)
		return (http_method)()

	def GET(cls): return "In class ", cls.__name__, ', received a GET request.'

	def PUT(cls): return "In class ", cls.__name__, ', received a PUT request.'

	def POST(cls): return "In class ", cls.__name__, ', received a POST request.'

	def DELETE(cls): return "In class ", cls.__name__, ', received a DELETE request.'


baseCRUD = MetaCRUD('baseCRUD',(),{})
root = baseCRUD

dynamic_class = {}

for d in ['legacy_dbi','new_dbi','some_other_dbi']:
	dynamic_class[d] = MetaCRUD(d,(),{})
	setattr(root,d,dynamic_class[d])

cherrypy.server.socket_port=8081
cherrypy.quickstart(root)

Here we’re using a metaclass, with CherryPy exposed methods, to generate a dictionary of dynamic classes. We set the root.classname = the_new_class by using the setattr() method.

After initialization, URL components and resources are fixed in this model. But wow, the awesome power we have during initialization, in 28 lines really rocks. I wrote this in 30 minutes, and realized again why I am so head-over-heels in love with this language.

When we hit these URLs:


http://localhost:8081/


http://localhost:8081/legacy_dbi/


http://localhost:8081/new_dbi/


http://localhost:8081/some_other_dbi/

We see this output:

In class baseCRUD, received a GET request.
In class legacy_dbi, received a GET request.
In class new_dbi, received a GET request.
In class some_other_dbi, received a GET request.

Let’s issue a POST request via curl, on the command line. The response is returned:

[gloriajw@g-monster ~]$ curl http://localhost:8081/some_other_dbi/ -d ""
In class some_other_dbi, received a POST request.

This model could be used for, say, reading the contents of the Postgres template1 databases list or the mysql ‘show databases’ command, and auto-generating a ReSTful CRUD interface for each. Access of each resources can be controlled via HTTP Auth methods. This is a great solution to providing, and restricting, legacy database access for new processes through a standard interface.

Way 3: Live, ever-dynamic determination of URL and associated component:

Some ReSTful URL models may need to be ‘run-time dynamic’, especially in the case where databases are dynamically created, and the associated resources per new database could vary. There is a simple example of a dynamic URL and resource model:

import cherrypy
import pprint

class ReSTPaths:
	@cherrypy.expose
	def __init__(self):
		pass

	@cherrypy.expose
	def client(self,*args,**kwargs):
		return "Your HTTP method was %s. Your args are: %s and your kwargs are: %s\n" \
		% (cherrypy.request.method, pprint.pformat(args), pprint.pformat(kwargs))

	@cherrypy.expose
	def address(self,*args,**kwargs):
		return "Your HTTP method was %s. Your args are: %s and your kwargs are: %s\n" \
		% (cherrypy.request.method, pprint.pformat(args), pprint.pformat(kwargs))

cherrypy.quickstart(ReSTPaths())

This allows for dynamic URLs such as:

http://localhost:8080/client/address/34567

http://localhost:8080/client/address?client_id=34567

http://localhost:8080/address/client?client_id=34567

http://localhost:8080/address/client/34567

http://localhost:8080/address/anything/anything_else

The output from this code looks like this:

Your HTTP method was GET. Your args are: ('address', '34567') and your kwargs are: {}
Your HTTP method was GET. Your args are: ('address',) and your kwargs are: {'client_id': '34567'}
Your HTTP method was GET. Your args are: ('client',) and your kwargs are: {'client_id': '34567'}
Your HTTP method was GET. Your args are: ('client', '34567') and your kwargs are: {}
Your HTTP method was GET. Your args are: ('anything', 'anything_else') and your kwargs are: {}

Notice that we only have keyword args (kwargs) when we pass a named parameter, such as client_id=34567

Let’s try a POST request from curl, on the command line:

[gloriajw@g-monster ~]$ curl -d "something_else=whatever_i_want" http://localhost:8080/address/anything/anything_else
Your HTTP method was POST. Your args are: ('anything', 'anything_else') and your kwargs are: {'something_else': 'whatever_i_want'}

In this code, the sky is the limit. You can place whatever code you like in these methods, dynamically creating classes and resources as needed, letting them only persist until the result is returned. This may add some inefficiency, but in exchange offer more secure network resources.

Code is attached, Enjoy!

Gloria

http://www.devchix.com/wp-content/uploads/2009/04/restfixedargs.py

http://www.devchix.com/wp-content/uploads/2009/04/restmeta.py

http://www.devchix.com/wp-content/uploads/2009/04/restvarargs.py

I Love Python: Twitter Feed/Content Auth and Scrape in One HTTP Request


PythonThoughts

This is some lingering code from a GrrlCamp project, written by Gaba. It is such a handy, great little nugget of code. It logs into a Twitter account and scrapes either the content or the RSS feed (your choice), in one fell swoop, using Base64 encoding of the login and password in the HTTP request header. Very clever:

import urllib2, base64
import sys
import feedparser
#import configuration

class Page:

    def __init__(self):
        self.data = {}
        self.data["url"] = 'http://www.twitter.com/gloriajw'
        self.data["username"] = 'gloriajw'
        self.data["password"] = 'XXXXXXXXXX'

        self.data["urlrss"] = 'http://twitter.com/statuses/user_timeline/18107956.rss'

    def __getitem__(self, key):
        return self.data[key]

    def __setitem__(self,key, value):
        self.data[key] = value

    def getContent(self):
        base64string = base64.encodestring('%s:%s' % (self.data['username'], self.data['password']))[:-1]
        authheader =  "Basic %s" % base64string

        req = urllib2.Request(self.data["url"])
        req.add_header("Authorization", authheader)
        try:
            handle = urllib2.urlopen(req)
        except IOError, e:                  # here we shouldn't fail if the username/password is right
            print "It looks like the username or password is wrong."
            sys.exit(1)

        return handle.read()

    def getRSS(self):
        base64string = base64.encodestring('%s:%s' % (self.data['username'], self.data['password']))[:-1]
        authheader =  "Basic %s" % base64string

        req = urllib2.Request(self.data["urlrss"])
        req.add_header("Authorization", authheader)
        try:
            handle = urllib2.urlopen(req)
        except IOError, e:                  # here we shouldn't fail if the username/password is right
            print "It looks like the username or password is wrong."
            sys.exit(1)

        return handle.read()

    def getData(self):
        """auth = urllib2.HTTPBasicAuthHandler()
        auth.add_password('BasicTest', 'twitter.com', self.data['username'], self.data['password'])

        return feedparser.parse('http://www.twitter.com/statuses/user_timeline/18107956.rss', handlers=[auth])
        """

        return feedparser.parse('http://%s:%s@twitter.com/statuses/user_timeline/18107956.rss' % (self.data['username'], self.data['password']))


To invoke it:

class Data:
    def __init__(self, entries):
        self.entries = entries

    def save(self):
        pass

    def parse(self):
        pass

    def imprimir(self):
        for item in self.entries:
            print item.title

And this:

def main():
    page = Page()

    statuses = page.getData().entries

    data = Data(statuses)
    data.save()

    data.imprimir()


if __name__ == "__main__":
    main()

This code will also be attached, in case of copy/paste mangling.

In both getContent() and getData(), Gaba constructs the HTTP response header so that the encoded username and password are passed in the Authorization section of the header. This is easier and more secure than making two requests, and maintaining session cookies. Very nice indeed. This can be used to sign into any web site which accepts HTTP Basic authentication headers (there are different types of HTTP authentication (BASIC, DIGEST, FORM, and CLIENT-CERT).

It is left as an exercise for you to get the content (not the feed) and use BeautifulSoup to extract the data portions. If you want to try this, and need help, post questions here.

Enjoy,
Gloria

PyCon is coming soon, watch for us!


Python

I’ll be doing a tutorial on Kamaelia on Wednesday, March 25:

http://us.pycon.org/2009/tutorials/schedule/1AM7/

Alex McFerron is my assistant.

The fee which I would normally collect to teach this tutorial at PyCon (something like $750) will be donated to Alex’s Math Circle Outreach for kids.

All tutorials will be videotaped and put somewhere on the net. Exciting, eh? Watch for us. We’re going to rock this out.

If you’re at PyCon, feel free to say hi and take me out for a beer, to which I’ll gladly reciprocate. I’m staying for all of the tutorials and code sprints. Woot! I can’t wait!

My email addy: gloriajw_66 at yahoo dot com

Gloria