Auto-creating MySql and Postgresql Databases with Powerful Python Tools


CherryPyMySQLPostgreSQLPythonThoughts

When designing and development web framework software for clients, I first spend a bit of time focused on the client data. I try to determine the traits of the client data, asking these questions:
Is it temporary?
If so, for how long? Should it expire and disappear automatically or manually?
Should it be archived?
How many users/applications will be entering data?
Is each user’s data mutually exclusive? If not, what is the logical grouping of users and data?
What level of control do users have over their own, or other user/app data?
How is the data used by the clients? Are there “super users”?

The database is such an essential part of the design, and being able to automatically create, archive, purge and delete databases is tremendously handy in so many of these designs.

I wrote this example to generate dynamic databases for short courses, training sessions, and demos. I use the example itself to demonstrate the functionality of CherryPy and SqlAlchemy.

First of all, I start my CherryPy process on my local port 7001, allowing me to do this:

http://grrlcamp.org:7001/dbi/create_my_db?db_type=postgres

or this:

http://grrlcamp.org:7001/dbi/create_my_db?db_type=mysql

First, the small bits of quirkiness in Postgresql and MySql

Starting a Postgresql session on the command line, we see what happens before the URL is referenced:

[cwc2008@localhost user1]$ psql -d template1 -U my_super_user
Welcome to psql 8.2.6, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

template1=# SELECT datname FROM pg_database ;
  datname
-----------
 postgres
 template1
 template0
(3 rows)

template1=#

and after (Note: I had to exit the command line session, run the URL, then re-enter the command line session):

template1=# SELECT datname FROM pg_database ;
   datname
--------------
 postgres
 template1
 template0
 cwcuser_7001
(4 rows)

template1=#

The cwcuser_7001 name is auto-generated from the user’s ID and port number, making it useful for training sessions with many users.

my_super_user was created using the Postgresql createuser command, and, of course, must be a super user to do this operation. The template1 database is the Postgresql master database, from which all other databases in that instance are visible.

Caveat: In Postgresql, only one user can access template1 at any given time. Violating this rule gives you this error:

template1=# create database blahblah;
ERROR:  source database "template1" is being accessed by other users
template1=#

Be sure to close your database descriptor when finished with template1. If this is not a production app, and/or you feel comfortable bouncing the instance, this lock can also be cleared by running:

-bash-3.2$ /usr/local/postgresql/8.2.6/bin/pg_ctl restart -m immediate -D/usr/local/postgresql/8.2.6/data

waiting for server to shut down... done
server stopped
server starting
-bash-3.2$ LOG:  database system was interrupted at 2008-02-14 15:07:47 EST
LOG:  checkpoint record is at 0/4B9054
LOG:  redo record is at 0/4B9054; undo record is at 0/0; shutdown FALSE
LOG:  next transaction ID: 0/2046; next OID: 57344
LOG:  next MultiXactId: 1; next MultiXactOffset: 0
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  record with zero length at 0/4B909C
LOG:  redo is not required
LOG:  database system is ready

This is nasty. In live production environments, do this with great care.

MySql is a bit forgiving when adding, removing or accessing new databases. It does not require the isolation level to be set, as in Postgresql.

We see what databases exist before:

/usr/local/mysql6/bin/mysql --socket=/tmp/mysql10.sock  -u my_super_user -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 170
Server version: 6.0.3-alpha-log Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.00 sec)

mysql>

and after:

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cwcuser_7001       |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

mysql>

MySql does not require my hopping in and out of the command line like Postgresql. It handles user scope much better, in my opinion. It does, however require each new non-super user to be granted access to each database they need, but this is (1) trivial, and (2) outside of the scope of this article.

The SqlAlchemy code to handle the quirky bits

This is invoked by CherryPy, but can be run in any other manner of Python execution. I have a Class called DBInterface, method is called create_engine:

class DBInterface:
    #~~~~~~~~~~~~~~~~~~~~~~~
    def create_engine(self,choice,local_dbname=None):

        if choice == 'super_mysql':

            '''
            Super user MySql commands do not require connection to a database.
            '''
            db = sqlalchemy.create_engine('mysql://%s:%s@%s' % ('my_super_user','some_password','127.0.0.1:3333'),strategy='thr
eadlocal')
            # 'self' is not threadsafe! Don't use it!
            #self.show_databases = "show databases ;"
            cherrypy.session['show_databases'] = "show databases ;"

        elif choice == 'mysql':
            db = sqlalchemy.create_engine('mysql://%s:%s@%s/%s' % ('my_super_user','some_password','127.0.0.1:3333',local_dbnam
e),strategy='threadlocal')

        elif choice == 'super_postgres':
            '''
            Super user Postgresql commands require that you connect to the
            built-in template1 database, and set the isolation level to 0.
            This can't be done in SqlAlchemy, so we do it in Psycopg.
            '''
            DSN_string = "dbname=template1 port=2222 user=my_super_user  host=127.0.0.1 password=some_password"
            def connection():
                self.local_conn = psycopg.connect(DSN_string)
                self.local_conn.set_isolation_level(0)
                return self.local_conn

            db = sqlalchemy.create_engine('postgres://',creator=connection, strategy='threadlocal')
            # NO!
            #self.show_databases = "SELECT datname FROM pg_database ;"
            cherrypy.session['show_databases'] = "SELECT datname FROM pg_database ;"

        elif choice == 'postgres':
            db = sqlalchemy.create_engine('postgres://%s:%s@%s/%s' % ('my_super_user','some_password','127.0.0.1:2222',local_db
name),strategy='threadlocal')

        else:
            db = None

        return db

The exposed method in the same class which invokes the create_engine method, is called create_my_db, which we saw in the URL above (I will address the CherryPy URL functionality after this point):

    @cherrypy.expose
    def create_my_db(self,db_type):

        '''
        Creating a db is a super user act.
        '''
        cherrypy.session['db_type'] = 'super_' + db_type
        cherrypy.session['my_dbname'] = "cwcuser_%s" % (cherrypy.request.config['server.socket_port'])

        db = self.create_engine(cherrypy.session['db_type'])
        db.execute("create database %s ;" % cherrypy.session['my_dbname'])
        databases = db.execute(cherrypy.session['show_databases'])

        dblist = []
        for d in databases:
             dblist.append(d[0])

        print ', '.join(dblist)
        self.dbclose(cherrypy.session['db_type'],databases)
        return ', '.join(dblist)

Because I clump together super user and non-super user database engine connection functionality, I distinguish the difference by modifying the users db type to a “super_” setting in a CherryPy session variable. This way, only one method is called in various internal modes of operation. When in non-super user mode, the SqlAlchemy connect string is the same for both Postgresql and MySql.

No more db quirks in this app: The rest is clean SqlAlchemy

All db operations after this point use the exact same SqlAlchemy code for both MySql and Postgresql. That is a profoundly beautiful thing.

The destroy_my_db method is generic:

    @cherrypy.expose
    def destroy_my_db(self,db_type):

        '''
        Destroying a db is a super user act.
        '''
        cherrypy.session['db_type'] = 'super_' + db_type

        if not cherrypy.session.has_key('my_dbname'):
            cherrypy.session['my_dbname'] = "cwcuser_%s" % (cherrypy.request.config['server.socket_port'])

        db = self.create_engine(cherrypy.session['db_type'])
        db.execute("drop database %s ;" % cherrypy.session['my_dbname'])
        databases = db.execute(cherrypy.session['show_databases'])

        dblist = []
        for d in databases:
             dblist.append(d[0])

        print ', '.join(dblist)
        self.dbclose(cherrypy.session['db_type'],databases)
        return ', '.join(dblist)

As above, all other database operations from this point forward are generic SqlAlchemy code. Yaaay!

The CherryPy layer

My CherryPy main module, which sets up the needed objects and assoiated URLs, looks like this:

import cherrypy
import pdb
import sys,os,re,traceback
import user_data
sys.path.append('../')

#hostname = os.environ.get('HOSTNAME')
#hostname = re.sub('\..*$','',hostname)

# Add your app here.
import db_wrapper

# Only generic headers and footers go here.
# To use generic templates in your app, you may need to symbolicly link
# them into your Templates dir, or references them by absolute path.
TemplatePath = os.path.join(os.path.dirname(__file__), '/cherrypy/dev/templates')

# Add your class here. Note that the 'root' should be the authentication module.

# To test, go to http://grrlcamp.org:1112/run_test
root = db_wrapper.WrapperTest()

# http://grrlcamp.org:port/dbi/create_my_db, destroy_my_db, etc.
root.dbi = db_wrapper.DBInterface()

# http://grrlcamp.org:port/user/get_user_data
root.user = user_data.UserData()

try:
        cherrypy.tree.mount(root)
        cherrypy.config.update(os.path.join(os.path.dirname(__file__),'cp.conf'))
        cherrypy.server.quickstart()
        cherrypy.engine.start()

except SystemExit:
        #smtpInterface.sendSysAdminEmail('cherrypy@xyz.com','sysadmin@xyz.com',"From %s: CherryPy is going down due to shutdown." % hostname,"")
        sys.exit()

except:
        print "ERROR: CherryPy crashed, error:"
        raise

finally:
        #smtpInterface.sendSysAdminEmail('cherrypy@xyz.com','sysadmin@xyz.com',"From %s: CherryPy is going down (from finally clause, check for another error message after this one)." % hostname,"")
        pass

The CherryPy “root” determines what objects are accessible via URL. I mounted a simple test under the root, and nested the above database code under root.dbi, hence the /dbi/ reference in the URL.

root.dbi contains an instance of the DBInterface class we saw earlier. All exposed methods in the DBInterface class become accessible via the /dbi/method_name URL when CherryPy is started.

I’ve attached the complete code used for training, configured to run on localhost, and connect to both MySql and Postgresql daemons. It contains some explicit redundancy and commented code for learning purposes. Your configuration variables will vary slightly. Enjoy.

auto-create-any-db.tar

RESTful Thoughts on a Web 2.0 Python Project


CherryPyPostgreSQLPythonThoughts

In September of last year, I was contracted by an independent investor to do the server side implementation of a Web 2.0 application from the ground up. It was an exciting new project, one which kept me up late at night through many nights, doing design and development. It had been a very long time since I had worked on a project that I found so exciting that it replaced my need for sleep. It was a geeks’ dream come true. It was great fun.

My first step was to get to know the front end Dojo guru who was working on the user interface. We were separated by thousands of miles and many time zones. We had never met in person (and still have not to this day). We were going to be spending long and occasionally stressful nights together in virtual space, so we spent time chatting to get to know each other. After the first month, we could both almost intuit when the other was tired, frustrated, or both. After a short time we worked so well together that communication was fast and efficient, and the work was not hindered by our physical distance. There were occasions where it would have been great to draw together on a real whiteboard. But we compensated for this loss fairly well. It was the first entirely virtual project I had worked on, and the social aspects of how we worked together were just as interesting as the technical aspects of the project.

We spent time evaluating tools which would achieve our goals of rapid design and development, and excellent performance both in the browser and on the server. For the server development I chose CherryPy and SqlAlchemy, using Postgresql. For the front end we chose Dojo and Javascript, using JSON to communicate between the browser and the server.

I started with the database schemas in Postgresql, then the wrappers which automatically generate one database per set of client data. The basics on how I do this in Postgresql are here:

http://www.devchix.com/2007/05/27/dynamic-web-content-what-you-can-do-in-postgresql-that-you-cannot-do-in-oracle/

This model of one database per data set per client worked so well with the client’s particular data that it was a natural fit. Scripting dynamic archiving and recovery took under one day in Python and Psycopg, and worked like a charm.

Here is a peek at how Postgresql can be used in CherryPy (or any other web framework, for that matter) to snapshot to backup tables, and subsequently to disk:

         def take_snapshot(self):
                try:
                        '''
                        All timestamps are UTC.
                        '''
                        snaptime = datetime.datetime.now()
                        snaptime = pytz.timezone(config.server_timezone).localize(snaptime)
                        snaptime = snaptime.astimezone(pytz.timezone('UTC'))
                        snaptime_str = snaptime.strftime("%Y%m%d_%H%M%S_%Z")

                        db = self.login.open_session_db()
                        #query = 'select datname from pg_database'
                        query = "select tablename from pg_tables where tablename not like 'pg_%' and tablename not like 'sql_%' and tablename not like '%_utc';"
                        our_tables = db.queryAndFetchAll(query)
                        mod = ""
                        for table in our_tables:
                                table = table[0]
                                '''
                                This table grows forever. Never recover it
                                from snapshot.
                                '''
                                if table in ('user_change'):
                                        continue

                                mod += "create table %s_%s as (select * from %s);\n" % (table,snaptime_str,table)
                        print "Taking snapshot: %s" % mod
                        db.executeMod(mod)

                except:
                        raise
                        db.close()
                        ft=open(TemplatePath + '/login')
                        showPage=ft.read()
                        ft.close()
                        showPage = "<P> Invalid User Name or Password.</P>" + showPage
                        return showPage

                db.close()
                return snaptime_str

        take_snapshot.exposed = True

Small snippets on the key concepts of flushing/retrieving snapshots from/to disk:

         def archive_to_disk(self,snaptime_str):
                        [...]
                        for archive_table in archive_tables:
                                archive_table = archive_table[0]

                                shmod += "mkdir -p %s; pg_dump -U tp -t %s -d -f %s/%s %s\n" % (archive_dir,archive_table,archive_dir,archive_table,dbname)
                                mod += "drop table %s;\n" % archive_table

                        print "Archiving old snapshot tables to disk: %s" % shmod
                        os.popen(shmod)
                        [...]

        def revert_to_snapshot(self,snaptime_str):
                        [...]
                        for revert_table in revert_tables:
                                '''
                                Move current tables to current snapshots.
                                Copy older snapshot tables to current tables.
                                '''
                                revert_table = revert_table[0]
                                table = re.sub('_%s' % snaptime_str,'',revert_table)
                                mod += "alter table %s rename to %s_%s; create table %s as (select * from %s);\n" % (table,table,curr_snaptime_str,table,revert_table)

                        print "Taking snapshot, and reverting to old snapshot: %s" % mod
                        db.executeMod(mod)
                        [...]

Why spend so much Web 2.0 article space talking about database concepts and client data handling? Because while designing Web 2.0 applications, all matters involving client data take up somewhere between one-third to 50% of the development time and energy. Client data is the very reason the application exists in the first place. The more time and energy placed into making sure it is kept, stored and manipulated in the best possible way for the application, means less headaches and refactoring as the product matures.

Once the database schema and wrappers were in place (much of those details were mercifully left out of this article), the business logic was next. When designing a RESTful application, a great bit of thought goes into the design of the state transactions between the browser and server. For our needs, passing massive XML structures back and forth to communicate state seemed like overkill. Passing JSON dictionaries with well defined fields back and forth seemed like a better solution, for speed and ease of customization.

Once the ‘how’ part is resolved, the next question is what will be passed back and forth between browser and server. In a REST implementation, the goal is to pass a complete transaction from browser to server, and a complete transaction from server back to browser. The definition of a transaction will vary based on the application. For RSS feeds, a transaction is only one-way, where the server constructs and sends a complete XML document to an RSS feed reader. This is probably the simplest REST implementation in existence right now. I’ll be addressing a more complex transaction model, where client and server pass each other data, and either side can manipulate the data and pass it back to the other side.

To achieve a RESTful state transaction model, in a nutshell, means passing enough data back and forth so that a user can click on any feature or function of the application, in any order, and the server will (1) know what to do every time without fail, and (2) will not ‘remember’ each client’s previous states. This implies that the server is ‘stateless’ with respect to specific client data. The server has a state of it’s own, and knows it’s own transaction state, of course. But it is only ‘aware’ of each client’s state when it is contacted by the individual clients, and the client notifies the server of it’s state.

It helps tremendously to work out the ‘chatter’ between the client and the server in human language before designing it. Here is an example of RESTful chatter between a client inventory application, and it’s corresponding server application:

server’s current state: “My current state is fifty boxes of paper clips in inventory.”

client one: “Hi server. Last time I contacted you, you had seventy boxes of paper clips in inventory. I am placing an order for sixty. I will accept a smaller quantity if you have >= forty in inventory. Bye!”

client two: “Hi server. I need twenty boxes of paper clips. I have no clue how many you had in inventory last time I contacted you, and I don’t really care. Fulfill this exact order or cancel it, no exceptions. Smell ya later.”

server: “Received client two’s request. Hi client two, you are properly authenticated, so I’ll look at your state. You want exactly twenty boxes of paper clips. You will make no exceptions. I currently have fifty boxes. I don’t care how many you have, I only care about how many you need. Your order is fulfilled, and I have thirty left. Bye.”

server: “Received client one’s request. Hi client one, you are properly authenticated, so I’ll look at your state. You want sixty boxes of paper clips. You will settle for a minimum of forty. My current state is only thirty in inventory. Your order is not fulfilled, and I have thirty left. Bye.”

Compare this to non-RESTful chatter between client and server:


client two:”Hi server.”

server: “Hi client two. You are authenticated, so I’ll continue to talk to you.” (server stores the state of talking to client two, properly authenticated.)

client one:”Hi server.”

server :”Hi client one. You are authenticated, so I’ll continue to talk to you.” (server stores the state of talking to client one, properly authenticated.)

client one:*BOOM* (crashed, blue screen of death, user restarts session once machine reboots)

client two: “server, hook me up with twenty boxes of paper clips.”

client two: *WHOMP* (browser crashes, user restarts browser)

server: “done, server two….hey wait, I can’t respond to you. How strange.”

client one: “Hi server.”

server: “Client one, you just authenticated, and you’re trying to authenticate again? I have to reject your request. Bye.”

client two: “Hi server.”

server: “client two: you have an outstanding transaction, but now your session ID is different. Are you trying to trick me? Get out of here. Bye.”

client two: “Huh? I just loaded, and have no idea what you’re talking about.”

server: “You are in a messed up state. Your authentication is rejected. Call customer support at 1-800-….”
client two: “????”

The point of the chatter is to ensure that the client is passing a complete transaction with every request, and that the server passes a complete transaction with every response. Similar to the CRC cards for OO design, the Chatter ensures that you have your states, transactions and interfaces well defined.

Note that a crash of the server in a REST implementation means that the client can’t reach the server with it’s complete transaction, so it waits and accumulates transactions, sending bulk transactions when the server is available once again. The crash of a client in a REST implementation means that if the server cannot acknowledge the client’s transaction, it can either drop it, knowing it will arrive again, or process it and ignore the duplicate transaction that may arrive again when the client is back up. There are no “partial states” which need to be resolved, the transaction was either accepted and completed, or rejected. Recovery in a REST implementation becomes trivial because of this simplicity, and this saves a tremendous amount of development time and energy.

Once your chatter seems complete, it can be translated into a header and data components. For this application, this means that the JSON structure coming from the client to the server has fixed fields with expected values, and data arrays/dictionaries with predefined formats.

The server receives the JSON data, decodes it into Python dictionaries and arrays, and first evaluates the ‘header’, as defined by the developers. The header tells the server the client’s current state, as well as the server’s last known state when the client last contacted the server. (For our application, this level of detail was necessary. Not all applications would need to know about the server state, and this needs to be decided while devising the chatter.)

Once the client’s full state is known, the client’s data can be processed against the server’s current state data.
The server constructs a reply header and data, as defined by the developers, comprised of expected fields and proposed values. In the case of inventory, let’s say client two needed to be notified of client one’s transaction, because these are franchise stores, and all inventory data is shared between them. The returned data set from server to client one would be the acknowledgment header (processed_order=False) plus the data received from other clients (other_orders_since_you_last_contacted_me=client two placed this order at that date and time).

CherryPy turned out to be an excellent framework for this task. It’s thin, fast, threaded architecture, and clean session handling made REST implementation quite easy. The same methods are called by every client, to contact the server and pass across it’s data. The same method is called by the server threads to respond to every client. The API between client and server is reduced down to about five methods. Granted, the object which checks the client’s current state against the server’s current state is quite large. But the transaction recovery code is quite small, and the objects are extensible, to handle new transaction types.

This article assumes the model of user authentication separate from transactions. The subject of cookieless authentication won’t be addressed here, but it is an option to consider, based on the application requirements.

This article also only addressed a Python implementation. Rails developers have a bit of an advantage, having a built-in REST methodology. I hope some articles to come will address Rails implementations.

Let’s All Evolve Past This: The Barriers Women Face in Tech Communities


BookCherryPyEventsIntroductionsReviewsSoftwareThoughts

Topics of this Article:

Introduction

This subject has been on the minds of many tech women for years. The issue is discussed regularly, almost cyclically at times, as we spin our collective wheels to try to find causes and solutions. I was reluctant to write about it, since I find the subject matter daunting, and the problem almost insurmountable at times. But three different sources approached me simultaneously, asking for this article. This article feels as if it is manifesting through me rather than from me, as a collective opinion and observation from the many tech women with whom I’ve worked and spoken. So many factors are in play when discussing this issue that I can only hope to address many of them without writing a tome.

My tendencies are to pick up on patterns, in human interaction, in data, in almost everything. I am a computer science/math major, and my brain loves to seek out the unobvious patterns in whatever I am observing. One of my favorite pastimes is to figure out broken elevator algorithms: what event causes the doors to close too quickly, how are the cars distributed amongst the people requesting the elevators, etc. One of the not-so-favorite puzzles my brain likes to do is to pick up on patterns of human behavior from both men and women which affect how tech women are treated both on and off the job. This article is all about the patterns I and other women have found in human interaction, office and online environments, which make them less conducive to tech women participation.

The less obvious

I won’t be addressing the more obvious problems affecting women in tech environments such as the pay scale gap between women and men, the blatantly inappropriate sexism and personal harassment that has taken place, and persists. My reasons are because I feel these issues have been properly and effectively addressed by other women in tech (they’re not resolved by any means, but at least public awareness is rising). With this article, I am attempting to address the less obvious or unobvious reasons why some tech environments are intolerable for many women.

The material for this article came about through my participation in both women-only and mixed gender groups of many kinds. When I wonder why tech groups aren’t tolerable for many women, I look at the inverse of the problem: What makes women-only tech groups more tolerable for women? My observations follow.

Why do women-only tech groups exist?

Over the years I had participated in many different types of women-only groups. Women-only drumming groups, women-only political groups, women-only tech groups, have all provided what women consider to be a “safe haven” to freely learn these arts, share ideas, expose each other to paid “gigs”, and help each other accomplish tasks. Women in these groups usually had nothing else in common except for the fact that they (1) were female, and (2) shared an interest and experience in drumming/politics/tech. Their professions, ages, skill levels, hobbies, sexual orientations, life experiences, marital status, children/grandchildren/no children, everything else about these women varied vastly.

My brain began to try and pick up on patterns which would explain why all of these different types of women feel as if they need a women-only group, and what such a group can provide that a mixed gender group cannot. Here are my observations.

Community plays an important and prevalent role in women-only/women-friendly groups.

No matter the group or the reason for gathering, _all_ of the women-only, and most of the successful women-friendly groups to which I have belonged had a strong sense of community. They make a tremendous effort to communicate well, to be fair with each other, and to provide support related to the groups goals, sometimes even extending outside of the groups goals.

This mindset is so common that women come to expect it when joining these groups, and foster it once they have joined. The implied message is that a strong, focused, collective effort will be spent to run things fairly and treat all members equally, and collective discussion happens when this is not accomplished. This is the lure to women-only groups.

Communication style is directly affected by this sense of community

I have never seen a woman harshly criticize another woman in these groups. Never have I seen or heard anything like “You suck”, “You’re wrong, idiot” when women in these groups communicate. Differences are usually discussed in a civilized manner. There is the occasional strong disagreement or ousting of a member now and again, but it happens after a discussion involving the entire group, and an effort to work out their differences. I am sure harsh criticism happens somewhere in some women’s groups. But I am also sure that it’s not tolerated for very long by other female members.

This style of communication is directly at odds with much of the harsh criticism and disdain found in predominantly male public comments, especially in most public online tech comment spaces, unfortunately.

Destructive criticism is the best way to keep a site predominantly male. It implies that there is no concern about whether a person can learn from a response or not, or whether they would find offense. It is an outward display of ego, a territorial “pissing rite” in which most women do not and will not participate.

That being said, there are many men who flock to women-only groups for the same reasons as women. They do not want to be subjected to the predominantly male style of communication where there is no sense of community, or even just simple accountability. They grow tired of the “pissing rite”, the absurd declarations of false boundaries, the outward display of insecurity through harsh criticism, implicit claims of “my way, my expertise, my right, never yours”, and poor display of ego. This mode of communication is an unproductive waste of time, and many men realize this as well. “I feel at home here because I really don’t want to deal with that male ego bullshit”, one male member of our political group stated to me.

Men who seek out women’s groups are usually welcome, or a splinter group is formed to accommodate these men, once it is determined that they do not seek membership for the wrong reasons. Some of the wrong reasons are:

(1) “I will be the only male member, and will therefore have my choice of ‘chicks’”. Nope. It’s not happening.
(2) “I will be the only male member, and I’ll guide/help/protect these lost/vulnerable/endangered women”. This is not only unnecessary, but laughable. Women find the implications of these assumptions both offensive and so primitive that it is hysterically funny.
(3) “I will infiltrate because I hate women, and want to try to dissolve the group in some way” This is very rare, but happens. The good news is that the motives of both men and women who attempt this become very obvious very quickly.

Women-only/women-friendly tech groups and gatherings offer a level of awareness of and accountability for behavior not found in most mixed gender tech groups/gatherings.

Awareness of and accountability for behavior in women’s groups means a lot more than just safety from sexual harassment, or discrimination. It means that if one is treated unfairly or harshly in any manner that a person finds offensive, the entire community will hear your claim. They will give you advice, opinions, and will collectively decide if action should be taken.

There has recently been a call for all public message board admins to get tougher about removing blatantly discriminatory, harassing, or sexually objectifying comments. This is a very necessary, damned good start. But to genuinely make an online tech community women-friendly, it needs even tighter moderation against harsh/demeaning criticism, elitist commentary, and exclusionist statements, the three most prevalent and women-unfriendly types of communication found in almost all moderated online tech message boards. There is no better way to give women a message that their comments are not welcome than implying that: (1) this is forbidden territory, women have no expertise here (2) your comments are stupid, wrong, or ridiculous, (3) we’re so much smarter than you.
Discussion, constructive criticism, even heated debate happens in women-only groups, but these methods of communication are avoided.

Both online and off, I have seen men who communicate this way with everyone, and men who only choose to communicate this way with women. I have also seen this behavior tolerated or ignored for the most part. Here are my observations on why this happens.

Men are generally very good at ignoring bad behavior.

This is both a blessing and a curse. In my most recent office environment, we had situations where a male colleague’s behavior was abrasive in one of these three ways mentioned. “That sucks, doesn’t it?” I asked another male colleague. “Yeah, but I just ignore it. That’s just the way he is. He is always like that” He responded. This is what I’ve seen as the general male way of coping with this poor communication style.

It’s a blessing that many men can ignore it, in the sense that most men do not get caught up in deep analysis of why this person said a specific thing, and what this person could have really meant, etc. When almost everything is taken at face value, and not overanalyzed, the ability to ignore communication issues makes it is easier to resolve the simple issues, and move on. I have seen some women in office environments do the over analysis, and take offense when there never was one given. I don’t see men do this very often, and it makes communication quicker and easier.

Ignoring communication issues is also a curse because one obnoxious person is allowed complete freedom to make excessive noise, be rude and disruptive, or explicitly offensive. Most men, online or in the office, will ignore it. Most women will notice it but not say or do anything about it, for a variety of reasons which are tangential to this article. The offender often thrives on the fact that no one told them to stop, so they continue. Sometimes the offender is not socially adept enough to pick up on the fact that ignoring implies intolerance at some level. They somehow missed the message most three year olds learn: I’m ignoring you because I don’t like your behavior, so they continue the intolerable behavior.

This is so prevalent in online tech communities that it is the primary reason why many women do not participate. The poor communication and behavior of even one boorish, ego-driven, elitist, socially inept geek is just simply intolerable for most women. Women generally tend to assume that everyone will be conscious of and annoyed by this behavior. Men tend to assume that everyone will ignore it. This causes problems in offices as well as in online communities, where women will complain about such behavior, and men will issue responses such as “toughen-up”, or “what’s the big deal?” because this is how they cope with the problem. A female-friendly group addresses and tries to resolve these issues, while the average group ignores it until/unless the person does something heinous.

The sense of community fosters a protective behavior within that community.

If you do something awful to one woman in a women-only community, all will hear and know about it, and you are ousted. Most of the time this is first discussed and voted on by many group members. Many times the women’s group will even make an effort to explain the offense to the oblivious offender. But if the offender is still oblivious and/or offending, the offender is out. This is done to protect the interests and goals of the group. Many male dominated online groups don’t run this way. Most if not all women’s groups run this way, whether online or off. This relates to the awareness and accountability mentioned before. It’s an essential element of all women-only groups, and seems necessary for women-friendly groups to draw women.

Women’s groups generally have a few vocal, and many silent, members
The vocal few express their opinions, and either gain support or do not gain support. The ones who gain support usually implicitly become the spokespeople for the silent many.
The silent many usually let the vocal few, with whom they agree, do the job of ousting, protecting the sense of community, and publicly representing the silent many. The silent many support the vocal few. The community in turn supports and protects the rights and privileges of the silent many.

Why this happens is again a dynamic which is tangential to this article. But it seems that many women in group participation give either their silent support or rejection, speaking up only occasionally. Because of this behavior, if a communication problem arises in any type of group, whether women-only or not, and there is not a vocal few who will attempt to resolve it, the silent many will often silently leave. The silent many often don’t want to complain, for fear of having to deal with the additional frustration of the unaware/unconcerned “toughen-up”, or “what problem?” type of responses. For the silent many, it’s easier and less frustrating to just leave. I think it is important for groups that want to advertise themselves as being women-friendly, to be aware of this pattern.

One of the challenges of any women-only/women-friendly group is encouraging the silent many to speak up. Many women deal with demeaning and discriminatory behavior so often in their lives that they are too emotionally exhausted to deal with even the possibility of an online onslaught of anonymous discriminatory and demeaning comments. Many women spend time observing online groups before deciding if they will participate, for this very reason. They want to ensure that they will not feel verbally attacked once speaking up, and that their issues, comments and contributions will be heard and handled fairly.

Women generally do not arm themselves for battle during tech discussions

Women generally do not work things out through verbal battle. By the time they
reach that point of wanting to argue, they are already so offended that they are in pure self-defense mode. Women treat the discussion of tech issues like the discussion of many other issues. It’s not competitive, and they wish to bi-directionally share information.

Many tech men envision a technical debate as a battle, and celebrate the supposed victory, exhibiting classic “Alpha Male” behavior. I have personally seen it so many times in my profession that I brace myself for it when discussing tech issues with new groups of men. So many of them arm themselves with weapons of aggression, demeaning comments, and behavior which encourage more of a filibuster than a healthy debate. The supposed tech discussion becomes a test of verbal and emotional endurance, where whomever can argue the hardest and last the longest wins.

They can shake hands afterwards and congratulate each other over a “good fight” after a technical debate. “I like the challenge of a good argument, which is why I do that” one male colleague explained to me. “I like a good technical debate too, but I don’t want to feel verbally or emotionally abused afterward. Women don’t fight for fun, they fight for personal issues.” I explained to my male colleague.

Unfortunately, the anonymity offered by many public wikis and message boards encourages the worst behavior in people. Even moderated tech chat areas and comment boards are rife with elitist, demeaning comments encouraging “the fight”. Some of it is due to oblivion, lack of knowledge that this is offensive to tech women. Some of it, unfortunately, is very intentional.

Apparently there are males online, in tech communities, who still believe that, like the cigar rooms of the Victorian Era, tech rooms should be male-only. Back then, the predominant purpose of smoking cigars in a common room was to have male-only space, and similarly today, the purpose of the demeaning and fight-provoking attempts is to maintain the male-only presence of some online tech spaces. I know for a fact this happens with intent in some online chat rooms and message boards. It is not simply an act of oblivion, but a concentrated, misogynistic effort between like-minded men to keep women out.

When I discuss this with people and we ask each other how this can be prevented, I feel overwhelmed. How do we stop any/all of the human behavior which prevents us from evolving further? I have no answer to this, but I am certain that if less of this behavior is tolerated online, we at least squeeze people who discriminate into their own, personal hidden online spaces. There is no reason why we need to be subjected to every single person’s beliefs or comments in the name of the First Amendment. We all have a right to remove from our lives anything and everything which holds us back in some way, even that which is subtly harmful or offensive. Web admins have a right to remove useless, demeaning, even subtly harmful comments in the best interest of an online community. The operative word here is “community”, and the appropriate questions is: Does your public comment space contribute to a community, or is it just an open toilet that everyone can vandalize and pollute?

Did you know?

When it was illegal for women to publish writing during various times in history throughout various countries, women published their work under male pseudonyms. Today, many tech women still use male pseudonyms when posting to lists or publishing tech articles. The reasons are to have their work read without bias, and to avoid misogynistic “hyper-scrutiny” of their work. I have experimented with this myself using a male pseudonym to post articles, and being told that the articles are informative, useful, great. Six months later I republish the exact same article, using a different title and a female pseudonym, and suddenly the article is horrible, technically incorrect, useless. It’s a fascinating study. I would love to see some prominent male techs publish under female pseudonyms, and watch the responses.

Women find it awkward to brag about their writing accomplishments published under male pseudonyms. For this reason, most of this work never gets credited to the correct person, and is never acknowledged on resumes or during job interviews. “How do I explain to a male ‘potential boss’ why I have chosen to use a male pseudonym, without bringing up the whole discrimination issue?” is what one female tech friend asked me. I had no answer for her. I have also let my work published under male pseudonyms fall between the cracks, into oblivion, not knowing what else to do.

To make an online community more women-friendly, try these suggestions:

(1) Monitor the public comments. Treat the public comments interface much like the
front door to your home. You don’t simply leave it open for any idiot to waltz in.
You can be selective regarding who comes in, and what they do once they’re in.

Useless comments get deleted as quickly as they appear. Any non-technical,
offensive, destructive, or off-topic comment is removed. This gives a clear
message about will and will not be tolerated. As useful comments accumulate,
useless ones are much less likely to appear.

(2) The technically correct but aggressive/demeaning/overly harsh comment gets returned
to the sender, asking the person to re-word using constructive criticism.
Sounds like overkill, but it’s not. The “You’re wrong, here’s the right answer”
type of response constitutes picking a battle that most women won’t fight, or won’t even bother dealing with.

(3) Treat your online space like a community. The web admin should act is if they’re on the board of chosen freeholders, voting on issues which affect themselves and the entire community. Don’t just throw up the comment space and leave it abandoned for vandals and other jerks. Maintain it according to the rules by which you want everyone to abide, and stick by your decisions. Have accountability for comments. Create a space where open discussion happens as if it were in an educational surrounding, not a seedy bar.

(4) Explicitly state that your site is women-friendly. Doing this will encourage the silent many to speak up. Kick out the jerks who don’t want your online space to take this direction.

For the men who care: Tips for communicating with women in Tech environments, online and Face-to-Face

(1) Tech women usually express great enthusiasm about their work. They do what they love, and they love what they do. When a woman gets enthusiastic about her work and shares that enthusiasm with you, it has absolutely nothing to do with you, or sex. I cannot tell you how often I have seen this. Some men mix up their incoming signals, and a women’s enthusiasm at work somehow translates to someone flirting with them at a bar. I have no idea how this happens, but it’s profoundly sad to see it happen again and again. If you’re lacking something in your life, please do not look to your female tech colleague to fill that niche. Do not even presume her mind is there even if yours is not, because hers is not, and your signal indicator needs serious recalibration.

(2) Leave your libido at the door. Please. Women tech colleagues want to be appreciated for their brains, their technical expertise, their contributions and accomplishments. Tech women do not give a flying shit about what their male colleagues think of their attire, their make-up or their body parts. Believe me when I say this is true. Women may give you a polite response, but on the inside they are offended, seething, and considering whether or not to go to their attorney. They will ask other women in the office or field if they too suffer from this problem, building an alliance against men in their company who do this. And soon you will have a legal problem. Leave it at the door, pick it up on your way out. No one else wants it.

(3) Some tech women dress up for work. It is NEVER for you. Many tech women wear clothing which makes them feel good. For some, comfort is paramount, if for example the tech female is crawling through the ceiling, moving dusty panels and running CAT5 cable. For other tech women who would not get their clothes ruined at work, they like to dress up. “It makes me feel confident. I look at myself in the mirror and I feel good.” my female colleague told me. For tech women at work, feeling “good” does not mean “sexy”, and it is not for you at all. It is entirely about self-confidence, self-encouragement, and giving one’s self the extra strength to prove they know their stuff in a technical environment. Note the emphasis on “self”: it is entirely for her, by her, and your reaction is entirely irrelevant.

I have heard males say horrible things in professional environments like “Well, you wore that dress, you do look great in it, that must be the reaction you wanted. Isn’t that why you wear that dress?” The answer is no, fool, get over yourself.

(4) Tech women are generally open-minded about what is commonly called “guy humor” and “guy socialization”. Guaranteed, many of them, myself included, have male friends with which they hang out on a regular basis, so this is far from a foreign concept to tech women. Chances are, the tech women of your group would enjoy your jokes and would like to be invited out for beers, as long as points (1) through (3) above are met. I personally enjoy and share many of my own raunchy or lewd jokes if I feel safe around the people with whom I’m joking. I enjoy hanging out afterwards over a beer or two, or going out late with “the guys” to a bar to welcome the “new guy”. These things could be fun for everyone if (1) through (3) are in order.

(5) To the men who do not do any of this: Thank you so much. We notice, and greatly appreciate this. I have been fortunate to work with some excellent men in tech, and I wanted to thank you and the many others for not being this way.

(6) No, women are not perfect. This article doesn’t imply or suggest that women are close to prefect and men are far from it. I know there are female stereotypes not mentioned in this article, mostly because I personally don’t find them in tech environments. Your experience may vary. All of these points can be applied to both genders. But the fact that I was asked by several different sources to write this article proves that there is a recognized gender divide in many tech spaces. All of what I have posted is what I and others have observed and experienced. None of it is fiction.

(7) Is someone making you feel uncomfortable? Speak up! If someone at work makes you feel uncomfortable, tell them so. If you feel discomfort coming from another person, and you think you’ve caused it inadvertently, say so. Make it clear and shove it out of the way as quickly as you can, so work can continue. This applies from/to men and women.

(8) But isn’t creating a women-only group, and using terms like ‘male behavior’ reverse sexism? Doesn’t this defeat the very goal you wish to achieve? My response is no, not if these tools/verbiage are used to try to ultimately achieve equality. If it’s used for mudslinging, or through some act of elitist exclusion, yes, it is reverse sexism.

Credits: Many thank yous to Carla Schroder for sharing her infinite wisdom and encouragement. A huge thank you to all of the women at LinuxChix.org for your tireless support of the cause over the years. Thank you to DevChix.com for giving my wayward articles a very worthy home. Thank you to the many readers who have left constructive criticism and comments.

Python Web Frameworks: Are Template Languages Worth using, or is Python enough?


CherryPyDjangoPythonThoughts

Last year I was tasked with the job of evaluating web frameworks and template languages, picking a set of tools, and using them to rewrite chunks of our web framework, one chunk at a time. It was a fun project (thanks be to Bill). I had months to try out various frameworks, content servers and template languages before we committed to anything and moved forward.

With respect to the web framework itself, certain things swayed my decision. We had an existing database and a whole suite of home-grown Psycopg objects to access this data the way we wanted. I was very reluctant to rewrite or replace all of that, because it contained much of the business logic in a way that the entire group understood and was used to seeing/changing.

I was tempted by Django, but realized that we would not need or use it’s very nice database administrative interface (even though I had no trouble whatsoever retrofitting our existing schema into Django. That was quite impressive). At the time, I wasn’t sure if we would use Django’s template language either. Then I discovered that Django did not have multiple database support. These factors made me shy from Django and back toward a thinner Python web framework. I wanted next to no explicit database support, and the freedom to choose whatever template language I wanted.

My best choices at the time were PythonPaste and CherryPy, which were running neck-and-neck in terms of rolling out new ease-of-use features, and WSGI support. I thought PythonPaste had a slight advantage over CherryPy at the time. I used it in earnest, only to find out that the bugs I posted to their user’s list were ignored. There were also times where the new release egg files were broken, and brought my prototype down to a grinding halt. I also found that the configuration file syntax was a bit obscure, and documentation was sparse. I was foced to ask the one guru, Ian Bicking, for help, and hope that he’d answer my post. Sometimes he did, and sometimes he did not. This was not comforting, to say the least. “Did he respond to your bug post yet?” My boss would ask. “Nope, I have to wait another day and see..” I responded time and again, until I finally dumped PythonPaste and gave up.

CherryPy had it’s own quirks. I experienced some header bugs, some oddities in cookie formatting between it and PHP (only to discover that CherryPy did it according to the official spec, and PHP did not). Its configuration syntax was also a bit obscure. But then CherryPy 3.0 was released, and this changed everything. WSGI support was easier to implement than ever in PythonPaste (although I think PythonPaste put the pressue on the CherryPy group to add this functionality, so PythonPaste did serve a good purpose in that respect). The configuration syntax, the root instantiation syntax, object and URL nesting all became much easier in 3.0. The support list is usually excellent, where several developers post responses in a day or two. There was no doubt that this was the direction to take, and it is still my favorite thin web framework.

CherryPy imposes no requirement to use it’s own template language, called CherryTemplate. This is a great! It gave me the opportunity to evaluate template languages as a separate entity. Here is what I found.

One of the goals of the new web framework was to migrate ourselves away from Zope. Much of our administrative interface was, and still is written in Zope using DTML. As part of the evaluation of web frameworks, I tried learning ZPT, which is a tag-based template language. Its syntax was a bit obscure, and it’s python interface was not as robust as I had hoped.

We were not generating dynamic XML content, so a tag based template laguage was not for us. I could see this as being a handy tool for XML content, but then again, there are other Python XML content generators with less obscure syntax. DTML is similarly obscure (although it has try/except syntax, which comes in handy when testing variable values). DTML is not pretty. It’s SQL interface is handy from within Zope, but not handy outside of that scope. Despite the fact that we used it for most of our web interface within Zope, none of us enjoyed using it. I found the magic DTML document online, and became decent at DTML, but I still did not like its obscure syntax. Also, all of us at various times were left wondering “Where did it get THAT value from?” I blame Zope for the variable ‘magic’ that had to constantly be deciphered when passing data to DTML, but it still made the whole experience one that none of us wanted to repeat.

In the end, it was difficult to justify either DTML or ZPT outside of the Zope scope, so both were out of the question in our new web framework.

I tried Cheetah, which seemed simple at the time, but too procedural for me. I wanted a template language which was more object-oriented in design, and as close to the Python syntax as possible. My logic was “Why should I learn/impose a new language on web developers if they already know and use Python?”, because in our small shop, the developers also handled web content.

Kid was the next template language I played with. It seemed interesting as an XML based template language. It has separate template and template generation components, nice for XML generation of fixed templates. But in our non-XML environment, many of our page templates are dynamic. They change based on whether the user is logged in as an administrative user, their group rights and access, etc. Kid turned out to be overkill, since for our application, it is much easier to dynamically generate both the template and the content based on the user’s login mode and group settings. Kid is nice, but not for this non-XML, dynamic template application.

MochiKit was next. I was thoroughly impressed with it’s ability to make javascript usable by a Python developer. Plus, the signal library sounds nifty. Contact the server when certain events happen, instead of pinging the server every N seconds, and letting the server figure out what event happened? Wow, how neat. Unfortunately this too was overkill for our particular application, and I could not justify it in the scope of the project. “Hey, who wants to learn javascript?” did not fly well in our group, so I skipped it.

Lumped along with CherryTemplate were some others, whose names escape me now, with the same philosophy: Provide Python-like syntax and variable expansion. This doesn’t offer much to native Python developers, where variable expansion is already so easy.

I ended up using no template language. Python seemed to serve our needs very well. I defined our own /Templates subdirectory for each module/URL path in CherryPy. Each template file contained syntax like this:

<table border="0" cellpadding="0" cellspacing="0">
<tr><td>

        <p>Client ID: %(client_id)s</p>
        <p>Account Expiry Information (%(VEDIT_OPTION)s)</p>

        <a href="%(STATIC_URL)s/%(inventory_link)s/ViewInventory?client_id=%(client_id)s">Unfulfilled Inventory Orders</a>
        <a href="%(STATIC_URL)s/%(history_link)s/ShowHistory?client_id=%(client_id)s">View Inventory History</a>

        </td>
        <td>

        etc...

The %(variable)s Python template syntax expands out to the variable name by the following syntax:

class MyCherryPyPageGenerator:

        def index(self):
                STATIC_URL = "https://my_site.whatever"
                TemplatePath=os.path.join(os.path.dirname(__file__)) + '/Templates' # set TemplatePath to current dir/Templates
                client_id=505
                inventory_link= 'manage/inventory'
                history_link = 'manage/history'

                if user.admin_mode:   # Optionally set certain links based on login mode.
                   VEDIT_OPTION = '''
                      a href="%(STATIC_URL)s/admin/EditAccount?client_id=%(client_id)s"&gt;Change Your Account Settings</a>
                   ''' % locals()    # Expand STATIC_URL variable in the VEDIT_OPTION variable now.
                else:
                    VEDIT_OPTION = "Access Denied"

                fd = open(TemplatePath + '/template_file')
                this_page = fd.read() % locals()
                fd.close()
                return this_page

        index.exposed=True

This syntax opens the template file shown above, conditionally sets the required local variables, expands them using the locals() syntax, and returns the CherryPy page to the browser. Ignoring the CherryPy syntax, you’ll see that the template logic involves setting local variables, reading in a template file, and expanding those variables.

This syntax, combined with CSS macros for standard page formatting, proves to be very powerful and simple. What is not showing up in my template, above, are CSS id tags, such as <td id=”leftcolumn”>, which are CSS macros, defined in a separate CSS template file.

For our needs, this seemed perfect. It is entirely in Python, not too bad to read when compared to reading another template language, and easy for everyone to modify and maintain. The advantage to this approach is that the templates themselves can be as dynamic as you wish. Entire section of a page can be displayed or not, based on your explicitly defined variables and logic. The formatting is entirely controlled by CSS, and can be overridden in Python when appropriate.

The downfall to this approach is that complex pages generate very long pieces of code, where the template and the business logic are intertwined. Is this a downfall? I am not sure, because I have yet to find any single template language which can cleanly and easily separate business logic from template code, and leave you with something legible.

My Conclusion

Each template language exists for a reason or a season. I think ZPT and DTML have had their days in the sun, and those days are now over. As AJAX grows in popularity, MochiKit seems like something very worthwhile, but requires an entirely separate development effort. It is also up against a very powerful Javascript library called Dojo, which offers a browser based editor and graphics package in addition to standard Javascript functional libraries. This is tough competition.

Many template languages are simple enough to provide the template abstraction you need, but are not robust enough to justify not using Python directly. So many of them promise to separate business logic from template design. But in reality I have yet to see this happen. Very often, the person making the template change has to know about the method returning it’s data, or even about the db schema itself. With the exception of CSS designers, I have not seen a template designer who did not need some level of programming experience and access to the interface code.

I see many template languages tout their speed and similarity to Python. My immediate question is: “Well, then why should I not simply use Python instead of your template language?” My wish for the future of template languages is that they do/don’t do the following:

1. Either make more use of CSS templates, or use a similar CSS ‘#’ macro syntax which can inherit from a CSS template passed into the template language. That would be so nice. CSS is not going anywhere anytime soon. And let’s face it, it rocks if you know how to use it.

2. Don’t just brag about how similar it is to Python, and how fast it is. Show me how it can help me truly separate business logic from content, without adding a level of obscurity.

3. Make it support DOM2, please! This one is personal for me, since I am doing DOM2 server-side manipulation in my current project. A template language with a good DOM2 structure, possibly one which can be passed to other Javascript libraries such as Dojo, would be so very nice.

Comments are welcome. Examples will be incorporated into this article with author credits, so please post or email some useful examples.

Did you work with a template language not mentioned here? Please let us know about your experience.