Caffeine Lab

In Play’s Eclipse plugin, I just added a simple integration with the test framework. Currently, all it does is opening the test page directly within Eclipse.

Play Tests in Eclipse

The obvious next step would be that, in case of a failure typically, links from the page brings you directly to your code at the correct line. That would be also very useful for error pages, just like Guillaume did in his Textmate integration. So I did some research, what is it possible to do with Eclipse?

I found a very old bug, opened in 2001, to support opening files directly from the command line. After nearly 10 years the bug is finally fixed: it’s in the not-yet-released 3.6 version of Eclipse. It’s such a basic feature it’s surprising they couldn’t do it earlier, but at least they eventually did it. So with a nightly build of Eclipse, you can open files like this:

eclipse -name Eclipse --launcher.openFile <qualified file name>

That’s the most important piece already in place, so it’s good news. But that’s not enough. To jump right to the error, you need to be able to open the file at a given line number. You don’t want to have to search in your code for the line responsible for the error. So I opened bug 305336. Looking at the patch for bug 4922, adding the line number seems trivial to do. I’ll probably submit a patch and cross my fingers to have it in 3.6 before it’s released.

The last piece we would need is Eclipse registering itself as a protocol handler. That would allow us to insert URLs such as:

eclipse://open?url=file://<qualified file name>

Again, I opened a bug (#305345). It’s not as critical as the other items because this one can be created manually. For Windows there’s a key to add to the registry, for Gnome it’s a key in gconf and for OSX I’m sure there is something similar. Of course, it would be better if it Eclipse registered itself so the user won’t have to mess with his registry.

Gandi Cloud is a cloud hosting service, similar to Amazon’s EC2. Since you get full access to a Linux installation, deployment is similar to using a dedicated server. They have prepackaged solutions for various frameworks but not for Play. Fortunately deploying Play in general is pretty straight-forward.

Note that, just like EC2, it’s not free hosting – you have to pay for it. But they have fairly reasonable prices.

1. Create a server on Gandi

Make sure you choose a solution that gives you root access to the machine, “expert mode”. In the rest I’ll assume you choose Ubuntu, but any distribution would work.

2. Install the Requirements

ssh to your server, and su to get root access. Now:

apt-get install openjdk-6-jre unzip

See how fast is the download, I bet they have a local proxy!

3. Download Play

exit to get back to regular user mode, then:

wget http://download.playframework.org/releases/play-1.0.1.zip
unzip play-1.0.1

4. Run It!

Now you’re ready to run your app! Copy your app to your server by using scp for example, switch to to prod mode and you can run:

play start yourAppFolder

That’s it, your application is running on your Gandi cloud server.

5. HTTP Port, database

Since it’s a Cloud environment, you can easily create one server for each app. In this case, just configure Play to run on port 80 and you’re done. But if one Gandi server part is too big for one application and you can’t afford buying too many parts, you can run a web front-end to host several Play apps on the same server.

See the Play documentation about running in production for instructions on how to do it.

For the database, you could configure one manually on the same server, but it’s much easier to create a separate MySQL server. You can create a server in “Gandi AI” mode, where the server is managed by Gandi and you don’t have direct access to it. That way, you can create a MySQL servers in a few clicks and point your Play app(s) to that separate server.

I will be at FOSDEM in Bruxelles on Saturday 6 February. I will make a talk about the Play framework in the Free Java session, at 17:15.

Busy as I was, I realized I didn’t blog about my recent employment change. I left Yoono 2 months ago to join a company called Zenexity (site in French). It’s really cool because after Flock and Yoono that were very similar (consumer oriented/social mashup/Mozilla technologies), I get to work on really different stuff: more server-side, and more business oriented. But still with a strong R&D component, and it’s something that really motivated me to get on board with Zenexity: they’re independent because they earn their own money (e.g. don’t live on VC money) but still spend a lot of effort in R&D projects. Projects for customers also are really state-of-the-art of the web.

Specifically, they (I mean “we”) have an Open Source project called the Play! Framework. It’s an MVC framework similar to Django or Ruby on Rails, in Java. Within the Java world, I think it’s pretty disruptive. It contrasts from bloated stacks, and manages to provide simplicity and productivity to Java web development. Also, it speaks the language of the web by making it easy to create RESTful web apps, pretty URLs and web services.

Here is a screencast I did last month for the 1.0 release.

A web app in 10 minutes using Play! from zenexity on Vimeo.

I recently installed the SDK for Android, Google’s OS for mobile phone. I’ve been following that since it was released; I’m pretty attracted by the platform but when I bought my last phone, Android was not really there yet.

One of my requirement is to be able to input Japanese. Currently a Google search about Japanese input will give you mostly links to some third party hack who get you there, but with a clunky interface. Well, I was happy to see in the SDK’s emulator that Android has support for Japanese input built in! You can select it by going to Menu->Settings->Locale & Text.

Now, that doesn’t tell me if I will be able to access it on an Android phone. Sometimes makers and phone operators are known to rip off parts of the OS or add crap for no good reason. If you have an Android phone, I’d love to hear what maker/model it is and whether you do have alternate input methods included.

Seeing the progress done by Chromium on Linux, I wanted to implement an equivalent of a Firefox extension that I like: automatically generating QR Code to send a link to my phone easily.

QRChrome Extension

So far, here is what I have been able to do:

• Add a page action icon in the URL bar
• Open a new tab with the QR Code for the URL you want

It seems straight-forward but the browser don’t seem to behave exactly as documented, and I didn’t found anyway to debug – at least on Linux. I assume it works better on Windows.

Here is what I want to do before the extension can be really useful:

• Just like the Mobile Barcoder, Show a popup on hover rather than opening a new tab.
• Add a item in the link context menu, so the user can generate a code from a link – not just from the URL bar. Useful for download links.
• Package the extension! That may sound stupid, but extensions are packaged from Chrome directly – and it doesn’t work on Linux. It’s not something you can do from command line… Well, actually you can package from command line but using the Chrome binary and that doesn’t work either

PS: The feed icon is a sample extension from Google. I don’t understand why it’s not in the core.

I recently learned that Kiyoshiro Imawano died last May, from a cancer at age 58. Most people outside Japan don’t know him, but he was a very famous and very controversial pop singer. He was unique, talented, funny and sometimes engaged.

He is famous for his punk version of the Japanese National anthem Kimi ga Yo, that became banned and can no longer be aired on Japanese media. That reminded me of Gainsbourg’s reggae version of La Marseillaise, very controversial too.

What I want to share today is a song about North Korea. To a US audience the part about world peace may sound a bit cheesy, but you have to understand that how Japanese people hate North Korea. For most Japanese thinking about friendship with North Korea is just insanity, and this is truly a courageous song.

As my friend Nicolas suggested, I decided to give some guidelines and pointers about SEO (Search Engine Optimization). Now, full disclamer: I’m not a SEO expert, so what I am going to write is just very basic rules.

Also, since Google is by far the most popular search engine, I will focus on it. I assume visibility on Google is the most important for most webmasters. Anyway, what is valid for Google is usually valid for other search engines as well.

Rule #1: Don’t Try to Trick Google

The reason why Google is so successful is that they managed to be smarter than shady webmasters who tricked search engines. With old search engines you could put 50 times the word “notebook” in a comment to get a decent placement on the search for “notebook”, and you could also hide “Pamela Anderson” in a completely unrelated website to get some visits. Google, with their pagerank and smarter analysis of pages, created better search results for users.

That’s what they are good at, and that’s why they make so much money. So you can bet they’re spending a lot of money and energy keeping this edge on unethical webmasters. I won’t get into details but there are tricks such as cloaking, link farming, page hijacking… You don’t want to use these techniques.

In short, you should play by the rules if you don’t want to get punished. Read Google’s webmaster guidelines. When you are done, you can also use their webmaster tools to check your website and create a sitemap.

Corollary: don’t hire a SEO consultant who would use shady techniques.

Rule #2: Have Good Content

It may sound obvious, but you can’t just generate some crappy pages automatically and expect search engines to index it well. Some people do that by shamelessly sucking content from blogs, but it doesn’t work.

Make sure your site is useful for real, human visitors. Also make sure that pages in itself are useful, that a user can jump in the middle of your site and get what he wants.

Rule #3: Be Readable

Search engine spiders don’t see pages like human do: they only understand plain text. That means:

• Avoid Flash, in particular for content. Search engines won’t be able to read it so they can’t index it.
• Avoid text as images. If you have to do it for titles because you want to use your fancy font, make sure put the text content in the alt= attribut.
• Avoid text generated by Javascript. Search engines spiders won’t run the Javascript.

You can somewhat simulate the experience of a search engine by looking at your website using the Links web browser. It’s text-only, not fancy, but you will learn a lot about your site.

Rule #4: Understand the Pagerank

The pagerank is a metric introduced by Google to measure the popularity of a page. It’s a number between 0 and 10. The inbound links you have, the highest this number is. If pages with a high pagerank link to your page, you get more “pagerank juice” and your pagerank is even higher.

Additionally, Google prefers inbound links from pages that are in the same category as yours. For example, a page about computer programming will benefit from links from other computer programming related pages, but not so much from pages about breeding dogs.

That means that you should develop your network – get in contact with people who have similar pages, make sure they know yours and that they link to you when it makes sense for their users.

For your website internally, that page rank system also works. A given page that gets popular may get a good pagerank, and your whole site must be architectured in a way that will redistribute the pagerank to other pages. That means not having more pages than necessary; if you do have too many pages (for example content broken into 5 pages when it would make sense to be one, or dynamic pages that don’t give much more information), your pagerank juice will be divided into those pages instead of channeled to the one page. So be careful when you think about what should be on its own page and what should be integrated to another one.

Rule #5: Optimize your Content

This section could go for over and over, so I’ll just limit myself to the basics.

• Design good titles. Not only that’s what your users see in the result page, but the keywords there weight a lot for search engines. Also put the relevant information first, like in this blog I put the article title before the blog title.
• Avoid duplicated content. That usually happens when you have several URLs that show the same content. For example, if you have http://www.mysite.com and http://mysite.com, that’s two pages for a search engine. Having both URLs working is a good thing, but make sure one redirects to the other.

Going Further

This is clearly just the basis, but if you manage to get enough inbound links you should be indexed decently. Next thing to do is to find out what keywords you want to optimize to, and monitor that on your favorite search engine. There are also a lot of books about SEO that can help you go deeping into search engine optimization.

A few days ago I’ve been invited to a launch party for a web product in Paris. While the product was nice and polished, it seems like the developers didn’t understand anything about scalability. They didn’t even understand my question when I asked them if the product could scale.

It’s probably not a big deal for them: they were presenting a CMS, so most of the time it will be installed for a limited user base. I guess most people will be happy to use it on a single server, so it’s probably OK for them not to be able to scale. However I noticed that while scalability is now a fairly solved problem, there are not that many articles explaining how to prepare to scalability on the web. So here I go. I will not try to replace a good book, but just to give the very basics.

What is scalability?

It’s important to get that out of the way. Scalability is not performance: it’s not about making good use of CPU and bandwidth, and it’s not about having the page being loaded quickly in the user’s browser. It’s about being able to balance the load between several servers. So when the load increases (more users creating accounts, more visitors, more page views) you can add additional servers to balance the load. You don’t just throw in a server, you need to design your software to work on a cluster of servers.

An other point is that you will rarely create a cluster of machines from scratch: when you launch a new website you will have few users so few machines (one or two), and as your load increase you will increase the number of servers. You will have to scale different parts of your system one after the other.

#1: the web front-end

Most of the time you start with a front-end (PHP, Python, Ruby, Java…) and a data layer (MySQL, PostgreSQL, CouchDB…). As your load increase, the front-end will be the first to break. Of course server-side caching will help, but at some point you will need several front-end servers.

The key for that is to ensure you don’t store any data on the front-end. The problem sometimes arise with sessions: a lot of PHP libraries store session information locally on the server, and that prevents from balancing the load. The idea is that in a session a user may hit a server for a given page, then an other for the next page. If the session is only accessible to the first server, you’re screwed. You want it to be somewhere else. That can be in the data layer or in a special sessions server. If you write a Facebook app you don’t need to care, because Facebook takes care of the session.

Now can have as many front-ends as we want, but we have a unique database server.

#2: the read operations on the database

Most applications will have many more reads than writes. For example in a blogging software, each visitor will trigger a read on the database (OK, not each visitor if there is a good cache), but writes only occur when the author writes a new post or someone leave a comment.

That’s good, because it’s much easier to scale reads than writes. Just make sure that in your code you have different settings for reads and writes. They can point to the same database at launch time, but when the time comes you can separate those. Writes will go to your “main” database, and reads will go to a copy. There are other approaches, but for example MySQL offers replications features. Once set up, the slaves will stay in sync with the master. You can have as many slaves as you need.

OK – several front-ends, several read-only databases, but still one master database for writes. If your applications has few reads it may be fine with a beefy database server, (and some major websites just have one master database), but if you have a lot of writes (highly social applications like Facebook or Twitter) you may want to continue the scaling process.

#3: the write database

Now we want to have several databases where we can write to. Obviously, we have to be careful not to introduce inconsistencies in the process. So having an old version of a blog post on a server and the new version on an other one is not great; what if some users see an old version of your post and others see the most recent one ?

There are various strategies to divide data in a safe and consistent way, including:

• Depending on the userid (or blogid, or whatever makes sense in your application), put the data on one server on an other. For example, all users with an even id go to server1 and all users with an odd id go to server2. Hint: make sure your algorithm lets you add more servers later, which is not the case with my example where you will be stuck at 2 servers
• Put some tables on a server, some others on an other. It doesn’t help you when a table is growing too much, but it can be combined with the previous point.

Conclusion

Here you go, the basics for building a scalable website. That’s not all you have to do, if your website continues growing you will face more problems such as having to scale your network. I’m not talking about outgoing bandwidth but communication between your servers (front-end and data layers). But if your code is efficient, those simple recommendation will get you to a server that can handle a fairly big load. I really recommend Building Scalable Websites, from O’Reilly if you want to know more.

FAQ

Q: Language X doesn’t scale, but language Y does!

A: Bullshit. It’s not the language that scales, it’s your code. Some languages may not perform as good as others, so you will have to add boxes more often but the way you scale is still the same.

Q: What about cloud computing? Virtualization? All these fancy buzzwords?

Virtualization means you run on virtual machines rather than on physical ones. The benefit is that you can easily add or remove machines. For example, using Amazon EC2 you can add as many machines as you want in a few minutes, and then remove them in no more time. With a classical hosting company, you need to make a phone call, ask for the machines and you get them in maybe one week. They’ll charge you for the set-up too, and if you no longer want it you still have to pay for a full term. So cloud computing offers are generally more flexible.

Q: Does Google App Engine make it easier to scale?

In short, yes. By not letting you access the machines, Google App Engine constrain you into writing scalable code. You also don’t have to request new machines when you need them or release when you no longer need them; you just pay what you use depending on the load of your application.

I am a big fan on Google App Engine but be careful, since it’s programmed in a particular way it’s not easy to move your project out of it. You may feel locked in after you project started.

It’s always a pleasure when a company offers a public bug tracker for users and third party developers. So it’s great to see that Facebook has a public bugzilla.

The problem with having a public bug tracker is that you’re supposed to be responsive and fix bugs; nothing worse that real bugs rotting in the tracker because the group behind the product doesn’t really care about that specific bugs.

So far I’ve filled a few bugs for the Facebook API:

• A security bug, letting anyone deactivate feed templates for any app (fixed)
• 5159: Incorrect error code returned for Stream.get – I can’t believe they won’t acknowledge it’s a real bug!
• 5624: Stream.get is not really using the updated time. In short, we miss data because the query doesn’t do what the doc says it does. They do recognize that should be fixed, but didn’t put a very high priority.

I have also filled a few feature requests, but it doesn’t really matters: they have obviously no obligation to implement them. On the other hand, it would be nice if they fixed the bugs even when they’re not security issues.

One interesting thing I’ve noticed is that not only they have a public bugzilla, but the code source for their API is Open Source. Let’s see: “Facebook Open Platform is a snapshot of the infrastructure that runs Facebook Platform. It includes the API infrastructure, the FBML parser, the FQL parser, and FBJS, as well as implementations of many common methods and tags.”

Cool. That means that if I’m motivated enough, I could fix the bugs and submit a patch! Let’s look at the buggy Stream.get method… Wait a minute – that source code is just a tiny subset of the Facebook API! Oh crap. Looks like I can’t fix it.