Blog

Why unit tests are cool

tero.heikkinen@iki.fi (Tero Heikkinen) — Sat, 03 Oct 2009 07:52:08 +0000

It has been a while since my last posting. Quite a lot of business and lack of personal time to write. There has been more important things like finishing up studies and master thesis.

One could be thinking why write those unit tests. Especially if you have buggy free code or libraries what is the catch there. It's labourfull to make those tests. But still you are most likely to do some interesting test cases that you were not thinking while implementing your original solutions.

I call these to "grey" areas. There might be cases where your function returns nothing or null when it should return true or false. This can happen quite often with PHP. You want to make simple function but then something goes wrong when the code above your basic function behaves badly.

In fact when writing your unit test you should have very destructive mindset and write also those "should be impossible" combinations. And what should your function do when something "impossible" happens? Should it try and correct the situation?

I've come to conclusion that strict approach is the best. Depending the programming language those grey areas are best addressed with for example exceptions. And why? The programmer above immediately gets feedback that something is going wrong thus preventing further damage. So put some informative message to that exception and be defencive.

Write those unit tests!

Open source and govermental funding

tero.heikkinen@iki.fi (Tero Heikkinen) — Sun, 12 Apr 2009 18:49:09 +0000

I've been recently thinking should Midgard project after it's transformation to a fundation apply for some govermental (TEKES) funding or something else from EU level.

Currently Midgard is developed with normal work (testing and stomping mines) or with my spare time (weekend hacking sessions) from my part. But in the big picture lot's of developement is funded by companies that are really and only using Midgard for their living.

R&D costs are considerable and I think that there should be some funding programs that coming Midgard Foundation can apply financing from. Commercial products do it shouldn't we also?

List of projects funded by tekes (in finnish)

New year comes with big changes

tero.heikkinen@iki.fi (Tero Heikkinen) — Sat, 07 Feb 2009 18:37:56 +0000

It has been a while since my last post. Mainly because great stress at school and my personal life. It has also been very interesting beginning of a new year. I bought my very own apartment and the company that I'm partner in bought majority of another company. That means that lots of new technology knowledge must be absorbed as quickly as possible.

Good thing is that it broadens up a perspective about how to make different kind of web services for consumers. That knowledge comes through all layers from the very top to the very bottom of the code itself. I've encountered some innovative solutions as well as the problems that will be encountered when the amount of data and traffic goes to another level.

Year 2009 means lots for the midgard itself. New Vinland (Midgard 2.0) will be released. The possibility to use Midgard as content repository of your application has became a reality. It's quite easy to do midgard + python + guid application that you can replicate to net. Also new codebase gives possibility to make enhancements to Midgard Core itself to make it very versatile tool for developers. But most importantly, new Core gives possibility to rethink some design rationales behind all. As far as I know when data storage grows and it's usage intensifies rules change at the very level of data storage and abstraction. How Midgard scales to this? By the developer meeting at Linkköping I'll try to find out.

Many aspects of testing

tero.heikkinen@iki.fi (Tero Heikkinen) — Sun, 21 Dec 2008 19:39:35 +0000

Christmas is almost here but there's still yet another exam ahead. This time it's about testing and quality assurance of software. The fact is that it can be done with quite workfull way. For example we had a exercise where we had to test a java program with black box methodology. When doing it properly it takes quite an effort.

But now when the sanity points and mental energy is almost gone after several exams there must be something wrong when reading dry text and I start laughing when stumbled following: "Soap-opera testing"

Luckily christmas vacation is only two days away.

Caching thoughts

tero.heikkinen@iki.fi (Tero Heikkinen) — Sat, 22 Nov 2008 11:09:13 +0000

I've previously written something about caching and it complexity. A short while ago I've heard from Bergie about interesting facts about memcache and also been bitten by it first hand.

It seems that with bigger sites with current MidCOM caching philosophy and level of atomicity are getting more heavier than just serving pages without any cache. It seems that if Memcached is heavily punished and if there's lots of keys in it it start so slow down and also reliability goes down. Basically it means that you don't always get data from cache or be able to store it in there resulting some database punishment and you can imagine what happens next.

This comes again to MidCOM 3 and the way caching should be handled there. I've taken a close look about how Zend Framework does it. They are useing tags as possible cache identifiers. Basically MidCOM's cache works quite same way but in fact every cache entry get's autotagged by lots of guids. So that is basically the problem.

One possible solution that I've pondered would be to give developer some additional tools to control caching in controllers. There could be some hint's for cache's level like if it's same for all users or per user basis. This helps caching to be done more intelligent way

But in addition to this there should be a concept for cache invalidation logic. In some cases you need more atomicity than others and the problem was with too big guid indexes. What if developer has in addition to tagging possibility to attach some callbacks to tags. Basically it means that you have possibility to call cache invalidation per tag but then those callbacks could do some logic whether or not tag entry really needs to be invalidated.

That would enable us to diminish the size of guid cache, put some intelligent invalidation where needed. Bit heavier process it is but if it's lighter than getting data and rendering it, then it's worth to explore.

How Midgard 2 talks between between machines

Sat, 25 Oct 2008 16:30:48 +0000

I gave today a workshop about the basic functionalities of Midgard 2. One of them is the dbus signaling and it's possibilities. After that we hade quite intense hacking session with XMPP and Midgard 2. And what did we archive?

I described in one of my blogposts how easy it is to signal. So when some code is added, I can send serialized midgard objects via XMPP

Basically there's some normal code for initialization for the dbus and twisted framework in python but the magic happens with

# initialization code
....

def replicate:
        # some preparing code for connection etc
        msg = domish.Element(("jabber:client", "message"))
        msg["to"] = "bergie@example.net"
        body = msg.addElement("body", content = replicationpackage)
        self.xmlstream.send(msg)

mbus = midgard.dbus("/mgdschema/midgard_page/create")
mbus.connect("notified", replicate, "foo")

Now we need a working receiver and probably Bergie is going to tell about is some more. This rocks we'll may have sooner than expected some P2P applications with Midgard.

Some documentation about installing Midgard 2

Sat, 25 Oct 2008 10:58:13 +0000

Required os: Ubuntu 8.04

Installing newest midgard is bit tricky. After all it's beta software so the installing scripts are not yet done.

1. Add following lines to your /etc/apt/sources.list
deb http://download.opensuse.org/repositories/home:/midgardproject:/ragnaroek/xUbuntu_8.04/ ./
# deb http://download.opensuse.org/repositories/home:/midgardproject:/vinland-dev/xUbuntu_8.04/ ./

2. apt-get update
3. apt-get install midgard libgda3-3 libgda3-common php5-memcache libgda3-mysql git-core python-gobject python2.5-dbus php5-syck
   - Ask yes for the midgard related questions
4. datagard
   - Enter your hostname when it's asked
       - Otherwise you won't be able to access web interface
   - Use default values except answer yes to MidCOM3 related question
5. Download and install all packages from http://download.opensuse.org/repositories/home:/midgardproject:/vinland-dev/xUbuntu_8.04/ that are related to your system architecture
6. Link /etc/midgard => /etc/midgard-2.0 and /usr/share/midgard => /usr/share/midgard-2.0

Now you got a midgard 2 installed on your system

6. git clone git://repo.or.cz/midcom.git (Installs MidCOM 3)
7. edit /etc/midgard-2.0/apache/vhost/hostname
   - add: php_value include_path .:/usr/share/php:/root/midcom
   - add: MidgardRootFile /root/midcom/midcom_core/httpd/midcom-root-midgard2.php
   - Comment other MidgardRootFiles off

8. Ensure that line extension=midgard.so is present at /etc/php5/apache2/php.ini

9. Add following file
/etc/dbus-1/system.d/midgard_dbus.conf
with content

"http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">





10. pear install http://phptal.motion-twin.com/files/PHPTAL-1.1.13.tar.gz
11.    pear channel-discover pear.phing.info
   pear install phing/phing
   pear channel-discover pear.phpunit.de
   pear install phpunit/PHPUnit

Midgard workshop at FSCONS

Tue, 21 Oct 2008 18:33:53 +0000

There will be a great Midgard related workshop at FSCONS. The focus of the workshop is to present the next generation of Midgard and great features that it offers.

What are those great features that are presented? Easiest and the most usable feature is the PHP & Python support and DBUS. Let's make a short snippet of code for python and for PHP that do the same thing

$cnc = new midgard_connection();

$cnc->open('midgard');

$cnc->set_sitegroup('mysitegroup');

import _midgard as midgard

import gobject

import dbus.mainloop.glib

cnc = midgard.connection()

cnc.open('midgard')

cnc.set_sitegroup('mysitegroup)

Both snippets do the exactly same thing. Open up connection to Midgard and after that the whole power of Midgard and it's features is at your disposal.

Let's make another short snippet that continue those I wrote earlier.

$page = new midgard_page();

$page->name = "Testing 123";

$page->create();

$page->name = "Final 123";

$page->update();

page = midgard.mgdschema.midgard_page()

page.name = "Testing 456"

page->create()

page.name = "Final 456"

page.update()

Again. Both do the very same thing. This same kind of data abstraction is available for C and soon for Mono (C#). But let's take this a bit further. What if I want have to put all latest articles to some index or search engine of my own?

Let's make one python snippet.

import _midgard as midgard

import gobject

import dbus.mainloop.glib

import my_indexer

def index(object, args):

# LOTS OF MAGIC

cnc = midgard.connection()

cnc.open('midgard')

cnc.set_sitegroup('mysitegroup)

mbus = midgard.dbus("/midgard_article/create")

mbus.connect("notified", index, "foo")

mainloop = gobject.MainLoop()

mainloop.run()

As you can see it's that easy to make a python application that listens for creations off midgard_article objects. And this means all articles even if they are created at PHP - level. Piotras deserves big applause for this archievement!

When you play with PHP I suggest that you use MidCOM 3 component framework. It provides nice services that take full leverage of Midgard's services. But think what you can nowadays do behind the scenes with nice C, Mono or Python daemons. Sky is the limit!

New Midgard, new documentation

Wed, 15 Oct 2008 21:02:52 +0000

The future Midgard + MidCOM release in the future needs also new documentation. Basically nowadays documentation is good if you are an experienced developer and are keen to understand how things work.

In a case for new employee for example the time required for learning must be shorter. And sometimes good more easily readable and structured documentation helps to clarify the thoughts of the new developer. It results better code and comprehension of the whole system.

I and Bergie had a short chat about FSCONS Midgard workshop. The goal is that we have new documentation more or less done by the workshop. That's a good place to test if it works.

The initial skeleton of the documentation comes from the company that I work in (Rohea Oy) training material. It has already been tested there once so lets see how well it works after complete rewrite.

Criminalized port scanning

Tue, 14 Oct 2008 12:44:45 +0000

Finland's Supreme Court made a decision spring 2003 about port scanning. A teenager had scanned local banks network without finding any holes. He did not manage even to try to break in to bank's systems.

Court sentenced teenager about trying to break in bank's systems. Also compensation for the bank was sentenced. Compensation was based on the expenses for the banks security audition that they "had to" take in order to be sure that their firewall is secure enough.

One may first think that this was right for the script kiddie that tried to mess around. Same kiddies tend to ddos whole network block down if they want to drop someone from the IRC for example. And same time true business damage occurs.

But there's one problem with the sentence. If you walk through a shopping mall during night and try every door. All are locked. By that same logic that court used, you are obliged to pay for the shop owners if they want to run security audition.

Interesting field of problem. But luckily when you are consultant and do the same thing. Port scan a machine and you'll get paid. Of course if the machine that you port scan is your customer's

If you are interested of the case is available in the net (in finnish).

My first conference talk

Wed, 08 Oct 2008 11:56:29 +0000

Yesterday a gave my very first conference talk. It was exciting since I've never been at any conference and I did not have any idea what is the expected level of the talks.

With Bergie's help I was able to formulate a set of slides. Presentation itself could have been better but the audience was very challenging.

It was not good idea for the organizers to couple lightningtalks and coctails. Audience were talking themselves and were not taking attention to the performers. Personally I see that audience did not give very much respect for the talks. In fact most of the performances would been greatly better but the situation ruined many good flows.

But Bergie recorded my lightning talk and it's at youtube. At least the beginning of my talk got the audience's attention.

Software patents

Sat, 27 Sep 2008 14:20:54 +0000

This writing is the second one in the series based on the course of sw-legistlation that I'm taking at school.

After copyrights the most natural path to continue exploring the wonders of legistlation is naturally patents. Patents tend to raise opinions since it has been proven that most cases patents in software fields are just not working as intended.

Most of the problems come from USA and the threat that similar conventions are applied someday in EU. It seems that patents are applied just in case for almost everything. When this is combined to the fact that the financing of the patent agency is based on the amount of patents they approve we'll have some interesting problems. Also the personnel that make the decisions are not often qualified to the field that the patent addresses.

What is the result then. In the US only the best patents are taken to the court. And about 70% of the patents are nullified there. What is the big picture then? It has been said that about 10% of the patents are truly valid.

What is the difference between the Europe and US? Well in Europe patent is given "first to file" basis and in the US it is "first to invent". You can tell what that means.

I personally consider patents as an arsenal of business cold war. Companies have plenty of patents for a fear factor. This leads to interesting cross licensing but also in the telecommunication field patents are used to regulate margets so that new and smaller players have trouble getting in. For example about every mobile communication standard has been patented.

Patents are controversial and seem to be a big mess. It remains to be seen how big mess it will become after EU decides to make some wise decisions.

Copyright in Software business

Mon, 22 Sep 2008 20:51:04 +0000

I've started a introduction course about legistlation in software business. I'll write some articles based on the memos that I've written during lectures. This first article is about copyright. One remark: the issues I discuss may contain my personal opinions and are more or less based on finnish law.

Copyright itself is an interesting concept. It is used to protect the rights of the person who does the work. This is very straightforward with classical art (writings, paintings...) The very software code itself does contain some more or less interesting exceptions.

Code has copyright because it equates to work of art. Yes! all coders are artists! But for who you work for? If you are a independent researcher, project worker or teacher at university or such your code that you do belongs to you and you only if you have not agreed otherwise. Also if you work as a private via your trade name it is not concidered as a standard working agreement so the copyrights will remain with you.

Naturally all technical plans and architectural charts are protected by copyright if they are unique and point one's creativity. But have you ever concidered if your detailed plans are stolen and someone makes implementation by using them? Months of carefull design is stolen and you know who did it and the he releases the software. Can you sue them?

This was quite suprising to know. No you cannot sue them. It's illegal to steal stuff but the implementation is completely legal since it is concidered as a separate work.

But what else is not protected? Implementations of some standard are not concidered as unique work. Even if you copy routines and code as it is and if that is the only way to implement the standard you are not making an infringement.

Last thing to point out is the copyright in opensource industry. There are some rules about screen of name and honour and requirement to give credits with good fashion. For example in Midgard Project the about screen and pear manifests implement quite well that requirement! However there is one exception. In embedded systems you cannot demand that your name will be shown. So if you've made an OS speech synthetizier that is used in elevator, you cannod force them that it should shout your name for exampe once an hour.

Next lecture is about software patents. Let's see if there is anything to write about them,

Source control matters

Sat, 13 Sep 2008 09:45:55 +0000

Recently I've been running problems when maintaining multiple concurrent branches or versions of software with subversion. Somehow the merging is a challenge since branches are not "real" branches but more or less just copies of folders. Also switching between branches in web developing can be more pain than gain.

As I started to play with MidCOM 3 I had to introduce myself to the git. The first impression was quite bad since we used repo.repo.or.cz as our storage. My bad impression came mainly because of the commit problems. But now as I've studied git some more it seems more and more the tool that I would like to adapt to every software product I'm dealing with.

Bergie blogged about the new idea where the "trunk" is stable and all new features, bugfixes, versions etc are made as new branches. First I felt a bit sceptical about this but now when I've studied and tried it a bit more I'm sold.

But lets take a look how does this work in your workgroup environtment? First of all instead of working in multiple branches things will not break up unexpected when you update. With one checkout morning checkout too often breaks things up. With branching you can leave it broken if it's your "personal" branch.

Git gives good and fast possibilities for branching. Another great advantage is that you can easily hop into your collegue's branch just by "git checkout another_branch". Merging conventions may need a bit more studying and especially branch branching. These are in my opinion project or organization related conventions.

Git could make a great improvement to software version controlling when compared to cvs or subversion. It however will need some thinking and adaptation to projects source control conventions if active branching and mergin has not been a used before.

Building a user level caching with dynamic content

Fri, 12 Sep 2008 10:36:13 +0000

Caching can be the most essential part of website developement since it can greatly affect to the site responsiviness and how well does your site cope with sudden increase of hits.

Things are quite simple when you have just static content or dynamic content where all data is common to all users. But what if some part of your site requires per user tailored content. Things start to get immediately a lot more interesting.

Lets take a normal page. There are for example following parts:

1. Header and the very beginnings of the page
2. Some personalized content. User's latest messages etc.
3. Latest news
4. Latest comments
5. The ending of the page

As a caching point of view this page should now be divided to three chunks.

1. Part #1 (public)
2. Part #2 (private)
3. Part #3-#5 (public)

So basically it means that when we got a request, caching engine should request those three chunks. There should be some logic around that kicks in at the second part. Perhaps if that code is not cached, then that code that generates it is fired.

The part two can as well be totally uncached if required so. But in the end the challenge comes how to invalidate those parts and how to keep cache as intelligent as possible. For example that private part can be same per each request. That's why for example cache keys cannot be totally request based.

Interesting challenge. Let's see how this all be done with MidCOM 3.

UI translations - Order to chaos

Tue, 09 Sep 2008 06:28:55 +0000

For MidCOM 3 the work of implementing gettext support is almost done. Most of the translation requests come from TAL templating. Some may come directly from the PHP - code and that requires some nice i18n API function.

A quick bus trip thinking about the flow of the translation spawned a requirement of a new helper component for MidCOM 3 that is midcom.helper.translator. If fact it's functionality is quite simple. It provides three key functionalities

Collector that goes through every component's all PHP and TAL files and looks for certain function calls. It also reads translations that are already made. Then it writes the results to database in order to provide easier access for the translation UI itself
The very UI for translations that makes it easy to browse and input translation entries. Especially strings that are not translated should be easily accessible
Exporter functionality is required for writing .po files to corresponding component's locales folder and running msgfmt command to that translation file in order to generate the gettext binary

This should provide nice Web based UI for translations management. Of course when doing things in right way this functionality can also be exposed via Neutron, WebDAV, Samba or other coming possible UI - interfaces

MidCOM 3 supports now X-Sendfile

Mon, 01 Sep 2008 09:54:10 +0000

Normally serving files directly from PHP can be quite resource consuming. X-Sendfile provides an alterntive way to serve files from PHP script. It's quite simple, you just send one more header line after setting your content type.

Apache's X-Sendfile module then intercepts the header and does it magic. Module also provides caching headers etc to the browser.

Setting things up is simple

Download the module from mod_xendfile's page
Compile and install it in your apache and do apache restart
Add following line to your MidCOM 3's configuration
- enable_xsendfile: true

Then all attachments should be served via xsendfile. It's a bit faster and on the second load when the browser cache hits in, it's even faster.

Pros:

Fast and low memory consuption when serving very large attachements
Allow using ACL's before serving

Cons:

Not an official apache module. Needs some tweaking.
File or any file is served without any intervention of Midgard or Midcom. So if you xsendfile directly, be careful!

Doing some lightweight test automatization

tero.heikkinen@iki.fi (Tero Heikkinen) — Sat, 23 Aug 2008 18:55:19 +0000

Most of the time when you are building up a website you constantly do some code and click testing. After you have finished up your work most of the testcases you could figure out should work.

Of course there are more elegant way to test things like writing unittest while doing some test driven developement. But still it does not make the functional testing go away. How the things work out within browser? Does this javascript snippet that I wrote work? There are plenty of places where things can go wrong.

I've been time to time tried to find out time to test what Selenium can do. Selenium is a tool for browser level testing automatisation. Basically it means that with selenium you can record the normal clickings that you do add some checkings for example if some element or text is present at the page.

There is a 2 minute video that shows how easy it is to start. I already installed Selenium IDE to my firefox and I'm quite sold.

How much more torque MidCOM 3 really gives?

tero.heikkinen@iki.fi (Tero Heikkinen) — Wed, 20 Aug 2008 12:23:12 +0000

I've been lately done lots of work with MidCOM 3. Currently MidCOM 3 has most of the important things done like basic ACL, templating, authentication and internationalization with gettext.

So how well does it perform? So far there are some optimizations to the web applications code to be made that I'm currently doing. But I still decided to make a little benchmark MidCOM 2.8 (content cache + memcached + bytecodecache) versus MidCOM 3.0 (no cache. Even Midgard page cache is disabled)

I did not have time or a suitable machine available this time to test MidCOM 2.9 with autoloading. So that part remains in the dark at this point.

Both test machine has same CPU power available. Only difference is that MidCOM 3 is run only with 256MB of RAM. Another machine has over 2GB. Both machines have somewhat similar background load.

Results

First in order to make worst possible MidCOM 2.8 run I made several independed requests with ab to get one single page after midcom-cache-invalidate.

Average request time with options ab -c 1 -n 1 done 10 times

MidCOM 2.8: 5.516 seconds
MidCOM 3.0: 0.270 seconds

Results with ab -c 1 -n 10. With this options MidCOM 2.8's caches can truly kick in.

MidCOM 2.8:
Requests per second:    3.28 [#/sec] (mean)
Time per request:       305.20 [ms] (mean)

MidCOM 3.0:
Requests per second:    3.59 [#/sec] (mean)
Time per request:       278.70 [ms] (mean)

Quite impressive. The page I loaded has 3 dynamic loads. So when MidCOM 3 has its caching engine up and running and some bytecode caching it _is_ going to fly.

The MidCOM 3 itself should not get much more functionality to be run on every request with sufficent planning, implementation and taking leverage of intelligent autoloading.

As the Midgard 2 and MidCOM 3 release comes nearer I'll take more look of the results. So far, looking very good.

What are the keypoints of business value generation with long term support?

tero.heikkinen@iki.fi (Tero Heikkinen) — Tue, 12 Aug 2008 12:59:47 +0000

As the Midgard LTS stabilization approaches it's always good to start thinking the most important factors that generate value of the product.

Because we are dealing up with an open source product the business itself is totally service oriented since no license fees are collected. Nowadays mixed business models have emerged that contain both licence and service fees. But with both cases there are more or less unexpected expenses that just occur.

Basically in the long term most of the software service expenses come from repeating tasks like installing, upgrading, bug tracking etc. With coming Midgard LTS version installation part has been quite well tested. Upgrading and bug tracking should be reviewed and discussed. How do we do software configuration management? And what are the expectations that LTS version should fullfill?

Git and OBS are good starting point for the improvement. However there still some important keypoins to address like Configuration control, status accounting, reviewing of the components and the very LTS process management as a whole. This is the way now and in the future to make better value when providing consultancy services or web services built on Midgard LTS. Try and minimize the probability that something unexpected happens. That should provide us better chance to focus on relevant matters instead just trying to put out acute flames somewhere.