TDDing Web applications: requirments vs implementation details

Now that my Asp.Net testing framework release is getting closer, it's time to think how we could incorporate it into our TDD precess. Specifically, I'm interested in the Red-Green-Refactor pattern.

Suppose I'm trying to develop a site map with a tree-like structure. How do I test the output? In the world of domain logic, or even WinForms UI, the result is more or less well-defined by user requirements: we have to display a table of search results, for example. There's still "more or less", however: we could use a DataGridView, or a third-party grid, or use the form's OnPaint event and draw the results manually. This is what Presenters are for: the actual output is the implementation details, the Presenter is what we test for the search results. In most cases, binding the data to the UI is so trivial that we don't have to test it.

With the HTML output, it's different. Retrieving the data is one thing, but my current task is to display it. So, I reformulate my story: my control has to display a hierarchic data "properly". Here "properly" means proper indentation for items that are deeper in the tree. Purely visual concept, absolutely untestable.

So, I come with several choices. Usually I think of the Asp.Net controls as a high-level structure, and the raw HTML output as the implementation details. So, in other situations, I'd prefer to test the high-level structure. For example, I don't care much if my DataList renders as a table or a set of spans, but I want to make sure that the first Item's DataItem is what I expect it to be. The DataList is a part of a user story, and the HTML output is private implementation details.

With the Web UI development, it's different. Looks like any decision we make is upfront design. Anyway, there are two choices, basically. I can expect some nested DataLists, or Repeaters, or a TreeView, but now this becomes the implementation. Or I can expect some HTML. And the more I think about it, the more I prefer HTML. Now, HTML remains implementation if you think in visual terms. Should I make a bullet list, a set of DIVs, or a table? After all, even a "b" tag will behave like a list item if you apply an appropriate CSS style. But many tags carry a semantical meaning, and if we look this way, the answer is obvious: since we have a list of items, let it be a list tag!

There's also a "purist" approach: you define your own tag, TDD your HTML structure, and apply CSS wisely to satisfy your customer. But the point is, you choose the raw output for testing.

Now that we know what to test, we write a test, and the Red phase is finished. At the Green phase, we just override the Render method and write what we need to make the test pass. At the Refactor phase, we could notice that we use a foreach loop and refactor it into a Repeater, or we could use a server HTML control and build it programmatically, etc. Each way, the refactoring would work only if we haven't hardcoded the entire HTML string into our test. We started with a semantic meaning of the list, let's keep it this way. For example, we can store the output in an XML document, and ignore the node attributes and extra tags like spans and such. Also note that all these server controls appear not as a part of upfront design, but at the refactoring phase, which is what we really want for a TDD process.

Looking at the whole picture, we discover something we didn't expect: our implementation details became a part of a user story, and vise versa. To tell you the truth, I didn't expect the proven TDD principles working in a new field (who said they should?), but they did!

Encoding disaster in Asp.Net

Yesterday I created a few pages that contain Russian text and uploaded it to my hosting provider which is located in The States.

The Russian letters turned into Kryakozyabras (weird text with umlauts). Such things happen when you try to read Russian letters with ASCII encoding (or just cut off the 8th bit).

Further study revealed the following facts:
The problem is only with the static text; dynamically added text appears fine.
The problem is with User controls and regular pages; master pages work fine.
The problem happens with all foreign hosting servers.

What's been really weird is:
Viewing the file in a built-in viewer of my ftp client showed that the file is fine.
Response encoding as well as charset was UTF-8.
The hex editor showed me that the umlauts were actually 2-byte, but different from Russian 2-byte letters.

And the answer is..
.. Visual Studio was saving my files in the default system encoding, that is, Windows-1251. While it was fine for the runtime on my PC, it certainly wasn't for the remote server. So, with all that talking about internationalization, Visual Studio doesn't make its own files encoded properly by default. What's even worse, I couldn't even find a global setting to change that -- I had to resave each file manually.

Tafiti

A few weeks ago I discovered the amazing new search engine by Microsoft. It uses Silverlight of course, and it is sort of beautiful (they improved over their initial design that looked quite childish). The main fun begins when you choose the Tree view. You actually see a tree with search results hanging like fruits. The tree spins slowly, so that you can wait till the link you want moves closer and then pick it. If you are overwhelmed by the quantity of the results, you can move the slider at the bottom, and the less relevant results will disappear.

Visit http://www.tafiti.com/ and see yourself.

ItemCreated Strikes Back

Being an old member of the DragNDrop fan club, I'm happily using things like ObjectDataSource in my Asp.Net pages. So, just yesterday I was fighting with the same problem for the, like, 10th time, so I decided to blog about it and maybe next time I'll be wiser.

The ObjectDataSource works great after you've tuned it for your needs. It was a long story about how I have married it with Neo (my ORM tool of choice), but now that all the plumbing code is autogenerated, I can create a simple administrative page in several seconds, while still keeping my UI, model, and data access separate. (Yes I know that you can do it even faster with Rails) The problem arises when you try to do some data processing that doesn't fit well into your model, since it is more view-related.

Let me explain. I'm trying to make a page that would allow to edit the schedule for a fitness club. The page has, naturally, a table: columns are days of the week, rows are hours, each cell has the trainer name and the course title. I've got two ObjectDataSources -- one for the hours and one for the days of the week. And I have a repeater inside a repeater.

Now, you see that there is no way of doing this via just declarative databinding. Even if there are no records about a particular hour/day combination, there should be a table cell. So, I tried to grab the data using the Repeater's ItemDataBound event. The Item here is of type RepeaterItem, and it has a DataItem property that gives me the day of the week. Next, we have to travel up the control hierarchy and retrieve the parent repeater's item, to get the corresponding Hour object. Now we are ready to retrieve the data for the corresponding cell. Next, we have to manually find the corresponding controls inside the cell and fill the data.

Doesn't sound like it's the year of 2007, eh? It actually reminds me of funny tricks I used to do with classic ASP.

The real fun, however, begins when you try to update the values. The truth is, you can't use the ItemDataBound event since it isn't fired again after you clicked the Update button. This makes sense, since the data has been fetched already, so there's no need to fetch it again. The ItemCreated event is fired, however. But the DataItem property is null (as it was the first time before the postback). So, thinking again, you don't have access to your data items at all: they are not persisted, what you have is just property values saved in the ViewState. Thus, no chance to retrieve the data in order to update the correct record with the new values.

There are several workarounds here: you can refetch the data and keep it in a private variable, you can persist HourID in the ViewState and retrieve it on postback, you can even use hidden fields, but all of these are equally ugly. The point is, the DragNDrop thing just doesn't belong here, let's not force it to work. Back in the Asp.Net 1.1, I'd pick a data source manually and have complete control over it. This is exactly what I'm going to do now.

So, the (trivial) lesson is: mainstream solutions work only in mainstream situations. If you have a straight table, go the ObjectDataSource way. If the table is complicated and the data doesn't quite fit, go the manual data source way. If the table is weird -- displays running totals every 10th row, for example, it might be even OK to remember the foreach loop with tr's and td's. Let's just hope it won't get too weird.

Sadly, if I were using TDD for ASP.Net, I won't get into that trap.

VB Express Limited

As I said in one of my previous posts, I decided to develop FreshReports using VB Orcas. Funny thing, I don't feel I miss any of Visual Studio's functionality, but I miss the add-ins a lot. I can live without TestDriven.Net, but I've got to use Refactor! a lot (they still don't have native refactoring tools in VB). But another thing I miss sometimes is the "attach to process" button. Together with the missing "Debug test" feature of TestDriven it makes it almost impossible to debug your tests. Almost, because you still can use the self-testing feature of MbUnit.

Anyway, at some point I discovered that I'm writing too much Console.WriteLine's and still can't get my test pass. Now, I keep saying it on every corner that I hate all limitations, but honestly, sometimes limiting possibilities can do good for dumbheads like me. So, the reason I tried to debug what's been happening in my code was a smell that something's wrong, but if I could debug it, I just wouldn't pay much attention (old habits). The amount of console output was like some annoying noise that finally woke me up.

I woke up and I told myself, why am I torturing myself? I know a better way of doing this, so let's do it!

Still, I miss the add-ins..

Neo is alive!

Finally Neo, my one and only ORM tool I've been using for several years, is resurrected! The project, originally hosted by Codehaus here, has been abandoned for some time, and even the Generics update by Paul Gielens hasn't made it into the main branch.

What makes it different from other ORM tools?

1. Built-in code generation with templated classes.
2. The model classes themselves are not POCOs but contain a lot of DAL functionality (some consider this bad).
3. Possibility to keep all objects in-memory. Very useful for testability.
4. Built-in sql-like query syntax, for both database and in-memory queries.
5. All changes can be saved in one line.
6. Automatic inverse relationships. Spielberg.DirectedMovies.Add(AI) is the same as AI.Director = Spielberg.
7. Child contexts. Say, you have a parent Context that can save data to a database, you can have a child context that can save data to its parent. Very useful in some WinForms scenarios.
8. Probably a lot more.

Some problems include performance issues and a few bugs. But now that the project is resurrected, we can hope that it becomes even better.

The project can be found here, but there are no downloadables yet, just the svn repository link.

Interaction vs stateful tests, or Falling for Need-Driven Development

A few days ago I started adding paging functionality to FreshReports, and immediately fell into a trap some people call Need-Driven Development (see also this paper). Actually it's not a trap but rather one of thу possible strategies in TDD. However, I felt trapped for some time until just now -- I decided to rewrite my paging functionality using a more stateful approach.

So, how do you do it with NDD? First, you formulate the story, as always. Mine is, "Suppose we have two sections in our report, then if the second doesn't fit on a page, it goes to the next page". Next, I start to code the Page' s CalculateLayout method, something like this:

For Each section In sections

...

If section.Fits() Then

'do something about it

So, we want to code the CalculateLayout method and not get distracted by the Fits method, which is probably a lot to code, so we just mock it out and get a green on CalculateLayout. Next we switch to the Fits method which probably depends on something else, so we mock this something and proceed with the Fits method until it's perfect.

This technique seems fine from a traditional development's point of view. In fact, it is more or less design-first approach. But when I tried to move this way, after programming the TDD way for some time, I was feeling very uneasy. In the TDD fashion, I should have made a simple test that covers everything (section setup and printing), and mocked just the call I am sure of (to the actual element printing method) (note that I could have mocked the call to the section's Print method instead and save time on adding elements to the sections, but only if I'm sure that it works fine with paging). With NDD, I had to write tests for several steps, and since I don't see the whole picture from the start, I don't even know how soon I get to the actual printing. With TDD, I would have let my tests do the design for me, and even while writing the first test I would have discovered that I need a separate Pager class. With NDD, I sort of know the design of the current step before I start writing the test, and that's wrong, because, for example, later I discover that my Fits method needs two additional arguments, and changing its signature will break my tests.

Why have I fallen for it? Partly because I just read an article in which some obviously smart guys told me it's good. Partly because it's tempting. In my case, while writing the first test, I didn't know how to setup this "doesn't fit" idea, so I just mocked it. It was quite easy to write a test, even easier to write the production code, but after that I didn't feel I achieved something, quite the opposite, I felt that I've just made things more complicated. So you see, it's all about feeling good or bad, so it's quite personal.

So, is it that bad? Of course, not. NDD is great for more structured guys. I was feeling helpless because I didn't have a clear picture of what's happening, but that's just me. I prefer to do something quick in a very chaotic way, and then refactor it to make it more structured. NDD is for those who prefer to move step by step.

In fact, I'm using NDD when I encounter an external dependency, such as FTP or file system. I resist a temptation to just save a file, instead I code an interface and use it. I still have to remember to implement it later, but at least it's an end point rather than part of a long chain of objects. So, to avoid this temptation, I made a rule for myself: never mock an interface you haven't implemented yet, save for such thin implementations of external dependencies.