So the main difference between agile and evolutionary prototyping is that in agile prototypes follow release cycle with a fixed length?

@Giorgio: as I explained in my answer, the differences are: (1) release cycle with a fixed length, (2) short iterations, (3) fully-functional product at the end of each iteration and (4) additional rules for specific Agile methodologies. IMO, there is no "main" difference among those four.

Since additional rules apply only to specific Agile methodologies, (4) does not characterize Agile. E.g., pair-programming does not characterize Agile. Furthermore, to me (1) seems to imply (3): if you release something it must be a fully-functional product, or is it possible to release a non-fully functional prototype in evolutionary prototyping? (2) is also important: iterations should be short whereas I guess in evolutionary prototyping there is freedom to choose long or short iterations. So, it seems to me that (1) and (2) characterize Agile (I had forgotten (2) in my previous comment).

But maybe I am wrong because I have interpreted (3) and (4) the wrong way.

@Giorgio: "if you release something it must be a fully-functional product" Not necessarily. When you release a prototype, it is not a fully-functional product, since some features actually available to end users may be completely broken. As for (4), I agree with you, it doesn't characterize Agile itself.

So if understand correctly, in Agile you only include in a release those features that are fully functional, whereas in evolutionary prototyping a release can contain some broken features.

@Giorgio: exactly. If it makes things clearer, in Agile, you may often use Continuous Deployment: it's similar to Continuous Integration, but the product is not only built and tested, but also pushed to production servers. On the other hand, you certainly don't want to push to production half-working prototypes you get in evolutionary prototyping.

Maybe I just interpret your diagram the wrong way: you have on the right "Release, All features, Final product". What are meant to be releases with broken / incomplete features? I would not call the previous, incomplete stages "releases".

What I mean by releases with broken / incomplete features are the ones marked as "Feedback" on the diagram. In other words, with evolutionary prototyping, you get the first prototype which contains only the core of the system, and give it for feedback to the stakeholders. Then you make the second prototype, which is the first one plus some features. Got the feedback - time to make the third prototype. Then feedback, then fourth one.

Each prototype is "released" to the stakeholders for the feedback, but it can't be a "real" release in the wild (i.e. to the end users/customers).

Example: you may give to stakeholders for their feedback a prototype of an e-commerce website where they can view different products, but once they press on "Purchase" button, the website crashes with HTTP 500.

Since it's a prototype, this is acceptable. You just tell the stakeholders: "You're fine browsing the products, but never ever press the Purchase button, or bad things would happen."

You can't do the same thing in a context of Continuous Deployment in Agile, because you can't tell the end users that they can browse the products, but shouldn't click on "Purchase".

Ok, so in Agile each prototype is (should be) also a potential release to the end user. I might sound a bit pedantic but there is a reason. I have been doing Agile for a few years in different projects now. Especially in the current project we always (at least try to) release a bug-free version at the end of each sprint.

On the other hand, we often have incomplete functionality in our releases: they can be shown to the stakeholders so they can provide feedback, but they are far from usable in practice. So maybe what we are doing it not really Agile.

Nothing prevents you from mixing Agile and incomplete functionality as far as the release at the end of the iteration doesn't contain the concerned functionality. Many companies (including Google) work this way. When they start implementing a feature which cannot possibly be shipped at the end of the iteration, they use flags which make it possible to activate and deactivate the incomplete feature.

The form of the flag can vary from project to project. For example, it can be as easy as a #define: if, as a developer, you want to see an incomplete feature on your local machine, you add the flag to the compiler to include the new code. Otherwise, the new code is not compiled. Of course, Continuous Integration server doesn't have the flag (or in more complex environments, it can create two versions: one with the flag, and one without).

In other projects, #define is not enough, so they go with a database option which enables or disables a feature for specific users or groups of users.

This is how Google can turn on an experimental feature for, say, users living in South America, but only to them. Or they may enable a feature for users who are interested in beta products. Once the experimental feature is tested on a small percentage of users, they turn it for everyone.

By the way, I imagine that you're using Scrum or similar methodology where each iteration takes one week or more. But when we talk about Continuous Integration, it may be 50 releases per day. In this context, you can't possibly develop any useful feature which is thoughtfully tested and works perfectly well for the next release.

Yes, we are using SCRUM and we have one-week iterations. We often build a very prototypical version of a feature that just does the job (e.g. an import function), and only later do we do a proper implementation. So e.g. we have worked with a primitive import function for months now.

It must be said that up to now our releases were only for internal use and the demos were only for our management. In a few months from now we are going to have a demo server that is going to be accessible by potential customers. We will not be releasing unusable functionality on the new server (at least I hope).

Releases for internal use sounds strange in a context of an Agile project.

The fact that you're going to have a demo server only in a few months after the project started is awkward too.

In general, iterations become regular from the beginning of the project (well, if a few of the very first iterations fail, it's OK, because it's usually very hard to start a new project and to find the ways to organize it correctly, as well as to have appropriate infrastructure you can rely on).

And of course, each iteration should be potentially releasable. This doesn't mean that you actually push it to production, but it means that if management decides that the last version should be in production immediately, your manager shouldn't come to a meeting with the boss, telling that there is nothing to deploy right now, because nothing works.

You touched two points.

1. Releases for internal use.

2. Potentially releasable.

2. We keep our code bug-free (we try at least, you can prove the presence but not the absence of bugs). We have lots of unit tests and at the end of each sprint we have a working system. We rarely have a non-working system for more than one day, also because we have lots of automatic tests that run each time we check something in.

One of the interesting criteria I can suggest is this one. Imagine tomorrow, your boss tells that the project is suspended for an unknown amount of time. Also, your team is shifted to a different project, which appears much more important for the company right now. Since you have been working for five months on the seven-months project, your boss expects that it's nearly finished, and despite the lack of some important features, he asks you to ship the product as-is. Can you?

1. The project started from scratch in an area in which there are other existing products built with different technologies. So during the first few months we only had to build every from scratch and our product would not have been considered for any real use. We had regular demos with potential stakeholders (once a month or so) but we had the feedback of our management and project coordinator during our weekly sprint reviews.

Now we are reaching a stage in which we have enough features that are mature enough to be considered by potential users. Showing them weekly demos, say, six months ago would have been useless.

Because many of the basic features they would have expected were simply missing.

So, if you ask if we could release now. I guess yes. Some important features are missing, but what is there is usable.

@Giorgio This is really great. Having a well-tested system which works at the end of each Sprint is a must have.

We have a build server: if we commit and a test is red we get an email within minutes.

@Giorgio "So during the first few months we only had to build every from scratch and our product would not have been considered for any real use." I've seen many companies having the same approach and I never found it valid in their cases. Every time they wanted "too much", forgetting that there could be a more basic product. For example, imagine you're doing an e-commerce website from scratch. Being able to purchase products by paying online is mandatory, right? Not really...

Regarding the questions you posed above. We have an internal demo server with the current version of the application. I think if our managers wanted to go live with that, we should just install it on another server accessible from outside. It would not take much. One question mark is we only have automatic tests. I would like to have an extensive manual test before releasing.

... Not really. Some e-commerce websites don't have the ability to pay online (if I remember well, MIT has a site where they sell books, and in order to pay, you have to do it "by hand" after they send you (manually) an invoice by e-mail).

Well, our company has been working in this area for several years (> 20). There is a minimum set of functions that a user expects otherwise they won't even look at your product.

Up to now most products were written in C++ as desktop applications. Now we are building a web-based application.

@Giorgio "So, if you ask if we could release now. I guess yes. Some important features are missing, but what is there is usable.": well, then great. The start of the project may be improved, but now, you're doing it in an Agile way.

I understand your example, but there are really some lower limits.

Well, maybe we were not agile at the beginning, but then I would argue that agile is not always applicable, it is not a silver bullet after all.

@Giorgio "One question mark is we only have automatic tests. I would like to have an extensive manual test before releasing": or have a suite of automated tests which test the interface as well. It's not a good thing to have manual tests in any project.

On the other hand, from the very beginning we had weekly feedback from our colleagues, many of which have worked in the area for more than 20 years and can provide qualified feedback.

@Giorgio "Well, our company has been working in this area for several years (> 20). There is a minimum set of functions that a user expects otherwise they won't even look at your product": wait, there is a difference between a piece of software you can release and a piece of software that marketing department will consider as actually useful for the end users. This is the gap between technical aspects of the project and business decisions.

We also have automated GUI tests. But certain things are much easier and less time consuming to test by using manual tests. We have tester teams (but not in our current project) and they can usually find bugs that automated tests can hardly find. I think the best is a combination of the two.

Sometimes, technically, you can release. Business-wise, you can't, because your boss told that buttons should be painted in pink, otherwise, it's a deal breaker.

@Giorgio "but then I would argue that agile is not always applicable, it is not a silver bullet after all." You're totally right.

OK, then technically we could release from week two or three. But nobody would've wanted to look at it, let alone buy it.

If you mean release from a technical point of view, we could release from the very beginning

From a business point of view, maybe a few months from now

@Giorgio "On the other hand, from the very beginning we had weekly feedback from our colleagues, many of which have worked in the area for more than 20 years and can provide qualified feedback.": I know: in fact, you started with evolutionary prototyping, and the project evolved into Agile. This makes it a perfect example for my answer on Programmers.SE. ;)

@Giorgio "But certain things are much easier and less time consuming to test by using manual tests.": the fact is that every (nearly every?) manual test can be automated. Automating the test can be time-consuming, but manual tests don't stay for long: when you start pushing several dozens of versions every day, testers just can't follow.

You may also be interested in Automation section in my answer related to testing: programmers.stackexchange.com/a/253548/6605

"when you start pushing several dozens of versions every day": Maybe you shouldn't do it.

Those are your commits.

So you should.

I do not commit several dozens of versions every day.

You, alone, maybe not. But what about your entire team?

The three of us. Maybe we get to 10 commits a day. Unless we have to fix some small things, then we could have more.

We try to commit when the code is in a relatively clean state. And we do code reviews and / or pair programming.

OK. Well, there are different modes of thoughts on that. Some developers try to avoid committing too often. Others don't hesitate to commit at every logical change.

So we may need to correct things, but that does not happen too often. Often we commit almost finished code, and we do an extra commit to fix a few things.

I find it a bit difficult to follow the changes when there are too many commits. However, one of my colleagues tends to commit more often than I do.

In any case, I would prefer to see only relevant changes in the svn log, not all kinds of intermediate steps that led my colleagues to the solution.

I see.

It's also important to know whether you run tests on your local machine or not. For example, the project I'm working on right now has tests which would take more than twenty minutes to run on my machine. When I commit, tests are distributed across servers and I have the results within two minutes. This, indeed, encourages to commit a lot.

If you can run tests locally, then you don't need as much commits.

Our tests take about half a minute to run. We run them locally each time we commit, before committing.

OK.

I understand

And remember where our discussion about tests started: that if you have manual tests, testers cannot follow you.

This depends on how you develop. Testers should test a finished feature. I do not see why they should waste their time on half-finished stuff.

In my previous project, we had a daily build.

Each morning the testers installed the latest build and started testing.

Sometimes they found bugs faster than we could fix them.

Imagine I'm doing some intensive refactoring. After each refactoring (and sometimes during the refactoring), I run automated tests. Three hours later, everything looks fine so I commit. I also spend the next hour fixing some bug (second commit) and the next four hours implementing a new feature (third commit). If manual testing is done the next day and detects a regression, I have no idea whether the regression is caused by the refactoring, the bug fixing or the new feature.

When bugs accumulated we stopped development and did a bug fixing sprint or two.

I'll ask testers to see if they can reproduce the regression with the revision n-1 and n-2.

Yes, but how long will you need to implement the manual tests as automatic tests?

If they found that the regression appeared during the refactoring, how much time would I spend searching for the source? (Given that the diff will be terrible)

GUI tests are extremely time consuming

And some times human testers find things you would have never expected.

@Giorgio "This depends on how you develop. Testers should test a finished feature. I do not see why they should waste their time on half-finished stuff.": as you see, later in my example, I'm talking about regressions, so they are actually testing finished features which were later broken.

I understand you would like to find a bug as soon as it appears.

So you want to have automatic tests for that.

@Giorgio "Yes, but how long will you need to implement the manual tests as automatic tests?": it depends. There are some which are very tricky to implement, indeed. But most are easy to automate, especially with the help of tools such as Selenium.

The point is whether this is feasible. Whether it takes less time to perform a manual test or to implement (and maintain) automatic tests.

Look we have an application that displays images. How to test if the correct image has been displayed with the correct annotations on it? Automatic tests should be able to analyze the image and recognize text on it. A human tester will do it instantaneously

"And some times human testers find things you would have never expected.": This is experimentation, and this is why you absolutely need testers even when you have automated 100% of your tests. You need human testers to discover new possibilities for failure, and once they find those ones, they automate it.

I did it once

I had a very difficult bug and I implement a quite complicated integration test.

@Giorgio "How to test if the correct image has been displayed with the correct annotations on it?": What about pdiff (pdiff.sourceforge.net) ?

No

You do not want to compare if two images are the same

Images are rendered dynamically

rotated, moved, and so on.

I mean compare the actual image with the one which is expected.

And some text is rendered on top of them.

The text is different every time.

You mean random?

And you want to know if the text is correct (a name).

Maybe you can do something but it is very tricky whereas a tester can just open the application and verify that the correct text is rendered.

And as soon as you change version the text might change.

So you have to change your image used for the automatic tests.

I'm not sure I understand, so I'll give an example. You develop a chart rendering application which draws a chart based on some computation from the data from the database: if data changes or the user switches the computation, the image changes as well. In order to test that, you create several reference images. Then you render one using your newly developed library, and pdiff the rendered ones comparing them to the reference images.

Given that you mock/stub the database and the computation, so you're sure that you need to get the exact chart that you have as a reference.

With the exact text and numbers and the exact curve.

Well, in our case it is 2d rendering of 3d images. I am not sure if it is feasible. Maybe it is.

May I ask you where you are from? I live in Germany.

I'm from France.

It's been very interesting to chat. I have to go now. Maybe catch up with you some other time.

Thanks for your time.

It was interesting for me as well, so thank you.

And you speak English very well. ;)

Have a nice evening.