Date: Fri, 16 Aug 2002 01:20:23 +0000
From: Julien Gilli
To: James A Morrison
Subject: Re: DMS and distcc


On Thu, 15 Aug 2002 17:20:01 -0400 (EDT)
James A Morrison wrote:

> I'm curious how does the work you are doing compare to distcc [1].

Don't be afraid by the length of the text, if you want a short answer, skip the first paragraphs :-).

Well, I will probably say a lot of things that _probably_ are, or _probably_ will be (because the project has just started), and since I don't speak english fluently, you might misunderstand me sometimes, so don't give too much sense to my words, and feel free to ask if something is not clear.

This project is currently a school project. We (three friends and me) are studying in a computer science school, and we are free, this time, to choose the project we would want to work on.

A project where you can do something interesting in 1 month and a half, that is still challenging and that let a lot more challenges opened after is not easy to find. We found that a software that could distribute jobs within a network would best suit our needs. Some time before our decision, we went to a compagny where people were using TeamBuilder. We really enjoyed the design of this program, and we thought that this could be really fun and usefull to write. Just after we decided to write such a software, we began searching for similar sofwtare and we found that actually a lot of people have worked on such tools. Teambuilder and distcc were the most interesting for us, because the others were patched version of *Make, or used middlewares such as PVM or MOSIX.

So the first millestone of our project will be very similar to TeamBuilder or distcc. We want to be able to distribute compilation jobs within a LAN, in order to decrease the compilation time. We want to support at least the C and C++ language, and the GCC and ICC compilers collection. The first millestone should also include a graphical monitoring tool, like the one which is provided by teambuilder.

One this millestone is rock solid, we want concentrante on several topics that will lead us (hopefully) to support more compilers, more programming languages, to deal with security problems, to increase the performance and scalability (distribute jobs across the internet), support building binaries for other architectures and more. However, we want to concentre first and mostly on the first millestone objectives, the objectives mentionned just above are just thoughs.

Currently, we manage to compile huge projects (emacs, perl, python, etc.) in a client/server fashion. We just need a basic scheduler to be able to distribute the jobs on several machines. Once the basic scheduler written (probably tomorrow night), we will test it and compare our results with the ones of distcc and teambuilder. If they are good enough, then we'll know that we are on the right way, and will begin to add features (C++ and ICC support, enhanced scheduler, stability, better overall source code, GUI and support for more OSes, probably linux, OpenBSD, FreeBSD first, and then IRIX, MacOS X and Solaris).

On the other hand, we don't want to duplicate work, and it is possible that when our project won't be a school project anymore (by the end of September), we will ask the author of distcc if he agree to merge our projects. Indeed, by that time, the goals of the two projects may be quite similar, and maybe we wil find that we can better work together.

The major problem that could prevent us to work together is that we use to very different programming languages. We use Python and distcc use plain c. Although Python has a good C API, it would be quite difficult to reuse code from the both projects, and merge it into a new one.

Well, I realize that I wrote all this text without answering your question. So how does our work compare to distcc ? Currently, it is difficult to say, because we've just started. But I think that the Python programming language will help us to add features really fast, and to concentrate only on what matters : scalability, portability, compilers and programming languages supports, etc. And we can see it everyday. The code is easylly readable and we can add features _really fast_. So we hope that some contributors will join us by the end of September.

I hope that the fact that Python is interpreted won't be a too big issue, since we follow the same design than distcc and TeamBuilder : each compilation command line invoke one of the client wich will ask a deamon to do the job. It means that if you distribute the job on fivteen computers, the client that called make will have fivteen Python interpreters running at the same time at most, which could lead to bad performance. However, we alredy thought about workarounds to this eventual problem.

We'll make a snapshot of the CVS when we'll manage to build, in a distributed way, a sufficient number of programs, so that you can check what's going on. it should be done before the end of the week-end. We'll also post news to tell about the evolution of the project, and if you want, I can keep you informed of important millestones.

I hope I didn't bothered you, and that I answered your question. Feel free to ask questions and to bring ideas. The only thing I really can't do is to accept source code contributions from you. Since it is a school project, we'll accept source code contributions by the end of September.

Regards.

P.S : you can find TeamBuilder at :
http://www.trolltech.com/products/teambuilder/index.html

--
Julien Gilli