»File sharing«, »cycle stealing«, »virtual communities« and »collective intelligence« are all terms which are currently generating a great deal of interest. All these concepts involve the joint use (»sharing«) of available computer resources and human resources. How should the basic infrastructure be organised? Central and hierarchical or distributed with equal partners as »client/server« or »peer-to-peer«?

netzspannung: What does the term »distributed systems« mean to you, and what is the current state of research?

Bernd Freisleben: To answer that question, I'd like to draw on two much-cited definitions of distributed systems, the first of which is Leslie Lamport's definition: » A distributed system is one in which I don't get any work done because some computer which I've never heard of has crashed«. The other definition is Andrew Tanenbaum's: »A distributed system is a group of independent computers which appear to the user as a single computer«.

The first definition refers to the processing of different parts of a task by a number of independent computer systems linked in a network, while in the second example, these computers work together to perform a single task. Both definitions contain the requirement that the actual distribution aspect remains hidden from the users - the technical term for this is the »single system image«.

A large amount of the current research work is concerened with achieving this »single system image«. For example, in the area of network computing, people are pursuing the vision of being able to call up functionality and processing power from anywhere via the Internet or WWW (see for example » The Grid Forum«. In the field of mobile and ubiquitous computing, the aim is to integrate specialised terminal devices as transparently as possible into the Internet. This means »smart devices« - everyday appliances with built-in processors and sometimes also sensors, as well as »Internet information appliances« - mobile, cordless networked devices such as mobile phones, PDAs, cameras and »wearable computers«.

How did distributed system architectures develop historically? What technical requirements and social parameters shaped the development of different approaches, for example client-server and peer-to-peer?

The star-shaped structure introduced at the beginning of the 1960s, in which the terminals were attached to a central computer (mainframe), can be viewed as the first »distributed« system architecture. This set-up is an early form of client/server architecture in which the functionality provided by the server is used by several clients.

With the introduction of the ARPAnet in 1971 and the development of the TCP/IP protocol in the early 80s, the first communications services were developed. These included email, Telnet and FTP. The FTP (File Transfer Protocol) was used for the direct transfer of data between different computers without an intermediary server in the ARPAnet i.e. in the subsequent Internet. This protocol can therefore be seen as the original conceptual form of peer-to-peer technology.

At the bottom line, there is a major criterion to which technical and social aspects of the development of distributed system architectures can be traced back, and that's the shared use of resources of all kinds (sharing).

In today's Internet-based peer-to-peer architectures, the shared use of resources refers to disk storage space or files (file sharing), processing power (cycle stealing) and human work or knowledge (collective intelligence). Well-known examples include gnutella, freenet and mojonation for file sharing, popular power, united devices and moneybee for cycle stealing systems, and worldstreet and opencola for collective intelligence.

What tools are available for supporting the shared use of computer resources as a basis for collaborative information and knowledge acquisition?

For system developers there is a large choice of different platforms and frameworks for developing distributed applications which use computer resources on the Internet. Examples of academic developments include Programmer´s Playground and XtremWeb, while companies are offering products such as COSM and FRONTIER. There is a considerable number of software systems for setting up shared workspaces on the Internet. These have come about as a result of several years of research in the fields of collaborative work, CSCW and groupware (see e.g. www.usabilityfirst.com/groupware/cscw.txl). Examples include TEAMWAVE, originating from academic work, tools from the not-for-profit organisation THE CONTACT CONSORTIUM, as well as GROOVE, FRIZZBY and CONSILIENT, which are offered by commercial companies. I have already mentioned software tools which provide end-user-relevant functionalities such as file sharing, cycle stealing and collective intelligence.

How are these approaches implemented in technical terms?

The best-known file sharing tool is NAPSTER. Indeed, one might say that the general idea of file sharing on the Internet has been definitively shaped by the exchange of music in mp3 format facilitated by NAPSTER. NAPSTER is not a true peer-to- peer system, but is based on a central directory containing the data offered by registered users (users make available a portion of their hard disk for this purpose). By entering a search query, the user receives a list of other participants offering the data he is looking for. A direct exchange can then take place. Gnutella, Freenet or MojoNation, on the other hand, are true peer-to-peer file sharing systems which, like NAPSTER, were started by individuals or by a group of interested people and do not use a central server with a central directory. In this case each computer is both a client and a server. Search queries are made directly to another computer which relays the query further to other computers until one is found which can answer the query directly.

The willingness to cooperate and to offer hard disk space which is necessary for such systems to work has now led to software products such as SHARESNIFFER. Such systems carry out Internet-wide searches to find »open« computers and call up content without the owner's consent. This is made possible by insufficient safety precautions and allows the widespread reading and modification of files over the Internet, particularly on computers using Windows operating systems.

The basic idea of cycle stealing comes from research into parallel processing. The resource management systems for networked workstations and workstation clusters developed in the 1990s contained components for automatically carrying out jobs on computers which were not being used. Commercial cycle stealing systems such as Moneybee, Popular Power or United Devices and academic projects such as GLOBUS and LEGION (for overview see www.gridcomputing.com) are based on the idea of using redundant processing time or the storage capacity of other computers on the Internet. Demanding processing jobs can thus be distributed or performed in parallel by a large number of networked computers without affecting the participants. Ideally, »third party« processing only takes place when the screensaver is activated. This is sometimes referred to as screen saver computation.

The term »collective intelligence« encompasses a wide range of possibilities. On one hand, it includes all software tools which enable Internet-based cooperation and collaboration, as I have already mentioned. The basic idea is to create a »shared workspace«, although this is associated with numerous technical problems which cannot be discussed here.

On the other hand, there are software tools which are designed specifically for collective information generation and knowledge acquisition. With OpenCOLA, for instance, search agents can be sent to other computers and the relevance of the documents found can be assessed by the agent, which may then also consult the agents appointed by other users. In addition, users can assess the usefulness of a document themselves. This in turn influences the assessment of other »peers« and other users from whom the information came. In time, a process of intelligent collaborative filtering of documents bringing in the recommendations of expert users develops. This attempt to improve knowledge and make decisions through the qualitative use of valuable expert opinions is used by the financial industry, for example, through the Worldstreet system. I believe that we will see many more applications of collective intelligence, both in academic and commercial environments, since it is an area which offers both exciting research subjects and interesting financial market potential.

What are the fundamental features of distributed system architectures?

The design of distributed systems takes account of user-specific requirements, which have a considerable effect on communication and coordination within the system. However, there is also a function distribution element which can be realised independently of the planned application and is influenced primarily by the geographic and organisational distribution of the system. An example would be the localisation and storage of data which can be carried out in a central, non-central or replicated manner and depends mainly on technical requirements. User administration, on the other hand, is often determined by organisational specifications and can be set up non-centrally, for example using directory services.

Communication
Numerous standardised and proprietary protocols are available for direct communication between two computers on the Internet and can be selected according to various technical parameters such as throughput, available bandwidth, response time, portability and the possibility of forwarding (firewall). Their main differences lie in the way in which they encode messages and they are based on the TCP/IP protocol, without which nothing works on the Internet these days. Nevertheless, the choice of a standardised and portable protocol (e.g. CORBA or SOAP) is certainly helpful when setting up an infrastructure for a heterogeneous computer environment such as those found on the Internet. Besides direct communication indirect communication models which use a third location as an intermediary, are being implemented as well. This is employed when additional functionalities provided centrally and not offered by the communication partners need to be used. Examples include buffer storage of content (proxy), filter functionalities (search engines, translators) or message distributors (relay servers, mail servers).

Coordination
The coordination of the participating computers is a much more complex issue, which is subject to intensive discussion. Between the classic client/server models (WWW) with their emphasis on centrally-organised coordination and the much-cited peer-to-peer models with their completely decentralised coordination lies a whole host of hybrid solutions such as hierarchical (directory services) and domain-based (e-mail infrastructure) models.

With respect to the previously mentioned function distribution element within a distributed syste three main groups can be identified. Certain computers offer a particular resource such as processing capacity, storage space or documents. These content servers can notify directory servers of the resources they offer so that a central register of the available documents can be set up. A third group, discovery servers, function like a telephone exchange by identifying available computers and resources.

Consideration of these features reveals a whole range of combinations in these systems which differ according to their aims and uses. A typical example of a non- centrally organised system is Gnutella (mentioned above). Here, each computer is simultaneously a content, directory and discovery server. This achieves a high degree of decoupling and redundancy and makes control impossible. Access to resources (file sharing) takes place along direct routes without detours via a third location. This naturally has considerable legal consequences concerning possible illegal or non-paying use of intellectual property in digital products of all kinds which I do not intend to deal with here. One technical disadvantage of this strategy lies in the high bandwidth requirement, since a search for resources can in principle involve every computer in the system. ICQ uses a hybrid solution whereby one central point assumes the functions of both the directory and the discovery servers. Actual communication then takes place directly between the participating computers. The centrally organised system group usually contains applications which have to get by without making many demands on the client. Internet-based online forums, for example, are therefore allocated a central server via which indirect communication between clients takes place and which handles registration and switching between clients.

How would you classify the architecture of netzspannung.org?

When other existing solutions are considered, it becomes clear that all these systems are developed for a relatively limited scenario with defined functionalities. netzspannung.org, on the other hand, constitutes an open system which is designed as a basis for developing a wide variety of Internet applications. This system integrates various distributed system architecture approaches. On one hand, it implements a domain- based structure which has been on the Internet almost from the very beginning. A domain is here a group of computers which is set up and managed by one person or institution in a central organisation. From this point of view, any web site which consists of several Internet and database servers is a domain. On the other hand, the domains are linked using a decentralised approach in which they behave like computers in a peer-to-peer network. The interesting thing about this is that the whole structure can be scaled at will from a pure peer-to-peer network where each domain contains precisely one computer to a central client/server system with one domain and many clients. This offers users vast potential for setting up their own projects and applications, depending on which available system functions are being used and which are to be newly generated, or what type of management is required for the resulting resources and applications. An example of this is the netzspannung model for networking existing media archives. The necessary basic system functions such as switching between heterogeneous data structures, user administration, access control, annotation, categorisation, document searching or exchange of awareness information can be applied to all integrated archives and displayed in a joint user interface. This not only enables searches beyond the limits of individual online archives, but also provides the basis for experiments with new visualisation and communication concepts in networked knowledge environments.

Thank you for the interview.