The Architecture of Lotus Notes

Figure 1 -- Lotus Notes architecture components

Figure 2 -- Lotus Notes' three models for communication: one-to-one messaging, sharing a common database, and distributed documents in replicated databases throughout the world

Lotus Notes technology is a departure from traditional database or messaging products. Traditional database systems are the backbones of most corporate information systems. They are designed to manage the tabular information generated by business operations, such as order processing, inventory control, and payroll management. These applications are data-centric, organizing information by breaking it into basic elements. Knowledge and information are gained only by sorting and querying these data elements in various ways. They are transaction-oriented, built to reflect the most current state of the data. They are generally not good at reflecting the changing states of information over time. If they are distributed, they require a single-system image so that if a number is debited in one place, it’s debited everywhere in a single transaction. In this sense they can’t easily handle access by disconnected users.

Traditional electronic mail and messaging systems, on the other hand, are designed for efficient transmission of messages from one place to another. They can handle simple and complex information, and they can deliver it to specific individuals or applications. However, they generally have no facility for capturing or tracking the information; they simply provide reliable delivery.

Most organizations need robust and scalable strategies for both databases and messaging, to manage and disseminate knowledge. This is the mission of Lotus Notes.

Because it is unique, fully exploiting Notes requires a new perspective. This article explains the structure of the Notes programs on the client and server and the structure of the Notes database (figure 1). This article bridges the gap between general architectural descriptions and detailed information which can only be found in the API documentation.

For groups of people to work together, it’s important that technology doesn’t obstruct communication. Therefore, Notes software is provided for a wide variety of operating systems, and it’s drivers support all the major network protocols. The result is an any-to-any connection matrix, allowing freedom of choice. Users can communicate anytime, anywhere, irrespective of their particular working environments.

Descriptions in this article relate to Lotus Notes version 3. However, the basic structures have been in place since version 1. layout, Notes still preserves backward compatibility. Future versions will continue to add features without changing the essential design. Documentation of the C data structures is in the Notes API Toolkit and isn’t duplicated here.

Client-server architecture

Lotus Notes is a true client-server system. The Notes server, running on Windows/NT, OS/2, NetWare NLM, Solaris, SCO, IBM AIX, or HP/UX, does more than simply take queries from database users and return result sets. It maintains application-defined indexes, stores and forwards electronic mail, replicates databases with other servers or workstations, performs agent tasks, and is extensible to accommodate user-created agents or third-party add-ins. The Notes client, running on Windows, OS/2, Macintosh, Solaris, SCO, IBM AIX, or HP/UX also contains most of the server functionality. This lets users replicate databases from a server and take those files away from the network; send and receive e-mail; create, modify and index documents locally; and synchronize replica databases over a LAN or WAN connection

In addition to giving you choice, client-server is inherently more secure and efficient than simple file-sharing. Access to the data is controlled by the server at a fine level of granularity, resulting in smaller amounts of data being passed between the user's workstation and the remote server.

Security

From the outset, Lotus Notes was developed as a secure system. Done in a manner that works across vendors’ operating environments, Notes utilizes technology licensed from RSA Data Security, Inc. of Redwood City, California, the industry leader in cryptographic algorithms and services. The RSA public key system enables user authentication, certification, and message signature. The RC2 and RC4 secret key systems enable encryption of documents with named keys, and allows for the creation of secret keys for secure channels of communication over the network.

Notes provides four classes of security:

Authentication : Identifies users using the industry-standard X.500 hierarchical naming syntax. Authentication in Notes is bi-directional. Servers authenticate the identity of users and users authenticate the identity of servers. Authentication is used whenever a user and a server or two servers are communicating with each other.
Digital Signatures : Guarantee that a given message is from whom it says it’s from, essentially a user-to-user form of authentication. In addition, this technology enables the computer to notarize all, or a portion, of a message and guarantees that the message has neither been forged nor altered in transit.
Access Control : Allows or denies access to shared databases, documents, views, forms, and fields. Server access can also be controlled for individual users by either allowing or denying access to specific Notes servers within the organization.
Encryption : Involves ciphering or scrambling information so that even if accessed by the wrong individuals, it can’t be understood. Encryption is available at three levels in the system. At the message level, individual messages can be encrypted for one or more intended recipients. At the network level, encryption prevents someone from promiscuously sniffing (tapping into) traffic on a LAN or dial-in line, because they can’t see anything intelligible. At the field level, databases can be designed to encrypt document fields so that only specified users can read them.

There are no back-doors into the system. Neither Lotus nor any hacker, foreign or domestic government agency, nor competitor has or will have a super-user capability in your information system.

Electronic messaging

Lotus Notes includes an e-mail system which uses its native objects, Notes documents, and provides store-and-forward. The Notes server network intelligently provides best-path routing analysis and fault tolerance. The Notes kernel and mail transfer agent (MTA, also referred to as ROUTER) implement features, such as blind copies, delivery confirmation, return receipts, encrypted messaging, and sender authentication. Electronic mail is tightly integrated into Notes applications.

For those who wish to retain an existing e-mail system, the Notes workstation software lets users send mail to any VIM-compliant system and (with the forthcoming VIM-to-MAPI converter) to any MAPI-compliant e-mail transport. The coexistence of Notes with alternate mail systems is transparent to the users. To provide back-end integration between different mail systems, Lotus and its business partners have developed a large number of mail gateways, connecting to all the popular LAN- and host-based e-mail products.

E-mail standards are very important to this. The coming Lotus Communications Server (LCS, based on the Notes 4.0 server) provides native X.400 and SMTP/MIME MTAs in addition to the existing Notes and cc:Mail transport protocols. LCS will also provide three standard mail interfaces for programmers: VIM, CMC, and MAPI SPI. This integrated architecture gives a robust and flexible solution to enterprise mail requirements.

Database replication

Store-and-forward e-mail is necessary but insufficient for many applications. When used as a method of database synchronization, a unidirectional message route is not robust. There needs to be a connection-oriented, bi-directional, method of document sharing.

Lotus Notes’ unique replication facility means that databases on different servers are automatically synchronized and change conflicts are resolved. This scales easily from two to thousands of replicas of a database. Replication does not require administrators to configure the individual database connections. Nor does it require a particular server-connection topology. When any two copies of a database are replicated, each exchanges the latest modifications with the other, guaranteeing that they both finish in synchronization. If the network link breaks during this process, there’s no need for commit and rollback types of protection, since replication will continue when any link between replica databases on different servers is re-established.

The combination of true replication with the Notes object store, where documents and application design elements are self-contained note objects, releases workgroups from constraints of time and geography (figure 2).

Notes servers can be configured to communicate with one another in peer-to-peer, hub-and-spoke, and ad hoc topology. Due to the flexible nature of Notes, topology planning can be independent of application or information strategy planning, thus simplifying deployment within a large organization.

Also, because of the peer nature of the Notes architecture, interconnection of servers to other servers or users to servers is treated identically. This is why it’s so easy and natural for mobile users to use a modem to connect their computers into the system. Notes is designed for occasionally-connected communications, whether the entity communicating is a user or another server.

Lotus Development Corp. is committed to making a Notes network manageable. Version 3 includes remote console administration and actively generates alerts that can be sent to network administrators through e-mail. Alerts indicate such things as low disk space, failed network connections, and communication bottlenecks. In early 1995, Lotus will provide notesView, an SNMP-based application to manage all aspects of the messaging system. Lotus messaging management facilities will support industry standard protocols such as SNMP, allowing the Lotus Communications Server to be managed by other vendors’ management systems.

Notes kernel services

The core modules of Notes are shared between client and server code. This kernel is exposed as the Notes C-language API. One function of the kernel is to insulate its clients from the underlying hardware and operating system. This approach is necessary in any architecture that must span existing and future operating-system choices. Insulation of the groupware application developer from the platform’s quirks is the only way to create a truly open system, where choice of software is not constrained by a particular hardware or OS vendor.

Application developers, whether working at the API level or in the Notes user interface, gain three key isolation layers: from the operating system, from the network transport, and from physical location of the object store (via a transparent form of remote procedure call).

Other kernel functions include security, database access, document structures, indexes, and full-text search capabilities.

All these core modules are available to higher layers of Notes, including API programs, in the client and in the server environments, on all the platforms supported by Notes. For example, Notes API code can be written to be completely portable between Windows, OS/2, various flavors of UNIX, and NetWare NLM. Figure 3 shows many of the components.

Figure 3 -- The Notes kernel, the major constituents of the Notes workstation and server, and the integration options available to API developers.

Important modules in the Notes kernel are:

OS : Operating system isolation layer. Provides platform-independent access to memory, shared resources, semaphores, environment information, and so on. Beneath the isolation layer are highly optimized implementations of these services for each individual operating system.
SEC : Security module. Gives access to user information, certificates, and encryption keys, based on the BSAFE security package from RSA Security, Inc.
NSF : Notes storage facility. Manages the Notes database, allowing its users to create, open, and delete databases; create, open, and delete documents; and store and retrieve information. Part of this function includes on-disk structure (ODS) management, which ensures portability of the .NSF file format across platforms and on the network. All the published NSF interfaces are independent of the database location; a remote-procedure call (RPC) system transparently redirects requests to the local disk or to the appropriate server. This RPC layer is also used by other kernel modules, such as NIF (the indexer) to retrieve a pre-built index from the appropriate server.
NET : Network transport layer. Provides a single interface to drivers for many networking protocols. It can initiate and receive phone calls or create LAN-protocol sessions for communication over ports defined by connection documents in the Name & Address Book.
COMPUTE : Performs calculations, using the Notes formula language. Notes API programs can ask COMPUTE to create and evaluate formulas and also to implement custom @ functions which are called by the COMPUTE module.
NIF : Notes index facility. Manages indexing of Notes documents into views. Views define the selection of particular documents, and columns containing information from those documents (or calculated values based on the documents). Columns can be sorted, or categorized (where similar values appear under one heading), and can use the hierarchy of main and response documents in a database. These collections of documents are indexed in a B-tree structure for presentation to the user as a Notes view. NIF is responsible for maintaining and using the indexes, adding and removing information as documents are modified.
FT : Full text index facility. Provides content-based retrieval and weighting with Boolean-logic search through the full text of any document.
NAME : User directory service. Not strictly part of the kernel code, but has some direct APIs which give access to the Name & Address Book. This is a special Notes database which contains user names and e-mail addresses. User identification uses a X.500-based naming model and conforms to X.509 certificate and authentication standards.

Workstation modules

The Notes client workstation software includes the Notes kernel and four major additional modules:

DESK : Notes desktop manager. Handles the layout of icons and the desktop, storage of user preferences, and general coordination of the Notes user interface.
EDIT : Notes document and forms editor. Also includes, creates, and renders compound documents.
VIEW : Notes view manager. Manages the user’s interaction with indexes of the database.
NEM : Graphical-environment manager. Provides high-level abstractions of the graphical user interfaces present on various platforms.

Server modules

The Notes server consists of one core server program. It manages the server’s other processes and threads, and users’ connections with the server. The server functionality is largely implemented with modules. Some are necessary; others are optional or developed by third parties.
Important modules include

replicator : schedules and connects to other servers and workstations to replicate databases
indexer : keeps indexes up-to-date for immediate access by workstations
router : directs mail between mailboxes and between servers
chronos : schedules agents to perform background tasks in Notes databases at the application designer’s request.

This server platform gives powerful, extensible intelligence. It can automatically connect to other servers to exchange mail and replicate databases. It uses the host operating system to exploit its particular features, such as SMP multiprocessor systems to spread the workload. It lets the clients be mobile, so a workstation can use local Notes databases when not connected to a network, and can connect to any server by LAN or dial up for direct server access or to replicate and exchange mail.

Client-server communication

The transport between client and server is implemented in a number of drivers under a common network layer. Selection of the appropriate driver for any named port is controlled by the user. Network port selection is also user-definable if the machine runs multiple network protocols and has a preferred hunting order. Selection of the appropriate serial port for a WAN connection between any two machines is controlled through connection records in the Name & Address Book.

When a connection is established, the client's identity must be verified. The server creates a random number and encrypts it using the alleged client's public key, then asks the client to decrypt. This decryption can only be performed using the client's private key; if the resulting value is returned to the server correctly, the user is authenticated. Then a secret key can be created for secure channel of communication between client and server. The server starts a thread to process this client's requests.

The client-server request protocol lets the client ask the server to perform various bulk actions—create a full text index, open or compact a database, categorize or search a number of documents. This is the first level of delegation of processing by the client.

The second level, a form of remote procedure call, occurs inside the kernel modules. For instance, if the workstation calls NIFOpenCollection to open an index, the internal RPC layer transfers this request to the server, which updates the index if it is out of date. And when retrieving a view, only the necessary portions are transmitted for display on screen. Due to the RPC layer, these processes are independent of whether the database is on the local hard disk or on a server, and of whether the server is connected by LAN or over a telephone line.

Notes database

Users compose, read, manipulate, forward, and interpret Notes documents. At the user interface, there are two main tools for document access.

Forms give structure to documents and provide the editing environment.
Views collate, categorize, and report on many documents in a database.

Other tools are available. Macros can run on the workstation and the server, on individual documents and on batches of documents. The full-text-retrieval engine is also an important tool.

Different users have differing perceptions of how this all works. The parable of the elephant in a darkened room seems appropriate. It is sometimes difficult for software professionals to "place" Notes in relation to other development tools, document management and workflow applications, relational databases, and legacy messaging systems.

User model

Here’s a user view of Notes, using terms from object-oriented design:

The user interacts with objects in the Notes system. The data of these object instances is held in Notes documents, in fields of various data types. The methods with which they manipulate the data are defined in Forms. Views are browsable collections of summary information from many object instances, allowing the user to search, sort, and relate the information in a database.
Notes’ formula language provides ways to manipulate the object data. These methods are augmented by OLE, and Datalens and ODBC drivers.
There are two ways to relate the document "instances" and the form "methods". First, Notes stores in each document the name of the form used to create or modify the document; this is a loose linkage. Second, an alternative tight linkage can be established by storing the form in the document. This is appropriate where a document and its form (an object instance and its methods) must be sent together through e-mail. The loose linkage is natural in most Notes applications, where each database contains both data (documents) and the associated design elements (forms and views).

This is a valid description of Notes as a document-object database. But internally, there’s really little distinction between data documents, form designs, view definitions and other design elements. They are just different Notes Classes. This abstraction means that replicated databases not only distribute the object data, but also the object methods, using one consistent infrastructure. Distribution of applications in this manner is one of Notes’ key strengths.

The simplified diagram in figure 4 illustrates how users interact with these documents, forms and views; and in the database, how the data and design elements are just different-colored instances of a more abstract Note.

Figure 4 -- Notes database, simplified, showing how users interact with major components.

Structure of the note

A note, whether a document, a form definition, or whatever, has a consistent and flexible structure. The reserved structures which are used to create the design notes are described in more detail later. A Note always consists of:

NOTEID : Identifies this Note in this particular database.
ITEMS : One or more fields containing data. In the Notes API, and this article, the fields in a Notes document are referred to as Items; the term field is reserved for describing field definitions on a Notes form.
ORIGINATORID : Identifies this Note in a globally unique way, so the same document can be identified in any replica of the database, and identifies the last modification, so replication conflicts due to simultaneous edits can be resolved.

The ORIGINATORID deserves more explanation. It is the structure that identifies all replicas of the same note. Here are its members and what they contain:

File : Unique random number, generated at the time the note is created.
Note : Date/time when the first copy of the note was stored into the first .NSF. (The structure to this point is also referred to as the UNID).
Sequence -- Sequence number used to keep track of the most recent version of the note for replicated data purposes.
SequenceTime : Sequence number qualifier that allows the replicator to determine which note is later, given identical Sequence’s. Both are required. The sequence number is needed to prevent someone from locking out future edits by setting the time/date to the future. The sequence time qualifies the sequence number for two reasons: It prevents two concurrent updates from looking like no update at all. And it forces all systems to reach the same decision as to which update is the latest version. These are the key decisions on which replication is based.

Every note in the database can have a different number of items, of different data types. Contrast this self-contained flexible document format to traditional relational database systems, in which the structure of a table is defined for all records. Relationships between documents are provided by some internal structures, such as the relationship between main documents and responses, and by any application-defined structures, such as lookups into Notes views or external databases, to give referential integrity.

The NOTEID is a compact and efficient identifier for a Note. It provides a record relocation vector (RRV), essentially a file position pointer to this note in this database. When you need to store lists of document IDs or to identify collections of documents or lists of unread documents, there’s a structure called IDTABLE, which is a compressed list of NOTEIDs.

Neither the ORIGINATORID nor its subset the UNID is not a file pointer, but is rather a unique identifier constructed from random numbers and timestamps. An internal index in the database gives fast access to a document by knowing its UNID (for example, so that the replicator can identify documents in common between database replicas). In relational database terms, the UNID is the primary key. To the user, the Views provide the important document indexes, each having application-defined keys.

A Notes database contains documents and design elements (Notes) which each contain and define their own structure. This structure can include an enormous variety of data from different sources. Certain internal structures build on this open object store. Notes applications provide the user with tools for more- or less-structured interaction.

Summary and non-summary items

Some data items contain simple data types and are used to contain summary information about documents: author name, creation date, title, e-mail recipients, and so on. Others contain compound information: rich text, embedded objects, graphics, and the like. This distinction is important in any object management system that has to index and sort documents for user accessibility. In Notes, this distinction is made by flagging items as summary or non-summary.

Summary items are available for computation in Notes formulas, for lookup into or from external data sources, and for indexing and presentation in views. Non-summary items cannot be evaluated in formula, nor can they be used for collation. They provide the storage for compound objects.

Object linking and embedding (OLE) is a powerful method of packaging and referring to documents from a server application in such a way that they can be embedded or linked in other client applications. For example, a Freelance presentation or a Paintbrush graphic can be embedded in a wordprocessing document. The embedded object can be edited by clicking on the object, launching the OLE server it is associated with. Various verbs implement the different modes in which an OLE object can be manipulated; for a sound or video object, verbs might be Play and Edit.

This packaging is important because it provides a way for applications to include data types they don’t directly handle. The client software need know nothing about the embedded object except its server name. The Registration Database in Windows gives more information about the server program, such as its location on disk and its verbs. OLE objects are opaque to the container application. A product such as Notes, designed to be a flexible object container, can store many OLE objects in many documents, but this gives little benefit to users unless they know something about the documents. It is not sufficient simply to know "This is a spreadsheet".

For this reason, Lotus developed the Notes/FX protocol. This enables bi-directional field exchange of summary information between the server and client. Using Notes/FX, a Notes application can store OLE objects and can also find important information about their contents, such as the number of pages, the values of spreadsheet ranges, and so on. In the Notes database, this summary information can be indexed and searched by users. So, a database of expense-report spreadsheets can be summarized and totaled by Notes. Field values from Notes documents can be placed into the spreadsheet, word processor, or server. For example, this can avoid duplicate storage or typing of addresses in a customer-service application. This gives true value to embedded objects. Microsoft has recently adopted Notes/FX in their product suite.

Documents

Document structures are usually defined by the forms with which they are edited. Additionally, there are important items for system use. These items include:

Form : Contains the name of the form used to create or edit this document. This may be overridden by specifying a form formula in a view, which can define the form used to display documents according to context. Additionally, the form definition can be stored as system items in the document itself.
$REF : Contains a reference to the document’s parent, if it is a response document. This item creates the response hierarchy, used in threaded discussions and to link (for example) correspondence documents with a customer-details document. The indexer uses this item to reflect the response hierarchy in Views.
$Header and $Footer : Contain print headers and footers for this document.
$File : Contains summary information about attached files and OLE objects, and pointers to their object RRVs.
$Readers : Contains the names of people authorized to read the particular document. This is in addition to other security provided by the ACL and encryption.
Other reserved item names are documented in the C API header file STDNAMES.H.

Electronic mail

E-mail is simply Notes documents which include particular reserved items and a flag which indicates to mail this document immediately. The reserved item names include:

SendTo : The name(s) of the primary recipients of this mail.
CopyTo : The name(s) of the secondary recipients of this mail.
BlindCopyTo : The name(s) of people who are to be sent blind carbon copies of the mail.
DeliveryPriority : Contains High, Medium or Low as appropriate.

To send a mail, the document is transferred by Notes into a special database named MAIL.BOX on the server. The mail router agent picks up the document, determines routing, and forwards the mail either to another server or to the recipient’s mail database as defined in the user information.

Form definitions

Form definition notes define the layout on a form on screen, including fields (through which the document items are displayed), formulas for field default values, input translation and validation, and form security. Form items include:

$Body : Most of the form definition in a rich text item.
$Title : The title of the form.
$WindowTitle : A formula which defines the title of the form. The title appears in the Notes user interface when the user is creating or reading documents with the form.
$FormUsers : The list of people who are authorized to use the form. This is in addition to other security provided by the ACL and encryption.

View definition notes define the selection of documents, and columns specifying calculation or field-retrieval formulas and sorting. Specific items in the view definition are:

$Title : The title of the view.
$Formula : The formula which defines selection of documents to appear in this view.
$ViewFormat : A detailed structure defining the structure of the view, its display options, and for each column: the column’s name, the formula which computes values to display in the column, and sorting/categorization flags.
$Collection : A pointer to the object which holds the index.

The view definition note doesn’t contain an index of documents in the database; the index is held as an object in the database, and maintained by the NIF subsystem. Thus, when view definitions replicate to other servers, the index collection is not replicated but rebuilt locally. This reduces connection cost and allows for selective replication and planned differences between replicas of one database.

Design note

The Design note is an internal index of important information about all the forms, views, and other design elements of the database. This provides quick access, for example, such as when forms must be loaded to display documents.

Other classes

Other design elements are classes of note, including:

Selective Replication Definition : Contains a formula used in replicating between pairs of servers and/or users, so that, for example, a mobile user need only replicate relevant documents to the laptop.
Shared Field : Holds the definition of a field which can be inherited by more than one form.
Icon : Contains the database’s icon.
Help-About : Also called the Policy Document, a special document to give users information on the database’s purpose.
Help-Using : A special document to give users information on the database’s usage.
ACL : Holding the access control information for the database.

Many of these note classes are single-instance; they occur once per database. This is marked with a special flag so they can be identified quickly, and so that the replicator can behave accordingly, not storing more than one copy per database.

Item data structures

The Summary items (text, text-list, number, number-range, timedate, timedate-range) are stored in very compact, efficient structures; this enables their contents to be indexed and computed very quickly. This also provide enormous flexibility in multi-value field storage.

The Compound Document (CD) data structure is designed for maximum flexibility and extensibility to incorporate the variety of rich objects which must be included in Notes rich-text. It also includes nesting of paired structures. For example, "hotspot begin" and "hotspot end", between which may be "graphic begin"..."end" and so on. The CD structure’s comprehensive rich-text support means that duplicate information may be stored if required. For example, a single graphic image can be stored in bitmap, metafile and PICT formats, for quick rendering on any Notes workstation on any operating system.

The extensibility of the Compound Document structure makes it possible to incorporate support for future objects in Notes documents. For example, a major part of the Notes 4.0 development is to provide excellent OLE 2 capability. Figure 5 shows the internal structure of these data types.

Figure 5 -- Internal structure of various Notes data types

The common data types include:

TYPE_TEXT : Plain text, stored in LMBCS (Lotus multi-byte character set, a variable-length-character encoding. In its European and North-American form, this corresponds to Codepage 850 with the exception of codes less than x20.
TYPE_TEXT_LIST : A multi-value item where each value is plain LMBCS text. In the Notes user interface, no great distinction is made between single-value and multi-value items. A single-element text list is transparently converted to a plain text item. In a Notes API program, you must treat these as two distinct data types. This also applies to multi-value number and timedate items.
TYPE_NUMBER : A 64-bit IEEE floating point number.
TYPE_NUMBER_RANGE : A compound multi-value structure: a list of numbers, and/or a list of number pairs.
TYPE_TIMEDATE : A comprehensive time stamp structure, containing the number of days since Julian Day 0 (January 1, 4713 BC); or the constant ANYDAY, the number of ticks (hundredths of a second) since midnight; or the constant ALLDAY, the timezone (with quarter-hour granularity), and the daylight-savings flag.
TYPE_TIMEDATE_RANGE : A list of TIMEDATEs, and/or a list of TIMEDATE pairs.
TYPE_COMPOSITE : The rich-text (compound document) field structure.

These highly flexible and efficient data types give enormous potential to the application developer.

Programmability

The above description shows how Notes makes a distinction between summary (computable, indexable) information and non-summary (compound, relatively opaque) information. These two types of information have different access requirements. Access to summary information about a note must be highly optimized. Notes accomplishes this by physically separating summary items from a note into a summary buffer, which is stored for high-speed access, and objects, which contain non-summary information and may be retrieved at lower speed. Thus, rich-text fields are stored as objects (BLObs) and other fields are held in a summary buffer.

Summary data can be manipulated very efficiently, by the Compute engine and the NIF indexer, for example. The Notes formula language, executed through the Compute module, allows application designers to build intelligence into forms, for computation at the workstation; into views, for selection and collation; and into macros, for agent or rules functionality at the client or scheduled on the server. The formula language is highly optimized and tailored for building group-collaboration and workflow applications.

Notes’ formula language will be supplemented with LotusScript in Notes 4.0. LotusScript is an object-oriented, BASIC-compatible language which is an important part of Lotus’ programmability strategy for the entire suite of desktop and communication products. It interacts with each host product (Notes, for example) by having the hosts publish various classes of object, which can be created and manipulated with LotusScript code. In Notes, developers will be able to write LotusScript programs, executed on forms (in buttons and other form elements) or as server-based agents. These programs will be able to manipulate Notes documents and data items in a very high level.

The combination of the Compute engine for optimized formula evaluation, and the LotusScript compiler for procedural and object-oriented scripting, results in a highly efficient and powerful development environment. The developer’s code is stored in the same Notes databases which it defines and manipulates.

Indexing and searching

The application designer can create multiple views of the data in a Notes database. Users have the ability to create private views where they want to see a particular selection or presentation of the documents.

Views are defined by the selection of documents which match a formula; by the hierarchy of parent and response documents, if this is required; and by columns which structure summary information. A column displays the contents of a field, or the result of a formula (which may involve one or more fields). Columns can be categorized in multiple levels, to group related documents together; they can be sorted, by date or alphabetically or by the results of any formula; and they can be totaled.

Views operate internally by the creation of a view definition note (which replicates with the database) and the maintenance of an index by the NIF subsystem. NIF uses B-tree index algorithms to hold coherent, browsable collections of summary buffers with the column calculation results, collated according to specifications in the view definition note.

If a column is sorted, it must be collated in the index. This is done in three ways.

KEY : sorting is collation by the value of a column, based on a summary item's value.
TUMBLER : is collation of list item data types. Each value of the list corresponds to a level in a hierarchical (outline) index. As collation is performed, only the Nth list value is collated if collating the Nth level of the hierarchical index. This causes the new index entry to be placed as many levels deep as there are list values. For example, a number list value of 1:2:3 places the index entry three levels down in the hierarchy. If the new index entry requires a subtree which does not yet exist, a ghost entry is created to act as a parent for the new entry at intermediate levels. The result is a hierarchical outline where index entries are created at a variety of different levels in the index depending on the number of values in their list data type.
CATEGORY : is collation of any data types (list or non-list) to create a hierarchical index. For each CATEGORY in the collating spec, ghost entries are created for each unique value of the specified item (only as many ghost entries as there are unique values), and all duplicate values are placed at the next lower level.

Unlike TUMBLER, CATEGORY collates list data types exactly the same way as KEY: Each list value is compared; if equal, the next one is compared to break ties; and so on until the list is exhausted.

As an example, if a collating spec consists of CATEGORY "Folder", CATEGORY ""Author", and KEY "Date", the top level of the index only contains as many ghost entries as there are unique Folder names. Below it, the next level contains all unique Author names within the folder, and below each Author, the next level contains all the index entries for each Author sorted by Date. The result is a three-level hierarchical outline where all index entries are always at the Nth level, and all intermediate category levels always contain only ghost entries.

The Topic Full Text Retrieval engine, licensed from Verity Inc., creates associative indexes of the full text of Notes documents. Full text search lets users search for words, phrases, numbers, and dates, as well as perform queries using wildcards, logical and proximity operators, and other advanced features. Search results are ranked for relevance and displayed in the view.

Full text search also allows Query By Form, entering logical queries into specific fields on a form.

Sequential search and replace is also available within a data-base, a view or a single document. This may be performed on the server or the workstation as efficiency dictates.

These flexible methods provide users with many ways to manage and arrange the contents of multiple databases. Their generic nature allows powerful applications to be built utilizing their various features.

Structure of the Notes database

A Notes database is the container for notes: documents, form designs, view definitions, and all the other classes of note previously described (figure 6). The flexible structures and features of the notes are tailored to the need to transport and identify them: unique identifiers, last-modified timestamp, summary information and so on. The container database also reflects its function: the need to quickly retrieve appropriate information, the need for extreme reliability, and the need to support replication.

Figure 6 -- Structure of a Notes database

The Notes database, a file that usually has the extension .NSF, begins with some header information and an allocation map. Important parts of the header include:.

Database ID : Uniquely identifies the file.
Replica ID : Uniquely identifies all replica copies of this file on all servers.
Creation : timestamp.
Last Modification timestamp : The last time that any of the database’s contents were modified.
Title
Category : For easy identification.
Design Class : Indicates that the design elements should be inherited from a central template database and maintained in synchronization with that template.

The replicator task and others use this information to efficiently decide whether a database should be replicated, and which database replicas exist on any particular server. The database contents can be scanned extremely quickly for the notes which have been added or modified since a certain time. This again is optimized for the replicator.

The rest of the database contains notes and other objects; summary buffers stored for rapid access, and non-summary items in another area of the file. There is another internal abstraction: Notes themselves are a specialized class of object. Other object classes include file attachments, OLE packages, and packed lists of documents (unread lists, for example). Every object is identified by a Record relocation vector (RRV) which is a file-position pointer; a NOTEID is simply the note’s RRV.

Some objects, such as the collections which constitute an index of the database, are never replicated, since they must be built appropriate to each local copy of the database.

System databases

Certain Notes databases are used by the system for specific tasks. These are no different from other databases, but the system configuration indicates that they have special uses. The major system databases are:

Name & Address Book : Usually called NAMES.NSF, this database holds the user directory, which is modeled on X.500 specifications. It contains various classes of document, named by their forms:
Person : Information about a user, such as name, e-mail data-base or forwarding address, public key, and so on.
Group : Named list of people, used for access control lists and for electronic mail.
Connection : Information about the ports and connection details (phone number or other network address) for other Notes servers and remote systems; also, scheduling specifications to connect between particular servers.
Mail Router Mailbox -- Usually called MAIL.BOX, this database is the forwarding store for Notes mail on a server. The ROUTER task (the mail transfer agent) is responsible for polling this database and forwarding the documents to other servers’ mailboxes or to users’ mail files. In version 4 of Notes, the router task will consist of various MTAs for native X.400 and SMTP/MIME connectivity, in addition to the existing Notes and cc:Mail transfer protocols. Gateway add-in tasks usually have their own mailbox, where mail is deposited by the Notes router according to definitions of the gateway in the Name & Address Book. The gateway task polls its mailbox and transfers mail accordingly.
Database Catalog : Usually called CATALOG.NSF, this database is a catalog of all databases on the server and is the policy document from each database. If replicated through a Notes domain, it becomes the company’s catalog of Notes databases. It is maintained by a server task called CATALOG, which usually runs once per day.
Notes Log : Usually called LOG.NSF, this database contains the activity log of the Notes server. Server tasks report their activity to the database, where they can be summarized and reported in views for the administrator. This is the most basic level of system administration log. Other administration utilities include administrator-defined events and alerts, and system-wide network management tools from Lotus and other vendors.

Notes’ inherent flexibility and extensibility ensures that the core databases can accommodate all present and future needs (as mandated in the X.500 specification, for example).

Relationships between documents

Lotus Notes recognizes that an object container only becomes useful when its contents can be indexed and related. The concept of summary fields, used to collate documents in views, is one part of the solution. The other part is to create structured methods of relating documents to one another.

One key relationship is between members of a hierarchy. The parent, response, and response-to-response hierarchy in Notes is used to relate topics in a discussion, to relate documents in a correspondence database, and so on for all the major types of workgroup application.

Other Notes applications can define their own structures for document relationships. To avoid duplication of storage, forms can perform lookups into other views to retrieve a column’s contents, and documents to retrieve individual values from key documents, using the @DbLookup and @DbColumn commands.

Version control is included in Notes through the ability to store prior versions, or the latest version, of a document as a response to the current, or previous, version.

Doclinks are hypertext links between documents and (recently, with the introduction of SmarText 3.0) from outside Notes altogether. They give users some ad-hoc ability to knit related documents together where there is no formal relationship.

If the structure of an application mandates referential integrity or normalized data structures, the design of the application is crucial. This might be the case if the application is to be your single database of all customer contact, or if the database must use key values to lookup into a relational database system. The flexible, diverse structure of Notes databases need not make your data any less structured that the data in a relational system -- everything is a consequence of the design. Notes is as suitable for those applications requiring standardized data structures, such as inter-enterprise workflow and EDI, as it is for those needing little structure.

Relationships outside the database

External linkages are essential to integration with transactional or legacy databases and other data sources. There are a number of these methods included in Notes, and open interfaces for third-party providers.

The highest level of integration is found at the platform-specific compound document format. Notes is the perfect container for OLE embedded objects, allowing live application objects to be embedded seamlessly in documents for sharing with other users.

In addition to its inter-application data exchange facilities, a second level of data integration is achieved through use of Notes’ rich set of import/export filters. By using these filters, users can copy information from application files directly into Notes databases or documents, retaining much of the original application's formatting information, or from Notes documents to application formats, again retaining the original look.

From Notes forms (field formulas, buttons and other elements), smarticons and macros, the @DbLookup and @DbColumn commands allow lookup into Notes databases. They also support the DataLens driver technology, which allows retrieval from external databases including Oracle, Sybase and Microsoft SQL Server, Informix, IBM Database Manager, Paradox, and dBASE. The @DbCommand function allows database queries in SQL and proprietary command languages, and it allows access to stored queries on the database server. DataLens also supports ODBC. These functions can be run from the workstation or, in background macros, on the server for agent-type integration.

The NotesSQL driver provides an ODBC interface to Lotus Notes data. Thus a wide range of third-party applications have access to Notes databases without special APIs.

Using the Notes API, many developers have created specific integration tools for their customers. Powerful tools available include Trinzic’s InfoPump, a scripted database integration utility which can migrate and synchronize data from a wide variety of sources, on an event-driven or scheduled basis.

Front-end development tools which can access data from Notes and integrate with other data sources include Lotus Notes ViP, Powersoft PowerBuilder, Gupta SQLWindows, Revelation OpenInsight, and many others. Lotus Development Corp. also sells a range of companion products, including the image-capable Lotus Notes:Document Imaging product, incoming and outgoing FAX gateways, PhoneNotes for developing interactive voice response applications integrated with the Notes environment, and an optical character recognition (OCR) server for scanning faxes or images into Notes documents.

Using these open tool sets, the application developer and the systems integrator can relate Notes information with other systems for integrity across the enterprise.

Notes applications

The combination of form and view design tools means that Notes applications are developed in much less time than with traditional 3GL and 4GL tools. Design is flexible even after the application has been placed into production because changes roll out through replication, at the same time as the data documents replicate. This unique feature, inherent in the design of Notes, means that applications can be refined continually to conform to user requirements, with no prototype-test-deploy-evaluate cycle; rather, continuous proto-cycling iterations can be used until everyone is happy.

The database templates that ship with Notes can be used to create identical applications or customizable applications. Design inheritance from standard templates and shared field definitions within a database provide data dictionary capabili-ties, enabling leverage and reuse of design elements, maintainability of database design over time, and the ability to set corporate design standards.

Figure 7 -- Overall Notes architecture

Lotus Notes and the Lotus Communications Server

Notes’ robust and scaleable server environment is also the basis of Lotus’ next generation of messaging servers. cc:Mail users will have the option of keeping the shared-file PostOffice environment or of migrating to a client-server system if this is more suited to their long-term needs. The integrated platform is called the Lotus Communications Server. It includes cc:Mail integration (with all the advantages of the cc:Mail system), multiple standards support in APIs, and a single platform for external gateways. It is also the fourth generation of the Lotus Notes server. This gives a uniquely powerful environment for communication.

Lotus Notes is a strategic system. Businesses deploy the Notes product for reasons which dramatically affect their performance. With any strategic software purchase, it is important for IT planners to understand the foundations of that software. The architecture of Lotus Notes is of long-term importance to the way people collaborate and do business. With this information you can make truly informed decisions about the enterprise strategies of Lotus Development Corp. and other software vendors.