The problem of music modeling in computer systems has been addressed several times in the literature for different purposes. Typical computer-based applications of music modeling are:
  • edit/print music for professional and home purposes with the goal of producing a paper version of music scores;
  • execute music to produce sounds by means of sound-boards;
  • acquire or produce music using Musical Instrumental Digital Interface (MIDI) for coding of sound, composition;
  • store/load music scores for analysis, maintenance, execution, versioning;
  • analyze music composition styles (musicologists and researchers);
  • synthesize music through a computer (by using mathematical and statistical models);
  • convert music from a paper score to a computer-based format, OMR (Optical Music Recognition) system similar to OCR (Optical Character Recognition);
  • teach music and its rules -- e.g., produce interactive educational CDs for teaching music;
  • manipulate music scores in order to change arrangements, transpose keys, etc.
  • ..., etc.
Some of the above applications can be used to build powerful tools for specific purposes; analyzing a music style in order to generate similar music; using a MIDI interface as a shortcut to compose music and generate a draft version of a music score.



On the basis of the above list, it can be seen that a music-based software application may use a music model having multiple interpretations and different manipulation mechanisms. In fact, publishers of music scores, composers, musicians, conductors, and computer music analysts, all regard music notation with different interests and use it for different purposes. In general, the elementary operations that can be performed on a music score are: editing, saving, loading, visualizing, executing, coding grabbed sounds, printing, analyzing, etc. In order to provide these capabilities, a specific music model must be defined.

The complexity of the music model needed obviously depends on the application for which the model is produced.
Different applications present different needs in terms of music modeling. Among the above-mentioned applications, one of the most complex is to edit music for professional publishing. Publishing requires a very high quality music score in terms of the number of symbols and their precise placement on the staff (set of parallel lines on which the music symbols are placed). Music scores have to be presented to professional users (musicians) with specific relationships and proportions among symbols. Visual relationships among symbols are the syntax of music notation. The formal relationships among symbols are frequently neglected or misused by most commercial computer-based editors, which frequently focus on producing music by means of MIDI interfaces or sound-boards, or acquiring music by means of a MIDI interface, or simply editing music on the computer for managing sound-boards, etc. Typically, in these cases, a limited number of music notation symbols are needed (e.g., notes, rests, accidentals, clefs and points -- approximately 50 symbols), since it is impossible to reproduce with a MIDI interface the effects specified by most music notation symbols.

The music symbols available in professional editors (e.g., Score, Finale, Sibelius, etc. for publishing music are: the   basic symbols  (notes, rests, beams, etc.),   instrumental and execution symbols, (symbols for describing the behavior of the musicians in playing a specific instrument -- e.g., up or down bow, with mute, without the mute, etc.) for: strings, harps, drums, flutes, etc.;   orchestral and complete repeat symbols , etc. These symbols,   and many others, are needed when main scores and parts for classical music, operas and ballets have to be produced, and absolutely essential when scores are used in music schools to train students to perform specific executions and interpretations. In commercial editors used to prepare music scores for publishing, the number of   elementary symbols is close to 300. These are frequently used as components for building more complex symbols. Typically, musicians have to personalize/prepare a score for the execution; thus, even this great number of symbols is not enough to satisfy all needs. For these reasons, some music editors provide a font editor for adding symbols; these, unfortunately, are treated as simple graphic entities.

The organization of many detailed symbols is frequently left to the user. Typically, music symbols (excluding notes, rests and a few other symbols) are mainly considered by music editors as graphic elements that can be placed in any position on a score page without addressing the problems related to the music notation; e.g., the visual and syntactic relationships among symbols.  Most music editors even allow the placement of several notation symbols in incorrect positions, resulting in the production of strange, incorrect music scores. These music editors give no support to the users. These incorrect music scores cannot be played by musicians, or correctly interpreted by style analysis algorithms, or correctly used to generate sound, and are, thus, professionally unusable. In several cases, only those symbols having a counterpart in the MIDI language are considered.
Commercial music editors for publishing are mainly oriented towards placing music symbols on the staff rather than collecting relationships among symbols and on that basis organizing the visual information.
Instead, they are mainly oriented towards printing music, since this is their most important application. They provide a complete set of symbols and instruments that a user skilled in both music notation and graphic computer user interface can use to produce a professional page of music.

For these reasons, professional music editors are powerful, but are difficult for non-expert users in music notation to employ. Typically, musicians are capable of reading music, but are not familiar with the exact rules for arranging symbols (e.g., when symbol  A  is associated with symbol  B  symbol  C  has to be moved up one-half unit). On the other hand, musicians have no problem correctly reading a correctly annotated score, but can be perplexed when non-perfect visual constructs are proposed. These musicians are artists, not music engraving and notation technicians. Conductors and composers are frequently more sensitive to problems of music notation and visual expressiveness. Archivists in music theaters, music engravers, and teachers of music notation are the experts to relied on for problems of music notation.

The above-mentioned problems also exist in general languages for storing/coding music. Concerning definition of relationships among symbols, the most flexible and complete languages are NIFF (Notation Interchange File Format) and SMDL (Standard Music Description Languages). Even these languages do not model all the relationships described above and allow placement of symbols in any point in the staff. There are several languages for coding music, as described in  SField97 , but unfortunately all of these languages suffer from the same problems.

In order to solve these problems, music notation should be considered a formal visual language in which errors in the placement of symbols are totally unacceptable because they are out of syntax - e.g.,  Chang87a . Music editors must provide support for the verification of the visual syntax. Moreover, a visual editor has to help the user to easily produce only correct music scores that are based on the visual grammar among symbols. These very important features can be easily obtained if the syntax and the relationships between the notational entities are formally defined. In the case of music, the definition of the visual grammar is a highly complex task that is not yet formalized. However, the existence of a formal model is a great improvement with respect to the state of the art.

To model all relationships of music notation is quite a complex task.
As the number of music symbols grows, the number of relationships among them and the related rules for visualizing them grows more than proportionally, since the presence of certain symbols influences the relationships among other symbols. The rules for adopting symbols are much more important than their availability as graphic elements to be inserted on a score. Therefore, the knowledge of the syntax and the semantics of music is mandatory for organizing music symbols on the staff.
For these reasons, very few music editors (Finale, MusicEase, Lime, etc.) address the problems of (i) automatic placement of symbols and (ii) music justification. Some of these are capable of correctly arranging a limited number of symbols.

In music, justification involves the placement of symbols both along the staff and the transversal of the staff, and can be performed on the basis of several different algorithms. These rules together with those for the justification of music have not yet been formalized. Several books have been written to solve this problem, but inconsistencies and incompleteness still exist.
For certain aspects, the automatic justification of music should be regarded as an open problem. Moreover, only a few symbols are placed automatically; for instance, in the music notation there are several symbols that have must be placed around the notehead. These symbols frequently must be placed by hand, whereas automatic arrangement could greatly simplify the music editing. For example, by selecting a given symbol and the note, the symbol has to be automatically placed in the right position with respect to the note and the other symbols involved.

Currently, even in very mature music editors, the positioning of music elements on the score presents many critical conditions. These music editors should provide a type of auto-correction, as is done for words in modern text editors.
Instead, the user has to work around the positioning of symbols, performing a very long and tedious process of adjustment.


New Needs


Recently, with the diffusion of computer technology into the artistic fields, new needs for computer-based applications of music have been identified. In 1994 in our research group in Florence, we began to research about a network of computer-based music lecterns oriented to musicians. These were studied for editing/showing music scores by means of a suitable video device, with the goal of totally removing the paper scores from musician stands (lecterns). Computerized music lecterns can be used by musicians to avoid transporting many kilograms of paper music scores, to save their work, to manage version, control, etc.
At first glance, this kind of application seems not much different from the music editors that are currently on the market. An important difference is in the user interface, which has to be much simpler than that of classical music editors, since it has to be   usable by musicians . The user interface must incorporate some of the mechanisms that are typical of editors for visual languages: (i) the presence of visual parser based on the grammatical rules of the symbols; and (ii) a visual analyzer for the automatic arrangements of symbols on the basis of lexical rules.

In the last three years, these problems have been studied by our research group, focused on the premise that the orchestras could be supported by a set of computer-based lecterns. If the lecterns are connected each other by means of a network, information shared among lecterns can be used to define some new and very interesting applications, such as:

  • lecterns for orchestras with an automatic and synchronous mechanism for turning pages and for cooperative manipulation of music scores,
  • cooperative editing of music scores for music publishers, and
  • interactive and constrained cooperative music editing for teaching music composition and execution in schools of music.
The main obstacle that we found to adopting the available commercial music editors in this way is the lack of a formal model of music. In fact, without a formal model, all relationships among visual notation constructs are difficult to maintain. In some cases, the visual syntax may change on the basis of the context and thus of the semantics of the statement under construction.
A distributed system with the above features must be capable of showing music in different formats and at the same time must support cooperative work of editing on different formats, showing the changes of one operator to the other in real time. Therefore, a model of music is needed to define and implement algorithms for the automatic:
  1. (i) Placement of attributes of the music notation by means a visual parser based on grammatical rules of music notation;
  2. (ii) Formatting of symbols when changes are made, for instance, when another symbol is added;
  3. (iii) Rapid conversion of a visualization mode to another depending on the user/musician that is reading the score or fo the score page dimension. Typically, scores for musicians and for the conductor are formatted differently.
This approach allowed the evolution of our research towards very new applications of music modeling.
We started with the definition of an object-oriented model and framework. This was initially used for building music editors for musicians and conductors. The present version is used as a basis for two main projects:
 
  • MOODS (Music Object-Oriented Distributed System) ESPRIT HPCN. The project has, among its goals, the implementation of a distributed system of lecterns for orchestras, publishers and schools of music. Partners of the MOODS consortium are: the well-known Teatro alla Scala of Milan, BMG Ricordi as music publisher, the School of Music of Fiesole, Shylock Projects as experts of music score databases (see projects CANTATE and HARMONICA), ELSEL as builder of computer-based lecterns, and DSI (Department of Systems and Informatics) as project coordinator.
  • O3MR (Object Oriented Optical Music Recognition), a project concerned with building an optical music recognition system for converting paper versions of music scores in a suitable format for music editors. O3MR is based on the adoption of the neural network for low-level recognition of symbols, while at high-level the constructs of music notation are recognized by using the object-oriented model of music discussed in this paper.
The activity on music at the Department of Systems and Informatics is partially supported by Hewlett Packard Italy.