Talk:MPDL Project XML Workflow

=Agenda Meeting 16.10.2008=

Project and current status

 * https://itgroup.mpiwg-berlin.mpg.de:8080/tracs/mpdl-project-content
 * https://itgroup.mpiwg-berlin.mpg.de:8080/tracs/mpdl-project-software

The XML Workflow project started September 2008. There are two main parts of the project:
 * Defining a working process (workflow) for production of XML texts (and documenting this process so that it can be reused)
 * Digitization of e.g. manuscripts
 * Transcription of the text presented on the manuscripts
 * Markup of parts of the XML texts
 * Enrichment of the XML texts


 * Development aspect by enabling tools and infrastructure to
 * enable access to documents, linking between documents and internal parts of the documents
 * building functions for searching, indexing and retrieval of relevant results

The main motivation is to standardize the working processes and develop a Center of competence that provides guidelines for transcription of texts.


 * Currently the project is in the initial design phase
 * One of the main goals is to use a repository functionality for all artefacts and enable easier reuse such as: importing the texts on which a work needs to be done into tool installed on a local system, as well as easily submitting the modified texts back to the repository

Introduction&status overview of eSciDoc project
Introduction into:


 * eSciDoc service infrastructure
 * eSciDoc data model
 * support for versioning
 * support for persistent identification
 * support for authorization
 * currently developed solutions

Presentations:
 * [[Image:CoreDataModelandServiceComposition.ppt|Core data model in eSciDoc and an example of a service composition]]
 * | eSciDoc Service Development

Repository
Functions that need to be provided are basically:
 * persistent storage for resources that come from various projects
 * versioning of resources
 * persistent identification of resources
 * possibility to access arbitrary portions of XML Documents
 * transformation of XML documents to XHTML for presentation purposes
 * enrichment of data such as:
 * links to language specific functionality
 * links to sources (available on the web)
 * expose resources publicly as soon as possible for further processing

Searching functionality (general)

 * project team considers as a core system to search within the XML documents the following: eXist database, Lucene or Oracle 11g
 * two types of queries need to be supported:
 * structural queries of XML documents (in particular subtrees, sets of subtrees)
 * full-text searching (integrated language technology)
 * support for different languages/scripts such as: Latin, Greek, Chinese, European languages, Sanscrit
 * Digilib - to be enabled as a service for viewing in-line images such as figures, diagrams
 * need to have the possibility to use quite mature/robust tools for working with images

Lemmatized search (more precise)

 * The functionality MPIWG intends to build is to enable searching by a word in its citation form
 * for modern Western European languages stemming may be acceptable
 * for other languages (e.g. Sanskrit, Latin) words need to be indexed by lemma (e.g. "tulit" → "fero")


 * NLP (Natural Language Processing)
 * enable generic interface to dictionaries
 * define/check on standards


 * Acting as XML Middleware
 * XML document is always indexed each time it is changed

Relations

 * as a minimum a need to have a core set of repository services to easily create objects and relate these objects

Overview of supported repository functionalities

 * eSciDoc alredy provides several functionalities stated within the high-level requirements:


 * a repository functionality that supports versioning, persistent identification and quite free modeling schema for content resources
 * possibility to define different workflows and schemas for different projects (via the usage of "Context" resource)
 * possibility to define fine-granular access level to resources and parts of resources
 * resources are managed via Resource Handlers exposed as web services
 * authorization is done for each service operation on the resource (as eSciDoc core services are stateless)
 * services are provided as SOAP or REST interfaces
 * the possibility to define Containers, Members and TOC is of great advantage for the XML Workflows
 * all metadata and selected formats of full-texts are searchable
 * Digilib is integrated with the item handler interface and parameters can be used appended to the (file) content retrieval request

Discussion

 * The TOC is an interesting feature also for XML Workflow project
 * The next release of the VIRR solution may serve also partly the needs of the XML Workflow project
 * XML Documents do not use TEI as standard, they partly depart from this standard
 * It is recommended that the substantial value for the outcome of this project (in general for editing XML documents) is to stick to (general or community) standards as much as possible


 * As XML Workflow project also considered usage of Fedora, it is recommended to use eSciDoc service infrastructure that provides extended functionality to Fedora especially in terms of content model, encapsulation of the complexity of objects structure
 * an image available as thumbnail, web-resolution or original-sized image can be in eSciDoc encapsulated as a single resource with three components - even though these are 4 different Fedora objects.
 * in this case the versioning is done on both component-level and resource level
 * in this case the PID can anyway be associated both at resource-level and component level

Development

 * The first stable 1.0 release of the eSciDoc infrastructure will be available by end of the year 2008
 * MPDL will inform XML Workflow project team on the availability
 * The XML Workflow project team can use DEMO version of the eSciDoc repository for first tests
 * For further development, to be independent of MPDL development efforts on other solutions, XML Workflow project team needs to set-up own development installation
 * The XML Workflow project team can use the productive installation of eSciDoc repository for real data and work i.e. after enablement of particular solutions
 * We need to further discuss the process of feedback and inclusion of evtl. new features into core services, MPDL developed solutions when the XML Workflow project team establish own development environment
 * XML Workflow project team is welcomed to use Colab (where this page is also written) - to share ideas with bigger community.
 * see also MPDLMediaWiki:Warming_Up

Other related projects

 * MPDL Solution VIRR (R1, possible relation to XML Workflow project)
 * see first Demo release


 * MPDL Project INTER (pls. contact Marc Snijders, MPI PL for further information on demo he made with Lexica)


 * DFG Viewer (available at DFG Viewer)
 * to check with DFG Viewer people to extend it with parameters to use the Digilib service (will be checked by Robert Casties)