Trip Report: ExLibris Developers Meeting 2008

MPDL,Aleph,ExLCustomers

Event: 1st ExLibris Developers' Meeting, Jerusalem, 12./13.11.2008

Participants MPDL: Erik, Inga

Summary: ExLibris Ltd invited a small group of people developing on their products (like Aleph, Primo, Metalib and SFX) to discuss the prospects and implications of the "Open-Platform Strategy".

Documents:
 * Lukas Koster wrote a short report for the IGeLU Newsletter 2-2008 (pdf)
 * Presentations expected to become available via Documentation Portal

Open architecture and the Ex Libris open-platform strategy, Shlomo Sanders
Software Development Kit (SDK) = API with some tools/classes/xslts on customer side. SDKs can be understood as an extension of an API because some source files and tools are supplied by the API provider to use the interface seamlessly. All future ExLibris tools/developments will come with some SDK support.

APIs. Goal: offer appropriate bindings for each functionalities, e.g. RESTfull services for resource oriented requests; X-Server/SOAP for more heavy workflows
 * "stateless has a direct relation to scalability"
 * stateless wherever possible; stateless wherever it should be stateless
 * Aleph: all ILS services to support REST interfaces as recommended by DLF, see ILS Discovery Interfaces

PlugIn-Manager: The objective is to manage the process of adding/removing a plug-in from the application, incl.
 * check of validity (of the plug-in for the specific setup)
 * transfer, unpack and place everything where it belongs
 * audit all changes

Data Services should contain all information which is required by more than one institute AND should provide services on the data. Examples:
 * is this item part of an FRBR work
 * usage data. (bx initiative to load usage statistic data to data services)

Primo deep search (3rd node) as an example for an Adapter. (plug-in is a step in a well-defined process, access to external data/application/etc.)

Reporting Interfaces (SQL-based)
 * read-only
 * improved performance (reporting-specific Oracle indexes)

UI customization - a crucial issue
 * Customization of html is "a catastrophe" (as it deprives the software upgrade process of making necessary changes - severe obstacle to version upgrade)
 * Ex Libris even considers no longer providing html at all
 * Template variables criticized (for example "$0100", not unique even within single template but only within template section) - (more) global variables required
 * Why not rely on xslt (global access to data via XPath)? - objection by Shlomo: "CPU gobbler" - transformation needs to be closely integrated with web server. Note taken down.

The realization of the open-platform program, Revital Marck

 * increased emphasis on SOA principles
 * expose everything to the upper layer
 * idea of sharing support tools, e.g. clients developed by ExLibris to be run locally to identify potential pitfalls and problems OR any other test suites
 * idea of running demo installations, where local solution may be deployed
 * setup of documentation templates for the sake of uniform documentation
 * establish development - documentation workflows
 * open interface section to development design templates (i.e. support open interfaces from the very beginning)
 * enhancement workflow: requirement - requirement document - functional specification (by developer, design template) - technical specification (by developer) - software
 * versioning of open interfaces: establish versioning standards to ensure robust solutions keeping backward compability
 * Ex Libris is moving forward in using REST. When? service, resource oriented versus business logic. Set guidelines to programmers: when to do apply which philosophy?
 * CRM and EL Commons are not related

Spicing up your interface with jQuery, Hanan Jacob

 * UI customization with primo 2.1
 * based on "tiles" which can be grouped and arranged via a web front end. all tiles "receive" a certain set of information, e.g. record id, session, search term
 * primo tiles (= predefined functionalities, e.g. search box, result list, static html, etc.)
 * custom tiles, e.g. to load external html code to a specific area on the user interface
 * test tile (which expose all available information


 * jQuery javascript library, see http://jquery.com/
 * DOM-based, supports tremendous DOM operations
 * produces very little code
 * might be quite powerful in combination with the "tiles"
 * layout manager

Mark Dehmlow, University of Notre Dame, USA
Mark presented the FindIt service, leveraging the Aleph X-Server to provide enhanced links, library maps, prefilled document delivery forms, and more from catalog searches.

Related presentation (from IGeLU 2008): http://igelu.org/files/webfm/public/documents/conference2008/presentations/20a_dehmlow.pdf

Daniel Forsman, Jönköping University, Sweden
Daniel gave a introduction to the extensions implemented for JULIA, the Aleph catalog used by the University of Jönköping. The developments focused on utilizing web services, including Google SOAP, Amazon REST and the API for the Swedish union catalog.

Notes:
 * JULIA search interface is integrated into the library web pages
 * Spell checking ("Did you mean") functionality by integrating yahoo's web service, see example
 * Hints on results screen: looking at most often search queries and collect recommendations from subject librarians to be displayed
 * Zeitgeist to display most popular titles
 * Run search additionally in external resources (meta-search, union catalog, etc.)
 * The full record display mashes up book covers, editorial reviews from amazon.com and further information from Google Books
 * Future plan
 * utilizing linked data offered by Swedish national library
 * integrating translation services

Inga Overkamp & Erik Altmann, Max Planck Society, Germany
We gave an overview on our work with MetaLib and SFX.

Presentation: http://edoc.mpg.de/395234

Peter van Boheemen, Wageningen University Library, Netherlands
Peter presented how the library use their Content Management System as an extra knowledge base for adding services to the SFX menu

Ere Maijala, University of Helsinki, Finland
Ere described several projects that take place in Finland on a national level. Current initiatives include a large-scale public interface project for all memory institutions (libraries, museums, and archives), a project aiming at getting a better understanding the user's needs, and the replacement for the Voyager-based union catalog with an Aleph system.

Strategy: Try to integrate the stuff into places where the users are.

John Osborn & Brian Thompson, University of Iowa, USA
Johna and Brian showed a variety of customizations they have developed in their implementation of Primo. This includes such things as "hotwords", integration of Primo into other search utilities and application, and the development of a pipe for our library web pages.

Note: Currently no "real-time" availability check (EEE catalog?)

Lukas Koster & Roxana Popistasu
see LibStats presentation


 * Idea: Collect/combine various types of sources (e.g. logfiles, database export, etc.)
 * Add statistics to database for evaluation
 * Access via MySQL admin, ODBC, etc.
 * Qualitative (session-based navigation patterns) and quantitative analysis

Working group SFX

 * Existing APIs:
 * XML API/SRU interface for A-Z list requested, this would also allow the integration of the SFX A-Z list as a searchable resource in MetaLib Anyway, it was quite interesting that just one of the participating customers is currently using the A-Z functionality of SFX
 * Availability check (sfx.response_type=service_exist) should be extended to objects without ISSN
 * Image based requests (sfx.response_type=image-large and sfx.response_type=image-short) is currently not documented in El Commons, but could be of potential interest as it provide a very easy way for integrating SFX availability information into external environments.


 * Improving Performance:
 * Caching availability information, e.g. the "pre-module to SFX" at the Wageningen UR Library) -> interesting idea, e.g. to provide very fast access to often requested objects
 * Re-organization of execution of service evaluation, e.g. do not execute plug-ins if you already know that another service is available (not to apply display logic at a very late point)?


 * Coming changes:
 * concept of organizing the SFX knowledgebase into three zones: LOCAL (only maintained by creator), SHARED (maintained by everybody and distributed with SFX updates) and GLOBAL (maintained by ExL kb team) In the discussion it became obvious that ExLibris is currently considering regarding the SHARED area, e.g. should changes really been automatically distributed to all customers. The group argued that this would require a high level of transparency
 * SFX will be further developed in PERL, a re-implementation in JAVA is not planned for SFX Version 4
 * SFX team considers to offer a list of "supported" and "not-supported" adaptations on customer side, e.g. parsers, plug-ins and templates can be localized while the core modules shouldn't be touched. They also agree that there are circumstances in which core modules need to be localized nevertheless...

Working group MetaLib

 * Openness/Interoperability:
 * mod_aleph takes to much control over other apache modules.
 * customers use X-services along with regular user interface (whereas Ex Libris presumed an approach of more isolated usage of interfaces, i.e. either customers present services under the regular user interface /V, or build their own solution based on /X).
 * some relation of /V and /X sessions was desired (though this is a security issue - normal users must not be able to generate a session which can take control over the full range of X-services).
 * proposal: views to oracle tables (as a kind of back office API, read-only access to data).
 * MetaLib enhancements/extensions contributed via EL Commons are often workarounds/hacks/UI tweaks, which ought to be transformed into proper software enhancements. Enhancement process/review by development should be triggered simultaneously.
 * product-specific API: problem to open services to third parties (security issue, X-server session too powerful).


 * Backwards compatibility:
 * of API (MetaLib X-Server) - tied to product life cycle/to specific versions - even deprecation of APIs in subsequent versions was suggested from customers' side (to achieve increased flexibility).
 * On the other hand, backwards compatibility even of system internals/back office components/logfiles/... would be appreciated (so that reporting tools, etc., may benefit).
 * Stability of deep links, html customization (broken by, for example, Find Database CJK support, distributed via SP) is a requirement.


 * Questions:
 * If moving towards new generation solutions - (how) will API access be maintained?
 * What's the life cycle of an API?
 * What's the relationship between stability/backwards compatibility and flexibility/new generation development?

Working group Primo

 * What web 2.0 tools might fill gaps?
 * additions that create context
 * share usability
 * rss
 * consider third party tools
 * good site map
 * comments on content (as rss? email?)
 * compile lists of desired features
 * More issues
 * get critical mass of data
 * development: sharing customized tiles (basic code unit of Primo UI development, maintained via web-based interface)..
 * "get it" links vs. title links (get resource rather than information about the resource)
 * tag cloud
 * how to seed
 * import tags
 * display facets as tag cloud
 * normalization rules (eye-readable, printout, PriView by Boston College), comparison between different sets of rules
 * batch-adding subjects from MetaLib as tags
 * desire for faster response times from server

Working group Aleph

 * API/Interfaces - advantages of X-Services, SOAP, REST
 * Batch loading of records
 * built-in manipulation of records using external software (MarcIT, rule-based modification)
 * Development
 * enhanced documentation - note: file internal documentation readable with Firefox add-on
 * share xsl, css in EL Commons

EL Commons
Notes on EL Commons as a new Ex Libris customers community platform.


 * Navigation considered awkward, even by experienced wiki users.
 * Still slow, error messages.
 * Developer Zone: Non-developers may be scared off, though content may still be of interest. In fact there's no need to agree to license if there's no intention to contribute code.
 * Question: Open EL Commons to third parties (non Ex Libris customers)? - Problem: A lot of code/APIs is not in a state to be used by third parties due to the security concept behind (i.e. designed to be used by customers only).
 * Desire for enhanced search for people, extended profiles to keep track of individual projects, person/project-related news.

Conclusions and future plans

 * Lack of documentation of database tables/data structures was criticized (tables not relational, not normalized, etc.) - proposal: publish documentation created by users? - However, time ought not to be spent on documentation of data, but rather on the documentation of Open Interfaces.
 * Direct (read) access to data is ok, need for views (= data API) is not critical - though certain backwards compatibility is expected.
 * Integration of EL Commons, IGeLU, Doc Portal, etc. - problem of authorization (question: openness to non Ex Libris customers? Has to be restricted at some point).
 * Temporary code solutions to be provided via EL Commons?
 * May obsolete code be deleted? - Yes.
 * BSD license: Contributor holds rights to own modifications, not to the entire (Ex Libris owned) file.
 * Need to integrate EL Commons with developers' working environments.
 * It was generally agreed that a low number of participants in the developers' meeting is an advantage. Contributions on EL Commons as a prerequisite to participate in future meetings?

Impressions
A few impressions and observations are compiled here.


 * Ex Libris: A competent team, with expertise in every field of web-based software development.
 * The team maintaining the SFX Knowledgebase is continuously growing and meanwhile includes about 20 members. Hugh additions and corrections are planned for the near future (e.g. eBooks), but they often face unsupportive information providers.
 * Good strategic approaches, but some of the ideas are very ambitious and it remains unclear how they want to conceptualize, implement and deliver all the envisioned services (Primo, SFX v4, MetaSearch NextGen, UML, etc.) in parallel.
 * Concepts for present maintenance/development do exist.
 * Question about transition to next generation unanswered.
 * A vision of customers buying components/modules (rather than products) was created, but not really discussed.
 * Diverging expectations among developers, especially regarding stability/life cycles of interfaces. There was some agreement on documentation/backwards compatibility of data structures (see above).
 * Maybe some sort of starting point for a community/"culture of sharing" has been established successfully.
 * Product-specific
 * It's questionable whether the core MetaLib search engine will be further developed. At least, it does not seem to be a demand of MetaLib customers (talking mainly about APIs, not about the core service itself).
 * Looks as if MetaLib was running towards dead end. Moving to next generation, we may be facing problems to preserve/transfer our work to a new tool/environment (detailed data processing instructions, interface configurations). But not tomorrow.
 * SFX v4 will mainly focus on implementing the new KB concept to allow direct user contributions. Anyway, the details are not very elaborated and decided at this point in time and will require some additional effort.

Links

 * IGeLU Newsletter 2008-2 Ex Libris "Developers Meet Developers" Meeting
 * The Ex Librian Newsletter January 2009 - The Ex Libris Open-Platform Strategy: Developer meets Developer