Escidoc-core Migration v1.1 to v1.2

From MPDLMediaWiki
Jump to navigation Jump to search

step-by-step instructioncs to migrate escidoc-core 1.1 to escidoc-core 1.2. (detailed information is related to the MPDL productive environment).

Affected servers[edit]

  • (PortgreSQL and Fedora)
  • ( (escidoc-core)
  • (PubMan)
  • (Faces)
  • (Virr)
  • (PubMan)
  • any other instance accessing

Required software packages[edit]

all available at

  • foxml_migration-jar-with-dependencies.jar
    • plus xsl folder containing xslt styesheets:
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
wget --no-check-certificate
    • plus file

available at

Pre-Migration requirements[edit]

  • shutdown all solutions accessing
    • stop JBoss application server on
    • stop JBoss application server on any other instance, which is accessing
  • shutdown the escidoc-core instance
    • stop JBoss application server on
  • shutdown Fedora instance
    • /X/fedora/tomcat/bin/ on
  • only

Main migration steps[edit]

Installation of Fedora 3.3.1[edit]

  • on change to directory /X
  • delete the symbolic link fedora
  • create a new directory fedora3.3
  • create a new symbolic link fedora pointing to the new directory fedora3.3
  • drop old fedora databases "fedora32" and "riTriples"
  • CREATE DATABASE "fedora3" WITH ENCODING='UTF8' OWNER="fedoraAdmin";
  • install Fedora 3.3.1
    • java -jar fcrepo-installer-3.3-fix01.jar
    • instructions can be found here:
    • when the installation is finished
      • copy the entire data directory located in the old Fedora instance (/X/fedora3.2.1/data) into the newly installed Fedora instance

Migration of the FOXML files[edit]

this will backup the Fedora data diectory!

  • go to escidoc-core-admin-1.2
  • edit the file
    • check Fedora properties
    • especially fedora-src.home=/X/fedora
  • ANT_OPTS=-Xmx1024m -Xms512m -XX:MaxPermSize=256m
  • call ant foxml-migration-from1.1-to1.2

Research Data Repository Only: Migrating old escidoc:TOC content model[edit]

  • locate the escidoc_TOC file inside /X/fedora/data/objects and open it for editing
    • replace all occurances of "TOC" with "toc"
    • check the creation date of the RELS-EXT.0 and the RELS-EXT.1 datastreams
    • if both dates "CREATED=<date>" are equal, decrease the one in RELS-EXT.0
  • update all references to escidoc:TOC in any resource
    • the current file has a list of all relevant resources
    • java -jar foxml_migration-jar-with-dependencies.jar cmodel

Rebuild Fedora[edit]

  • edit <fedora-dir>/server/bin/
exec_cmd="exec \"$java\" -server -Xmn64m -Xms256m -Xmx1024m \
  • start fedora: <fedora-dir>/tomcat/bin/
  • rebuild the Fedora database and the Fedora resource index
    • Location: <fedora-dir>/server/bin
    • Script:
      • CAUTION! causes NoClassDefFoundError on "exotic" variants of JDK. (Let the SUN shine in ...)
      • rebuild first the database
      • rebuild next the resource index

Post-rebuild Fedora steps[edit]

  • create new tablespaces
    • check if tablespaces for directories are created
     mkdir /var/lib/pgsql/data/tables
     mkdir /var/lib/pgsql/data/tables/fedora
     mkdir /var/lib/pgsql/data/tables/triples
     mkdir /var/lib/pgsql/data/tables/escidoc-core
     mkdir /var/lib/pgsql/data/tables/statistics
     mkdir /var/lib/pgsql/data/indexes
     mkdir /var/lib/pgsql/data/indexes/fedora
     mkdir /var/lib/pgsql/data/indexes/triples
     mkdir /var/lib/pgsql/data/indexes/escidoc-core-large
     mkdir /var/lib/pgsql/data/indexes/escidoc-core-normal
     mkdir /var/lib/pgsql/data/indexes/statistics
     cd /var/lib/pgsql/data/
     chown -R postgres:postgres *
    • run the script (as postgres user) (create new tablespaces)
  • Fedora database
    • run the scripts (as postgres user) (re-set fedora tables tablespace) (re-set fedora indexes tablespace)

  • RiTriples database
    • run the scripts (as postgres user)
    • NOTE: Recommended is to re-run the select queries in these scripts and to create additional scripts.
    • Data in triples are changed daily, so are the tables (in case there are new data or predicates).
    • this has to be done AFTER Fedora resource index is rebuilt. (re-set triples tables tablespace) (re-set triples indexes tablespace)

Migration of the escidoc-core database[edit]

  • ensure there is enough disk space, because the db-migration ant task requires far too much of it!
  • backup the escidoc-core database on !!!
    • log-in as postgres-user to the system
    • issue
       pg_dump -f escidoc_core_database.dmp -C escidoc-core

in a desired backup directory, you need to know the password of user postgres in the pg database

  • The database migration uses certain fingerprints in order to determine the current version of the db. The layout of our databases sometimes differs from the one assumed by the db-migration tool.
  • To check if the database layout is fine
    • go to escidoc-admin tool
    • getrepository info
    • usually (except for very old versions of escidoc-admin tool) there is an information whether the database is consistent or not
  • IF the database is not consistent run the following script prior to the db-migration:
  • Modify
    • Check the following entries:
    • datasource.script.prefix=
    • datasource.url=check-which-one
    • datasource.driverClassName=org.postgresql.Driver
    • datasource.username=check-which-one
    • datasource.password=check-which-one
    • creator_id=the user-id of a sysadmin user (e.g. escidoc:user42 for roland)
    • persistentHandle=the handle taken from aa.user_login_data
    • escidoc.database.tablespace.list=pg_default
  • go to admin-tool
  • Get repository info (shall tell if the fingerscript is different)
    • DO NOT RUN db-migration unless database is consistent!!!
      • if this is the case, compare the fingerprint script of 1.1.0 with the fingerprint script /tmp/fingerprint.xml
      • this should give an idea what to modify manually
      • repeat get-repository-info as long as consistency is achieved
  • call ant db-migration
    • NOTE: for the research data repository, prior to calling ant-db-migration please drop table aa.user_account_backup. All the rest is prepared. --Natasa 12:20, 17 September 2010 (UTC)
    • DO NOT USE THE JAR FILE with the db-migration argument!

JBoss patch[edit]

  • backup
  • copy everything inside jboss-patch-1.2/server to /usr/share/jboss on, ignore jboss-patch-1.2/bin
  • remove old file wstx-asl-3.2.4.jar from <JBOSS_HOME>/server/default/lib.
  • Change properties in according to backup

Post-Migration steps[edit]

  • shutdown core-service
  • shutdown fedora
  • Postgresql shall run
  • vacuum analyze all tables in escidoc-core, riTriples, Fedora database (all as postgres user)
       helper: issue following sql to get the script (on each of the mentioned databases):
       select 'vacuum analyze '||table_schema||'.'||table_name||'; ' from information_schema.tables where table_schema in ('list', 'aa', 'public')
       copy/paste result in a new PgAdmin SQL Editor
       replace all " with nothing (Edit->Find and Replace->Find what " -> Replace All) 
       run all commands together as PgScript (or use F6)
  • change the tablespaces in escidoc-core database
      run the script (as postgres user)
      re-set escidoc-core tables tablespace
  • drop indexes (chance for faster recache), create some aa.indexes, list.filter indexes
      run the script (as postgres user)
      postmigration-pre-recache script

escidoc-core deployment[edit]

  • copy the escidoc-core-bin-1.2/ear/escidoc-core.ear file into /usr/share/jboss/server/default/deploy on

Recache and reindex[edit]

  • Change Property for Lucene Indexing stylesheet in
gsearch.escidoc.indexingStylesheet =
  • start JBoss on
  • start Fedora on
  • launch the escidoc-core admin tool at
  • recache (ensure the "clear cache" checkbox is selected!) (MUST BE RUN FIRST, see afterwards POST-RECACHE below)
  • reindex (ensure the "clear index" checkbox is selected!) (MUST BE RUN SECOND, as it reads the SOAP representation from the cache)


  • NOTE: POST-recache procedure can run in parallel with "reindexing" from eSciDoc Admin (DONE on coreservice pubman)
  • create new indexes on
      run the script (as postgres user on psql from command line)
      set_list_property_indexes script

  • re-vacuum analyze databases escidoc-core, riTriples, Fedora database
            vacuum analyze
  • run the script with changed policies (as postgres user)
               change of policies

  • (optional) see Performance
    • NOTE: the escidoc-core database on production is prepared, no need to change anything, use these tipps for the other databases

deployment of solutions[edit]

  • pror to the deployment of Faces and Virr
    • the Java version on and on has to be upgarded to version 1.6
  • deploy the appropriate solution ear files to pubman, faces and virr
  • apply necessary updates to the solution specific properties files
  • start the Jboss instances for the three solutions