Migration pubman70

MPDL

Angaben von Willi
Angaben von FIZ

1.2 to 1.3 upgrade guide


 * transition towards QA
 * due to JBoss isolation problems with eSciDoc 1.3.2, qa-pubman will remain solely within its own JBoss instance.
 * qa-coreservice has to be migrated to eSciDoc 1.3.2
 * Java JDK (SUN !!!) > 1.6.0
 * upgrade Postgres > 8.4
 * upgrade Fedora to 3.4.2
 * apply most recent JBoss patch
 * migrate escidoc-core database
 * deploy escidoc-core.ear version 1.3.2
 * deploy AdminTool version 1.1.1

LVM für backup erstellen
(namenskonvention für logical volumes VVnnDDDx wo VV = vm, nn = vm-nummer DDD = physikalischen laufwerk, x = fortlaufender nummer z.B vm13sda3 = dritter logical volume auf vm13).
 * yast
 * partitionieren
 * + bei volume-verwaltung und l1/l2/l3 (logical volume group nicht mit namenskonvention definiert)


 * über "hinzufügen" + name auf logical volume group verfügbarer kapazität abfragen.

mkdir /data/backup mount /dev/xvdd /data/backup/

fedora
derzeit INFO 2011-10-05 13:16:36.906 [main] (Server) Server home is /opt/fedora/server INFO 2011-10-05 13:16:36.954 [main] (BasicServer) Fedora Version: 3.1

du -sh auf srv01 machen, nicht von vm05 (schlechte performance von nfs)

srv01:5 15:36:27 /X/vm05_fedora # ls client  data  docs  install  server  target  tomcat srv01:5 15:36:28 /X/vm05_fedora # du -sh * 30M	client 143G	data 38M	docs 100M	install 7.0M	server 4.0K	target 124M	tomcat

jboss
/data/jboss-4.2.2.GA > du -sh 1.6G	.

wovon hauptsächlich lucene indices

/data/jboss-4.2.2.GA > du -sh server/default/data/index/ 1.1G	server/default/data/index/

postgres
vm05:4 15:41:40 /data/pgsql/data # du -sh * 11G	base 400K	global 136K	pg_clog 4.0K	pg_hba.conf 4.0K	pg_ident.conf 68K	pg_log 28K	pg_multixact 152K	pg_subtrans 4.0K	pg_tblspc 4.0K	pg_twophase 4.0K	PG_VERSION 129M	pg_xlog 20K	postgresql.conf 4.0K	postmaster.opts 4.0K	postmaster.pid 683M	tablespaces

Postgres

 * dumpen von laufenden Postgres in backup verzeichnis, mit user postgres, mit schreibrecht im Verzeichnis
 * pg_dumpall -f vm05_escidoc122_migration.sql (auf qa: 6 minuten)

postgres runterfahren

(als root) rcpostgresql stop

für rollback-prozedur auch Daten-sichern.

# mkdir /data/backup/postgres_rollback_data

Jboss
jboss runterfahren (rcjboss stop)

fedora runterfahren (/opt/fedora/tomcat/bin/shutdown.sh)

/data/backup # mkdir jboss_backup

rsync -ahv ../../jboss-4.2.2.GA/ /data/backup/jboss_backup/

dauer (auf vm05 z. 2 minuten)

fedora
auf srv01 sicherung ausserhalb baum machen srv01:5 16:10:37 /X # pwd /X # mkdir vm05_escidoc122_backup_fedora # rsync -ahv /X/vm05_fedora/ /X/vm05_escidoc122_backup_fedora/

dauer (auf srv01) 144GB z.140 minuten

rollback procedure
auf vm05 nfs mount von /data von srv01.mpdl.mpg.de:/X/vm05_fedora

auf srv01.mpdl.mpg.de:/X/vm05_escidoc122_backup_fedora

PostgreSQL Upgrade

 * deinstallieren bzw. upgraden von postgreSQL_server, contrib, basic_client, doc und Abhängigkeiten mit yast2 (nicht yast).
 * data-verzeichnis plattmachen
 * /etc/init.d/rcpostgresql start erzeugt die postgres und template0 und template1 dbs.
 * Verzeichnis für Tablespaces anlegen und Rechte vergeben
 * (mkdir /data/pgsql/data/tablespaces)
 * (chown postgres:postgres /data/pgsql/data/tablespaces)
 * dump-datei wieder laden als Postgres (psql -f /data/backup/postgres_dump/vm05_escidoc122_migration.sql) template1

Dauer auf vm05 (11GB) 20 Minuten)

/data/backup/postgres_rollback_data # cp pg_hba.conf ../../pgsql/data/pg_hba.conf


 * danach method 'ident sameuser' in 'ident' ändern

postgresql.conf manuell ändern

listen_address = 'localhost,xxx.xxx.xxx.xxx'

shared_buffers = 48MB

temp_buffers = 16MB

work_mem = 32MB

wal_buffers = 8MB

effective_cache_size = 532MB

autovacuum = on

autovacuum_vacuum_threshold = 5000

autovacuum_analyze_threshold = 5000

autovacuum_vacuum_scale_factor = 0.0

autovacuum_analyze_scale_factor = 0.0

datestyle = 'iso, dmy'

fedora upgrade
fedora upgrade guide

FIZ Installation walkthrough (for fedora 3.1)

fedora install for 342

vorher in  CREATE DATABASE "fedora3" WITH ENCODING='UTF8' OWNER="fedoraAdmin"; CREATE DATABASE "riTriples" WITH ENCODING='SQL_ASCII' OWNER="fedoraAdmin" TEMPLATE=template0;
 * pgadmin3 / psql ritriples DB droppen und fedora3 erstellen

mkdir /opt/fedora342 <--- achtung, lokaler festplatte

wget http://downloads.sourceforge.net/fedora-commons/fcrepo-installer-3.4.2.jar

wget http://www.escidoc-project.de/software/fedora/mptstore-0.9.4.jar

wget https://www.escidoc.org/artifactory/libs-releases-local/de/escidoc/admintool/AdminTool/1.1.1/AdminTool-1.1.1.war

wget http://www.escidoc.org/software/builds/stable-releases/1.3.2/escidoc-core-1.3.2-bin.zip (and extract.

copy

admintool.war, escidoc-core-1.3.2-build122.ear to $JBOSS_HOME/server/default/deploy

copy mptstore-0.9.4.jar to $FEDORA_HOME/tomcat/tomcat/webapps/fedora/WEB-INF/lib

export FEDORA_HOME=/opt/fedora342 export CATALINA_HOME=/opt/fedora342/

java -jar fcrepo-installer-3.4.2.jar


 * select 'custom' installer
 * overwrite
 * fedoraAdmin=fedoraAdmin
 * fedora-server-host : qa-coreservice.mpdl.mpg.de
 * namespace /fedora (default)
 * jdbc url jdbc:postgresql://localhost/fedora3
 * feSL Authn: false
 * fesl authz: false
 * low-level-storage: legacy-fs
 * enable resource index: true <-- creates ritriples
 * enable messaging: false

laufen lassen, erzeugt tomcat, fedora.war, usw.


 * /opt/fedora342/server/config/fedora.fcfg editieren. von

 The root directory for the internal storage of Fedora objects. This value should be adjusted based on your installation environment. This value should not point to the same location as                       datastream_store_base. 

auf

 The root directory for the internal storage of Fedora objects. This value should be adjusted based on your installation environment. This value should not point to the same location as                       datastream_store_base. 

/opt/fedora342/data/fedora-xacml-policies/repository-policies/default # wget http://www.escidoc-project.de/software/fedora/deny-everything-if-not-administrator.xml

/opt/fedora342/tomcat/bin # ./startup.sh

/opt/fedora342/tomcat/bin # ./shutdown.sh <-- erstmalige erstellung des repositories.

/opt/fedora342/server/bin # ./fedora-rebuild.sh

zuerst option 2) rebuild sql database Anfang 14:25 ingest-dauer 4 sek pro 100 objecte ende 14:55

SUCCESS: 58927 objects rebuilt.

danach option 1) rebuild the resource index. anfang 14:55 15:05

apply most recent JBoss patch
from fiz download page


 * runterladen mit wget, unzip,


 * /tmp/jboss/jboss-patch-1.3.2 # cp -Rf /usr/share/jboss

Migrate the escidoc database
download tool from download for release 1.3 wget https://www.escidoc.org/artifactory/libs-releases-local/org/escidoc/core/migration-tool/1.3.2/migration-tool-1.3.2.zip

unzip

as described at FIZ


 * modify admin-tool.properties (no modifications were necessary on qa-coreservice.

in unzipped main directory run:

ant db-migration

This fails. Correct as below and rerun until the db-migration works. Duration on qa-coreservice: 5 minutes

Procedure to correct the db-structure
Background: the escidoc database migration utility contains a (sensible) check, that the db-structure of the existing database is as expected by the migration tool. On qa-coreservice and live coreservice, this check will fail, due to the presence of various unexpected a) indices, b) tablespaces c) primary key constraints

the current db-structure (from the point of view of the migration tool) is dumped to /tmp/fingerprint.xml. This is compared with the required structure in escidoc-core.ear 01_common.jar//META-INF/db/fingerprints/PostgreSQL/1.2.0.xml.


 * compare the files with a graphical diff editor, manually resolve any differences.
 * manually resolve any differences with pgAdmin / psql scripts Notes on this
 * tablespaces: Are probably not visible / relevant to the migration tool, but are relevant to the psql script. Use SET default_tablespace = '';, and similar
 * indices. Need to be created as required by the fingerprint.xml
 * primary key constraints: on adm.version in qa-coreservice. Dropped.
 * useful tip: use exec pgscript in PGAdmin3 to ensure script continues after errors.

correct bug in migration tool as documented by FIZ
run the update statements in pgadmin

Migrating the lucene indexes
vi /data/jboss-4.2.2.GA/server/default/conf/search/config/index/escidoc_all/index.properties change fgsindex.defaultUpdateIndexDocXslt             = http://coreservice.mpdl.mpg.de/mpdlEscidocXmlToLucene
 * add the customised indexing stylesheet to the lucene properties

In adminconsole1.1.1 rebuild indexes. Dauer: in etwa 1500 Items pro Stunde

Open Points on QA-Coreservice

 * 1) set the fedora environment system-wide to /opt/fedora342 erledigt--Bourke 11:15, 18 October 2011 (CEST)
 * 2) change the jboss-internal datastorage from hsqldb to postgresql Change_Hypersonic_DB_on_Jboss_to_another_DB
 * 3) Do we have a jdbc driver for postgresql 9.x ? Answer: no, but it shouldn't matter. Currently installed on qa-coreserive is jdbc driver version 8.1.63.17.1. But almost all the newer jdbc driver features listed at the postgresql site are just implementing jdbc3 or jdbc4 features, which we don't use anyway