Running VuFind

MPDL,Admin

= Installation =

 inst_script=${inst_dir}/vufind.sh   vufind_instances="vf001 vf002 vf003"   VUFIND_HOME=`readlink -f \`dirname $0\`` export VUFIND_HOME
 * service path /srv/web/vufind
 * service data /srv/web/vufind/svn_checkout/
 * other service-related directories
 * /srv/web/vufind/bin - shell scripts for all instances (might be replaced by, for example, /usr/local/bin/vufind)
 * /srv/web/vufind/etc - MPG IP list, config files (files might be moved to /etc, /etc/vufind)
 * /srv/web/vufind/logs - store anonymized log data
 * /srv/web/vufind/www - Apache doc roots of single or multiple instances (note that /srv/www/htdocs is not an option)
 * instances /srv/web/vufind/inst/ - example for : vf003 (MPG.ReNa, production)
 * start /etc/init.d/vufind [ ]
 * -> uses /vufind.sh see /etc/init.d/vufind
 * config /etc/sysconfig/vufind
 * note: if is not specified, instances listed in /etc/sysconfig/vufind are started, for example
 * /vufind.sysconf is called by /vufind.sh and creates environment (note: vufind.sysconf might be symlinked to generic copy because there are no instance-specific directives)

VUFIND_INST_NAME=`basename $VUFIND_HOME` VUFIND_INST_ID=`echo $VUFIND_INST_NAME | sed -e 's/^.*[^0-9]\([0-9][0-9]*\)/\1/g'`

JETTY_RUN=$VUFIND_HOME export JETTY_RUN

JETTY_CONSOLE=${VUFIND_HOME}/vufind.log export JETTY_CONSOLE

JETTY_PORT=`expr 8080 + $VUFIND_INST_ID` export JETTY_PORT   root    21888  0.0 19.8 1513996 419024 ? Sl  Feb25  20:37 /usr/lib64/jvm/java/bin/java -server -Xms1024m -Xmx1024m -XX:+UseParallelGC -XX:NewRatio=5 -Djetty.port=8081 -Dsolr.solr.home=/srv/web/vufind/inst/vf001/solr -Djetty.logs=/srv/web/vufind/inst/vf001/solr/jetty/logs -Djetty.home=/srv/web/vufind/inst/vf001/solr/jetty -Djava.util.logging.config.file=/srv/web/vufind/inst/vf001/solr/jetty/etc/logging.properties -jar /srv/web/vufind/inst/vf001/solr/jetty/start.jar /srv/web/vufind/inst/vf001/solr/jetty/etc/jetty.xml     meaning that http://localhost: /solr may be accessed from localhost only (i.e. is available to VuFind)
 * example of running service
 * note: jetty.host is presently hardcoded in solr/jetty/etc/jetty.xml

= Instance =

Multiple instances supported

Instances may be started independently, see Installation - start

Components inst/ /.svn_checkout -> ../../svn_checkout  ... /var/log/apache2/rena-access.log { daily dateext rotate 90 compress delaycompress postrotate /etc/init.d/apache2 reload /srv/web/vufind/bin/anonymize.sh rena-access.log config_vufind_access.py   endscript } ... </PRE> Alias / /srv/web/vufind/inst/ /web <Directory /srv/web/vufind/inst/ /web/> Options FollowSymLinks All Options -Indexes AllowOverride All Order Allow,Deny Allow from all ... <Location ~ "/[Vv][Uu][Ff][Ii][Nn][Dd]/[Aa][Dd][Mm][Ii][Nn]/.+"> AuthUserFile /srv/web/vufind/inst/vf003/web/services/Admin/.htpasswd AuthName "VuFind Administration" AuthType Basic require valid-user </Location> </PRE>  mysql   21367  0.0  2.0 400112 44324 pts/0    Sl   11:19   0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/srv/mysql/vufind --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysql/mysqld.log --pid-file=/srv/mysql/vufind/mysql.pid --socket=/srv/mysql/vufind/mysql.sock --port=3307 </PRE>  [Site] path           = / url            = http:// / (see httpd-vufind.conf) local          = /srv/web/vufind/inst/ /web ... admin_enabled  = false
 * symlink to service data in svn working copy
 * Apache, general
 * presently no caching (will be introduced asap)
 * Apache, rotation of logfiles and anonymization
 * logrotate script /etc/logrotate.d/apache
 * logfiles vufind-*.log, rena-*.log, separate entry for each logfile
 * anonymize.sh
 * appends current date to logfile name passed
 * if gzipped logfile does not exist in log data directory /srv/web/vufind/logs
 * anonymizes current logfile in /var/log/apache2 by calling anonymizer.py (symlink in /srv/web/vufind/bin), see anonymizer
 * copies anonymized logfile to log data directory, gzips logfile
 * creates protocols /srv/web/vufind/bin/anonymize/anon-*.log
 * protocols are sent by email once a day, see Cronjobs
 * notes
 * current logfile must not be compressed on rotate because anonymizing a compressed logfile .gz in place may corrupt logdata, see anonymizer/readme.txt, known issues
 * rotate 90 might be reduced
 * 1) !! Please retain option delaycompress !! ##
 * Apache config /etc/apache2/vhosts.d/rena.conf (MPG.ReNa production)
 * AddDefaultCharset utf-8
 * AddType application/rss+xml .rss
 * DocumentRoot /srv/web/vufind/www/rena
 * includes single httpd-vufind.conf Include /srv/web/vufind/inst/vf003/httpd-vufind.conf</PRE>
 * single instance vf003 RedirectMatch ^/$ http://rena.mpdl.mpg.de/rena</PRE>
 * Apache config /etc/apache2/vhosts.d/vufind.conf (MPG.ReNa Demo, other VuFind instances - note: if required, an instance may have its own VirtualHost configuration, doc root, etc)
 * AddDefaultCharset utf-8
 * AddType application/rss+xml .rss
 * DocumentRoot /srv/web/vufind/www/vufind
 * includes multiple /httpd-vufind.conf Include /srv/web/vufind/inst/ /httpd-vufind.conf</PRE>
 * default instance vf001 RedirectMatch ^/$ http://vufind-demo.mpdl.mpg.de/vufind</PRE>
 * Apache config VuFind /httpd-vufind.conf
 * access control for /srv/web/vufind/inst/ /web is done here
 * note: !! Be careful !! default admin pw not changed, admin access disabled at VuFind level, see /web/conf/config.ini - admin_enabled = false
 * PHP 5.3.17 (known to be working with...), Smarty, PEAR, etc.
 * Solr (Lucene) 3.5 /solr - Index at solr/biblio/index, connected to VuFind via http, test curl http://localhost: /solr/biblio/select?q=ID%3AERS000000001</PRE>
 * Jetty, jetty-6.1.11, see solr/jetty, Port 8080 + inst ID - example: vf001, Port 8081 - Monitor in place, on MPG.ReNa see solrStatusMPDL.php, code - output "ONLINE" 200 OK, "OFFLINE" 500 Internal Server Error if Jetty/Solr is unavailable, otherwise "ERROR: ..." 500 Internal Server Error.
 * MySQL (PHP sessions, user data, tags, etc. - data is actually stored in Solr index) - mysql process, example
 * SolrMarc, config see /import/import.properties
 * VuFind config /web/conf/config.ini - settings

[Index] engine         = Solr url            = http://localhost: /solr local          = /srv/web/vufind/inst/ /solr default_core   = biblio

[Database] database         = mysql:// : @127.0.0.1:3307/ schema_location  = /srv/web/vufind/inst/ /web/conf class_location   = /srv/web/vufind/inst/ /web/services/MyResearch/lib

[Logging] ... file          = /var/log/vufind/ /messages.log:alert,error,notice,debug ...


 * MPDL sections ;;;;

[MARCData] marcUrl = http(s)://.../<MARC file> ... [PredefinedSets] predefWs = http(s)://.../predef.cgi ... </PRE>
 * MARC data
 * Predefined Sets

See config.ini_LOCAL_DO_NOT_USE for customized settings.

= Services =
 * vf001: demo instance of MPG.ReNa, http://vufind-demo.mpdl.mpg.de (Apache rewrite -> http://vufind-demo.mpdl.mpg.de/vufind see Apache, default instance) - MUST be available
 * vf002: temporary instance of VuFind - data is indexed sporadically, displayed, discussed, etc. - http://vufind-demo.mpdl.mpg.de/vufindtmp - SHOULD be available
 * vf003: production instance of MPG.ReNa, http://rena.mpdl.mpg.de (Apache rewrite -> http://rena.mpdl.mpg.de/rena see Apache, default instance) - MUST be available
 * vf099: test instance - to test whether multiple instance setup is working properly, and instances may be easily cloned - NOT USED

Current auto-start of instances see /etc/sysconfig/vufind vufind_instances="vf001 vf002 vf003"</PRE>

DEFECT environment does not seem to be created properly on auto-start, i.e. though VuFind is up and running, JETTY is not. If so, http://rena.mpdl.mpg.de etc will issue error "Connection refused". To get content displayed, please login as root, or see vLib trac, and restart VuFind manually  [> sudo -i] </PRE> SOLVED, auto-start of services tested successfully (08/12/2014).
 * 1) /etc/init.d/vufind restart

= Cronjobs = /etc/cron.d/vufind  00 05 * * * root /srv/web/vufind/bin/IpLoader.sh 00 09 * * * root /srv/web/vufind/bin/email.sh "VuFind anonymize.sh" /srv/web/vufind/bin/anonymize/ 05 09 * * * root /srv/web/vufind/bin/ip_download.sh 00 06 * * 1-5 root /srv/web/vufind/inst/vf001/update-files-mpdl.sh 05 06 * * 1-5 root /srv/web/vufind/inst/vf001/update-index-mpdl.sh 10 06 * * 1-5 root /srv/web/vufind/inst/vf003/update-files-mpdl.sh 15 06 * * 1-5 root /srv/web/vufind/inst/vf003/update-index-mpdl.sh 20 06 * * 1-5 root /srv/web/vufind/inst/vf001/update-synd-mpdl.sh /srv/web/vufind/www/vufind/rss 25 06 * * 1-5 root /srv/web/vufind/inst/vf003/update-synd-mpdl.sh /srv/web/vufind/www/rena/rss 30 06 * * 1-5 root /srv/web/vufind/inst/vf001/update-anchors-mpdl.sh 35 06 * * 1-5 root /srv/web/vufind/inst/vf003/update-anchors-mpdl.sh 00 09 * * * root /srv/web/vufind/bin/OAGoldList.sh </PRE>
 * 1) update IpLoader output formats
 * 1) send Apache log anonymization info by email
 * 1) download current IP list
 * 1) update data & files from aleph
 * 1) update resource feeds
 * 1) update EZB anchors
 * 1) OAGoldList update

Code see update-files-mpdl.sh, update-index-mpdl.sh, update-synd-mpdl.sh

= Logfiles =

 [Site] ... timezone      = "Europe/Berlin" ... [Logging] ... file          = /var/log/vufind/ /messages.log:alert,error,notice,debug </PRE>  JAVA_OPTIONS="... -Djava.util.logging.config.file=$VUFIND_HOME/solr/jetty/etc/logging.properties " </PRE> logging.properties  .level = INFO
 * Apache see /etc/apache2/vhosts.d
 * VuFind see config.ini
 * Jetty request - default
 * Jetty console output, logging
 * console output JETTY_CONSOLE=VUFIND_HOME/vufind.log
 * logging see solr/jetty/etc/logging.properties - note: problem with processing of JETTY_ARGS=-D... hence appended to JAVA_OPTIONS in vufind.sh
 * 1) Default global logging level:

org.apache.solr.update.processor.LogUpdateProcessor.level = FINE
 * 1) Log every update command (add, delete, commit, ...)

handlers = java.util.logging.FileHandler
 * 1) Where to log (space separated list).

java.util.logging.FileHandler.level = FINE

java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter

java.util.logging.FileHandler.limit = 100000000
 * 1) set to max. 100 MB

java.util.logging.FileHandler.pattern = solr/jetty/logs/solr%u.log
 * 1) Log to the logs directory, with log files named solrxxx.log

java.util.logging.FileHandler.append=true java.util.logging.FileHandler.count = 10 </PRE>
 * 1) append onto existing logfiles, rotate max. 10 times

= Info =


 * VuFind 1.x documentation
 * MPG.ReNa documentation