AWOB Preparation Functionalities

MPDL,GAVO

On this page the usage scenarios for AWOB (Astronomer's Workbench) will be collected.

Summary of long term requirements

 * Install and configure an instance of AWOB for a project’s use
 * Create project pages
 * Create sub projects
 * Register user
 * Assign users to projects with particular (standard) types of privileges
 * Have standard functions available for
 * attaching a relational database to a project
 * visualising standard astronomical data products (Aladin, SpecView, VOPlot, ...)
 * Upload data products and associate metadata	.

Specific use cases
Here we list a number of use cases extracted form real examples of web sites created by scientists to support their work and that of their collaborators. In the next section we will use this to extract 1/2 pilot projects.

(z)COSMOS

 * Contact: Angela Bongiorno (MPE)
 * Description: The COSMOS1 project aims to ... It is divided over a number of “sub-projects”, one of which is zCosmos. zCosmos aims to ...The participants in this project have paid quite some effort on setting up a collaborative environment. There is a central web site for the COSMOS projects with links to sites of the sub-projects (see Figure 4) thumb|Figure 4 Homepage of the COSMOS project.
 * Current Features:
 * specific authorisation per sub-project
 * public archival data available under search service at IRSA
 * some shared data on common site through PI
 * work of individuals on individually managed web sites, notification by email
 * papers as PDFs on sub-project’s area
 * project proposals on (sub-)project (TBD find out) web site
 * The missing components are:
 * central overview/search over all available resources
 * Use cases
 * Setting up overall COSMOS environment (using a special AWOB instance)
 * PI and Co-Is have decided on a project and install an AWOB instance to support it.
 * A customised main project area is defined with links to sub-projects etc.
 * Sub-projects are defined and created below the main project
 * Co-Is, (PIs of the sub-projects) associate users to their sub-projects; these are automatically associated to the main project.
 * Some specialised applications are deployed and made accessible from the main project:
 * search on core data products
 * Defining a science project
 * Angela connects to her sub-project (zCosmos)
 * She defines a new science project there.
 * An area is created for it,
 * Angela describes the goal of the project, gives it a title etc.
 * The project has status: PROJECT PROPOSED
 * Other users associated to the zCosmos sub-project are able to participate in the project, which they do by registering/associating themselves to the project. This participation can not be refused.
 * Discussions are held between the participants, this is supported by a discussion area for the project.
 * Work starts, which moves the science project into a different state (WORK STARTED)
 * Data are produced and uploaded, thumbnails added to the uploaded files, documentation written.
 * Paper is written, PDF versions of it are attached to the project.
 * Required Functionality
 * some hierarchy of projects, with associated privilege inheritance etc should be supported.

Small Magellanic Cloud (SMC)
thumb|Screen shot of querying an external service in SMC.
 * Contact: Frank Haberl (MPE)
 * Description: A facility which manages a source catalog of an ongoing project and provides search capability on it. Since the catalog is still private data only authorized people have access to it. The service also enables for a user to add new results to the catalog. thumb|Screen shot of form based query interface in SMC.
 * Use cases thumb|Screen shot of customize query interface in SMC. thumb|Screen shot of query result page in SMC.
 * Frank creates a source catalog in FITS format, and creates a ASCII file which contains source classification. The ASCII file is updated as new source classifications are available.
 * Combine the FITS binary table and the ASCII table to produce a single table in the database by using C++ code and upload the data into the database by using php script.
 * On top of the database set up, add browse and search capabilities on the tables.
 * User can do search via form based query parameters, or via custom queries written by oneself.
 * For a select source
 * a detailed view page brings up the related images stored in the pre defined local directory. Also in this page one can update metadata both in the table in the db and ASCII file.
 * Link to an external web service for further inquiries.
 * Always maintain the synchronicity between input ASCII file and FITS files and the database.
 * Required Functionality thumb|Screen shot of detail page in SMC.
 * Must keep the synchronicity between the input files (ASCII, FITS) and the database tables.
 * Allow to update contents of ASCII files including comments and references, and the corresponding database table via web browser.
 * Define a database table explicitly from reading a FITS table and an ASCII file.
 * VOTable-like metadata
 * Upload data in a FITS binary table into table(s) in a database.
 * Standard SQL query interface
 * Documentation page for each table, generated from metadata about table
 * Form based query for each table, including sub-selecting columns etc
 * Form based query can be executed immediately, or the generated query can be returned in standard query page for further editing.
 * Enable the query result metadata flexible.
 * Provide name resolver by linking SIMBAD and the result fills up the query form.
 * Provide tools to view images.
 * Provide query services on external web services like VISIER/SIMBAD.
 * Maintain reusable bibliographies in LaTex format.
 * Password protection to the service.

The Lockman Hole multi-wavelength survey project
thumb|Figure 1 Opening page Manolis Rovilos' portal to Lockman hole data products. thumb|Figure 2 Page with links to images from different observatories and link to download FITS file for selected observatory. thumb|Figure 3 Links to source catalogs.
 * Contact: Manolis Rovilos (MPE)
 * Description: Aims to gather data about the “Lockman hole”, an area on the sky with particular low concentrations of galactic (foreground) HI gas, therefore allowing particular deep views of the extra-galactic universe1. This project gathers observations by many instruments in many different wavelength regimes to obtain a multi-wavelength view of this area. Some data from pre-existing archives, some obtained specially for this project [TDB check whether this is true!]. Manolis has created a simple web portal (see Figure 1) to collections of data products. The site requires authentication.
 * Use cases: Maintenance shared data site.
 * Manolis create a new project and calls it “Lockman Hole”.
 * He creates various pages, similar to the ones shown in the description above.
 * He uploads files of various types:
 * FITS files
 * tabular catalogs
 * images to serve as illustrations to these.
 * He creates links for simple download of the data, but also, for the FITS files, links that will open the file in a visualiser applet, for example Aladin for images and SpecView for spectra.
 * The tabular files are loaded in a relational database and standard queries are enabled.
 * Other users of the site are added to the project upon request.
 * Required Functionality
 * Definition project, not necessarily for a paper.
 * Assigning privileges to users for your project.
 * Upload of data of various types (images, tables) and associated functions for accessing these (visualisation, database query, 2D-3D plots)

SAM comparison project

 * Contact: Mike Boylan-Khinchin (MPA)
 * Use cases
 * Mike logs in to AWOB
 * He has the appropriate rights and creates a new project, which he calls “SAM Comparison”
 * The project must be connected to a “data-database” (i.e. a separate database where users can upload data products and which can be queried).
 * Mike creates a table called FOFTrees in the database, uploads a CSV file to the system and inserts its contents (in bulk mode) into the table.
 * Mike writes documentation pages
 * In the mean time Mike’s collaborators have registered themselves with AWOB and Mike finds their user credentials and adds them to his project. They get all privileges (apart from ownership/admin rights …?) that Mike also has.
 * The users query the FOFTable and retrieve the result in their favourite format (from the supported ones: CSV, BSV, VOTable, …).
 * They analyse the data, run their own SAM, upload results to project database.
 * Required Functionality
 * User Registration
 * Project creation
 * Database management and association to project
 * User association+rights/privilege allocation
 * Add personal page for associated user (by default?)
 * File upload/attachment to project, and association of known format (possibly other metadata) to file, so that tools can be used to manipulate the file.
 * Loading tabular (CSV, VOTable, ..) file into relational database
 * Querying database and retrieval of results in desired format.
 * Authoring of web pages in system.

MAXI (Multivarite Archive of X-ray Images)

 * Contact: Alexis Finoguenov (MPE)
 * Description: Upload a science product into the database, and publish it via web service. The science product is the analysis result of an observation of an X-ray cluster, and the results are multidimensional matrix of multiple derived physical properties of the regions which represent spatial divisions of a cluster. Each physical property could be represented as an  image.
 * Use cases
 * From the FITS file with region definition and ASCII files containing the scientific results create database to store all the necessary data.
 * Build a web service which list all the clusters published, each of which has an individual page including a table of the properties and links to related images.
 * In each page one can download the results in FITS format as well as view them via Aladin.
 * Required Functionality
 * Self-upload request
 * Each observation could have a different set of physical properties to be loaded.
 * The request requires database operations including creating, and loading tables as well as storing metadata of each table created by the request.
 * May require to store images in the database.
 * Load multiples images of selected properties of clusters in a single instance of Aladin applet for easy comparison between images.
 * Upon request generate data of a selected property in a FITS binary table on-the-fly and be available.

Jens

 * WIKI
 * FITS ingestor+SQL querying on FITS database
 * database for derived data: upload source catalogue
 * cross match to external catalogues

Student supervision
(following discussion with Marat Gilfanov)

Interesting selling point: Usage by supervisors and students.

Summary of specific use cases
Here we list types of functionality that have been used in the above projects and eventually could(should?) make it into AWOB. This is not our proposal for the pilots, those follow in the next section.

Project Management

 * create projects
 * assign users to projects with particular privileges
 * sub-projects (e.g. Cosmos->zCosmos->science projects)
 * (for science projects only?) state management
 * each user gets personal workspace, "private project"

"Wiki" functionality

 * Allow creation/editing of documentation pages "fully" as in wiki
 * mailing lists
 * workspaces
 * simplefile attachment ...

Smart file attachment & editor association
interpret uploaded files and "do something more with them"
 * "FITS ingestor": keyword extraction and loading into RDB
 * FITS viewer including header and images (eg.,, ), FITS images: viewer (Aladin, ...)
 * zip/tar/...: extract and associate files (MAXI)
 * pdf,word/doc,postscript,plain text, ...
 * Tex, LaTex

Relational database usage

 * table creation through form; metadata association, UCDs for example
 * tabular file (csv,bsv,tsv,FITS binary table, VOTable,...) upload into table (possibly add table structure inference from files (maybe can use STIL)
 * Query, use IVOA's TAP/ADQL as template for support
 * SQL (includes view generation, requires (for metadata inference) an SQL interpreter. Are being built in IVOA TAP/ADQL efforts).
 * Form based (include possibility to only generate SQL form form, allow modification, see SMC)
 * Allow generation of query form from table/view

Usage of external services

 * Simbad: object-name -> position-on-sky resolver
 * Vizier: query of astronomical catalogues and tables
 * SDSS SkyServer: interface to SDSS results, images, spectra, catalogues
 * ADS: Astronomical publications service
 * Los Alamos preprint service
 * (I)VO(A)
 * registries
 * tools
 * services

Publication

 * to IVOA: implementing standards, SIA, SSA, SCS, TAP
 * to MPG: long-term archiving, persistent IDs?
 * to world: as custom web site

System requirements
List of functionalities without very specific ideas.
 * logging
 * history
 * ability to add custom plugins/application modules
 * authentication
 * backups
 * versioning
 * messaging/email notification to collaborators if new data are available.

PP 1
We realise that this list or requirements may be more than can be treated in 1 pilot phase. We have tried ordering the requirements in block of related functionality, where we think the first 3 blocks should be done if possible, and later blocks depend on the availability of earlier ones (not necessarily all).

Description

 * Build based on Small Magellanic Cloud(SMC) use case: The reason to choose this case is that it is a common practice for observational astronomers to create a source catalog for on going project, and keep adding and updating the catalog as new results are available.

Requirements
1. Minimal Installation and Project Management Requirements thumb|Figure 1 Schematic drawing of project overview page.
 * 1) Able to install AWOB in our system so that we can do development.
 * 2) Sysdmin should be able to define users with certain privileges
 * 3) System should be login protected, on per project basis
 * 4) To be discussed: in Pilot can we assume 1 AWOB installation per project?

2. Minimal Wiki(-like) Functionality thumb|Schematic drawing of attach file page.
 * 1) Attach File (+thumbnail if applicable)
 * 2) Describe File:
 * 3) English text
 * 4) Relavant metadata eg., file type (csv, tsv, fixed length, votable, FITS)
 * 5) List of attached files (with thumbnails)
 * 6) Download selected file

3. Minimal RDB Functionalities thumb|Schematic diagram of create user page. thumb|Schematic diagram of database management page.
 * 1) Possible to associate a database to a project
 * 2) Users can be assinged database roles.
 * 3) RDB Management:
 * 4) Create Table (from ddl, from votable, from GUI, from FITS)
 * 5) Describe table
 * 6) Load Data from attached files
 * 7) View schema
 * 8) SQL interface :
 * 9) Depending on the role allow: query, insert, update, delete, create view, create index.
 * 10) select format of result (CSV, VOTable, HTML, LaTeX, FITS)

4. Form based Query Interface
 * 1) In general
 * 2) form allows querying on a single table (or view): no join
 * 3) SELECT: subset of columns (standard multi-select list, see SMC eg)
 * 4) FROM: fixed for table (or view)
 * 5) WHERE: allow selection on subset of all columns, HTML components to add values based on data type of column.
 * 6) User can design this form:
 * 7) select table
 * 8) select subset of columns that can show up in WHERE clause
 * 9) Issue: layout of form. (wiki? AJAX?)
 * 10) We generate the SQL from the parameters
 * 11) show SQL for further refinements
 * 12) execute SQL

5. FITS management
 * 1) view key words
 * 2) open image viewer (Aladin, other?)
 * 3) open spectrum viewer (SpecView, SPLAT, other?)
 * 4) FITS ingestor

6. External Services
 * 1) Link to SIMBAD/NAD/CDS for name resolving or more details
 * 2) Serve as VOPortal (i.e. query Registries, find DAL services, execute them, download results ...)