Throw down the SWORD

With the Orbital project at its end, and plans for a University research information / research data service afoot, I’m reviewing the excellent work carried out by our (now-departed) developers Harry Newton and Nick Jackson – work which linked up CKAN, the Orbital ‘bridge’ application, and the Lincoln Repository (EPrints) using SWORD – described in earlier blog posts here and here.

“One important piece of work that we’re undertaking at the moment in Orbital is the facility to deposit the existence of a dataset, from CKAN and the University’s new Awards Management System (AMS), into our (EPrints) Repository via SWORD – at the same time requesting a DOI for the dataset via theDataCite API. The software at the centre of this operation is what we refer to as Orbital Bridge.”

This deposit workflow is now broadly working as it should – I think only a few tweaks would be necessary now to turn this into a working tool for the University of Lincoln.

Most urgent is the need for the University to sign up with the DataCite DOI service, which would secure a DOI for each dataset record deposited from CKAN and hence formally published by the University. This subscription should form part of the new research information service.

The underlying code could be used for other SWORD-enabled deposit from sources of metadata (e.g. the Library’s discovery system, Find it at Lincoln), to the Lincoln Repository as the University’s bibliographic ‘system of record’.

Warning: this is an extremely screenshot-heavy blog post! Click on any one of the screenshots below to view a larger image.

Here’s a step-by-step walkthrough of the entire process of adding a dataset to CKAN, and depositing it as a record in the Lincoln Repository.

  1. Go to the Researcher Dashboard at: https://orbital.lincoln.ac.uk/ and click on “Sign In”.
    Screenshot from the Researcher Dashboard
  2. Enter your staff accountID and password to sign in to the Researcher Dashboard.
    Screenshot from the Researcher Dashboard
  3. Once you have been signed in and returned to the Researcher Dashboard, click on your name (in the top right-hand corner) and then click on “My Projects”.
    Screenshot from the Researcher Dashboard
  4. You will see an overview of your research projects – both funded projects (derived from the AMS), and unfunded projects you have added locally. Click on the name of the project you want to add data to.
    Screenshot from the Researcher Dashboard
  5. You will be taken to a page for that research project. On the right-hand side of this page, under the heading “Options”, click on “Create Research Data Environment”.
    Screenshot from the Researcher DashboardImage7
  6. You will be taken to the University’s CKAN research data platform, where a page/group will have been created which corresponds to your project in the Researcher Dashboard. Sign in to CKAN using your staff accountID (there is currently no single sign-on between the Researcher Dashboard and CKAN) and password and you should be returned to the same page. However you will probably be sent instead to the CKAN home page, in which case you will have to look again for your project under the “Groups” menu.
    Screenshot from CKAN
  7. Toward the top of the project screen in CKAN, click on “Add Dataset” > “New Dataset…”.
    Screenshot from CKAN
  8. Fill in the form with information about the overall dataset, including the following fields:
    • Title
    • URL
    • License (N.B. US spelling!)
    • Description
      Screenshot from CKAN
  9. Then click on “Add Dataset”
    Screenshot from CKAN
  10. If you now click on “Further information” tab on the left-hand menu, you can add the following additional information about the dataset (this is not obvious from the initial dataset form):
    • Author
    • Author email
    • Maintainer
    • Maintainer email
    • Version
    • Summary [of changes]
      Screenshot from CKAN
  11. To attach individual data document(s)—which CKAN refers to as “resources”—to the dataset, scroll down the page and click on “Upload a file” (there are other options) > “Choose file” > “Upload”.
    Screenshot from CKAN
  12. Then fill in the form with the following basic information about the “resource”:
    • Name
    • Description
    • Format
    • Resource Type
    • Datastore enabled (ticked by default)
    • Mimetype
    • Mimetype (Inner)
    • “Extra Fields” (user-defined, or used by Orbital)
      Screenshot from CKAN
  13. To deposit a record for this dataset in the Lincoln Repository, go back to the Orbital Researcher Dashboard at: https://orbital.lincoln.ac.uk/ and navigate to your project. Toward the bottom left of the page you should now see a table containing the dataset(s) you have created in CKAN for this project. Choose which dataset you want to deposit, and hit the “Publish to Lincoln Repository” button.
    Screenshot from the Researcher Dashboard
  14. The Researcher Dashboard will then display a deposit form containing the following fields (some of which should be being autopopulated from CKAN fields but which do not appear to be):
    • Title
    • Description
    • Type of Data
    • Keywords
    • Subjects
    • Divisions
    • Metadata visibility [Show|Hide]
    • People
      Screenshot from the Researcher Dashboard
      “Publishing will publicly announce the existence of your dataset on the Lincoln Repository, as well as start the process of long-term preservation of your data.“Usually you should only publish a dataset either at the end of a research project, or if the data is being cited in a paper. Publishing a dataset will place some restrictions on the changes you can make to the dataset in the future, such as removing your ability to delete the data. It will also generate a DOI, which allows your dataset to be uniquely identified and located using a simple identifier.“Please check the information in this form and make any necessary changes, as this is the information which will be entered into the published record of the dataset.“If you have any questions about this process please contact a member of the research services team for advice or assistance.”
  15. When you hit the “Publish Dataset” button, the dataset record from CKAN will be used to create a record in the Lincoln Repository. The record will be submitted for review by the Repository team, who will then make it live. N.B. for the time being, you will see an error “Validation errors: [doi] is a required string” – this happens because the University does not currently have access to the live DataCite DOI service, which would secure a DOI for each dataset record deposited from CKAN. This should form part of the new research information service.
    Screenshot from the Researcher Dashboard
  16. Here’s an example of a record in the Lincoln Repository, created from a CKAN dataset and made live by the Repository team.
    Screenshot from the Lincoln Repository

Problems with the deposit process as it currently stands:

  1. Permissions are not correctly cascaded from a project the Researcher Dashboard to a group in CKAN.
  2. There is currently no single sign-on between the Researcher Dashboard and CKAN.
  3. When CKAN challenges a user to log in to a group, they should be redirected back to the group page after logging in – instead they get sent back to the CKAN home page, in which case they will have to look again for their project under the “Groups” menu.
  4. A minor one – in CKAN “License” (noun) appears in US spelling (should be “Licence”).
  5. In order to add all the information needed to deposit a dataset from CKAN, user has to click  “Further information” tab on the left-hand menu (this is not obvious from the initial dataset form).
  6. Some of the field labels in CKAN are a bit opaque or use technical terms (“Mimetype”) which could do with explanation.
  7. When depositing to EPrints, some of the deposit fields should be being autopopulated from CKAN fields – this does not appear to be happening. The fields affected are:
    • “Description” (could be derived from CKAN dataset/resource Description fields)
    • “Type of Data” (could be derived from CKAN resource Format field)
  8. Repository records created from CKAN have the data “Creator” attached, but not the “Maintainer”.
  9. Repository records created from CKAN don’t have a link back to the CKAN dataset (should go in the EPrints “Official URL” field) – this will be required to provide access to the data.
  10. After deposit, users see the error message “Validation errors: [doi] is a required string” – the University does not currently have access to the live DataCite DOI service, which would secure a DOI for each dataset record deposited from CKAN.

Research data documentation and training materials

The final within-project version of the Orbital Research Data Management training materials are now live on the Orbital Researcher Dashboard website. They have been written collaboratively by the Orbital project team, and draw on a lot of existing RDM training and guidance material from across the web (in particular, from the DCC).

We intend that these materials will continue to be maintained and developed as part of the new University-wide research information service mentioned in a previous blog post.

Screenshot of the Researcher Dashboard

The training materials can be accessed at https://orbital.lincoln.ac.uk/ and cover the following areas:

  1. What is research data?
  2. The research data lifecycle
  3. Policies affecting your research data
  4. Data Management Planning (DMP)
  5. Data search and discovery tools
  6. Data storage and security
  7. Legal and ethical issues
  8. Tools for working with your data
  9. Data publishing and citation
  10. Licences for sharing your data
  11. Data curation and preservation
  12. Workshops and training events
  13. Help and support

The source text for each page is stored in an open Github repository (at http://github.com/unilincoln/rdm) in Markdown format. The page admin tools in the Researcher Dashboard can then be used to link to the source document, which is then formatted in the University’s Common Web Design.

These web pages will be used to support the ongoing RDM training for postgraduate students, which will shortly be rolled out to University staff.

“Managing Your Research Data” – training for postgrad students

As part of the JISC-funded Orbital project, we are starting to offer introductory training to (initially) postgraduate students, on how to look after their research data.

The first workshop is on 23 January 2013 at 10.00 in the Graduate School classroom, and there are further workshops every couple of weeks throughout 2013.

I’ll be arranging further workshops aimed more at staff in due course.

MANAGING YOUR RESEARCH DATA

The Graduate School – University of Lincoln Multiple dates throughout 2013

Research data management is an important part of the research process, and a vital part of academic practice. This one-hour workshop will include a presentation and discussion of what you should consider when creating, looking after, and sharing/publishing your research data.

The workshop will cover:

  • What do we mean by research data?
  • Policies affecting your data
  • Data Management Planning (DMP)
  • The research data lifecycle
  • Practical tools for looking after your data
  • Data publishing and citation
  • Where to go for help

Postgraduate students can book a place on a workshop, online at: http://uolresearchdata.eventbrite.co.uk/

Orbital deposit of dataset records to the Lincoln Repository: workflow

Further to yesterday’s blog post about linking our CKAN datastore with our EPrints Repository (to allow researchers to deposit permanent, public, citable records of their datasets), here’s a fleshed-out diagram of the proposed dataset deposit workflow process.

At the moment, this assumes a one-time “fire and forget” deposit. At some point, we’re going to have to tackle versioning.

The original diagram is available on Lucidchart. See the table in my previous blog post for details of which data fields are involved in the process (i.e. passed between CKAN, Orbital Bridge, the DataCite API, and EPrints).

This is a proposal and still has to be road-tested. Comments welcome.

Diagram of the dataset deposit process

Stages in the proposed deposit process:

  1. User enters project metadata in AMS
  2. AMS creates project container in CKAN
  3. User creates dataset record in CKAN
  4. Nucleus adds user metadata to CKAN
  5. User deposits data in CKAN
  6. User presses “DEPOSIT DATASET” button in CKAN
  7. Orbital Bridge requests DOI
  8. DataCite API returns DOI
  9. Orbital Bridge adds DOI to dataset record in CKAN
  10. User reviews and approves dataset metadata (making changes if necessary)
  11. Orbital Bridge writes changes back to dataset record in CKAN
  12. Orbital Bridge creates a new EPrints record via SWORD
  13. EPrints confirms existence of new record
  14. Orbital Bridge writes EPrints record URL back to CKAN dataset record

Research data training at the University of Lincoln

As part of the Orbital project to build a pilot Research Data Management (RDM) infrastructure at the University of Lincoln, I’m looking particularly at support, training and documentation.

We aim to start offering—early in 2013—an introductory 1-hour workshop on managing your research data, aimed at early-career researchers and postgraduate research students. In particular, we want to promote this training through three avenues:

  1. As part of the Lincoln Graduate School‘s standard timetable of postgrad training;
  2. Directly, to PhD students in the School of Engineering (our pilot group);
  3. To researchers who completed our Data Asset Framework questionnaire.

The training will be supported by documentation (written and maintained through WordPress and a dedicated RDM reading list), presented through the main Orbital “bridge” site, which we’re starting to treat as a VRE.

Here’s an outline of the initial workshop. I’m meeting the Graduate School this afternoon to agree this.

“Managing your research data”

  1. Definitions, terminology and scope (what do we mean by research data?)
  2. Policies and laws affecting your data
  3. The “research data lifecycle
  4. Data Management Planning (DMP)
  5. Practical tools for looking after your data
  6. Data publishing and citation
  7. Where to go for further help and support

Comments welcome!