Orbital – Page 2 – A JISC-funded Managing Research Data project

The Researcher Dashboard

Joss Winn / May 3, 2013May 3, 2013 / Development

At the JISC MRD final programme meeting, I demonstrated the work we’ve done to integrate disparate research information systems at Lincoln and begin to develop a workflow between them for the deposit of datasets. Below, are two videos which run through the website and application that was called ‘Orbital Bridge’ and is now referred to as the ‘Researcher Dashboard‘.

The screencasts are a bit rough and ready – I’m no voice actor – but it should give you an idea of what we’ve done and the way that this work points forward to further integration and an increased aggregation and re-presentation of research information, of which datasets are a part. Feel free to post questions in the comments box below. Thanks.

The first video (7 mins) discusses the website in general and the ‘My Profile’ section of the Researcher Dashboard.

The second video (13 mins) discusses the ‘My Projects’ section of the Dashboard and gives an overview of the integration and workflow between different research information systems. I haven’t gone into any detail on the actual use of CKAN as we’re using a fairly vanilla 1.7.2 version, with just some additional branding and authentication work. If you’re interested in how CKAN works, I recommend that you try http://demo.ckan.org. Much of our development with CKAN up to now has been through interaction with its APIs to set up groups and users from the Researcher Dashboard and pull information about datasets from CKAN into the Dashboard.

We’re just about to advertise for a Research Services Developer post (c. £30K/yr), and look forward to that person picking up this work and developing the application and CKAN further. More details on the new role will be posted here in due course.

There are a couple of things that I left out of these videos that I should also mention:

Information about a project in the Researcher Dashboard can be edited from the project page and provides a place where researchers can add further information about the project which is not being collected by the Awards Management System. It also allows the Researcher to add people to the project and assign them roles so they can edit that information, too.If I haven’t emphasised it enough already, one of the ‘features’ of the Dashboard is that it’s an application that allows us to collect more information about research at Lincoln and starts to link it all together. For the first time, information about people, projects, funding, research outputs, datasets and metrics are being brought together in a structured way. With this information, we can go on to build other applications (e.g. a database of research expertise) based on information provided by researchers themselves, enhanced by some simple text mining and clever semantic tagging.

Finally, the documentation on the website is managed by a simple ‘content management system’ that we built for Librarians and other staff who support research at Lincoln. All of the training materials are easily accessible to a non-technical user and can be edited on the website or optionally, managed through Github. This way the site’s content can remain up-to-date and accessible to content authors without having to ask developers of the site to add/edit content for them.

Research data documentation and training materials

Paul Stainthorp / April 26, 2013April 26, 2013 / Training

The final within-project version of the Orbital Research Data Management training materials are now live on the Orbital Researcher Dashboard website. They have been written collaboratively by the Orbital project team, and draw on a lot of existing RDM training and guidance material from across the web (in particular, from the DCC).

We intend that these materials will continue to be maintained and developed as part of the new University-wide research information service mentioned in a previous blog post.

The training materials can be accessed at https://orbital.lincoln.ac.uk/ and cover the following areas:

The source text for each page is stored in an open Github repository (at http://github.com/unilincoln/rdm) in Markdown format. The page admin tools in the Researcher Dashboard can then be used to link to the source document, which is then formatted in the University’s Common Web Design.

These web pages will be used to support the ongoing RDM training for postgraduate students, which will shortly be rolled out to University staff.

Team meeting notes: Winding down, tidying up, moving forward

Joss Winn / March 28, 2013March 28, 2013 / Meetings

We had a quick Orbital team meeting this afternoon. Not everyone could attend, but here’s what was reported:

Joss is writing up the final report for JISC and, along with Paul Stainthorp working with the Dean of Research and University Librarian to establish and resource a new ‘Research Information Service’. This service will bring together people, systems, and policy relating to ePrints, the REF, RDM, bibliometrics and scholarly communication. Joss presented this to the JISC MRD Programme meeting on Tuesday. The slides from the conference are below.

Orbital Project: From RDM to a Research Service from Joss Winn

Joss and Nick will also be creating a screencast and advocacy materials to explain and promote the work of the project and new Service before the end of April, when the project officially ends.

Nick is working with the DCC on DMPOnline integration into Orbital’s ‘Researcher Dashboard’. He’s also documenting code and generally tidying up before the end of the project. He’s also working with ePrints Services to add a new user permission to allow the Researcher Dashboard to submit work to ePrints “on behalf” of someone else.

Harry has a one month extension to his contract to undertake some work commissioned by Janet, evaluating different storage services. More on that later in April.

Bev and Paul have been working on the upgrade to ePrints and attending to REF work. The data warehouse (‘Nucleus’) developed as part of the Orbital project looks likely to be used as a source of information for the REF, in particular the university staff profiles, which run on top of Nucleus. (Orbital/Nick, along with combined effort from two other JISC projects at Lincoln (Linkey/Alex Bebop/Dale), built a new university staff profile directory. Something that I keep forgetting! My example can be seen here).

RDM training for post-graduates continues each month and similar training for staff is being arranged through HR.

Robotics data now stored in CKAN

Joss Winn / March 21, 2013March 21, 2013 / Development, Impact and Benefits

I’ve had to delay this post until confirmation of Tom’s project funding came through, but I’m pleased to be able to say that we’ve published our first complete research dataset(s) on CKAN.

Some months ago, Researchers, Tom Duckett and Feras Dayoub, came to us asking if we could host their data to support two publications and an EU grant application they were about to submit. We quickly stuck the data on one of our servers, they knocked up some HTML pages and we advised them on licensing the data so that it could be re-used. It was a temporary solution but we assured them that their root domain name would always act as a proxy to the final resting place of their data and so they started to tell the world about it. I’m told there was much interest in their data on specialist mailing lists and we were invited to submit a paper which discussed the data and the process of its publication. Their consortium bid for EU funding was also successful. Here’s what Tom had to say:

I believe that publishing our datasets for long-term robotic mapping has helped us: 1) to achieve greater awareness of our work (we were among the first groups in the world to study long-term mapping by mobile robots, in research from 2004-present), enabling other researchers worldwide to use our data, 2) to increase citations to our REF-able research papers in this area, and 3) to play our part in successfully applying for a 4-year FP7 IP project in collaboration with 7 other partners, by showing that we already have a track record in hosting such datasets. (STRANDS project – joint PIs at Lincoln: Marc Hanheide and Tom Duckett). One of the requirements of this project will be to publish even larger datasets of robot data, so we look forward to collaborating with Joss and colleagues again in future to address the challenges of hosting and curating “big data” for robotics research.

Prior to switching to CKAN, we were just about to move Tom and Feras’ data across to our own Orbital software, which met their minimal requirements, but having now switched to integrating with CKAN, we’ve moved the datasets to their permanent home at https://ckan.lincoln.ac.uk.

Just as we promised, Tom and Feras are still able to direct people to the original web address we gave them which points to their research pages, but the data itself is now hosted on CKAN. Having seen Tom’s data presented in this way, his colleague Greg published his data in the same way, using our WordPress platform to build a site explaining the data and CKAN as the actual data store.

This all happened before we had our Orbital Bridge publishing workflow in place (a post on that in a couple of weeks) and in the absence of a working Orbital application, I uploaded the data on Tom and Feras’ behalf. I spent quite some time using CKAN and can make the following observations about version 1.7.x, which is what we currently use.

Batch uploads: The data was zipped up into four collections of zip files. My task was to duplicate the organisation of the data which made sense to the researchers. This was possible as you can see, but it was tedious uploading each of the 29 zip files, many of which were over 1GB each. There were no problems doing so, it was just tedious and better batch upload/edit operatios in CKAN would make this much easier. Ideally, I’d like to have uploaded the zip files from each of the four collections of data, catalogued them by batch where they shared the same information and then individually edited attributes like the title of each zip file, for example. Having been an Archivist on and off for the last decade, this is one of the main gripes we have with library and archive systems. When dealing with collection of things, we need to be able to operate on them as collections and not have to deal with each object individually. I’ve spoken to CKAN developers about this and there are work-arounds, using scripts and a form extension, but it’s not something CKAN offers to most users with ease. Yet! 🙂
Research Groups and projects: The v1.7.x version of CKAN understands the concept of ‘dataset’ e.g. https://ckan.lincoln.ac.uk/en/dataset/ltmro-1 and of that dataset containing discreet resources. e.g. https://ckan.lincoln.ac.uk/en/dataset/ltmro-1/resource/92cbf22b-3293-45a3-b1de-f7782e581fe8 CKAN also understands the concept of ‘groups’ e.g. https://ckan.lincoln.ac.uk/en/group/lincoln-centre-for-autonomous-systems which datasets can be attached to. Groups are simply a label you apply to a dataset. You can add people to a group with specific read/write permissions over the group and you can add datasets to the group, too. CKAN also maintains a history of the actions of that group e.g. https://ckan.lincoln.ac.uk/en/group/history/lincoln-centre-for-autonomous-systems However, currently, CKAN does not (yet) understand ‘projects’, i.e. an organisational concept that is role-based and allows a user to administer other users and work. Groups are not synonymous with projects, but we think that a new feature in CKAN v2.0, due for release in a month or so, will resolve this. As I understand it, CKAN organisations will work like Github organisations and if so, that’s good. On Github, our research group, LNCD, is an ‘organisation’ and within that organisation I can add/remove people, give them roles, create private and public repositories (‘datasets’) and we can be members of more than one organisation, too. e.g. http://github.com/lncd and http://github.com/josswinn There is already a CKAN extension that implements organisations, but we’re waiting for this work to be merged into the core code.
Citations: If you look at Tom’s original web pages for their data, they are pretty clear in providing details about how to cite their data. This is so important to academics. CKAN does not offer a way to automatically generate a suggested citation for people who use the data. EPrints, on the other hand, offers the citation details of a research paper right at the top of the publication record e.g. http://eprints.lincoln.ac.uk/6046/ Some work on citations for CKAN has happened – there were conversations a few weeks ago on the IRC channel – but it’s something we need to work on, too. As a temporary solution, I have added the paper citation details as additional fields in the dataset record. CKAN is nice in that it allows you to add adhoc key-value pairs when cataloguing. However, this doesn’t address the citation details for the actual datasets themselves, but rather the publications.

In the near future, our ‘Researcher Dashboard’ application (codenamed ‘Orbital Bridge’) will handle the data deposit workflow from project creation to grabbing a datacite DOI to setting up a CKAN environment, to depositing a record of the data in ePrints for curation and preservation by the university. However, the upload and cataloguing of data will still be done by the researcher using CKAN, with Orbital aggregating information about the project, publications and data into a ‘dashboard’ for the researcher. Something like this below, which is an actual screenshot of another project that we’re using to test the ‘Researcher Dashboard’. More on this soon…

APIs first!

Harry Newton / March 15, 2013March 27, 2013 / Development

After testing the new SWORD2 endpoint for our new ePrints 3.3 instance, we found that a significant change was needed for the SWORD library. Minor changes included the endpoint, which became …/id/contents instead of …/sword-app/deposit/inbox, and the structure of the XML changing from <eprint> tags to <entry> tags. The main change was the implementation of how the XML was posted. The SWORD library swordappv2-php-library was forked from the github repository so that an XML string could be posted. This was because our current method posted a string, which the endpoint read as a file rather than metadata. So the dataset had the XML attached to it as a file, with no metadata. We have made additions to the library, changing it to post a string of XML metadata rather than a file. This fixed the problem, giving the dataset metadata once posted rather than attaching it as a separate file.

Now heres the main problem. The dataset gets posted to ePrints in a deposited state, which ePrints classes as ‘in Review’. Now, ePrints requires a minimal set of metadata before a dataset can be ‘in review’. But only if the dataset is made manually within ePrints. Not via the API. Over the API, you can post a dataset straight to ‘in review’ without the mandatory set of metadata. Which brings me to the title of this post; APIs first! API driven development would mean that the APIs are built first so this kind of situation would be avoided.

Another problem we came across during the change was that the test account we had for testing deposits no longer existed due to the migration of user accounts skipping it. This is fine, as an unauthorised response should be received on an attempted deposit. This was not the case, as we got an ‘Invalid XML’ response. Which was unusual, as the XML was valid and everything we tried was to no avail. It was by chance that we found the solution, by switching to an account we knew existed and the deposit working as planned. What had happened was that the depositing had failed, due to the account not existing, but the wrong error message being sent back.

So I reiterate; APIs first. Knowing what the response is, and that the functionality of the application works first, is the most important aspect of said application.