Author: Joss Winn

I work in the School of Education at the University of Lincoln.

Hello CKAN

Joss Winn / / Announcements, Development, Meetings, News

On Wednesday, we hosted three people from the Open Knowledge Foundation, to discuss the Orbital project and their software, CKAN. It was a very engaging and productive day spent with Peter Murray-Rust (on the Advisory Board of OKFN), Mark Wainwright (community co-ordinator) and Ross Jones (core developer). We asked them at the start of the day to challenge us about our technical work on Orbital so far and I described the day to them as an opportunity to evaluate our work developing the Orbital software so far. We didn’t touch on the other aspects of the Orbital project such as policy development and training for researchers.

To cut to the chase, the Orbital project will be adopting CKAN as the primary platform for further development of the technical infrastrcuture for RDM at Lincoln. This is subject to approval by the Steering Group, but the reasons are compelling in many ways and I am confident that the Steering Group will accept this recommendation. More importantly, the Implementation Plan that was approved by the Steering group and submitted to JISC remains unchanged.

The raw notes from our meeting are available here. Remember these are raw notes written throughout the day, primarily for our own record. They probably mean more to us than they do to you! Thanks to Paul Stainthorp for his fanatical note taking 🙂

Here’s the list of attendees and our agenda:

Present

Peter Murray-Rust (OKFN)
Mark Wainwright (OKFN)
Ross Jones (OKFN)
Joss Winn (University of Lincoln, CERD)
Nick Jackson (University of Lincoln, CERD)
Harry Newton (University of Lincoln, CERD)
Jamie Mahoney (University of Lincoln, CERD)
Alex Bilbie (University of Lincoln, ICT services)
Paul Stainthorp (University of Lincoln, Library)

Agenda

09.30 Introductions
10.00 Orbital introduction and context: Student as Producer, LNCD; Orbital bid and pilot project; Discussion of Orbital approach, the data we’re using, user needs etc.
10.30 CKAN introduction and context
11.00 Technical discussion – Orbital
12.00 LUNCH
12.30 Technical discussion – CKAN
13.30 Discussion – should Orbital adopt CKAN?
14.00 data[.lincoln].ac.uk
15.00 Next steps; Opportunities for collaboration/funding?

What is probably of most interest to people reading this are the pros & cons of the Orbital project adopting CKAN. I’ll provide more context further into the post, but here’s a summary copied from our notes:

Continue reading “Hello CKAN”

ownCloud: An ‘academic dropbox’?

Joss Winn / / Development, Research

Following up on our post from the MRDHack day, what follows is an evaluation of ownCloud as an institutional alternative to Dropbox. Our DAF survey showed that researchers at Lincoln require better managed storage space than the current 1GB FTP “H: Drive” provided to each staff member. Many of them are using portable drives, USB sticks and cloud-based services such as Dropbox to store and share their research data. Services like Dropbox provide compelling advantages to more traditional storage. Dropbox provides, for free, double the storage on offer to Lincoln researchers at present; it is always backed up, existing on both the local machine and on Amazon’s servers, it offers versioning for files up to 30 days old with the free account and ‘forever’ for paid accounts, it is accessible from almost all devices with Linux, OS X, Windows, iOS and Android clients available. Files can be published to the web or shared privately with other Dropbox users.

We know, however, that researchers using Dropbox are doing so without a clear understanding of the terms and conditions of the service and would ideally like a similar service to be provided internally by the University, where we can retain control of the data and its associated security.

We were first made aware of ownCloud when D’Arcy Norman blogged about his initial trial of it. ownCloud is an open source tool, which provides the same features as Dropbox (and more). With the release of version 4 in May, it appears to be a credible alternative to Dropbox for institutions wishing to provide a modern storage solution for their staff and students. D’Arcy’s initial experience with ownCloud was promising but he found issues with the syncing of files. Our recent tests of ownCloud have found that these problems have now been resolved with a recent update and what follows is an evaluation of ownCloud version 4.0.6, looking at it from three perspectives:

  1. ownCloud as a general purpose storage technology for an academic community
  2. ownCloud as a storage technology for research data
  3. ownCloud as a technology for integration with Orbital

Storage for an academic community?

ownCloud is an AGPLv3 licensed open source project, which started in January 2010. The project is run by a company, also called ownCloud, which provides commercial services and support for its software. The company resides in both Germany and the USA. The development of ownCloud is also open and supported by standard tools for open source projects: a source code repository, bug tracker, IRC channel, mailing list, wiki and forum. There are currently 13 core members of the project and 34 contributing developers. Development of the code is currently very active with changes made several times a day. The ownCloud project was started by the KDE community (it has no dependencies on KDE) and therefore benefits from the involvement of experienced open source developers. The software is written in PHP and can use MySQL, PostgreSQL or SQLite for its database. As of version 4, ownCloud has the following features:

  • Web user interface for file uploads and management of account and other features. Files can also be uploaded from an existing URL.
  • Windows/Mac/Linux/Android/iOS synchronisation clients.
  • WebDav integration for direct access to file storage. ownCloud has its own WebDAV server.
  • Folder and/or file sharing: publish to the web or share with groups or individuals.
  • File versioning.
  • An API for application integration.
  • Previewing for a number of filetypes.
  • Server side file encryption.
  • LDAP integration.
  • Notifications.
  • ownCloud can be installed in a PHP/MySQL environment on both Linux and Windows servers.

ownCloud also provides a number of other applications that can be activated or not, including a calendar, task list, contacts management, a built in text editor, image management, and experimental support for FTP, Google Drive and Dropbox integration. These are all available from the built in ‘app store’ and configurable by administrators. Maximum quotas for each account can be specified on an account-by-account basis, and a maximum file upload size can also be specified if needed.

The roadmap for version 5, due in August 2012, list the following:

  • Inter-ownCloud Sharing
  • Ajax interface
  • Mozilla Sync Integration
  • Improved permissions
  • Mounting of Dropbox and Google Drive
  • Improved version control

Continue reading “ownCloud: An ‘academic dropbox’?”

Version 0.3 released

Joss Winn / / Announcements, Development

Our aim is to release a new version of Orbital every month until the end of the year. Yesterday, we released version 0.3, which, as well as many small improvements and bug fixes,  improves the handling of dynamic datasets and begins work on implementing and integrating ownCloud with Orbital. Here’s the changelog.

  • Improvements to project activity timelines:
    • Public/private modes
    • Calendar events
  • Improvements to filetype handling and file uploads
  • Improvements to file management, collections and private/public modes
  • Dynamic datasets:
    • A working query builder
    • Queries can be saved and re-run against data
    • CSV output of data for use by external tools e.g. Matlab
  • Working Datasets:
    • Preparation for ownCloud integration (integration with Lincoln SSO, evaluation of product, contact with developers)

The plan for version 0.4 is full ownCloud integration with Orbital via the respective APIs, which will provide the first part of the overall Orbital workflow: ‘Working Data’ -> ‘Dynamic Data’ -> Archive Files. During two weeks in August we’ll also be setting up our own private in-house cloud using OpenStack and moving Orbital in-house from Rackspace.

Orbital Team meeting notes 26-07-12

Joss Winn / / Meetings

Agenda & notes

  1. Application update: Orbital v0.3 will be release today. New dynamic dataset features. Import/export/query data. Already auto importing Siemens’ sensor data for analysis. Integration with MatLab will be presented to the Siemens/Lincoln research group on August 1st. We have set up ownCloud as an alternative to Dropbox and integrated with it with Lincoln SSO. Working on full integration with Orbital for Orbital v0.4 release next month. Nick, Harry and other members of LNCD will take third/fourth weeks in August to set up and build OpenStack private cloud for R&D/Academic Computing. Following that, Orbital servers on Rackspace will be moved in-house to OpenStack. Initially starting with three servers (26 cores, 140GB RAM) and 30TB storage. We intend to provide a Continuous Integration (CI) environment (Gitorious/Jenkins) for staff and student research, as well as  support LNCD R&D.
  2. Edinburgh/OR12:  Presented a paper at OR12, which was well received. Came away thinking that RDM has ‘arrived’ and that platforms for RDM need to be more integrated into research process than research output repositories have been. Orbital is well placed for this e.g. ‘dynamic datasets’. Discussed systems with OKFN/CKAN representative at OR12 and have invited OKFN to Lincoln to discuss CKAN and Orbital. Initial Skype meeting Monday 30th July. We have asked OKFN to challenge/persuade us as to why we should adopt CKAN rather than continue to develop Orbital. Need to think about the pros/cons of eight more months of Orbital development vs. eight months contributing to CKAN development. May lead to Lincoln extending CKAN for academic RDM, re-using bits of Orbital. Still not clear. Nick has also contacted Patrick at Soton about his winning proposal for DevCSI developer challenge. Patrick’s proposal is very similar to Orbital and both Nick and Patrick are keen to work on dynamic data and visualisations together with others.
  3. RDM Policy: Joss and Annalisa met with Andrew S, Head of Research & Enterprise Office. Reviewed draft RDM Policy. Will make minor amendments but on the whole agreement on way forward.
  4. Training: Joss, Annalisa, Melanie, Bev, Paul to meet early September and bring together RDM training materials produced by other projects. Evaluate, synthesise, extend and re-package for Lincoln. Agreed to arrange RDM training workshops via HR for staff every three weeks from late September to test and inform the development of this project deliverable. Annalisa/Melanie to arrange meeting and workshops.
  5. Metadata: Discussion about adopting BL’s minimum metadata requirements for DataCite. Agreement that the mandatory and optional attributes should be part of Orbital. Need to talk to/confirm with Bev about this and add relevant tasks to Pivotal Tracker. CKAN (see above) meeting relevant to this. If CKAN is used to publish datasets, need to ensure it meets this requirement.
  6. Research Information/Systems integration: Research Information Management (RIM) at the university is dependent on three systems: new Awards Management System (AMS) (Research and Enterprise Office), Orbital, and EPrints (Library). Need to contact Worktribe about API access to AMS for Orbital. EPrints work planned to add the ‘REF plugin’ will give us better data. SWORD2 deposit from Orbital planned. Again, need to consider in light of CKAN conversations. Other work going on in university to build business intelligence dashboards. Lee to arrange meeting with Registry DBAs and Orbital team to discuss data warehousing and dashboards. Lots of overlap in interest/experience/skills. Not enough talking.
  7. Business Case: Joss still waiting on storage modelling costs from ICT to present to SMT in September. Will start process of writing Business Case following that meeting.

 

Eating Your Own Dog Food: Building a repository with API-driven development

Joss Winn / / Dissemination, Research

We’re in Edinburgh, at Open Repositories 2012, and will be presenting our paper at 9am tomorrow morning (yes, that’s right, the morning after the conference dinner!). Here’s the paper we’ll be discussing.

As part of its project to develop a new research data management system the University of Lincoln is embracing development practices built around APIs – interfaces to the underlying data and functions of the system which are explicitly designed to make life easy for developers by being machine readable and programmatically accessible.

http://eprints.lincoln.ac.uk/5962/

Eating Your Own Dog Food

View more presentations from Nick Jackson