We’re in Edinburgh, at Open Repositories 2012, and will be presenting our paper at 9am tomorrow morning (yes, that’s right, the morning after the conference dinner!). Here’s the paper we’ll be discussing.
As part of its project to develop a new research data management system the University of Lincoln is embracing development practices built around APIs – interfaces to the underlying data and functions of the system which are explicitly designed to make life easy for developers by being machine readable and programmatically accessible.
The Orbital project team met today (24 May 2012) and agreed the following:
Documentation
User documentation will focus on the “why”s of Research Data Management, rather than being a point-and-click guide to the Orbital UI (which should not require detailed explanations).
JW will create a changelog (human readable text file) for each major release of Orbital, so that documentation for each feature is review if that feature is updated.
PS will lead on writing documentation (as HTML pages, stored in the GitHub repository), with documentation for release v0.N completed and available by the launch of v0.N+1
PS will email colleagues from the Library and Research/Enterprise for assistance on writing documentation.
Training
JW will invite Melanie Bullock and David Sheppard on to the Orbital working group. He is meeting Annalisa Jones to discuss RDM training for staff.
Releases/development
Orbital v0.1.1 (including bug fixes) met all of the initial ‘minimum viable product‘ requirements specified by Dr Tom Duckett, and also includes the basics of project administration.
v0.2 will include improvements to the file upload/management, project management, and license management interfaces, as well as clearer distinction between language files and operating code.
NJ demoed the current version of Orbital to Siemens staff. He now has access to Siemens machine data for testing within Orbital.
The group discussed the LNCD plans for internal servers/private cloud, and about the disk space requirements and costs.
Integration
The current version of the DMPOnline tool has been installed on a test server. The group discussed our approach to integration between external tools/software (such as DMPOnline, R, Gephi) and Orbital.
NJ is going to email Adrian Richardson at the DCC to ask when the DMPOnline APIs will become available.
RDM policy
JW presented the draft policy to the University RIEC committee. The committee have been asked to send comments to Joss. (One comment at the committee meeting was that our having a policy too geared around the requirements of the Research Councils may not be appropriate for Lincoln, which generates a lot of non-RC income. However it was noted that the good practice specified by the RCs is good practice for management of all research data, whatever the funding source.)
NJ and JW are presenting a paper at OR12 (Open Repositories 2012). PS will attend for ~1 day of the conference.
JW is chairing a planning event with Ken Evans, looking at data/schema modelling for all the University’s research information systems (Orbital, EPrints, the Awards Management System, people systems) using an Object Role Modelling (ORM) approach.
JW will convene a sub-group to consider the responses in detail, and plan follow-up interviews.
Business case
JW is currently gathering costs for long-term data storage. This will form the first strand of the Orbital business case, which will be presented to University SMT (along with the agreed RDM policy) in September 2012.
The JISC-funded Orbital project is building on earlier work at the University of Lincoln to develop a state-of-the-art research data management infrastructure, piloted with the first purpose-built School of Engineering in the UK in over 20 years.
Orbital (figure c) differs from traditional database applications in three significant ways:
Orbital Core uses MongoDB, a document-oriented, schema-less, so-called ‘NoSQL’ database. MongoDB offers flexibility in that it is capable of accepting an object representing any kind of data (e.g. tabular data, survey results, images) without the need to develop a schema beforehand. MongoDB also includes useful features which can boost performance and resiliency, namely sharding – slicing data across multiple servers so a request may be processed by multiple servers in parallel – and replication — keeping multiple identical copies of data on different servers in case one of them fails. Orbital is also designed to be able to spread the ‘core’ – the application which does the heavy lifting – and the ‘manager’ – the front-end user interface – across multiple servers without causing stress. In our experience MongoDB, combined with the Sphinx search engine to perform full-text searching, is also extremely fast and allows us to develop simple, attractive APIs which we can expose to user applications.
Orbital Core mediates access to the data via an open source OAuth 2 server we have developed and implemented at Lincoln. The use of OAuth 2 allows access to the data from multiple authorised systems providing that the owner of the data has given permission, instantly opening the Orbital application to third-party extension. This method establishes the identity, authentication and authorisation of users, providing direct access to individual data sets or portions of data sets (e.g. specific rows/columns) through APIs on Orbital Core.
The design and development of Orbital Core is API-driven, resulting in an application that offers 100% of its functionality through APIs, whether to our own Orbital Manager or a third-party application, each of which are treated equally by Orbital Core (figure c). As far as Orbital Core is concerned there is no functional difference between Orbital Manager (the front-end) and an application that a researcher has developed to meet a specific need; they are subject to the exact same access controls, restrictions, sanity checking and limitations. We have therefore eschewed some of the traditional approaches of building a database application, where access to the database is either provided via a stand-alone application (figure a) or via an API bolted on to the database (figure b). Orbital is also designed to be both stateless, i.e. all of the API functions are RESTful and thus represent a complete transaction with no requirement for session affinity, are not reliant on SQL features like transactions and joins, and have a reduced requirement for referential integrity.
Under this design, the API is the only way to interface with the data and functionality of the system. This API-driven approach offers several benefits:
Architecture is better: We are forced to think about data types and methods early on. Consistent behaviour across the application is easier to achieve.
Development is easier: Calling a well designed API is simple; error messages become cleanly captured by design; APIs encourage code reuse at both API and application end.
Updates become simpler: We can run two or more API versions concurrently; tweak the API back-end and all front-end applications (‘official’ and 3rd party) benefit at once.
The APIs are better: The APIs must include everything we want our application to be able to do. Reliability of the API is now critical which encourages better design of resiliency and error handling; and usability of the API is essential which encourages better documentation.
The challenges of this approach are that every time we want to build user-facing functionality we have to assess our APIs and work out where the functionality belongs as well as ensuring that we have lightweight data transfer and reliable error handling designed into the application. We also have to double up on some areas of development, writing both the respective Core and Manager parts of the system.
Illustrations
Figure a: The only way to interact with this application is to either be a user, or pretend to be one (for example via screen-scraping).Figure b: The most common form of API, consisting of a ‘second view’ on the data and functionality of an application. This style of API often exposes a limited subset of the application’s functionality.Figure c: In an API-driven model the API is the only way to interface with the application.
