Author: Nick Jackson

One thing that Orbital is focussed on is the notion that data should remain as raw and accessible as possible throughout the research cycle, with as few steps as possible between the source of the data and its storage. We don’t want our rich sensor data being turned into Excel spreadsheets unnecessarily, and we don’t want to have to manually run reports, extract data and then load the data into something else just to get work done.

When we were building our own platform this was achieved with something we called Dynamic Datasets; a MongoDB cluster with a RESTful API bolted on top which would accept massive amounts of data and re-expose it on demand, including powerful filtering and output options. In CKAN we’re using the DataStore API, powered by ElasticSearch (at the moment, this will change in the future), to do the same thing. The DataStore API was originally intended to provide a searchable view on data such as CSV files as part of the Recline preview, but we’re using it for a slightly different purpose as the sole repository of data without any corresponding originating file. Data comes straight from the source, through any sanitation process which we need to run, and into DataStore.

To test the principle whilst we’re finishing evaluating security for our more confidential research data (where we get to play with big engineering data) we’ve started loading data which is being used by On Course into CKAN. Specifically we’re ingesting our course data and our organisational structure — and we’re doing it all automatically.

Once a day we run a set of queries against our Nucleus institutional data warehouse to gather the required data. This is our ‘sensor’ in the model, representing the source of the data prior to any analysis or further work. The data is then manipulated into ElasticSearch’s bulk update API format, and injected directly into DataStore. Check out some of our resources, maintained automatically by direct interaction between our source data and CKAN’s DataStore:

• Awards
• Modules
• Colleges
• Faculties
• Schools

We can also use this data to run ‘real-time’ queries against the data — something researchers are likely to find useful.For example, we can ask for the first 100 modules taught by the School of Computing with “Games” in their title. This query is always as accurate as the last update of the DataStore API, which means that (as an engineering example) researchers can ask for data such as “every sensor value for machines recorded in the last day where the temperature was over 40”, and the data will be hot off the press every single time, with no requirement to re-export the information.

There are still a few rough edges we’re going to do our best to iron out, but all in all I’m fairly impressed with the ease of getting going.

One of the things that Orbital set out to do is to prove that agile development of software and solutions can arrive at the same outcome as a more traditional ‘waterfall’ method of project planning. A big part of this is the “release often” approach to development, shipping new versions far more quickly than is usually the case in academia. We’re actually on the slightly slow side of agile development (some companies push updates several times a day), but still aiming to ship a major point release every month for our users to have a play with and comment on.

On top of the point releases we’re also churning out additional maintenance releases with bug fixes and minor features roughly every week, our latest being 0.2.1 which shipped yesterday. These follow a fairly common pattern of odd-numbered releases (0.1.1, 0.2.1, 0.2.3 etc) being patch releases which fix bugs and even-numbered (0.2.2, 0.2.4 etc) adding evolutionary updates to existing functionality.

There are a few major benefits to doing things this way:

• Orbital never gets the chance to stray far from user requirements – we can’t go off and spend 6 months developing something that doesn’t do what people want, because they can tell us if we’re going wrong at least once a month (and often more frequently than that).
• Users who report bugs don’t have to wait several months for the next ‘service patch’ which rolls up thousands of changes, the next maintenance release will fix things and is usually only a few days out.
• The gap between a feature request and implementation is dramatically reduced, so the majority of feature requests are delivered whilst the requirement is fresh in the minds of the user. This results in more immediate usage and feedback.
• Our code is refactored and refined more often. Instead of building a massive codebase all at once and never going back to improve things we spend a small amount of time each release making sure that code is clean and sane, interfaces are well defined and so-on.
• Our continuous integration server won’t let us ship a product which doesn’t meet minimum requirements of code quality, documentation and testing. Being forced through this process on a regular basis means that we never get a chance to build up a significant backlog of problems.
• The use of our code repository and feature branches (there’s a post on this coming up later) means that every ‘merge’ of development code with our staging code is checked over by a developer other than the one who wrote the feature. When we ‘merge’ the staged and tested code with our production code the changes are checked yet again.
• More granular releases make it easier to roll back when things go wrong. Moving from v0.2.1 to v0.2.3 doesn’t need any database changes, so if something isn’t working as expected its a simple matter to move back to an earlier release. In contrast, if we only ever moved from major releases v1 to v2 (which will almost inevitably involve changing a database schema) then performing a rollback becomes much more challenging.

Thus far Orbital has made four distinct releases (v0.1, v0.1.1, v0.2 and v0.2.1), with v0.2.2 due out next week. If you’re interested in seeing (roughly) what’s in the pipeline don’t forget our Pivotal Tracker can tell you more.

You may remember a while back that I blogged about how Orbital thinks of research data, using our “Smarties not tubes” approach. We then went away for a bit, and in our first functional release included a way of storing your tubes, but nothing about the Smarties. This understandably caused some confusion amongst those who had paid close attention to our poster and were expecting a bit more than a file uploader.

The reason behind this seeming disconnect in our plans was simple: a researcher had a bunch of files they wanted to publish through Orbital, and it made a lot more sense to build something that would let them do what they wanted straight out of the gate rather than devote our efforts to breaking up their nice datasets again and storing individual bits and pieces. Fortunately, our next functional release is planned to include our magical Smarty storage system. Here’s a quick overview.

Dynamic Data (as we’re calling it) uses a document storage database to keep tabs on individual data points within a research project. It’s designed specifically so that you can fill it up with fine-grained information as individual records rather than storing a single monolithic file. We think this is the best way to go about storing and managing research data during the lifetime of a research project for a few reasons:

• It’s easier to find relevant stuff. Instead of trying to remember if the data you were looking for was in 2011-Nov-15_04_v2.xls or 2011-Nov-15_04_v3.xls you can instead just search the Dynamic Dataset.
• It’s an awful lot easier for us to ensure a Dynamic Dataset is stored reliably than a bunch of files, due to databases’ tendencies to have good replication and resiliency options.
• We can scale for storing individual files up to a few tens of gigabytes per file at most before things start to get silly, although we can store a lot of files at that size. We can scale a single Dynamic Dataset until we run out of resources.
• Data can be reproduced in a number of standard ways from a single source. The same source can easily be turned into a CSV, or XML document, or JSON, or any other format we can write a structured description for.
• With a little work, data can be reproduced in a number of non-standard ways from a single source. Templating engines can allow researchers to describe their own output formats.
• Data can be interfaced with at a much more ‘raw’ level with only basic programming skills. Equipment such as sensors can automatically load data to a Dynamic Dataset, survey responses can be captured automatically and more. Data can be retrieved and analysed in the same way, for example scheduling analysis of a week’s worth of data.

The data in release v0.2 is manipulated purely at an API level through Orbital Core, although upcoming versions will have cool ways of manually entering and querying the data through the web interface. Data is then quickly processed to add things like tracking metadata (upload time etc) and shovelled off to our storage cluster of MongoDB servers.