A JISC-funded Managing Research Data project

Posts tagged Cloud storage

Following up on our post from the MRDHack day, what follows is an evaluation of ownCloud as an institutional alternative to Dropbox. Our DAF survey showed that researchers at Lincoln require better managed storage space than the current 1GB FTP “H: Drive” provided to each staff member. Many of them are using portable drives, USB sticks and cloud-based services such as Dropbox to store and share their research data. Services like Dropbox provide compelling advantages to more traditional storage. Dropbox provides, for free, double the storage on offer to Lincoln researchers at present; it is always backed up, existing on both the local machine and on Amazon’s servers, it offers versioning for files up to 30 days old with the free account and ‘forever’ for paid accounts, it is accessible from almost all devices with Linux, OS X, Windows, iOS and Android clients available. Files can be published to the web or shared privately with other Dropbox users.

We know, however, that researchers using Dropbox are doing so without a clear understanding of the terms and conditions of the service and would ideally like a similar service to be provided internally by the University, where we can retain control of the data and its associated security.

We were first made aware of ownCloud when D’Arcy Norman blogged about his initial trial of it. ownCloud is an open source tool, which provides the same features as Dropbox (and more). With the release of version 4 in May, it appears to be a credible alternative to Dropbox for institutions wishing to provide a modern storage solution for their staff and students. D’Arcy’s initial experience with ownCloud was promising but he found issues with the syncing of files. Our recent tests of ownCloud have found that these problems have now been resolved with a recent update and what follows is an evaluation of ownCloud version 4.0.6, looking at it from three perspectives:

  1. ownCloud as a general purpose storage technology for an academic community
  2. ownCloud as a storage technology for research data
  3. ownCloud as a technology for integration with Orbital

Storage for an academic community?

ownCloud is an AGPLv3 licensed open source project, which started in January 2010. The project is run by a company, also called ownCloud, which provides commercial services and support for its software. The company resides in both Germany and the USA. The development of ownCloud is also open and supported by standard tools for open source projects: a source code repository, bug tracker, IRC channel, mailing list, wiki and forum. There are currently 13 core members of the project and 34 contributing developers. Development of the code is currently very active with changes made several times a day. The ownCloud project was started by the KDE community (it has no dependencies on KDE) and therefore benefits from the involvement of experienced open source developers. The software is written in PHP and can use MySQL, PostgreSQL or SQLite for its database. As of version 4, ownCloud has the following features:

  • Web user interface for file uploads and management of account and other features. Files can also be uploaded from an existing URL.
  • Windows/Mac/Linux/Android/iOS synchronisation clients.
  • WebDav integration for direct access to file storage. ownCloud has its own WebDAV server.
  • Folder and/or file sharing: publish to the web or share with groups or individuals.
  • File versioning.
  • An API for application integration.
  • Previewing for a number of filetypes.
  • Server side file encryption.
  • LDAP integration.
  • Notifications.
  • ownCloud can be installed in a PHP/MySQL environment on both Linux and Windows servers.

ownCloud also provides a number of other applications that can be activated or not, including a calendar, task list, contacts management, a built in text editor, image management, and experimental support for FTP, Google Drive and Dropbox integration. These are all available from the built in ‘app store’ and configurable by administrators. Maximum quotas for each account can be specified on an account-by-account basis, and a maximum file upload size can also be specified if needed.

The roadmap for version 5, due in August 2012, list the following:

  • Inter-ownCloud Sharing
  • Ajax interface
  • Mozilla Sync Integration
  • Improved permissions
  • Mounting of Dropbox and Google Drive
  • Improved version control

(more…)

Last month, Nick wrote about how Orbital is being designed as an application to run in the cloud. This week, we met with Andy Powell from Eduserv to discuss the use of their ‘Education Cloud‘ for the Orbital project.

In the run up to this meeting, we’d been talking to colleagues in ICT Services about our need for more flexibility and autonomy when we required servers in order to do our work. Outside of work we’re quite used to spinning up servers on Rackspace or AWS to try things out and increasingly we’ve been looking for ways to take control of our servers in this way at work. We’re not the only Researchers who need this flexibility; colleagues in LiSC have also been telling us that for some of their work, the scalability and reliability of cloud services is looking increasingly attractive.

This is not to say that ICT services is inflexible and unreliable by any means. I’ve always found my colleagues very willing to help where and when they can, but I think we’d all agree that a central ICT department in a university, with the multivarious responsibilities it has, is not the same as a dedicated cloud provider and, in our case, does not offer the resilience nor the scalability that Rackspace or Amazon are offering for example. The availability of resources, the business model and available support are quite different. When I joined the University in 2007, ICT Services were implementing a new VMWare server farm, which has given us more flexibility than having to work with physical boxes in every instance. Typically, if I want a Linux server with 4GB RAM and 100GB of HDD, I put in a request, transfer approx. £1200, and some time later, a virtual server is provided to me at no further cost. If I need more RAM or HDD, I put in another request, transfer some money, and some time later, I get what I need. This process can take weeks or months.

However, our VMWare farm is now almost five years old and nearing ‘end of life’ and I know that ICT are thinking about the next five year cycle and how cloud computing fits into their future plans. Colleagues in the Online Services team have been using Rackspace recently as a CDN for the Common Web Design framework as well as hosting our popular Gateway website, and have been very impressed with the service. The main hurdle was not technical but organisational: billing for the use of the CDN is by credit card and Pay As You Go (PAYG), meaning we don’t know exactly how much it will cost each month. This is in contrast to how departments normally make payments which are known in advance and invoiced in arrears. Nevertheless, that hurdle has been overcome and hopefully set a precedence.

So the meeting we had with ICT Services was in light of all this and we recognised and agreed that Orbital was a timely and appropriate project by which the university could pilot a more extensive use of cloud services and look at how we might integrate servers in the cloud with our existing server farm. It would also allow us to think about new business models where the real costs of running a server are more transparent to everyone, rather than being absorbed by ICT as the server ages.

Nick has been setting up the Orbital development environment and basic architecture (more on that in another post) using Rackspace and the Orbital project pays for this each month via our departmental credit card. This works fine if a) the department is happy to use the credit card in this way; and b) we have dedicated project funds for this, but it’s no way to run a long-term service that is to be sustained by the institution. Our interest is not really in whether we use Rackspace or Eduserv for hosting during the period of our project – both offer Linux boxes afterall – rather we’re interest in working with ICT to ensure that by the end of the project, there are formal processes in place for a) running sustained services in the cloud; and b) providing researchers with the ability to spin up and manage adhoc servers as and when they are required.

The plan is to evaluate both Rackspace and Eduserv over the coming months, looking at which service fits best with the future plans of ICT Services. Rackspace has a much more mature offering, but we’re really keen to work with Eduserv too, recognising that they’re a new not-for-proft provider of cloud services, running on JANET and with a long history of providing hosting and other technical services to HE and government.

At our meeting with Andy, he went through much the same presentation that Nick and I had seen at the MRD start-up meeting, answering our specific questions along the way. He also demonstrated (for the first time??) the vCloud Director interface for setting up and managing the servers, and this should, in principle, integrate with our existing VSphere system. One of the nice things about the Eduserv offering is that unlike most other cloud providers, they provide the entire vCloud Director application to their customers, including a full API, rather than a cut-down interface. We’ve yet to see how vCloud Director will allow us to create access controls for different types of users, but that’s what the Orbital project will be helping to investigate and I’m pleased that we’re able to work with our ICT department in this way. There are other important questions, too, around data protection and liabilities, and Andy was keen that we review Eduserv’s Terms and Conditions and SLA and feed back our thoughts on it.

This experience will allow me to better understand the business model of the cloud and how to make the business case for developing and running cloud-based services. As Nick previously said, it also allows us to make our costs more transparent, too, so that the actual costs (per Gigabyte and per Gigahertz) of managing research data are clearer to both Researchers and the institution. Having a clearer idea of the costs will help us create a more sustainable service in the long run.

With the Orbital project we’re looking at taking a leap into the brave new world (well, at least as far as university projects go) of cloud hosting. Now, I hate the word “cloud” when used to describe most services because people bander it about like it’s some magical world-fixing technology when in actual fact all they mean is “it’s on the internet”. We have “cloud” services which are fundamentally no different from doing the same thing on a server kept on a desk somewhere; but Orbital isn’t going to be like that.

I hope.

Instead, Orbital will be a true ‘cloud’ service in that it’s a resource which end users can tap into with no care at all for the underlying technologies. It’ll scale up and down with demand, extending both processing power and storage space as needed. Should one of our servers fail for any reason it won’t be met with a week of downtime whilst we rebuild things, but instead a seamless transition of work to one of the redundant, load balanced alternatives. If a process stops working then instead of the entire system crashing down it’ll adapt, queueing tasks until things are restored. Alongside this, the use of common standards is something that is essential to development. RESTful APIs follow well understood principles for interacting with data, and authentication using OAuth (the same authentication method used by Twitter, Facebook, Google and Microsoft) is core to how things behave.

Whilst the Orbital application itself is built to run in a cloudy manner using these loosely coupled methods and Rambo architecture, we’re also going to be hosting the thing in the cloud. This helps us with a few things including the aforementioned scalability, improved resiliency and the ability to properly analyse how the cloud works for higher education.

There’s also an unexpected benefit of this cloudy approach to Orbital: we gain the ability to pin a real-world cost on the storage of research data since we are quite literally being charged by the GB. At the moment researchers tend to treat storage as a one-off cost – for example buying a pile of hard disks – with less understanding of what it actually costs to keep them spinning. Since Orbital will know more about the intricacies of the stored data than the researchers we will be able (for the first time) to offer a number both in terms of how much it is costing to store data and also the estimated carbon impact.

Both these numbers are something that we want to be able to give researchers to help them understand that hanging on to research data has a cost, but also that it’s probably more efficient to hang on to it in a central, cloud-based platform. Of course, we also want to give people a clean exit strategy so we’re also going to be looking at ways of easily creating ‘hard’ copies for offline, non-cloud storage whilst still maintaining a virtual presence for the purposes of referencing and metadata.