Is open data the same as big data? Well no, not really.
Open data is stuff that is out there which can be freely used, shared and messed about with by anyone, anywhere, at any time and for whatever purpose they choose. It is the wildebeest of the data world, able to roam free across the plains. The definition of Open Data is described more eloquently and more succinctly by the Open Knowledge foundation which has been around since 2005.
Big data on the other hand as we all know is the analysis of whopping great data sets to reveal patterns, trends, links and opportunities that may be used to improve service, improve efficiency or that are just down right interesting.
So, they are different yet they are clearly related. Big data may not be open yet open data can be big.
I’m raising this with myself as my big data work continues to struggle to get off the ground and that may be because I have confused it in my mind with an open data idea. I have always believed that there are patterns and opportunities within our data just waiting to be found yet we may not be the right people to find them. To get the most out of what we have we might need to set it free, release it into the wild and let others who are more capable search out what can be of use. My first belief leans towards big data yet my second relies upon open data.
Perhaps this is where I have been going wrong. On stopping and thinking about it I’ve managed to achieve very little in with a big or open way. I’d thought my lack of progress was on the back of not being able to persuade my colleagues of the value within the data yet it may be because I don’t have the thoughts straight inside my head. Perhaps it is me that needs to be persuaded about what needs to be done.
So where do I go from here? Firstly, I’ve commissioned a small amount of work detailing how other authorities have addressed open data and I have met up with several people to try and understand our position. I must say that the feedback doesn’t paint us in the best of light but then again I knew that really. That bit wasn’t much of surprise.
Steve and I have talked about what this all means and we’ve arranged another meeting of the big data team for January.
What I have come to realise is that what I (we) could do with is an open data project that allows our data to be played with in a big data way. We need to focus on getting the data out there. The feedback showed us some good examples of where this has been done and at low cost. I’m going to reinvigorate the project by flexing the report to reflect my new thinking and look for a sponsor external to the service. Steve and I are going to visit some exemplar authorities and we’re going to set up a local data community to let us know what data they want.
This has given me hope.
Thoughts from a Guerrilla Worker 