All posts tagged Open Data

WikiLeaks Endangers Open Data?


The 250,000 plus dispatches on SIPRNet, or the Secret Internet Protocol Router Network, have recently become part of the public domain via intrepid reporting by WikiLeaks. While the recent cables do not contain any classified information aside from time, recipients, and the subject title, the amount and frequency of cables sent between various U.S. Embassies raises the question: does publishing this information constitute an uncomfortable and potentially ill-advised breach of open data’s unstated rules of privacy?

Whatever anyone thinks of the U.S.’s foreign policy (which, admittedly, has never won a ‘Ms. Congeniality’ award) publishing this type of information begins to infringe on the sovereign rights of a nation-state. Open data is a movement dedicated primarily to helping people easily access information about their communities in an effort to improve their lives. Very few people, however, wish to have their tax records in the public realm. Not everything is made for public consumption; some communication should always remain private.

By publishing this information, WikiLeaks and the open data movement by association are making a bold move which will backfire. By antagonizing governments instead of working with them, those who push too far too fast engender ill will. Nothing is less desirable than an angry, wounded hegemon. While the release of the cables will not in of itself necessarily pose an immediate threat to world political relations (describing Kim Jong-il as a“flabby old chap” is a heart warming moniker compared to his daily online nicknames), the anger at those who breached confidentiality will not simply disappear. Those who strive to increase transparency using government data may encounter swift and undeserved retribution for the actions of their peers. The old fogies may not understand computers, but they do understand social relationships.
The open data movement must not only continue working to increase the amount of data available, but also it must dedicate effort toward fostering warmer social relationships and respecting each government’s particular boundaries. The U.S. undoubtedly needs to rethink its foreign policy, but it needs to come to that decision without having the equivalent of a virtual stick shoved in its eye.

Top 3 Challenges Facing Open Data

The Big Challenge

In only a few short years, open data has changed the way people interact with their governments. By increasing transparency and demanding well-designed software application, the open data/open government movement has essentially given birth to a virtual town square, where anyone with an internet connection and an interest in civic affairs can shape their communities. But as with any growing enterprise, there are difficulties ahead. Here are the top three challenges for 2011.

3. Data Amalgamation/Formatting

While some data sets are easily transferred into PDF format, a slew of older forms of data are decidedly analog, requiring significant amounts of effort in order to parse them into readable electronic form. One of the greatest difficulties facing open data portals is getting all the data out there, and designing systems which in the future will make data entry and filing far faster and simpler. Most civil servants are enthusiastic about sharing their data, but they still have to overcome the initial hurdle of developing systems and linkages which make that process efficient.

2. Proprietary Internet

In mid 2010, Google and Verizon announced a plan to develop and begin charging for a two-tier internet. While this has not yet come to fruition, the development of such a system could impact open data by making it far more costly to access information. Currently, one of the benefits of the open data movement is that all information is freely available. While many countries, including Canada, have pledged to maintain a free internet service for accessing public information, proprietary technology could eventually hamper the efforts of open data designers to distribute information efficiently. Those who are passionate about maintaining and expanding open data to other cities and countries should actively protest any developments in this direction.

1. Publicity

Anyone who is involved in open data/open government can forget that many other unaffiliated people are currently clueless about what has transpired in the last few years. However, each citizen should be aware of the information that is available to them. Major open data portals would be wise to invest in some publicity to inform their citizenry, and continue to inspire similar movements in other countries around the globe.

Obama Opens Up

Obama - Standard of Openness

While the White House has been derided for being too removed emotionally, its contributions to the Open Data / Open Government movement have been truly astonishing. One of the president’s first executive actions was to draft a memo calling for greater transparency in government relations, followed shortly thereafter by a law making it legal for government agencies to stage public meetings over the internet. In his recent spate of meetings in Asia, Obama took time to emphasize the importance of open government in a meeting held in Mumbai, India, a country whose technological capabilities currently outpace its entrenched class outlook.

The president’s attention to this matter in a political climate that at best could be described as “intense” is admirable. As a budding technology, open data has already shown considerable promise in allowing individuals greater input into how their communities come together and function. It’s unprecedented for a U.S. administration to put so much effort into making this type of access standard, especially in a global context. The White House has an extensively catalogued page on its official website dedicated to its efforts in open data / open government, which can be found under the heading “”

The difficulties of managing disparate agendas, especially in an increasingly interconnected globe, will undoubtedly continue to pose challenges in how we perceive the role of government in our daily lives. By sharing information openly, President Obama has encouraged a methodology that may ultimately prove beneficial in solving long standing infrastructural issues that are currently hindering the implementation of a functional global government strategy in the 21st century.

Although some government data must always remain classified, seeking to broaden the average citizen’s understanding of their place within their society may ultimately encourage people to take greater responsibility for what they can change, and place pressure on those agencies that are not achieving their stated goals. Perhaps Obama’s seeming emotional reticence is due less to his ‘elitism’ than his exhaustion at bearing the brunt of responsibility for every possible facet of life on earth. With the help of open data, it may be time for the world to start governing itself.

Can Open Data Replace Politicians?

Stephen Harper

In pre-Open Data/Open Government days, information about one’s local community was often difficult to find. In some countries, data was locked behind an old fashioned pay-wall. Politicians were empowered by the community to tackle issues and institute beneficial policies, mainly because they had the time to talk to everyone and could generate the connections needed to gather up information. In the Open Data age, this information is now increasingly available to ordinary people with very limited amounts of time. Thanks to apps and mash-ups, questions that would have taken at least a day and a few bags of cash to answer are now available online.

Which leads to the question: can open data replace the majority of politicians? Or, to be fair, has this happened already? It seems that the real politicians are the ones who are actually getting things done in the community (San Francisco’s publicly generated Apps), not the officials who award themselves $800,000 annual salaries by slurping up public funds (here’s looking at you, city of Bell, California).

• What Are Politicians for, Exactly?

In theory, politicians function as the wise arbiters of conflicting agendas. Farmer A wants to tap into a creek that runs along his land; the city wants to use that land to build a freeway to increase city commerce. Who gets the use of the land? A politician is supposed to find the common ground between these two parties and work out a compromise that allows Farmer A to water his crops, and the city to increase traffic and reduce congestion. Except that politics is now conducted mainly between Abstract Colored Territory Number 1 and the Retrogressive Angry Uninformed, leaving Farmer A with a bunch of unanswered emails.

Naturally, politicians are motivated by the size and complexity of the region they are representing. Asking any one person to be responsible for the needs of two million or even two hundred thousand distinct individuals is inherently ridiculous, which is why “the system” is in dire need of an overhaul.

• An Outsider’s Guide to “The System”

Open data is only a few years old, but already people are utilizing it to create a better quality of life for their communities. This quality of life issue—which seems to be the motivation behind any grass roots political group—is invariably trampled by the realities of “the system.” But what is the system, and why is it needed?

The system is a group of professionals who understand how to manipulate each other and other motivated players to achieve their own personal goals. In international affairs, this set-up is necessary; but for virtually anything else, it produces weird policies that read more like in-jokes than actual solutions.

This cosseted “system” generally lives apart from the people they represent. Given enough time, say 20 consecutive years in office, it’s understandable that a person’s thinking becomes hindered by the limits of their insular realm. Open data, on the other hand, is organized by people directly connected to that data. People who work with open data tend to live in their city or region most of the time, and are generally not seeking any reward except the solution to their individual civic problem. In this new world of open data, traditional politicians are expensive and obsolete: they provide nothing that can’t be dredged up by a free iPhone app.

•Leaders vs. the Clingy, Corrupt, and Unfulfilled

It should be noted that people could not have started to utilize open data in such a way without the enthusiasm and hard work of civil servants and some very progressive elected officials. The British government earns particular points for opening up its doors to technology; the Prime Minister is aware and supportive of the Open Data project.

Actual leaders—people who can inspire other people to take action—are always in short supply. Anyone who demonstrates that rare combination of charisma, intelligence and hammy compassion should seriously consider public service.

But all the other lackeys and back-scratchers and people who weren’t pretty enough to be a professional model but really need extra attention should find alternative careers that do not involve crafting legislation. With open data, people now have the technology they need to govern themselves. The power-hungry need not apply.

The Top 5 Open Data Portals

Top 5

How do you spot an excellent open data portal in a field that is still largely in its beta phase? Although the technology is still developing, clear leaders are already starting to emerge. We selected these five sites based on criteria including design, accessibility, and availability of information. We then focused on a feature or trend that distinguishes each particular site from its peers. In an impressive finish, Canada boasts two of the finest open data portals worldwide.

The World Bank’s ability to concisely encapsulate detailed data in a Google ‘one-box’ format places it in a category of its own for achievement in data visualization. However, unlike most open data portals, the World Bank has a far more regulated data set, with decades of experience gathering and collating the data. Despite this advantage, the data display is still magnificent.

The city of San Francisco’s open data site is notable for the impressive amount of useful apps developed by the site’s users. From recycling center finders to health inspection scores for restaurants, the city’s data has been eagerly and cleverly parsed by the community.

Similar to the World Bank’s efforts with data visualization, the city of Toronto’s recently launched open data site has placed an emphasis on creating ‘mash-ups,’ which are conglomerations of separate data sets that, when combined, provide useful information. A study of a city’s population mashed-up with local school building density, for example, would provide information about potential overcrowding in schools.

Known throughout the world for its innovative open source coding, is impressive not only for its sensible display, but for the scope of the information on the site. Attempting to present a kingdom’s worth of data in a format that is both engaging and useful is, to put it mildly, challenging.

Cited by the World Bank’s Neil Fantom as a site that helped guide his design process, the city of Edmonton stands apart for its remarkably intuitive catalogue index. This grounded design sets up the site for easy expansion in the future, and hopefully will serve as a template for other open data portals.’s Richard Stirling In Conversation

Since went public in January of 2010, it’s attracted worldwide attention for its usability, its user-friendliness, and its underlying mission to provide citizens with the information they need in the format that they need it. British civil servant Richard Stirling, who inadvertently has become one of the more recognizable faces of the open source/open data community, took some time to talk about the challenges of data presentation, including curated versus uncurated data, the most efficient methods of presentation, and app contests. In our conversation, he also briefly touched on why chose Drupal over Joomla, and the wonderful unpredictability of what the future may bring.

JI: I wanted to ask you about data visualization. Would your ultimate goal be to present all of your data in form that has a signature look, like the World Bank?

RS: It’s not something that we have as an explicit goal for all of the data on the site. If nothing else then what we’re doing is dealing with lots of different types of data sets. We have some data, like National Statistics, which is quite well curated and has pretty good standardization and there are some common things which you can pull together. But we also have some other stuff, like admin data, it’s pretty noisy. It wouldn’t fit into that visualization mechanism. We have real time energy data coming from the headquarters buildings of the government that’s surfaced on the site. Combining those two things into one viewer would be really difficult and a bit of a hard challenge. What we are focusing on is how we can make the whole process of using much easier, much simpler. Some of that will be around making the data easier to explore and navigate, and bringing some of the serendipity back in. If you vaguely know what you’re looking for, it will jump out at you. Additionally, we want to make it easier to show what is on that data. That might include some visualization of the data sets with ways of giving a sample of the data that’s there, or indeed surfacing some of the good work that other people have done. We have an app gallery on the site precisely for the lay user to get into the data set. We’re working on how to tie together the data set and the applications.

JI: You ask the public to submit their applications in terms of displaying data. What is your favorite app, or at least one that surprised you with how it tied together different information strands?

RS: Obviously, they’re all my favorite. The one that was not something that I expected is an application that lets you get a bit of sleep or work done while you’re on public transport. Essentially, your phone knows where you are. We published the locations of all those public transport nodes, including train stations, bus stops, that sort of thing. Really simply you tell your phone where you’re going, and it sets off an alarm 8 minutes before you’re due to arrive. I do a lot of train travel, that’s invaluable, but certainly, not something I had in my mind originally.

JI: In terms of open data sets there’s been, practically every day, another country or region in the world that is trying to put together a new open portal. It seems like your site in particular gets cited as an inspiration for them to move forward. Is it fun for you to watch this international proliferation of open data? It seems like you’re turning into a leading figure in that movement.

RS: It’s fantastic to see so many different countries opening up their data. I spotted the other day, I think it’s Kenya, has an open data site, which is something I hadn’t looked for, and it’s good. I’m happy. One of the things we’re trying to do with our site is be completely open about what we’re doing and open up our code. So if anyone wants to follow our particular design pattern or have a look at the code to see how we’ve build stuff, they can do. The front end runs on Drupal, the back end runs on an open-source registry called We’ve open-sourced the code that joins those two things together so there’s a Drupal module that queries CKAN and exposes it to the Drupal. That’s open and available on our blog and will be on

JI: Why Drupal and not Joomla, for example?

RS: It seems a long time ago now. It was just a case of we did a quick feature comparison. Drupal came out on top on that. We had some experience in deploying it from within the team. It seemed like the right fit.

JI: In terms of open source code, do you think that will enter another phase of innovation or do you think people will start to try and carve it up and make it proprietary?

RS: The history of open source is one of continuous innovation and development, with people always trying new business models and putting things together. I don’t see the rate of innovation slowing down. You look at where Linux was ten years ago and where it is now, I certainly don’t feel like the pace of innovation of open source has slowed down, if anything I feel like there are more people involved, and the pace is picking up.

JI: Do you worry about trying to meet longer term goals? Do you have an idea about what the site will look like in 10 years, or you’re just open to whatever comes in and excited about that journey?

RS: We think about the longer term in terms of what are the strategic problems that need to be fixed. What is the role that we should be aiming to do well? At the moment the development focus is very much on how do we make people’s lives better now. What’s the problems we should be focusing on now: how do we improve our offering, how do we make people’s lives easier when they want to find data, when they want to work with data, when they want to see what other people have done with it. When they want to surface their data to the public, how do we make that journey easier? Those long term goals are about making it findable, usable and opening it up people so they can create good things. In terms of setting out on the wireframe what the site would look like in ten years, that would be very challenging, yeah?

JI: [Laughs in agreement.]

RS: If you were trying to sit down ten years ago and set out what the site would look like today, I’m not sure you could do that. Or if you did, it would look completely different. For a start, you wouldn’t have people in there. There wouldn’t be quite so much focus on joining up the user base, as we’ve done at the moment.

Show Me the Money: Financial Transparency


Open data has spread to the Rockies. Colorado, infamous for its mountainous peaks and occasionally controversial politics, has become a purveyor of open data, with a group of local citizens known as Colorado Smart Communities assembling to create The site features groups of data relevant to the concerns of the citizenry, including a list of liquor licenses, road construction zones, bike lanes and financial transparency. As with all open data portals, the character of the community is on full display.

Perhaps the unintended result of so many open data portals opening all over the world is that truly global concerns can now be literally itemized. If a survey were to be taken of all the open data portals currently operating, a list could be made of the concerns which affect all cities and governments, regardless of geography or cultural history. Luckily, we are prepared to make an unofficial itemization, based on our survey of open data portals.

Unsurprisingly, the most frequently listed item is the issue of financial transparency. From San Francisco to the U.K. to Poland to the World Bank, issues of how money is spent and who receives it have become a motivating force in opening up the flow of information. Many citizens want to make better use of pre-existing budgets, and identify groups or associations who are abusing the public trust. This emphasis on financial transparency is especially relevant in a slow economic period, when capital is quietly warehoused in large reserves as power structures slowly shift to accommodate the new infrastructure. Each of the major financial power houses is waiting to see who will move first on the economic stage before committing their capital; however, no one is yet making a move.

Open data sites help ordinary citizens create the quality of life they want for their communities, without the interference of profiteering moguls or corporations whose chief interest is in accumulating more lucre. Growing an integrated, functional community is not about a formula. It is about building relationships between idiosyncratic individuals, each with his or her own skill set. Communities that welcome and support diversity thrive; those that focus on a single-minded pursuit eventually become ghost towns.

Is It Time for Corporate Open Data?

Corporate Transparency

Governments have begun to benefit from open data portals; will corporations follow suit? Gap, the ubiquitous clothing retailer, recently encountered the wrath of the virtual consumer, due to a poll about a proposed new logo on Facebook. The heavy negative reaction to the new logo prompted the company to pull its suggested redesign, and offer statements of apology for “going about the process the wrong way.”

Prior to the internet, corporations rarely retracted business decisions unless there was a lawsuit or a massive letter writing campaign. While Facebook has more in common with an instantaneous letter writing campaign than a formal legal action, the company’s apology is curious, as if Gap is somehow admitting that the only way to conduct business is through public opinion polls. Admittedly, consumers have always voted with their dollars; but have we reached an era where they can now simply vote online?

In many places, corporations now act with the power formerly accorded to nation-states. Since officially sanctioned nations are now successfully exploring open data portals, is it time for corporations to start thinking about integrating greater transparency into their business processes? In the same way that ordinary citizens have come up with applications to make better use of public data, perhaps it is time for corporations to open a similar channel for the public, to keep both the corporations and its consumers from drifting too far apart in their goals.

The old model of capitalism emphasized a conquer and destroy outlook. When taken to its end stage, this model results in creating pockets of hyper-wealth for a few dozen people, and crushing economic conditions for the remaining 6 billion. Perhaps new corporations should strive less to conquer and destroy than to develop and collaborate, resulting in healthy profits for the company, and healthier living conditions for everyone else.

The idea of employment itself could be radically reconfigured; a roving ‘skills app’ which allows people who possess certain abilities in a given geographical area to list themselves as looking for work. Corporations may improve more than their bottom line if they choose to embrace an open approach to technology and data.

Poland’s Open Data Project: Transparency Camp Polska

Transparency Camp Poland

On November 18, 2010, Poland will host “Transparency Camp Polska,” a conference and collaborative forum aimed at launching an official Polish open data site. The conference has been organized with flexibility in mind, with three parallel stages featuring numerous guest speakers and presenters in tandem with conference participants. The idea is to create a place where individuals who are interested in creating a greater level of transparency in public life can gather and exchange ideas. Those who currently work with public sector information in Poland and from other parts of the world are encouraged to attend.

The conference’s confirmed speakers constitute an array of the world’s top experts in open data, including Richard Stirling of The decision to hold the Transparency Camp followed a discussion which took place in March with several of Poland’s top government officials. The usual issues surrounding roadblocks to open data were identified, namely lagging technology and recalcitrant mental attitudes. Polish politicians hope that open data will create greater citizen support for the government’s administration of public affairs, while technology experts are hopeful that greater transparency will aid in improving the quality of life.

The decision to attempt to create an open data portal seems well-timed. Poland is something of a breeding ground for innovation, especially in the arts. Aside from Wassily Kandinksy, in the last few years Poland has become a quietly simmering hotbed of fomenting talent. Both David Lynch and Frank Gehry had or have ongoing projects based in the country which interact with local talent.

Aside from making government processes more transparent, open data portals are wonderful because no two are alike. Each one is a reflection of the attitudes and goals of its particular region and environment. Although the Transparency Camp is attempting to gather the resources it needs to establish the most basic building materials, the world stage will undoubtedly benefit from having such a creative and innovative group of people try their hand at displaying publicly collected data. Each open data portal not only contributes to the health of its regional community, but advances the sophistication of complex data display globally.

The World Bank Open Data Portal

The World Bank

In the era of Google Instant, internet real estate has become increasingly valuable, partially because it is so ephemeral. A single key stroke can instantly dispatch a page of results. In order to remain relevant within the “blink and you miss it” parameter of popular information research, the need for concise yet informative displays of complex topics such as poverty and climate change is vital.

The World Bank’s open data portal is a marvel of efficiency and intelligence. Officially launched April 20th, 2010, the site is exceptionally easy to navigate, and provides in-depth information in a format that is easy to comprehend visually. I spoke with Livia Barton, Jeff McCoy, and Neil Fantom, who each played a role in developing the World Bank open data portal. We discussed their favorite open data portal, the Google one-box, and the upcoming ‘Apps for Development’ competition.
I began by asking for a brief overview of the development process.

Livia: The bank has always had a set of 54 indicators freely available to the public. These 54 indicators have been on our website on our for years, but also have been sold in a publication called the “Little Data Book.” It’s a very popular publication among the academic and statistician audience. We started to talk to Google about two years ago. I have to give the credit to Jeff, who saw that the U.S. Census was putting their unemployment rates into the Google search ‘one-box.’ We thought it would be really cool if we had our population, G.D.P., CO2 emissions, and other similar statistics in the same Google one-box.

Jeff: A couple of years ago we put out a first version of an API with the 54 free indicators. That sort of caught Google’s eye, that we were reaching out in that way. It was a much smaller set than we’re providing right now.

JI: How difficult it is to maintain the portal on a daily basis, in terms of making sure that everything’s accurate and updating information?

Neil: Having the portal is not difficult to maintain because it draws data directly from the database, through the API.

Jeff: All of the country, topic and indicator pages that you see on the World Bank site are all driven directly from the API. As long as those databases are updated, our pages are okay.

Neil: This leverages the business model that we in the data group have had for many years. We had these 54 indicators that were free, but we also had about another thousand indicators that were available to use through subscription. That data set was developed to support a publication. Over time, of course, it’s interesting that the paper publications become less important than the electronic media. In fact, the data set has become more important than the product that it was created to support. We’ve always put a great deal of effort into maintaining that data set. We don’t change it on the fly; we do it twice a year, once on April, and once in September. If there are things that are wrong that we see, of course we update them immediately.

JI: In terms of visitors to the website, the stats said you had 132,000 unique visitors in the first month. Has interest remained high since the website launched? Has it increased?

Livia: The traffic we are getting from the Google one-box, particularly on the population indicator, is the number one performer. Since launching, it’s maintained about a 90% increase over the previous site that was there instead. It’s steadily increasing, and we’re seeing more interest with the upcoming Apps for Development. We’re launching some great new features which will allow people to build a chart and use an embed code and put it into a blog post or story and write a good story behind it.

JI: How long have each of you been working for the World Bank?

Livia: I came from the marketing/ad agency world. Moving to the world bank was more of a strategic move. I wanted to sell fewer Dunkin Donuts and reduce poverty instead.

Jeff: I’ve been here about a dozen years, lots of different roles over the years, including web development.

Neil: I’ve been at the bank 8 years. Before I joined the bank, I worked at various countries in statistical offices overseas. I was in Africa for almost 10 years, and I worked for a European office, and the U.K. statistical office. I wanted to come here because a lot of the data that statistical offices produce winds up here; it seems like the Google ‘one-box.’ For me, it was a natural progression.

JI: How have governments reacted to having information so widely available? Have they had any reaction to that?

Neil: We haven’t had any negative reaction. People who have seen it are very excited by it. We’ve been encouraging governments to go down a similar route. We obviously learned when did this from other organizations like the U.S. and U.K. You don’t see quite as much of this in the developing world. But it’s something we’ve been working with other countries to promote.

JI: As web developers, do you have contact with other web developers for other large government open data portals?

Neil: I’ve had a lot of contact with the U.S. government on this, specifically on their open data portal. We listen to what they’ve told us. Andrew McLaughin made a post about our data portal. He was excited about the way we’ve done it.

Jeff: Locally here in D.C. there are some active data open initiatives with the D.C. government. We’ve had some contact with them to see how they’re using it, sort of in a local context. It’s interesting to see the applications that they’re using.

Livia: We’ve seen such great things come from the Canadian government. We have high hopes.

Neil: The one really nice open data site which we used a lot, in fact we were very impressed with their terms of use, is Edmonton City Council. They use an open standard for all their data sets. It’s a really great site.

JI: Livia, you were talking about the Apps for Developers?

Livia: We have the FAQ out there which informs the public that the competition is coming up. We’ll do the official announcement on October 7th that will announce all the rules and prizes, the deadlines and the categories. Our goal here is really for the people to take a piece of the World Bank data and combine it with other data sets to produce a tool, application or software app, or anything of the like that helps the world to learn more about the state of the economy and the state of development. And even improve it.

Neil: We had a get together with some developers last week, and it was interesting to hear the sorts of things they’d like to do with our data. I think we’ll see applications that take the data and visualize it on a global scale. I think we’ll also see a lot of people very interested to do things at a national scale. They’ll try to discover new data sets that are available nationally, and try to merge those with the data sets we have in order to make an impact on people’s lives day to day. I think we’ll see a lot of mobile apps, particularly because they’re becoming more relevant for Africa. I think there will be a lot of incentive for people to find new data sets at a national level around the world. I think that will be one of the exciting things that comes out of this challenge.

Livia: Part of our open agenda will be more publicly announced on October 7th. We’re hosting an open forum which is a live web cast. We’re doing a series of announcements about open data and apps for development and a whole series of things. We’re actually opening up quite a few of our research tools. It will be an opportunity for people all across the world to interact with each other.