On Friday it was announced that further development on the data platform will come to a halt in the new year. So this is the last missive from your parliamentary data mouses. We’ve had a decent run. And, I think, spent three years doing some of the best work in the public sector. There is a good deal to be proud of, and regret that we can’t continue.
The final jigsaw piece of Anya’s work with the Procedural Publishing Unit clipped into place this week when Monday’s House of Commons Order Paper was the first in history to contain actual anchor links to business items. Like how the web works. It all seemed to go down rather well. Though we would like to point out, contrary to some rumours, we’ve long been aware of the power of hypertext. It means we’ve now covered off the Order Paper, Future Business and Votes and Proceedings. Work like this is important. No order was dispatched from on high. Just some people chatting and exploring ways to make a thing just that little bit better. Somewhere between self evident and profound. Top work Anya and Mark.
Way back in September, team:Samu headed off to an Ontotext meetup to chat about ontology engineering, the architecture of our Open Linked Data Platform, the tooling around it, its readiness for production workloads, monitoring and all other things related to building resilient semantic web infrastructure. Wojciech, Mike, Chris and Matthieu all had their say. Ontotext have now published a video from the event which is a pretty good record of three years of solid work. We suspect Samu might be proud of the team he built. We’re proud of it too.
Elsewhere our Samu was on the receiving end of a Twitter question about tallying counts from the data.parliament divisions API. He went in with a reply that managed to pull off a very nice triple negative. Michael chipped in with something a little more pragmatic. We aim, to the last, to be helpful.
Tuesday saw Anya, Samu, Robert, Wojciech, Mike, Matthieu, Jianhan, Oli and Michael meet with Oliver Heath, a professor of politics at Royal Holloway. Oliver’s working on an ESRC funded project to map census data onto constituency boundaries from 1851 to 2018 and is looking for organisations to partner. We volunteered to help where possible. But that was when we still had a data platform. Which we no longer do.
Robert and Michael had another meeting with Sarah, Simon and Laurence about progress with the formal body, formal body affiliation and business item models. All in the general direction of trying to help build web pages for committees. Although they’re not completely convinced why. They’re still in search of a label for the overlapping group of groups that covers formal bodies, answering bodies, laying bodies and government organisations. Again they ask why.
On Tuesday, Anya, Robert and Michael popped across St James’s Park to the Lego castle that houses the Parliamentary Computational Section for a chat with Bex, Christine and Matt. Together they attempted to persuade Robert that saying that a work package could follow a procedure was not untrue or indeed misleading. An hour later they emerged with a browbeaten Robert and a single new predicate. But happy.
Following on from Chris’s monster query last week, Samu made us a sweet little URL shortener. So the work of Chris’s SPARQL spanner can now be invoked by api.parliament.uk/s/b89efe80 and Samu’s epic procedure schema visualisation query as api.parliament.uk/s/e756472e. Like much else, the shortener uses Pugin, the marvellous beta front end courtesy of Usman.
Whilst releasing the URL shortener, Samu took the opportunity to update the search service to also use Pugin. Which is here for your delectation.
On the subject of search, more things on the service are now nicely addressable since our Samu added HTTP accept header based content negotiation for search results. For the less CURL minded it also works if you tack on a file extension. Which means you can now get back JSON, Atom, RSS and good old HTML. Beautiful. Properly nice.
The OpenSearch document reflects these options and adds some additional metadata for good luck. As does the OpenApi document, which is itself content negotiable for lovers of YAML or JSON. There’s even a public UI for interacting with the service which uses the OpenApi definition. Open your browser. Have a tinker.
Samu would like to add that none of this work required any changes to the website because both the data platform and the website use open standards to exchange search requests and responses. The OpenSearch standard has a service description mechanism, allowing the API publisher - team:Samu - to tell consumers - team:Jamie - how it all works and what to expect. The work here is merely to embrace the standard. Past that point, changes become trivial. Which is why we love open standards. Filthy for them, some might say.
On a final search related note, Alex departed last week, but not before gifting us a set of improved search hints which are now live. Date hints also went live round about the same time. His changes can be found at api.parliament.uk/s/37b8dde7 because Samu’s URL shortener also works for github/ukparliament. Obviously. No amateurs here.
Team:Samu have also been busy building a vocabulary browser for use by the Indexing and Data Management Section of the House of Commons Library. And beyond. It’s been another collaborative effort with our Matthieu working closely with librarian Liz.
IDMS do sterling work subject indexing parliamentary material and tying together disjointed data. But to date the output of that work has only been half visible behind a search form. And not visible to the wider web. We also published the thesauri on data.parliament, but given that was just a list of subject headings and carried no indexed content, it was less than useful.
The new browser spans across the data platform triple store and SOLR - where the subject indexings live - so you can now browse parliamentary material by subject heading in a way that’s never been possible before. For those interested in approaching Parliament by topic, rather than by business or document type, it’s probably a better bet than the website.
The vocabulary browser is based on a lot of thinking and a lot of work over a lot of time. Matthieu has been beavering away on exporting our existing vocabulary, converting it to SKOS, importing it into a triple store and making the whole thing queryable by SPARQL. We think it’s not too far off being able to replace search material and in time could replace the internal only parliamentary search. And it could totally provide the backbone for topic pages on beta. Take a look at what’s been subject indexed under Brexit and we think you’ll get our drift.
It’s built on our standard technology stack and makes use of:
GraphDB, a standard compliant graph database engine supporting OWL inferencing and SPARQL querying
a standard ontology for expressing the basic structure and content of concept schemes such as thesauri, classification schemes, taxonomies and other similar types of controlled vocabulary in the shape of SKOS
Samu’s Dynamic graph implementation which he contributed as an extension to the dotNet RDF library
a standard, containerised .NET Core web application with fully automated deployment
another outing for Usman’s Pugin framework
a simple Lucene connector for search
It’s another clarion call for the benefits of having a team dedicated to working in the open and working with open standards. And long may that continue. As the Lego bricks were deployed, building standards compliant, accessible applications became more and more like intelligently assembling strong blocks of technology. And every new project benefited from updates to our stack. Though obviously that was last week. In a world with no data platform development, we’re less certain how this work continues to be supported.
Liz and Matthieu gave a brief tour to librarian Phil and his team who were pleased to see the vocabulary available in an accessible and good looking UI. And the adoption of SKOS seems like a good thing to everyone with a bit of subject matter expertise in knowledge management.
It’s also lending a helpful hand with improving our vocabulary, as Matthieu and Liz collaborate on tidying and pruning the hierarchy.
Every week, more brilliant work when librarians met team:Samu.
We could tell you. But we’d have to shoot you.
This week we’ve been mostly listening to Insight. It seems to match our current festive mood.