weeknotes.data-search

Week 19: 01-12-2017

Leave the Capitol

Your final reminder that our 5th Parliament, data and democracy meetup is on Wednesday 6th December, this time at ODI Leeds.

Speakers lined up are our Dan, Cristina Leston-Bandeira (Professor of Politics at the University of Leeds), Tom Forth (Head of Data at ODI Leeds) and Edward Wood (Director of Research in the House of Commons Library).

More in the way of interesting talks, nice people and free beer. Come along. Bring a friend.

International, continental, Parliament in the area

Dan didn’t get to follow up on the multiple international leads he got in Geneva because he spent lots of time talking to people about the website, corporate data, and dashboards (website like things that no-one ever clicks on for the avoidance of doubt).

Community

Robert popped out to see Libby Miller. Libby made a terrible music box. Robert supplied the box.

Michael met with Edward Wood and a PhD student from King’s who’s looking at social media engagement with Parliament. Several proxies for engagement were discussed.

Anya and Michael met with Stephen Betts from the NYPL to talk about libraries and archives and searching and browsing. Dan popped by to say hello.

Corpcomms

Dan and Robert met in real life (and not on Slack or indeed on Yammer) to talk about Yammer. A plan emerged. A ‘yam plan’ if you will. I for one am not asking.

Who is L Ron Hubbard?

Nobody knows mate.

One world, one web, one team

Aidan and Dan were both low on ceremony attendance this week. Because of circumstances. Michael went to several. Pazuzu was still un-summoned. Cthulhu popped by but just sat there yawning.

Michael gave his talk on web and information architecture to a variety of designers and product managers and etc. Many clouds got yelled at.

Anya and Michael gave the website team an update on progress with the question and answer model. It seemed to go quite well. Anya has offered to give a couple of the team a tour round the Q&A system.

Domain modelling

Anya and Michael continued to tidy up the domain model for questions. And indeed answers.

On Thursday they were joined by Ned who was very helpful around questions of declarations of interest and ‘tabling’ in general. Thanks Ned. They’ve now pulled out tabling into a separate model. Though it’s still in need of comments.

Anya and Michael also met with Robert to talk through work on the questions and answers domain model.

Angela and Michael did a little more work autopsying the House of Commons committee management database. They have 8 tables still to go. Coherence could not be said to be emergent.

Data platform

Jianhan continued working on the OData service which now supports most of OData query options.

Raphael continued with the Berlin SPARQL benchmark tests. He contacted the triplestore vendor to confirm some behaviour that was causing issues in the test.

Chris gave Robert an education on assorted graph issues.

Alex wrote a couple of scripts to aid the task of manually processing missing oral and written parliamentary questions. Currently a lot of questions aren’t being automatically processed due to a date issue within the question files. After having some issues at first, Samu helped to make the code more functional, and after a (nearly) complete rewrite the scripts were ready for every day use.

The first script works by scraping our public oral and written data sources to find the relevant UINs for a specific date. These UINs are then sent in JSON form with some other identifying data to MyJSON.com (a public JSON store + API). The second script then accesses this JSON data and compares the UIN values against those in the private RDF feed. It then automatically downloads and renames the missing RDF files, ready for processing.

Dia and Robert wrote a first draft of ‘a strategy note for search’ which they’re trying to align with the Digital Strategy.

Jamie, Robert and Michael met to chat about search and browse and URLs and subsidiary resources and stuff.

The first step in making Deposited Papers searchable happened.

Measuring things

Sara spent time analysing the A/B testing results for search. The analysis software has limitations when it comes to calculating certain statistical measures (median absolute deviation for example) or statistical tests (t-tests, power analysis). That said, it allows you to run R scripts which means we can do the analysis in a single environment. Apparently this is good.

Corporate data

Mat helped the finance team with the integration of the backend of the new online shop with sales processing.

He met with Sam Jones and provided all the details on the EPMO (something about enterprise and projects and management and offices) dashboard.

He also started work on a new feed from the HR system to ensure that job information for the staff of the House of Commons is correct. Still in HR land, he created a first pass data integration to correct email addresses. Noel and Mat are working together on the testing and will both do the release.

Mat met with the asset management integration team to agree on how requirements for data integrations will be captured.

Mat raised the fact that the House of Lords finance system does not have any integration services on. As part of delivering a fully automated integration they’d would have to be turned on and tested.

Capability

Julie, Sara, Mattieu, Chris and Dan went along to the Silicon Milkroundabout recruitment event. Lots of people registered an interest. Julie would like to say thank you to all who helped.

Julie and Aidan started the Dynamic Purchasing framework conversation.

Did the machines learn owt?

Not strictly speaking.

Raphael started a series of write ups on the NLP and supervised learning aspects of the data toolkit.

Michael tried to understand how much learning the machines might have done but the whole conversation turned into a taxonomy of data science and ML.

I think it’s safe to say the singularity will not start here.

Did anyone say blockchain?

These folks certainly did. At Parliament, not so much.

Strolls

Dan went for a stroll with Robert. This time it included stopping for lunch which isn’t something that Dan usually does. He does eat, for the avoidance of doubt. It was great.

Anya and Michael took a walk down to Tate Britain but forgot to eat lunch. It was also great.

Meta

Robert has suggested that weeknote numbering should really follow common standards. Michael agrees but is conflicted because COOL URIs DON’T CHANGE.

Things that caught our eye