Reconstructing the decision-making processes of a city council based on references between documents

Co-citation of documents is found by identifying agenda items and entries containing (near) duplicate documents. Duplicate detection is performed by finding documents with the same filename and/or the same displayname in the metadata fields. For this set of documents it was ensured that all documents have a similar filesize, defined as not deviating by more than 5% from the average filesize in the set. This resulted in 43507 unique documents. Before grouping, filenames and displaynames were normalised by removing file extensions and any trailing white spaces. The requirement for similar file sizes prevented cases where a generic filename (e.g. ‘Proposal.pdf’) refers to completely different documents.

The algorithm models each agenda item as a small timeline, and for each (near) duplicate document a new timeline is created that merges all timelines that co-cite it. If the original timeline contained multiple duplicates, then the same timeline will be merged into multiple new timelines. Therefore an iterative process is started to identify and merge timelines that contain new duplicates. This yielded 3,006 timelines that are especially strong at showing the history of individual policy proposals, including the weekly meetings and documents.

Figure depicting a vertical timeline where each item is a meeting at a displaye date. For each meeting, the user can fold open a view displaying the documents discussed during this meeting. Users can click links to the council information system they use, to the minutes of this meeting, and to the video of this meeting.

Figure 6 shows the processing pipeline for textual references. There are three types of textual references, which we define and extract as follows:

Figure displaying that first text is extracted from a document, and then all references are found. This can be done by ID (by recognising the standardised URL format), and by identifying document titles (which authors denoted within quotation marks), and by extracting dates (by using the HeidelTime software). All documents matching these documents are retrieved, and if multiple documents are retrieved from references in the same sentence, we combine the different types of references to disambiguate the target document or meeting. Then the original document and identified can be linked, and included in the same timeline.

References by ID are very specific, and refer to a unique document. ID extraction is done by detecting hyperlinks in the text using pdfminer in python [19]. The contents of the URL indicate what type of meeting or document is referenced, as well as its unique ID.

References by title are less specific, as documents may share titles or have non-descriptive and generic titles. Title extraction is performed using regular expressions to identify text strings enclosed with quotation marks (e.g. ’ " ‘). As document titles were usually shorted, we tested if each string is a substring of an existing document title.

References by date are the least specific, as sets of documents are typically submitted as a batch on the same date. Additionally, it is ambiguous whether a date refers to a document or a meeting. Dates were extracted using HeidelTime [20]. Relative expressions such as ’yesterday’ were normalized to the document's upload date. Because documents are uploaded in batches, the exact target is typically ambiguous. Additionally, it can be unclear whether a date refers to a document or a meeting.

Ambiguous references (e.g. multiple documents are uploaded on the same date, or share the same title) are disambiguated to the target document using other references within the same sentence. We ignore references that we cannot disambiguate to a single target to maintain a high precision (rate of true positives) in our timelines. This increases user trust, and it prevents the co-citation approach from including irrelevant documents.

The resulting timelines typically show the progression between council decisions, on a time-scale of months to years. The textual references allowed us to merge the 3,006 timelines found through co-citation into 2,751 timelines. There is an overlap in the links found by both types of references, but the co-citation approach is particularly adept at finding short-term connections whereas the textual references are better at finding long-term connections. On average the timelines consist of 4.55 agenda items and entries, spanning a period of 12.5 months. Two outliers spanned 64 months and 32 months. Two others spanned less than 1 month.

We evaluate what proportion of in-text references we successfully extracted from documents, and how many of these we disambiguated successfully. We annotated all textual references in ten randomly chosen manually constructed manual timelines (as other document genres contained fewer references). These annotations were compared to the extracted references.

Table 1 shows that we successfully extracted 91% of references from the documents, but that we were only able to disambiguate 39% of references to an exact document. References by URL were identified and trivially disambiguated, but a part of these include dead links (to pages no longer in the dataset) or link to pages outside of the dataset. References by date and title often refer to a subset of documents, including the target document. In the following user study we investigate whether the references that we found resulted in relevant documents.

Three main themes emerged: inclusion criteria; completeness and conciseness; and overview perspective. These themes and the main points are briefly summarized in Table 2.

Inclusion criteria [RQ1]:. In this setting it was not important to find whether a document was relevant to the topic or useful to the task. Instead, a document should be included in the timeline if both the document and the associated policy proposal can trace their provenance (as introduced in section 2.2) from the same council decision. This provenance is not explicitly tracked, but it is of such importance that authors of council documents are consistent (and trained) in referencing to previous decisions and documents. Documents typically state why the council has to read them. The importance of council decisions in particular were reflected in every interview, as shown e.g. when P3 stated “[this document] is important because the council apparently has thoughts about this". P5 termed the document types which contain council decisions as “milestone documents", and references to these milestones when persuading other civil servants to take their requests seriously (e.g. in emails).

Participant P4 illustrated the importance of provenance over relevance, stating that “Sometimes the council asks questions about a different topic". In these cases “There can be a whole [internal] discussion on [which civil servant] should address that question, but we don't want to record that". Such semi-related documents need to be included “in the same package", to keep track of “who is responsible for it". P9 was reluctant to use relevance as an organising principle as “[something] can be relevant, but it's a side issue", and “there comes a point where everything is relevant to everything".

Participants rated nearly all documents found by our approach as relevant, and no difference between any document's relevance or usefulness. The only documents that were not relevant were 1) a single entry that was incorrectly included because the duplicate detection malfunctioned (and the co-citation approach included an non-duplicate document), and 2) a few agenda items during administrative meetings. These administrative meetings should not be included in the timeline, because these same items were discussed during a separate meeting later that week. After excluding this type of meeting from the generated timelines, 97% of timeline items were found to be relevant. This precision score reflects that we do not need to account for weak links between documents, as references between documents are only created when they are directly related, according to staff with expertise. Participants always agreed in their assessments of items, with the exception of P5. P5 only considered council decisions as relevant, and other documents (those of an informative nature) as semi-relevant. All participants gave input around this theme.

Completeness and conciseness [RQ2]:. Participants noted both the importance of conciseness (P6: “Less is better") and completeness (P4: “Maybe I'm too careful, but I want the complete picture"). P5 described how these relate: “There is information at two ranks. Decisions are at the first rank... informing [documents are] at the second rank". They suggested a layered overview, which initially shows a timeline of the decision layer, which can be expanded to also show a provenance layer by toggling a button. The decision layer displays the main items, and expanding the provenance layer displays the sub-steps. This is illustrated in Figure 7 The decision layer encompasses the decisions made by the council, as recorded in motions, policy proposals, formal questions to alderpeople and formal promises by alderpeople. These result in duties and activities from municipal staff, which in turn lead to documents whose provenance is displayed in the provenance layer.

The figure has two parts. In the first, we see a timeline that contains two council decisions and an button to expand the view. In the second, we see the same timeline but now we also see the meetings that occurred in between those council decisions. We also see the documents attached to each meeting.

P5 suggested that the decision layer should support people who “primarily need to know what the council decided, and the current state of affairs". As such, the decision layer should include which decisions have been resolved, and the latest information on those unresolved. P5 noted that such information also serves as a form of accountability, where the staff shows “we haven't been idle". Note that ‘latest information’ refers to the latest official document sent to the council, as more recent working documents “are usually not ready yet to show [to the council]" (P10). Conversely, the provenance layer is more comprehensive, providing background information and showing how individual decisions were made. P10 noted the history of individual decisions can be important “when something seems awkward" about them. All participants had responses about this theme.

Overview perspective:. Although some decisions follow a linear process, P8 highlighted that “sometimes multiple [time]lines converge, and I can't see if [these documents] are in the same line". Therefore, overviews should display the different lines, and how they interrelate (akin to the metro maps timelines [18]). P6 shared their experience tracing document references without an overview perspective, noting that “[they] couldn't see the forest for the trees". P10 cited an example where this approach took them 2-3 hours to understand what lead to a policy proposal. P7 appreciated that a timeline draws attention to time gaps between documents, which they considered important clues that “something might be missing" and could prompt further investigation. P4 commended on the robustness of the timeline against changes in the keywords in documents, highlighting an example when “the building changed names" which would be less obvious when searching by keywords. Seven participants had responses about this theme.

Comparing types of timelines:. No obvious differences emerged between participants who used different types of timeline (manual, generated, combined).

We reflect upon the practical implications, and then the theoretical implications of the study. We find two themes with regards to practical implications:

How to structure council decisions:. The importance of provenance explains why authors of council documents are consistent in referencing past documents in the text of their documents. This practice ensures that we can generate timelines based on references to documents (RQ3). Although previous work typically generated timelines based on document similarity, our approach can leverage the provenance information that is provided explicitly in the council's work processes. The importance of provenance in this setting suggests that similarity based approaches are less appropriate for decision-making processes. Provenance ties into the main responsibilities of council members: shaping policies that generate the municipality's activities, and overseeing the execution of those activities. Hence a timeline should be presented as an overview with two layers: a decision layer that conveys these two things (decisions and current state of activities), and a provenance layer which includes more comprehensive information.

Investigating decision-making:. To quantify the value of an overview perspective, future work could investigate how much information is available at the moment of making a decision (with or without the overview), and how the availability of information affected the decisions that were made. The establishment of reliable overviews of council decisions also facilitates further research on the nature of those decisions. For instance on identifying critical decision moments [4], and whether these moments are more likely to arise when the topic is discussed in information sources like the news.

With regards to the theoretical implications, we reflect:

Generalisation:. The proposed method depends on the process of how decisions are made, as well as how they are recorded and archived. The decision-making process is similar for many governmental organisations at the municipal and (sub)national level in Country, although smaller organisations may record these in less detail as they have less administrative staff available. The approach likely generalises to many (country) governmental organisations, especially as most municipalities in the country use the same two meeting planner systems and hence already structure their data similarly.

The method itself is domain-agnostic, and could be adapted for similar decision-making processes at other organisations. Potential future work could investigate whether this approach could be adapted broadly, specifically for use in organisations that have integrated software for their email, calendar and content management (e.g. organisations using SharePoint). Although these processes will be less structured and recorded less accurately, a timeline might still be a useful way to organise and revisit information.

Limitations:. Although our approach achieved a high precision in retrieving relevant documents (97%), it is only an explorative step towards generating overviews of policy information. The main limitation is that, although 91% of references are extracted from the text, only 39% of total references find exact matches. Future work should improve the detection of true positives, both in the reference disambiguation (e.g. using the domains given in document metadata), and in the duplicate detection. One approach to disambiguate references and identify missing documents is to involve a domain expert in the loop (RQ4).

As the domain experts reference the most vital documents known to their expertise, we assume that our timelines cover the most important moments towards a decision. Future work could include documents that did not directly lead to a given council decision, but might still include contextual information. Improving the recall of these timelines could be done by first generating high-precision timelines, and using these documents as the basis for content-based recommendations. A domain expert could prepare a timeline for council members by first disambiguating references, and then expanding the timelines based on recommendations.

A limitation of the experimental setup is that each user only used one timeline. Although no obvious differences between the manual and generated timelines emerged, properly investigating these differences would require a comparative study.

Finally we wish to highlight the limited number of participants in the study (n = 10). Although this is a useful sample size for an iterative design process, later stages in this research direction will require studies with a larger participant pool to quantify how such an overview aids in task performance compared to searching individual documents.