Parsing webmentions

"Parsing Webmentions" by @adactio: http://adactio.com/journal/6495/ A step-by-step explanation of how to receive #webmentions and incremental steps you can take afterwards like: * displaying links to posts that mention yours * displaying such posts in their entirety with attribution Well done Jeremy.

adactio: this is awesome! http://adactio.com/journal/6495/

Jeremy has recently implemented Webmention on adactio.com, and posted an explanation of the small piece of code involved. I I love the simplicity of Webmention, and I love the Indieweb idea of connecting our conversations in the simplest possible way whilst still publishing to our own sites, owning our data. I intend to implement it on this site as soon as I can: both to test it out, and to offer a way of commenting without all the hassle of actually managing comments (sort of).

I do have one teensy tiny criticism though: what’s that little bit of tech-specific junk doing in the Webmention URL, Jeremy? “Dot PHP”? Cool URI:s don’t change, but publishing platforms and script languages do, right? Excuse me sir, your CMS is showing! :-)

In 2007 and 2011 I wrote a pair of articles where I tried to articulate my thoughts on what I saw as the dawning social web. The earliest article was about Google’s OpenSocial API, and the latter, about what I saw as Google’s new social network, built around profiles, rel=me and Buzz.

Neither came to pass. OpenSocial died a slow, painful death, while Buzz was kicked in the teeth, then taken out back and shot. But, what they demonstrated was Google’s attempts at socialising the web itself. Of course, this dream died when Google+ was released as a Facebook-clone and Google returned to only being an advertising company.

But, others have been continued working on freeing the web for us.

IndieWeb

Individuals like Tantek Çelik, Barnaby Walters, Jeremy Keith and many others are tirelessly working on ensuring that the content we create is recognised as ours, controlled by us and, in the end, owned by us. This work is best articulated by the principles of the IndieWeb.

One of the latest pieces of work developed by Jeremy Keith is webmentions.

Webmentions

Webmentions allows conversations, that would usually occur in comments, to take place on your own website. Pinging back responses to the original article allows readers to follow the discussion of articles and comments, while the content itself continues to be hosted on the respondent’s website and owned by them - fulfilling the IndieWeb principles.

Google Buzz performed this kind of aggregation and connecting. It allowed conversations to happen across the web and be followed in one place.

Webmentions picks up where Buzz left off and adds the ability to host the conversation itself to the mix.

A fantastic tool that I can’t wait to implement on this site.

@sandeepshetty @pfefferle cweiske Something new to consider: Jeremy Keith added a webmention sending form to his journal entries to help people who’s websites don’t support webmention already. Being able to test and use webmention through a human visible, interactable form is a huge benefit of using HTTP form encoded data.

We can make this an even stronger case by encouraging success and error responses to be full HTML documents with helpful copy.

See also * http://tantek.com/2013/258/t2/web-protocols-should-use-http-params-no-xml * http://adactio.com/journal/6469/ * http://adactio.com/journal/6495/

@benwerd Thanks! Credit for the inline webmention box goes to @adactio! http://adactio.com/journal/6495/

This site does not have a way for readers to post comments under each article. And I plan not to implement a comment section. Instead, now Parallel Transport accepts webmentions, so you can write and publish responses on your own place on the web and link it back to the original article here. In fact this very article appears as a response on Jeremy Keith’s post.

Comments & Feedback Most comments are useless. They do not contribute much to the original content. They’re mostly one-line quips about liking or disliking the post. No context, no feedback, no thought. Having a comment form makes it very easy to post a random comment on some website. Combine that with the anonymity it affords, comment sections are breeding grounds for trolls, unconstructive arguments, flaming, name-calling, and shouting matches. No wonder the bottom-half of the internet is so despised. Second; why this expectation that every blog must be a discussion forum? Public discussions can be had on social platforms like Facebook, Twitter, Google+. If you really have some response to what I write, you can use email or catch me on one of the aforementioned social site places. Or better yet, write your own post about it on your own blog, site, journal, social-network-thingie!

You write on your site; I write on mine. That’s a response.

—John Gruber on I’ll Tell You What’s FairIf your response, adds something of value to the original content, I’ll post it here. I have transfered many such valued comments over from my old blog. Not all of them agreed with what I had to say, but they were considered responses, instead of just ‘I like it!’ or ‘This is shit!’. Webmention So suppose you have published a response on your own site. Webmention is a way to notify me of your response. You send the URL of your response along with the URL of my original article as a POST request to my web server. My server verifies that the response post exists and that it links to the original article. Now, I can do something with your response. I could simply link to it below my article, or repost your entire response. I have chosen to take a middle-ground and display a little snippet with a nice link back to your original post. If all goes well, you would have responded to my article with one of your own, published on your own site, and still have a discussion that links back and forth. Since, most web-publishing platforms don’t support sending mentions automatically, following Jeremy Keith, I have a small form at the bottom of every article that you can use to send me the link to your response post. If you are interested in the details of how this works, take a look at the webmention spec, and the informative discussion at the IndieWeb. If, like me, you use and love Python, Panayotis Vryonis has written a good tool to handle webmentions. You can also see my own webmention code, which extends Vryonis’ webmentiontools to do a few more things. Feel free to make suggestions, test, extend and use it. And if you do, let me know. Or better yet, write about it and link it back here!

Related reading: Comments still off —Matt Gemmell Why I dont’t have comments —Seth Godin Why you can’t comment here—Panayotis Vryonis Parsing webmentions — Jeremy Keith. Also, read the wonderful webmention-ed articles at the end. IndieWeb Principles Comparison table of linkback methods on Wikipedia.

Tuning up my Little Piece of the Indieweb

Thanks to Jeremy Keith for providing this URL for testing h-entry markup.

Testing my automated webmention sending to posts I reply to. Fingers crossed.

Testing my grabbing of reply context with sending a webmention.

Testing again. Apologies for the spamming.

Testing once more as a queued job. Auto-webmention sending.

 Jeremy’s probably cursing me with my spamming of his page for my webmention testing.

Hi Jeremy, just testing my webmentions…

@adactio @adactioJournal Thanks Jeremy

In a world before social media, a lot of online communities existed around blog comments. The particular community I was part of – web standards – was all built up around the personal websites of those involved.

As social media sites gained traction, those communities moved away from blog commenting systems. Instead of reacting to a post underneath the post, most people will now react with a URL someplace else. That might be a tweet, a Reddit post, a Facebook emission, basically anywhere that combines an audience with the ability to comment on a URL.

Oh man, the memories of dynamic text replacement and the lengths we went to just to get some non-standard text. https://t.co/f0whYW6hh1

— One Bright Light ☣️ (@onebrightlight) July 13, 2017

Whether you think that’s a good thing or not isn’t really worth debating – it’s just the way it is now, things change, no big deal. However, something valuable that has been lost is the ability to see others’ reactions when viewing a post. Comments from others can add so much to a post, and that overview is lost when the comments exist elsewhere.

This is what webmentions do

Webmention is a W3C Recommendation that solves a big part of this. It describes a system for one site to notify another when it links to it. It’s similar in concept to Pingback for those who remember that, just with all the lessons learned from Pingback informing the design.

The flow goes something like this.

1. Frankie posts a blog entry.
2. Alex has thoughts in response, so also posts a blog entry linking to Frankie’s.
3. Alex’s publishing software finds the link and fetches Frankie’s post, finding the URL of Frankie’s Webmention endpoint in the document.
4. Alex’s software sends a notification to the endpoint.
5. Frankie’s software then fetches Alex’s post to verify that it really does link back, and then chooses how to display the reaction alongside Frankie’s post.

The end result is that by being notified of the external reaction, the publisher is able to aggregate those reactions and collect them together with the original content.

The reactions can be comments, but also likes or reposts, which is quite a nice touch. For the nuts and bolts of how that works, Jeremy explains it better than I could.

Beyond blogs

Not two minutes ago was I talking about the reactions occurring in places other than blogs, so what about that, hotshot? It would be totally possible for services like Twitter and Facebook to implement Webmention themselves, in the meantime there are services like Bridgy that can act as a proxy for you. They’ll monitor your social feed and then send corresponding webmentions as required. Nice, right?

Challenges

I’ve been implementing Webmention for the Perch Blog add-on, which has by and large been straightforward. For sending webmentions, I was able to make use of Aaron Parecki’s PHP client, but the process for receiving mentions is very much implementation-specific so you’re on your own when it comes to how to actually deal with an incoming mention.

Keeping it asynchronous

In order for your mention endpoint not to be a vector for a DoS attack, the spec highly recommends that you make processing of incoming mentions asynchronous. I believe this was a lesson learned from Pingback.

In practise that means doing as little work as possible when receiving the mention, just minimally validating it and adding it to a job queue. Then you’d have another worker pick up and process those jobs at a rate you control.

In Perch we have a central task scheduler, so that’s fine for this purpose. My job queue is a basic MySQL database table, and I have a scheduled task to pick up the next job and process it once a minute.

I work in publishing, dhaaaling

Another issue that popped up for me in Perch was that we didn’t have any sort of post published event I could hook into for sending webmentions out to any URLs we link to. Blog posts have a publish status (usually draft or published in 99% of cases) but they also have a publish date which is dynamically filtered to make posts visible when the date is reached.

If we sent our outgoing webmentions as soon as a post was marked as published, it still might not be visible on the site due to the date filter, causing the process to fail.

The solution was to go back to the task scheduler and again run a task to find newly published posts and fire off a publish event. This is an API event that any other add-on can listen for, so opens up options for us to do this like auto-tweeting of blog posts in the future.

Updating reactions

A massive improvement of webmentions over most commenting systems is the affordance in the spec for updating a reaction. If you change a post, your software will re-notify the URLs you link to, sending out more webmention notifications.

A naive implementation would then pull in duplicate content, so it’s important to understand this process and know how to deal with updating (or removing) a reaction when a duplicate notification comes along. For us, that meant also thinking carefully about the moderation logic to try to do the right thing around deciding which content should be re-moderated when it changes.

Finding the target

One interesting problem I hit in my endpoint code was trying to figure out which blog post was being reacted to when a mention was received. The mention includes a source URL (the thing linking to you) and a target URL (the URL on your site they link to) which in many cases should be enough.

For Perch, we don’t actually know what content you’re displaying on any given URL. It’s a completely flexible system where the CMS doesn’t try to impose a structure on your site – you build the pages you want and pull out the content you want onto those pages. From the URL alone, we can’t tell what content is being displayed.

This required going back to the spec and confirming two things:

1. The endpoint advertised with a post is scoped to that one URL. i.e. this is the endpoint that should be used for reacting to content on this page. If it’s another page, you should check that page for its endpoint.
2. If an endpoint URL has query string parameters, those must be preserved.

The combination of those two factors means that I can provide an endpoint URL that has the ID of the post built into it. When a mention comes in, I don’t need to look at the target but instead the endpoint URL itself.

It’s possible that Bridgy might not be compliant with the spec on this point, so it’s something I’m actively testing on this blog first.

Comments disabled

With that, after about fifteen years of having them enabled, I’ve disabled comments on this blog. I’m still displaying all the old comments, of course, but for the moment at least I’m only accepting reactions via webmentions.

Reaktionen sichtbar machen Der Austausch über Blogartikel steht für mich schon immer im Mittelpunkt. In den Artikeln werden Ideen, Gedanken und Erfahrungen beschrieben, diese stehen jedoch nicht nur für sich selbst, sondern werden von den Reaktionen der Leserinnen und Leser ergänzt. Anfangs wurde engagiert kommentiert. Durch Twitter und Facebook fand schließlich der Austausch meist auf den entsprechenden Plattformen statt, und beschränkte sich in vielen Fällen auf den Ausdruck des Gefallens. Viele Kommentarspalten blieben leer, und die Artikel wirkten leblos. Die Reaktionen darauf war im Blog selbst nicht sichtbar. Social-Buttons mit ihren Tracking-Skripten stellten für mich keine Lösung dieses Problems dar. Den Austausch über die Inhalte verwandeln sie in Zahlen, die Qualität desselben bleibt jedoch weiterhin verborgen. Einen Mehrwert für das Blog und dessen Community stellen sie nicht dar, sondern dienen vor allem den Plattformen als Datenlieferanten. Bereits vor einigen Jahren erfuhr ich von der IndieWeb-Bewegung, die vernetzte Inhalte sichtbar machen wollte, den Zugriff auf die eigenen Daten betonte und an vielen Stellen interessante Projekte hervorbrachte. In diesem Zusammenhang las ich bei Jeremy Keith über Webmentions. Die Annahme über zu wenig Wissen und Zeit zu verfügen um mich selbst an die Implementation von Webmentions zu machen, führte dazu diesen Bereich ruhen zu lassen, und schließlich nichts dergleichen zu unternehmen. Erst als ich im Nachklang der Smashing Conference in Freiburg den Artikel »Implementing Webmentions« von Drew McLellan las, fasste ich den Entschluss die Reaktionen auf Inhalte in Blogs endlich wieder sichtbar zu machen. Da ich gerade an ein paar Projekten arbeite die auf WordPress basieren und ich meine eigenen Blogs ebenfalls damit betreibe, widme ich mich an dieser Stelle der Implemantation von Webmention in mit WordPress betriebene und selbst gehostete Blogs. Was ist zu tun? Um Webmention mit WordPress zu benutzen sind vier Schritte nötig. Installation des Webmention-Plugins Installation des Plugins für Semantic-Linkbacks Social-Media-Profile via Bridgy mit dem Blog verbinden Das eigene Comment-Template anpassen Das Webmention-Plugin Das Webmention-Plugins von Matthias Pfefferle erweitert die Diskussion-Einstellung eines WordPress-Blogs, und legt die Grundlage dafür, dass ein Blog Webmentions senden und empfangen kann. Nachdem das Plugin heruntergeladen, installiert und aktiviert ist stehen unter Einstellungen > Diskussion ein paar weitere Einstellungen zur Verfügung. Achte auch darauf, dass die allererste Einstellung »Versuche, jedes in Beiträgen verlinkte Weblog zu benachrichtigen« ausgewählt ist. Unter Webmention Settings befinden sich vier Checkboxen und eine Auswahl mit deren Hilfe sich das Verhalten der Webmentions einstellen lassen. Das Plugin für Semantic-Linkbacks Das Plugin für Semantic-Linkbacks bietet die Möglichkeit die empfangenen Webmentions je nach Typ unterschiedlich zu behandeln. Es weist den empfangenen Erwähnungen einen entsprechenden Typ – beispielsweise Like, Repost oder Mention – zu, der als Filter und für die Styles zur Verfügung steht. Eine Übersicht der Einstellungen bringt dieses Plugin nicht mit, es verrichtet seine Dienste sobald es aktiviert ist. Im IndieWeb-Wiki erfährst Du etwas über die Möglichkeiten die zugewiesenen Typen in Deinem Theme zu verwenden. Wie ich die Typen verwende beschreibe ich etwas später in diesem Artikel. Update 15.10.2017: Seit Version 3.5.0 unterstützt dieses Plugin die Darstellung von Likes, Mentions und Reposts in eigenen Listen. Dadurch wurde die Anpassung meines Comment-Template obsolet. Brücken bauen Der Dienst Bridgy baut die Brücken zwischen Deinen Social-Media-Profilen und Deinem Blog. Er überwacht die verbundenen Social-Media-Profile und sendet Erwähnungen ans Blog. Deine Social-Media-Profile müssen öffentlich sein und die URL Deines Blogs muss dort hinterlegt sein. Wichtig ist auch, dass die Post öffentlich sind, nur so können sie von Bridgy ausgelesen werden. Auf brid.gy wählst Du also die Social-Media-Kanäle aus über die Du Erwähnungen empfangen möchtest. Sobald Du Bridgy den Zugriff auf Deine Profile gestattet hast, beginnt es damit alle 30 Minuten nachzusehen ob es dort eine Erwähnung Deines Blogs gibt. Wird es fündig sendet es die Erwähnung als Kommentar an Dein Blog. Das Comments-Template Um die Erwähnungen zu Bündeln habe ich mein Comments-Template und das Stylesheet etwas angepasst. Zur Anpassung meines Comment-Templates habe ich mich an diesem Code von Michael Bishop orientiert. Mein aktuelles Comments-Template kannst Du in diesem Gist ansehen. Likes und Reposts sammle ich in einer Liste und verlinke lediglich die Profilbilder mit den jeweiligen Accounts. Danach gebe ich eine Liste der Kommentare aus – in dieser Liste erscheinen bei mir Antworten von Twitter, Kommentare von Facebook und die klassischen Kommentare. In einer Liste unter den Kommentare gebe ich Erwähnungen des Artikels in anderen Blogs aus. Verbesserungen Bei meinen ersten Schritten mit Webmention sehe ich Verbesserungspotential in den folgenden Punkten: Links auf den Namen der Autorinnen korrigieren Baumstruktur der Kommentare darstellen Profilbilder optimieren und auf den eigenen Server speichern Den Callback der manuellen Webmention-Eingabe optimieren Ich freue mich von Euch zu hören wenn Ihr Ideen habt wie einer dieser Punkte gelöst werden kann, aber auch wenn Euch noch weitere Verbesserungen einfallen. Veröffentlicht am 15.09.2017 von Daniel in Blogs, IndieWeb, Internet, WordPress.