2012-07-06

Using rss2email and github feeds to only notify of issues

Long story short:
  • We want any github issue notification related to the libusbx github project being pushed as an e-mail to libusbx-devel mailing list.
  • The default notification mechanism of github is next to useless to do anything like that, especially if you're working with a project that is associated with a github organization.
  • As a result, we have created a libusbx-devel user that, monitors the libusbx project as a pull-only participant, and that receives NewsFeed about issues among other things. This NewsFeed can then be used as RSS/Atom (https://github.com/libusbx-devel.private.atom), and fed through rss2email to send e-mail notifications to the mailing list.
  • However, besides report about issues, the NewsFeed also contains reports about wiki updates and whatnot, i.e. loads of polluting events that we don't want the mailing list to get notified about. And neither github or rss2email offer the possibility to filter events on their own.
  • A quick look at the feed with curl indicates that RSS/Atom entries are tagged, with something like: <id>tag:github.com,2008:IssuesEvent/1570141350</id> or <id>tag:github.com,2008:IssueCommentEvent/1570141345</id> for the ones that are of interest to us, and <id>tag:github.com,2008:GollumEvent/1570245314</id> for the ones that aren't
  • Thus, if you use a dedicated copy of rss2email for the github feed, it is possible to modify the rss2email.py code, and add a filter that ensures only entries that have an issue related tag are processed, with:
    for entry in r.entries:
            id = getID(entry)
            if id[:25] != "tag:github.com,2008:Issue": continue
            
            # If TRUST_GUID isn't set, we get back hashes of the content.
    so that anything that isn't issue related is filtered out.
Of course, the proper way would be to add an INCLUDE_TAGS/EXCLUDE_TAGS section in rss2email's config.py, that can take wildcards, and then cross reference these in the code above, but since we don't have a all day...

Bonus 1: If you need authentication with rss2email, assuming that libusbx-devel is your user, the following is an example of the URI you should use for the RSS: https://libusbx-devel:PASSWORD@github.com/libusbx-devel.private.atom
Bonus 2: In case you want to play with the github JSON API, that pertains to issues, rather than the RSS, and provide libusbx/libusbx is your project, you can issue something like

curl -i -u libusbx-devel:PASSWORD https://api.github.com/repos/libusbx/libusbx/issues/events

4 comments:

  1. Hi Pete,

    Thanks for your post. This sounds exactly what I've been looking for to set up for github.com/SOCI/ organisations, so our soci-devel mailing list can be notified about new issues, especially.

    Would you mind to explain in more details how/where you set up all parts of your solution?

    So far, here is what I've grasped:
    1. A virtual user https://github.com/libusbx-devel with libusbx at googlemail dot com e-mail address, but this e-mail is not used for notification purposes

    2. You have installed rss2mail somewhere on a server (BTW, do you host it privately or use a free hosting?)

    3. rss2mail runs as a cronjob and checks the news feed of the libusbx-devel user, and posts new items to libusbx-devel mailing list

    Am I missing anything?

    Thanks,
    Mat

    ReplyDelete
  2. Hi Mateusz,

    You're pretty much spot on. The github user (which we created using a gmail address) was added to the project so that it receives github notifications, i.e. the ones you get within github if you logon as this user.

    Then on a _private_ server we have rss2email running as a cron job.

    Now, ever since I wrote this post, it seems like github have made some changes to their RSS configuration, as our notification process is no longer working. When I contacted them on another issue related to this, github also indicated that they weren't really planning on supporting the old RSS API, and instead invited us to move to the new API, which would be fine, if it didn't mean having to write a complete notification client, instead of simply piggybacking on rss2email.

    I'm still hoping github will understood that e-mail notifications on issue updates can be hugely beneficial for their hosted projects, and added the feature themselves...

    ReplyDelete
  3. Hi Pete
    sorry for offtopic about earlier post debug usb 3.0 have you seen this "We (Synopsys) would like to donate the attached code for driving the debug
    capability of the xHCI controller" http://marc.info/?l=linux-usb&m=135940793913182

    ReplyDelete
  4. Thanks for the link. I hadn't seen it, and as a matter of fact, I purchased my first USB 3.0 control with debug support today (should be about a month before I receive it though).

    Unfortunately, I don't think I'll have a chance to play with USB 3.0 debug for a long time, as I have way too many concurrent things going on, and too little time to do them all. But hey, you never know...

    ReplyDelete