Proposal around pubsub (as a BB)

The Question: Should PubSub be its own BB? (And is it needed for any of the currently defined use cases?)

Context 1

The “PubSub Layer” has always been a bit of a special case in the IM building block because any implementation of a inter-application pubsub server would need to be “merely” another application (e.g., “Cool Cloud PubSub v2.3”) which provides a service (e.g., POST to /api/event-inbox) and is registered under a member (e.g., “Ministry of IT”). It follows all the same rules as any other application that’s registered on the InfoMed layer. In a sense, PubSub is a building block of a modern, distributed, GovStack implementation.

Context 2

Right now, @areitsakas and I can’t think of any of the currently defined use cases that require PubSub. While we (along with @psrk and @mcarlson ) have been vocal supporters of PubSub as the long-term way forward, architecturally, not only haven’t there been use cases which make use of the functionality provided by pub sub, there’s also been strong pushback against it, mostly because in a government context it’s very rare to actually “broadcast” a message:

  • a big part of pubsub is that the event emitter doesn’t know/care about the receivers, and the event receiver doesn’t know/care about the emitters;
  • a big part of Government (with a capital G :slight_smile: ) is that they never want to emit information without knowing exactly who they are sending that information to and what that party is doing with it. (This is the push back that we’ve been getting from other folks in the groups.)

It should be noted that “publishing” in a PubSub system is not a public broadcast, the only potential subscribers who could opt-in to receiving that message are registered members organizations/applications on the Information Mediator, but my understanding of the push back is that even then no ministry would ever use PubSub.

We’ve included a new concept in the spec for our “GovPubSub” , that requires that each event have not only a type but a sender and that in order for someone to subscribe to messages of a certain type they also must gain the required legal access for subscribing to events from that specific sender.

While we haven’t nailed down all the technical requirements, on paper, this sounds :ok_hand: to me. There may even be support for stuff like this in some of the many off the shelf pubsub options:

  1. What is Pub/Sub Messaging?
  2. https://cloud.google.com/pubsub
  3. Azure Web PubSub – WebSocket Web Publishing | Microsoft Azure

But…

Considering context 1 (it’s a separate set of applications that sit on the IM and play by the same rules as every other BB) and context 2 (it’s yet to be fully defined and is not being used by any of the use cases) we’re wondering why we don’t extricate the existing specs and diagrams from the IM specification, shift them to a new “PubSub” BB document/repo, and then put a pin in them until they’re needed.

The thought is that this would accelerate our ability to deliver on these initial use cases and get a demo up and running. As soon as a use case is developed that requires PubSub, we could pick it back up and allocate resources to its finalization.

Next steps

We’d love to hear some push back on this: use cases we’ve missed, reasons to keep developing the spec now, etc. If we don’t get around to it until next week I’ll raise this in the cross-team standup so we can get some more voices/opinions in here.

Hi guys, quick one from here in Dublin, Ireland.
Not seeing any great appetite for pub/sub model within/across government departments here yet.
Likely to be a digital maturity issue and can foresee the need arising in due course but not now.
Hope that helps, Tony

I think that this is worth a discussion, actually, and I’d like to understand why we would not consider starting with one of the most widely adopted tools, Kafka, as a start if/when we go down this path.

Hi @mcarlson , @jwatson , @tonyshannon , first… thank you so much for engaging around this.

After a series of conversations with @psrk and @areitsakas , and the call with Kristo, we’ve got a new working specification for PubSub that provides, IMO, the flexibility we need and still adheres to standard best-practices around PubSub.

The full (sub)spec is in our Google Drive: PubSub Proposal - Google Documenten

And the image, cause images get more clicks :wink: is below…
image

1 Like