Apex Code Development You need to sign in to do that
Don't have an account?
ScottWells Reliable semi-real time outbound message queue options?
We're in the process of building a composite app that needs to synchronize data managed in a Force.com UI with an external data source in near-real time. Basically Force.com acts as the application's configuration UI but the heavy lifting is done elsewhere based on that configuration. Guaranteed, well-ordered message delivery is critical for us. My initial thought was to use post-DML triggers to write entries into a custom object that represents a message queue entry, then have scheduled Apex drain the queue frequently by invoking a simple REST API on the external system with the DML operation, sObjectType, IDs of the affected objects.
Coincidentally I attended a session at DreamForce '11 last week on a very similar topic (REST for Others: Design Patterns for External REST API Integrations) where the presenter showed an evolutionary design for a solution to this problem. He started with future callouts from triggers and ended up with a similar approach to what I describe above, ultimately using scheduled Apex and the batch API to work around governor limitations on batch sizes, number of outbound calls per process and org/day, number of scheduled processes, etc. At the end of the session I asked whether anyone knew of other solutions to this problem given how prevelant it must be and someone suggested I look into workflow outbound messaging.
Outbound messaging looks promising, but I have a few reservations. First, I'm bummed that it uses SOAP, and in particular that it uses SOAP without a dynamic WSDL. Rather than get into a philosophical debate, though, let me just say that I would love to see a version of outbound messaging that can invoke a RESTful service with a set of JsonObjects so that the sObjectType isn't wired into the WSDL. My second concern is the lack of ordering in messages ("Messages are retried independent of their order in the queue. This may result in messages being delivered out of order."). Obviously we can set it up to send over the modification date and order things appropriately on the receiver end, though. Also, retries get exponentially longer. How big an issue is this in practice if you have a reliable receiver, though? Perhaps it's not at all, and when a real issue exists, the queue can be flushed manually. Also, is the queue per-user session or per-org? Hopefully the former because otherwise it seems that an issue with one user might cause a backup for all other users in the org.
Does anyone know of other good options for this? I'd really prefer not to build something if I don't have to (that's one of the major selling points for the Force.com PaaS, right?), but like I said, near-real time, guaranteed, well-ordered message delivery is going to be critically important.
Thanks much for any insights you can offer!!!
One thing about the message ordering is that the message always contains the latest data (and not neccasirly the data when the workflow fired), so the lack of ordering is less of an issue for most people, also a given data record can only appear in teh queue once, so between these 2 things, you'll never get new then stale data for a given record. (which is what most ordering requirements are based around).
Remember that the WSDL is just a description of the message we'll send, you can choose to use it, not use it, alter it and use it, and so on, as long as you accept the message and respond correctly, we really don't care how you build your listener, i've built a number of generic listeners that don't use the wsdl at all.
The exponential back-off is capped (at 2 hours, IIRC), and the initial back-off is pretty small, so you need a number of consequetive failures before the back-off gets significant (e..g > 10 minutes).
The queue is per org, yes a single user can dominate the queue, but even large updates are chunked and so queue entries from other users tend to get interspersed within the bulk updates into the queue, and the bulk messaging (upto 100 notifications in a single soap round-trip) means the queue can move very quickly if the listener is upto the task. I know of customers that pump 1M+ messages though outbound message daily.
Oh, and the other technology to consider is the new streaming API, where you can register a topic (soql query) and get pushed updates over a long-poll connection (ala cometd)