Onyx 0.5.3 is out today with a new, critical feature called Flow Conditions. Briefly, Flow Conditions are a construct that extracts routing conditionals out of your distributed program. Following Onyx's aggressive data driven approach, Flow Conditions are a data structure with its own documented information model. This is a rich, novel approach that offers runtime composition, parameterization, and a light form of pattern matching. If you want to dive right in, you can read the Flow Conditions User Guide section.

The Problem

Flow conditions are used for isolating logic about whether or not segments should pass through different tasks in a workflow. These conditions are often scattered throughout your program in the form of:

(defn my-task [argument]
  (cond
    (blue? argument)
    (send-to-downstream-tasks [:a :b :c] (transform argument))
    (green? argument)
    (send-to-downstream-tasks [:b :c] (transform argument))))

I wanted the ability to express these kinds of often very complex and tangled conditions in my Onyx programs without resorting to cluttering up my transformation functions. It was very tempting to take the above approach - but in the context of a situation where good decisions have already been made, it's easier to keep making good decisions.

We express the above as the following data structure:

[{:flow/from :some-task
  :flow/to [:a :b :c]
  :flow/predicate :my.ns/blue?}
 {:flow/from :some-task
  :flow/to [:b :c]
  :flow/predicate :my.ns/green?}]

This new data structure can optionally be submitted as part of your Onyx job along with your workflow and catalog. Predicates are keywords that resolve to Clojure functions, which take at least two parameters (an execution context, a segment, and any optional values). The predicate form supports rich, nested composition with :and, :or, and :not. Additionally, these predicates can take runtime parameters - making them as powerful and flexible as the rest of your Onyx program.

[{:flow/from :some-task
  :flow/to [:a :b]
  :flow/predicate [:not :my.ns/yellow?]}
 {:flow/from :some-task
  :flow/to [:d]
  :my/runtime-parameter 42
  :flow/predicate [:my.ns/orange? :my/runtime-parameter]}
 {:flow/from :some-other-task
  :flow/to [:z]
  :my/runtime-parameter 42
  :flow/predicate [:and :my.ns/orange? [:or :my.ns/white? :my.ns/black? [:not :my.ns/red?]]]}]

This is just a taste of what Flow Conditions can do. If you want more, see the full documentation and examples on Multi-matching, Key Exclusion, Short Circuiting, and Predicate Composition.

The Road Ahead

You're probably wondering what's next - and I'd love to tell you. We're heads-down with the 0.6.0 Onyx release, steadily working our way towards a fantastic performance story. We've confirmed our decision to drop HornetQ as a dependency and adopt an approach similar to Apache Storm. Also up on our roadmap is an open, reusable AWS benchmarking framework to take the sting out out testing your Onyx workflows under heavy load.

These are exciting times, and I'm extremely grateful to everyone who's helped make this possible. Join us in Gitter, on the mailing list, or sign up for the newsletter!