When a job is submitted to Onyx, it will only begin when there are enough virtual peers to sustain its execution. How many are enough? That depends. One virtual peer may work on at most one task at any given time. By default, each task in your workflow requires one available virtual peer to run it. If a grouping task is used, or if you set :onyx/min-peers on the task’s catalog entry, the number may be greater than 1. You can statically determine how many peers a job needs for execution with this simple helper function available in Onyx core.

As an example, if you submit a job with 10 tasks in its workflow, you need at least 10 available virtual peers for your job to begin running. If you only have 8, nothing will happen. If you have more than 10 (assuming that 10 is truly the minimum that you need), Onyx may attempt to dedicate more resources to your job beyond 10 peers, depending on which scheduler you used You can control the number of virtual peers that you’re running by modifying the parameters to onyx.api/start-peers.

We don’t emit a warning that your cluster is underprovisioned in terms of number of peers because its masterless design makes this difficult to asynchronously detect. This has been discussed in an issue. Onyx is designed to handle under and over provisioning as part of its normal flow, so this is not considered an error condition.

See the Scheduling chapter of the User Guide for a full explanation.

You’ve hit this case if you have more than one job running concurrently. When you start your peers, you specified a job scheduler through the configuration key :onyx.peer/job-scheduler. If you specified :onyx.job-scheduler/greedy, you’ve asked that Onyx dedicate all cluster resources to the first submitted job until that job is completed. Then the job is completed, all resources will be assigned to the next job available, and so on. If you want more than one job running at a time, consider using the Balanced (onyx.job-scheduler/balanced) or Percentage (onyx.job-scheduler/percentage) job schedulers. See the Scheduling chapter of this User Guide for a full explanation.

true?

If any of your lifecycle hooks for :lifecycle/start-task? return false or nil, you’ll want to change them to eventually return true based upon some condition.

As per step 2, a plugin’s constructor will be called to open any relevant stateful connections, such as a database or socket connection. Many connection calls, such as a JDBC SQL, will block for prolonged periods of time before failing, unless otherwise configured. If the task that appears to be stuck is an input or output task, this is likely the cause. You may want to reconfigure your initial connections to fail faster to make it more obvious as to what’s happening.

One thing that you can do for extra visibility is to log all incoming messages from input tasks. This is inadvisable for production, but can be useful for temporary debugging. You can specify an :onyx/fn transformation function to any task, including inputs and outputs. It can be useful to specify a debug function on your input tasks to see which messages are entering the system. Be sure that this function returns the original segment! For example, you can define a function:

Then add :onyx/fn ::spy to your input catalog entries.

Unless otherwise configured by your Lifecycles, if any user-level code throws an exception, the job will be killed and is no longer elligible for execution. Check onyx.log on all your peers to ensure that no exceptions were thrown. If this were the case, you’d see messages lower in the log reading:

If you’re using the Kafka input plugin, make sure that you’re reading from a reasonable starting offset of the topic. If you’ve set :kafka/force-reset? to true in the catalog entry, and you’ve also set :kafka/offset-reset to :largest, you’ve instructed Onyx to begin reading messages from the end of the topic. Until you place more messages into the topic, Onyx will sit idle waiting for more input. The starting offset for each input task using Kafka is echoed out in onyx.log.

Messages are replayed from the input source if they do not complete their route through the cluster within a particular period of time. This period is controlled by the :onyx/pending-timeout parameter to the catalog entry, and it’s default is 60 seconds. You can read about its specifics in the Cheatsheet. You should set this value high enough such that any segment taking longer than this value to complete is highly likely to have encountered a failure scenario.

Most exceptions will kill the job in question. If you are simply monitoring progress by reading from an output data source through Onyx, you should check all of the peer onyx.log files for exceptions that may have killed the job.

Any implementations of :onyx/fn that are blocking will halt progress of all other segments that are directly lined up behind it. Ensure that user level functions finish up in a timely manner.

To get started, see the full section on how and why messages are being replayed. In short, messages will be replayed in 60 seconds if they are not completed. You may be experiencing initial success, followed by a runtime error that is causing temporarily lost segments before replay.

If you’re using a core.async output plugin writing to a channel that will block writes when the buffer is full, you have run enough messages to put onto the channel such that core.async writes are now blocking, and hence stalling Onyx.

Ensure that the host and port that the peer advertises to the rest of the cluster for incoming connections is correct. If it is incorrect, only tasks that are colocated on the same machine will have a chance of working. Remember that Onyx uses UDP as its port, so make sure that your security settings are allowing traffic to run through that protocol.

The host is configured via the :onyx.messaging/bind-addr key, and the port is configured via the :onyx.messaging/peer-port key.