All notable changes to this project will be documented in this file. This change log follows the conventions of keepachangelog.com.
- Fixed publishing of metrics for batch consumption
- Fixed the startup logic for batch consumption - only routes provided in Ziggurat init-args will be started
- Standardized naming for Kafka Consumer API configs
- Adds support for consuming Kafka messages in batches using Kafka Consumer API
- Fixes the logic during RabbitMQ disconnection - Ziggurat now retries (publishing a message)
infinitely till RMQ recovers. This changes the present behaviour where Kafka Streams were being stopped
during disconnection with RabbitMQ.
- Log the stream record metadata
- Investigate if transform() affects how stream joins behave
- Fixes issue #56
- Adds functionality to stop and restart KafkaStreams using nREPL.
- Fixes issue #142
- Adds improved error message responses for Deadset API calls
- Releases Stable Ziggurat version with support for RabbitMQ clusters
- Moves stream joins behind an alpha feature flag
- Defaults RabbitMQ queue replication to
(n/2) + 1
nodes, where n
is the number of nodes in the cluster
- Fixes false positive exception thrown by messaging when an actor abnormally terminates
- Remove defrecord wrappers
- Uses default ha-config values if nothing is provided
- Adds RabbitMQMessaging implementation to support connection with RabbitMQ clusters
- Adds support for setting up HA policies for queues and exchanges
- Removes the use of the old protobuf library in favor of the new one
- Makes messaging implementation configurable
- Adds a new protocol for Messaging
- Refactors rabbitmq specific logic to the messaging.rabbitmq package
- Adds unit tests for rabbitmq specific namespaces
- Adds test annotations to messaging integration tests
- Support for Kafka Stream KStream-KStream Join
- Introduces a swagger middleware on the HTTP server.
- Makes metrics library configurable, exposes a metrics interface and provides an
implementation for clj-statsd library.
- Updates dependency of flatland/protobuf and puts the change behind a alpha-features configuration flag
- Adds support for all configurations of kafka-producer.
(for a more details list of changes look at the changelogs of 3.3.0-alpha. entries)
- Adding support for all configurations supported by Kafka Producer
- Remove
[org.flatland/protobuf "0.8.1"]
from test dependencies
- Fixes Issue
- Adds alpha feature flags
- Makes metrics implementation configurable
- passes "all" metric-name to update timing for all metrics
- Adds docs for MetricsProtocol
- Defines an interface for metrics.
- Changes the dropwizard implementation to use the metrics interface
- Adds a metrics interface implementation for clj-statsd library.
- Fixes metrics initialization
- Refactors Metrics.clj
- Moves dropwizard metrics logic to its own namespace
- Moves statsd state (transport and reporter) to dropwizard namespace
- Removes wrap-with-metrics middleware from HTTP router
- Fixes bug Issue
to avoid confusion between datadog and statsd
- Fixes bug for this Issue
- Releases metrics for http requests in a stable release
- Added metrics for recording http requests served
- Changes Exponential backoff config contract.
- Adds a
:type
key to retry-config - Adds a limit on the number of retries possible in exponential backoff
- Releasing exponential backoff as an alpha feature
- Fixes issue where dead-set replay doesn't send the message to the retry-flow
- Fixes issue by updating tools.nrepl dependency to nrepl/nrepl
- Fixes this bug where dead set replay broke on Ziggurat upgrade from 2.x to 3.x .
- Fixes this bug in RabbitMQ message processing flow
- Adds support for exponential backoffs in channels and normal retry flow
- exponential backoffs can be enabled from the config
- Fixes issue where dead-set replay doesn't send the message to the retry-flow
- Fixes issue by updating tools.nrepl dependency to nrepl/nrepl
- Fixes this bug where dead set replay broke on Ziggurat upgrade from 2.x to 3.x .
- Fixes this bug in RabbitMQ message processing flow
- Adds support for exponential backoffs in channels and normal retry flow
- exponential backoffs can be enabled from the config
- Adds tracing support. With Jaeger as the default tracer
- Adds a JSON middleware to parse JSON serialized functions
- Renames report-time to report-histogram and adds deprecation notice on report-time
- Makes metrics backward compatible with 2.x and 3.0.0 . Ziggurat now publishes metrics in 2 formats similar to version 2.12.0 and above.
- Fixes metrics publishing for custom metrics (i.e. string metric namespaces): In 2.x ziggurat appended the service_name
to a string metrics namespace (e.g. "metric" -> "service_name.metric"), we changed the contract in 3.0 by
removing the service_name from the metric name and instead adding a tag for it. To have backward compatibility with both
2.x and 3.0 we now send metrics in both formats
- Reintroduces old metrics format (statsd). Ziggurat now pushes metrics in both formats (statsd and prometheus like).
- Reverts the changes for exponential backoff, the current implementation was broken and a new PR is being raised with the correct approach.
- Renames report-time to report-histogram while being backward compatible
- JSON middleware has been added.
- Adds custom delay (Issue#78)
for processing messages from RabbitMQ channels and
adds exponential backoff strategy (configurable) for channel retries.
- Adds tracing support to the framework.
- Updates kafka streams - 1.1.1 -> 2.1.0
- Changes metrics format
- Instead of having service name and topic in the metric name, everything is now added to tags
- Middleware
- Handler-fn will now receive the message as a byte array
- Channel-fns will now receive the message as a byte array
- We have provided middlewares, that can be used to deserialize the messages
- Deadset-get api will now get serialized messages
- Java functions
- Java functions are now exposed for all public functions in namespaces
- Dependency simplification
- Removes dependency overrides.
- Removes dependency overrides and conflicts
- Adds pedantic warn to generic profile
- Adds pedantic abort to uberjar profile
- Fix increment/decrement count to accept both number and map
- Exposes Java methods for init, config, producer, sentry, metrics, fixtures namespaces
- Adds util methods for converting data types between java and Clojure
- Remove old metrics from being sent
- Adds middleware support
Breaking Change!
Mapper-function will now receive deserialised message if middleware is not applied- Deadset view will now return serialized messages
- Fixes a bug that incorrectly checked for additional-tags in ziggurat.metrics/merge-tags
- Fixes a bug where calling inc-or-dec count without passing additional tags raised and exception
- Upgrades kafka streams to version 2.1. Please refer this to upgrade
- Fix increment/decrement count to accept both number and map
- Fix functions to either take vector or string as input
- Fixes a bug that incorrectly checked for additional-tags in ziggurat.metrics/merge-tags
- Fixes a bug where calling inc-or-dec count without passing additional tags raised and exception
- Add support for providing a topic-name label in the metrics
- Multiple Kafka producers support in ziggurat (#55)
- Validate stream routes only when modes is not present or it contains stream-server (#59)
- Actor stop fn should stop before the Ziggurat state (#53)
- Running ziggurat in different modes (#46)
- Adds config to change the changelog topic replication factor
- dont close the channel on shutdown listener. it is already closed when connection is broken. this prevents topology recovery
- catch message production exception in rabbitmq publisher
- Adds nippy as dependency instead of carmine
- Adds macro for setting thread-local context params for logs
- Adds deployent stage on CI pipeline
- Initialize reporters before running actor start fn
- Initialize reporters before running actor start fn
- Adds deployent stage on CI pipeline
- Updates changelog for older releases
- Releases using java 8
- This release has been compiled using java 10. It will not work with
older versions of java.
- Adds oldest-processed-message-in-s config
- Adds capabiltiy to filter message based on timestamp
- Fixes bug in deadset API for channel enabled
- Changes namespace of
transformer
into timestamp-transformer
- Handle Deadset API when retry is disabled
- Fixing message being retried n + 1 times
- Fixing kafka delay calculation
- Upgrades kafka streams to 1.1.1
- Adds stream integration tests
- Adds API to flush messages from dead-letter-queue in RabbitMQ
- Starts sentry-reporter in on application initialization
- removes executor dependency as it was not being used
- updates readme and contribution guidelines
- refactors config files to remove gojek specific configs
- Removes sentry dependency and instead uses sentry-clj.async
- Merges lambda commons and adds default configs for missing application specified configs
- Users using
lambda-commons.metrics
should now start using ziggurat.metrics
to send custom metrics.
- Fixes bug where connection to rabbitmq fails when stream routes is not passed in mount/args
- Adds support for multipart params over actor routes, moves lein-kibit and eastwood to dev plugins
- Changed the order of starting up of ziggurat and actor. First config will be initialized, then actor function will start up and ziggurat start function will start up.
- Apps with (mount/start) in their
start-fn
will no longer work correctly. Users should start using mount/only
instead.
- Removes Yggdrasil, bulwark and ESB log entities dependency
- Removes the
make-config
function from ziggurat.config
namespace. Users should now use config-from-env
function instead.
- Overrides and exposes kafka streams config: buffered.records.per.partitions and commit.inteval.ms
- Fixes bug where rabbitmq connection is established even when retry is disabled and channels are absent in consumer/start-subscribers
- Add configuration to read data from earliest offset in kafka
- Fixes rabbitmq queue creation when retries are disabled and channels are present
- Fixes rabbitmq intialization when retry is disabled but channels are present
- Fixes dead set management api to validate the channel names
- Starts up rabbitmq connection when channels are present or retry is enabled
- Fixes bug around reporting execution time for handler fn
- Adds arbitrary channels for long running jobs
- Fix parallelism for retry workers
- Starts sending expiration per message instead of setting it on the queue
- Starts calculating timestamp from kafka metadata
- removes deprecated config variables in kafka-streams
- Upgraded lambda commons library to 0.3.1
- Upgraded lambda commons library
- Adds metrics to skipped and retried message
- Retry message when actor raises an expection
- Add support for multi stream routes
- Fixes replay of messages in dead letter queue.
- Bumps up lambda-common version to 0.2.2
- Fixes converting message from kafka to clojure hash
- Fixes converting message from kafka to clojure hash
- Instruments time of execution of mapper function
- Increments the esb-log-entites version to fetch from 3.18.7 and above
- Fixes the consumer to retry the mapper-fn
- Uses WallclockTimestampExtractor as timestamp extractor for the streams
- Always fetches the esb-log-entites version greater than or equal 3.17.11
- Bumps up the esb log entities version to 3.17.11
- Fetches config from yggdrasil and if not found fallbacks to env
Configs added
{
:ziggurat {:yggdrasil {:host "http://localhost"
:port [8080 :int]
:connection-timeout-in-ms [1000 :int}}
}
- Bumps up the esb log entities version
- Adds ability to pass actor specific routes
- Changes dependency from esb-log-client to esb-log-entities
- Adds metrics to count throughput
- Changes the job name getting pushed to NR
- Adds an
v1/dead_set
to view the dead set messages
- Bump version of
com.gojek/sentry
- Fixed a bug in application shutdown: the actor's start-fn was being called instead of the stop-fn.
- Made some functions private.
- Added some docstrings.
- Added Gotchas section to the README.
- Added ziggurat.sentry/report-error to be used by actors.
- Upgrade esb-log-client version to latest (1.103.0).
- Various internal refactorings: removed dead code, fixed some spelling mistakes, made some functions private.
- Flag to enable retries and conditionally start the rabbitmq states depending on this flag.
- Namespace framework configs under
:ziggurat
Can you improve this documentation? These fine people already did:
Anirudh, Kartik Gupta, shubhang.balkundi, Michael Angelo Calimlim, prateek.khatri, Sandilya Jandhyala, Nivedita Priyadarshini, WickedBrat, Saptanto Sindu, prashant, Rooba Limsa, Anirudh Vyas, hogaur, Soham Kamani, Anshuman Srivastava, Akshay Gupta, maulik.soneji, Kishan Sharma, Gowtham Sai, rajnishdashora & dark-bytesEdit on GitHub