write_kafka: check for partition availability before selecting one

When a partition is unavailable, sending to it will just lead to a lost
metric. Therefore, after selecting the partition, check if it is
available. If not, select the next one until we tried them all.

A future iteration may use consistent hashing to avoid to double the
work done on a partition when the previous one is unavailable.

diff --git a/src/write_kafka.c b/src/write_kafka.c
index ba76d71f6c727e4f1b19a5514e74e1e6eb36926e..e1d2f1229a6684a340b4cf3da40b71f9a9ca09c4 100644 (file)
--- a/src/write_kafka.c
+++ b/src/write_kafka.c
@@ -76,8 +76,13 @@ static int32_t kafka_partition(const rd_kafka_topic_t *rkt,
                                int32_t partition_cnt, void *p, void *m)
 {
     u_int32_t key = *((u_int32_t *)keydata );
+    u_int32_t target = key % partition_cnt;
+    int32_t   i = partition_cnt;
 
-    return key % partition_cnt;
+    while (--i > 0 && !rd_kafka_topic_partition_available(rkt, target)) {
+        target = (target + 1) % partition_cnt;
+    }
+    return target;
 }
 
 static int kafka_write(const data_set_t *ds, /* {{{ */