Untitled

Elastic Reconciliation Report for Failed Events - 

Server - ST2(microservice was running in 45)

Targetted Events - {"addIndividual","addOrganization","addPartyRole","addSubscriber","createBatchOrder","createCustomerOrder","enrichAddon","enrichAttribute","enrichPayer","enrichUser","orderLineCease","terminatePartner","updateCustomerOrder","updatePartyRole","updatePartyRoleByAssocId","updateSalesOrder","updateSubscriber","orderCompletion","updateAccount","updateCustomer","updateOrderLines","createOrder","addContact","addCustomer","addAccount","updateOrder"}

Daily Average Failure Count = 200-400

Observations - Some times the failure count is around 3000-5000 due to service failures like Kafka down or else EventEnricher Failure

Reason for Events Failure in Microservice - While kafka trying to send the message with the key and payload provided Specifically, it says "Expiring 16 record(s) for salesorder.reconstruct.request-0:120000 ms has passed since batch creation." This indicates that the Kafka producer attempted to send a batch of 16 records related to the salesorder.reconstruct.request topic but was not able to complete it within the allowed time (120000 ms).



Error - 2023-07-27 09:12:01.790 ERROR 19352 --- [ad | producer-1] c.t.telecom.tib.producer.EventPublisher  : Message transmission failed for eventID:a47128cb-ab78-410d-be3c-8ad10fb9dcfd, eventType:updateSalesOrder, eventDate:2023-07-27T09:11:00+0530
2023-07-27 09:12:01.790 ERROR 19352 --- [ad | producer-1] c.t.telecom.tib.producer.EventPublisher  : Message transmission failed because of:Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Expiring 16 record(s) for salesorder.reconstruct.request-0:120000 ms has passed since batch creation
2023-07-27 09:12:01.790 DEBUG 19352 --- [ad | producer-1] c.t.telecom.tib.producer.EventPublisher  : event with eventstatus:{"event":{"eventType":"updateSalesOrder"},"request":{"buID":"DEFAULT","createdDate":"27-Jul-2023 09:11:00","orderId":"SALES_ORDER_10072023_01","opID":"HOB","userID":"SALES_ORDER_BATCH","subStatus":"COMPLETED","status":"COMPLETED"},"eventID":"a47128cb-ab78-410d-be3c-8ad10fb9dcfd","eventDate":"2023-07-27T09:11:00+0530","eventStatus":{"topic-publish":"FAILED"}}
2023-07-27 09:12:01.790 ERROR 19352 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='{"event":{"eventType":"updateSalesOrder"},"request":{"buID":"DEFAULT","createdDate":"27-Jul-2023 09:...' to topic salesorder.reconstruct.request:

Observations - Kafka getting down frequently in 45 server


Reconciliation Scripts - currently two scripts running in st2

Script-1(captureElasticFailedEvents)
  - captures the targetted events on reconCriteria = 'tibsearchindex': 'SUCCESS'

Script-2(publishElasticFailedEvents)
  - publishes the targetted events on reconCriteria = 'tibsearchindex': 'SUCCESS'
  - The threshold count for retries is setted as 5
  - kafka batch size = 25

Observations - max at reties of 2 all the failed events are getting published