Consumer lifecycle

The Quick start section provides a very simple use case to introduce the main concepts and steps required when using the Kafka API.

This section describes recommended practices for using the API in a real-life scenario.

The example in this use case assumes:

• Two partitions: 0 and 1
• Two consumers: c1 and c2
• One consumer group: g1

1. Create a consumer instance and specify:
   • A new consumer instance (c1)
   • A consumer group (g1)
   • Parameter auto.offset.reset: earliest
     This will ensure that the new consumer instance will start reading logs from the beginning of the partition. Note that this parameter is ignored if the group already exists and an offset was committed.
2. Get the list of available partitions.
   This will determine the number of consumer instances that need to be created. Nuance recommends that the number of consumer instances that you create should be less than or equal to the number of available partitions in a topic.
3. Call the Assign Partitions endpoint to manually assign partition 0 to consumer instance c1.
4. Call the Get Records endpoint in a loop to fetch the records for consumer c1, but do not process them at this point to avoid consumer performance issues.
5. Call the Get Assigned Partitions endpoint to get the list of partitions assigned to consumer c1.
6. Call the Commit Offsets endpoint to commit the offset for the last fetched record.
7. Create a consumer instance as described in step 1, using c2 as the consumer instance.
   Creating and managing consumer instance c2 can be done in parallel with creating and managing consumer instance c1.
8. Call the Assign Partitions endpoint to manually assign partition 1 to consumer instance c2.
9. Call the Get Records endpoint in a loop to fetch the records for consumer c2, as described in step 4.
10. When completed, do a graceful shutdown of the consumers:
    1. Delete consumer c1.
    2. Delete consumer c2.

The following diagram summarizes this lifecycle:

Restarting consumers

To restart consumers, repeat all the steps above. The consumers will read new records from the last commit offsets.