[BUG] Kinesis: LATEST shard iterator never returns records written after the iterator is obtained #1900

Description

GetShardIterator with ShardIteratorType: LATEST returns an iterator that never yields any records, even for records written after the iterator was obtained. Following NextShardIterator and polling does not help.

Per the AWS Kinesis API, LATEST starts reading "just after the most recent record in the shard", so a record written after the iterator is obtained must eventually be returned.

TRIM_HORIZON on the same shard at the same moment returns the record, so the write itself succeeds — only the LATEST iterator is affected.

Impact

This breaks the standard "observe an event as it happens" pattern: obtain a LATEST iterator, trigger the action that produces the record, then poll. It is the pattern AWS documents for tailing a stream, and a common one in integration/E2E tests, so a suite that asserts on freshly produced records fails against Floci while passing on real AWS.

The only workaround we found is to re-read the whole shard with TRIM_HORIZON and filter client-side, which is not equivalent (it re-reads all retained records and cannot express "only what happens after this point").

Reproduction

Self-contained; needs Docker and AWS CLI v2. Exits 0 either way and prints both the LATEST result and the TRIM_HORIZON control.

#!/usr/bin/env bash
set -euo pipefail

docker run -d --rm --name floci-repro -p 4566:4566 floci/floci:1.5.33 >/dev/null

export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1

until curl -sf "$AWS_ENDPOINT_URL/_localstack/health" >/dev/null; do sleep 1; done

aws kinesis create-stream --stream-name my-stream --shard-count 1
aws kinesis wait stream-exists --stream-name my-stream
SHARD=$(aws kinesis list-shards --stream-name my-stream --query 'Shards[0].ShardId' --output text)

# 1. obtain a LATEST iterator BEFORE any record is written
IT=$(aws kinesis get-shard-iterator --stream-name my-stream --shard-id "$SHARD" \
  --shard-iterator-type LATEST --query ShardIterator --output text)

# 2. write a record AFTER the iterator was obtained
aws kinesis put-record --stream-name my-stream --partition-key pk \
  --data "$(echo -n hello | base64)" >/dev/null

# 3. poll for 8s, following NextShardIterator (well within the 5 min iterator expiry)
N=0
for i in $(seq 1 16); do
  RES=$(aws kinesis get-records --shard-iterator "$IT" \
    --query "[length(Records), NextShardIterator]" --output text)
  N=$(echo "$RES" | cut -f1)
  IT=$(echo "$RES" | cut -f2)
  if [ "$N" != "0" ]; then
    echo "LATEST:       found $N record(s) after $(( (i - 1) * 500 ))ms"
    break
  fi
  sleep 0.5
done
if [ "$N" = "0" ]; then
  echo "LATEST:       0 records after 8s"
fi

# control: TRIM_HORIZON on the same shard, same moment
IT2=$(aws kinesis get-shard-iterator --stream-name my-stream --shard-id "$SHARD" \
  --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
echo "TRIM_HORIZON: $(aws kinesis get-records --shard-iterator "$IT2" \
  --query 'length(Records)' --output text) record(s)"

docker rm -f floci-repro >/dev/null

Notes on the method, to rule out the usual false positives: the shard ID comes from ListShards rather than being hardcoded; the stream is awaited with wait stream-exists; empty GetRecords responses are treated as normal and the loop follows NextShardIterator; and the whole poll finishes in 8s, far inside the 5 minute iterator expiry.

Expected

LATEST:       found 1 record(s) after 0ms
TRIM_HORIZON: 1 record(s)

Actual

LATEST:       0 records after 8s
TRIM_HORIZON: 1 record(s)

Affected versions

Reproduced on every version tested, so this does not look like a recent regression:

Running the identical script with the image swapped to localstack/localstack:4.14.0 prints the expected output above (found 1 record(s) after 0ms).

Environment

• Image: floci/floci:1.5.33 (also reproduced on 1.5.29 and nightly)
• Host: Linux, Docker
• Client: AWS CLI v2