Skip to content

Commit b3ce77e

Browse files
committed
Fix five issues flagged in second AI code review
- MessagePackParser: always reset unpacker for byte[] sources (|| src instanceof byte[]) to remove subtle dependency on the ThreadLocal-clearing fix; regression test added - MessagePackGenerator: use bb.duplicate().get(data) for non-array-backed ByteBuffers to avoid advancing the caller's buffer position - MessagePackGenerator: release OutputStream reference in _releaseBuffers() via messageBufferOutput.reset(null), avoiding retention without the 23% regression that messageBufferOutputHolder.remove() caused - MessagePackExtensionType: add toString() so extension-type map keys appear meaningfully in read-context error messages - preexisting-issues.md: document fixes and add issue #12 (ByteBuffer non-array path)
1 parent caad184 commit b3ce77e

5 files changed

Lines changed: 95 additions & 13 deletions

File tree

msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackExtensionType.java

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,12 @@ public int hashCode()
5555
return result;
5656
}
5757

58+
@Override
59+
public String toString()
60+
{
61+
return "MessagePackExtensionType(type=" + type + ", data.length=" + (data == null ? 0 : data.length) + ")";
62+
}
63+
5864
public static class Serializer extends StdSerializer<MessagePackExtensionType>
5965
{
6066
public Serializer()

msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackGenerator.java

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -400,7 +400,7 @@ else if (v instanceof ByteBuffer) {
400400
}
401401
else {
402402
byte[] data = new byte[len];
403-
bb.get(data);
403+
bb.duplicate().get(data);
404404
messagePacker.packBinaryHeader(len);
405405
messagePacker.addPayload(data);
406406
}
@@ -965,6 +965,15 @@ protected void _closeInput() throws IOException
965965
@Override
966966
protected void _releaseBuffers()
967967
{
968+
OutputStreamBufferOutput messageBufferOutput = messageBufferOutputHolder.get();
969+
if (messageBufferOutput != null) {
970+
try {
971+
messageBufferOutput.reset(null);
972+
}
973+
catch (IOException e) {
974+
throw _wrapIOFailure(e);
975+
}
976+
}
968977
}
969978

970979
@Override

msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackParser.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ public MessagePackParser(
105105
messageUnpacker = MessagePack.newDefaultUnpacker(input);
106106
}
107107
else {
108-
if (StreamReadFeature.AUTO_CLOSE_SOURCE.enabledIn(streamReadFeatures) || messageUnpackerTuple.first() != src) {
108+
if (StreamReadFeature.AUTO_CLOSE_SOURCE.enabledIn(streamReadFeatures) || messageUnpackerTuple.first() != src || src instanceof byte[]) {
109109
messageUnpackerTuple.second().reset(input);
110110
}
111111
messageUnpacker = messageUnpackerTuple.second();

msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackParserTest.java

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1108,4 +1108,25 @@ public void testByteArrayThreadLocalClearedAfterClose()
11081108
List<Integer> result = objectMapper.readValue(bytes, new TypeReference<List<Integer>>() {});
11091109
assertEquals(Arrays.asList(1, 2, 3), result);
11101110
}
1111+
1112+
@Test
1113+
public void testByteArrayReuseResetsUnpackerWhenAutoCloseSourceDisabled()
1114+
throws IOException
1115+
{
1116+
ObjectMapper objectMapper = MessagePackMapper.builder(new MessagePackFactory())
1117+
.disable(tools.jackson.core.StreamReadFeature.AUTO_CLOSE_SOURCE)
1118+
.build();
1119+
1120+
byte[] bytes = objectMapper.writeValueAsBytes(Arrays.asList(1, 2, 3));
1121+
1122+
// First parse succeeds
1123+
List<Integer> first = objectMapper.readValue(bytes, new TypeReference<List<Integer>>() {});
1124+
assertEquals(Arrays.asList(1, 2, 3), first);
1125+
1126+
// Second parse with the same byte[] instance and AUTO_CLOSE_SOURCE disabled.
1127+
// The unpacker is not reset (src identity match, no AUTO_CLOSE_SOURCE trigger),
1128+
// so it continues from EOF and fails to produce the expected result.
1129+
List<Integer> second = objectMapper.readValue(bytes, new TypeReference<List<Integer>>() {});
1130+
assertEquals(Arrays.asList(1, 2, 3), second);
1131+
}
11111132
}

plans/preexisting-issues.md

Lines changed: 57 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -54,12 +54,11 @@ Also fixed in the same pass:
5454
never reset after the first `writeName`, making every subsequent `writeName` return
5555
false.
5656

57-
Note: `messageBufferOutputHolder` ThreadLocal was NOT fixed here — calling
58-
`messageBufferOutputHolder.remove()` in `_releaseBuffers()` caused a 23% serialization
59-
regression by defeating the ThreadLocal caching (each close allocates a new
60-
`OutputStreamBufferOutput` on the next generator creation). Dropped in favour of
61-
accepting the minor OutputStream retention, which is only observable when a thread
62-
creates exactly one generator and never creates another.
57+
Note: `messageBufferOutputHolder` ThreadLocal OutputStream retention — FIXED (see
58+
preexisting-issues section 4 below). `messageBufferOutputHolder.remove()` caused a 23%
59+
regression; instead `_releaseBuffers()` now calls `messageBufferOutput.reset(null)` to
60+
release the OutputStream reference while keeping the `OutputStreamBufferOutput` object
61+
alive for reuse.
6362

6463
### 5. `writeString(Reader, int)` len≥0 path allocates unbounded buffer — FIXED
6564

@@ -143,12 +142,27 @@ identically in msgpack-jackson. They should be addressed in both modules togethe
143142

144143
## 1. `MessagePackParser`: Same byte-array input skips unpacker reset
145144

146-
**File:** `msgpack-jackson/src/main/java/org/msgpack/jackson/dataformat/MessagePackParser.java:130`
145+
**Files:**
146+
- `msgpack-jackson/src/main/java/org/msgpack/jackson/dataformat/MessagePackParser.java:130`
147+
- `msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackParser.java:108` — FIXED
148+
149+
When `AUTO_CLOSE_SOURCE` is disabled and the same `byte[]` instance is parsed more
150+
than once, the condition `messageUnpackerTuple.first() != src` is false, so the unpacker
151+
is not reset. The second parse continues from where the first left off (typically EOF)
152+
instead of from the beginning.
153+
154+
Bug only manifests when all three conditions hold: (1) `reuseResourceInParser=true`,
155+
(2) `AUTO_CLOSE_SOURCE` disabled (non-default), (3) the same `byte[]` reference reused.
156+
157+
**Note on msgpack-jackson3:** In practice, issue #2's fix (clearing the `byte[]` from the
158+
ThreadLocal on `close()`) incidentally prevents this bug from manifesting — after close,
159+
the cached src becomes `null`, so the next parse always sees `null != bytes` and resets.
160+
Fixed explicitly anyway (`|| src instanceof byte[]`) to make intent clear and remove the
161+
subtle dependency between the two fixes. Regression test:
162+
`MessagePackParserTest.testByteArrayReuseResetsUnpackerWhenAutoCloseSourceDisabled`.
147163

148-
When `AUTO_CLOSE_SOURCE` is disabled and the same byte-array instance is parsed more
149-
than once (e.g. reused buffer), the condition `messageUnpackerTuple.first() != src`
150-
is false, so the unpacker is not reset. The second parse continues from where the first
151-
left off instead of from the beginning.
164+
Needs the same explicit fix in msgpack-jackson (where issue #2 is also not yet fixed, so
165+
both bugs compound).
152166

153167
## 2. `MessagePackParser`: ThreadLocal retains last byte-array payload per thread
154168

@@ -259,3 +273,35 @@ Fixed in msgpack-jackson3 using `bb.arrayOffset() + bb.position()`; needs the sa
259273
with a `ByteBuffer` field that was sliced or partially consumed will silently produce
260274
corrupt serialized output. No exception is thrown.
261275

276+
## 11. `MessagePackFactory`: byte-array parser copies the input slice unnecessarily
277+
278+
**Files:**
279+
- `msgpack-jackson/src/main/java/org/msgpack/jackson/dataformat/MessagePackFactory.java:143`
280+
- `msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackFactory.java` — FIXED
281+
282+
`_createParser(byte[], int, int)` calls `Arrays.copyOfRange(data, offset, offset + len)` to
283+
extract the slice, then wraps the copy in `InputStreamBufferInput`. `ArrayBufferInput` already
284+
accepts `(data, offset, len)` directly and avoids the copy entirely.
285+
Fixed in msgpack-jackson3 by constructing `ArrayBufferInput(data, offset, len)` and passing it
286+
to the `MessageBufferInput`-taking constructor; needs the same fix in msgpack-jackson.
287+
288+
**Practical impact:** Low for small payloads; proportional to payload size for large messages
289+
since the entire slice is copied on every `readValue` call.
290+
291+
## 12. `MessagePackGenerator`: non-array-backed ByteBuffer read advances caller's position
292+
293+
**Files:**
294+
- `msgpack-jackson/src/main/java/org/msgpack/jackson/dataformat/MessagePackGenerator.java:373`
295+
- `msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackGenerator.java` — FIXED
296+
297+
When serializing a `ByteBuffer` whose `hasArray()` is false (e.g. direct buffers), the
298+
generator calls `bb.get(data)` which advances the buffer's position as a side effect. The
299+
`hasArray()` fast path is non-destructive (`writePayload(bb.array(), bb.arrayOffset() +
300+
bb.position(), len)`), so the two paths are inconsistent.
301+
Fixed in msgpack-jackson3 using `bb.duplicate().get(data)``duplicate()` shares the
302+
backing store without copying data (O(1)) but has its own independent position.
303+
Needs the same fix in msgpack-jackson.
304+
305+
**Practical impact:** Low. Direct ByteBuffers are uncommon in typical POJO fields, and the
306+
mutation is only observable if the caller inspects or reuses the buffer after serialization.
307+

0 commit comments

Comments
 (0)