Skip to content

ref(openai): Only handle streamed results when applicable#5553

Open
alexander-alderman-webb wants to merge 10 commits intowebb/openai/input-parameterfrom
webb/openai/streaming-response
Open

ref(openai): Only handle streamed results when applicable#5553
alexander-alderman-webb wants to merge 10 commits intowebb/openai/input-parameterfrom
webb/openai/streaming-response

Conversation

@alexander-alderman-webb
Copy link
Contributor

@alexander-alderman-webb alexander-alderman-webb commented Feb 26, 2026

Description

Remove iterator patches from shared output-handling code, and only apply them in the dedicated functions for patching streamed responses. Only keep code relevant to the Completions and Responses APIs in the respective iterator wrappers.

Add stream=True to tests where a streamed response is returned from openai functions.

Issues

Reminders

@github-actions
Copy link
Contributor

github-actions bot commented Feb 26, 2026

Semver Impact of This PR

🟢 Patch (bug fixes)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


Bug Fixes 🐛

Openai

  • Avoid consuming iterables passed to the Completions API by alexander-alderman-webb in #5489
  • Avoid consuming iterables passed to the Embeddings API by alexander-alderman-webb in #5491

Other

  • (anthropic) Fix token accounting by shellmayr in #5490
  • (google-genai) Remove agent spans for simple requests by alexander-alderman-webb in #5443
  • (grpc) Read method from handler_call_details for grpcio >= 1.76 compat by yeung108 in #5521
  • (pydantic-ai) Adapt to missing ToolManager._call_tool by sentrivana in #5522
  • (utils) Use HEROKU_BUILD_COMMIT env var for default release by ericapisani in #5499

Documentation 📚

  • Add debugging advice by alexander-alderman-webb in #5517
  • New integration guide by alexander-alderman-webb in #5476

Internal Changes 🔧

Agents

  • Add security-review skill to agent configuration by ericapisani in #5498
  • Add sentry skills to be used by warden in CI reviews by ericapisani in #5485

Openai

  • Only handle streamed results when applicable by alexander-alderman-webb in #5553
  • Extract input in API-specific functions by alexander-alderman-webb in #5546
  • Separate output handling by alexander-alderman-webb in #5543

Openai Agents

  • Remove set_data_normalized for primitive attributes by alexander-alderman-webb in #5509
  • Expect new tool fields by alexander-alderman-webb in #5471

Other

  • (ai) Add configuration for dotagents by ericapisani in #5480
  • (anthropic) Remove set_data_normalized for primitive attributes by alexander-alderman-webb in #5504
  • (github) Add warden configuration by ericapisani in #5484
  • (pydantic-ai) Remove set_data_normalized for the gen_ai.response.model attribute by alexander-alderman-webb in #5512
  • (repo) Add .serena to .gitignore by ericapisani in #5464
  • 🤖 Update test matrix with new releases (02/24) by github-actions in #5524
  • 🤖 Update test matrix with new releases (02/23) by github-actions in #5503
  • 🤖 Update test matrix with new releases (02/19) by github-actions in #5483
  • 🤖 Update test matrix with new releases (02/18) by github-actions in #5475

🤖 This preview updates automatically when you update the PR.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 26, 2026

Codecov Results 📊

1851 passed | ⏭️ 166 skipped | Total: 2017 | Pass Rate: 91.77% | Execution Time: 3m 16s

All tests are passing successfully.

❌ Patch coverage is 0.00%. Project has 11846 uncovered lines.

Files with missing lines (173)
File Patch % Lines
langchain.py 3.28% ⚠️ 590 Missing
openai.py 4.87% ⚠️ 567 Missing
utils.py 0.00% ⚠️ 479 Missing
__init__.py 5.08% ⚠️ 374 Missing
starlette.py 5.34% ⚠️ 337 Missing
mcp.py 5.24% ⚠️ 253 Missing
anthropic.py 9.09% ⚠️ 250 Missing
__init__.py 3.91% ⚠️ 221 Missing
utils.py 81.23% ⚠️ 171 Missing and 47 partials
langgraph.py 5.29% ⚠️ 215 Missing
span_processor.py 0.00% ⚠️ 205 Missing
strawberry.py 8.54% ⚠️ 182 Missing
__init__.py 6.01% ⚠️ 172 Missing
aws_lambda.py 16.50% ⚠️ 167 Missing
spark_driver.py 0.00% ⚠️ 166 Missing
huggingface_hub.py 8.89% ⚠️ 164 Missing
sanic.py 9.60% ⚠️ 160 Missing
tracing_utils.py 80.03% ⚠️ 117 Missing and 33 partials
aiohttp.py 10.84% ⚠️ 148 Missing
ai_client.py 0.00% ⚠️ 145 Missing
litellm.py 0.00% ⚠️ 138 Missing
litestar.py 9.59% ⚠️ 132 Missing
starlite.py 8.33% ⚠️ 132 Missing
scope.py 91.23% ⚠️ 72 Missing and 41 partials
falcon.py 8.94% ⚠️ 112 Missing
flask.py 12.50% ⚠️ 112 Missing
asgi.py 20.71% ⚠️ 111 Missing
cohere.py 12.70% ⚠️ 110 Missing
dramatiq.py 0.00% ⚠️ 110 Missing
arq.py 9.92% ⚠️ 109 Missing
asgi.py 0.00% ⚠️ 109 Missing
pymongo.py 10.17% ⚠️ 106 Missing
beat.py 8.62% ⚠️ 106 Missing
caching.py 0.00% ⚠️ 106 Missing
client.py 87.45% ⚠️ 63 Missing and 41 partials
utils.py 0.00% ⚠️ 103 Missing
templates.py 0.00% ⚠️ 100 Missing
asyncpg.py 11.61% ⚠️ 99 Missing
quart.py 16.10% ⚠️ 99 Missing
gcp.py 0.00% ⚠️ 98 Missing
otlp.py 0.00% ⚠️ 97 Missing
utils.py 13.39% ⚠️ 97 Missing
agent_run.py 0.00% ⚠️ 97 Missing
models.py 4.95% ⚠️ 96 Missing
pyramid.py 13.76% ⚠️ 94 Missing
tornado.py 14.55% ⚠️ 94 Missing
__init__.py 0.00% ⚠️ 93 Missing
bottle.py 11.65% ⚠️ 91 Missing
middleware.py 0.00% ⚠️ 90 Missing
agent_run.py 0.00% ⚠️ 90 Missing
tools.py 0.00% ⚠️ 88 Missing
tracing.py 87.37% ⚠️ 60 Missing and 27 partials
_wsgi_common.py 31.50% ⚠️ 87 Missing
runner.py 0.00% ⚠️ 87 Missing
transaction_profiler.py 78.21% ⚠️ 73 Missing and 13 partials
loguru.py 11.58% ⚠️ 84 Missing
ray.py 0.00% ⚠️ 84 Missing
beam.py 0.00% ⚠️ 83 Missing
__init__.py 4.65% ⚠️ 82 Missing
invoke_agent.py 0.00% ⚠️ 79 Missing
clickhouse_driver.py 17.58% ⚠️ 75 Missing
rq.py 14.12% ⚠️ 73 Missing
pure_eval.py 0.00% ⚠️ 72 Missing
ariadne.py 14.46% ⚠️ 71 Missing
gql.py 10.13% ⚠️ 71 Missing
hub.py 67.36% ⚠️ 63 Missing and 7 partials
sqlalchemy.py 10.26% ⚠️ 70 Missing
fastapi.py 15.85% ⚠️ 69 Missing
graphene.py 12.82% ⚠️ 68 Missing
spotlight.py 55.56% ⚠️ 64 Missing and 3 partials
transactions.py 0.00% ⚠️ 67 Missing
huey.py 17.72% ⚠️ 65 Missing
boto3.py 14.86% ⚠️ 63 Missing
utils.py 16.00% ⚠️ 63 Missing
transport.py 88.35% ⚠️ 46 Missing and 16 partials
streaming.py 0.00% ⚠️ 62 Missing
httpx.py 16.44% ⚠️ 61 Missing
continuous_profiler.py 84.03% ⚠️ 50 Missing and 9 partials
chalice.py 16.18% ⚠️ 57 Missing
propagator.py 0.00% ⚠️ 57 Missing
server.py 0.00% ⚠️ 56 Missing
spark_worker.py 0.00% ⚠️ 56 Missing
graph_nodes.py 0.00% ⚠️ 52 Missing
_async_common.py 0.00% ⚠️ 51 Missing
views.py 0.00% ⚠️ 50 Missing
caches.py 16.95% ⚠️ 49 Missing
invoke_agent.py 0.00% ⚠️ 46 Missing
traces.py 47.06% ⚠️ 45 Missing
_asgi_common.py 16.67% ⚠️ 45 Missing
signals_handlers.py 0.00% ⚠️ 44 Missing
utils.py 85.88% ⚠️ 36 Missing and 5 partials
client.py 0.00% ⚠️ 40 Missing
_sync_common.py 22.00% ⚠️ 39 Missing
client.py 0.00% ⚠️ 38 Missing
_queue.py 68.89% ⚠️ 28 Missing and 8 partials
_span_batcher.py 35.71% ⚠️ 36 Missing
server.py 0.00% ⚠️ 34 Missing
monitoring.py 62.20% ⚠️ 31 Missing
launchdarkly.py 0.00% ⚠️ 31 Missing
trytond.py 0.00% ⚠️ 30 Missing
integration.py 0.00% ⚠️ 30 Missing
error_tracing.py 0.00% ⚠️ 29 Missing
envelope.py 92.31% ⚠️ 15 Missing and 13 partials
tools.py 0.00% ⚠️ 27 Missing
ai_client.py 13.33% ⚠️ 26 Missing
execute_tool.py 0.00% ⚠️ 26 Missing
_openai_completions_api.py 19.35% ⚠️ 25 Missing
redis_cluster.py 26.47% ⚠️ 25 Missing
typer.py 0.00% ⚠️ 24 Missing
tasks.py 0.00% ⚠️ 24 Missing
statsig.py 0.00% ⚠️ 23 Missing
queries.py 25.81% ⚠️ 23 Missing
api.py 89.26% ⚠️ 16 Missing and 6 partials
session.py 91.11% ⚠️ 8 Missing and 14 partials
__init__.py 80.00% ⚠️ 18 Missing and 4 partials
rust_tracing.py 90.21% ⚠️ 14 Missing and 8 partials
sessions.py 90.98% ⚠️ 12 Missing and 9 partials
worker.py 82.61% ⚠️ 16 Missing and 5 partials
logging.py 97.39% ⚠️ 4 Missing and 17 partials
unraisablehook.py 0.00% ⚠️ 21 Missing
openfeature.py 0.00% ⚠️ 20 Missing
execute_tool.py 0.00% ⚠️ 20 Missing
unleash.py 0.00% ⚠️ 19 Missing
_werkzeug.py 51.85% ⚠️ 13 Missing and 5 partials
model_request.py 0.00% ⚠️ 18 Missing
wsgi.py 91.34% ⚠️ 11 Missing and 6 partials
_batcher.py 86.25% ⚠️ 11 Missing and 5 partials
utils.py 0.00% ⚠️ 16 Missing
atexit.py 56.25% ⚠️ 14 Missing and 1 partials
excepthook.py 56.25% ⚠️ 14 Missing and 1 partials
redis.py 25.00% ⚠️ 15 Missing
cloud_resource_context.py 91.03% ⚠️ 13 Missing
gnu_backtrace.py 88.24% ⚠️ 6 Missing and 7 partials
__init__.py 27.78% ⚠️ 13 Missing
serializer.py 95.16% ⚠️ 9 Missing and 3 partials
executing.py 81.58% ⚠️ 7 Missing and 5 partials
threading.py 92.63% ⚠️ 7 Missing and 4 partials
redis_py_cluster_legacy.py 26.67% ⚠️ 11 Missing
asyncio.py 90.00% ⚠️ 8 Missing and 2 partials
utils.py 41.18% ⚠️ 10 Missing
handoff.py 0.00% ⚠️ 10 Missing
utils.py 85.19% ⚠️ 8 Missing and 2 partials
monitor.py 91.80% ⚠️ 5 Missing and 4 partials
__init__.py 72.41% ⚠️ 8 Missing and 1 partials
_types.py 76.67% ⚠️ 7 Missing and 1 partials
consts.py 0.00% ⚠️ 7 Missing
rb.py 30.00% ⚠️ 7 Missing
_openai_responses_api.py 33.33% ⚠️ 6 Missing
logger.py 94.87% ⚠️ 2 Missing and 2 partials
dedupe.py 95.00% ⚠️ 2 Missing and 2 partials
stdlib.py 99.25% ⚠️ 1 Missing and 3 partials
consts.py 0.00% ⚠️ 4 Missing
__init__.py 20.00% ⚠️ 4 Missing
__init__.py 0.00% ⚠️ 4 Missing
feature_flags.py 93.75% ⚠️ 2 Missing and 1 partials
serverless.py 94.44% ⚠️ 2 Missing and 1 partials
socket.py 96.00% ⚠️ 2 Missing and 1 partials
__init__.py 0.00% ⚠️ 3 Missing
__init__.py 40.00% ⚠️ 3 Missing
__init__.py 0.00% ⚠️ 3 Missing
__init__.py 0.00% ⚠️ 3 Missing
__init__.py 0.00% ⚠️ 3 Missing
_compat.py 94.87% ⚠️ 2 Missing
_log_batcher.py 100.00% ⚠️ 2 partials
consts.py 99.16% ⚠️ 2 Missing
debug.py 95.65% ⚠️ 1 Missing and 1 partials
scrubber.py 100.00% ⚠️ 2 partials
agent_workflow.py 71.43% ⚠️ 2 Missing
_init_implementation.py 100.00% ⚠️ 1 partials
_metrics_batcher.py 100.00% ⚠️ 1 partials
argv.py 100.00% ⚠️ 1 partials
consts.py 0.00% ⚠️ 1 Missing
consts.py 0.00% ⚠️ 1 Missing

Generated by Codecov Action

@alexander-alderman-webb alexander-alderman-webb marked this pull request as ready for review February 26, 2026 15:28
@alexander-alderman-webb alexander-alderman-webb requested a review from a team as a code owner February 26, 2026 15:28
Copy link
Member

@ericapisani ericapisani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small cleanups but approving so as not to block


def new_iterator() -> "Iterator[ChatCompletionChunk]":
nonlocal ttft
count_tokens_manually = True
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't appear to be needed in this iterator function - we never set it to false, so the conditional on line 644 will always run.

Copy link
Contributor Author

@alexander-alderman-webb alexander-alderman-webb Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! always great to remove some more dead code 👍

addressed in 3fb20f0 + 82843e2


async def new_iterator_async() -> "AsyncIterator[ChatCompletionChunk]":
nonlocal ttft
count_tokens_manually = True
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same situation here as above - this is never set to False so the conditional further down on line 688 will always run.

Copy link
Contributor Author

@alexander-alderman-webb alexander-alderman-webb Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants