Skip to content

fix: nan is not properly handled for Athena connections [sc-89372] #37071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ramiroaquinoromero wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix: nan is not properly handled for Athena connections [sc-89372] #37071

ramiroaquinoromero wants to merge 1 commit into from

Conversation

@ramiroaquinoromero
Copy link

@ramiroaquinoromero ramiroaquinoromero commented Jan 13, 2026

Summary

Fix NaN Handling in DataFrame Conversion for All Database Connections

Problem Description

When executing queries that produce NaN (Not a Number) values—such as division by zero operations—Superset would fail when attempting to save the results as a dataset or serialize the data to JSON. This issue affected all database connections, but was particularly noticeable with Amazon Athena, where queries like SELECT 0.00 / 0.00 return IEEE 754 NaN values.

The root cause was that Python's float('nan') values cannot be serialized to valid JSON without special handling. When users attempted to save query results containing NaN values as datasets, the JSON serialization would fail, resulting in errors displayed in the UI.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codeant-ai-for-open-source
Copy link
Contributor

codeant-ai-for-open-source bot commented Jan 13, 2026

CodeAnt AI is reviewing your PR.

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@bito-code-review
Copy link
Contributor

bito-code-review bot commented Jan 13, 2026 •
edited
Loading

Code Review Agent Run #1c6679

Actionable Suggestions - 0
Review Details
  • Files reviewed - 1 · Commit Range: dfb1ebe..dfb1ebe
    • superset/dataframe.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful
Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

  • /review - Manually triggers a full AI review.

  • /pause - Pauses automatic reviews on this pull request.

  • /resume - Resumes automatic reviews.

  • /resolve - Marks all Bito-posted review comments as resolved.

  • /abort - Cancels all in-progress reviews.

Refer to the documentation for additional commands.

Configuration

This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.

Documentation & Help

AI Code Review powered by Bito Logo

@dosubot dosubot bot added the data:databases Related to database configurations and connections label Jan 13, 2026
@codeant-ai-for-open-source
Copy link
Contributor

codeant-ai-for-open-source bot commented Jan 13, 2026

Nitpicks 🔍

🔒 No security issues identified
⚡ Recommended areas for review

  • Large integers not detected
    The helper _convert_big_integers only checks for built-in int. NumPy integer types (e.g., numpy.int64) or other integer-like types returned by pandas may not be recognized and therefore won't be stringified when exceeding JS_MAX_INTEGER, which can lead to numeric precision loss or incorrect behavior in JS consumers.

  • Infinity values not handled
    The new NaN-to-None logic uses pd.isna which covers NaN/NA/NaT, but does not cover positive/negative infinity (inf, -inf). Such values are also not valid JSON numbers and should be normalized (e.g., to None or a safe string) before serialization.

  • Potential type coercion needed
    Values coming from dframe.to_dict(orient="records") may include numpy scalars (e.g., numpy.float64, numpy.int64) and pandas extension scalars. The code currently treats values as plain Python types which may leave numpy scalars as-is; consider normalizing to native Python types when appropriate to ensure consistent downstream JSON serialization and isinstance checks behave as expected.

@netlify
Copy link

netlify bot commented Jan 13, 2026

✅ Deploy Preview for superset-docs-preview ready!

Name Link
🔨 Latest commit dfb1ebe
🔍 Latest deploy log https://app.netlify.com/projects/superset-docs-preview/deploys/6965e4581ab999000887e193
😎 Deploy Preview https://deploy-preview-37071--superset-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

codeant-ai-for-open-source[bot]
codeant-ai-for-open-source bot reviewed Jan 13, 2026
View reviewed changes
@codeant-ai-for-open-source
Copy link
Contributor

codeant-ai-for-open-source bot commented Jan 13, 2026

CodeAnt AI finished reviewing your PR.

@codecov
Copy link

codecov bot commented Jan 13, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.86%. Comparing base (481bfa0) to head (dfb1ebe).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files 
@@             Coverage Diff             @@
##           master   #37071       +/-   ##
===========================================
+ Coverage        0   67.86%   +67.86%     
===========================================
  Files           0      642      +642     
  Lines           0    47989    +47989     
  Branches        0     5248     +5248     
===========================================
+ Hits            0    32568    +32568     
- Misses          0    14138    +14138     
- Partials        0     1283     +1283
Flag Coverage Δ
hive 42.88% <50.00%> (?)
mysql 65.88% <100.00%> (?)
postgres 65.93% <100.00%> (?)
presto 46.46% <50.00%> (?)
python 67.83% <100.00%> (?)
sqlite 65.65% <100.00%> (?)
unit 100.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@betodealmeida betodealmeida Awaiting requested review from betodealmeida

@msyavuz msyavuz Awaiting requested review from msyavuz

1 more reviewer

@codeant-ai-for-open-source codeant-ai-for-open-source[bot] codeant-ai-for-open-source[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

data:databases Related to database configurations and connections size/S

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ramiroaquinoromero