Skip to content

Comments

CloudWatch: fix Scheduler shutdown#13822

Open
bentsku wants to merge 3 commits intomainfrom
fix-cloudwatch-shutdown
Open

CloudWatch: fix Scheduler shutdown#13822
bentsku wants to merge 3 commits intomainfrom
fix-cloudwatch-shutdown

Conversation

@bentsku
Copy link
Contributor

@bentsku bentsku commented Feb 23, 2026

Motivation

While working on S3, I had a test interacting with SQS where LocalStack wouldn't shutdown. I tracked it down to CloudWatch re-start its scheduler after it was tasked to shut it down:

2026-02-23T11:52:49.470  INFO --- [  MainThread] localstack.runtime.shutdown : [shutdown] Stopping all services
2026-02-23T11:52:49.470 DEBUG --- [  MainThread] localstack.services.cloudwatch.provider_v2 : stopping cloudwatch scheduler
2026-02-23T11:52:49.474 DEBUG --- [  MainThread] localstack.runtime.runtime : [shutdown] Shutting down runtime server ...
2026-02-23T11:52:49.909 DEBUG --- [PoolThread-twisted.internet.reactor-2] localstack.services.cloudwatch.provider_v2 : starting cloudwatch scheduler
2026-02-23T11:52:49.909  INFO --- [PoolThread-twisted.internet.reactor-2] localstack.utils.bootstrap : Execution of "require" took 525.07ms
2026-02-23T11:52:49.913  INFO --- [et.reactor-2] l.request.internal.aws     : AWS cloudwatch.PutMetricData => 200; 000000000000/us-east-1; PutMetricDataInput({'Namespace': 'AWS/SQS', 'MetricData': [{'MetricName': 'SentMessageSize', 'Dimensions': [{'Name': 'QueueName', 'Value': 'test-queue-cf800955'}], 'Timestamp': datetime.datetime(2026, 2, 23, 10, 52, 49, tzinfo=datetime.timezone.utc), 'Value': 218.0, 'Unit': 'Bytes'}]}, headers={'Accept-Encoding': 'identity', 'X-Amz-Target': 'GraniteServiceVersion20100801.PutMetricData', 'Content-Type': 'application/x-amz-json-1.0', 'x-amzn-query-mode': 'true', 'User-Agent': 'Boto3/1.42.54 md/Botocore#1.42.54 md/awscrt#0.31.1 ua/2.1 os/macos#25.3.0 md/arch#arm64 lang/python#3.13.7 md/pyimpl#CPython m/N,b,Z,E,e cfg/retry-mode#standard Botocore/1.42.54', 'x-localstack-data': '{}', 'host': 'localhost:4566', 'X-Amz-Date': '20260223T105249Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=000000000000/20260223/us-east-1/monitoring/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target;x-amzn-query-mode;x-localstack-data, Signature=9dbdd9ee6137d85c46086b4c4404ffa585afd31ca9f40a31aa5fb8546aac70c0', 'amz-sdk-invocation-id': 'd7525d8e-e0b8-47e6-88d2-5bd235d68e34', 'amz-sdk-request': 'attempt=1', 'Content-Length': '202', 'x-moto-account-id': '000000000000'}); Response({}, headers={'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '2', 'x-amzn-RequestId': '97f45fae-79ba-4c88-ab93-f18878d9fe63', 'x-localstack': 'true'})
2026-02-23T11:52:49.913 DEBUG --- [PoolThread-twisted.internet.reactor-2] localstack.aws.protocol.serializer : No accept header given. Using request's Content-Type (application/x-amz-json-1.0) as preferred response Content-Type.
2026-02-23T11:52:49.914  INFO --- [  MainThread] localstack.testing.pytest.in_memory_localstack : waiting for runtime to stop

We can see that the scheduler is being shutdown then restarted straight away.

I've modified the logic a bit to avoid this issue, and also registered the scheduler to with the shutdown handlers, so that the thread wouldn't block in case something goes wrong when shutting down the service.

Changes

  • register scheduler thread with shutdown hooks
  • modify the logic to not recreate the whole scheduler at once but instead rely on the scheduler to start and stop itself

Tests

Test the tests.aws.services.s3.test_s3_api.TestS3BucketNotificationConfiguration.test_bucket_notification_with_invalid_filter_rules test with and without this fix.

Related

@bentsku bentsku self-assigned this Feb 23, 2026
@bentsku bentsku added aws:cloudwatch Amazon CloudWatch semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Feb 23, 2026
@github-actions
Copy link

github-actions bot commented Feb 23, 2026 •
edited
Loading

Test Results - Preflight, Unit

23 066 tests   - 57   21 175 ✅  - 77   6m 8s ⏱️ -2s
     1 suites ± 0    1 891 💤 +20 
     1 files   ± 0        0 ❌ ± 0 

Results for commit 3cc349e. ± Comparison against base commit a857a31.

This pull request removes 74 and adds 17 tests. Note that renamed tests count towards both. 
tests.unit.aws.protocol.test_op_router ‑ test_create_op_router_works_for_every_service[evidently]
tests.unit.aws.protocol.test_op_router ‑ test_create_op_router_works_for_every_service[iotanalytics]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-BatchEvaluateFeature]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-CreateExperiment]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-CreateFeature]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-CreateLaunch]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-CreateProject]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-CreateSegment]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-DeleteExperiment]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[evidently-rest-json-DeleteFeature]
…

tests.unit.aws.protocol.test_op_router ‑ test_create_op_router_works_for_every_service[signer-data]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-data-rest-json-GetRevocationStatus]
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_double_dollar_raises
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_empty_expression
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_filter_expression_with_variable
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_filter_predicate_bare_dollar_equals_zero
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_filter_predicate_with_context_variable
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_lone_dollar_ignored
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_multiple_variables
tests.unit.services.stepfunctions.test_jsonata_variable_references.TestExtractJsonataVariableReferences ‑ test_no_variables
…
This pull request skips 19 tests. 
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-AddProfilePermission]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-CancelSigningProfile]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-DescribeSigningJob]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-GetRevocationStatus]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-GetSigningPlatform]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-GetSigningProfile]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-ListProfilePermissions]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-ListSigningJobs]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-ListSigningPlatforms]
tests.unit.aws.test_service_router ‑ test_service_router_works_for_every_service[signer-rest-json-ListSigningProfiles]
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Feb 23, 2026 •
edited
Loading

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 2s ⏱️ -3s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit 3cc349e. ± Comparison against base commit a857a31.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Feb 23, 2026 •
edited
Loading

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   49m 8s ⏱️ - 1h 12m 33s
1 243 tests  - 4 089  1 165 ✅  - 3 799  78 💤  - 290  0 ❌ ±0 
1 245 runs   - 4 089  1 165 ✅  - 3 799  80 💤  - 290  0 ❌ ±0 

Results for commit 3cc349e. ± Comparison against base commit a857a31.

This pull request removes 4089 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Feb 23, 2026 •
edited
Loading

Test Results (amd64) - Integration, Bootstrap

    5 files  ±    0      5 suites  ±0   1h 5m 6s ⏱️ - 1h 35m 43s
1 267 tests  - 4 470  1 191 ✅  - 4 022  76 💤  - 448  0 ❌ ±0 
1 273 runs   - 4 470  1 191 ✅  - 4 022  82 💤  - 448  0 ❌ ±0 

Results for commit 3cc349e. ± Comparison against base commit a857a31.

This pull request removes 4470 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Feb 23, 2026 •
edited
Loading

Test Results - Alternative Providers

180 tests   - 1 071    39 ✅  - 671   2m 30s ⏱️ - 28m 35s
  1 suites  -     3   141 💤  - 400 
  1 files    -     3     0 ❌ ±  0 

Results for commit 3cc349e. ± Comparison against base commit a857a31.

This pull request removes 1071 tests. 
tests.aws.services.cloudformation.api.test_changesets ‑ test_autoexpand_capability_requirement
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_and_then_remove_non_supported_resource_change_set
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_and_then_remove_supported_resource_change_set
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_and_then_update_refreshes_template_metadata
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_change_set_create_existing
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_change_set_invalid_params
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_change_set_missing_stackname
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_change_set_no_changes
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_change_set_update_nonexisting
tests.aws.services.cloudformation.api.test_changesets ‑ test_create_change_set_update_without_parameters
…

♻️ This comment has been updated with latest results.

@bentsku bentsku marked this pull request as ready for review February 23, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pinzon pinzon Awaiting requested review from pinzon pinzon is a code owner

@steffyP steffyP Awaiting requested review from steffyP steffyP is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@bentsku bentsku

Labels

aws:cloudwatch Amazon CloudWatch docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bentsku