Skip to content

Speedup, refactoring and modernization#76

Merged
fjwillemsen merged 40 commits intopython-constraint:masterfrom
fjwillemsen:speedup
Aug 30, 2023
Merged

Speedup, refactoring and modernization#76
fjwillemsen merged 40 commits intopython-constraint:masterfrom
fjwillemsen:speedup

Conversation

@fjwillemsen
Copy link
Member

@fjwillemsen fjwillemsen commented Aug 25, 2023 •
edited
Loading

Hi all! A while ago I implemented python-constraint in Kernel Tuner, an auto-tuning framework where we need to obtain all solutions on user-specified parameters and constraints before starting optimization. python-constraint works great, but it had some room for improvement, especially performance-wise.
I thus decided to fork python-constraint and make some changes, the following:

  • Cythonized the package
  • Added the OptimizedBacktracking solver based on issue #62
  • Added type-hints to improve Cythonization
  • Added the MaxProd and MinProd constraints
  • Improved pre-processing for the MaxSum constraint
  • Optimized the Function constraint
  • Added getSolutionsOrderedList and getSolutionsAsListDict functions for efficient result shaping
  • Overall optimization of common bottlenecks
  • Split the codebase into multiple files for convenience
  • Switched from setup.py to pyproject.toml
  • Achieved and requires test coverage of at least 65%
  • Added nox for local testing against all supported Python versions
  • Added ruff for codestyle testing
  • Dropped Python 3.4, 3.5, 3.6, 3.7 support

In all, this results in a huge speedup of >150x on a benchmark of 112 synthetic searchspaces:
image

I'd like to hear what the mainainers think about these changes and would be happy to see them integrated! Before releasing this, I think it would be a good idea to revise the package publishing process, and consider switching to GitHub Actions (I can help with this). If there are any questions, please let me know!

fjwillemsen added 30 commits June 14, 2023 16:42
@fjwillemsen
Added optimizations by @maciejlibraryx
45cffba
@fjwillemsen
Local Cythonizing working, split code to clean init file
ee21d71
@fjwillemsen
Refactored package and module structure
8ab7f8d
@fjwillemsen
Updated project metadata
81e23b1
@fjwillemsen
>10x speedup by rewrite and Cythonization of FunctionConstraint
923aa77
@fjwillemsen
Refactored from one monolithic file to appropriate split files, impro…
ed6377d 
…ved comments.
@fjwillemsen
Recompiled code
4dce751
@fjwillemsen
Added function for obtaining the solutions in a specified order
a8a17ae
@fjwillemsen
Optimized getSolutions
4952466
@fjwillemsen
Added a bruteforce solver
eb07608
@fjwillemsen
Attempt to use Cython Parallelism
054c9b8
@fjwillemsen
Rolled back parallel and bruteforce implementations: does not work we…
e586e00 
…ll outside GIL; optimized non-forwardchecking instead
@fjwillemsen
Added MaxProdConstraint to constraint
e7ca5e6
@fjwillemsen
Optimizations via type hints
9e90b94
@fjwillemsen
Attempt to improve performance by mapping variable names to integers
fb0bdd1
@fjwillemsen
Revert to previous commit
11faaa8
@fjwillemsen
Revert to previous commit
0e91422
@fjwillemsen
Revert "Revert to previous commit"
d0c72bc 
This reverts commit 0e91422.
@fjwillemsen
Revert "Revert to previous commit"
abb94f2 
This reverts commit 11faaa8.
@fjwillemsen
Revert "Attempt to improve performance by mapping variable names to i…
e3e7ead 
…ntegers"

This reverts commit fb0bdd1.
@fjwillemsen
Properly typed function signatures
fe9eb3b
@fjwillemsen
Added MinProdConstraint, removed MaxProd and MinProd preprocessing be…
6df4c58 
…cause it can result in invalid outcomes
@fjwillemsen
Fixed an issue with MaxSumConstraint preprocessing that resulted in i…
d6b3725 
…nvalid outcomes, added preprocessing to MaxProdConstraint
@fjwillemsen
Fixed an issue in MinProdConstraint that caused invalid outcomes
4920372
@fjwillemsen
Added preprocessing for MaxProd and MinProd, improved getSolutionsAsL…
5f47f88 
…istDict
@fjwillemsen
Complete switching to pyproject, added Nox tests, fix with type hints…
88c2e1e 
… for Python <3.9
@fjwillemsen
Added and satisfied codestyle tests, pyproject.toml tests, improved p…
84457a9 
…yproject.toml
@fjwillemsen
Added coverage checking, added tests
a8e86b0
@fjwillemsen
Minor change to solvers
5c9b1e3
@fjwillemsen
Updated changelog
9fef5d4
@s-celles
Copy link
Member

s-celles commented Aug 25, 2023

Hi @fjwillemsen

Very pleased to see such a contribution. I'm no longuer maintaining this repository. I sent you and invite to be member of https://github.com/python-constraint/ GH org and also grant you write access to https://github.com/python-constraint/python-constraint/
Feel free to take the lead on the project and improve it.
If you need something more just let me know.

Kind regards

Sébastien

@fjwillemsen
Copy link
Member Author

fjwillemsen commented Aug 25, 2023

@scls19fr thank you very much, I'll do my best :)
Perhaps it is not practical, but given that you will no longer maintain, could you make me owner instead of member? It appears I currently do not have access to the repository secrets, which prohibits a release on PyPI

@s-celles
Copy link
Member

s-celles commented Aug 25, 2023

Done.

@s-celles
Copy link
Member

s-celles commented Aug 25, 2023

https://pypi.org/project/python-constraint/ belongs to @niemeyer so I can't give you access.
You can either ask him or create a new PyPi project python-constraint2 or 3 for example.

@fjwillemsen
Copy link
Member Author

fjwillemsen commented Aug 25, 2023

Thank you! I have contacted @niemeyer via email, for continuity it would be ideal to keep it on the same package, I hope this will be possible.
Thank you very much for all of your contributions to python-constraint throughout the years!

@fjwillemsen fjwillemsen merged commit cd42626 into python-constraint:master Aug 30, 2023
@s-celles
Copy link
Member

s-celles commented Aug 30, 2023

Great work! Thanks @fjwillemsen
Renaming master branch to main should also be considered
https://github.com/github/renaming

@fjwillemsen
Copy link
Member Author

fjwillemsen commented Aug 30, 2023

Good suggestion! Renamed master branch to main

@fjwillemsen fjwillemsen deleted the speedup branch August 30, 2023 06:39
@fjwillemsen
Copy link
Member Author

fjwillemsen commented Oct 11, 2023

My emails to @niemeyer regarding PyPI access seem to go unanswered. @scls19fr could you perhaps get in touch with @niemeyer, or provide me with an alternative means of contact? My contact details are on my website.

@s-celles
Copy link
Member

s-celles commented Oct 11, 2023

what about a new name?
python-constraint2 or 3 or something else.
So we could be totally free.
I think it will be the simplest solution.

@fjwillemsen
Copy link
Member Author

fjwillemsen commented Oct 11, 2023

Yes, definitely, I had already created a python-constaint2 for beta testing. However, python-constraint had 148,000 downloads in the past month, which is a very solid user base I would prefer not to abandon if we have a choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fjwillemsen @s-celles