Planet Python

Last update: November 19, 2024 01:43 PM UTC

November 19, 2024

Mike Driscoll

How to Debug Your Textual Application

Textual is a great Python package for creating a lightweight, powerful, text-based user interface. That means you can create a GUI in your terminal with Python without learning curses! But what happens when you encounter some problems that require debugging your application? A TUI takes over your terminal, which means you cannot see anything from Python’s print() statement.

Wait? What about your IDE? Can that help? Actually no. When you run a TUI, you need a fully functional terminal to interact with it. PyCharm doesn’t work well with Textual. WingIDE doesn’t even have a terminal emulator. Visual Studio Code also doesn’t work out of the box, although you may be able to make it work with a custom json or yaml file. But what do you do if you can’t figure that out?

That is the crux of the problem and what you will learn about in this tutorial: How to debug Textual applications!

Getting Started

To get the most out of this tutorial, make sure you have installed Textual’s development tools by using the following command:

python -m pip install textual-dev --upgrade

Once you have the latest version of textual-dev installed, you may continue!

Debugging with Developer Mode

When you want to debug a Textual application, you need to open two terminal windows. On Microsoft Windows, you can open two Powershell or two Command Prompts. In the first terminal, run this command:

textual console

The Textual console will listen for any Textual application running in developer mode. But first, you need some kind of application to test with. Open up your favorite Python IDE and create a new file called hello_textual.py. Then enter the following code into it:

from textual.app import App, ComposeResult
from textual.widgets import Button


class WelcomeButton(App):

    def compose(self) -> ComposeResult:
        yield Button("Exit")

    def on_button_pressed(self) -> None:
        self.mount(Button("Other"))


if __name__ == "__main__":
    app = WelcomeButton()
    app.run()

To run a Textual application, use the other terminal you opened earlier. The one that isn’t running Textual Console in it. Then run this command:

textual run --dev hello_textual.py

You will see the following in your terminal:

If you switch over to the other terminal, you will see a lot of output that looks something like this:

Now, if you want to test that you are reaching a part of your code in Textual, you can add a print() function now to your on_button_pressed() method. You can also use self.log.info() which you can read about in the Textual documentation.

Let’s update your code to include some logging:

from textual.app import App, ComposeResult
from textual.widgets import Button


class WelcomeButton(App):

    def compose(self) -> ComposeResult:
        yield Button("Exit")
        print("The compose() method was called!")

    def on_button_pressed(self) -> None:
        self.log.info("You pressed a button")
        self.mount(Button("Other"))


if __name__ == "__main__":
    app = WelcomeButton()
    app.run()

Now, when you run this code, you can check your Textual Console for output. The print() statement should be in the Console without you doing anything other than running the code. You must click the button to get the log statement in the Console.

Here is what the log output will look like in the Console:

And here is an example of what you get when you print() to the Console:

There’s not much difference here, eh? Either way, you get the information you need and if you need to print out Python objects, this can be a handy debugging tool.

If you find the output in the Console to be too verbose, you can use -x or --exclude to exclude log groups. Here’s an example:

textual console -x SYSTEM -x EVENT -x DEBUG -x INFO

In this version of the Textual Console, you are suppressing SYSTEM, EVENT, DEBUG, and INFO messages.

Launch your code from earlier and you will see that the output in your Console is greatly reduced:

Now, let’s learn how to use notification as a debugging tool.

Debugging with Notification

If you like using print() statements then you will love that Textual’s App() class provides a notify() method. You can call it anywhere in your application using self.app.notify() , along with a message. If you are in your App class, you can reduce the call to simply self.notify().

Let’s take the example from earlier and update it to use the notify method instead:

from textual.app import App, ComposeResult
from textual.widgets import Button


class WelcomeButton(App):

    def compose(self) -> ComposeResult:
        yield Button("Exit")

    def on_button_pressed(self) -> None:
        self.mount(Button("Other"))
        self.notify("You pressed the button!")


if __name__ == "__main__":
    app = WelcomeButton()
    app.run()

The notify() method takes the following parameters:

• message – The message you want to display in the notification
• title – An optional title to add to the message
• severity – The message’s severity, which translates to a different color for the notification. You may use “information”, “error” or “warning”
• timeout – The timeout in seconds for how long to show the message

Try editing the notification to use more of these features. For example, you could update the code above to use this instead:

self.notify("You pressed the button!", title="Info Message", severity="error")

Textual’s App class also provides a bell() method you can call to play the system bell. You could add this to really get the user’s attention, assuming they have the system bell enabled on their computer.

Wrapping Up

Debugging your TUI application successfully is a skill. You need to know how to find errors, and Textual’s dev mode makes this easier. While it would be great if a Python IDE had a fully functional terminal built into it, that is a very niche need. So it’s great that Textual included the tooling you need to figure out your code.

Give these tips a try, and you’ll soon be able to debug your Textual applications easily!

The post How to Debug Your Textual Application appeared first on Mouse Vs Python.

November 19, 2024 01:09 PM UTC

Ned Batchelder

Loop targets

I posted a Python tidbit about how for loops can assign to other things than simple variables, and many people were surprised or even concerned:

params = {

    "query": QUERY,

    "page_size": 100,

}



# Get page=0, page=1, page=2, ...

for params["page"] in itertools.count():

    data = requests.get(SEARCH_URL, params).json()

    if not data["results"]:

        break

    ...

This code makes successive GET requests to a URL, with a params dict as the data payload. Each request uses the same data, except the “page” item is 0, then 1, 2, and so on. It has the same effect as if we had written it:

for page_num in itertools.count():

    params["page"] = page_num

    data = requests.get(SEARCH_URL, params).json()

One reply asked if there was a new params dict in each iteration. No, loops in Python do not create a scope, and never make new variables. The loop target is assigned to exactly as if it were an assignment statement.

As a Python Discord helper once described it,

While loops are “if” on repeat. For loops are assignment on repeat.

A loop like for <ANYTHING> in <ITER>: will take successive values from <ITER> and do an assignment exactly as this statement would: <ANYTHING> = <VAL>. If the assignment statement is ok, then the for loop is ok.

We’re used to seeing for loops that do more than a simple assignment:

for i, thing in enumerate(things):

    ...



for x, y, z in zip(xs, ys, zs):

    ...

These work because Python can assign to a number of variables at once:

i, thing = 0, "hello"

x, y, z = 1, 2, 3

Assigning to a dict key (or an attribute, or a property setter, and so on) in a for loop is an example of Python having a few independent mechanisms that combine in uniform ways. We aren’t used to seeing exotic combinations, but you can reason through how they would behave, and you would be right.

You can assign to a dict key in an assignment statement, so you can assign to it in a for loop. You might decide it’s too unusual to use, but it is possible and it works.

November 19, 2024 10:40 AM UTC

November 18, 2024

PyCharm

JetBrains AI Assistant 2024.3 is here! A highlight of this release is the flexibility to choose your preferred chat model. Select between Google Gemini, OpenAI, or local models to tailor interactions for a more customized experience. 

This update also brings advanced code completion for all major programming languages, improved context management, and the ability to generate inline prompts directly within the editor.

More control over your chat experience: Choose between Gemini, OpenAI, and local models 

You can now select your preferred AI chat model, choosing from cloud model providers like Google Gemini and OpenAI, or connect to local models. This expanded selection allows you to customize the AI chat’s responses to your specific workflow, offering a more adaptable and personalized experience. 

Google’s Gemini models now available

The lineup of LLMs used by JetBrains AI now includes Gemini 1.5 Pro 002 and Flash 002. These models are designed to deliver advanced reasoning capabilities and optimized performance for a wide range of tasks. The Pro version excels in complex applications, while Flash is tailored for high-volume, low-latency scenarios. Now, AI Assistant users can leverage the power of Gemini models alongside our in-house Mellum and OpenAI options.

Local model support via Ollama

In addition to cloud-based models, you can now connect the AI chat to local models available through Ollama. This is particularly useful for users who need more control over their AI models, offering enhanced privacy, flexibility, and the ability to run models on local hardware.

To add an Ollama model to the chat you need to enable Ollama support in AI Assistant’s settings and configure the connection to your Ollama instance. 

Improved context management

In this update, we’ve made context handling in AI Assistant more transparent and intuitive. A revamped UI lets you view and manage every element included as context, providing full visibility and control. The open file and any selected code within it are now automatically added to the context, and you can easily add or remove files as needed, customizing the context to fit your workflow. Additionally, you can attach project-wide instructions to guide AI Assistant’s responses throughout your codebase.

Cloud code completion with broader language support

JetBrains has released its own large language model (LLM) model, Mellum, specifically designed to enhance cloud-based code completion for developers. This new model, specialized for coding tasks, has expanded support for several new languages, including JavaScript, TypeScript, HTML, C#, C, C++, Go, PHP, and Ruby. Now, the code completion experience is unified across JetBrains IDEs, offering syntax highlighting for suggested code, the flexibility to accept suggestions token by token or line by line, and overall reduced latency.

Local code completion enhancements: Multi-line support for Python and contextual improvements

Local code completion has significantly improved, now offering multi-line suggestions for Python. Additionally, optimizations have been made across other programming languages. For Kotlin, retrieval-augmented generation (RAG) enables the model to pull information from multiple project files, ensuring the most relevant suggestions. The support for JavaScript, TypeScript, and CSS has also seen enhancements to their existing RAG functionality. Furthermore, local code completion has been introduced for HTML.

These improvements mean that suggestions appear faster across all languages, creating a more seamless coding experience. Best of all, local code completion is included for free in your IDE, allowing you to start utilizing these powerful features immediately.

Streamlined in-editor experience with inline AI prompts

The new inline AI prompt feature in AI Assistant introduces a direct way to enter your prompts right in the editor. Just start typing your request in natural language and the AI Assistant will recognize it and generate a suggestion. Inline AI prompts are context-aware, automatically including related files and symbols for more accurate code generation. This feature supports Java, Kotlin, Scala, Groovy, JavaScript, TypeScript, Python, JSON, YAML, PHP, Ruby, and Go file formats, and is available to all AI Assistant users.

We also improved the visibility of changes applied. There is now a purple mark in the gutter next to lines changed by AI Assistant, so you can easily see what has been updated.

Make multiple file-wide updates easily

AI Assistant now offers file-wide code generation, enabling streamlined edits across an entire file. This functionality allows for modifications across multiple code sections, including adding necessary imports, updating references, and defining missing declarations.Currently available for Java and Kotlin, it is triggered by the Generate Code action when no specific selection is made in the editor, offering a seamless experience for broad, file-wide adjustments.

Get instant answers about IDE features and settings in AI Chat

Say goodbye to searching through settings or documentation! With the new /docs command, you can now access documentation-based answers directly in the AI chat. Simply ask AI Assistant about a feature, and it will provide interactive step-by-step guidance.

AI-powered quick-fix for faster error resolution

When a JetBrains IDE inspection flags a problem – whether it’s a syntax error, missing import, or something else – it suggests a quick-fix directly within the editor. With the latest update, Fix with AI takes this a step further. This new capability uses AI context awareness to suggest fixes that are more precise and applicable to your specific coding context, making it faster and easier to resolve coding problems without any manual input.

Explore AI Assistant and share your feedback

Explore these updates and let AI Assistant streamline your development workflow even further. As always, we look forward to hearing your feedback. You can also tell us about your experience via the Share your feedback link in the AI Assistant tool window or by submitting feature requests or bug reports in YouTrack.

Happy developing!

November 18, 2024 07:41 PM UTC

Python Morsels

Python's pathlib module

Python's pathlib module is the tool to use for working with file paths. See pathlib quick reference tables and examples.

>>> import pathlib
>>> path = Path("/home/trey/proj/readme.md")
>>> relative = Path("readme.md")
>>> base = Path("/home/trey/proj")
>>> new = Path("/home/trey/proj/sub")
>>> target = path.with_suffix(".txt")  # .md -> .txt
>>> pattern = "*.md"
>>> name = "sub/f.txt"

Read the full article: https://www.pythonmorsels.com/pathlib-module/

November 18, 2024 05:00 PM UTC

Real Python

Interacting With Python

There are multiple ways of interacting with Python, and each can be useful for different scenarios. You can quickly explore functionality in Python’s interactive mode using the built-in Read-Eval-Print Loop (REPL), or you can write larger applications to a script file using an editor or Integrated Development Environment (IDE).

In this tutorial, you’ll learn how to:

• Use Python interactively by typing code directly into the interpreter
• Execute code contained in a script file from the command line
• Work within a Python Integrated Development Environment (IDE)
• Assess additional options, such as the Jupyter Notebook and online interpreters

Before working through this tutorial, make sure that you have a functioning Python installation at hand. Once you’re set up with that, it’s time to write some Python code!

Get Your Code: Click here to get the free sample code that you’ll use to learn about interacting with Python.

Take the Quiz: Test your knowledge with our interactive “Interacting With Python” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Interacting With Python

In this quiz, you'll test your understanding of the different ways of interacting with Python. By working through this quiz, you'll revisit key concepts related to Python interaction in interactive mode using the REPL, through Python script files, and within IDEs and code editors.

Hello, World!

There’s a long-standing custom in computer programming that the first code written in a newly installed language is a short program that displays the text Hello, World! to the console.

In Python, running a “Hello, World!” program only takes a single line of code:

Python

print("Hello, World!")

Copied!

Here, print() will display the text Hello, World! in quotes to your screen. In this tutorial, you’ll explore several ways to execute this code.

Running Python in Interactive Mode

The quickest way to start interacting with Python is in a Read-Eval-Print Loop (REPL) environment. This means starting up the interpreter and typing commands to it directly.

When you interact with Python in this way, the interpreter will:

• Read the command you enter
• Evaluate and execute the command
• Print the output (if any) to the console
• Loop back and repeat the process

The interactive session continues like this until you instruct the interpreter to stop. Using Python in this interactive mode is a great way to test short snippets of Python code and get more familiar with the language.

• Windows
• Linux + macOS

When you install Python using an installer, the Start menu shows a program group labeled Python 3.x. The label may vary depending on the particular installation you chose. Click on that item to start the Python interpreter.

Alternatively, you can open your Command Prompt or PowerShell application and type the py command to launch it:

Windows PowerShell

PS> py

Copied!

To start the Python interpreter, open your Terminal application and type python3 to launch it from the command line:

Shell

$ python3

Copied!

If you’re unfamiliar with this application, then you can use your operating system’s search function to find it.

After pressing Enter, you should see a response from the Python interpreter similar to the one below:

Python

Python 3.13.0 (main, Oct 14 2024, 10:34:31) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Copied!

If you’re not seeing the >>> prompt, then you’re not talking to the Python interpreter. This could be because Python is either not installed or not in the path of your terminal window session.

Note: If you need additional help to get to this point, then you can check out the How to Install Python on Your System: A Guide tutorial.

If you’re seeing the prompt, then you’re off and running! With these next steps, you’ll execute the statement that displays "Hello, World!" to the console:

1. Ensure that Python displays the >>> prompt, and that you position your cursor after it.
2. Type the command print("Hello, World!") exactly as shown.
3. Press the Enter key.

Read the full article at https://realpython.com/interacting-with-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 18, 2024 02:00 PM UTC

Go Deh

There's the easy way...

Best seen on a larger than landscape phone

Someone blogged about a particular problem:

The solution they blogged used a sort which meant it could not be O(n) in time, but the problem looked good so I gave it some thought.

Sets! sets are O(1) in Python and are good for looking things up.

What if when looking at the inputted numbers, one at a time, you also looked for other ints in the input that would extend the int you have to form a longer  range?  Keep tab of the longest range so far and if you remove ints from the pool as they form ranges, when the pool is empty, you should know the longest range.

I added the printout of the longest range too.

My code

Sample output

Another Algorithm

What if, you kept and extended ranges untill you amassed all ranges then chose the longest? I need to keep the hash lookup. dict key lookup should also be O(1).  What to look up? Look up ints that would extend a range!

If you have an existing (integer) range, say 1..3 inclusive of end points then finding 0 would extend the range to 0..3 or finding one more than the range maximum, 4 would extend the original range to 1..4 

So if you have ranges then they could be extended by finding rangemin - 1 or rangemax +1. I call then extends

If you do find that the next int from the input ints is also an extends value then you need to find the range that it extends, (by lookup), so you can modify that range. - use a dict to map extends to their range and checking if an int is in the extends dict keys should also take O(1) time.

I took that sketch of an algorithm and started to code. It took two evenings to finally get something that worked and I had to work out several details that were trying. The main problem was what about coalescing ranges? if you have ranges 1..2 and 4..5 what happens when you see a 3? the resultant is the single range 1..5. It took particular test cases and extensive debugging to work out that the extends2range mapping should map to potentially more than one range and that you need to combine ranges if two of them are present for any extend value being hit.

So for 1..2 the extends being looked for are 0 and 3. For 4..5 the extends being looked for are 3, again, and 6. The extends2ranges data structure for just this should look like:

The Code #2

Its Output

This second algorithm gives correct results but is harder to develop and explain. It's a testament to my stubbornness as I thought there was a solution there, and debugging was me flexing my skills to keep them honed.

END.

November 18, 2024 10:11 AM UTC

Python Software Foundation

Help power Python and join in the PSF year-end fundraiser & membership drive!

The Python Software Foundation (PSF) is the charitable organization behind Python, dedicated to advancing, supporting, and protecting the Python programming language and the community that sustains it. That mission and cause are more than just words we believe in. Our tiny but mighty team works hard to deliver the projects and services that allow Python to be the thriving, independent, community-driven language it is today. Some of what the PSF does includes producing PyCon US, hosting the Python Packaging Index (PyPI), awarding grants to Python initiatives worldwide, maintaining critical community infrastructure, and more.

To build the future of Python and sustain the thriving community that its users deserve, we need your help. By backing the PSF, you’re investing in Python’s growth and health, and your contributions directly impact the language's future. Is your community, work, or hobby powered by Python? Join this year’s drive and power Python’s future with us by donating or becoming a Supporting Member today.

• Keep Python thriving 
• Support CPython and PyPI progress 
• Increase security across the Python ecosystem 
• Bring the global Python community together 
• Make our community more diverse and robust every year

Highlights from 2024:

• A record-making PyCon US - We produced the 21st PyCon US, in Pittsburgh, US, and online, and it was a huge success! For the first time post-2020, PyCon US 2024 sold out with over 2,500 in-person attendees.
• Advances in our Grants Program - 2024 has been a year of change and reflection for the Grants Program, starting with the addition of Marie Nordin to the grants administration team who has supported the PSF in launching several new grants initiatives. We set up Grants Program Office Hours, published a Grants Program Transparency Report for 2022 and 2024, invested in a third-party retrospective, launched a major refresh of all areas of our Grants program and updated our Grants Workgroup Charter. With more changes to come, we are thrilled to share that we awarded a record-breaking amount of grant funds in 2024!
• Empowering the Python community through Fiscal Sponsorship - We are proud to continue supporting our 20 fiscal sponsoree organizations with their initiatives and events all year round. The PSF provides 501(c)(3) tax-exempt status to fiscal sponsorees such as PyLadies and Pallets, and provides back office support so they can focus on their missions. Consider donating to your favorite PSF Fiscal Sponsoree and check out our Fiscal Sponsorees page to learn more about what each of these awesome organizations is all about!
• Connecting directly through Office Hours - The current PSF Board has decided to invest more in connecting and serving the global Python community by establishing a forum to have regular conversations. The board members of the PSF with the support of PSF staff are now holding monthly PSF Board Office Hours on the PSF Discord. The Office Hours are sessions where folks from the community can share with us how we can help your regional community, express perspectives, and provide feedback for the PSF.
• Paying more engineers to work directly on Python, PyPI, and security - We welcomed Petr Viktorin, Deputy Developer in Residence (DiR), and Serhiy Storchaka, Supporting DiR. It’s been exciting to begin to realize the full vision of the DiR program, with special thanks to Bloomberg for making it possible for us to bring Petr on board. The DiR team is taking an active role in shaping the development of the language, and with three people on the team each DiR can now also spend a percentage of their time on feature work aligned with their interests.
• Continuing to enhance Python’s security through Developers-in-Residence - Seth Larson, PSF Security Developer in Residence (DiR) had a busy year thanks to continued support from Alpha-Omega. Seth worked on a variety of projects including the creation of SBOMs for Source and Windows CPython artifacts, implementing build reproducibility for CPython source artifacts, and auditing and migrating Sigstore, to name just a few. Check out Seth's blog to keep up to date with his work. Mike Fiedler, PyPI Safety & Security Engineer, also worked on a variety of projects such as two-factor authentication for all users on PyPI, an audit of PyPI, made significant progress on malware response and reporting, collaborated on the PSF’s submission for the Cybersecurity and Infrastructure Security Agency (CISA)’s Request for Information (RFI), and more! Thanks to AWS and Georgetown for making Mike’s PyPI security accomplishments possible. Stay up to date with Mike's work on the PyPI blog.
• New PSF Staff dedicated to critical infrastructure - We established the PyPI Support Specialist role, filled by Maria Ashna. Over the past 23 years, PyPI has seen essentially exponential growth in traffic and users, relying for the most part on volunteers to support it. The load far outstretched volunteers and prior staff capacity, so we are very excited to have Maria on board. We also filled our Infrastructure Engineer role, welcoming Jacob Coffee to the team, to ensure PSF-maintained systems and services are running smoothly.
  

We appreciate you and we’re so excited to see where we can go together in the year to come!

November 18, 2024 09:56 AM UTC

Python Bytes

#410 Entering the Django core

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong><a href="https://buttondown.com/carlton/archive/thoughts-on-djangos-core/?featured_on=pythonbytes">Thoughts on Django’s Core</a></strong></li> <li><strong><a href="https://pypi.org/project/futurepool/?featured_on=pythonbytes">futurepool</a></strong></li> <li><strong><a href="https://snarky.ca/dont-use-named-tuples-in-new-apis/?featured_on=pythonbytes">Don't return named tuples in new APIs</a></strong></li> <li><strong><a href="https://ziglang.org/news/migrate-to-self-hosting/?featured_on=pythonbytes">Ziglang: Migrating from AWS to Self-Hosting</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=j-q31u9G3Ds' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="410">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/?featured_on=pythonbytes"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://courses.pythontest.com/p/the-complete-pytest-course?featured_on=pythonbytes"><strong>The Complete pytest Course</strong></a></li> <li><a href="https://www.patreon.com/pythonbytes"><strong>Patreon Supporters</strong></a></li> </ul> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a> <strong>/</strong> <a href="https://bsky.app/profile/mkennedy.codes?featured_on=pythonbytes"><strong>@mkennedy.codes</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a> <strong>/</strong> <a href="https://bsky.app/profile/brianokken.bsky.social?featured_on=pythonbytes"><strong>@brianokken.bsky.social</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a> <strong>/</strong> <a href="https://bsky.app/profile/pythonbytes.bsky.social"><strong>@pythonbytes.bsky.social</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Monday</strong> at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it. </p> <p><strong>Brian #1:</strong> <a href="https://buttondown.com/carlton/archive/thoughts-on-djangos-core/?featured_on=pythonbytes">Thoughts on Django’s Core</a></p> <ul> <li>Carlton Gibson</li> <li>Great discussion on <ul> <li>Django and Core vs Plugins</li> <li>Sustainability with limited people</li> <li>Keeping core small</li> <li>The release cycle</li> <li>eembrace plugins vs endorsing plugins.</li> </ul></li> </ul> <p><strong>Michael #2:</strong> <a href="https://pypi.org/project/futurepool/?featured_on=pythonbytes">futurepool</a></p> <ul> <li>via Pat Decker</li> <li>Takes the concept of multiprocessing Pool to the async/await world.</li> <li><p>Create a pool then delegate the work:</p> <pre><code>async with FuturePool(2) as fp: result = await fp.map(async_pool_fn, range(10)) </code></pre></li> <li><p>I would LOVE to see something like this in a broader background asyncio worker pool concept.</p></li> <li>But that concept doesn’t exist in asyncio in Python and that’s a failing of the framework IMO.</li> </ul> <p><strong>Brian #3:</strong> <a href="https://snarky.ca/dont-use-named-tuples-in-new-apis/?featured_on=pythonbytes">Don't return named tuples in new APIs</a></p> <ul> <li>Brett Cannon</li> <li>First off, I’m grateful for any post that talks about APIs and the API is a module, class, or package API and not a Web/REST API. The term API existed long before the internet.</li> <li>“e.g., get_mouse_position() very likely has a two-item tuple of X and Y coordinates of the screen”</li> <li>“it actually makes your API more complex for both you and your users <em>to use</em>. For you, it doubles the data access API surface for your return type as you have to now support index-based and attribute-based data access forever (or until you choose to break your users and change your return type so it doesn't support both approaches)”</li> <li>“… you probably don't want people doing with your return type, like slicing, iterating over all the items …”</li> <li>Alternatives <ul> <li>class</li> <li>dataclass</li> <li>dictionary</li> <li>TypedDict</li> <li>SimpleNamespace</li> </ul></li> <li>“My key point in all of this is to prefer readability and ergonomics over brevity in your code. That means avoiding named tuples except where you are expanding to tweaking an existing API where the named tuple improves over the plain tuple that's already being used.”</li> </ul> <p><strong>Michael #4:</strong> <a href="https://ziglang.org/news/migrate-to-self-hosting/?featured_on=pythonbytes">Ziglang: Migrating from AWS to Self-Hosting</a></p> <ul> <li>The Rust Foundation for example, reports that they spent $404,400 on infrastructure costs in 2023.</li> <li>Zig lang has decided to use a single big cloud machine + mirrors</li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li>Changing the Python Test community <ul> <li>Was started to answer questions for Test &amp; Code listeners years ago. </li> <li>Primarily pytest questions</li> <li>Used to be Slack. Then moved to Podia forum. </li> <li>Now I’m trying to work out a Discord solution that is both sustainable and usable.</li> </ul></li> </ul> <p>Michael:</p> <ul> <li><a href="https://bsky.app/profile/wang.social/post/3lb346uyzdc2r?featured_on=pythonbytes">PWang Bsky essay</a></li> <li><a href="https://theworkitem.com/blog/building-a-business-from-python-expertise-michael-kennedy/?featured_on=pythonbytes">Building A Business From Python Expertise - Michael Kennedy on Work Item Podcast</a></li> <li>Subscribe to package releases, just put .atom on the end of their releases URL, for example: <ul> <li><a href="https://github.com/mikeckennedy/jinja_partials/releases?featured_on=pythonbytes">github.com/mikeckennedy/jinja_partials/releases</a> ← add .atom for RSS</li> </ul></li> <li><a href="https://pypi.org/project/pytest-bdd/8.0.0/#data">pytest-bdd 8.0.0</a> was just released via Jamie Thomson <ul> <li>The big feature (in Jamie’s opinion) is the addition of data tables https://github.com/pytest-dev/pytest-bdd/blob/master/CHANGES.rst#800---2024-11-14</li> </ul></li> </ul> <p><strong>Joke:</strong> <a href="https://devhumor.com/media/breaking-javascript-developer-commits-to-framework-for-record-breaking-3-weeks?featured_on=pythonbytes">Breaking: JavaScript Developer Commits to Framework for Record-Breaking 3 Weeks</a></p>

November 18, 2024 08:00 AM UTC

James Bennett

Introducing DjangoVer

Version numbering is hard, and there are lots of popular schemes out there for how to do it. Today I want to talk about a system I’ve settled on for my own Django-related packages, and which I’m calling “DjangoVer”, because it ties the version number of a Django-related package to the latest Django version that package supports.

But one quick note to start with: this is not really “introducing” the idea of DjangoVer, because I know I’ve used the name a few times already in other places. I’m also not the person who invented this, and I don’t know for certain who did — I’ve seen several packages which appear to follow some form of DjangoVer and took inspiration from them in defining my own take on it.

Django’s version scheme: an overview

The basic idea of DjangoVer is that the version number of a Django-related package should tell you which version of Django you can use it with. Which probably doesn’t help much if you don’t know how Django releases are numbered, so let’s start there. In brief:

• Django issues a “feature release” — one which introduces new features — roughly once every eight months. The current feature release series of Django is 5.1.
• Django issues “bugfix releases” — which fix bugs in one or more feature releases — roughly once each month. As I write this, the latest bugfix release for the 5.1 feature release series is 5.1.3 (along with Django 5.0.9 for the 5.0 feature release series, and Django 4.2.16 for the 4.2 feature release series).
• The version number scheme is MAJOR.FEATURE.BUGFIX, where MAJOR, FEATURE, and BUGFIX are integers.
• The FEATURE component starts at 0, then increments to 1, then to 2, then MAJOR is incremented and FEATURE goes back to 0. BUGFIX starts at 0 with each new feature release, and increments for the bugfix releases for that feature release.
• Every feature release whose FEATURE component is 2 is a long-term support (“LTS”) release.

This has been in effect since Django 2.0 was released, and the feature releases have been: 2.0, 2.1, 2.2 (LTS); 3.0, 3.1, 3.2 (LTS); 4.0, 4.1, 4.2 (LTS); 5.0, 5.1. Django 5.2 (LTS) is expected in April 2025, and then eight months later (if nothing is changed) will come Django 6.0.

I’ll talk more about SemVer in a bit, but it’s worth being crystal clear that Django does not follow Semantic Versioning, and the MAJOR number is not a signal about API compatibility. Instead, API compatibility runs LTS-to-LTS, with a simple principle: if your code runs on a Django LTS release and raises no deprecation warnings, it will run unmodified on the next LTS release. So, for example, if you have an application that runs without deprecation warnings on Django 4.2 LTS, it will run unmodified on Django 5.2 LTS (though at that point it might begin raising new deprecation warnings, and you’d need to clear them before it would be safe to upgrade any further).

DjangoVer, defined

In DjangoVer, a Django-related package has a version number of the form DJANGO_MAJOR.DJANGO_FEATURE.PACKAGE_VERSION, where DJANGO_MAJOR and DJANGO_FEATURE indicate the most recent feature release series of Django supported by the package, and PACKAGE_VERSION begins at zero and increments by one with each release of the package supporting that feature release of Django.

Since the version number only indicates the newest Django feature release supported, a package using DjangoVer should also use Python package classifiers to indicate the full range of its Django support (such as Framework :: Django :: 5.1 to indicate support for Django 5.1 — see examples on PyPI).

But while Django takes care to maintain compatibility from one LTS to the next, I do not think DjangoVer packages need to do that; they can use the simpler approach of issuing deprecation warnings for two releases, and then making the breaking change. One of the stated reasons for Django’s LTS-to-LTS compatibility policy is to help third-party packages have an easier time supporting Django releases that people are actually likely to use; otherwise, Django itself generally just follows the “deprecate for two releases, then remove it” pattern. No matter what compatibility policy is chosen, however, it should be documented clearly, since DjangoVer explicitly does not attempt to provide any information about API stability/compatibility in the version number.

That’s a bit wordy, so let’s try an example:

• If you started a new Django-related package today, you’d (hopefully) support the most recent Django feature release, which is 5.1. So the DjangoVer version of your package should be 5.1.0.
• As long as Django 5.1 is the newest Django feature release you support, you’d increment the third digit of the version number. As you add features or fix bugs you’d release 5.1.1, 5.1.2, etc.
• When Django 5.2 comes out next year, you’d (hopefully) add support for it. When you do, you’d set your package’s version number to 5.2.0. This would be followed by 5.2.1, 5.2.2, etc., and then eight months later by 6.0.0 to support Django 6.0.
• If version 5.1.0 of your package supports Django 5.1, 5.0, and 4.2 (the feature releases receiving upstream support from Django at the time of the 5.1 release), it should indicate that by including the Framework :: Django, Framework :: Django :: 4.2, Framework :: Django :: 5.0, and Framework :: Django :: 5.1 classifiers in its package metadata.

Why another version system?

Some of you probably didn’t even read this far before rushing to instantly post the XKCD “Standards” comic as a reply. Thank you in advance for letting the rest of us know we don’t need to bother listening to or engaging with you. For everyone else: here’s why I think in this case adding yet another “standard” is actually a good idea.

The elephant in the room here is Semantic Versioning (“SemVer”). Others have written about some of the problems with SemVer, but I’ll add my own two cents here: “compatibility” is far too complex and nebulous a concept to be usefully encoded in a simple value like a version number. And if you want my really cynical take, the actual point of SemVer in practice is to protect developers of software from users, by providing endless loopholes and ways to say “sure, this change broke your code, but that doesn’t count as a breaking change”. It’ll turn out that the developer had a different interpretation of the documentation than you did, or that the API contract was “underspecified” and now has been “clarified”, or they’ll just throw their hands up, yell “Hyrum’s Law” and say they can’t possibly be expected to preserve that behavior.

A lot of this is rooted in the belief that changes, and especially breaking changes, are inherently bad and shameful, and that if you introduce them you’re a bad developer who should be ashamed. Which is, frankly, bullshit. Useful software almost always evolves and changes over time, and it’s unrealistic to expect it not to. I wrote about this a few years back in the context of the Python 2/3 transition:

Though there is one thing I think gets overlooked a lot: usually, the anti-Python-3 argument is presented as the desire of a particular company, or project, or person, to stand still and buck the trend of the world to be ever-changing.

But really they’re asking for the inverse of that. Rather than being a fixed point in a constantly-changing world, what they really seem to want is to be the only ones still moving in a world that has become static around them. If only the Python team would stop fiddling with the language! If only the maintainers of popular frameworks would stop evolving their APIs! Then we could finally stop worrying about our dependencies and get on with our real work! Of course, it’s logically impossible for each one of those entities to be the sole mover in a static world, but pointing that out doesn’t always go well.

But that’s a rant for another day and another full post all its own. For now it’s enough to just say I don’t believe SemVer can ever deliver on what it promises. So where does that leave us?

Well, if the version number can’t tell you whether it’s safe to upgrade from one version to another, perhaps it can still tell you something useful. And for me, when I’m evaluating a piece of third-party software for possible use, one of the most important things I want to know is: is someone actually maintaining this? There are lots of potential signals to look for, but some version schemes — like CalVer — can encode this into the version number. Want to know if the software’s maintained? With CalVer you can guess a package’s maintenance status, with pretty good accuracy, from a glance at the version number.

Over the course of this year I’ve been transitioning all my personal non-Django packages to CalVer for precisely this reason. Compatibility, again, is something I think can’t possibly be encoded into a version number, but “someone’s keeping an eye on this” can be. Even if I’m not adding features to something, Python itself does a new version every year and I’ll push a new release to explicitly mark compatibility (as I did recently for the release of Python 3.13). That’ll bump the version number and let anyone who takes a quick glance at it know I’m still there and paying attention to the package.

For packages meant to be used with Django, though, the version number can usefully encode another piece of information: not just “is someone maintaining this”, but “can I use this with my Django installation”. And that is what DjangoVer is about: telling you at a glance the maintenance and Django compatibility status of a package.

DjangoVer in practice

All of my own personal Django-related packages are now using DjangoVer, and say so in their documentation. If I start any new Django-related projects they’ll do the same thing.

A quick scroll through PyPI turns up other packages doing something that looks similar; django-cockroachdb and django-snowflake, for example, versioned their Django 5.1 packages as “5.1”, and explicitly say in their READMEs to install a package version corresponding to the Django version you use (they also have a maintainer in common, who I suspect of having been an early inventor of what I’m now calling “DjangoVer”).

If you maintain a Django-related package, I’d encourage you to at least think about adopting some form of DjangoVer, too. I won’t say it’s the best, period, because something better could always come along, but in terms of information that can be usefully encoded into the version number, I think DjangoVer is the best option I’ve seen for Django-related packages.

November 18, 2024 02:04 AM UTC

Armin Ronacher

Playground Wisdom: Threads Beat Async/Await

It's been a few years since I wrote about my challenges with async/await-based systems and how they just seem to not support back pressure well. A few years later, I do not think that this problem has subsided much, but my thinking and understanding have perhaps evolved a bit. I'm now convinced that async/await is, in fact, a bad abstraction for most languages, and we should be aiming for something better instead and that I believe to be thread.

In this post, I'm also going to rehash many arguments from very clever people that came before me. Nothing here is new, I just hope to bring it to a new group of readers. In particular, you should really consider these who highly influential pieces:

• Bob Nystrom's What Color is Your Function post, which makes a very strong case that having two types of functions, which are only compatible in one direction, causes problems.
• Ron Pressler's Please stop polluting our imperative languages with pure concepts which I think is probably the single most important talk on that topic.
• Nathaniel J. Smith's Notes on structured concurrency, or: Go statement considered harmful which does a really good job laying out the motivation for structured concurrency.

def move_mouse():
    while mouse.x < 200:
        mouse.x += 5
        sleep(10)

def move_cat():
    while cat.x < 200:
        cat.x += 10
        sleep(10)

move_mouse()
move_cat()

function moveMouseBlocking() {
  while (mouse.x < 200) {
    mouse.x += 5;
    sleep(10);  // a blocking sleep
  }
}

function moveMouseAsync() {
  return new Promise((resolve) => {
    function iterate() {
      if (mouse.x < 200) {
        mouse.x += 5;
        sleep(10).then(iterate);  // non blocking sleep
      } else {
        resolve();
      }
    }
    iterate();
  });
}

async function moveMouseAsync() {
  while (mouse.x < 200) {
    mouse.x += 5;
    await sleep(10);
  }
}

let neverSettle = new Promise((resolve) => {
  // this function ends, but we never called resolve
});

const semaphore = new Semaphore(10);

async function execute(f) {
  let token = await semaphore.acquire();
  try {
    await f();
  } finally {
    await semaphore.release(token);
  }
}

def decode_object(idx):
    header = indexes[idx]
    object_buf = buffer[header.start:header.start + header.size]
    return brotli.decompress(object_buf)

November 18, 2024 12:00 AM UTC

November 17, 2024

Django Weblog

2025 DSF Board Election Results

The 2025 DSF Board Election has closed, and the following candidates have been elected:

• Abigail Gbadago
• Jeff Triplett
• Paolo Melchiorre
• Tom Carrick

They will all serve two years for their term.

Directors elected for the 2024 DSF Board, Jacob, Sarah, and Thibaud are continuing with one year left to serve on the board.

Therefore, the combined 2025 DSF Board of Directors are:

• Jacob Kaplan-Moss
• Sarah Abderemane
• Thibaud Colas
• Abigail Gbadago*
• Jeff Triplett*
• Paolo Melchiorre*

• Tom Carrick*

• Elected to a two (2) year term

Congratulations to our winners, and a huge thank you to our departing board members Çağıl Uluşahin Sonmez, Chaim Kirby, Kátia Yoshime Nakamura, Katie McLaughlin.

Thank you again to everyone who nominated themselves. Even if you were not successful, you gave our community the chance to make their voices heard in who they wanted to represent them.

November 17, 2024 11:56 PM UTC

Paolo Melchiorre

Thoughts on my election as a DSF board member

My thoughts on my election as a member of the Django Software Foundation (DSF) board of directors.

November 17, 2024 11:00 PM UTC

Real Python

Using the Python zip() Function for Parallel Iteration

>>> dir(__builtins__)
['ArithmeticError', 'AssertionError', 'AttributeError', ..., 'zip']

>>> numbers = [1, 2, 3]
>>> letters = ["a", "b", "c"]
>>> zipped = zip(numbers, letters)
>>> zipped  # Holds an iterator object
<zip object at 0x7fa4831153c8>

>>> type(zipped)
<class 'zip'>

>>> list(zipped)
[(1, 'a'), (2, 'b'), (3, 'c')]

>>> s1 = {2, 3, 1}
>>> s2 = {"b", "a", "c"}
>>> list(zip(s1, s2))
[(1, 'a'), (2, 'c'), (3, 'b')]

>>> zipped = zip()
>>> zipped
<zip object at 0x7f196294a488>

>>> list(zipped)
[]

Read the full article at https://realpython.com/python-zip-function/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 17, 2024 02:00 PM UTC

Test and Code

223: Writing Stuff Down is a Super Power

Taking notes well can help to listen better, remember things, show respect, be more accountable, free up mind space to solve problems.

This episode discusses

• the benefits of writing things down
• preparing for a meeting
• taking notes in meetings
• reviewing notes for action items, todo items, things to follow up on, etc.
• taking notes to allow for better focus
• writing well structured emails
• writing blog posts and books

 Learn pytest

• pytest is the number one test framework for Python.
• Learn the basics super fast with Hello, pytest!
• Then later you can become a pytest expert with The Complete pytest Course
• Both courses are at courses.pythontest.com

November 17, 2024 01:55 AM UTC

November 16, 2024

Real Python

Using the len() Function in Python

>>> help(len)
Help on built-in function len in module builtins:
len(obj, /)
    Return the number of items in a container.

>>> greeting = "Good Day!"
>>> len(greeting)
9

>>> office_days = ["Tuesday", "Thursday", "Friday"]
>>> len(office_days)
3

>>> london_coordinates = (51.50722, -0.1275)
>>> len(london_coordinates)
2

>>> len("")
0
>>> len([])
0
>>> len(())
0

>>> len(range(1, 20, 2))
10

Read the full article at https://realpython.com/len-python-function/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 16, 2024 02:00 PM UTC

November 15, 2024

Real Python

The Real Python Podcast – Episode #228: Maintaining the Foundations of Python & Cautionary Tales

How do you build a sustainable open-source project and community? What lessons can be learned from Python's history and the current mess that the WordPress community is going through? This week on the show, we speak with Paul Everitt from JetBrains about navigating open-source funding and the start of the Python Software Foundation.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 15, 2024 12:00 PM UTC

Talk Python to Me

#485: Secure coding for Python with SheHacksPurple

What do developers need to know about AppSec and building secure software? We have Tonya Janca (AKA SheHacksPurple) on the show to tell us all about it. We talk about what developers should expect from threat modeling events as well as concrete tips for security your apps and services.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/posit'>Posit</a><br> <a href='https://talkpython.fm/bluehost'>Bluehost</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Tanya on X</b>: <a href="https://x.com/shehackspurple?featured_on=talkpython" target="_blank" >@shehackspurple</a><br/> <b>She Hacks Purple website</b>: <a href="https://shehackspurple.ca/?featured_on=talkpython" target="_blank" >shehackspurple.ca</a><br/> <b>White House recommends memory safe languages</b>: <a href="https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/press-release-technical-report/?featured_on=talkpython" target="_blank" >whitehouse.gov</a><br/> <b>Python Developer Survey Results</b>: <a href="https://lp.jetbrains.com/python-developers-survey-2023/?featured_on=talkpython" target="_blank" >jetbrains.com</a><br/> <b>Bandit</b>: <a href="https://github.com/PyCQA/bandit?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>Semgrep Academy</b>: <a href="https://academy.semgrep.dev/?featured_on=talkpython" target="_blank" >academy.semgrep.dev</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=ocnFl_Nt-ic" target="_blank" >youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/485/secure-coding-for-python-with-shehackspurple" target="_blank" >talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

November 15, 2024 08:00 AM UTC

Matt Layman

Heroku To DigitalOcean - Building SaaS #206

In this episode, I began a migration of my JourneyInbox app from Heroku to DigitalOcean. The first step to this move, since I’m going to use Kamal, is to put the app into a Docker image. We got the whole app into the Docker image, then cleaned up local development and the CI system after making changes that broke those configurations.

November 15, 2024 12:00 AM UTC

November 14, 2024

Python Morsels

Inspecting objects in Python

I rely on 4 functions for inspecting Python objects: type, help, dir, and vars.

Table of contents

1. Inspecting an object's structure and data
2. How to see an object's class
3. Looking up documentation with help
4. Getting the methods and attributes for an object
5. Inspecting direct attributes of an object
6. Base classes, module paths, and more
7. Inspect Python objects with type, help, dir, and vars

from argparse import ArgumentParser
from collections import Counter
from pathlib import Path
import re

def count_letters(text):
    return Counter(
        char
        for char in text.casefold()
        if char.isalpha()
    )

def main():
    parser = ArgumentParser()
    parser.add_argument("file", type=Path)
    args = parser.parse_args()
    letter_counts = count_letters(args.file.read_text())
    breakpoint()
    for letter, count in letter_counts.most_common():
        print(count, letter.upper())

if __name__ == "__main__":
    main()

~ $ python3 letter_counter.py frankenstein.txt
> /home/trey/letter_counter.py(18)main()
-> breakpoint()
(Pdb) interact
*pdb interact start*
>>>

>>> letter_counts
Counter({'e': 46043, 't': 30365, 'a': 26743, 'o': 25225, 'i': 24613, 'n': 24367, 's': 21155, 'r': 20818, 'h': 19725, 'd': 16863, 'l': 12739, 'm': 10604, 'u': 10407, 'c': 9243, 'f': 8731, 'y': 7914, 'w': 7638, 'p': 6121, 'g': 5974, 'b': 5026, 'v': 3833, 'k': 1755, 'x': 677, 'j': 504, 'q': 324, 'z': 243})

Read the full article: https://www.pythonmorsels.com/inspecting-python-objects/

November 14, 2024 09:49 PM UTC

Django Weblog

Django’s technical governance challenges, and opportunities

As of October 29th, two of four members of the Django Software Foundation Steering Council have resigned from their role, with their intentions being to trigger an election of the Steering Council earlier than otherwise scheduled, per our established governance processes.

To our departing members, Simon and Adam, thank you for your contributions to Django and its governance ❤️. The framework and our community owes a lot to your dedication, and we’re confident our community will join us in celebrating your past contributions – and look forward to learning about your future endeavors in the Django ecosystem. And thanks to the remaining members, James and Andrew, for their service over the years.

Our governance challenges

Governance in open source is hard, and community-driven open source even more so. We’re proud that Django’s original two Benevolent Dictators For Life (BDFLs) both retired from the role and turned things over to community governance ten years ago now. The BDFL model can provide  excellent technical governance, but also has its flaws. So the  mantle of technical governance then went on to the Core Developers and the Technical Board (renamed to Steering Council) was introduced.

However, time has revealed flaws in the Steering Council’s governance model and operations. The Steering Council was able to provide decision-making – tiebreaking when the developer community couldn’t lead to consensus – but didn’t provide more forward-looking leadership or vision. Disagreements over how – or if – the Steering Council should approach this part of leadership led us to the current situation, with no functioning technical governance as of a few weeks ago. Even before those recent events, those flaws were also a common source of frustration for our contributors, and a source of concern for Django users who (rightly or not) might have expectations of Django’s direction – such as the publication of a “roadmap” for Django development.

The Django Software Foundation Board of Directors is and was aware of those issues, and recently made attempts to have the Steering Council rectify them, in coordination with other established community members. The DSF Board has tried to be hands-off when it comes to technical leadership, but in retrospect we should have been getting involved sooner, or more decisively. The lack of technical leadership is an existential threat to Django – a slow moving one, but a threat nonetheless. It’s our responsibility to address this threat.

Where we’re heading

We now need new Steering Council members. But we also need governance reform. There’s a lot about the Steering Council that is good and might only need minimal changes. However, the overall question of the Steering Council’s remit, and how it approaches technical leadership for the Django community, needs to be resolved.

We’re going to hold early elections of the Steering Council, as soon as we’ve completed the ongoing 2025 DSF Board elections. Those elections will follow existing processes, and we will want a Steering Council who strives  to meet the group’s intended goals:

1. To safeguard big decisions that affect Django projects at a fundamental level.
2. To help shepherd the project’s future direction.

We expect the new Steering Council will take on those known challenges, resolve those questions of technical leadership, and update Django’s technical governance. They will have the full support of the Board of Directors to address this threat to Django’s future. And the Board will also be more decisive in intervening, should similar issues keep arising.

How you can help

We need contributors willing to take on those challenges and help our community come out ahead. It’s a big role, impactful but demanding. And there are strict, often annoying eligibility rules for the Steering Council.

To help you help us, we’ve set up a form: Django 6.x Steering Council elections - Expression of interest.

If you’re interested in stepping up to shepherd Django’s technical direction, fill in our expression of interest form. We’ll let you know whether or not you meet those eligibility rules, take the guesswork out of the way. You get to focus on your motivation for taking on this kind of high-purpose, high-reward governance role.

And once the elections start and we get to candidate registrations, we’ll be able to reuse details submitted here (if you want to) so the process is smoother for everyone.

Django 6.x Steering Council elections - Expression of interest

How everyone can help

Those elections will be crucial for the future of Django, and will be decided thanks to the vote of our Django Software Foundation Individual Members. If you know people who contribute to ​​the DSF’s mission but aren’t Individual Members already -- use our form to nominate them as Individual Members, so they’re eligible to vote. If you’re that person, do nominate yourself. We consider all contributions towards our mission: advancing and promoting Django, protecting the framework’s long-term viability, and advancing the state of the art in web development.

Any questions? Comment on our forum thread, Discussion thread for “Django’s technical governance challenges, and opportunities” blog post, or reach out via email to foundation@djangoproject.com.

November 14, 2024 05:00 PM UTC

PyCharm

Inline AI Prompting, Coding Assistance for the dataclass_transform Decorator (PEP 681), and More in PyCharm 2024.3!

Code smarter, optimize performance, and stay focused on what matters most with the latest updates in PyCharm 2024.3. From enhanced support for AI Assistant and Jupyter notebooks to new features like no-code data filtering, there’s so much to explore. 

Learn about all the updates on our What’s New page, download the latest version from our website, or update your current version through our free Toolbox App.

Key features of PyCharm 2024.3

AI Assistant

Inline AI prompting

Get help with code, generate documentation, or write tests by prompting AI directly in PyCharm’s editor. Just type your request on a new line and hit Enter.

Edits made by AI are marked in purple in the gutter, so changes are easy to spot. Need a fresh suggestion? Press Tab, Ctrl+/ ( ⌘/ on macOS), or manually edit the purple input text yourself. This feature is available for Python, JavaScript, TypeScript, JSON, YAML, and Jupyter notebooks.

For a personalized AI chat experience, you can now also choose from Google Gemini, OpenAI, or your own local models. Moreover, enhanced context management now lets you control what AI Assistant takes into consideration. The brand-new UI auto-includes open files and selected code and comes with options to add or remove files and attach project-wide instructions to guide responses across your codebase.

Ability to convert for loops into list comprehensions

Refactor your code faster with AI Assistant, which can now help you change massive for loops into list comprehensions. This feature works for all for loops, including nested and while loops.

Local multiline AI code completion PyCharm Professional

PyCharm Professional now provides local multiline AI code completion suggestions based on the proprietary JetBrains ML model used for Full Line Code Completion. Note that we don’t use your data to train the model.

Local multiline code completion typically generates 2–4 lines of code in scenarios where it can predict the next sequence of logical steps, such as within loops, when handling conditions, or when completing common code patterns and boilerplate sections.

Coding assistance for the dataclass_transform decorator (PEP 681)

PyCharm now supports intelligent coding assistance for custom data classes created with libraries using the dataclass_transform decorator. Enjoy the same support as for standard data classes, including attribute code completion and type inference for constructor signatures.

Jupyter Notebook PyCharm Professional

Auto-installation for multiple packages 

PyCharm 2024.3 makes it easier to install packages that are imported in your code. A new quick-fix is available for bulk auto-installations, allowing you to download and install several packages in one click.

Ability to open Jupyter table outputs in the Data View window

View Jupyter table outputs in the Data View tool window to access powerful features like heatmaps, formatting, slicing, and AI functions for enhanced dataframe analysis. Just click on the Open in Data View icon to get started. 

No-code data filtering 

Effortlessly filter data in the Data View tool window or within dataframes without writing any code. Just click the Filter icon in the upper-right corner, choose your filter options and see results in the same window. This functionality works with all supported Python frameworks, including pandas, Polars, NumPy, PyTorch, TensorFlow, and Hugging Face Datasets.

Debug port specification PyCharm Professional

PyCharm now allows you to specify a single debugger port for all communications, simplifying debugging in restricted environments like Docker or WSL. After you set the port in the debugger settings, the debugger runs as a server and all communication between it and the IDE flows through the specified port.

Visit our What’s New page or check out the full release notes for more features and additional details about the features mentioned here. Please report any bugs on our issue tracker so we can address them promptly.

Connect with us on X (formerly Twitter) to share your thoughts on PyCharm 2024.3. We look forward to hearing from you!

November 14, 2024 01:42 PM UTC

Real Python

Quiz: Namespaces and Scope in Python

In this quiz, you’ll test your understanding of Python Namespaces and Scope.

You’ll revisit how Python organizes symbolic names and objects in namespaces, when Python creates a new namespace, how namespaces are implemented, and how variable scope determines symbolic name visibility.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 14, 2024 12:00 PM UTC

PyPy

Guest Post: Final Encoding in RPython Interpreters

Introduction

This post started as a quick note summarizing a recent experiment I carried out upon a small RPython interpreter by rewriting it in an uncommon style. It is written for folks who have already written some RPython and want to take a deeper look at interpreter architecture.

Some experiments are about finding solutions to problems. This experiment is about taking a solution which is already well-understood and applying it in the context of RPython to find a new approach. As we will see, there is no real change in functionality or the number of clauses in the interpreter; it's more like a comparison between endo- and exoskeletons, a different arrangement of equivalent bones and plates.

Overview

An RPython interpreter for a programming language generally does three or four things, in order:

1. Read and parse input programs
2. Encode concrete syntax as abstract syntax
3. Optionally, optimize or reduce the abstract syntax
4. Evaluate the abstract syntax: read input data, compute, print output data, etc.

Today we'll look at abstract syntax. Most programming languages admit a concrete parse tree which is readily abstracted to provide an abstract syntax tree (AST). The AST is usually encoded with the initial style of encoding. An initial encoding can be transformed into any other encoding for the same AST, looks like a hierarchy of classes, and is implemented as a static structure on the heap.

In contrast, there is also a final encoding. A final encoding can be transformed into by any other encoding, looks like an interface for the actions of the interpreter, and is implemented as an unwinding structure on the stack. From the RPython perspective, Python builtin modules like os or sys are final encodings for features of the operating system; the underlying implementation is different when translated or untranslated, but the interface used to access those features does not change.

In RPython, an initial encoding is built from a hierarchy of classes. Each class represents a type of tree nodes, corresponding to a parser production in the concrete parse tree. Each class instance therefore represents an individual tree node. The fields of a class, particularly those filled during .__init__(), store pre-computed properties of each node; methods can be used to compute node properties on demand. This seems like an obvious and simple approach; what other approaches could there be? We need an example.

Final Encoding of Brainfuck

We will consider Brainfuck, a simple Turing-complete programming language. An example Brainfuck program might be:

[-]

This program is built from a loop and a decrement, and sets a cell to zero. In an initial encoding which follows the algebraic semantics of Brainfuck, the program could be expressed by applying class constructors to build a structure on the heap:

Loop(Plus(-1))

A final encoding is similar, except that class constructors are replaced by methods, the structure is built on the stack, and we are parameterized over the choice of class:

lambda cls: cls.loop(cls.plus(-1))

In ordinary Python, transforming between these would be trivial, and mostly is a matter of passing around the appropriate class. Indeed, initial and final encodings are equivalent; we'll return to that fact later. However, in RPython, all of the types must line up, and classes must be determined before translation. We'll need to monomorphize our final encodings, using some RPython tricks later on. Before that, let's see what an actual Brainfuck interface looks like, so that we can cover all of the difficulties with final encoding.

Before we embark, please keep in mind that local code doesn't know what cls is. There's no type-safe way to inspect an arbitrary semantic domain. In the initial-encoded version, we can ask isinstance(bf, Loop) to see whether an AST node is a loop, but there simply isn't an equivalent for final-encoded ASTs. So, there is an implicit challenge to think about: how do we evaluate a program in an arbitrary semantic domain? For bonus points, how do we optimize a program without inspecting the types of its AST nodes?

What follows is a dissection of this module at the given revision. Readers may find it satisfying to read the entire interpreter top to bottom first; it is less than 300 lines.

Core Functionality

Final encoding is given as methods on an interface. These five methods correspond precisely to the summands of the algebra of Brainfuck.

class BF(object):
    # Other methods elided
    def plus(self, i): pass
    def right(self, i): pass
    def input(self): pass
    def output(self): pass
    def loop(self, bfs): pass

Note that the .loop() method takes another program as an argument. Initial-encoded ASTs have other initial-encoded ASTs as fields on class instances; final-encoded ASTs have other final-encoded ASTs as parameters to interface methods. RPython infers all of the types, so the reader has to know that i is usually an integer while bfs is a sequence of Brainfuck operations.

We're using a class to implement this functionality. Later, we'll treat it as a mixin, rather than a superclass, to avoid typing problems.

Monoid

In order to optimize input programs, we'll need to represent the underlying monoid of Brainfuck programs. To do this, we add the signature for a monoid:

class BF(object):
    # Other methods elided
    def unit(self): pass
    def join(self, l, r): pass

This is technically a unital magma, since RPython doesn't support algebraic laws, but we will enforce the algebraic laws later on during optimization. We also want to make use of the folklore that free monoids are lists, allowing callers to pass a list of actions which we'll reduce with recursion:

class BF(object):
    # Other methods elided
    def joinList(self, bfs):
        if not bfs: return self.unit()
        elif len(bfs) == 1: return bfs[0]
        elif len(bfs) == 2: return self.join(bfs[0], bfs[1])
        else:
            i = len(bfs) >> 1
            return self.join(self.joinList(bfs[:i]), self.joinList(bfs[i:]))

.joinList() is a little bulky to implement, but Wirth's principle applies: the interpreter is shorter with it than without it.

Idioms

Finally, our interface includes a few high-level idioms, like the zero program shown earlier, which are defined in terms of low-level behaviors. In an initial encoding, these could be defined as module-level functions; here, we define them on the mixin class BF.

class BF(object):
    # Other methods elided
    def zero(self): return self.loop(self.plus(-1))
    def move(self, i): return self.scalemove(i, 1)
    def move2(self, i, j): return self.scalemove2(i, 1, j, 1)
    def scalemove(self, i, s):
        return self.loop(self.joinList([
            self.plus(-1), self.right(i), self.plus(s), self.right(-i)]))
    def scalemove2(self, i, s, j, t):
        return self.loop(self.joinList([
                self.plus(-1), self.right(i), self.plus(s), self.right(j - i),
                self.plus(t), self.right(-j)]))

Interface-oriented Architecture

Applying Interfaces

Now, we hack at RPython's object model until everything translates. First, consider the task of pretty-printing. For Brainfuck, we'll simply regurgitate the input program as a Python string:

class AsStr(object):
    import_from_mixin(BF)
    def unit(self): return ""
    def join(self, l, r): return l + r
    def plus(self, i): return '+' * i if i > 0 else '-' * -i
    def right(self, i): return '>' * i if i > 0 else '<' * -i
    def loop(self, bfs): return '[' + bfs + ']'
    def input(self): return ','
    def output(self): return '.'

Via rlib.objectmodel.import_from_mixin, no stressing with covariance of return types is required. Instead, we shift from a Java-esque view of classes and objects, to an OCaml-ish view of prebuilt classes and constructors. AsStr is monomorphic, and any caller of it will have to create their own covariance somehow. For example, here are the first few lines of the parsing function:

@specialize.argtype(1)
def parse(s, domain):
    ops = [domain.unit()]
    # Parser elided to preserve the reader's attention

By invoking rlib.objectmodel.specialize.argtype, we make copies of the parsing function, up to one per call site, based on our choice of semantic domain. Oleg calls these "symantics" but I prefer "domain" in code. Also, note how the parsing stack starts with the unit of the monoid, which corresponds to the empty input string; the parser will repeatedly use the monoidal join to build up a parsed expression without inspecting it. Here's a small taste of that:

while i < len(s):
    char = s[i]
    if char == '+': ops[-1] = domain.join(ops[-1], domain.plus(1))
    elif char == '-': ops[-1] = domain.join(ops[-1], domain.plus(-1))
    # and so on

The reader may feel justifiably mystified; what breaks if we don't add these magic annotations? Well, the translator will throw UnionError because the low-level types don't match. RPython only wants to make one copy of functions like parse() in its low-level representation, and each copy of parse() will be compiled to monomorphic machine code. In this interpreter, in order to support parsing to an optimized string and also parsing to an evaluator, we need two copies of parse(). It is okay to not fully understand this at first.

Composing Interfaces

Earlier, we noted that an interpreter can optionally optimize input programs after parsing. To support this, we'll precompose a peephole optimizer onto an arbitrary domain. We could also postcompose with a parser instead, but that sounds more difficult. Here are the relevant parts:

def makePeephole(cls):
    domain = cls()
    def stripDomain(bfs): return domain.joinList([t[0] for t in bfs])
    class Peephole(object):
        import_from_mixin(BF)
        def unit(self): return []
        def join(self, l, r): return l + r
        # Actual definition elided... for now...
    return Peephole, stripDomain

Don't worry about the actual optimization yet. What's important here is the pattern of initialization of semantic domains. makePeephole is an SML-style functor on semantic domains: given a final encoding of Brainfuck, it produces another final encoding of Brainfuck which incorporates optimizations. The helper stripDomain is a finalizer which performs the extraction from the optimizer's domain to the underlying cls that was passed in at translation time. For example, let's optimize pretty-printing:

AsStr, finishStr = makePeephole(AsStr)

Now, it only takes one line to parse and print an optimized AST without ever building it on the heap. To be pedantic, fragments of the output string will be heap-allocated, but the AST's node structure will only ever be stack-allocated. Further, to be shallow, the parser is written to prevent malicious input from causing a stack overflow, and this forces it to maintain a heap-allocated RPython list of intermediate operations inside loops.

print finishStr(parse(text, AsStr()))

Performance

But is it fast? Yes. It's faster than the prior version, which was initial-encoded, and also faster than Andrew Brown's classic version (part 1, part 2). Since Brown's interpreter does not perform much optimization, we will focus on how final encoding can outperform initial encoding.

JIT

First, why is it faster than the same interpreter with initial encoding? Well, it still has initial encoding from the JIT's perspective! There is an Op class with a hierarchy of subclasses implementing individual behaviors. A sincere tagless-final student, or those who remember Stop Writing Classes (2012, Pycon US), will recognize that the following classes could be plain functions, and should think of the classes as a concession to RPython's lack of support for lambdas with closures rather than an initial encoding. We aren't ever going to directly typecheck any Op, but the JIT will generate typechecking guards anyway, so we effectively get a fully-promoted AST inlined into each JIT trace. First, some simple behaviors:

class Op(object): _immutable_ = True

class _Input(Op):
    _immutable_ = True
    def runOn(self, tape, position):
        tape[position] = ord(os.read(0, 1)[0])
        return position
Input = _Input()

class _Output(Op):
    _immutable_ = True
    def runOn(self, tape, position):
        os.write(1, chr(tape[position]))
        return position
Output = _Output()

class Add(Op):
    _immutable_ = True
    _immutable_fields_ = "imm",
    def __init__(self, imm): self.imm = imm
    def runOn(self, tape, position):
        tape[position] += self.imm
        return position

The JIT does technically have less information than before; it no longer knows that a sequence of immutable operations is immutable enough to be worth unrolling, but a bit of rlib.jit.unroll_safe fixes that:

class Seq(Op):
    _immutable_ = True
    _immutable_fields_ = "ops[*]",
    def __init__(self, ops): self.ops = ops
    @unroll_safe
    def runOn(self, tape, position):
        for op in self.ops: position = op.runOn(tape, position)
        return position

Finally, the JIT entry point is at the head of each loop, just like with prior interpreters. Since Brainfuck doesn't support mid-loop jumps, there's no penalty for only allowing merge points at the head of the loop.

class Loop(Op):
    _immutable_ = True
    _immutable_fields_ = "op",
    def __init__(self, op): self.op = op
    def runOn(self, tape, position):
        op = self.op
        while tape[position]:
            jitdriver.jit_merge_point(op=op, position=position, tape=tape)
            position = op.runOn(tape, position)
        return position

That's the end of the implicit challenge. There's no secret to it; just evaluate the AST. Here's part of the semantic domain for evaluation, as well as the "functor" to optimize it. In AsOps.join() are the only isinstance() calls in the entire interpreter! This is acceptable because Seq is effectively a type wrapper for an RPython list, so that a list of operations is also an operation; its list is initial-encoded and available for inspection.

class AsOps(object):
    import_from_mixin(BF)
    def unit(self): return Shift(0)
    def join(self, l, r):
        if isinstance(l, Seq) and isinstance(r, Seq):
            return Seq(l.ops + r.ops)
        elif isinstance(l, Seq): return Seq(l.ops + [r])
        elif isinstance(r, Seq): return Seq([l] + r.ops)
        return Seq([l, r])
    # Other methods elided!

AsOps, finishOps = makePeephole(AsOps)

And finally here is the actual top-level code to evaluate the input program. As before, once everything is composed, the actual invocation only takes one line.

tape = bytearray("\x00" * cells)
finishOps(parse(text, AsOps())).runOn(tape, 0)

Peephole Optimization

Our peephole optimizer is an abstract interpreter with one instruction of lookahead/rewrite buffer. It implements the aforementioned algebraic laws of the Brainfuck monoid. It also implements idiom recognition for loops. First, the abstract interpreter. The abstract domain has six elements:

class AbstractDomain(object): pass
meh, aLoop, aZero, theIdentity, anAdd, aRight = [AbstractDomain() for _ in range(6)]

We'll also tag everything with an integer, so that anAdd or aRight can be exact annotations. This is the actual Peephole.join() method:

def join(self, l, r):
    if not l: return r
    rv = l[:]
    bfHead, adHead, immHead = rv.pop()
    for bf, ad, imm in r:
        if ad is theIdentity: continue
        elif adHead is aLoop and ad is aLoop: continue
        elif adHead is theIdentity:
            bfHead, adHead, immHead = bf, ad, imm
        elif adHead is anAdd and ad is aZero:
            bfHead, adHead, immHead = bf, ad, imm
        elif adHead is anAdd and ad is anAdd:
            immHead += imm
            if immHead: bfHead = domain.plus(immHead)
            elif rv: bfHead, adHead, immHead = rv.pop()
            else:
                bfHead = domain.unit()
                adHead = theIdentity
        elif adHead is aRight and ad is aRight:
            immHead += imm
            if immHead: bfHead = domain.right(immHead)
            elif rv: bfHead, adHead, immHead = rv.pop()
            else:
                bfHead = domain.unit()
                adHead = theIdentity
        else:
            rv.append((bfHead, adHead, immHead))
            bfHead, adHead, immHead = bf, ad, imm
    rv.append((bfHead, adHead, immHead))
    return rv

If this were to get much longer, then implementing a DSL would be worth it, but this is a short-enough method to inline. The abstract interpretation is assumed by induction for the left-hand side of the join, save for the final instruction, which is loaded into a rewrite register. Each instruction on the right-hand side is inspected exactly once. The logic for anAdd followed by anAdd is exactly the same as for aRight followed by aRight because they both have underlying Abelian groups given by the integers. The rewrite register is carefully pushed onto and popped off from the left-hand side in order to cancel out theIdentity, which itself is merely a unifier for anAdd or aRight of 0.

Note that we generate a lot of garbage. For example, parsing a string of n '+' characters will cause the peephole optimizer to allocate n instances of the underlying domain.plus() action, from domain.plus(1) up to domain.plus(n). An older initial-encoded version of this interpreter used hash consing to avoid ever building an op more than once, even loops. It appears more efficient to generate lots of immutable garbage than to repeatedly hash inputs and search mutable hash tables, at least for optimizing Brainfuck incrementally during parsing.

Finally, let's look at idiom recognition. RPython lists are initial-coded, so we can dispatch based on the length of the list, and then inspect the abstract domains of each action.

def isConstAdd(bf, i): return bf[1] is anAdd and bf[2] == i

def oppositeShifts(bf1, bf2):
    return bf1[1] is bf2[1] is aRight and bf1[2] == -bf2[2]

def oppositeShifts2(bf1, bf2, bf3):
    return (bf1[1] is bf2[1] is bf3[1] is aRight and
            bf1[2] + bf2[2] + bf3[2] == 0)

def loop(self, bfs):
    if len(bfs) == 1:
        bf, ad, imm = bfs[0]
        if ad is anAdd and imm in (1, -1):
            return [(domain.zero(), aZero, 0)]
    elif len(bfs) == 4:
        if (isConstAdd(bfs[0], -1) and
            bfs[2][1] is anAdd and
            oppositeShifts(bfs[1], bfs[3])):
            return [(domain.scalemove(bfs[1][2], bfs[2][2]), aLoop, 0)]
        if (isConstAdd(bfs[3], -1) and
            bfs[1][1] is anAdd and
            oppositeShifts(bfs[0], bfs[2])):
            return [(domain.scalemove(bfs[0][2], bfs[1][2]), aLoop, 0)]
    elif len(bfs) == 6:
        if (isConstAdd(bfs[0], -1) and
            bfs[2][1] is bfs[4][1] is anAdd and
            oppositeShifts2(bfs[1], bfs[3], bfs[5])):
            return [(domain.scalemove2(bfs[1][2], bfs[2][2],
                                       bfs[1][2] + bfs[3][2],
                                       bfs[4][2]), aLoop, 0)]
        if (isConstAdd(bfs[5], -1) and
            bfs[1][1] is bfs[3][1] is anAdd and
            oppositeShifts2(bfs[0], bfs[2], bfs[4])):
            return [(domain.scalemove2(bfs[0][2], bfs[1][2],
                                       bfs[0][2] + bfs[2][2],
                                       bfs[3][2]), aLoop, 0)]
    return [(domain.loop(stripDomain(bfs)), aLoop, 0)]

This ends the bonus question. How do we optimize an unknown semantic domain? We must maintain an abstract context which describes elements of the domain. In initial encoding, we ask an AST about itself. In final encoding, we already know everything relevant about the AST.

The careful reader will see that I didn't really answer that opening question in the JIT section. Because the JIT still ranges over the same operations as before, it can't really be slower; but why is it now faster? Because the optimizer is now slightly better in a few edge cases. It performs the same optimizations as before, but the rigor of abstract interpretation causes it to emit slightly better operations to the JIT backend.

Concretely, improving the optimizer can shorten pretty-printed programs. The Busy Beaver Gauge measures the length of programs which search for solutions to mathematical problems. After implementing and debugging the final-encoded interpreter, I found that two of my entries on the Busy Beaver Gauge for Brainfuck had become shorter by about 2%. (Most other entries are already hand-optimized according to the standard algebra and have no optimization opportunities.)

Discussion

Given that initial and final encodings are equivalent, and noting that RPython's toolchain is written to prefer initial encodings, what did we actually gain? Did we gain anything?

One obvious downside to final encoding in RPython is interpreter size. The example interpreter shown here is a rewrite of an initial-encoded interpreter which can be seen here for comparison. Final encoding adds about 20% more code in this case.

Final encoding is not necessarily more code than initial encoding, though. All AST encodings in interpreters are subject to the Expression Problem, which states that there is generally a quadratic amount of code required to implement multiple behaviors for an AST with multiple types of nodes; specifically, n behaviors for m types of nodes require n × m methods. Initial encodings improve the cost of adding new types of nodes; final encodings improve the cost of adding new behaviors. Final encoding may tend to win in large codebases for mature languages, where the language does not change often but new behaviors are added frequently and maintained for long periods.

Optimizations in final encoding require a bit of planning. The abstract-interpretation approach is solid but relies upon the monoid and its algebraic laws. In the worst case, an entire class hierarchy could be required to encode the abstraction.

It is remarkable to find a 2% improvement in residual program size merely by reimplementing an optimizer as an abstract interpreter respecting the algebraic laws. This could be the most important lesson for compiler engineers, if it happens to generalize.

Final encoding was popularized via the tagless-final movement in OCaml and Scala, including famously in a series of tutorials by Kiselyov et al. A "tag", in this jargon, is a runtime identifier for an object's type or class; a tagless encoding effectively doesn't allow isinstance() at all. In the above presentation, tags could be hacked in, but were not materially relevant to most steps. Tags were required for the final evaluation step, though, and the tagless-final insight is that certain type systems can express type-safe evaluation without those tags. We won't go further in this direction because tags also communicate valuable information to the JIT.

Summarizing Table

Credits

Thanks to folks in #pypy on Libera Chat: arigato for the idea, larstiq for pushing me to write it up, and cfbolz and mattip for reviewing and finding mistakes. The original IRC discussion leading to this blog post is available here.

This interpreter is part of the rpypkgs suite, a Nix flake for RPython interpreters. Readers with Nix installed can run this interpreter directly from the flake:

$ nix-prefetch-url https://github.com/MG-K/pypy-tutorial-ko/raw/refs/heads/master/mandel.b
$ nix run github:rpypkgs/rpypkgs#bf -- /nix/store/ngnphbap9ncvz41d0fkvdh61n7j2bg21-mandel.b

November 14, 2024 08:42 AM UTC

Python Bytes

#409 We've moved to Hetzner write-up

<strong>Topics covered in this episode:</strong><br> <ul> <li><a href="https://github.com/willmcgugan/terminal-tree?featured_on=pythonbytes"><strong>terminal-tree</strong></a></li> <li><strong><a href="https://posting.sh?featured_on=pythonbytes">posting: The API client that lives in your terminal</a></strong></li> <li><strong>Extra, extra, extra</strong></li> <li><strong><a href="https://micro.webology.dev/2024/11/03/uv-does-everything.html?featured_on=pythonbytes">UV does everything or enough that I'm not sure what else it needs to do</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=vg6VLG0jKek' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="409">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by:</p> <ul> <li><a href="https://pythonbytes.fm/scout"><strong>ScoutAPM</strong></a> - Django Application Performance Monitoring</li> <li><a href="https://pythonbytes.fm/codeium"><strong>Codeium</strong></a> - Free AI Code Completion &amp; Chat </li> </ul> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Monday</strong> at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</p> <p><strong>Michael #1:</strong> <a href="https://github.com/willmcgugan/terminal-tree?featured_on=pythonbytes"><strong>terminal-tree</strong></a></p> <ul> <li>An experimental filesystem navigator for the terminal, built with <a href="https://github.com/textualize/textual?featured_on=pythonbytes">Textual</a></li> <li>Tested in macOS only at this point. Chances are very high it works on Linux. Slightly lower chance (but non-zero) that it works on Windows. <ul> <li>Can confirm it works on Linux</li> </ul></li> </ul> <p><strong>Brian #2:</strong> <a href="https://posting.sh?featured_on=pythonbytes">posting: The API client that lives in your terminal</a></p> <ul> <li>Also uses Textual</li> <li>From Darren Burns</li> <li>Interesting that the installation instructions recommends using uv: <ul> <li>uv tool install --python 3.12 posting</li> </ul></li> <li>Very cool. Great docs. Beautiful. keyboard centric, but also usable with a mouse.</li> <li>“Fly through your API workflow with an approachable yet powerful <strong>keyboard-centric</strong> interface. Run it locally or <strong>over SSH</strong> on remote machines and containers. Save your requests in a readable and <strong>version-control friendly</strong> format.”</li> <li>Able to save multiple environments</li> <li>Great colors</li> <li>Allows scripting to run Python code before and after requests to prepare headers, set variables, etc.</li> </ul> <p><strong>Michael #3:</strong> <strong>Extra, extra, extra</strong></p> <ul> <li><a href="https://training.talkpython.fm/courses/getting-started-with-spacy?featured_on=pythonbytes">spaCy course</a> swag give-away, <a href="https://forms.gle/MJPWh3VCB58Peegj7?featured_on=pythonbytes">enter for free</a></li> <li>New essay: <a href="https://mkennedy.codes/posts/opposite-of-cloud-native-is-stack-native/?featured_on=pythonbytes">Opposite of Cloud Native is?</a></li> <li>News: <a href="https://talkpython.fm/blog/posts/we-have-moved-to-hetzner/?featured_on=pythonbytes">We've moved to Hetzner</a></li> <li>New package: <a href="https://mkennedy.codes/posts/introducing-the-chameleon-flask-package/?featured_on=pythonbytes">Introducing chameleon-flask package</a></li> <li>New release: <a href="https://github.com/mikeckennedy/listmonk?featured_on=pythonbytes">Listmonk Python client</a></li> <li><a href="https://www.tiobe.com/tiobe-index/?featured_on=pythonbytes">TIOBE Update</a></li> <li><a href="https://peps.python.org/pep-0750/?featured_on=pythonbytes">PEP 750 – Template Strings</a></li> <li><a href="https://canarymail.io?featured_on=pythonbytes">Canary email</a></li> <li>Left Omnivore, for Pocket, left Pocket for, …, landed on <a href="https://www.instapaper.com?featured_on=pythonbytes">Instapaper</a> <ul> <li>Supports direct import from Omnivore and Pocket</li> <li>Though <a href="https://hoarder.app/?featured_on=pythonbytes">Hoarder</a> is compelling</li> </ul></li> <li>Trying out <a href="https://zen-browser.app/?featured_on=pythonbytes">Zen Browser</a> <ul> <li>Wasn’t a fan of Arc (<a href="https://www.yahoo.com/tech/arc-browser-creator-moving-project-151945233.html?featured_on=pythonbytes">especially</a><a href="https://www.yahoo.com/tech/arc-browser-creator-moving-project-151945233.html?featured_on=pythonbytes"> now)</a> but the news turned me on to Zen</li> </ul></li> </ul> <p><strong>Brian #4:</strong> <a href="https://micro.webology.dev/2024/11/03/uv-does-everything.html?featured_on=pythonbytes">UV does everything or enough that I'm not sure what else it needs to do</a></p> <ul> <li>Jeff Triplett</li> <li>“UV feels like one of those old infomercials where it solves everything, which is where we have landed in the Python world.”</li> <li>“My favorite feature is that UV can now bootstrap a project to run on a machine that does not previously have Python installed, along with installing any packages your application might require.”</li> <li>Partial list (see Jeff’s post for his complete list) <ul> <li>uv pip install replaces pip install</li> <li>uv venv replaces python -m venv</li> <li>uv run, uv tool run, and uv tool install replaces pipx</li> <li>uv build - Build your Python package for pypi</li> <li>uv publish - Upload your Python package to pypi, replacing twine and flit publish</li> </ul></li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li><a href="https://nedbatchelder.com/blog/202411/coveragepy_originally.html?featured_on=pythonbytes">Coverage.py originally </a>was just one file</li> <li>Trying out BlueSky <a href="https://bsky.app/profile/brianokken.bsky.social?featured_on=pythonbytes">brianokken.bsky.social</a> <ul> <li>Not because of Taylor Swift, but nice. </li> <li>There are a lot of Python people there.</li> </ul></li> </ul> <p><strong>Joke:</strong> <a href="https://devhumor.com/media/how-programmers-sleep?featured_on=pythonbytes">How programmers sleep</a></p>

November 14, 2024 08:00 AM UTC

Stefan Scherfke

Publishing to PyPI with a Trusted Publisher from GitLab CI/CD

PyPA’s Trusted Publishers let you upload Python packages directly from your CI pipeline to PyPI. And you don’t need any long-lived secrets like API tokens. This makes uploading Python packages not only easier than ever and more secure, too.

In this article, we’ll look at what Trusted Publishers are and how they’re more secure than using API tokens or a username/password combo. We’ll also learn how to set up our GitLab CI/CD pipeline to:

• continuously test the release processes with the TestPyPI on every push to main,
• automatically perform PyPI releases on every Git tag, and
• additionally secure the process with GitLab (deployment) environments.

The official documentation explains most of this, but it doesn’t go into much depth regarding GitLab pipelines and leaves a few details unexplained.

Why should I want to use this?

API tokens aren’t inherently insecure, but they do have a few drawbacks:

• If they are passed as environment variables, there’s a chance they’ll leak (think of a debug env | sort command in your pipeline).
• If you don’t watch out, bad co-maintainers can steal the token and do mischief with it.
• You have to manually renew the token from time to time, which can be annoying in the long run.

Trusted Publishers can avoid these problems or, at the very least, reduce their risk:

• You don’t have to manually renew any long-lived tokens.
• All tokens are short-lived. Even if they leak, they can’t be misused for long.

After we’ve learned how Trusted Publishers and protected GitLab environments work, we will take another look at security considerations.

How do Trusted Publishers work?

The basic idea of Trusted Publishers is quite simple:

• In PyPI’s project settings, you add a Trusted Publisher and configure it with the GitLab URL of your project.
• PyPI will then only accept package uploads if the uploader can prove that the upload comes from a CI pipeline of that project.

The technical process behind this is based on the OpenID Connect (OIDC) standard.

Essentially, the process works like this:

• In your CI pipeline, you request an ID token for PyPI.
• GitLab injects the short-lived token into your pipeline as a (masked) environment variable. It is cryptographically signed by GitLab and contains, among other things, your project’s path with namespace.
• You use this token to authenticate with PyPI and request another token for the actual package upload.
• This API token can now be used just like “normal” project-scoped API tokens.

The Trusted Publishers documentation explains this in more detail.

One problem remains, though: An ID token can be requested in any pipeline job and in any branch. Malicious contributors could sneak in a pipeline job and make a corrupted release.

This is where environments come in.

Environments

GitLab environments represent your deployed code in your infrastructure. Think of your code running in a container in your production or testing Kubernetes cluster; or your Python package living on PyPI. :-)

The most important feature of environments in this context is access control: You can protect environments, restricting deployments to them. For protected environments, you can define users or roles that are allowed to perform deployments and that must approve deployments. For example, you could restrict deployments (uploads to PyPI) to all maintainers of your project, but only after you yourself have approved each release.

Note

Protected environments are a premium feature.

Non-profit open source projects/organizations can apply for a free ultimate subscription.

It seems that very old projects also have this feature enabled. Otherwise I can’t explain why I have it for Typed Settings but not for my other projects…

To use an environment in your CI/CD pipeline, you need to add it to a job in the .gitlab-ci.yml.

If we also store the name of the environment in the PyPI deployment settings, only uploads from that environment will be allowed, i.e. only uploads that have been authorized by selected people.

Security Considerations

The last two sections have already hinted at this: GitLab environments are only truly secure if you can protect them.

Let’s take a step back and consider what threats we’re trying to protect against, so that we’ll then be able to choose the right approach:

1. Random people doing a merge request for your project.
2. Contributors with the developer role committing directly into your project.
3. Co-maintainers with more permissions then a developer.
4. A Jia Tan which you trust even more than the other maintainers.

What can we do about it?

1. Code in other people’s forks doesn’t have access to your project’s CI variables nor can it request OIDC ID tokens in your project’s name. But you need to carefully review each MR!
2. Contributors with only developer permissions can still request ID tokens. If you cannot use protected environments, using an API token stored in a protected CI/CD variable is a more secure approach. You should also protect your main branch and all tags (using the * pattern), so that devleopers only have access to feature branches. You’ll find it under Settings → Repository → Protected branches/tags.
3. Protected CI/CD variables do not protect you from malicious maintainers, though. Even if you only allow yourself to create tags, other maintainers still have access to protected variables. Protected environments with only a selected set of approvers is the most secure approach.
4. If a very trusted co-maintainer becomes malicious, there’s very little you can do. Carefully review all commits and read the audit logs (Secure → Audit Events).

So that means for you:

• If you are the only maintainer of a small open source project, just use a Trusted Publisher with (unprotected) environments.
• If you belong to a larger project with multiple maintainers, consider applying for GitLab for Open Source and use a Trusted Publisher with a protected environment.
• If there are multiple contributors and you don’t have access to protected environments, use an API token stored in a protected CI/CD variable and try only grant developer permissions to contributors.

See also

Please also read about the security model and considerations in the PyPa docs.

Putting it all together

Configuring your GitLab project to use a trusted publisher involves three main steps:

1. Update your project’s publishing settings on PyPI and TestPyPI.
2. Update the CI/CD settings for your GitLab project.
3. Update your project’s pyproject.toml and .gitlab-ci.yml.

PyPI Settings

Tell PyPI to trust your GitLab CI pipelines.

1. Log in to PyPI and go to your account’s Publishing settings. Here, you can manage and add trusted publishers for your project.

2. Add a new trusted publisher for GitLab as shown in the screenshot below.

   Enter your project’s namespace (your GitLab username or the name of your organization), the project name, the filename of your CI def (usually .gitlab-ci.yml).

   Use release as the environment name!

3. Repeat the same steps for the TestPyPI, but use release-test as environment name.

GitLab CI/CD Settings

You need to create two environments and protect the one for production releases.

1. Open your project in GitLab, then go to Operate → Environments and click Create an environment to create the production environment:

   • Title: release
   • Description: PyPI releases (or whatever you want)
   • External URL: https://pypi.org/project/{your-project}/ (the URL is displayed in a few places in GitLab and helps you to quickly navigate to your project on PyPI.)

   Click Save.

2. Click New environment (in the top right corner) to create the test environment:

   • Title: release-test
   • Description: TestPyPI releases (or whatever you want)
   • External URL: https://test.pypi.org/project/{your-project}/

   Click Save.

3. If protected environments are available (see the note above), navigate to Settings → CI/CD and open the Protected environments section. Click the Protect an environment button.

   • Select environment: release
   • Allowed to deploy: Choose a role or user, e.g. Maintainers.
   • Approvers: Choose a role or user, e.g. yourself.

Changes in Project Files

In order to be able to upload each commit to the TestPyPI, we need a different version for each build. To achieve this, we can use hatch-vcs, setuptools_scm, or similar.

In the following example, we are going to use hatchling with hatch-vcs as the build backend and uv for everything else.

1. We configure the build backend in our pyproject.toml as follows:

   [build-system]
   requires = ["hatchling", "hatch-vcs"]
   build-backend = "hatchling.build"
   
   [tool.hatch.version]
   source = "vcs"
   raw-options = { local_scheme = "no-local-version" }  # TestPyPI lacks support for this
   
   [project]
   dynamic = ["version"]
   

   Hint

   Versions with a local component cannot be uploaded to to (Test)PyPI, so we must disable this feature.

2. Now lets open our project’s .gitlab-ci.yml which we’ll edit during the next steps.

   Hint

   The snippets in the next steps only show fragments of the .gitlab-ci.yml. I’ll post the complete file at the end of the article.

3. We need at least a build and a deploy stage:

   stages:
     - 'build'
     # - 'test'
     # - ...
     - 'deploy'
   

4. Python build tools usually put their artifacts (binary wheels and source distributions) into dist/. This directory needs to be added to your pipeline artifacts, so that these files are available in later pipeline jobs:

   build:
     stage: 'build'
     script:
       - 'uv build --out-dir=dist'
     artifacts:
       paths:
         - 'dist/'
   

5. For our use-case, we need two release jobs: One that uploads to the TestPyPI on each push (release-test) and one that uploads to the PyPI in tag pipelines (release).

   Since both jobs are nearly the same, we’ll also define an “abstract base job” .release-base which the other two extend.

   Hint

   To improve readability and avoid issues with excaping, we’ll use YAML multiline strings.

   The >- operator joins the following lines without a line break and strips additional whitespace.

   See yaml-multiline.info for details.

   .release-base:
     # Abstract base job for "release" jobs.
     # Extending jobs must define the following variables:
     # - PYPI_OIDC_AUD: Audience for the ID token that GitLab
     #   issues to the pipeline job
     # - PYPI_OIDC_URL: PyPI endpoint for retrieving a publish
     #   token with GitLab’s ID token
     # - UV_PUBLISH_URL: PyPI endpoint for the actual upload
     stage: 'deploy'
     id_tokens:
       PYPI_ID_TOKEN:
         aud: '$PYPI_OIDC_AUD'
     script:
       # Use the GitLab ID token to retrieve an API token from PyPI
       - >-
         resp="$(curl -X POST "${PYPI_OIDC_URL}" -d "{\"token\":\"${PYPI_ID_TOKEN}\"}")"
       # Parse the response and extract the token
       - >-
         publish_token="$(python -c "import json; print(json.load('${resp}')['token'])")"
       # Upload the files from "dist/"
       - 'uv publish --token "$publish_token"'
       # Print the link to PyPI so we can quickly go there to verify the result:
       - 'version="$(uv run --with hatch-vcs hatchling version)"'
       - 'echo -e "\033[34;1mPackage on PyPI:\033[0m ${CI_ENVIRONMENT_URL}${version}/"'
   

6. Now we can add the release-test job. It extends .release-base, defines variables for the base job, and rules for when the job should run:

   release-test:
     extends: '.release-base'
     rules:
       # Only run if it's a pipeline for the default branch or a tag:
       - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_TAG'
     environment:
       name: 'release-test'
       url: 'https://test.pypi.org/project/typed-settings/'
     variables:
       PYPI_OIDC_AUD: 'testpypi'
       PYPI_OIDC_URL: 'https://test.pypi.org/_/oidc/mint-token'
       UV_PUBLISH_URL: 'https://test.pypi.org/legacy/'
   

7. The release job looks very similar, but the variables have different values and the job only runs in tag pipelines.

   release:
     extends: '.release-base'
     rules:
       # Only run in tag pipelines:
       - if: '$CI_COMMIT_TAG'
     environment:
       name: 'release'
       url: 'https://pypi.org/project/typed-settings/'
     variables:
       PYPI_OIDC_AUD: 'pypi'
       PYPI_OIDC_URL: 'https://pypi.org/_/oidc/mint-token'
       UV_PUBLISH_URL: 'https://upload.pypi.org/legacy/'
   

That’s it. You should now be able to automatically create PyPI releases directly from your GitLab CI/CD pipeline. 🎉

If you run into any problems, you can

• check if the settings on PyPI match your GitLab project,
• read the Trusted Publishers docs,
• read the GitLAB CI/CD YAML syntax reference,
• read the docs for GitLab environments and GitLab OIDC authentication.

You can leave comments over at Mastodon or Bluesky.

And, as promised, here is the complete (but still minimal) .gitlab-ci.yml from the snippets above. If you want to see a real-world example, you can take a look at Typed Settings pipeline definition.

# .gitlab-ci.yml

stages:
  - 'build'
  # - 'test'
  # - ...
  - 'deploy'

build:
  stage: 'build'
  script:
    - 'uv build --out-dir=dist'
  artifacts:
    paths:
      - 'dist/'

.release-base:
  # Abstract base job for "release" jobs.
  # Extending jobs must define the following variables:
  # - PYPI_OIDC_AUD: Audience for the ID token that GitLab issues to the pipeline job
  # - PYPI_OIDC_URL: PyPI endpoint for retrieving a publish token with GitLab’s ID token
  # - UV_PUBLISH_URL: PyPI endpoint for the actual upload
  stage: 'deploy'
  id_tokens:
    PYPI_ID_TOKEN:
      aud: '$PYPI_OIDC_AUD'
  script:
    - >-
      resp="$(curl -X POST "${PYPI_OIDC_URL}" -d "{\"token\":\"${PYPI_ID_TOKEN}\"}")"
    - >-
      publish_token="$(python -c "import json; print(json.load('${resp}')['token'])")"
    - 'uv publish --token "$publish_token"'
    - 'version="$(uv run --with hatch-vcs hatchling version)"'
    - 'echo -e "\033[34;1mPackage on PyPI:\033[0m ${CI_ENVIRONMENT_URL}${version}/"'

release-test:
  extends: '.release-base'
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_TAG'
  environment:
    name: 'release-test'
    url: 'https://test.pypi.org/project/typed-settings/'
  variables:
    PYPI_OIDC_AUD: 'testpypi'
    PYPI_OIDC_URL: 'https://test.pypi.org/_/oidc/mint-token'
    UV_PUBLISH_URL: 'https://test.pypi.org/legacy/'

release:
  extends: '.release-base'
  rules:
    - if: '$CI_COMMIT_TAG'
  environment:
    name: 'release'
    url: 'https://pypi.org/project/typed-settings/'
  variables:
    PYPI_OIDC_AUD: 'pypi'
    PYPI_OIDC_URL: 'https://pypi.org/_/oidc/mint-token'
    UV_PUBLISH_URL: 'https://upload.pypi.org/legacy/'

November 14, 2024 06:18 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Mathspp Pydon'ts"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Ponne / Coding The Past
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Düvel
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Ledermann
  • Christian Scholz
  • Christoph Schiessl
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • CodersLegacy
  • Coding Diet
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • DataWars.io
  • Dave Beazley
  • David Amos
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Declassed Art
  • Denis Kurov
  • Django Weblog
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • HoloViz
  • Hugo van Kemenade
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Inspired Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Velichko
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lintel Technologies
  • Linux Stans
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Mariatta
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Nsukami Patrick
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Kennedy
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyPodcats
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Docs Editorial Board
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python GUIs
  • Python Insider
  • Python Morsels
  • Python People
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Reuven Lerner
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rishi Maker
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Seth Michael Larson
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stefanie Molin
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The Digital Cat
  • The Lunar Cowboy
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Coding Blog
  • The Python Papers
  • The Python Show
  • The Three of Wands
  • Thibauld Nion
  • ThisHosting.Rocks
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • TypeThePipe
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pygame
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • scikit-learn
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github