Become A Expert in Python Comprehensions

When it comes to expressiveness in the world of computer programming and coding, Python has its significant advantages. Developers can create elegant solutions with little complexity that, if carefully designed, read like a novel.

If I had to pick a single sentence from the Zen of Python defining the significance of Python language, that would be “simple is better than complex”. We should follow the golden rule as much as possible, that is, a code is read more often than it is written, whether by the author, colleagues, or external contributors. As a result, be straightforward, avoid frills, and ensure that your teammates enjoy what you have written today.

This post will expand on this idea with a set of examples of one of the most significant data analysis tasks: data cleaning, with a focus on readability and simplicity.

Loops

Repeating something over and over until a particular condition is met or satisfied is called looping. In python, a for loop is a control flow statement that is used to execute a set of statements repeatedly as long as the condition is met. Iterative statements are another name for this type of statement. Thus, we can say, a for loop is an iterative statement.

The first example will involve adding one to the even elements of a numeric range.

data = range(10)
added = []
for num in data:
	if num %2 == 0:
		added.append(num + 1)
#[1, 3, 5, 7, 9]

Even though this is a reasonably straightforward problem, we had to do several operations:

1. Create an empty list.
2. Iterate over each data point.
3. Examine the element to see if it is even.
4. At each step, make sure to update the inserted item.

It does the job, but we can make it more compact and easier to understand. Our key goal is to communicate our intent simply by looking at the code.

List Comprehension

They excel at expressing transformations simply and directly that developers may apply to lists, sets, dictionaries, and generators by just using various symbols.

We can define and build the collection on the fly by passing functions and conditions.

list_added = [num + 1 for num in data if num%2 ==0]
#[1, 3, 5, 7, 9]
set_added = {num + 1 for num in data if num%2 == 0}
#{1, 3, 5, 7, 9}
dict_added = {str(num): num + 1 for num in data if num%2 == 0}
# {'0': 1, '2': 3, '4': 5, '6': 7, '8': 9}

When reading this code, notice how logic is the major focus. The objective of the code is obvious at first look. The data object can then be wrapped in this logic and converted into other structures.

When it comes to comprehensions, one tiny trick I like to use is pulling an element from a list if it exists.

item = next(
	iter(num + 1 for num in data if num == 4),
	None,
)
#5

The comprehension logic is picked up by this gist and turned into an iterator. Python uses next to jump from one element to the next in an iterator. As a result, we can use our filtering logic to return the element we're looking for and either obtain it or None if it's not present. This method allows us to handle type hints and conditions gracefully later in the code.

Returning to comprehensions. Is this to say that we should abandon loops in favor of higher-order functions? Not in the least. Developers must consider how intricate the reasoning becomes as a result of various approaches. Each tool has its purpose, and with practice, one may figure out which designs are the easiest to maintain.

Map & Filter

Higher-order functions accept functions as arguments and return functions as outpmakeIt’s a mouthful, but it’s critical to take use of Python’s ability to hanmakefunctions like any other object.

Filter adds the logic to skip specific items whereas map helps us run a function on each element of a sequence. We'll write our functions in lambda for convenience, but the identical example will work with named definitions as well.

# First, let's see how we can sum 1 to each number
added = map(lambda x: x + 1, data)
list(added)  # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Now, sum 1 to each even number
# Not the most readable approach...
even_added = map(lambda x: x + 1, list(filter(lambda x: x % 2 == 0, data)))
list(even_added)  # [1, 3, 5, 7, 9]
# Finally, sum 1 to each even number
# But using two steps for readability
even = list(filter(lambda x: x % 2 == 0, data))
even_added = map(lambda x: x + 1, even)
list(even_added)  # [1, 3, 5, 7, 9]

It’s crucial to remember from this snippet that just because we can create a one-liner doesn’t imply we should.

It’s worth noting how much easier it is to comprehend what’s going on in the last few lines, as well as the significance of using meaningful names. Even if one does not understand what filter performs, we are assisting our readers by utilizing the word rather than diving into implementation specifics. Following this logic, I should have done a better job in the lambda expressions and utilized num instead of x.

Ternary Operations

We’ve shown how to filter elements using conditions. Another common method is to apply the requirements to each element individually as a means of transformation. Ternary Operators assist us in achieving this goal.

parsed = ["even" if num % 2 == 0 else "odd" for num in data]
# ['even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd']

We changed a list of numbers to a list of strings depending on the item’s attributes after inspecting each of the items. This example may appear simple, but when combined with filtering and data descriptors like Pedantic, it creates a wonderful recipe.

The most readable and straightforward text is always the winner. And it’s far more elegant than the alternative in this case. It is shorter not because it is shorter, but because it better communicates the code’s goal.

parsed = []
for num in data:
    if num % 2 == 0:
        parsed.append("even")
    else:
        parsed.append("odd")
# ['even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd']

Assignment Expressions

PEP 572 brings us to a close with a terrific contribution. Assignment expressions, which have been available since Python 3.8, allow developers to give an expression result a name (assign a variable), which is very handy when dealing with comprehensions.

Consider the case of a function that returns None for some inputs. After that, we must apply the function to a list and filter out the missing values.

from typing import Optional
def maybe_none(num: int) -> Optional[int]:
    return num if num > 4 else None
naive = [maybe_none(num) for num in data if maybe_none(num) is not None]
assigned = [res for num in data if (res := maybe_none(num)) is not None]
# [5, 6, 7, 8, 9]

The naïve technique would require running the function twice:

1. First to check if the output is None for filtering, and
2. Then again to save the valid return values.

If the function takes a long time to perform or requires an external component, we should avoid using this method because repeating the procedure is not an option.

With assignment expressions (also known as the walrus operator), we gain all of the advantages of comprehension without compromising any function performance. The function is only called once, and the resulting list can be built locally using the res variable.

Conclusion

Python is a very versatile language, with several ways to achieve the same result. As a result, it’s critical to understand the options available to us. Otherwise, anything appears like a nail if you have a hammer.

We’ve looked at a few different ways to filter and convert data in this post, including for loops, map & filter, and comprehensions.

The difficult issue is that there is no such thing as a golden rule. We need to figure out how to communicate our objective in the most effective way possible. It’s not about being clever, but about being concise.

When it comes to expressiveness in the world of computer programming and coding, Python has its significant advantages. Developers can create elegant solutions with little complexity that, if carefully designed, read like a novel.

If I had to pick a single sentence from the Zen of Python defining the significance of Python language, that would be “simple is better than complex”. We should follow the golden rule as much as possible, that is, a code is read more often than it is written, whether by the author, colleagues, or external contributors. As a result, be straightforward, avoid frills, and ensure that your teammates enjoy what you have written today.

This post will expand on this idea with a set of examples of one of the most significant data analysis tasks: data cleaning, with a focus on readability and simplicity.

Loops

Repeating something over and over until a particular condition is met or satisfied is called looping. In python, a for loop is a control flow statement that is used to execute a set of statements repeatedly as long as the condition is met. Iterative statements are another name for this type of statement. Thus, we can say, a for loop is an iterative statement.

The first example will involve adding one to the even elements of a numeric range.

data = range(10)
added = []
for num in data:
	if num %2 == 0:
		added.append(num + 1)
#[1, 3, 5, 7, 9]

Even though this is a reasonably straightforward problem, we had to do several operations:

1. Create an empty list.
2. Iterate over each data point.
3. Examine the element to see if it is even.
4. At each step, make sure to update the inserted item.

It does the job, but we can make it more compact and easier to understand. Our key goal is to communicate our intent simply by looking at the code.

List Comprehension

They excel at expressing transformations simply and directly that developers may apply to lists, sets, dictionaries, and generators by just using various symbols.

We can define and build the collection on the fly by passing functions and conditions.

list_added = [num + 1 for num in data if num%2 ==0]
#[1, 3, 5, 7, 9]
set_added = {num + 1 for num in data if num%2 == 0}
#{1, 3, 5, 7, 9}
dict_added = {str(num): num + 1 for num in data if num%2 == 0}
# {'0': 1, '2': 3, '4': 5, '6': 7, '8': 9}

When reading this code, notice how logic is the major focus. The objective of the code is obvious at first look. The data object can then be wrapped in this logic and converted into other structures.

When it comes to comprehensions, one tiny trick I like to use is pulling an element from a list if it exists.

item = next(
	iter(num + 1 for num in data if num == 4),
	None,
)
#5

The comprehension logic is picked up by this gist and turned into an iterator. Python uses next to jump from one element to the next in an iterator. As a result, we can use our filtering logic to return the element we're looking for and either obtain it or None if it's not present. This method allows us to handle type hints and conditions gracefully later in the code.

Returning to comprehensions. Is this to say that we should abandon loops in favor of higher-order functions? Not in the least. Developers must consider how intricate the reasoning becomes as a result of various approaches. Each tool has its purpose, and with practice, one may figure out which designs are the easiest to maintain.

Map & Filter

Higher-order functions accept functions as arguments and return functions as outpmakeIt’s a mouthful, but it’s critical to make use of Python’s ability to handle functions like any other object.

Filter adds the logic to skip specific items whereas map helps us run a function on each element of a sequence. We'll write our functions in lambda for convenience, but the identical example will work with named definitions as well.

# First, let's see how we can sum 1 to each number
added = map(lambda x: x + 1, data)
list(added)  # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Now, sum 1 to each even number
# Not the most readable approach...
even_added = map(lambda x: x + 1, list(filter(lambda x: x % 2 == 0, data)))
list(even_added)  # [1, 3, 5, 7, 9]
# Finally, sum 1 to each even number
# But using two steps for readability
even = list(filter(lambda x: x % 2 == 0, data))
even_added = map(lambda x: x + 1, even)
list(even_added)  # [1, 3, 5, 7, 9]

It’s crucial to remember from this snippet that just because we can create a one-liner doesn’t imply we should.

It’s worth noting how much easier it is to comprehend what’s going on in the last few lines, as well as the significance of using meaningful names. Even if one does not understand what filter performs, we are assisting our readers by utilizing the word rather than diving into implementation specifics. Following this logic, I should have done a better job in the lambda expressions and utilized num instead of x.

Ternary Operations

We’ve shown how to filter elements using conditions. Another common method is to apply the requirements to each element individually as a means of transformation. Ternary Operators assist us in achieving this goal.

parsed = ["even" if num % 2 == 0 else "odd" for num in data]
# ['even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd']

We changed a list of numbers to a list of strings depending on the item’s attributes after inspecting each of the items. This example may appear simple, but when combined with filtering and data descriptors like Pedantic, it creates a wonderful recipe.

The most readable and straightforward text is always the winner. And it’s far more elegant than the alternative in this case. It is shorter not because it is shorter, but because it better communicates the code’s goal.

parsed = []
for num in data:
    if num % 2 == 0:
        parsed.append("even")
    else:
        parsed.append("odd")
# ['even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd']

Assignment Expressions

PEP 572 brings us to a close with a terrific contribution. Assignment expressions, which have been available since Python 3.8, allow developers to give an expression result a name (assign a variable), which is very handy when dealing with comprehensions.

Consider the case of a function that returns None for some inputs. After that, we must apply the function to a list and filter out the missing values.

from typing import Optional
def maybe_none(num: int) -> Optional[int]:
    return num if num > 4 else None
naive = [maybe_none(num) for num in data if maybe_none(num) is not None]
assigned = [res for num in data if (res := maybe_none(num)) is not None]
# [5, 6, 7, 8, 9]

The naïve technique would require running the function twice:

1. First to check if the output is None for filtering, and
2. Then again to save the valid return values.

If the function takes a long time to perform or requires an external component, we should avoid using this method because repeating the procedure is not an option.

With assignment expressions (also known as the walrus operator), we gain all of the advantages of comprehension without compromising any function performance. The function is only called once, and the resulting list can be built locally using the res variable.

Conclusion

Python is a very versatile language, with several ways to achieve the same result. As a result, it’s critical to understand the options available to us. Otherwise, anything appears like a nail if you have a hammer.

We’ve looked at a few different ways to filter and convert data in this post, including for loops, map & filter, and comprehensions.

The difficult issue is that there is no such thing as a golden rule. We need to figure out how to communicate our objective in the most effective way possible. It’s not about being clever, but about being concise.