Optimizing Python Scripts

This topic was published by and viewed 677 times since "". The last page revision was "".

Viewing 1 post (of 1 total)
  • Author
    Posts
  • DevynCJohnson
    DevynCJohnson
    Keymaster
    • Topics - 444
    • @devyncjohnson

    Most (if not all) programmers want their program to be fast. No body enjoys using slow programs that perform tasks slowly. Obviously, programmers would want to know everything they can to speed-up their program. Many strategies and tips are available. Some of these tips may break backwards compatibility with Python2 and other tips may be fine to use. Keep in mind that this article discusses Python3 for the CPython implementation (the default Python interpreter).

    When optimizing Python programs, it helps to be able to measure the performance. IPython is a graphical console for Python that offers various tools for programmers. The one useful to readers of this article is "%timeit". When typing a one-line command, place "%timeit" before the command and separated with a space. For multiple lines of code, place "%%timeit" (yes, two "%") on the first line and then the code below the first line. After pressing enter once or twice, the code will execute, and then, the time needed to run the code is returned. Before timing code, be sure that the code works properly and do not have many applications running. For instance, running some resource intensive applications will make the results look worse than their true values.

    The time needed to execute code will differ with every processor type, operating system, hardware, etc. Thus, the timed results I get and the results you get may be different.

    These three IF-constructs perform the same action - "if type(VAR) == TYPE:", "if isinstance(VAR, TYPE)", and "if VAR.__class__ is TYPE:". However, which one is faster? To test this code, I will use IPython3. First, I will set a variable and then test that my code works. Afterwards, I will time each form of the IF-construct for type-testing. Developers could time these type-tests like I did for the commands #2-5 or #6-8. Notice that I also measured the time it takes to look-up a variable (#9) and execute "type(text)" (#10).

    In [1]: text = 'Sometext'
    
    In [2]: if type(text) == str:
    ...: x = 3 + 7
    ...:
    
    In [3]: %%timeit
    ...: if type(text) == str:
    ...: x = 3 + 7
    ...:
    10000000 loops, best of 3: 132 ns per loop
    
    In [4]: %%timeit
    ...: if isinstance(text, str):
    ...: x = 3 + 7
    ...:
    10000000 loops, best of 3: 123 ns per loop
    
    In [5]: %%timeit
    ...: if text.__class__ is str:
    ...: x = 3 + 7
    ...:
    10000000 loops, best of 3: 81.3 ns per loop
    
    In [6]: %timeit type(text) == str
    10000000 loops, best of 3: 123 ns per loop
    
    In [7]: %timeit isinstance(text, str)
    10000000 loops, best of 3: 117 ns per loop
    
    In [8]: %timeit text.__class__ is str
    10000000 loops, best of 3: 73.2 ns per loop
    
    In [9]: %timeit text
    10000000 loops, best of 3: 24.1 ns per loop
    
    In [10]: %timeit type(text)
    10000000 loops, best of 3: 90 ns per loop

    So, when testing if a single variable is of one type, "if VAR.__class__ is TYPE" is the fastest code.
    To test if an object belongs to one of multiple types, programmers could use "isinstance(VAR, (TYPE1, TYPE2, TYPE3))".

    If a boolean is saved to a variable, then what is the fastest way to use that variable in an IF-construct - "if VAR == True:", "if VAR is True:", "if VAR is not False:", or "if VAR:"?

    In the code below, notice that if the code contains a syntax error (#14), then the code will not be timed. As we can see, "if VAR:" is faster than the other choices. According to the way programs operate, if an IF-construct evaluates to "True", then the code will be executed. For instance, when the code "if 7 > 3:" is executed, the "7 > 3" will be evaluated first. Because the statement is correct, "7 > 3" is replaced with "True". "if" will only allow its enclosed code to execute when it is accompanied with "True". In summary, our "if VAR:" is faster because that is the fully evaluated statement. "if VAR == True:" must first evaluate "VAR == True" which tests if the two objects have the same value. Then, "if BOOL:" is processed.

    In [11]: _bool = True
    
    In [12]: %timeit _bool == True
    10000000 loops, best of 3: 40.8 ns per loop
    
    In [13]: %timeit _bool is True
    10000000 loops, best of 3: 32.4 ns per loop
    
    In [14]: %timeit _bool not False
    File "<unknown>", line 1
    _bool not False
    ^
    SyntaxError: invalid syntax
    
    In [15]: %timeit _bool is not False
    10000000 loops, best of 3: 31.9 ns per loop
    
    In [16]: %timeit _bool
    10000000 loops, best of 3: 22.7 ns per loop

    As for testing if any of the given expressions are true, it is best to use the format "if EXPR or EXPR or EXPR:". Also, keep in mind that Python allows short-circuit logic. This means an IF-construct with multiple expressions will be "True" if any of the expressions are true. So, in an expression like "if X or Y or Z:", X is evaluated first, then Y, and next Z. If X is evaluated as "True", Y and Z will not be evaluated because the IF-construct indicated that any of the values can be "True". With this info, it would be best to place expressions that will likely be true near the beginning of the IF-construct (as seen in #18).

    In [17]: %timeit 7 < 3 or 9 == 0 or 37 >= 1
    10000000 loops, best of 3: 79.1 ns per loop
    
    In [18]: %timeit 37 >= 1 or 7 < 3 or 9 == 0
    10000000 loops, best of 3: 35.5 ns per loop
    
    In [19]: %timeit any((7 < 3, 9 == 0, 37 >= 1))
    10000000 loops, best of 3: 181 ns per loop
    
    In [20]: %timeit any([7 < 3, 9 == 0, 37 >= 1])
    1000000 loops, best of 3: 221 ns per loop

    Below is a very basic example to help better visualize the concept of short-circuit logic.

    #Start

    7 < 3 or 9 == 0 or 37 >= 1

    9 == 0 or 37 >= 1

    37 >= 1

    True

    #End

    #Start

    37 >= 1 or 7 < 3 or 9 == 0

    True

    #End

    This can be proven with the "dis.dis" command (import dis). Notice in the disassembled output below that the constant on each side of the equality test is loaded (LOAD_CONST). Then, they are compared (COMPARE_OP). If the expression is true, then the IF-construct will execute the code; if not, then the next set of expressions are evaluated (JUMP_IF_TRUE_OR_POP).

    In [45]: def test():
    ...: 7 < 3 or 9 == 0 or 37 >= 1
    
    In [46]: dis.dis(test)
    2 0 LOAD_CONST 1 (7)
    3 LOAD_CONST 2 (3)
    6 COMPARE_OP 0 (<)
    9 JUMP_IF_TRUE_OR_POP 33
    12 LOAD_CONST 3 (9)
    15 LOAD_CONST 4 (0)
    18 COMPARE_OP 2 (==)
    21 JUMP_IF_TRUE_OR_POP 33
    24 LOAD_CONST 5 (37)
    27 LOAD_CONST 6 (1)
    30 COMPARE_OP 5 (>=)
    >> 33 POP_TOP
    34 LOAD_CONST 0 (None)
    37 RETURN_VALUE

    When "is" and "==" can be replaced with the other, it may be difficult for programmers to decide which to use when either one would be valid. Therefore, the real question is "Which is faster?".

    In [49]: %timeit text.__class__ == str
    10000000 loops, best of 3: 87.3 ns per loop
    
    In [50]: %timeit text.__class__ is str
    10000000 loops, best of 3: 78.8 ns per loop

    To convert a decimal integer to a number of another base (like octal, in this example), which should be used - "''.join(['%o' % INT])", "'%o' % INT", "oct(INT)", or "'{0:o}'.format(INT)"? All four convert an integer to an octal number as a string. "oct()" returns its output with an "0o" at the beginning of the string, unlike the other three alternatives.

    In [79]: %timeit ''.join(['%o' % 37])
    10000000 loops, best of 3: 133 ns per loop
    
    In [80]: %timeit '%o' % 37
    100000000 loops, best of 3: 14.1 ns per loop
    
    In [81]: %timeit oct(37)
    10000000 loops, best of 3: 82.3 ns per loop
    
    In [82]: %timeit '{0:o}'.format(37)
    1000000 loops, best of 3: 323 ns per loop

    One-line IF-constructs for very simple code -
    When writing a very simple IF-construct that can be written on one-line, which of the below forms is the fastest?

    # x = True; y = False; a&b = expressions

    • (lambda:y, lambda:x)[a > b]()
    • {True: x, False: y}[a > b]
    • (y, x)[a > b]
    • x if a > b else y

    Assume that a = 7 and b = 3, and x will be a string ('This is true') and y is also a string ('This is wrong').

    • (lambda:'This is wrong', lambda:'This is true')[7 > 3]()
    • {True: 'This is true', False: 'This is wrong'}[7 > 3]
    • ('This is wrong', 'This is true')[7 > 3]
    • 'This is true' if 7 > 3 else 'This is wrong'
    In [94]: %timeit (lambda:'This is wrong', lambda:'This is true')[7 > 3]()
    1000000 loops, best of 3: 267 ns per loop
    
    In [95]: %timeit {True: 'This is true', False: 'This is wrong'}[7 > 3]
    1000000 loops, best of 3: 208 ns per loop
    
    In [96]: %timeit ('This is wrong', 'This is true')[7 > 3]
    10000000 loops, best of 3: 51.8 ns per loop
    
    In [97]: %timeit 'This is true' if 7 > 3 else 'This is wrong'
    10000000 loops, best of 3: 39.4 ns per loop
    
    In [108]: %%timeit # added for comparison
    ...: if 7 > 3:
    ...: 'This is true'
    ...: else:
    ...: 'This is wrong'
    ...:
    10000000 loops, best of 3: 32.6 ns per loop

    Therefore, when the value of a variable depends on certain conditions, it is better to assign a value to a variable like below. (only if the IF-construct is written on one line.)

    In [98]: x = 'This is true' if 7 > 3 else 'This is wrong'
    In [99]: x
    Out[99]: 'This is true'

    To execute a command, use a form like "eval(bin(37)) if 7 > 3 else eval(hex(87))", but remember to use "eval()", not "exec()", if you want the value/output returned. Below shows "exec()" being used instead in a different form of a one-line IF-construct to perform a task and not produce an output.

    In [99]: x
    Out[99]: 'This is true'
    
    In [100]: {True: exec('del x')} [7 > 3]
    
    In [101]: x
    ---------------------------------------------------------------------------
    NameError Traceback (most recent call last)
    <ipython-input-101-401b30e3b8b5> in <module>()
    ----> 1 x
    NameError: name 'x' is not defined

    However, just because code has fewer lines does not indicate speed. As we saw in #108, the multi-line IF-construct (4 lines) executed in less time than the IF-constructs that are one-line.

    NOTE: In real code, programmers would want to use the "return" command or save the strings to a variable. For example, in #108, the strings 'This is true' and 'This is wrong' would normally be saved in a variable or come after the return command (if inside a function). In this article, the code is simplified to allow readers to more easily see and understand the main points. However, all this code will still execute without errors.

    When testing if the length ("len()") is zero or not, use "if len(obj):" instead of "if len(obj) != 0:", or anything of that form. A length of zero is "False" and any value larger than zero is "True".

    In [137]: %%timeit
    ...: if len('') == 0:
    ...: 'True'
    ...: else:
    ...: 'False'
    ...:
    10000000 loops, best of 3: 66.1 ns per loop
    
    In [138]: %%timeit
    ...: if len(''):
    ...: 'Do this when length is not zero'
    ...: else:
    ...: 'Length is zero'
    ...:
    10000000 loops, best of 3: 51 ns per loop
    
    In [139]: %%timeit
    ...: if not len(''):
    ...: 'Do this when length is zero'
    ...: else:
    ...: 'Do this when length is not zero'
    ...:
    10000000 loops, best of 3: 52.6 ns per loop
    
    In [140]: if not len(''):
    ...: print('True')
    ...: else:
    ...: print('False')
    ...:
    True

    When iterating, lists are slower than strings.

    In [142]: %timeit 'e' in 'abcdef'
    10000000 loops, best of 3: 36 ns per loop
    
    In [143]: %timeit 'e' in ['a', 'b', 'c', 'd', 'e', 'f']
    10000000 loops, best of 3: 88.5 ns per loop

    The "__debug__" built-in variable (constant) is a boolean. When it equals "True", various debugging code may execute. To make this constant "False" to disable debugging, use the "-O" parameter when invoking Python. If running from a script, append " -O" to the hashpling.

Viewing 1 post (of 1 total)