Run A Python Script Inside A Bash Script

12 Apr 2016

It seems odd to want to do it: wrapping a python script inside a bash and run it. Why bother?

In my job, I encountered a situation which I created a python script that has dependencies on some internal libraries. In particular, I need the libraries to be loaded before python process starts. Of course, one obvious solution is to create a bash script that sets environment variable LD_LIBRARY_PATH and calls the python script within:

export LD_LIBRARY_PATH="/path/to/some/library/"
$@

However, this is a bit cumbersome if you want to run python scripts that need this dependency frequently and only need this dependency when running those scripts, since you need to type one more script name every time (for example, if this wrapping bash script is called runpy.sh, then you need to run $ runpy.sh somepythonscript.py every time).

One interesting hack I learned from one of my colleagues is as follows:

#!/bin/bash

"""echo" "-n"

export LD_LIBRARY_PATH=/path/to/the/shared/library/you/need/
export PYTHONPATH=$PYTHONPATH:/some/imaginary/path

exec `which python` "$0" "$@"
"""

import sys
import os

if __name__ == '__main__':
    print '----- SYS PATHS:  -----'
    for p in sys.path: print p

    print '----- OS ENVIRON:  -----'
    print os.environ['PYTHONPATH']
    print os.environ['LD_LIBRARY_PATH']

Detailed Explanations

The script above is both a bash and python script at the same time! In fact, it is a bash script that will execute itself as a python script!

The first shbang line tells the OS it is a bash script and therefore will be executed in a new bash process spawned by the current shell. This is the standard behavior of how a bash script is executed.

Then, in the new bash process, the environment variables LD_LIBRARY_PATH and PYTHONPATH are set and only set for this process, not its parent process. In other words, this environment is only visible/accessible inside this new bash process, whatever statements or commands run within this process will inherit this environment.

Next, something interesting happened: the new bash process exec itself into a python process! It is still the same process but the process image was replaced by a python interpreter. That means, this python process still has the same environment before exec but now executing the same script as a python script! Note that exec does not return if it succeeds, which means the bash interpreter before exec are not swapped back into the process image and continue execution after python interpreter terminates! This also implies the new process (which becomes a python interpreter process after exec) terminates as soon as the python interpreter finishes execution, and THEN, the control returns back to the parent shell.

While the python interpreter executes the script, it sees the original bash statements understood by bash interpreter are wrapped around in triple quotes, therefore treates them as doctring and executes the statements that follow, which are normal python statements printing some information to the console.

Conclusion

Here is the output on the console:

----- SYS PATHS:  -----
/home/henry/Work/repos/tools
/usr/lib/python2.7/dist-packages
/home/henry/Work/repos/tools
/some/imaginary/path
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/PILcompat
/usr/lib/python2.7/dist-packages/gst-0.10
/usr/lib/python2.7/dist-packages/gtk-2.0
/usr/lib/pymodules/python2.7
----- OS ENVIRON:  -----
:/some/imaginary/path
/path/to/the/shared/library/you/need/at/runtime

As you can see, the environment variables set are indeed visble to the python interpreter as you can spot the set paths in the os.environ and sys.path attributes. Mission complete, interesting and convenient hack!