←back to thread

311 points melodyogonna | 1 comments | | HN request time: 0s | source
Show context
MontyCarloHall ◴[] No.45138920[source]
The reason why Python dominates is that modern ML applications don't exist in a vacuum. They aren't the standalone C/FORTRAN/MATLAB scripts of yore that load in some simple, homogeneous data, crunch some numbers, and spit out a single result. Rather, they are complex applications with functionality extending far beyond the number crunching, which requires a robust preexisting software ecosystem.

For example, a modern ML application might need an ETL pipeline to load and harmonize data of various types (text, images, video, etc., all in different formats) from various sources (local filesystem, cloud storage, HTTP, etc.) The actual computation then must leverage many different high-level functionalities, e.g. signal/image processing, optimization, statistics, etc. All of this computation might be too big for one machine, and so the application must dispatch jobs to a compute cluster or cloud. Finally, the end results might require sophisticated visualization and organization, with a GUI and database.

There is no single language with a rich enough ecosystem that can provide literally all of the aforementioned functionality besides Python. Python's numerical computing libraries (NumPy/PyTorch/JAX etc.) all call out to C/C++/FORTRAN under the hood and are thus extremely high-performance, and for functionality they don't implement, Python's C/C++ FFIs (e.g. Python.h, NumPy C integration, PyTorch/Boost C++ integration) are not perfect, but are good enough that implementing the performance-critical portions of code in C/C++ is much easier compared to re-implementing entire ecosystems of packages in another language like Julia.

replies(8): >>45139364 #>>45140601 #>>45141802 #>>45143317 #>>45144664 #>>45146179 #>>45146608 #>>45146905 #
pjmlp ◴[] No.45146905[source]
I was there when Perl and Tcl were the main actors, that is why VTK used Tcl originally.

Python dominates, because 25 years ago places like CERN started to adopt Python as their main scripting language, and eventually got used for more tasks than empowered shell scripts.

It is like arguing why C dominates and nothing else can ever replace it.

replies(1): >>45147181 #
1. cdavid ◴[] No.45147181[source]
I agree ability to use python to "script HPC" was key factor, but by itself would not have been enough. What really made it dominate is numpy/scipy/matplotlib becoming good enough to replace matlab 20 years ago, and enabled an explosion of tools on top of it: pandas, scikit learn, and the DL stuff ofc.

This is what differentiates python from other "morally equivalent" scripting languages.