Hello @Rendiyono Wahyu Saputro
Yes, you can import python libraries and use them in Spark, which supports a full Python API via the pyspark shell. For instance, if you wanted to load and use the python scikit-fuzzy library to run fuzzy logic, then you just:
1) Download python library, either using maven update to local repo, or directly via github, and add the library to your Spark classpath
2) Kick off job with pyspark shell (Example: $ pyspark --jars /path/to/scikit-fuzzy.jar )
3) Import python library in your code (Example: "import skfuzzy as fuzz")
4) Use the library
More information about scikit-fuzzy library here:
https://pypi.python.org/pypi/scikit-fuzzy
Hints about dependencies and install:
Scikit-Fuzzy depends on
- NumPy >= 1.6
- SciPy >= 0.9
- NetworkX >= 1.9
and is available on PyPi! The lastest stable release can always be obtained and installed simply by running
$ pip install -U scikit-fuzzy