Modifying the Apache Spark source code to improve UX around a SparkSession display error.|

Playing with PySpark on a Jupyter Notebook, one can do this:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("read_taxi")
    .master("local[*]")
    .getOrCreate()
)

Evaluating spark to print it, in the jupyter-lab notebook, calls SparkSession._repr_html_(), a method that lives in python/pyspark/sql/session.py of the Apache Spark repository.

class SparkSession(...):
    ...
    def _repr_html_(self) -> str:
        return """
            <div>
                <p><b>SparkSession - {catalogImplementation}</b></p>
                {sc_HTML}
            </div>
        """.format(
            catalogImplementation=self.conf.get("spark.sql.catalogImplementation"),
            sc_HTML=self.sparkContext._repr_html_(),
        )

Which, as seen, calls SparkContext._repr_html_(), that does:

class SparkContext:
    ...
    def _repr_html_(self) -> str:
        return """
        <div>
            <p><b>SparkContext</b></p>

            <p><a href="{sc.uiWebUrl}">Spark UI</a></p>

            <dl>
              <dt>Version</dt>
                <dd><code>v{sc.version}</code></dd>
              <dt>Master</dt>
                <dd><code>{sc.master}</code></dd>
              <dt>AppName</dt>
                <dd><code>{sc.appName}</code></dd>
            </dl>
        </div>
        """.format(
            sc=self
        )

And so the html is shown in the notebook:

Note that it uses the sc variable.

Now, no one ever checked whether sc is available.

When, in the notebook, one does spark.stop(), and then spark again, an ugly error is thrown.

Now, we can improve the UX around this if we add something like this:

    def _repr_html_(self) -> str:
        if self._sc._jsc is None:
            return """
                <div>
                    <p><b>SparkSession - stopped</b></p>
                    <p>SparkContext is no longer active.</p>
                </div>
            """
        return """
            ...
        """.format(
            catalogImplementation=...
            sc_HTML=self.sparkContext._repr_html_(),
        )

And so we get a nicer UX here: