{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f792412a",
   "metadata": {},
   "source": [
    "\n",
    "<a id='chap-tpfouilletextellm'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "469ac617",
   "metadata": {},
   "source": [
    "# Travaux pratiques - Fouille de textes 2 : GloVe vs BERT avec SparkNLP\n",
    "\n",
    "Références externes utiles :\n",
    "\n",
    "> - [Documentation SparkNLP BertEmbeddings](https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/embeddings/bert_embeddings/index.html)  \n",
    "- [Modèles BERT disponibles dans SparkNLP](https://sparknlp.org/models?task=Embeddings&language=en&q=bert)  \n",
    "- [Jeu de données IMDb Large Movie Review](https://ai.stanford.edu/~amaas/data/sentiment/)  \n",
    "- [Article BERT original](https://arxiv.org/abs/1810.04805)  \n",
    "\n",
    "\n",
    "\n",
    "**L’objectif** de cette séance est de comparer, sur une tâche d’analyse de sentiments, deux représentations vectorielles de textes dans SparkNLP : les plongements lexicaux GloVe (vus dans le TP précédent) et les plongements contextuels BERT.\n",
    "\n",
    "Nous utilisons le jeu **IMDb Large Movie Review** [[ML11]](#ml11), constitué de 25 000 critiques de films positives et 25 000 négatives. Ce problème est **difficile** : les deux classes partagent le même vocabulaire et le même style d’écriture, seule la tonalité change. Des phénomènes comme l’ironie, la négation longue portée (« ce film n’est pas *sans* défauts ») ou la polysémie de certains adjectifs évaluatifs rendent la tâche sensible à la qualité de la représentation contextuelle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ec74ff0",
   "metadata": {},
   "source": [
    "# Note\n",
    "\n",
    "Pour pouvoir faire le TP sur une machine avec assez peu de ressources (le JupyterHub du Cnam alloue 1 cœur CPU et 4 Go de RAM à chacun.e) nous avons apporté des modifications au TP initial : peu de `cache()`, forte réduction du nombre d’exemples, emploi des encodages les plus courts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e317a1e",
   "metadata": {},
   "source": [
    "## Mise en place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d05b721f",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import os, sys\n",
    "\n",
    "os.system(\"mkdir -p tpFouilleTexteLLM/data\")\n",
    "os.system(\"wget -q -nc https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz -P tpFouilleTexteLLM/data/\")\n",
    "os.system(\"tar -xzf tpFouilleTexteLLM/data/aclImdb_v1.tar.gz -C tpFouilleTexteLLM/data/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c215ed",
   "metadata": {},
   "source": [
    "Adaptation de la version de SparkNLP (>= 7) à celle de Spark (>= 4) :\n",
    "\n",
    "> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e184cea",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install spark-nlp==7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c474b7b",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "os.environ['PYSPARK_PYTHON'] = sys.executable\n",
    "os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable\n",
    "\n",
    "import sparknlp\n",
    "\n",
    "spark = sparknlp.start(\n",
    "        memory=\"2G\",  # défaut 16G -> garbage collector ne s'active pas avant épuisement mémoire\n",
    "        params={\n",
    "                \"spark.master\": \"local[2]\",      # au lieu de local[*]\n",
    "                \"spark.sql.shuffle.partitions\": \"4\",\n",
    "                \"spark.kryoserializer.buffer.max\": \"256m\",\n",
    "        })\n",
    "sc = spark.sparkContext\n",
    "print(sc.getConf().get(\"spark.driver.memory\"), sc.defaultParallelism)\n",
    "\n",
    "print(\"SparkNLP :\", sparknlp.version(), \"| Spark :\", spark.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8532718b",
   "metadata": {},
   "source": [
    "\n",
    "## Chargement et exploration des données\n",
    "\n",
    "Le jeu IMDb est disponible sous la forme d’un répertoire de fichiers texte (un fichier par critique). La structure est `aclImdb/{train,test}/{pos,neg}/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3278dae5",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.functions import lit, col, length\n",
    "from pyspark.sql.types import DoubleType\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "\n",
    "def charger_imdb(repertoire, label):\n",
    "    \"\"\"Charge un répertoire de critiques et ajoute une étiquette.\"\"\"\n",
    "    return spark.read.text(repertoire) \\\n",
    "                .withColumnRenamed(\"value\", \"text\") \\\n",
    "                .withColumn(\"label\", lit(label).cast(DoubleType()))\n",
    "\n",
    "train_pos = charger_imdb(\"tpFouilleTexteLLM/data/aclImdb/train/pos\", 1.0)\n",
    "train_neg = charger_imdb(\"tpFouilleTexteLLM/data/aclImdb/train/neg\", 0.0)\n",
    "test_pos  = charger_imdb(\"tpFouilleTexteLLM/data/aclImdb/test/pos\",  1.0)\n",
    "test_neg  = charger_imdb(\"tpFouilleTexteLLM/data/aclImdb/test/neg\",  0.0)\n",
    "\n",
    "train_data = train_pos.union(train_neg)\n",
    "test_data  = test_pos.union(test_neg)\n",
    "#train_data = train_pos.union(train_neg).cache()\n",
    "#test_data  = test_pos.union(test_neg).cache()\n",
    "\n",
    "print(f\"Etiquettes dans train :\")\n",
    "train_data.groupBy(\"label\").count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13ef20c0",
   "metadata": {},
   "source": [
    "## Question :\n",
    "\n",
    "Quelle est la distribution des classes ? Le jeu est-il équilibré ? Calculez la longueur moyenne des critiques (en caractères). Y a-t-il une différence notable de longueur entre critiques positives et négatives ?\n",
    "\n",
    "Pour diminuer fortement l’espace mémoire occupé nous travaillons sur un échantillon de 200 critiques en entraînement (100 par classe) et évaluons sur 200 critiques de test :\n",
    "\n",
    "..Pour garder des temps de calcul raisonnables nous travaillons sur un échantillon de 4 000 critiques en entraînement (2 000 par classe) et évaluons sur la totalité des 25 000 critiques de test :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d0b4f72",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "fractions = {0.0: 200/12500, 1.0: 200/12500}\n",
    "train_ech = train_data.stat.sampleBy(\"label\", fractions, seed=42).cache()\n",
    "test_ech = test_data.stat.sampleBy(\"label\", fractions, seed=42)\n",
    "print(f\"Échantillon d'entraînement : {train_ech.count()} critiques\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7c7d23c",
   "metadata": {},
   "source": [
    "### Partie 1 : *Baseline* avec GloVe\n",
    "\n",
    "Nous construisons d’abord une *baseline* avec GloVe, en reprenant la méthodologie du TP précédent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a0cea19",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from pyspark.ml import Pipeline\n",
    "from sparknlp.base import DocumentAssembler, EmbeddingsFinisher\n",
    "from sparknlp.annotator import (Tokenizer, StopWordsCleaner,\n",
    "                                 WordEmbeddingsModel, SentenceEmbeddings)\n",
    "\n",
    "# Pipeline GloVe (identique au TP précédent)\n",
    "documentAssembler = DocumentAssembler() \\\n",
    "    .setInputCol(\"text\").setOutputCol(\"document\") \\\n",
    "    .setCleanupMode(\"shrink\")\n",
    "\n",
    "tokenizer = Tokenizer() \\\n",
    "    .setInputCols([\"document\"]).setOutputCol(\"token\")\n",
    "\n",
    "stopWords = StopWordsCleaner.pretrained(\"stopwords_en\", \"en\") \\\n",
    "    .setInputCols([\"token\"]).setOutputCol(\"token_clean\")\n",
    "\n",
    "# Si les traitements suivants prennent trop de temps avec glove_6B_300,\n",
    "#    employer glove_100d (à comparer ensuite à small_bert_L2_128)\n",
    "glove = WordEmbeddingsModel.pretrained(\"glove_100d\", \"en\") \\\n",
    "    .setInputCols([\"document\", \"token_clean\"]) \\\n",
    "    .setOutputCol(\"glove_embeddings\")\n",
    "#glove = WordEmbeddingsModel.pretrained(\"glove_6B_300\", \"xx\") \\\n",
    "#    .setInputCols([\"document\", \"token_clean\"]) \\\n",
    "#    .setOutputCol(\"glove_embeddings\")\n",
    "\n",
    "sentenceEmb = SentenceEmbeddings() \\\n",
    "    .setInputCols([\"document\", \"glove_embeddings\"]) \\\n",
    "    .setOutputCol(\"sentence_embeddings\") \\\n",
    "    .setPoolingStrategy(\"AVERAGE\")\n",
    "\n",
    "finisher = EmbeddingsFinisher() \\\n",
    "    .setInputCols([\"sentence_embeddings\"]) \\\n",
    "    .setOutputCols([\"features\"]) \\\n",
    "    .setOutputAsVector(True) \\\n",
    "    .setCleanAnnotations(False)\n",
    "\n",
    "pipeline_glove = Pipeline(stages=[\n",
    "    documentAssembler, tokenizer, stopWords,\n",
    "    glove, sentenceEmb, finisher\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62f2c20b",
   "metadata": {},
   "source": [
    "Encodage avec la *Pipeline* GloVe :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a11d4444",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "t0 = time.time()\n",
    "modele_glove_pipe = pipeline_glove.fit(train_ech)\n",
    "train_glove = modele_glove_pipe.transform(train_ech) \\\n",
    "                               .select(\"label\", col(\"features\")[0].alias(\"features\")).cache()\n",
    "test_glove  = modele_glove_pipe.transform(test_ech) \\\n",
    "                               .select(\"label\", col(\"features\")[0].alias(\"features\"))\n",
    "#test_glove  = modele_glove_pipe.transform(test_data) \\\n",
    "#                               .select(\"label\", col(\"features\")[0].alias(\"features\")).cache()\n",
    "train_glove.count(); test_glove.count()\n",
    "t_glove_emb = time.time() - t0\n",
    "print(f\"Embeddings GloVe calculés en {t_glove_emb:.1f} s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c4f6e07",
   "metadata": {},
   "source": [
    "### Question :\n",
    "\n",
    "Pourquoi supprime-t-on les *stop words* avant de calculer la moyenne des vecteurs GloVe ? Cette suppression pourrait-elle être **préjudiciable** pour l’analyse de sentiments (pensez aux expressions comme « not bad », « not at all disappointing »)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "179c7479",
   "metadata": {},
   "source": [
    "#### Classification GloVe\n",
    "\n",
    "Ici encore, pour réduire le temps de calcul nous ne gardons qu’une seule valeur de l’hyperparamètre de régularisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b711e02",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import LogisticRegression\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "import numpy as np\n",
    "\n",
    "# Régression logistique (binaire)\n",
    "lr_glove = LogisticRegression(\n",
    "               featuresCol=\"features\", labelCol=\"label\",\n",
    "               maxIter=50, family=\"binomial\")\n",
    "\n",
    "grid_glove = ParamGridBuilder() \\\n",
    "                 .addGrid(lr_glove.regParam, [0.1]) \\\n",
    "                 .build()\n",
    "#grid_glove = ParamGridBuilder() \\\n",
    "#                 .addGrid(lr_glove.regParam, [0.01, 0.1]) \\\n",
    "#                 .build()\n",
    "\n",
    "evaluateur = BinaryClassificationEvaluator(\n",
    "                 labelCol=\"label\", metricName=\"areaUnderROC\")\n",
    "\n",
    "cv_glove = CrossValidator(\n",
    "               estimator=lr_glove,\n",
    "               estimatorParamMaps=grid_glove,\n",
    "               evaluator=evaluateur,\n",
    "               numFolds=5, seed=42)\n",
    "\n",
    "t0 = time.time()\n",
    "modele_glove = cv_glove.fit(train_glove)\n",
    "t_glove_clf = time.time() - t0\n",
    "print(f\"Classification GloVe : {t_glove_clf:.1f} s\")\n",
    "print(f\"Meilleur regParam : \"\n",
    "      f\"{modele_glove.getEstimatorParamMaps()[np.argmax(modele_glove.avgMetrics)][lr_glove.regParam]}\")\n",
    "\n",
    "glove_predictions = modele_glove.transform(test_glove)\n",
    "#glove_predictions = modele_glove.transform(test_glove).cache()\n",
    "auc_glove = evaluateur.evaluate(glove_predictions)\n",
    "print(f\"AUC GloVe sur test : {auc_glove:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "046cfb40",
   "metadata": {},
   "source": [
    "#### Question :\n",
    "\n",
    "Pourquoi utilise-t-on l’AUC (aire sous la courbe ROC) plutôt que *accuracy* pour évaluer ce modèle ? Quelle valeur d’AUC correspondrait à un classifieur aléatoire ? Et à un classifieur parfait ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c5e051",
   "metadata": {},
   "source": [
    "### Partie 2 : *Embeddings* contextuels BERT\n",
    "\n",
    "Nous remplaçons maintenant GloVe par BERT. La différence fondamentale est que BERT produit une représentation **différente** pour chaque token selon son **contexte** dans la phrase."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64f2a561",
   "metadata": {},
   "source": [
    "#### Tokenisation BERT\n",
    "\n",
    "BERT emploie une tokenisation par sous-mots, différente de la tokenisation classique. SparkNLP fournit un annotateur `BertEmbeddings` qui gère cette tokenisation en interne. Si la construction des *embeddings* de textes dure trop longtemps avec `bert_base_cased`, utiliser plutôt `small_bert_L2_128` en limitant le nombre de *tokens* par critique à 128 (en anglais, 128 *tokens* BERT ~ 90-100 mots) avec `.setMaxSentenceLength(128)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b76abb1",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from sparknlp.annotator import BertEmbeddings, SentenceDetector\n",
    "\n",
    "# Modèle BERT léger : small_bert_L2_128 (2 couches, dimension 128)\n",
    "bert = BertEmbeddings.pretrained(\"small_bert_L2_128\", \"en\") \\\n",
    "           .setInputCols([\"document\", \"token\"]) \\\n",
    "           .setOutputCol(\"bert_embeddings\") \\\n",
    "           .setMaxSentenceLength(128)   # tronquer à 128 tokens ~ 90-100 mots\n",
    "\n",
    "# Modèle BERT : bert_base_cased (12 couches, dimension 768)\n",
    "#bert = BertEmbeddings.pretrained(\"bert_base_cased\", \"en\") \\\n",
    "#           .setInputCols([\"document\", \"token\"]) \\\n",
    "#           .setOutputCol(\"bert_embeddings\") \\\n",
    "#           .setMaxSentenceLength(512)   # tronquer à 512 tokens\n",
    "\n",
    "sentenceEmbBert = SentenceEmbeddings() \\\n",
    "    .setInputCols([\"document\", \"bert_embeddings\"]) \\\n",
    "    .setOutputCol(\"sentence_embeddings_bert\") \\\n",
    "    .setPoolingStrategy(\"AVERAGE\")\n",
    "\n",
    "finisherBert = EmbeddingsFinisher() \\\n",
    "    .setInputCols([\"sentence_embeddings_bert\"]) \\\n",
    "    .setOutputCols([\"features\"]) \\\n",
    "    .setOutputAsVector(True) \\\n",
    "    .setCleanAnnotations(False)\n",
    "\n",
    "# Pas de stop words supprimés pour BERT : la séquence entière est importante\n",
    "pipeline_bert = Pipeline(stages=[\n",
    "    documentAssembler, tokenizer,    # réutilisation depuis Partie 1\n",
    "    bert, sentenceEmbBert, finisherBert\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1da60a34",
   "metadata": {},
   "source": [
    "#### Question :\n",
    "\n",
    "Pourquoi ne supprime-t-on **pas** les *stop words* dans le *pipeline* BERT, alors qu’on les supprimait dans le *pipeline* GloVe ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dd4a1d6",
   "metadata": {},
   "source": [
    "#### Question :\n",
    "\n",
    "Le paramètre `setMaxSentenceLength(n)` tronque les critiques à n tokens (sous-mots). Une critique IMDb moyenne fait environ 230 mots. Quel est l’impact attendu de cette troncature sur les performances ? Comment pourrions-nous atténuer cet effet ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b776dd01",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "t0 = time.time()\n",
    "modele_bert_pipe = pipeline_bert.fit(train_ech)\n",
    "train_bert = modele_bert_pipe.transform(train_ech) \\\n",
    "                             .select(\"label\", col(\"features\")[0].alias(\"features\")).cache()\n",
    "test_bert  = modele_bert_pipe.transform(test_ech) \\\n",
    "                             .select(\"label\", col(\"features\")[0].alias(\"features\"))\n",
    "#test_bert  = modele_bert_pipe.transform(test_ech) \\\n",
    "#                             .select(\"label\", col(\"features\")[0].alias(\"features\")).cache()\n",
    "train_bert.count(); test_bert.count()\n",
    "t_bert_emb = time.time() - t0\n",
    "print(f\"Embeddings BERT calculés en {t_bert_emb:.1f} s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d6d3f8",
   "metadata": {},
   "source": [
    "#### Classification BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f8a4345",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "lr_bert = LogisticRegression(\n",
    "              featuresCol=\"features\", labelCol=\"label\",\n",
    "              maxIter=50, family=\"binomial\")\n",
    "\n",
    "grid_bert = ParamGridBuilder() \\\n",
    "                .addGrid(lr_bert.regParam, [0.1]) \\\n",
    "                .build()\n",
    "\n",
    "cv_bert = CrossValidator(\n",
    "              estimator=lr_bert,\n",
    "              estimatorParamMaps=grid_bert,\n",
    "              evaluator=evaluateur,\n",
    "              numFolds=5, seed=42)\n",
    "\n",
    "t0 = time.time()\n",
    "modele_bert = cv_bert.fit(train_bert)\n",
    "t_bert_clf = time.time() - t0\n",
    "print(f\"Classification BERT : {t_bert_clf:.1f} s\")\n",
    "print(f\"Meilleur regParam : \"\n",
    "      f\"{modele_bert.getEstimatorParamMaps()[np.argmax(modele_bert.avgMetrics)][lr_bert.regParam]}\")\n",
    "\n",
    "bert_predictions = modele_bert.transform(test_bert)\n",
    "#bert_predictions = modele_bert.transform(test_bert).cache()\n",
    "auc_bert = evaluateur.evaluate(bert_predictions)\n",
    "print(f\"AUC BERT sur test : {auc_bert:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0587a0fd",
   "metadata": {},
   "source": [
    "### Partie 3 : Comparaison et analyse qualitative"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed038fe2",
   "metadata": {},
   "source": [
    "## Tableau comparatif"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "092073f3",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "print(f\"{'Méthode':<20} {'AUC test':>10} {'Temps emb. (s)':>16} {'Temps clf. (s)':>16}\")\n",
    "print(\"-\" * 80)\n",
    "print(f\"{'GloVe':<20} {auc_glove:>10.4f} {t_glove_emb:>16.1f} {t_glove_clf:>16.1f}\")\n",
    "print(f\"{'BERT':<20} {auc_bert:>10.4f} {t_bert_emb:>16.1f} {t_bert_clf:>16.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e985d129",
   "metadata": {},
   "source": [
    "## Question :\n",
    "\n",
    "Interprétez les différences observées entre GloVe et BERT sur les aspects suivants : qualité de la représentation (à travers l’AUC), coût de calcul (temps de *embedding*), dimension du vecteur. Le gain de BERT justifie-t-il son coût supplémentaire ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e7ac1b4",
   "metadata": {},
   "source": [
    "## Analyse qualitative : exemples différenciateurs\n",
    "\n",
    "Identifions des critiques positives où les deux modèles divergent : cas où BERT réussit (score > 0.5) mais GloVe échoue (score < 0.5), et inversement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd48fa25",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import abs as F_abs\n",
    "from pyspark.ml.functions import vector_to_array\n",
    "\n",
    "# Récupérer les scores de confiance (probabilité de classe positive) pour chaque modèle\n",
    "preds_glove = glove_predictions.select(\"label\", vector_to_array(col(\"probability\"))[1].alias(\"score_glove\"))\n",
    "preds_bert  = bert_predictions.select(\"label\", vector_to_array(col(\"probability\"))[1].alias(\"score_bert\"))\n",
    "\n",
    "# Joindre sur l'indice (les DataFrames ont le même ordre car même split)\n",
    "from pyspark.sql.functions import monotonically_increasing_id\n",
    "preds_glove = preds_glove.withColumn(\"id\", monotonically_increasing_id())\n",
    "preds_bert  = preds_bert.withColumn(\"id\", monotonically_increasing_id())\n",
    "\n",
    "comparaison = preds_glove.drop(\"label\").join(preds_bert, \"id\") \\\n",
    "                         .withColumn(\"delta\", F_abs(\n",
    "                             col(\"score_bert\") - col(\"score_glove\")))\n",
    "\n",
    "# Cas où BERT prédit correctement (p > 0.5) et GloVe se trompe (p < 0.5) sur une critique positive\n",
    "bert_gagne = comparaison.filter(\n",
    "        (col(\"label\") == 1.0) &\n",
    "        (col(\"score_bert\") > 0.5) &\n",
    "        (col(\"score_glove\") < 0.5)\n",
    "    ).orderBy(\"delta\", ascending=False).cache()\n",
    "print(f\"Nombre cas où BERT prédit bien et GloVe se trompe sur une critique positive : {bert_gagne.count()} \")\n",
    "\n",
    "# Récupérer les textes correspondants\n",
    "test_avec_id = test_data.withColumn(\"id\", monotonically_increasing_id())\n",
    "bert_gagne_texte = bert_gagne.join(test_avec_id, \"id\") \\\n",
    "                             .select(\"text\", \"score_glove\", \"score_bert\")\n",
    "bert_gagne_texte.show(truncate=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26ff528c",
   "metadata": {},
   "source": [
    "Le même type d’analyse peut être réalisé pour les critiques négatives."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f97f346",
   "metadata": {},
   "source": [
    "## Question :\n",
    "\n",
    "Examinez les critiques où BERT réussit et GloVe échoue. Identifiez les constructions linguistiques en cause : ironie, négation, comparaison, polysémie. Expliquez pourquoi l’attention de BERT permet de mieux traiter ces constructions par rapport à la moyenne des vecteurs GloVe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90694c28",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "# Cas inverses : GloVe réussit, BERT échoue\n",
    "glove_gagne = comparaison.filter(\n",
    "    (col(\"label\") == 1.0) &\n",
    "    (col(\"score_glove\") > 0.5) &\n",
    "    (col(\"score_bert\") < 0.5)\n",
    ").orderBy(\"delta\", ascending=False).cache()\n",
    "print(f\"Nombre cas où GloVe prédit bien et BERT se trompe sur une critique positive : {glove_gagne.count()} \")\n",
    "\n",
    "glove_gagne_texte = glove_gagne.join(test_avec_id, \"id\") \\\n",
    "                                .select(\"text\", \"score_glove\", \"score_bert\")\n",
    "glove_gagne_texte.show(truncate=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed687ed",
   "metadata": {},
   "source": [
    "## Question :\n",
    "\n",
    "Trouvez-vous des cas où GloVe surpasse BERT ? Si oui, proposez une explication."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d8263f6",
   "metadata": {},
   "source": [
    "## Courbes ROC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8593a99",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "#matplotlib.use('Agg')\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "\n",
    "glove_y_score = glove_predictions.select(vector_to_array(\"probability\")[1]).rdd.keys().collect()\n",
    "glove_y_true = glove_predictions.select(\"label\").rdd.keys().collect()\n",
    "glove_fpr, glove_tpr, glove_thresholds = roc_curve(glove_y_true, glove_y_score)\n",
    "\n",
    "bert_y_score = bert_predictions.select(vector_to_array(\"probability\")[1]).rdd.keys().collect()\n",
    "bert_y_true = bert_predictions.select(\"label\").rdd.keys().collect()\n",
    "bert_fpr, bert_tpr, bert_thresholds = roc_curve(bert_y_true, bert_y_score)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 6))\n",
    "\n",
    "plt.plot([0, 1], [0, 1], \"k--\", label=\"Aléatoire\")\n",
    "plt.plot(glove_fpr, glove_tpr,\n",
    "         label=f\"GloVe (AUC={roc_auc_score(glove_y_true, glove_y_score):.3f})\")\n",
    "plt.plot(bert_fpr, bert_tpr,\n",
    "         label=f\"BERT (AUC={roc_auc_score(bert_y_true, bert_y_score):.3f})\")\n",
    "plt.xlabel(\"Taux de faux positifs (1 - spécificité)\")\n",
    "plt.ylabel(\"Taux de vrais positifs (sensibilité)\")\n",
    "plt.title(\"Courbes ROC pour IMDb Sentiment Analysis\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"tpFouilleTexteLLM/roc_glove_vs_bert.png\", dpi=150)\n",
    "plt.show()\n",
    "print(\"Figure sauvegardée dans tpFouilleTexteLLM/roc_glove_vs_bert.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bfc8069",
   "metadata": {},
   "source": [
    "Pour obtenir les courbes ROC sur les données de **test** nous avons opté ici pour l’utilisation de fonctions issues de Scikit-learn. Le modèle de régression logistique qui a été appris peut fournir les tableaux de FPR et TPR permettant de tracer la courbe ROC (`modele_glove.bestModel.summary.roc.select(\"FPR\", \"TPR\").toPandas()`), mais pour les données d’apprentissage !"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8da1dec9",
   "metadata": {},
   "source": [
    "## Question (bilan) :\n",
    "\n",
    "Résumez les différences fondamentales entre GloVe et BERT en termes de (1) nature de la représentation (statique vs contextuelle), (2) gestion des mots hors vocabulaire, (3) capacité à capturer les relations syntaxiques, (4) coût de calcul. Pour quels types de tâches et de contraintes préféreriez-vous l’un ou l’autre ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c4947ca",
   "metadata": {},
   "source": [
    "## Références\n",
    "\n",
    "<a id='ml11'></a>\n",
    "\\[ML11\\] Maas, A. L., R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. *Learning word vectors for sentiment analysis*. In Proc. 49th Annual Meeting of the Association for Computational Linguistics, pages 142–150, 2011.\n",
    "\n",
    "<a id='dcl18'></a>\n",
    "\\[DCL18\\] Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. *BERT: Pre-training of deep bidirectional transformers for language understanding*. In Proc. NAACL-HLT 2019, pages 4171–4186, 2019.\n",
    "\n",
    "<a id='psm14'></a>\n",
    "\\[PSM14\\] Pennington, J., R. Socher, and C. D. Manning. *GloVe: Global vectors for word representation*. In EMNLP, pages 1532–1543, 2014.\n",
    "\n",
    "<a id='vsp17'></a>\n",
    "\\[VSP17\\] Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. *Attention is all you need*. Advances in Neural Information Processing Systems 30, 2017."
   ]
  }
 ],
 "metadata": {
  "date": 1791976213.5148337,
  "filename": "tpFouilleTexteLLM.rst",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python"
  },
  "title": "Travaux pratiques - Fouille de textes 2 : GloVe vs BERT avec SparkNLP"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}