粘合作业失败，出现`JohnSnowLabs Spark-NLP依赖项未找到‘随机错误

Glue job failed with `JohnSnowLabs spark-nlp dependency not found` error randomly(粘合作业失败，出现`JohnSnowLabs Spark-NLP依赖项未找到‘随机错误)

问题描述

我正在使用AWS Glue运行一些pyspark python代码，它有时成功，但有时失败，出现依赖错误：Resource Setup Error: Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: JohnSnowLabs#spark-nlp;2.5.4: not found]，错误日志如下：

:: problems summary ::
:::: WARNINGS
        module not found: JohnSnowLabs#spark-nlp;2.5.4

    ==== local-m2-cache: tried

      file:/root/.m2/repository/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom

      -- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:

      file:/root/.m2/repository/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.jar

    ==== local-ivy-cache: tried

      /root/.ivy2/local/JohnSnowLabs/spark-nlp/2.5.4/ivys/ivy.xml

      -- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:

      /root/.ivy2/local/JohnSnowLabs/spark-nlp/2.5.4/jars/spark-nlp.jar

    ==== central: tried

      https://repo1.maven.org/maven2/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom

      -- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:

      https://repo1.maven.org/maven2/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.jar

    ==== spark-packages: tried

      https://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom

      -- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:

      https://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: JohnSnowLabs#spark-nlp;2.5.4: not found

        ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: JohnSnowLabs#spark-nlp;2.5.4: not found]

从成功运行的日志中，我可以看到GLUE能够从https://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom下载依赖项，失败的作业也曾尝试从中下载，但失败了。 这个问题上周似乎自己解决了，但最近几天又出现了，到目前为止还没有自己解决。有没有人见过这个奇怪的问题？谢谢。

推荐答案

Spark-Package于2021年5月1日移动。在我的Scala项目中，我不得不添加一个不同的解析器，如下所示。它必须在Java中类似。

resolvers in ThisBuild ++= Seq(
  "SparkPackages" at "https://repos.spark-packages.org"
 ## remove -> "MVNRepository"  at "https://dl.bintray.com/spark-packages/maven"
)

https://dl.bintray.com/spark-packages/

这篇关于粘合作业失败，出现`JohnSnowLabs Spark-NLP依赖项未找到‘随机错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！