![]() ![]() I have a simple HTTPS request that works well on my local envoiremnt. * The following is the error I get when trying different options listed above.ĪnalysisException: .ql.metadata.HiveException: : Unable to execute HTTP request: Connect to failed: connect timed out Read from jdbc table via Glue contextĭynamic_frame = glue_context.create_dynamic_frame_from_options(Ĭonnection_type=jdbc_config, # "postgresql" * And then I think I should be able to pass the connection info to either a Glue or Spark function to read from the table. Jdbc_config = glue_context.extract_jdbc_conf(connection_name=glue_connector) ![]() Glue_context = GlueContext(spark_session.sparkContext) Glue_context.spark_(session_name)ĭef get_jdbc_table(spark_session: SparkSession, glue_connector: str, db_table: str, logger) -> DataFrame: Spark_conf = SparkConf().setAll(pairs=list(spark_config_ems())) Glue_context = GlueContext(spark_context) Spark_context = SparkContext.getOrCreate() The spark session is returned from another util function and then is passed into this function, which is why I am getting the glue context from the spark context instead of the other way around.įrom pyspark.sql import DataFrame, SparkSession * Retrieving information about the connection I think should work using **glue_context.extract_jdbc_conf()**. * Testing a Glue connection is successful. ![]()
0 Comments
Leave a Reply. |