Airflow FileSensor taks 不工作,文件路径出来很奇怪

扎眼的阳光 python 230

原文标题Airflow FileSensor taks is not working,The file path is coming out weird

Airflow FileSensor taks 不工作,一直处于运行状态。我创建了一个文件连接,如下所示。

连接 ID:file_sensor 连接类型:文件(路径)额外:{“路径”:“/opt/airflow/sensor”}

我不知道为什么 AIRFLOW_CTX_DAG_OWNER 显示为 ****。

*** Reading local file: /opt/airflow/logs/sensor_test/sensor_a/2022-03-24T07:52:07.069026+00:00/1.log
[2022-03-24, 16:52:08 KST] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: sensor_test.sensor_a manual__2022-03-24T07:52:07.069026+00:00 [queued]>
[2022-03-24, 16:52:08 KST] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: sensor_test.sensor_a manual__2022-03-24T07:52:07.069026+00:00 [queued]>
[2022-03-24, 16:52:08 KST] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2022-03-24, 16:52:08 KST] {taskinstance.py:1244} INFO - Starting attempt 1 of 1
[2022-03-24, 16:52:08 KST] {taskinstance.py:1245} INFO - 
--------------------------------------------------------------------------------
[2022-03-24, 16:52:08 KST] {taskinstance.py:1264} INFO - Executing <Task(FileSensor): sensor_a> on 2022-03-24 07:52:07.069026+00:00
[2022-03-24, 16:52:08 KST] {standard_task_runner.py:52} INFO - Started process 960 to run task
[2022-03-24, 16:52:08 KST] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'sensor_test', 'sensor_a', 'manual__2022-03-24T07:52:07.069026+00:00', '--job-id', '349', '--raw', '--subdir', 'DAGS_FOLDER/***SensorTest.py', '--cfg-path', '/tmp/tmp5wiingk3', '--error-file', '/tmp/tmplhcg37hy']
[2022-03-24, 16:52:08 KST] {standard_task_runner.py:77} INFO - Job 349: Subtask sensor_a
[2022-03-24, 16:52:08 KST] {logging_mixin.py:109} INFO - Running <TaskInstance: sensor_test.sensor_a manual__2022-03-24T07:52:07.069026+00:00 [running]> on host b05251258af1
[2022-03-24, 16:52:08 KST] {taskinstance.py:1431} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=sensor_test
AIRFLOW_CTX_TASK_ID=sensor_a
AIRFLOW_CTX_EXECUTION_DATE=2022-03-24T07:52:07.069026+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-03-24T07:52:07.069026+00:00
[2022-03-24, 16:52:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:53:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:54:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:55:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:56:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:57:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:58:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:59:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:00:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:01:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:02:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:03:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:04:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:05:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:06:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:07:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:08:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:09:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:10:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:11:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:12:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:13:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:14:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:15:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:16:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:17:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:18:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:19:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:20:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:21:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:22:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:23:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:24:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:25:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt

原文链接:https://stackoverflow.com//questions/71599448/airflow-filesensor-taks-is-not-working-the-file-path-is-coming-out-weird

回复

我来回复
  • Elad的头像
    Elad 评论

    *** 只是按照文档中的说明进行掩蔽。

    如果您有一个密码为 a 的连接,那么您日志中字母 a 的每个实例都将被替换为 ***。要禁用屏蔽,您可以将 hide_sensitive_var_conn_fields 设置为 false。

    处理这个问题的正确方法是不要使用airflow作为密码。你应该使用更复杂的密码——它会给你更多的保护,也会在日志中显示实际的文件路径。

    至于它为什么会戳 – 可能这意味着该文件不存在。

    2年前 0条评论