Untitled
unknown
plain_text
a year ago
6.2 kB
5
Indexable
Never
attempts)
vm6.template
*** Reading local file: /var/segmel_ml/ml_ops/airflow/logs/dag_id=realtime_runs/run_id=scheduled__2023-04-24T14:16:00+00:00/task_id=update_fs/attempt=1.log
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1090} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: realtime_runs.update_fs scheduled__2023-04-24T14:16:00+00:00 [queued]>
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1090} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: realtime_runs.update_fs scheduled__2023-04-24T14:16:00+00:00 [queued]>
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1288} INFO -
--------------------------------------------------------------------------------
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1289} INFO - Starting attempt 1 of 4
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1290} INFO -
--------------------------------------------------------------------------------
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1309} INFO - Executing <Task(BashOperator): update_fs> on 2023-04-24 14:16:00+00:00
[2023-04-24, 14:17:00 UTC] {standard_task_runner.py:55} INFO - Started process 2338346 to run task
[2023-04-24, 14:17:00 UTC] {standard_task_runner.py:82} INFO - Running: ['***', 'tasks', 'run', 'realtime_runs', 'update_fs', 'scheduled__2023-04-24T14:16:00+00:00', '--job-id', '16604', '--raw', '--subdir', 'DAGS_FOLDER/realtime_runs.py', '--cfg-path', '/tmp/tmpc8_ilqp2']
[2023-04-24, 14:17:00 UTC] {standard_task_runner.py:83} INFO - Job 16604: Subtask update_fs
[2023-04-24, 14:17:00 UTC] {task_command.py:389} INFO - Running <TaskInstance: realtime_runs.update_fs scheduled__2023-04-24T14:16:00+00:00 [running]> on host vm6.template
[2023-04-24, 14:17:00 UTC] {taskinstance.py:1516} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=segmel
AIRFLOW_CTX_DAG_ID=realtime_runs
AIRFLOW_CTX_TASK_ID=update_fs
AIRFLOW_CTX_EXECUTION_DATE=2023-04-24T14:16:00+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2023-04-24T14:16:00+00:00
[2023-04-24, 14:17:00 UTC] {subprocess.py:63} INFO - Tmp dir root location:
/tmp
[2023-04-24, 14:17:00 UTC] {subprocess.py:75} INFO - Running command: ['/usr/bin/bash', '-c', 'python3.9 /var/segmel_ml/data_ml_pipelines/data/feature_store/real_time_features_update.py']
[2023-04-24, 14:17:00 UTC] {subprocess.py:86} INFO - Output:
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - 2023-04-24 17-17-01 realtime_session_features: start
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/1687082018.1681810239’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/260724567.1682331851’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/1986808320.1682344003’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/406102036.1682343979’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/754120029.1682343894’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/1579168920.1682343597’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - find: ‘/var/www/html/.tmp/1790003826.1673249460’: No such file or directory
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - Traceback (most recent call last):
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - File "/var/segmel_ml/data_ml_pipelines/data/feature_store/real_time_features_update.py", line 38, in <module>
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - main()
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - File "/var/segmel_ml/data_ml_pipelines/data/feature_store/real_time_features_update.py", line 21, in main
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - ldf_date = loader.load_session_features_for_active_users()
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - File "/usr/local/lib/python3.9/dist-packages/segmel/data/dataloaders.py", line 190, in load_session_features_for_active_users
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - active_users = ', '.join(self._get_active_users())
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - File "/usr/local/lib/python3.9/dist-packages/segmel/data/dataloaders.py", line 152, in _get_active_users
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - subprocess
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - File "/usr/lib/python3.9/subprocess.py", line 528, in run
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - raise CalledProcessError(retcode, process.args,
[2023-04-24, 14:17:01 UTC] {subprocess.py:93} INFO - subprocess.CalledProcessError: Command 'find /var/www/html/.tmp -mmin -2 -type f -exec basename {} \;' returned non-zero exit status 1.
[2023-04-24, 14:17:01 UTC] {subprocess.py:97} INFO - Command exited with return code 1
[2023-04-24, 14:17:01 UTC] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/airflow/operators/bash.py", line 196, in execute
raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2023-04-24, 14:17:01 UTC] {taskinstance.py:1327} INFO - Marking task as UP_FOR_RETRY. dag_id=realtime_runs, task_id=update_fs, execution_date=20230424T141600, start_date=20230424T141700, end_date=20230424T141701
[2023-04-24, 14:17:01 UTC] {standard_task_runner.py:100} ERROR - Failed to execute job 16604 for task update_fs (Bash command failed. The command returned a non-zero exit code 1.; 2338346)
[2023-04-24, 14:17:01 UTC] {local_task_job.py:212} INFO - Task exited with return code 1
[2023-04-24, 14:17:01 UTC] {taskinstance.py:2596} INFO - 0 downstream tasks scheduled from follow-on schedule check