登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
AI 队友
登录
注册
Gitee 2025年度开源项目评选启动,快来选出你心中的最佳开源项目!
代码拉取完成,页面将自动刷新
开源项目
>
人工智能
>
计算机视觉/人脸识别
&&
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
4
Star
38
Fork
12
MindSpore Lab
/
mindcv
代码
Issues
2
Pull Requests
1
Wiki
统计
流水线
服务
JavaDoc
PHPDoc
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
我知道了,不再自动展开
更新失败,请稍后重试!
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息,仓库外成员不可访问
AttributeError: None object has no attribute: size
待办的
#I7SSFW
Vencen-31
创建于
2023-08-13 23:15
1. 问题说明: 我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码,导入了Gmlp模型。修改如图所示:  图 1 代码修改 2. 在ModelArts中的环境如下: os:欧拉 mindspore:1.10.1 python:3.7 Ascend:910 cann:  3. 两次尝试的报错 1. 这里是在 **jupyter lab** 中通过命令:mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt **得到以下报错** ``` ---------------------------------------- Num devices: 2 Distributed mode: True Num training samples: 1281168 Num classes: 1000 Num batches: 5005 Batch size: 128 Auto augment: autoaug Model: mobilenet_v2_035_224 Model param: 73075392 Num epochs: 300 Optimizer: adamw LR: 0.001 LR Scheduler: cosine_decay ---------------------------------------- Start training Traceback (most recent call last): File "train_with_func.py", line 506, in <module> train(args) File "train_with_func.py", line 296, in train log_interval=args.log_interval, File "train_with_func.py", line 410, in train_epoch loss, logits = train_step(data, label) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__ phase = self.compile(args_list, self.fn.__name__) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True) AttributeError: None object has no attribute: size ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat): # 0 In file train_with_func.py:389 (loss, logits), grads = grad_fn(data, label) ^ # 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554 return grad_(fn_, weights)(*args, **kwargs) ^ # 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498 outputs = fn(*args) ^ # 3 In file train_with_func.py:373 loss = loss_fn(logits, label) ^ # 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51 loss_logits = F.cross_entropy( # 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0 ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod Traceback (most recent call last): File "train_with_func.py", line 506, in <module> train(args) File "train_with_func.py", line 296, in train log_interval=args.log_interval, File "train_with_func.py", line 410, in train_epoch loss, logits = train_step(data, label) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__ phase = self.compile(args_list, self.fn.__name__) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True) AttributeError: None object has no attribute: size ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat): # 0 In file train_with_func.py:389 (loss, logits), grads = grad_fn(data, label) ^ # 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554 return grad_(fn_, weights)(*args, **kwargs) ^ # 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498 outputs = fn(*args) ^ # 3 In file train_with_func.py:373 loss = loss_fn(logits, label) ^ # 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51 loss_logits = F.cross_entropy( # 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0 ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[28690,1],1] Exit code: 1 -------------------------------------------------------------------------- ``` 2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息 ``` ---------------------------------------- Num devices: 1 Distributed mode: False Num training samples: 1281167 Num classes: 1000 Num batches: 10010 Batch size: 128 Auto augment: autoaug Model: mobilenet_v2_035_224 Model param: 73075392 Num epochs: 300 Optimizer: adamw LR: 0.001 LR Scheduler: cosine_decay ---------------------------------------- Start training ---------------------------------------- Num devices: 1 Distributed mode: False Num training samples: 1281167 Num classes: 1000 Num batches: 10010 Batch size: 128 Auto augment: autoaug Model: mobilenet_v2_035_224 Model param: 73075392 Num epochs: 300 Optimizer: adamw LR: 0.001 LR Scheduler: cosine_decay ---------------------------------------- Start training Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 0 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 0 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 0 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd [WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again Traceback (most recent call last): File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module> train(args) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train log_interval=args.log_interval, File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch loss, logits = train_step(data, label) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__ phase = self.compile(args_list, self.fn.__name__) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True) AttributeError: None object has no attribute: size ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat): # 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389 (loss, logits), grads = grad_fn(data, label) ^ # 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554 return grad_(fn_, weights)(*args, **kwargs) ^ # 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498 outputs = fn(*args) ^ # 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373 loss = loss_fn(logits, label) ^ # 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51 loss_logits = F.cross_entropy( # 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0 ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown len(cache)) -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[43151,1],1] Exit code: 1 -------------------------------------------------------------------------- ``` 如下图所示可以看到data and label 都不为None  图 2 debug 展示 感谢各位大佬帮忙!
1. 问题说明: 我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码,导入了Gmlp模型。修改如图所示:  图 1 代码修改 2. 在ModelArts中的环境如下: os:欧拉 mindspore:1.10.1 python:3.7 Ascend:910 cann:  3. 两次尝试的报错 1. 这里是在 **jupyter lab** 中通过命令:mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt **得到以下报错** ``` ---------------------------------------- Num devices: 2 Distributed mode: True Num training samples: 1281168 Num classes: 1000 Num batches: 5005 Batch size: 128 Auto augment: autoaug Model: mobilenet_v2_035_224 Model param: 73075392 Num epochs: 300 Optimizer: adamw LR: 0.001 LR Scheduler: cosine_decay ---------------------------------------- Start training Traceback (most recent call last): File "train_with_func.py", line 506, in <module> train(args) File "train_with_func.py", line 296, in train log_interval=args.log_interval, File "train_with_func.py", line 410, in train_epoch loss, logits = train_step(data, label) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__ phase = self.compile(args_list, self.fn.__name__) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True) AttributeError: None object has no attribute: size ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat): # 0 In file train_with_func.py:389 (loss, logits), grads = grad_fn(data, label) ^ # 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554 return grad_(fn_, weights)(*args, **kwargs) ^ # 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498 outputs = fn(*args) ^ # 3 In file train_with_func.py:373 loss = loss_fn(logits, label) ^ # 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51 loss_logits = F.cross_entropy( # 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0 ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod Traceback (most recent call last): File "train_with_func.py", line 506, in <module> train(args) File "train_with_func.py", line 296, in train log_interval=args.log_interval, File "train_with_func.py", line 410, in train_epoch loss, logits = train_step(data, label) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__ phase = self.compile(args_list, self.fn.__name__) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True) AttributeError: None object has no attribute: size ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat): # 0 In file train_with_func.py:389 (loss, logits), grads = grad_fn(data, label) ^ # 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554 return grad_(fn_, weights)(*args, **kwargs) ^ # 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498 outputs = fn(*args) ^ # 3 In file train_with_func.py:373 loss = loss_fn(logits, label) ^ # 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51 loss_logits = F.cross_entropy( # 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0 ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[28690,1],1] Exit code: 1 -------------------------------------------------------------------------- ``` 2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息 ``` ---------------------------------------- Num devices: 1 Distributed mode: False Num training samples: 1281167 Num classes: 1000 Num batches: 10010 Batch size: 128 Auto augment: autoaug Model: mobilenet_v2_035_224 Model param: 73075392 Num epochs: 300 Optimizer: adamw LR: 0.001 LR Scheduler: cosine_decay ---------------------------------------- Start training ---------------------------------------- Num devices: 1 Distributed mode: False Num training samples: 1281167 Num classes: 1000 Num batches: 10010 Batch size: 128 Auto augment: autoaug Model: mobilenet_v2_035_224 Model param: 73075392 Num epochs: 300 Optimizer: adamw LR: 0.001 LR Scheduler: cosine_decay ---------------------------------------- Start training Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 1 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 0 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 0 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd Traceback (most recent call last): File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main old_handlers) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one code = spawn._main(child_r) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main prepare(preparation_data) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module> from gmlp import gMLPVision File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module> class SpatialGatingUnit(nn.Cell): File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit act=nn.Identity(), File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__ super(Identity, self).__init__() File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__ init_pipeline() RuntimeError: Device 0 call rtSetDevice failed, ret[507033]. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- E39999: Inner Error, Please contact support engineer! E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243] TraceBack (most recent call last): TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644] Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973] Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315] new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336] rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd [WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again Traceback (most recent call last): File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module> train(args) File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train log_interval=args.log_interval, File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch loss, logits = train_step(data, label) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__ phase = self.compile(args_list, self.fn.__name__) File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True) AttributeError: None object has no attribute: size ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat): # 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389 (loss, logits), grads = grad_fn(data, label) ^ # 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554 return grad_(fn_, weights)(*args, **kwargs) ^ # 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498 outputs = fn(*args) ^ # 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373 loss = loss_fn(logits, label) ^ # 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51 loss_logits = F.cross_entropy( # 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0 ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown len(cache)) -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[43151,1],1] Exit code: 1 -------------------------------------------------------------------------- ``` 如下图所示可以看到data and label 都不为None  图 2 debug 展示 感谢各位大佬帮忙!
评论 (
1
)
登录
后才可以发表评论
状态
待办的
待办的
进行中
已完成
已关闭
负责人
未设置
标签
未设置
标签管理
里程碑
未关联里程碑
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
分支 (9)
标签 (8)
main
gh-pages
v0.5.0
v0.4.0
mint_dev
0.3.0
0.3.1
test/ms2.2
0.2.x
v0.3.0
v0.2.2
v0.2.1
v0.2.0
v0.1.1
v0.1
v0.0.1-beta
v0.0.1-alpha
开始日期   -   截止日期
-
置顶选项
不置顶
置顶等级:高
置顶等级:中
置顶等级:低
优先级
不指定
严重
主要
次要
不重要
参与者(1)
1
https://gitee.com/mindspore-lab/mindcv.git
git@gitee.com:mindspore-lab/mindcv.git
mindspore-lab
mindcv
mindcv
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大,如何减小?
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release(发行版)
将 PHP 项目自动发布到 packagist.org
评论
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号,请先登录后再操作。
立即登录
没有帐号,去注册