You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
env: MEDLEYDB_PATH=/home/ec2-user/SageMaker/medleydb
/home/ec2-user/SageMaker/music-trees
Global seed set to 42
/home/ec2-user/SageMaker/medleydb/medleydb/__init__.py:64: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
INST_TAXONOMY = yaml.load(fhandle)
/home/ec2-user/SageMaker/medleydb/medleydb/__init__.py:72: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
MIXING_COEFFICIENTS = yaml.load(fhandle)
/home/ec2-user/SageMaker/medleydb/medleydb/__init__.py:94: UserWarning: The medleydb audio was not found at the expected path: None This module will still function, but without the ability to access any of the audio.
UserWarning
Global seed set to 42
2021-11-12:13:59:08,293 INFO [tune.py:748] Initializing Ray automatically.For cluster usage or custom Ray initialization, call `ray.init(...)` before `tune.run`.
2021-11-12 13:59:09,772 WARNING experiment.py:272 -- No name detected on trainable. Using DEFAULT.
2021-11-12 13:59:09,773 INFO registry.py:70 -- Detected unknown callable for trainable. Converting to class.
2021-11-12 13:59:09,780 INFO logger.py:606 -- pip install 'ray[tune]' to see TensorBoard files.
2021-11-12 13:59:09,780 WARNING callback.py:115 -- The TensorboardX logger cannot be instantiated because either TensorboardX or one of it's dependencies is not installed. Please make sure you have the latest version of TensorboardX installed: `pip install -U tensorboardx`
(pid=24752) Global seed set to 42
(pid=24752) /home/ec2-user/SageMaker/medleydb/medleydb/__init__.py:64: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
(pid=24752) INST_TAXONOMY = yaml.load(fhandle)
(pid=24752) /home/ec2-user/SageMaker/medleydb/medleydb/__init__.py:72: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
(pid=24752) MIXING_COEFFICIENTS = yaml.load(fhandle)
(pid=24752) /home/ec2-user/SageMaker/medleydb/medleydb/__init__.py:94: UserWarning: The medleydb audio was not found at the expected path: None This module will still function, but without the ability to access any of the audio.
(pid=24752) UserWarning
(pid=24752) Global seed set to 42
== Status ==
Current time: 2021-11-12 13:59:09 (running for 00:00:00.15)
Memory usage on this node: 1.6/15.4 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 60000.000: None
Resources requested: 1.0/4 CPUs, 1.0/1 GPUs, 0.0/8.63 GiB heap, 0.0/4.31 GiB objects (0.0/1.0 accelerator_type:T4)
Result logdir: /home/ec2-user/SageMaker/music-trees/runs/height-v1-11.12.2021/height-v1
Number of trials: 5/5 (4 PENDING, 1 RUNNING)
+---------------------+----------+----------------------+----------+
| Trial name | status | loc | height |
|---------------------+----------+----------------------+----------|
| DEFAULT_b5663_00000 | RUNNING | 172.16.105.192:24752 | 4 |
| DEFAULT_b5663_00001 | PENDING | | 2 |
| DEFAULT_b5663_00002 | PENDING | | 0 |
| DEFAULT_b5663_00003 | PENDING | | 3 |
| DEFAULT_b5663_00004 | PENDING | | 1 |
+---------------------+----------+----------------------+----------+
(ImplicitFunc pid=24752) PARSING KNOWN ARGS
(ImplicitFunc pid=24752) root
(ImplicitFunc pid=24752) ├── aerophones
(ImplicitFunc pid=24752) │ ├── free aerophones
(ImplicitFunc pid=24752) │ │ └── interruptive free aerophones
(ImplicitFunc pid=24752) │ │ └── interruptive free aerophones_
(ImplicitFunc pid=24752) │ │ ├── accordion
...
(ImplicitFunc pid=24752) ├── male rapper
(ImplicitFunc pid=24752) ├── male screamer
(ImplicitFunc pid=24752) ├── male singer
(ImplicitFunc pid=24752) ├── male speaker
(ImplicitFunc pid=24752) └── vocalists
(ImplicitFunc pid=24752)
2021-11-12 13:59:13,924 ERROR trial_runner.py:924 -- Trial DEFAULT_b5663_00000: Error processing event.
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 890, in _process_trial
results = self.trial_executor.fetch_result(trial)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 788, in fetch_result
result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/worker.py", line 1625, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=24752, ip=172.16.105.192, repr=<types.ImplicitFunc object at 0x7f1bbef16400>)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/trainable.py", line 224, in train_buffered
result = self.train()
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/trainable.py", line 283, in train
result = self.step()
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/function_runner.py", line 381, in step
self._report_thread_runner_error(block=True)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/function_runner.py", line 529, in _report_thread_runner_error
("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=24752, ip=172.16.105.192, repr=<types.ImplicitFunc object at 0x7f1bbef16400>)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/function_runner.py", line 262, in run
self._entrypoint()
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/function_runner.py", line 331, in entrypoint
self._status_reporter.get_checkpoint())
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ray/tune/function_runner.py", line 599, in _trainable_func
output = fn()
File "music_trees/search.py", line 110, in run_trial
return mt.train.train(hparams, use_ray=True)
File "/home/ec2-user/SageMaker/music-trees/music_trees/train.py", line 107, in train
output_filename=exp_dir / 'profiler-report.txt'),
TypeError: __init__() got an unexpected keyword argument 'output_filename'
The text was updated successfully, but these errors were encountered:
I get this error when running the training
Stacktrace:
The text was updated successfully, but these errors were encountered: