can not load Video-XL-2 model

[laoxie001]

You are using a model of type qwen2 to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
Loading vision tower: /share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384
/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384
Traceback (most recent call last):
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/utils/hub.py", line 470, in cached_files
    hf_hub_download(
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384'. Use `repo_type` argument if needed.

model has been download to local, but i get this error when i run the example inference code in README.md.

[UnableToUseGit]

You are using a model of type qwen2 to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
Loading vision tower: /share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384
/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384
Traceback (most recent call last):
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/utils/hub.py", line 470, in cached_files
    hf_hub_download(
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384'. Use `repo_type` argument if needed.

model has been download to local, but i get this error when i run the example inference code in README.md.

Hi Alice : )
To resolve this issue, please set the mm_vision_tower parameter to 'google/siglip-so400m-patch14-384' in the config.py file.

[LeiPang0817]

where is config.py file？ Is this referring to the config.json in Hugging Face? The parameter in it is already set to this value.

[UnableToUseGit]

where is config.py file？ Is this referring to the config.json in Hugging Face? The parameter in it is already set to this value.

Yep, is config.json. The value of the parameter is "/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384", you should change it to "google/siglip-so400m-patch14-384"

[LeiPang0817]

where is config.py file？ Is this referring to the config.json in Hugging Face? The parameter in it is already set to this value.

Yep, is config.json. The value of the parameter is "/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384", you should change it to "google/siglip-so400m-patch14-384"

Thanks, I got it.

[laoxie001]

You are using a model of type qwen2 to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
Loading vision tower: /share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384
/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384
Traceback (most recent call last):
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/utils/hub.py", line 470, in cached_files
    hf_hub_download(
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384'. Use `repo_type` argument if needed.

model has been download to local, but i get this error when i run the example inference code in README.md.

Hi Alice : ) To resolve this issue, please set the mm_vision_tower parameter to 'google/siglip-so400m-patch14-384' in the config.py file.
thanks, it works :)
but i get a cuda out of memory error now. Is there any way to reduce the usage of cuda memory?

Traceback (most recent call last):
  File "/home/server/XWF/Video-XL/Video-XL-2/test_video.py", line 8, in <module>
    model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map=device,quantization_config=None,attn_implementation="sdpa").to(torch.bfloat16)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 309, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4573, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5030, in _load_pretrained_model
    disk_offload_index, cpu_offload_index = _load_state_dict_into_meta_model(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 842, in _load_state_dict_into_meta_model
    _load_parameter_into_model(model, param_name, param.to(param_device))
                                                  ^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 260.00 MiB. GPU 0 has a total capacity of 23.69 GiB of which 135.69 MiB is free. Including non-PyTorch memory, this process has 23.16 GiB memory in use. Of the allocated memory 22.87 GiB is allocated by PyTorch, and 38.91 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[UnableToUseGit]

sry, the demo inference code is old. we have update it in our huggingface page , you counld find it and try it again. The memory peak of the new code is 23.7GB.

…

---- Replied Message ---- | From | ***@***.***> | | Date | 06/07/2025 09:42 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [VectorSpaceLab/Video-XL] can not load Video-XL-2 model (Issue #55) | laoxie001 left a comment (VectorSpaceLab/Video-XL#55) You are using a model of type qwen2 to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors. Loading vision tower: /share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384 /share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384 Traceback (most recent call last): File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/utils/hub.py", line 470, in cached_files hf_hub_download( File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn validate_repo_id(arg_value) File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id raise HFValidationError( huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/share/LXRlxr0_0/code/videoxlturbo2.0/videoxl/google/siglip-so400m-patch14-384'. Use `repo_type` argument if needed. model has been download to local, but i get this error when i run the example inference code in README.md. Hi Alice : ) To resolve this issue, please set the mm_vision_tower parameter to 'google/siglip-so400m-patch14-384' in the config.py file. thanks, it works :) but i get a cuda out of memory error now. Is there any way to reduce the usage of cuda memory? Traceback (most recent call last): File "/home/server/XWF/Video-XL/Video-XL-2/test_video.py", line 8, in <module> model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map=device,quantization_config=None,attn_implementation="sdpa").to(torch.bfloat16) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 309, in _wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4573, in from_pretrained ) = cls._load_pretrained_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5030, in _load_pretrained_model disk_offload_index, cpu_offload_index = _load_state_dict_into_meta_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/server/software/miniconda3/envs/nerfstream/lib/python3.12/site-packages/transformers/modeling_utils.py", line 842, in _load_state_dict_into_meta_model _load_parameter_into_model(model, param_name, param.to(param_device)) ^^^^^^^^^^^^^^^^^^^^^^ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 260.00 MiB. GPU 0 has a total capacity of 23.69 GiB of which 135.69 MiB is free. Including non-PyTorch memory, this process has 23.16 GiB memory in use. Of the allocated memory 22.87 GiB is allocated by PyTorch, and 38.91 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>