a h @sddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*edGddde j+Z,d@e j+ej-ej-ej-eej-e.e.dddZ/Gddde j+Z0e#GdddeZ1ee#ddGd d!d!eZ2Gd"d#d#e j+Z3Gd$d%d%e j+Z4Gd&d'd'e j+Z5e j6e,d(Z7Gd)d*d*eZ8Gd+d,d,e j+Z9e#Gd-d.d.e1Z:e#Gd/d0d0eZ;Gd1d2d2e j+Zee#d9dGd:d;d;e!Z?e#dd>e;eZ@gd?ZAdS)AN) dataclass)CallableOptionalUnion)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple torch_int) AutoModel)InternVLConfigInternVLVisionConfigZRMSNormcs.eZdZdfdd ZddZddZZS) InternVLVisionRMSNormư>cs&ttt||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__j/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/internvl/modeling_internvl.pyr .s zInternVLVisionRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)Zkeepdim) dtypetor#Zfloat32powmeanZrsqrtr&r%)r' hidden_statesZ input_dtypeZvariancer,r,r-forward6s  zInternVLVisionRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler%shaper&r'r,r,r- extra_repr=sz InternVLVisionRMSNorm.extra_repr)r)__name__ __module__ __qualname__r r4r8 __classcell__r,r,r*r-r,sr)modulequerykeyvalueattention_maskscalingdropoutc Ks|}|} t||dd|} |durV|ddddddd|jdf} | | } tjj| dd} tjj| ||jd} t| | } | dd } | | fS)Nrrr.dim)ptrainingr) r#matmul transposer6r! functionalZsoftmaxrDrI contiguous) r>r?r@rArBrCrDkwargs key_states value_states attn_weightsZ causal_mask attn_outputr,r,r-eager_attention_forwardAs & rScsNeZdZdZedfdd Zd ejeejeeje e dddZ Z S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigcs@t||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ |j }|j }d|_ tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_t|j|j|_|dkrt|nt|_ |rt|jnt|_|r2t|jnt|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFZbiasr)rr rVr( embed_dimZnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropoutZ use_qk_norm is_causalr!LinearZattention_biasq_projk_projv_projprojection_layerDropoutIdentityrq_normk_norm)r'rVZ proj_dropoutZqk_normr*r,r-r _s.   z InternVLVisionAttention.__init__N)r3rBoutput_attentionsrNcKs |\}}}||}||} ||} ||}|| } ||||j|j dd}| |||j|j dd} | |||j|j dd} t } |j j dkrt|j j } | ||| | |f|jsdn|j|jdd|\} } | |||j} || }||}|r|| fn|df}|S)Nrreagerr=F)rDrCr_)sizerarbrcrgrhreshaperYrZrKviewrSrVZ_attn_implementationrrIr]r\rXrdr^)r'r3rBrirN batch_sizeseq_len_Z query_statesrOrPZattention_interfacerRrQoutputoutputsr,r,r-r4{s<           zInternVLVisionAttention.forward)NN) r9r:r;__doc__rr r#Tensorrrr r4r<r,r,r*r-rT\srTcsHeZdZUeed<dZdZdZdgZdZ dZ dZ dZ fddZ ZS)InternVLVisionPreTrainedModelrVZinternvl_vision pixel_valuesTInternVLVisionLayercst|t|trP|jj|jdur8|jj|jdur~|jjn.t|t r~|j j |j j |jj |j j dS)zInitialize the weightsN)r _init_weights isinstanceInternVLVisionEmbeddings cls_tokendataZzero_ mask_tokenposition_embeddingsrwlambda_1Zfill_rVlayer_scale_init_valuelambda_2)r'r>r*r,r-rxs       z+InternVLVisionPreTrainedModel._init_weights)r9r:r;r__annotations__base_model_prefixZmain_input_namesupports_gradient_checkpointingZ_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrxr<r,r,r*r-rus ruz7 Class for outputs of [`InternVLVisionModel`]. )Z custom_introc@seZdZdZdS)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)r9r:r;rsr,r,r,r-rsrcs4eZdZdZfddZejejdddZZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}|d|d|d|d}|d|d|d|df}||_||_||_||_||_tj ||||d|_ dS)Nrr)Z kernel_sizeZstride) rr image_size patch_size num_channelsr( num_patches patch_shaper!ZConv2d projection)r'rVrrrr(rrr*r,r-r s   z&InternVLVisionPatchEmbeddings.__init__)rvreturnc Cs^|j\}}}}||jkr td||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r6rr[rflattenrK) r'rvrnrheightwidth embeddings patch_height patch_widthr,r,r-r4s  z%InternVLVisionPatchEmbeddings.forward) r9r:r;rsr r#rtr4r<r,r,r*r-rs rcs^eZdZdZeddfdd ZejeeejdddZ d eje ej ejd d d Z Z S) rzzc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. NrVrcstttdd|j|_|jrBttdd|j|_ nd|_ t ||_ |j |_ t |jtjjrp|jn |j|jf|_|j j}|jrttd|d|j|_nd|_t|j|_dS)Nr)rr r!r"r#Zzerosr(r{Zuse_mask_tokenr}rpatch_embeddingsrryr collectionsabcIterablerZ use_absolute_position_embeddingsr~rehidden_dropout_probrD)r'rVrr*r,r-r s    z!InternVLVisionEmbeddings.__init__)rrrrc Cs|jdd}|jjdd}tjs>||kr>||kr>|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr.r?rrZbicubicF)rkmodeZ align_cornersrF)r6r~r#Zjit is_tracingrrrlpermuter!rLZ interpolatermcat) r'rrrrZ num_positionsZclass_pos_embedZpatch_pos_embedrGZ new_heightZ new_widthZsqrt_num_positionsr,r,r-interpolate_pos_encodings(   z1InternVLVisionEmbeddings.interpolate_pos_encoding)rvbool_masked_posrcCs|j\}}}}||\}\}}|\} } }|durj|j| | d} |d| } |d| | | }|j| dd} tj | |fdd}|j dur|| |||}| |}|||ffS)Nr.rrF) r6rrkr}expand unsqueezeZtype_asr{r#rr~rrD)r'rvrrprrrrrrnroZ mask_tokenswZ cls_tokensr,r,r-r4:s  z InternVLVisionEmbeddings.forward)N)r9r:r;rsrr r#rtintrr BoolTensorr4r<r,r,r*r-rzs+rzcs0eZdZfddZejejdddZZS)InternVLVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rr rVrZ hidden_act activation_fnr!r`r(Zintermediate_sizefc1fc2r'rVr*r,r-r Us   zInternVLVisionMLP.__init__)r3rcCs"||}||}||}|Sr)rrr)r'r3r,r,r-r4\s   zInternVLVisionMLP.forward)r9r:r;r r#rtr4r<r,r,r*r-rTs r) layer_normZrms_normcsXeZdZdZeddfdd Zd ejee e eje ejejffddd Z Z S) rwz?This corresponds to the Block class in the timm implementation.Nrcst|j|_d|_t||_t||_t|j |j |j d|_ t|j |j |j d|_ |j}tj|t|j dd|_tj|t|j dd|_t|j|_dS)Nrr)T)Z requires_grad)rr Zchunk_size_feed_forwardZ seq_len_dimrT attentionrmlpNORM2FNZ norm_typer(layer_norm_epslayernorm_beforelayernorm_afterrr!r"r#r$rrrerrD)r'rVZ init_valuesr*r,r-r is   zInternVLVisionLayer.__init__F)r3rircCsl|j|||d\}}|j|}||}||}||}||}|jdur\|j|}||}||fS)N)ri)rrrrrrDr)r'r3riZattention_outputZattention_weightsZ layer_outputr,r,r-r4xs       zInternVLVisionLayer.forward)F) r9r:r;rsrr r#rtboolrr5r4r<r,r,r*r-rwfsrwcsHeZdZeddfdd Zed ejeee e e fdddZ Z S) InternVLVisionEncoderNrcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr,)rw).0irUr,r- z2InternVLVisionEncoder.__init__..F) rr rVr!Z ModuleListrangeZnum_hidden_layerslayerZgradient_checkpointingrr*rUr-r s  zInternVLVisionEncoder.__init__F)r3rioutput_hidden_statesrc Csz|rdnd}|rdnd}t|jD]:\}}|r8||f}|||}|d}|r"||df}q"|rl||f}t|||dS)Nr,rrlast_hidden_stater3 attentions) enumeraterr ) r'r3rirZall_hidden_statesZall_self_attentionsrZ layer_moduleZ layer_outputsr,r,r-r4s      zInternVLVisionEncoder.forward)FF)r9r:r;rr rr#rtrrr5r r4r<r,r,r*r-rs rc sdeZdZeddfdd ZddZeed ej e ej e e e e e eefddd ZZS) InternVLVisionModelNrcsTt|||_t||_t||_|jr4t ntj |j |j d|_ |dS)Nr)rr rVrzrrencoderZuse_mean_poolingr!rf LayerNormr(r layernorm post_initrr*r,r-r s   zInternVLVisionModel.__init__cCs|jjSr)rrr7r,r,r-get_input_embeddingssz(InternVLVisionModel.get_input_embeddings)rvrrirrc Csn|dur |n|jj}|dur |n|jj}|j||d\}}|j|||d}|d}||}t||j|jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)r)rirrr) rVrirrrrrr3r) r'rvrrirZembedding_outputrpZencoder_outputsZsequence_outputr,r,r-r4s  zInternVLVisionModel.forward)NNN)r9r:r;rr rrrr#rtrrrrr5rr4r<r,r,r*r-rs rc@s6eZdZUeed<dZdZdZdZdZ dZ dZ dZ dS)InternVLPreTrainedModelrVTpast_key_valuesN) r9r:r;rrrrZ_skip_keys_device_placementrrZ_can_compile_fullgraphrrr,r,r,r-rs rcs*eZdZedfdd ZddZZS)InternVLMultiModalProjectorrUcsztt|jjtd|jd|_t |jjtd|jd|j j|_ t |j |_t |j j|j j|_dS)Nrr)rr r!r vision_configr(rdownsample_ratiorr` text_configlinear_1rZprojector_hidden_actactlinear_2rr*r,r-r s " z$InternVLMultiModalProjector.__init__cCs,||}||}||}||}|Sr)rrrr)r'image_featuresr3r,r,r-r4 s     z#InternVLMultiModalProjector.forward)r9r:r;rr r4r<r,r,r*r-rs rzM Base class for InternVL outputs, with hidden states and attentions. c@s$eZdZUdZdZeejed<dS)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) r9r:r;rsrrr# FloatTensorrr,r,r,r-rs  rzx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. cs$eZdZddiZedfdd ZddZdd Zd d Zd d Z de j e e eeefe edddZe je j e j dddZeede je j e e je e je ee e j e e eeefe ee ee ee ee ee e jeee eefdddZde jedddZZ S) InternVLModelzlanguage_model.modellanguage_modelrUcs>t|t|j|_t||_t|j|_ | dSr) rr r from_configr vision_towerrmulti_modal_projectorrrrrr*r,r-r 1s   zInternVLModel.__init__cCs |jSr)rrr7r,r,r-r9sz"InternVLModel.get_input_embeddingscCs|j|dSr)rset_input_embeddingsr'rAr,r,r-r<sz"InternVLModel.set_input_embeddingscCs ||_dSrrr'decoderr,r,r- set_decoder?szInternVLModel.set_decodercCs|jSrrr7r,r,r- get_decoderBszInternVLModel.get_decoderNrvvision_feature_layervision_feature_select_strategyc Ks|dur |n|jj}|dur |n|jj}|j|jd}|jj}|dkrV|j|dj}n|j|dj |}|dkr|ddddddf}|j d}t |d}|j d} | | ||d}|j ||d }| | d|j d}||}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. N)r/r.)rvdefaultrrr) scale_factor)rVrrr0r/rrrZ vision_modelr3r6rrl pixel_shuffler) r'rvrrrNrvision_featureschannelsZ feature_sizernr,r,r-get_image_featuresEs*    z InternVLModel.get_image_features) input_ids inputs_embedsrcCs|dur8||tj|jjtj|jdk}|d}n ||jjk}|}| d | |j}|j d|j d}|| | krtd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)r/devicer.rrz6Image features and image tokens do not match: tokens: z , features )rr#ZtensorrVZimage_token_idlongrallsumrZ expand_asr0r6Znumelr[)r'rrrspecial_image_maskZn_image_tokensZn_image_featuresr,r,r-get_placeholder_mask{s  z"InternVLModel.get_placeholder_mask)rrvrB position_idsrrrr use_cacherir return_dictcache_positionrNrc Ks | dur | n|jj} | dur | n|jj} | dur4| n|jj} |durH|n|jj}|dur\|n|jj}|du|duAr|td|dur||}|dur|j|||d}| |j |j }|j |||d}| ||}|jf||||| | | d| d |}t|j|j|j|j|dur|nddS)Nz:You must specify exactly one of input_ids or inputs_embedsr)rrT) rBrrrrrirrr)rrr3rr)rVriruse_return_dictrrr[rrr0rr/rZmasked_scatterrrrrr3r)r'rrvrBrrrrrrrirrrrNrrrrr,r,r-r4sZ    zInternVLModel.forwardr)rrc Cs|\}}}}||dks(||dkr0td|||t||t||}|dddd}||t||t||t||d}|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rkr[rmrrrM)r'rrrnrrrr,r,r-rs $zInternVLModel.pixel_shuffle)NN) NNNNNNNNNNNNN)r)!r9r:r;_checkpoint_conversion_mappingrr rrrrr#rrrrliststrr LongTensorrrrrtrrrr r5rr4floatrr<r,r,r*r-r)s` 7  FrzT Base class for InternVL causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<dS) InternVLCausalLMOutputWithPasta] loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsrr3rr)r9r:r;rsrrr#rrrrrr3r5rrr,r,r,r-rs rzV The INTERNVL model which consists of a vision backbone and a language model. csbeZdZdddddZdgZedfdd Zd d Zd d Ze j dddZ ddZ ddZ d%ejeeeeefeedddZeddZeddZeddZeed&ejejeejeejeeeejeeeeefeeeejeeeeeeeeeejeeejfeeje e!ee"e#fd d!d"Z$d'fd#d$ Z%Z&S)( InternVLForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightrUcs<t|t||_tj|jj|jjdd|_ | dS)NFrW) rr rmodelr!r`rr( vocab_sizerrrr*r,r-r ,s  z)InternVLForConditionalGeneration.__init__cCs |jSr)rrr7r,r,r-r2sz5InternVLForConditionalGeneration.get_input_embeddingscCs|j|dSr)rrrr,r,r-r5sz5InternVLForConditionalGeneration.set_input_embeddings)rcCs|jSr)rr7r,r,r-get_output_embeddings8sz6InternVLForConditionalGeneration.get_output_embeddingscCs|j|dSr)rrrr,r,r-r;sz,InternVLForConditionalGeneration.set_decodercCs |jSr)rrr7r,r,r-r>sz,InternVLForConditionalGeneration.get_decoderNrcKs|jjf|||d|S)Nr)rr)r'rvrrrNr,r,r-rAsz3InternVLForConditionalGeneration.get_image_featurescCs|jjSr)rrr7r,r,r-rPsz/InternVLForConditionalGeneration.language_modelcCs|jjSr)rrr7r,r,r-rTsz-InternVLForConditionalGeneration.vision_towercCs|jjSr)rrr7r,r,r-rXsz6InternVLForConditionalGeneration.multi_modal_projectorr)rrvrBrrrrrlabelsrrirrrlogits_to_keep image_sizesrNrcKs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durH|n|jj}|dur\|n|jj}|jf||||||||| | | d||d|}|d}t|trt | dn|}| |dd|ddf}d}| dur|j f|| |jj j d|}t|||j|j|j|jdS)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NT)rrvrBrrrrrrrirrrr r)rrr)rrrr3rr)rVrirrrrrryrslicerZ loss_functionrrrrr3rr)r'rrvrBrrrrrrrrirrrr r rNrrr3Z slice_indicesrrr,r,r-r4\s\9 z(InternVLForConditionalGeneration.forwardc  s8tj|f|||||d|} |ddkr4|| d<| S)N)rrrBrr rrv)rprepare_inputs_for_generation) r'rrrrvrBrr rNZ model_inputsr*r,r-r s  z>InternVLForConditionalGeneration.prepare_inputs_for_generation)NN)NNNNNNNNNNNNNNrN)NNNNNN)'r9r:r;rZ_tied_weights_keysrr rrr!Modulerrrr#rrrrrrrpropertyrrrrrrrtrrrrr5rr4r r<r,r,r*r-rs      nr)rurrrr)r=)Bcollections.abcr dataclassesrtypingrrrr#Ztorch.nnr!Z activationsrZ cache_utilsrZ generationr Z integrationsr Zmodeling_flash_attention_utilsr Zmodeling_layersr Zmodeling_outputsr rrZmodeling_utilsrrZprocessing_utilsrutilsrrrrrautorZconfiguration_internvlrrr rrtrrSrTrurrrzrrrrwrrrrrrrr__all__r,r,r,r-s           K &^ 0&5RF