a h-U@sdZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZmZernddlZddlmZeeZdd Ze rerdd lmZn dd lmZGd d d eZGdddeZdaddZddZddZddZ ddZ!ddZ"d"ddZ#d#d d!Z$dS)$z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nncCs@tjddu}|rzDHfTrainerDeepSpeedConfig.trainer_config_finalize..r hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rCrDg?rE z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_stepsZ warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.N)lenhasattrr2rJmaxrKrLr&r>is_zero3intr3Zget_warmup_stepsr%join)rrAmodelnum_training_stepsZhidden_size_based_keysZhidden_size_auto_keysrJr%rr'rtrainer_config_finalizesV       z0HfTrainerDeepSpeedConfig.trainer_config_finalize)NT)F) rrrr rr(r-r3rr>rBrWr!rrrrr"Rs    Jr"cCst|adSr#)weakrefref_hf_deepspeed_config_weak_ref)Zhf_deepspeed_config_objrrrrsrcCsdadSr#)rZrrrrunset_hf_deepspeed_configsr[cCs$tdurtdurtSdSdS)NF)rZrRrrrris_deepspeed_zero3_enabled$s r\cCs"tdurtdurtjSdSdSr#)rZr2rrrrdeepspeed_config+sr]csRt|dd|}dur"|_gd tjdfdd ||ddS) z Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` _metadataNF)modulec sdur inddi}||d<||dggf}trtfdd|Ddkrddl}t|jddddfd d|D}t|dkr|jj|dd (tj dkr|j |Wdn1s0Y|j D]&\}} | dur| ||d |qdS) Nassign_to_params_buffersTcsg|]}|r|qSr) startswith)rFkey)prefixrrrHJrIzC_load_state_dict_into_zero3_model..load..rF)reZrecursecsg|]}|vr|qSrr)rFk)named_parametersrrrHPrI)Z modifier_rank.)r/r\rOr dictrgzeroZGatheredParametersr@ distributedZget_rankZ_load_from_state_dictZ_modulesitems) r` state_dictrerbZlocal_metadatarAr Zparams_to_gathernamechildZ error_msgsloadr)rgrerrqCs   (z/_load_state_dict_into_zero3_model..load)rb)r_F)getattrcopyr^rModule)Z model_to_loadrmrrpr!_load_state_dict_into_zero3_model2s ruc sddlm}m}|j}d}d|vr<|jr0td||d}n"|rNtd }d|d <d} d |vrt||} n4t ||rfd d } ||| d } nj |d} || fS)zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.)paramszDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)TZzero_allow_untested_optimizerZ schedulercs"t}d|_|j|d}|S)NrVrx)rs lr_schedulercreate_scheduler)rxZ trainer_copyr{rVtrainerrr_lr_scheduler_callables  z5deepspeed_optim_sched.._lr_scheduler_callable)Zlr_scheduler_callablerz) Zaccelerate.utilsrvrwr2Z adafactorr&Z is_offloadloggerinfoZcreate_optimizer isinstancer|) r~hf_deepspeed_configrArVmodel_parametersrvrwr2rxr{rrr}rdeepspeed_optim_schedbs.     rFc Csddlm}|j}|j}|jjjj}||||| | |rv| sTt d| d| dd\}}d} nnd|_|jdid d } | d krddl} | j|| ||jd }ttd d |} t||||| \}}||fS)a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrxr{)NNNZtensor_parallelZ autotp_size)rUZtp_sizer(r2cSs|jSr#)Z requires_grad)prrrrIz deepspeed_init..)Zdeepspeed.utilsrrUrAZ acceleratorstateZdeepspeed_pluginZ hf_ds_configrWsetLevelZget_process_log_levelrRr&Zdel_config_sub_treerxr2r/r Z tp_model_initr(listfilter parametersr) r~rVZ inferenceZ ds_loggerrUrArrxr{rZdeepspeed_tp_sizer rrrdeepspeed_inits8     rTcCsxddl}t||d}t|dkrftd||j||ddd\}}|durttd|ntd|dS)Nrz /global_step*zAttempting to resume from T)load_module_strictZload_optimizer_statesZload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z!Can't find a valid checkpoint at )globsortedrOrrZload_checkpointr&)Zdeepspeed_engineZcheckpoint_pathrrZdeepspeed_checkpoint_dirsZ load_pathrrrrdeepspeed_load_checkpoints  r)F)T)%r rsimportlib.metadatarr importlib.utilr rX functoolsrZdependency_versions_checkrutilsrrrr@rZ get_loggerrrrZaccelerate.utils.deepspeedrZDeepSpeedConfigbuiltinsrr"rZrr[r\r]rurrrrrrrs6       B0= C