a hG@s>dZddlZddlmZddlmZddZGdd d ZdS) zA Loss functions for linear models with raw_prediction = X @ coef N)sparse) squared_normcCsZ|jd}t|r8|jtj|df||fd|S|dddf|}|j|SdS)z/Compute the sandwich product X.T @ diag(W) @ X.rshapeN)rrissparseT dia_matrixZtoarray)XW n_samplesZWXr ]/var/www/html/assistant/venv/lib/python3.9/site-packages/sklearn/linear_model/_linear_loss.py sandwich_dots  rc@sleZdZdZddZdddZddZd d Zd d ZdddZ dddZ dddZ dddZ dddZ dS)LinearModelLossa General class for loss functions with raw_prediction = X @ coef + intercept. Note that raw_prediction is also known as linear predictor. The loss is the average of per sample losses and includes a term for L2 regularization:: loss = 1 / s_sum * sum_i s_i loss(y_i, X_i @ coef + intercept) + 1/2 * l2_reg_strength * ||coef||_2^2 with sample weights s_i=1 if sample_weight=None and s_sum=sum_i s_i. Gradient and hessian, for simplicity without intercept, are:: gradient = 1 / s_sum * X.T @ loss.gradient + l2_reg_strength * coef hessian = 1 / s_sum * X.T @ diag(loss.hessian) @ X + l2_reg_strength * identity Conventions: if fit_intercept: n_dof = n_features + 1 else: n_dof = n_features if base_loss.is_multiclass: coef.shape = (n_classes, n_dof) or ravelled (n_classes * n_dof,) else: coef.shape = (n_dof,) The intercept term is at the end of the coef array: if base_loss.is_multiclass: if coef.shape (n_classes, n_dof): intercept = coef[:, -1] if coef.shape (n_classes * n_dof,) intercept = coef[n_features::n_dof] = coef[(n_dof-1)::n_dof] intercept.shape = (n_classes,) else: intercept = coef[-1] Shape of gradient follows shape of coef. gradient.shape = coef.shape But hessian (to make our lives simpler) are always 2-d: if base_loss.is_multiclass: hessian.shape = (n_classes * n_dof, n_classes * n_dof) else: hessian.shape = (n_dof, n_dof) Note: If coef has shape (n_classes * n_dof,), the 2d-array can be reconstructed as coef.reshape((n_classes, -1), order="F") The option order="F" makes coef[:, i] contiguous. This, in turn, makes the coefficients without intercept, coef[:, :-1], contiguous and speeds up matrix-vector computations. Note: If the average loss per sample is wanted instead of the sum of the loss per sample, one can simply use a rescaled sample_weight such that sum(sample_weight) = 1. Parameters ---------- base_loss : instance of class BaseLoss from sklearn._loss. fit_intercept : bool cCs||_||_dS)N) base_loss fit_intercept)selfrrr r r__init__hszLinearModelLoss.__init__NcCsZ|jd}|jj}|jr"|d}n|}|jjrFtj|||f|dd}ntj|||d}|S)aAllocate coef of correct shape with zeros. Parameters: ----------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. dtype : data-type, default=None Overrides the data type of coef. With dtype=None, coef will have the same dtype as X. Returns ------- coef : ndarray of shape (n_dof,) or (n_classes, n_dof) Coefficients of a linear model. F)rdtypeorder)rr)rr n_classesr is_multiclassnpZ zeros_like)rr r n_featuresrn_dofcoefr r rinit_zero_coefls  zLinearModelLoss.init_zero_coefcCs|jjs.|jr$|d}|dd}qd}|}nV|jdkrP|j|jjdfdd}n|}|jr|dddf}|ddddf}nd}||fS)aHelper function to get coefficients and intercept. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). Returns ------- weights : ndarray of shape (n_features,) or (n_classes, n_features) Coefficients without intercept term. intercept : float or ndarray of shape (n_classes,) Intercept terms. Nrrr)rrrndimreshaper)rr interceptweightsr r rweight_intercepts z LinearModelLoss.weight_interceptcCs<||\}}|jjs$|||}n||j|}|||fS)aiHelper function to get coefficients, intercept and raw_prediction. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. Returns ------- weights : ndarray of shape (n_features,) or (n_classes, n_features) Coefficients without intercept term. intercept : float or ndarray of shape (n_classes,) Intercept terms. raw_prediction : ndarray of shape (n_samples,) or (n_samples, n_classes) )r'rrr)rrr r&r%raw_predictionr r rweight_intercept_raws z$LinearModelLoss.weight_intercept_rawcCs&|jdkr||nt|}d||S)z5Compute L2 penalty term l2_reg_strength/2 *||w||_2^2.rg?)r#r)rr&l2_reg_strengthZnorm2_wr r r l2_penaltyszLinearModelLoss.l2_penaltyr!rc Cs\|dur|||\}} }n||\}} |jj||d|d} tj| |d} | |||S)aCompute the loss as weighted average over point-wise losses. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. y : contiguous array of shape (n_samples,) Observed, true target values. sample_weight : None or contiguous array of shape (n_samples,), default=None Sample weights. l2_reg_strength : float, default=0.0 L2 regularization strength n_threads : int, default=1 Number of OpenMP threads to use. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). If provided, these are used. If None, then raw_prediction = X @ coef + intercept is calculated. Returns ------- loss : float Weighted average of losses per sample, plus penalty. NZy_truer( sample_weight n_threadsr&)r)r'rlossraverager+) rrr yr-r*r.r(r&r%r0r r rr0s'zLinearModelLoss.losscCs\|j|jj\}} } | t|j} |dur>|||\} } }n||\} } |jj||||d\}}|durp|nt |}| |}|| | |7}||}|jj stj || j d}|j||| |d| <|jr| |d<nptj| | f| j dd}|j||| |ddd| f<|jr<|j dd|dddf<|jd krT|jdd }||fS) a\Computes the sum of loss and gradient w.r.t. coef. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. y : contiguous array of shape (n_samples,) Observed, true target values. sample_weight : None or contiguous array of shape (n_samples,), default=None Sample weights. l2_reg_strength : float, default=0.0 L2 regularization strength n_threads : int, default=1 Number of OpenMP threads to use. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). If provided, these are used. If None, then raw_prediction = X @ coef + intercept is calculated. Returns ------- loss : float Weighted average of losses per sample, plus penalty. gradient : ndarray of shape coef.shape The gradient of the loss. Nr,rr rrrrZaxisrr")rrrintrr)r' loss_gradientrsumr+r empty_likerremptyr#ravel)rrr r2r-r*r.r(r rrrr&r%r0grad_pointwisesw_sumgradr r rr7 s6*  "  zLinearModelLoss.loss_gradientcCs>|j|jj\}} } | t|j} |dur>|||\} } }n||\} } |jj||||d}|durl|nt |}||}|jj stj || j d}|j ||| |d| <|jr| |d<|Stj| | f| j dd}|j ||| |ddd| f<|jr|j dd|dddf<|jd kr6|jdd S|SdS) aComputes the gradient w.r.t. coef. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. y : contiguous array of shape (n_samples,) Observed, true target values. sample_weight : None or contiguous array of shape (n_samples,), default=None Sample weights. l2_reg_strength : float, default=0.0 L2 regularization strength n_threads : int, default=1 Number of OpenMP threads to use. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). If provided, these are used. If None, then raw_prediction = X @ coef + intercept is calculated. Returns ------- gradient : ndarray of shape coef.shape The gradient of the loss. Nr,r3r rr4rr5rr")rrrr6rr)r'gradientrr8rr9rrr:r#r;)rrr r2r-r*r.r(r rrrr&r%r<r=r>r r rr?Xs4' "  zLinearModelLoss.gradientc CsB|j|jj\} } } | t|j} | dur>|||\}}} n||\}}|durX| nt|}|dur~tj ||j dd}nF|j|jkrt d|jd|jdn|jj r|j jst dn|}|j}|durtj||f|j d}nZ|j||fkrt d ||fd |jdn,|jj r>|j js>|j js>t d n|}|jj sr|jj|| ||d \}}||}||}tj|d k|ddk}t|}|j||||d| <|jr||d<|r|||fSt|||d| d| f<|d kr0|j jrdnd}|jd|dd| | | d|7<|jr8|j|}||dddf<||dddf<||d<n|jj|| ||d \}}||}|j| | fdd}|j||||ddd| f<|jr|jd d|dddf<|jdkr|jdd}|dur||}nd|}t| D]}|dd|fd|dd|f|}t||||| | | || | | f<|jr|j|}|||| | | | | |f<||| | ||| | | f<||| | || | |f<t|d| D]}|dd|f |dd|f|}t||||| | | || | | f<|jr|j|}|||| | | | | |f<||| | ||| | | f<||| | || | |f<||d| |d| f||d| |d| f<qq"|d kr4|j jrdnd}|jd|dd| d| | | | d|7<d}|||fS)a~Computes gradient and hessian w.r.t. coef. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. y : contiguous array of shape (n_samples,) Observed, true target values. sample_weight : None or contiguous array of shape (n_samples,), default=None Sample weights. l2_reg_strength : float, default=0.0 L2 regularization strength n_threads : int, default=1 Number of OpenMP threads to use. gradient_out : None or ndarray of shape coef.shape A location into which the gradient is stored. If None, a new array might be created. hessian_out : None or ndarray of shape (n_dof, n_dof) or (n_classes * n_dof, n_classes * n_dof) A location into which the hessian is stored. If None, a new array might be created. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). If provided, these are used. If None, then raw_prediction = X @ coef + intercept is calculated. Returns ------- gradient : ndarray of shape coef.shape The gradient of the loss. hessian : ndarray of shape (n_dof, n_dof) or (n_classes, n_dof, n_dof, n_classes) Hessian matrix. hessian_warning : bool True if pointwise hessian has more than 25% of its elements non-positive. Nrr4z4gradient_out is required to have shape coef.shape = z; got .z"gradient_out must be F-contiguous.r3z'hessian_out is required to have shape (z); got hessian_out.shape=zhessian_out must be contiguous.r,rr/g?r Cr"r)r r r5g?rF)rrrr6rr)r'rr8r9r ValueErrorrflags f_contiguoussizer: c_contiguousgradient_hessianr1absrrr$gradient_probar#r;range)rrr r2r-r*r.Z gradient_outZ hessian_outr(r rrrr&r%r=r>nZhessr<hess_pointwiseZhessian_warningrZXhprobaswkhlr r rrGs7               "  &  (       &       4  z LinearModelLoss.gradient_hessianc sj jj\}t j \ }} dur@|nt  jjs@ jj || |d\} } | } | } tj  j d} j |  | d< jr| | d<| t rt j| df||fdn| ddtjf jr&ttjddt fdd } n jj|| |d\} | } tjf j d d } | j  | dddf< jr| jdd| dddf< f d d } jd kr| jd d| fS| | fS)aComputes gradient and hessp (hessian product function) w.r.t. coef. Parameters ---------- coef : ndarray of shape (n_dof,), (n_classes, n_dof) or (n_classes * n_dof,) Coefficients of a linear model. If shape (n_classes * n_dof,), the classes of one feature are contiguous, i.e. one reconstructs the 2d-array via coef.reshape((n_classes, -1), order="F"). X : {array-like, sparse matrix} of shape (n_samples, n_features) Training data. y : contiguous array of shape (n_samples,) Observed, true target values. sample_weight : None or contiguous array of shape (n_samples,), default=None Sample weights. l2_reg_strength : float, default=0.0 L2 regularization strength n_threads : int, default=1 Number of OpenMP threads to use. Returns ------- gradient : ndarray of shape coef.shape The gradient of the loss. hessp : callable Function that takes in a vector input of shape of gradient and and returns matrix-vector product with hessian. Nr,r3r rrr5cst|}tr4j|d|d<n$tjj|dg|d<|d|d7<jr|d|d7<|d|d|d<|S)Nr )rr9rrrZlinalgZ multi_dotr)sret)r hXhX_sum hessian_sumr*rrr rhessps   $  z7LinearModelLoss.gradient_hessian_product..hessprr4cs|jdfdd}jr>|dddf}|ddddf}nd}|j|}| |jddddtjf7}|9}dur|ddtjf9}tjf jdd}|j ||dddf<jr|jdd |dddf<jdkr|j ddS|SdS)Nr rr"rrr5r4) r$rrr8rnewaxisr:rr#r;)rRZ s_intercepttmpZ hess_prod) r rr*rrrrMr-rr=r&r rrWs"$&  rr")rrrr6rr)rr8rrGr9rrrrr rXZsqueezeZasarrayZ atleast_1drIr:r#r;)rrr r2r-r*r.r r%r(r<rLr>rWr )r rrTrUrVr*rrrrMr-rr=r&rgradient_hessian_productsV       "  z(LinearModelLoss.gradient_hessian_product)N)Nr!rN)Nr!rN)Nr!rN)Nr!rNNN)Nr!r)__name__ __module__ __qualname____doc__rrr'r)r+r0r7r?rGrZr r r rr%s@B '   ; S N r) r^numpyrZscipyrZ utils.extmathrrrr r r rs