a ip @s UdZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZddlZddlmZmZmZddlmZmZddlmZmZmZmZdd lmZed d d Zd gZejeeddgdgdejejddee Z!e"dd Z#e"ddZ$e%e"ddZ&e"dd'Z(e"dd'Z)e"de"dd'Z*e"dd'+Z,dZ-e.d e.d!e-e.d"ej/0e-e.d#e1ej/0e-rze2e-Z3e.d$e3ej/4e-d%Z5ej/4e-d&Z6e.d'ej/0e5e.d(ej/0e6ej/0e6rVe7e6Z8e.d)e8j9d*e.d+e:e8j;Wn4e<yZ=ze.d,e=WYdZ=[=n dZ=[=00ne.d-e-e.d.ed d/d0d1Ze>Z?ej@jAd2d3d4d5ZBe?Cd6eBe?Cd7eBe d8d9d:d;ZDiZEdd?d@dAdBZGdCdDdEdFdGdHZHeIdIdJdKZJeIdLdMdNZKeLdOdPdQZMeIdRdSdTZNeIdUdVdWZOeIdXd>dYdZd[ZPeId\d]d^ZQeId_d`daZReIdbdcddZSeIded>dfdgdhZTeIdieddjdkeddjdkfd>dldldmdndoZUeIdpedqdrdkfdCdsdtduZVeIdvedqdrdkfdCdsdwdxZWGdydzdzZXdCdCd{d|d}ZYGd~ddeZZGdddeZ[ej\dOe[dddddZ]eIdd>dYddZ^e_dddZ`e dkrddlaZaeajbeddddS)ux Assistant Gateway (FastAPI) — Moodle + RAG + OpenAI Versão atualizada com correção definitiva do path INDEX_DIR Funcionalidades: - Endpoints Moodle (/courses, /course-contents, etc.) - RAG com FAISS + SentenceTransformers (/ask) - LLM integration (OpenAI compatible) - Debug endpoints para diagnóstico - Path absoluto fixo para resolver problemas de working directory ) annotationsN) lru_cache)AnyDictListOptional)FastAPI HTTPExceptionQuery) BaseModel validator)rr r Response)CORSMiddlewarezAssistant Gatewayz1.0.0)titleversionzhttps://lms.ed-consulting.aoT*)Z allow_originsZallow_credentialsZ allow_methodsZ allow_headersz4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformat MOODLE_URL MOODLE_TOKENREQUEST_TIMEOUTZ25 MODEL_API_URL MODEL_API_KEY MODEL_NAMEZ OPENAI_MODELz gpt-4o-miniMODEL_PROVIDERZopenaiz/var/www/html/assistant/indexesz=== GATEWAY CONFIGURATION === INDEX_DIR: INDEX_DIR exists: zCurrent working directory: zFiles in INDEX_DIR: zcourse_3.faisszcourse_3_meta.jsonzcourse_3.faiss exists: zcourse_3_meta.json exists: zcourse_3_meta.json size: z bytesz course_3_meta.json permissions: zError listing INDEX_DIR: zINDEX_DIR does not exist: z=== END CONFIGURATION ===1.2.0z0Moodle + RAG + LLM Gateway - Fixed Paths Version)rr description )Zpool_connectionsZ pool_maxsizeZ max_retrieszhttp://zhttps://)maxsizec Csz.ddlm}td|d}td|WStyd}ztd|WYd}~n>d}~0ty}ztd|WYd}~n d}~00dS) z:Cache do modelo SentenceTransformer para evitar recarregarrSentenceTransformerz$Loading SentenceTransformer model...z&sentence-transformers/all-MiniLM-L6-v2z-SentenceTransformer model loaded successfullyz&Failed to import SentenceTransformer: Nz*Failed to load SentenceTransformer model: )sentence_transformersr&loggerinfo ImportErrorerror Exception)r&modeler/"/var/www/html/assistant/gateway.pyget_sentence_transformerxs   r1zDict[int, 'Retriever']retrievers_cacheintz 'Retriever')courseidreturncCs,|tvr$td|t|t|<t|S)u4Cache dos retrievers para evitar recarregar índicesz"Creating new retriever for course )r2r(r) Retrieverr4r/r/r0 get_retrievers r8strDict[str, Any]r) wsfunctionparamsr5c Cststdtddtd}td|d}zdtj|||td}|| }t |t r| drtd |td d | d d |WSt jytdtddYnHt jy}z,td|tdd|WYd}~n d}~00dS)z%Chama Moodle REST API e retorna JSON.zMOODLE_TOKEN not configureduMOODLE_TOKEN não configuradoz/webservice/rest/server.phpjson)ZwstokenZmoodlewsrestformatr;)r<datatimeout exceptionzMoodle API error: iz Erro Moodle: messagezErro desconhecidozTimeout connecting to MoodleizTimeout ao contactar MoodlezMoodle request failed: zFalha ao contactar Moodle: N)rr(r+r rsessionpostrraise_for_statusr> isinstancedictgetrequestsZTimeoutRequestException)r;r<urlqsrr?r.r/r/r0 call_moodles,    rOz/healthcCs dttdttjtdS)zHealth check endpointokr)status timestampr index_dirindex_dir_exists)r3time INDEX_DIRospathexistsr/r/r/r0healths   rZz /debug-configc Cs6tttttjtttott t t dj dS)z"Debug configuration (safe version)sys)Z moodle_urlZhas_moodle_tokenrSrTZhas_model_configZ model_nameZ python_cwdpython_version) rboolrrVrWrXrYrrrgetcwd __import__rr/r/r/r0 debug_configs   r`z/askcCs tddS)N) status_code)r r/r/r/r0 options_asksrcz /cors-testcCsdttddS)z!Endpoint simples para testar CORSuCORS está funcionando!T)rBrRZorigin_allowed)r3rUr/r/r/r0 cors_tests rdz /debug-routesc CsTg}tjjD]>}z&||jttt|dgdWq tyHYq 0q d|iS)zList all available routesmethods)rXreroutes) appZrouterrfappendrXsortedlistgetattrr,)rfZrouter/r/r0 debug_routess   rlz/debug-rag/{courseid}r7c Csltjtd|d}tjtd|d}|t||tj|tj|td}d|fd|ffD]\}}tj|rdzJt|}|j||d<t|j dd ||d <t |tj ||d <|j ||d <|dkrzt |d dd}|}t||d<t|dkr*|d ddn||d<t|} d|d<t| j|d<t| ttfrnt| |d<Wd n1s0YWn2ty} zt| |d<WYd } ~ n d } ~ 00Wqdty} zt| ||d<WYd } ~ qdd } ~ 00qdi} z"dd l} dt| ddd| d<Wn8ty`} zd t| d!| d<WYd } ~ n d } ~ 00zdd"lm}d#di| d$<Wn8ty} zd t| d!| d$<WYd } ~ n d } ~ 00| |d%<zlt|}||d&<|r|j d'd(d)}t||d*<|r |d|d+<d,|d-<n|!|d.<d/|d-<Wn:tyf} z t| |d0<d1|d-<WYd } ~ n d } ~ 00|S)2z6Debug endpoint ATUALIZADO para verificar status do RAGcourse_.faiss .meta.json)r4rS index_path meta_path index_exists meta_existscwdindexmeta_sizeNZ _permissions _readableZ _modifiedrNutf-8encodingZmeta_content_length...Z meta_previewTZmeta_json_validZ meta_typeZ meta_itemsZmeta_read_error_errorrOK __version__unknown)rQrfaissERROR)rQr+r%rQr'Z dependenciesZ retriever_okz test queryr#kZtest_search_countZtest_search_sampleSUCCESSZretriever_statusZretriever_errorZFAILEDZretriever_exceptionZ EXCEPTION)"rWrXjoinrVrYr^statst_sizeoctst_modeaccessR_OKst_mtimeopenreadlenr>loadstype__name__rGrjrHr,r9rrkr*r'r&r8rPsearch last_error)r4rprqr)Zpath_keyZ file_path stat_infofcontentZ meta_dataZ read_errorr.Zdepsrr& retrieverZ test_resultsr/r/r0 debug_ragsx       & 0&*( (        rz /site-infocCs tdiS)zGet Moodle site informationZcore_webservice_get_site_inforOr/r/r/r0 site_infoDsrz/coursescCs tdiS)zGet all coursescore_course_get_coursesrr/r/r/r0coursesIsrz /courses_slimcCstdi}dd|DS)z&Get courses with essential fields onlyrcSs*g|]"}|d|d|ddqS)id shortnamefullname)rrr)rI).0cr/r/r0 Rs z courses_slim..r)r?r/r/r0 courses_slimNs rz /user-coursesuseridcCstdd|iS)zGet courses for a specific userZcore_enrol_get_users_coursesrrrr/r/r0 user_courses[srz/course-contentsz true/false)rz Optional[str])r4includecontentsexcludemodulescCsnd|i}d}|dur.[,\s]+z courseids[]Zmod_assign_get_assignmentsresplitstrip enumeraterOrZidsr<iZcidr/r/r0 assignmentsus rz/calendar/eventscCsZddtd|D}i}t|D](\}}d|d|d<||d|d<q&td|S) zGet calendar events for coursescSsg|]}|rt|qSr/rrr/r/r0rrz#calendar_events..rZcoursezevents[events][z ][eventtype]z][courseids][0]Z!core_calendar_get_calendar_eventsrrr/r/r0calendar_events~s rc@s^eZdZdZddddZddZdd Zd d d d Zdd ddZdddddddZ dS)r6zA Retriever corrigido com path absoluto e logs detalhados r3r7cCst||_tjtd|jd|_tjtd|jd|_d|_d|_ t d|dt d|jt d|j| dS) NrmrnroFz&*** INITIALIZING RETRIEVER FOR COURSE  ***z Index path: z Meta path: ) r3r4rWrXrrVrprq_init_ok _init_errr(r) _initialize)selfr4r/r/r0__init__s zRetriever.__init__c Cs&zddl}tdtd|jtj|j}td|td|jtj|j}td||std|j|std |jt |jtj st d |jt |jtj st d |jtd td ||_| |j|_ td|j jdtdt|_tdtdt|jddd}t|}Wdn1st0Ytdt|||d|_td|jdWnhty }zNd|_||_td|jd|dddl}td|WYd}~n d}~00dS)u:Initialize com logging detalhado e verificações robustasrNzFAISS imported successfullyzChecking if index exists: zIndex exists: zChecking if meta exists: z Meta exists: zFAISS index file not found: zMetadata file not found: zCannot read index file: zCannot read metadata file: zFile permissions OKzLoading FAISS index...z!FAISS index loaded successfully: z vectorsz$Getting SentenceTransformer model...zSentenceTransformer readyzLoading metadata file...rNrzr{zMetadata loaded: type=Tz?*** RETRIEVER INITIALIZATION COMPLETED SUCCESSFULLY FOR COURSE rFz/*** RETRIEVER INITIALIZATION FAILED FOR COURSE z: Full traceback: )rr(r)rprWrXrYrqFileNotFoundErrorrrPermissionErrorZ read_indexruZntotalr1embedrr>loadr_process_metadatarr4r,rr+ traceback format_exc)rrrrrsrrvr.rr/r/r0rsL      * zRetriever._initializec sttr<|_ddtD|_tdtdnttrڈ|_zPt dd D}fddt |d D|_td td |Wqt t fyt|_td td Yq0n g|_i|_td tdS)z%Process and normalize metadata formatcSsi|]\}}t||qSr/)r9)rritemr/r/r0 rz/Retriever._process_metadata..z%Metadata processed: list format with z itemscss|]}|rt|VqdS)N)isdigitr3)rrr/r/r0 rz.Retriever._process_metadata..csg|]}t|iqSr/)rIr9)rrrvr/r0rrz/Retriever._process_metadata..r#z%Metadata processed: dict format with z keys, max_index=z keys, non-numeric indiceszUnknown metadata format: N)rGrj meta_listr meta_dictr(r)rrHmaxkeysrange ValueError TypeErrorvalueswarningr)rrvZmax_keyr/rr0rs   zRetriever._process_metadatar])r5cCs|jS)z*Check if retriever is properly initialized)rrr/r/r0rPsz Retriever.okr9cCs|jrt|jSdS)zGet last initialization errorzNo error)rr9rr/r/r0rszRetriever.last_errorList[Dict[str, Any]])questionrr5c Cs6|jstd|td|ddd|d|j|g}|j||\}}g}t |dD]\}}|dkrzqhd} |t |j kr|j |} nt ||j vr|j t |} | st|d |d d } t| } | |t|t|d|tdd t|d|d || qhtdt |d|S)z"Search for top-k relevant passageszRetriever not initialized: zSearching for: 'N2z...' (k=)rzChunk z (no metadata))rtextg?)rankZ faiss_idxZdistanceZ similarityzSearch completed: found z results)r RuntimeErrorrr(r)rencoderurrrrr9rr3rHupdatefloatrrh) rrrZqvDIresultsrrrresultr/r/r0rs4   zRetriever.searchN)r) r __module__ __qualname____doc__rrrrPrrr/r/r/r0r6s<r6)promptr5c Cstr tr tstddSdtdd}tdddd |dgd d d }ztd ttjt||td}| | }| digd di dd }|stdWdStdt |d|WStjy}z,td|tdd|WYd}~n d}~00dS)zCall OpenAI-compatible LLM APIz0LLM not configured, returning simulated responseuwResposta simulada (LLM não configurado). Configure MODEL_API_URL, MODEL_API_KEY e MODEL_NAME para usar um modelo real.zBearer zapplication/json) Authorizationz Content-TypesystemugVocê é um assistente educacional. Responda com base nos trechos fornecidos de forma clara e objetiva.)roleruserg?i)r-messagesZ temperatureZ max_tokenszCalling LLM API: )headersr>r@choicesrrBrrzEmpty response from LLMzResposta vazia do modelo.zLLM response received: z charszLLM API call failed: rCzFalha ao contactar modelo: N)rrrr(rr)rDrErrFr>rIrrrJrKr+r )rrpayloadrNr?answerr.r/r/r0call_llm_openai!s>   & rc@sZeZdZUded<ded<dZded<eddd Zedd d Zedd d ZdS) AskRequestr9rr3r4rz Optional[int]top_kcCs0|r |stdt|dkr(td|S)NuPergunta não pode estar vaziaiu.Pergunta muito longa (máximo 2000 caracteres))rrrclsvr/r/r0validate_questionWs   zAskRequest.validate_questioncCs|dkrtd|S)NrzCourse ID deve ser positivorrr/r/r0validate_courseid_szAskRequest.validate_courseidcCs$|dur |dks|dkr td|S)Nr#r!ztop_k deve estar entre 1 e 20rrr/r/r0validate_top_keszAskRequest.validate_top_kN) rrr__annotations__rr rrrr/r/r/r0rRs    rc@s6eZdZUded<ded<ded<ded<ded <d S) AskResponser9rr3r4rpassagesrr:metadataN)rrrrr/r/r/r0rks r)Zresponse_model)requestc Cst}tdtd|jdddtd|jtd|jz*tdt|j}|sd |jd |d |j}t |t d |td t t d|jpdd}|j |j|d}tdt|d|s tdt|j|jgdt|dddWStdg}t|d|dD]\}}|dpn|dpn|dpnd} |dp|dp|d pd!|} | r@|d"|d#| d$| d%q@|st|j|j|d&t|d'ddWSd(|} d)|jd*| d+} td,t| } t|}td-|d.d/t|j|j|| |t|t| trptnd0d1d2dWSt yYnbty}zHt d3|d4dl}t d5|t d d6|WYd}~n d}~00dS)7zJ RAG endpoint atualizado com melhor logging e tratamento de erros z=== PROCESSING ASK REQUEST ===z Question: Ndr~z Course ID: zTop K: zGetting retriever...u"RAG não disponível para o curso z. Erro: u3. Para gerar o índice, execute: python3 ingest.py r=z"Retriever OK, performing search...r#rr!rzSearch completed: z passages foundzNo relevant passages founduINão foram encontrados conteúdos relevantes para esta pergunta no curso.Z no_content)processing_timerQ)rr4rrrzBuilding context for LLM...rrZsnippetrsourcefilerLz Documento u [CONTEÚDO z] z (Fonte: z) uROs conteúdos encontrados não possuem texto relevante para responder à pergunta.Zno_text z Pergunta: u[ Com base nos seguintes conteúdos do curso, forneça uma resposta educativa e completa: u' Instruções para a resposta: - Use apenas as informações fornecidas nos conteúdos acima - Seja claro, educativo e bem estruturado - Cite as fontes quando relevante (Fonte: ...) - Se a informação não estiver completa nos conteúdos, mencione isso - Organize a resposta de forma didáticazGenerating LLM response...z=== REQUEST COMPLETED IN z.2fzs ===Z simulatedZsuccess)r Zpassages_foundZcontext_lengthZ model_usedrQzUnexpected error in /ask: rrzErro interno: )rUr(r)rr4rr8rPrr+r minrrrrrrrIrrhrrrr,rr)rZ start_timerZ error_msgrrZ context_partsrZpassagerr contextrrr r.rr/r/r0askus         (. &     rz/courses/{courseid}/rag-statusc Cstjtd|d}tjtd|d}|dtj|tj|dtd}|dr|drzJt|}||d <ttj|tj||d <|d s| |d <Wn0t y}zt ||d <WYd}~n d}~00|S) z,Check RAG availability for a specific coursermrnroFN)r4 rag_availablerrrs last_modifiedrSrrrsrrr+) rWrXrrVrYr8rPrgetmtimerr,r9)r4rprqrQrr.r/r/r0 rag_statuss,      "rstartupc stdtdtdttdtjttdttdtttdtt ont tdt tjtrz,t t}d d |D}td |Wn4t y}ztd |WYd }~n d }~00tdd S)zApplication startup eventz"=== ASSISTANT GATEWAY STARTING ===zVersion: 1.2.0rrz MOODLE_URL: zHas MOODLE_TOKEN: zHas LLM config: zWorking Directory: cSs.g|]&}|dr|ds&|dr|qS)rmrnz _meta.json) startswithendswith)rrr/r/r0rrz!startup_event..zAvailable course indexes: z#Could not list INDEX_DIR contents: Nz=== STARTUP COMPLETE ===)r(r)rVrWrXrYrr]rrrr^listdirr,r)filesZ course_filesr.r/r/r0 startup_events     &r__main__z 127.0.0.1i@)hostport)cr __future__rrWr>rUrlogging functoolsrtypingrrrrrJZfastapirr r Zpydanticr r r Zfastapi.middleware.corsrrgZALLOWED_ORIGINSZadd_middleware basicConfigINFO getLoggerrr(getenvrrr3rrrrrlowerrrVprintrXrYr^rrrZcourse_3_faissZ course_3_metarrrrrr,r.ZSessionrDZadaptersZ HTTPAdapterZadapterZmountr1r2rr8rOrIrZr`optionsrcrdrlrrrrrrrrr6rrrrErrZon_eventruvicornrunr/r/r/r0s          &      "     V      1 m