rbsetiawan
/
LDAPWeb
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases Wiki Activity
GeoNetAgent, LDAPWeb, server-audit, server-connection
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
14 Commits
1 Branch
0 Tags
19 MiB
 
 
 
 
 
 
Tree: d500db95b3
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'd500db95b3'
${ noResults }
LDAPWeb/server-connection/server/ollama-nas-project-search-r...

34 KiB
Raw Blame History Escape

Requirement & Rekomendasi — Ollama + Pencarian Kontekstual NAS

Tanggal: 2026-06-15
Use case: Pencarian proyek/file di share QNAP berdasarkan konteks natural language
Contoh query: "Carikan projek Pertamina → lokasi folder dan kapan dibuat"
Integrasi target: geonet-console + PostgreSQL (10.100.1.25) + Ollama inference host terpisah

Ringkasan keputusan

Pertanyaan Jawaban
Apakah Ollama bisa langsung scan \\10.100.1.10\project? Tidak — perlu indexer + database
Bisa multi-folder (project, Marketing, dll.)? Ya — via env indexer, bukan env Ollama
Host Ollama POC & produksi CPU VM 10.100.1.26 (aiopr) — Ubuntu 22.04, dedicated
Host Ollama GPU (target) Migrasi ke Windows 11 + GPU (~6 bulan), ganti OLLAMA_BASE_URL
Model Ubuntu → Windows portable? Ya — copy folder models/ atau ollama pull ulang
Indeks RAG ikut pindah ke PC Windows? Tidak perlu — tetap di PostgreSQL .25
Tanya license code / tanggal expired dari PDF sertifikat? Ya — via License extraction + license_registry (lihat section khusus)

Arsitektur

┌─────────────────────────────────────────────────────────────────┐
│  User → geonet-console (chat UI) / API v1                       │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│  Project Search Service (Laravel / worker)                      │
│  - Terima query natural language                                │
│  - Filter scope source (project, marketing, …)                  │
│  - Audit log setiap pencarian                                   │
└──────────────┬──────────────────────────────┬───────────────────┘
               │                              │
               ▼                              ▼
┌──────────────────────────┐    ┌─────────────────────────────────┐
│  PostgreSQL 10.100.1.25  │    │  Ollama 10.100.1.26 (aiopr)     │
│  - project_index         │    │  - qwen2.5 (chat)               │
│  - project_chunks        │    │  - nomic-embed-text (embedding)   │
│  - license_registry      │    │  OLLAMA_BASE_URL → .26:11434    │
│  - pgvector              │    │                                   │
└──────────────────────────┘    └─────────────────────────────────┘
               ▲
               │
┌──────────────────────────┐
│  Indexer Worker (.26 POC)│
│  - Mount SMB QNAP          │
│  - Crawl incremental       │
│  - Ekstrak teks dokumen    │
│  - Generate embedding      │
└──────────────┬─────────────┘
               │
               ▼
┌──────────────────────────┐
│  QNAP TS-932X 10.100.1.10│
│  SMB shares              │
└──────────────────────────┘

Prinsip: Ollama = lapisan bahasa alami. Indexer + PostgreSQL = otak pencarian. NAS = storage saja.

Sumber data NAS (QNAP)

Share utama — use case project search

SMB path Path internal NAS Catatan
\\10.100.1.10\project /share/CACHEDEV3_DATA/project Struktur per tahun 2019–2026
Ukuran (audit 2026-06-15) ~2,6 TB, ~1,4 juta file Indeks incremental wajib

Share tambahan (multi-source)

SMB path Path internal NAS
\\10.100.1.10\Marketing /share/CACHEDEV2_DATA/Marketing
\\10.100.1.10\Marketing\Nanda\DATA SALES subfolder Marketing
\\10.100.1.10\GIS /share/CACHEDEV2_DATA/GIS
\\10.100.1.10\sales /share/CACHEDEV3_DATA/sales
\\10.100.1.10\development /share/CACHEDEV3_DATA/development

Lihat symlink lengkap: SSH admin@10.100.1.10 → ls -la /share/

Contoh folder Pertamina (verifikasi 2026-06-15)

  • project\2020\Pertamina EP
  • project\2021\Pertamina EP, Pertamina Hulu Rokan (PHR)
  • project\2023\PT. Pertamina Geothermal Energy Tbk, PHKT, PHE Holding, dll.
  • project\2025\PT. Pertamina Geothermal Energy Tbk, PERTAMINA EP
  • project\2026\PT. Pertamina Geothermal Energy Tbk

Fase implementasi

Fase 1 — Metadata search (1–2 minggu)

Scope: path, nama folder/klien, tahun (dari parent folder), mtime/ctime, owner.

Item Requirement
Crawler Scan SMB, simpan metadata ke PostgreSQL
Pencarian Keyword + filter tahun/source
LLM Ollama format jawaban dari hasil terstruktur (bukan tebak path)
Jawaban contoh Lokasi UNC + tahun folder + tanggal modifikasi

Catatan tanggal "kapan dibuat": birth time di QNAP sering tidak andal. Gunakan:

  1. Tahun dari path (/2023/...)
  2. mtime folder
  3. Tanggal file tertua di dalam folder (opsional)

Fase 2 — RAG penuh (produksi)

Scope: ekstrak teks .docx, .pdf, .xlsx, .msg → chunk → embedding → semantic search.

Item Requirement
Embedding model nomic-embed-text via Ollama
Vector store PostgreSQL + pgvector di 10.100.1.25
Chat model qwen2.5:7b (CPU) / qwen2.5:14b (GPU)
Chunk strategy Per dokumen, max ~512–1024 token, metadata source_id

Fase 2b — License / Certificate extraction (prioritas tinggi)

Scope: parse terstruktur dokumen sertifikat & license dari folder proyek — akurat untuk kode license, pemilik, dan tanggal expired.

Lihat section License / Certificate extraction di bawah.

License / Certificate extraction

Use case

User bertanya dalam bahasa natural; sistem jawab dari data terstruktur (bukan tebakan LLM):

Pertanyaan contoh Jawaban yang diharapkan
"License AzqehoVMExxV24ww milik siapa?" PT. Kalimantan Prima Persada
"Kapan expired AzqehoVMExxV24ww?" End date 09/06/2027
"Sertifikat Global Mapper PO A17944 untuk siapa?" PT. Kalimantan Prima Persada, license AzqehoVMExxV24ww
"Certificate ESRI No 583582?" Pertamina Hulu Kalimantan Timur

File verifikasi (2026-06-15)

Path SMB:

\\10.100.1.10\project\2026\PT Kalimantan Prima Persada #AzqehoVMExxV24ww\Sertifikat Software Global Mapper PT Kalimantan Prima Persada - PO A17944.pdf

Hasil ekstrak teks (template Geonet / Blue Marble):

Field Nilai
Template geonet_global_mapper_v1
End User PT. Kalimantan Prima Persada
License Number AzqehoVMExxV24ww
Produk Global Mapper V27 Network License (qty 4)
Produk Global Mapper Pro add-on V27 Network License (qty 4)
Start Date 10/06/2026
End Date 09/06/2027
PO A17944 (dari nama file)
Penerbit PT. Geonet Infomedia

PDF ini berbasis teks — ekstraksi pypdf/pdfplumber cukup, tanpa OCR.

Metadata folder: nama folder mengandung #AzqehoVMExxV24ww → Fase 1 bisa link license ke klien tanpa buka PDF; end date tetap butuh parse dokumen.

Prinsip desain

PDF/MSG/DOCX  →  Deteksi template  →  Parser terstruktur  →  license_registry
                                                          ↘
                                                    project_chunks (RAG opsional)
                                                          ↓
User query  →  Lookup license_registry (exact)  →  Ollama format jawaban natural
Layer Fungsi
Template detector Cocokkan header/frasa khas per vendor (ESRI, Geonet GM, dll.)
Structured parser Ekstrak field: license_code, client_name, start_date, end_date, …
license_registry Sumber kebenaran untuk kode license & tanggal — jangan andalkan LLM saja
Ollama Hanya merangkai jawaban dari baris DB yang match
RAG (project_chunks) Pelengkap untuk pertanyaan naratif di luar field terstruktur

Aturan akurasi: untuk license_code, end_date, client_name → jawab hanya dari license_registry. Jika tidak ada di DB, jawab "tidak ditemukan di indeks" — jangan hallucinate.

Alur indexer — dokumen license

  1. Crawl file dengan pola nama/path (lihat Prioritas indeks).
  2. Ekstrak teks:
    • PDF teks → pypdf / pdfplumber
    • PDF scan / JPG / JPEG → OCR (Tesseract ind+eng)
    • .msg → extract-msg / Outlook parser
    • .docx → python-docx
  3. detect_template(extracted_text, filename) → template_id
  4. parse(template_id, text) → satu atau banyak baris license_registry
  5. Simpan raw_excerpt + confidence_score untuk audit
  6. (Opsional) chunk teks ke project_chunks untuk RAG

Skema license_registry

-- Enum-like: gunakan VARCHAR + CHECK atau tabel license_vendors

CREATE TABLE license_vendors (
    id              VARCHAR(32) PRIMARY KEY,   -- 'esri', 'blue_marble', 'mapinfo', ...
    display_name    VARCHAR(128) NOT NULL
);

INSERT INTO license_vendors (id, display_name) VALUES
    ('esri', 'Esri / ESRI Indonesia'),
    ('blue_marble', 'Blue Marble Geographics (Global Mapper)'),
    ('geonet_reseller', 'PT. Geonet Infomedia (Certificate of Authenticity)'),
    ('mapinfo', 'Precisely MapInfo'),
    ('safe_software', 'Safe Software (FME)'),
    ('other', 'Lainnya / belum diklasifikasi');

CREATE TABLE license_document_templates (
    id              VARCHAR(64) PRIMARY KEY,   -- 'geonet_global_mapper_v1', 'esri_certificate_id_v1', ...
    vendor_id       VARCHAR(32) NOT NULL REFERENCES license_vendors(id),
    display_name    VARCHAR(255) NOT NULL,
    detection_rules JSONB NOT NULL,            -- frasa/header regex untuk auto-detect
    parser_class    VARCHAR(128) NOT NULL,     -- FQCN parser di worker
    is_active       BOOLEAN NOT NULL DEFAULT true
);

CREATE TABLE license_registry (
    id                  BIGSERIAL PRIMARY KEY,
    source_id           VARCHAR(64) NOT NULL,       -- sama dengan project_index
    template_id         VARCHAR(64) REFERENCES license_document_templates(id),
    vendor_id           VARCHAR(32) NOT NULL REFERENCES license_vendors(id),

    -- Identitas license (index utama pencarian)
    license_code        VARCHAR(128),               -- AzqehoVMExxV24ww, EUN#, SA#, dll.
    license_code_norm   VARCHAR(128) GENERATED ALWAYS AS (upper(trim(license_code))) STORED,
    esri_auth_number    VARCHAR(64),                -- nomor certificate ESRI jika ada (583582)
    order_number        VARCHAR(64),                -- PO A17944, Order # 216760-AXGWF5T5
    product_name        VARCHAR(255) NOT NULL,
    product_version     VARCHAR(64),                -- V27, 10.9.1, ArcGIS 11.0
    license_type        VARCHAR(64),                -- network, single_use, concurrent, maintenance
    quantity            INT,

    -- Pemilik / end user
    client_name         VARCHAR(255) NOT NULL,
    client_email        VARCHAR(255),
    client_address      TEXT,

    -- Periode
    start_date          DATE,
    end_date            DATE,                       -- maintenance end / expired
    date_format_hint    VARCHAR(32) DEFAULT 'D/M/Y', -- format asli di dokumen

    -- Lokasi file sumber
    relative_path       TEXT NOT NULL,
    full_smb_path       TEXT NOT NULL,
    file_name           VARCHAR(512) NOT NULL,
    file_ext            VARCHAR(16),
    file_mtime          TIMESTAMPTZ,
    file_sha256         CHAR(64),                   -- deduplikasi re-index

    -- Audit ekstraksi
    raw_excerpt         TEXT,                       -- cuplikan teks yang diparse (max ~4 KB)
    confidence_score    SMALLINT CHECK (confidence_score BETWEEN 0 AND 100),
    parse_warnings      JSONB,                      -- field yang gagal / ambigu
    indexed_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    parser_version      VARCHAR(32) NOT NULL,

    UNIQUE (source_id, full_smb_path, license_code, product_name)
);

-- Pencarian cepat license code (pertanyaan paling umum)
CREATE INDEX idx_license_registry_code ON license_registry (license_code_norm);
CREATE INDEX idx_license_registry_esri_auth ON license_registry (esri_auth_number) WHERE esri_auth_number IS NOT NULL;
CREATE INDEX idx_license_registry_client ON license_registry USING gin (client_name gin_trgm_ops);
CREATE INDEX idx_license_registry_end_date ON license_registry (end_date);
CREATE INDEX idx_license_registry_order ON license_registry (order_number) WHERE order_number IS NOT NULL;

-- View: license akan expired dalam 90 hari
CREATE VIEW license_expiring_soon AS
SELECT *
FROM license_registry
WHERE end_date IS NOT NULL
  AND end_date BETWEEN CURRENT_DATE AND CURRENT_DATE + INTERVAL '90 days'
ORDER BY end_date;

Normalisasi license_code: simpan asli + license_code_norm (uppercase, trim). Pertanyaan user case-insensitive.

Satu PDF → banyak baris: template Global Mapper Geonet menghasilkan 2+ produk dengan license code sama — insert satu baris per produk (lihat contoh verifikasi).

Daftar template dokumen

Template di bawah ini ditemukan di share \\10.100.1.10\project (sampling 2019–2026). Implementasi parser per prioritas bisnis.

Prioritas P1 — implementasi pertama

template_id Vendor Pola file / deteksi Field utama
geonet_global_mapper_v1 Geonet / Blue Marble Certificate of Authenticity, PT. Geonet Infomedia, tabel License Number + Maintenance Period license_code, client_name, start_date, end_date, product_name, quantity
esri_certificate_id_v1 ESRI Indonesia Certificate ESRI Indonesia, Certificate Esri No, ESRI's License Certificate esri_auth_number, client_name, product_name, periode tahun di judul
esri_license_msg_v1 Esri File .msg subject Esri License File for ArcGIS order_number, product_name, client_name (dari subject/body)

Contoh path NAS (P1):

  • ...\2026\PT Kalimantan Prima Persada #AzqehoVMExxV24ww\Sertifikat Software Global Mapper ... PO A17944.pdf → geonet_global_mapper_v1
  • ...\2023\PT. Pertamina Hulu Kalimantan Timur\license 10.9.1\Certificate Esri No 583582 ....pdf → esri_certificate_id_v1
  • ...\2023\PHE Holding\ESRI's License Certificate Pertamina Hulu Energi Holding 2021-2022.pdf → esri_certificate_id_v1
  • ...\2023\PHE Holding\Esri License File for ArcGIS 10.x (...).msg → esri_license_msg_v1

Prioritas P2 — fase berikutnya

template_id Vendor Pola file / deteksi Field utama
geonet_global_mapper_msg_v1 Blue Marble .msg subject Blue Marble Geographics License, License Instruction Global Mapper license_code, order_number, client_name
esri_sertifikat_legacy_v1 ESRI Sertifikat License ESRI, Sertifikat ESRI - client_name, periode di nama file
mapinfo_certificate_v1 MapInfo / Precisely Sertifikat MapInfo, Certificate MapInfo client_name, product_name, periode
geonet_license_docx_v1 Geonet License.docx, License.pdf di subfolder order license_code, client_name

Contoh path NAS (P2):

  • ...\2023\PT. ADARO INDONESIA ...\Sertifikat GM_PT Adaro Indonesia.pdf
  • ...\2023\PT. Ciptakridatama\...\License.pdf, License.docx
  • ...\2023\PT. Minamas Plantation\Sertifikat MapInfo PT. Anugerah Sumber Makmur.pdf
  • ...\2023\PT. GEOTRAV BHUANA SURVEY\Sertifikat PT. geotrav Bhuana Survey_05-08-2022.pdf

Prioritas P3 — butuh OCR atau parser khusus

template_id Vendor Pola file / deteksi Catatan
license_image_ocr_v1 Generic license.JPG, port license *.JPG, *.jpeg screenshot aktivasi Wajib OCR; confidence lebih rendah
safe_fme_v1 Safe Software FME license file Jika muncul di share
geocalculator_v1 Blue Marble GeoCalculator license (sering di bundle GM) Parse dari email/PDF terkait
concurrent_license_server_v1 Various Concurrent License Server *.exe, ZIP license Metadata dari sidecar PDF/MSG

Contoh path NAS (P3):

  • ...\2023\Star Energy Wayang Windu\license.JPG
  • ...\2023\PT. Pertamina Geothermal Energy Tbk\Globalmapper\port license 27000.JPG
  • ...\2023\PT. Pamapersada Nusantara\*Server License*.jpeg

Prioritas indeks file license

Indexer tidak parse semua 1,4 juta file sekaligus. Prioritas:

1. **/project/*/*certificate*
2. **/project/*/*sertifikat*
3. **/project/*/*license*
4. **/project/*/*licence*
5. Folder dengan pola *#<license_code>* di nama folder
6. Ekstensi: .pdf, .msg, .docx (P1–P2)
7. .jpg, .jpeg, .png → antrian OCR (P3)

Env indexer:

LICENSE_INDEX_PATTERNS=**/certificate*,**/sertifikat*,**/license*,**/*#*
LICENSE_INDEX_EXTENSIONS=.pdf,.msg,.docx
LICENSE_OCR_EXTENSIONS=.jpg,.jpeg,.png
LICENSE_PARSER_VERSION=1.0.0

API — license lookup

Endpoint Method Fungsi
/api/v1/license-search/ask POST Natural language → lookup license_registry + Ollama format
/api/v1/license-search/code/{code} GET Lookup exact license_code
/api/v1/license-search/expiring GET Daftar license expired ≤ N hari
/api/v1/license-search/reindex POST Trigger re-parse file (admin, audit)

Request contoh:

{
  "query": "license AzqehoVMExxV24ww milik siapa dan kapan expired?"
}

Response contoh (dari data terverifikasi):

{
  "answer": "License AzqehoVMExxV24ww milik PT. Kalimantan Prima Persada. End date 09/06/2027 (maintenance 10/06/2026 – 09/06/2027). Produk: Global Mapper V27 Network License dan Global Mapper Pro add-on V27.",
  "matches": [
    {
      "license_code": "AzqehoVMExxV24ww",
      "client_name": "PT. Kalimantan Prima Persada",
      "product_name": "Global Mapper V27 Network License",
      "quantity": 4,
      "start_date": "2026-06-10",
      "end_date": "2027-06-09",
      "order_number": "A17944",
      "template_id": "geonet_global_mapper_v1",
      "full_smb_path": "\\\\10.100.1.10\\project\\2026\\PT Kalimantan Prima Persada #AzqehoVMExxV24ww\\Sertifikat Software Global Mapper PT Kalimantan Prima Persada - PO A17944.pdf",
      "confidence_score": 98
    }
  ],
  "source": "license_registry"
}

Parser geonet_global_mapper_v1 — field mapping

Deteksi: teks mengandung Certificate of Authenticity dan PT. Geonet Infomedia dan License Number.

Label di PDF Kolom DB
End User Name client_name
Email client_email
Address client_address
License Number license_code
Maintenance Period Start / End Date start_date, end_date
Kolom Product product_name
Qty quantity
Nama file PO A17944 order_number (regex PO [A-Z0-9]+)

Format tanggal dokumen: D/M/Y (contoh 09/06/2027 → simpan ISO 2027-06-09).

Parser esri_certificate_id_v1 — field mapping

Deteksi: Certificate ESRI Indonesia, Certificate Esri No, atau ESRI's License Certificate.

Sumber Kolom DB
Certificate Esri No 583582 esri_auth_number
Nama klien di judul / body client_name
Periode di judul 2021-2022 start_date, end_date (approx year range)
Versi ArcGIS di path folder 10.9.1 product_version

Kendala & mitigasi

Kendala Mitigasi
PDF hasil scan (image) OCR + flag confidence_score rendah
Satu license code, banyak produk Multi-row di license_registry
Tanggal format bervariasi Parser per template; simpan date_format_hint
File .msg Outlook Parser khusus; fallback ekstrak subject + attachment PDF
LLM salah tanggal Wajib jawab dari license_registry, bukan dari memory model
Re-index setelah replace file Bandingkan file_sha256; update baris jika hash berubah

Checklist License extraction

  • Tabel license_vendors, license_document_templates, license_registry di .25
  • Parser P1: geonet_global_mapper_v1, esri_certificate_id_v1
  • Index folder project/2025, project/2026 (prioritas tahun terbaru)
  • Endpoint /api/v1/license-search/code/{code}
  • Alert license_expiring_soon (opsional → n8n / email)
  • Audit log query license (data sensitif)

Spesifikasi host

VM Ollama 10.100.1.26 (aiopr) — audit live 2026-06-15

Resource Nilai saat ini Target implementasi
Hostname aiopr —
IP 10.100.1.26/24 (ens18) —
OS Ubuntu 22.04.5 Desktop Ubuntu Server 22.04 (disarankan) atau minimal tanpa GUI
CPU 8 vCPU Xeon E5-2699 v4, AVX2 ✅ 16 vCPU (wajib naik)
RAM 62 GB (~60 GB free) ✅ 64 GB — sudah memenuhi
Disk / 503 GB (468 GB free) ✅ Sisakan ≥ 100 GB untuk OLLAMA_MODELS
GPU Tidak ada (virtual VGA) Tidak wajib fase CPU; GPU di Windows nanti
Virtualisasi KVM —
Jaringan → .10 NAS Ping OK Wajib untuk indexer
Jaringan → .25 PostgreSQL Port 5432 OK Wajib
Jaringan → .24 geonet-console Ping OK Consumer OLLAMA_BASE_URL
Ollama Belum terinstall Install + systemd
Port 11434 Belum listen Bind internal VLAN saja
Mount SMB Belum ada cifs-utils + mount read-only (indexer)
SSH Key-based (root / geonet) ✅ .\scripts\ssh-ollama.ps1

Verdict: Host cocok untuk POC dan produksi CPU (Opsi B). Reverse proxy 10.100.1.24 tidak dipakai untuk Ollama — resource terbatas (15 GB RAM, Docker penuh).

Kebutuhan tambahan di 10.100.1.26 (selain naik CPU ke 16)

# Kategori Item Prioritas Catatan
1 Compute vCPU 8 → 16 P0 Target Opsi B; inference + indexer paralel
2 OS Ubuntu Desktop → Server atau matikan GUI/snap P1 Hemat ~2 GB RAM + CPU idle
3 Ollama Install Ollama + systemd enable P0 curl -fsSL https://ollama.com/install.sh | sh
4 Model ollama pull nomic-embed-text P0 Embedding Fase 1 + 2
5 Model ollama pull qwen2.5:3b (POC) lalu qwen2.5:7b (produksi CPU) P0 POC bisa mulai 3b, naik ke 7b setelah CPU 16
6 Env Ollama OLLAMA_HOST=0.0.0.0:11434, OLLAMA_MODELS=/data/ollama/models P0 Lihat section Env Ollama server
7 Storage model Folder /data/ollama/models (≥ 100 GB) P0 qwen2.5:7b + nomic-embed-text ≈ 8–10 GB
8 Firewall ufw aktif: allow 22, allow 11434 dari 10.100.1.0/24 saja P0 Jangan expose ke internet
9 Integrasi app OLLAMA_BASE_URL=http://10.100.1.26:11434 di .env geonet-console (.24) P0 Edit hanya di server
10 Indexer SMB cifs-utils, smbclient, mount /mnt/qnap read-only P0 POC Kredensial NAS hanya di server
11 Indexer worker Service crawl metadata → PostgreSQL .25 P0 POC Bisa di VM ini untuk POC; pisah saat produksi penuh
12 PostgreSQL .25 Extension pgvector + tabel project_index P0 Fase 1 / P1 Fase 2 Bukan di host .26, tapi wajib untuk stack
13 Monitoring Zabbix agent atau healthcheck ollama ps P2 Opsional POC
14 NTP systemd-timesyncd aktif P1 Audit log konsisten

Tidak perlu di .26: Docker (Ollama native cukup), GPU, mirror 2,6 TB NAS ke disk lokal, Nginx publik.

Reverse proxy 10.100.1.24 — bukan host Ollama

Resource Nilai
CPU 8 vCPU (KVM, tanpa GPU)
RAM 15 GB total, ~12 GB available
Disk / 50 GB, ~32 GB free
Docker RAM ~1,5 GB (n8n, grafana, geonet-console, dll.)
Peran Gateway 39+ domain *.gisportal.id + geonet-console

Verdict: Tetap sebagai reverse proxy & app host saja. Jangan jalankan Ollama atau indexer 1,4 juta file di sini.

Opsi A — POC minimum (tanpa GPU) — fase awal di .26

Resource Spesifikasi Status .26
Host 10.100.1.26 ✅
vCPU 8 (sementara) → 16 (target) ⚠️ 8/16
RAM 16–32 GB ✅ 62 GB
Disk 100 GB SSD ✅ 503 GB
OS Ubuntu 22.04 LTS ⚠️ Desktop (perlu optimasi)
Model qwen2.5:3b + nomic-embed-text ❌ belum pull
User 5–10 internal —

Opsi B — Produksi Fase 2 RAG (CPU only) — target host .26

Resource Spesifikasi
Host 10.100.1.26
vCPU 16 (AVX2)
RAM 64 GB
Disk 200 GB SSD
OS Ubuntu 22.04 LTS
Model qwen2.5:7b + nomic-embed-text
User 10–30 internal
Latensi estimasi 5–15 detik/query (7B CPU)

Opsi C — Produksi optimal (GPU) — target 6 bulan

Resource Spesifikasi
Host PC Windows 11 Pro (always-on)
vCPU 8–16
RAM 32–64 GB
GPU NVIDIA min 16 GB VRAM (T4, L4, RTX 4060 Ti 16GB, A4000)
Disk 200 GB NVMe
Model qwen2.5:14b Q4 atau llama3.1:8b + nomic-embed-text
Latensi estimasi 2–8 detik/query

VM Indexer worker (terpisah dari Ollama)

Resource Spesifikasi
vCPU 4–8
RAM 8–16 GB
Disk 50 GB (hanya metadata, tidak mirror NAS)
Network Akses SMB ke 10.100.1.10, API ke .25
Job Crawl incremental, ekstrak teks, kirim embedding

QNAP TS-932X — jangan jalankan Ollama produksi

Alasan
ARM Cortex-A57, inferensi LLM lambat
Storage pool 83–93% penuh
Peran NAS = storage, bukan compute AI

Multi-folder via environment

Ollama tidak punya env untuk daftar folder SMB. Konfigurasi di indexer / geonet-console (.env server).

Contoh .env indexer

# Mount CIFS di Linux indexer
NAS_MOUNT_BASE=/mnt/qnap
NAS_SMB_HOST=10.100.1.10
NAS_SMB_USER=admin
# NAS_SMB_PASS — hanya di server, jangan commit

# Daftar path (Linux mount path)
PROJECT_SEARCH_PATHS=/mnt/qnap/project,/mnt/qnap/Marketing/Nanda/DATA SALES,/mnt/qnap/GIS

# Label untuk UI & filter RBAC
PROJECT_SEARCH_SOURCES=project:Project NAS,marketing_sales:Marketing DATA SALES,gis:GIS

# Exclude
PROJECT_SEARCH_EXCLUDE=@Recycle,*.qcow2,*.iso,~$*

# Jadwal crawl (cron)
PROJECT_INDEX_CRON_PROJECT=0 2 * * *
PROJECT_INDEX_CRON_MARKETING=0 3 * * 0

Contoh .env aplikasi (geonet-console)

OLLAMA_BASE_URL=http://10.100.1.26:11434
OLLAMA_CHAT_MODEL=qwen2.5:7b
OLLAMA_EMBED_MODEL=nomic-embed-text
PROJECT_SEARCH_DEFAULT_SOURCES=project

Ganti IP saat migrasi Ubuntu → Windows GPU — indeks PostgreSQL tidak perlu rebuild.

Model Ollama — daftar rekomendasi

Model Fungsi RAM/VRAM approx Fase
nomic-embed-text Embedding semantic ~0,3 GB 1 + 2
qwen2.5:3b Chat POC ~4 GB POC di .26
qwen2.5:7b Chat produksi CPU ~8 GB Fase 2 di .26 (Opsi B)
qwen2.5:14b Chat produksi GPU ~16 GB VRAM Fase 2 (Opsi C)

Simpan daftar ini di repo; saat migrasi Windows gunakan ollama pull atau copy folder models.

Skema database (PostgreSQL .25)

-- Fase 1: metadata
CREATE TABLE project_index (
    id              BIGSERIAL PRIMARY KEY,
    source_id       VARCHAR(64) NOT NULL,   -- 'project', 'marketing_sales', ...
    relative_path   TEXT NOT NULL,
    full_smb_path   TEXT NOT NULL,          -- \\10.100.1.10\project\...
    client_name     VARCHAR(255),
    year_folder     SMALLINT,
    is_directory    BOOLEAN NOT NULL DEFAULT true,
    file_ext        VARCHAR(16),
    size_bytes      BIGINT,
    mtime           TIMESTAMPTZ,
    indexed_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (source_id, relative_path)
);

CREATE INDEX idx_project_index_client ON project_index USING gin (client_name gin_trgm_ops);
CREATE INDEX idx_project_index_source_year ON project_index (source_id, year_folder);

-- Fase 2: RAG chunks (butuh extension pgvector)
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE project_chunks (
    id              BIGSERIAL PRIMARY KEY,
    source_id       VARCHAR(64) NOT NULL,
    file_path       TEXT NOT NULL,
    chunk_index     INT NOT NULL,
    chunk_text      TEXT NOT NULL,
    embedding       vector(768),
    indexed_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

License / sertifikat: tabel license_registry, license_vendors, license_document_templates — lihat section License / Certificate extraction.

API requirement (geonet-console)

Endpoint Method Fungsi
/api/v1/project-search/ask POST Query natural language + optional sources[]
/api/v1/project-search/index/status GET Status indeks per source
/api/v1/project-search/sources GET Daftar source dari env
/api/v1/license-search/ask POST Query license natural language (lihat section License)
/api/v1/license-search/code/{code} GET Lookup exact license code
/api/v1/license-search/expiring GET License mendekati expired

Request contoh:

{
  "query": "carikan projek Pertamina",
  "sources": ["project"],
  "limit": 20
}

Response contoh:

{
  "answer": "Ditemukan 9 folder proyek Pertamina di \\\\10.100.1.10\\project ...",
  "results": [
    {
      "source_id": "project",
      "path": "\\\\10.100.1.10\\project\\2025\\PT. Pertamina Geothermal Energy Tbk",
      "year_folder": 2025,
      "mtime": "2025-03-05T09:56:00+07:00"
    }
  ]
}

Wajib enterprise:

  • Auth LDAP / API token
  • Audit log: user, query, sources, IP, duration
  • Throttle API
  • RBAC per source (mis. marketing_sales hanya role Sales)

Migrasi Ollama: Ubuntu 22 → Windows 11 + GPU

Yang portable

Item Portable
File model (~/.ollama/models/) ✅
Custom Modelfile ✅
Indeks pgvector di .25 ✅ tetap, tidak pindah
Metadata project_index ✅ tetap

Langkah migrasi

  1. Catat model: ollama list → simpan ke repo
  2. Deploy Ollama Windows + driver NVIDIA
  3. Opsi A: copy models/ dari Ubuntu
    Opsi B (disarankan): ollama pull qwen2.5:7b + nomic-embed-text
  4. Update OLLAMA_BASE_URL di .env geonet-console
  5. Update Nginx internal jika perlu
  6. Verifikasi GPU: Task Manager / log Ollama saat inference

Path model

OS Path
Ubuntu ~/.ollama/models/
Windows C:\Users\<user>\.ollama\models\

Env Ollama server (bukan folder NAS)

OLLAMA_HOST=0.0.0.0:11434
OLLAMA_MODELS=/data/ollama/models
OLLAMA_NUM_PARALLEL=2
OLLAMA_MAX_LOADED_MODELS=2

Bind 11434 hanya jaringan internal (10.100.1.0/24). Jangan expose ke internet.

Keamanan

Item Requirement
Kredensial SMB NAS Hanya di .env server indexer
Ollama API Internal VLAN saja
Data sales / project RBAC per source_id
Audit Log setiap query + hasil count
Password NAS Rotasi jika pernah dibagikan di chat

Timeline rekomendasi

Tahap Durasi Deliverable
POC metadata 1–2 minggu Index project saja, chat di console
Multi-source +1 minggu Tambah Marketing/GIS via env
Fase 2 RAG 2–4 minggu pgvector + ekstrak dokumen
License extraction P1 +1–2 minggu license_registry + parser GM & ESRI
Migrasi GPU ~6 bulan Windows 11 + GPU, ganti OLLAMA_BASE_URL

Host referensi (repo)

Host IP Peran
QNAP NAS 10.100.1.10 SMB storage
Ollama VM 10.100.1.26 Ollama inference + indexer POC (aiopr)
Reverse proxy 10.100.1.24 Nginx, Docker, geonet-console
Main database 10.100.1.25 PostgreSQL + pgvector
geonet-console via .24 UI + API — consumer OLLAMA_BASE_URL

SSH Ollama: .\scripts\ssh-ollama.ps1 atau ssh ollama
SSH NAS: .\scripts\ssh-qnap.ps1
SSH reverse proxy: .\scripts\ssh-remote.ps1

Checklist sebelum go-live

Host 10.100.1.26

  • vCPU dinaikkan ke 16
  • OS dioptimasi (Server 22.04 atau tanpa GUI)
  • Ollama terinstall + systemd aktif
  • Model nomic-embed-text + qwen2.5:3b/7b ter-pull
  • OLLAMA_HOST bind internal; ufw allow 11434 dari 10.100.1.0/24
  • Folder /data/ollama/models siap
  • Mount SMB QNAP read-only (/mnt/qnap) untuk indexer POC
  • Indexer worker jalan (metadata project → PostgreSQL)

Stack keseluruhan

  • PostgreSQL pgvector + tabel project_index di .25
  • OLLAMA_BASE_URL=http://10.100.1.26:11434 di geonet-console (.env server .24)
  • Audit log pencarian aktif
  • RBAC source sensitif (Marketing/Sales)
  • Rencana migrasi Windows GPU terdokumentasi
  • Tabel license_registry + parser P1 (geonet_global_mapper_v1, esri_certificate_id_v1)
  • Index license folder project/2025–2026 terlebih dahulu

Referensi terkait

  • server/10.100.1.10_audit.txt — audit QNAP
  • server/reverse-proxy-audit.txt — audit reverse proxy
  • server/main-database-audit.txt — PostgreSQL .25
  • server/host.txt — daftar SSH host