Skip to content

RESEARCH_CONTEXT.md

Latest commit

 

History

History
307 lines (230 loc) · 14.5 KB

RESEARCH_CONTEXT.md

File metadata and controls

307 lines (230 loc) · 14.5 KB
Raw

EduSync: Research Context for Paper Generation

Purpose: This document provides technical context for a reasoning AI to generate a research paper. It focuses exclusively on ECE and AI logic—no auth or routing boilerplate.

1. Project Abstract

  • Name: EduSync
  • Core Concept: Edge-deployed Learning Management System (LMS) using:
    • Local Llama-3 (GPU) for quiz, summary, and flashcard generation
    • MediaPipe (CPU) for vision-based head pose estimation during quizzes
    • EasyOCR (CPU) for text extraction from PDFs/images
  • Key Innovation:
    1. Split-Compute Architecture: Vision runs on CPU to leave GPU free for Llama-3; both pipelines execute on the same edge node.
    2. Multimodal Engagement Tracking: Vision (head pose during quizzes) + Behavior (scroll velocity during material viewing). Both streams logged with timestamps and user identifiers for correlation analysis.

Research objectives (for capstone and paper)

  1. Objective 1 — Edge AI performance and vision-based attention: Demonstrate measurable inference performance (latency, throughput) for OCR and LLM operations, and vision-based head-pose attention monitoring with summary statistics (Table 2) and time-series/distribution figures.
  2. Objective 2 — Multimodal engagement and correlation with quiz performance: Demonstrate that scroll engagement (DSP) and vision-based attention (focused ratio) correlate with quiz score across users/sessions; show scatter plots with Pearson r and p-values (N ≥ 20–30 pairs).

Figures (paths and one-line captions)

All under backend/research_data/figures/ (generated by backend/scripts/visualize_research_data.py):

Path Caption
engagement_vs_quiz_scatter.png Scatter plot of average engagement score vs quiz score with linear fit and Pearson r.
engagement_distribution.png Histogram of engagement scores across sessions.
latency_by_operation.png Bar chart of latency (avg or p50 ms) per operation (OCR, summary, flashcards, quiz).
attention_timeseries.png Head pose (yaw and pitch) vs time for a sample quiz session.
attention_distribution.png Histograms of yaw, pitch, focused ratio per student, and attention state distribution.
attention_vs_quiz_scatter.png Scatter plot of focused ratio vs quiz score with linear fit and Pearson r.

2. System Architecture (Technical)

Split-Compute Strategy

The edge node runs two heavy workloads simultaneously:

  • LLM Pipeline (GPU): Llama-3 8B Q4 quantized handles quiz/summary/flashcard generation. GPU is fully allocated (n_gpu_layers=-1).
  • Vision Pipeline (CPU): MediaPipe Face Mesh + OpenCV solvePnP for head pose. Runs on CPU to avoid GPU contention.
  • OCR Pipeline (CPU): EasyOCR extracts text from documents. No GPU used.

Result: Students can take quizzes (vision attention monitoring) while the teacher triggers AI content generation without resource conflicts.

Hardware Constraints (Optimized For)

  • Target: Laptop edge node (e.g., RTX 3060 6GB + Ryzen 7 5800H)
  • Llama-3 load: n_ctx=4096, n_batch=512, use_mlock=True, n_threads=8
  • Vision: MediaPipe CPU, min_detection_confidence=0.5
  • All processing local—no cloud round-trip; enabling low-latency on-device inference.

Models Used

Model Type Hardware Purpose
Meta-Llama-3-8B-Instruct-Q4_K_M.gguf Quantized (Q4_K_M) GPU Quiz, summary, flashcards
MediaPipe Face Mesh Pretrained CPU 2D face landmarks for head pose
EasyOCR (en) Pretrained CPU OCR for PDFs/images

3. Key Algorithms (Code Snippets)

3.1 Vision Pipeline (vision_service.py)

3.1.1 Moving Average Smoothing Filter

Per-student buffer of last N frames (default 10). Thresholds applied to averaged yaw/pitch to reduce false positives from single-frame noise.

class HeadPoseSmoother:
    def update(self, yaw: float, pitch: float, student_id: str) -> dict:
        buf = self._buffers[student_id]
        buf.push(yaw, pitch)
        n = len(buf.yaw_buf)
        yaw_avg = sum(buf.yaw_buf) / n if n else yaw
        pitch_avg = sum(buf.pitch_buf) / n if n else pitch
        if abs(yaw_avg) > YAW_THRESHOLD or abs(pitch_avg) > PITCH_THRESHOLD:
            attention_score = 0   # distracted
        else:
            attention_score = 100 # focused
        return {"yaw_avg": yaw_avg, "pitch_avg": pitch_avg, "attention_score": attention_score, ...}

3.1.2 solvePnP and Euler Angle Extraction

6-point 3D face model (nose, chin, eye corners, mouth corners) mapped to 2D MediaPipe landmarks. solvePnP yields rotation vector; Rodrigues + arctan2 yield yaw, pitch, roll in degrees.

# Generic 3D face model (mm)
FACE_3D_MODEL = np.array([
    (0.0, 0.0, 0.0),           # Nose tip
    (0.0, -330.0, -65.0),      # Chin
    (-225.0, 170.0, -135.0),   # Left eye left
    (225.0, 170.0, -135.0),    # Right eye right
    (-150.0, -150.0, -125.0),  # Left mouth
    (150.0, -150.0, -125.0),   # Right mouth
], dtype=np.float64)

# solvePnP
success, rvec, tvec = cv2.solvePnP(
    FACE_3D_MODEL, points_2d, camera_matrix, dist_coeffs,
    flags=cv2.SOLVEPNP_ITERATIVE,
)
yaw, pitch, roll = _rotation_vector_to_euler(rvec)

# Euler from rotation matrix (convention: yaw Y, pitch X, roll Z)
def _rotation_vector_to_euler(rvec):
    rmat, _ = cv2.Rodrigues(rvec)
    sy = np.sqrt(rmat[0,0]**2 + rmat[1,0]**2)
    yaw   = np.degrees(np.arctan2(rmat[1,0], rmat[0,0]))
    pitch = np.degrees(np.arctan2(-rmat[2,0], sy))
    roll  = np.degrees(np.arctan2(rmat[2,1], rmat[2,2]))
    return yaw, pitch, roll

3.1.3 Attention Rule

  • |yaw| > 20° OR |pitch| > 15° → distracted (attention_score = 0)
  • Else → focused (attention_score = 100)

Applied to smoothed values when student_id is provided.

3.2 Multimodal Logic (signal_processor.py)

3.2.1 Scroll Engagement Pipeline

Scroll velocity sampled at 1 Hz (pixels/second). FIR moving-average filter, then band-pass classification:

  • Idle: ≤ 2 px/s
  • Reading: 2–100 px/s
  • Skimming: > 100 px/s

3.2.2 Engagement Score Formula

# Base score from reading ratio (0–60)
base_score = reading_ratio * 60

# Energy bonus (0–25), normalized
energy_normalized = min(energy / 1000, 1.0)
energy_bonus = energy_normalized * 25

# ZCR bonus (0–15), optimal ZCR ≈ 0.3
optimal_zcr = 0.3
zcr_factor = 1 - abs(zcr - optimal_zcr) / optimal_zcr
zcr_bonus = zcr_factor * 15

# Final Scroll Engagement Score (0–100)
engagement_score = base_score + energy_bonus + zcr_bonus
engagement_score = clamp(engagement_score, 0, 100)

3.2.3 Fusion with Vision Data

  • Vision: attention_log.csv — per-frame head pose, student_id, timestamp_iso, attention_state (0/1).
  • Scroll: engagement_metrics.jsonl — per-session scroll signal, user_id, scroll_engagement_score, timestamp.
  • Correlation: Join by student_id = user_id and align by time for multimodal analysis (e.g., scatter: focused_ratio vs scroll_engagement_score).

3.3 LLM Pipeline (llm_service.py)

3.3.1 Prompt Engineering

Llama-3 Instruct format: <|start_header_id|>system<|end_header_id|>...<|eot_id|><|start_header_id|>user<|end_header_id|>...<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Quiz (5 MCQs):

prompt = f"""<|start_header_id|>system<|end_header_id|>
You are a strict JSON generator. Output exactly 5 multiple-choice questions in a list based on the text provided.
Do not include any text outside the JSON. Keys must be exactly: "question", "options", "correct_answer".
"options" must be a list of 4 strings. "correct_answer" is the index (0-3).
<|eot_id|><|start_header_id|>user<|end_header_id|>
Context: {content_text[:6000]}
<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
output = llm(prompt, max_tokens=512, temperature=0.4, stop=["<|eot_id|>"])

Summary:

prompt = f"""<|start_header_id|>system<|end_header_id|>
You are an educational assistant. Summarize the following study material as key bullet points that are easy to learn.
Use concise bullet points only, not paragraphs. Cover main concepts and important details.
<|eot_id|><|start_header_id|>user<|end_header_id|>
{content_text[:8000]}
<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
output = llm(prompt, max_tokens=512, temperature=0.5, stop=["<|eot_id|>"])

Flashcards:

prompt = f"""<|start_header_id|>system<|end_header_id|>
You are a strict JSON generator. Output exactly 5 valid JSON objects in a list.
Each object must have exactly: "front" (question/term) and "back" (answer/definition).
<|eot_id|><|start_header_id|>user<|end_header_id|>
Context: {content_text[:8000]}
<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
output = llm(prompt, max_tokens=512, temperature=0.4, stop=["<|eot_id|>"])

4. Data Schema & Samples

4.1 attention_log.csv

Header (current/new format):

timestamp,timestamp_iso,student_id,yaw,pitch,roll,attention_state,assignment_id

Sample rows (mock + real-like):

timestamp,timestamp_iso,student_id,yaw,pitch,roll,attention_state,assignment_id
1771165319397,2026-02-14T10:21:59.397Z,6,-8.08,8.81,172.01,1,1
1771165321254,2026-02-14T10:22:01.254Z,6,-8.37,7.45,169.53,1,1
1771165323367,2026-02-14T10:22:03.367Z,6,-8.47,9.23,170.19,1,1
1771165731463,2026-02-14T10:28:51.463Z,10,-176.16,-21.19,-147.09,0,1
1771165733433,2026-02-14T10:28:53.433Z,10,151.36,-40.43,-147.73,0,1
1771165735535,2026-02-14T10:28:55.535Z,10,179.16,21.13,-156.03,0,1
1771165739475,2026-02-14T10:29:39.475Z,10,-6.48,12.24,152.97,1,1

Note: attention_state 1 = focused, 0 = distracted.

4.2 inference_metrics.jsonl

Fields per line:

timestamp, operation, duration_ms, duration_s, input_chars, output_chars, chars_per_second, model, gpu_layers, platform

Sample rows (real):

{"timestamp": "2026-02-14T10:12:24.140422", "operation": "ocr", "duration_ms": 167.3, "duration_s": 0.167, "input_chars": 1769772, "output_chars": 14788, "chars_per_second": 88392.74, "model": "llama-3-8b-q4", "gpu_layers": -1, "platform": "edge"}
{"timestamp": "2026-02-14T10:15:05.325130", "operation": "summary", "duration_ms": 111667.16, "duration_s": 111.667, "input_chars": 14788, "output_chars": 1701, "chars_per_second": 15.23, "model": "llama-3-8b-q4", "gpu_layers": -1, "platform": "edge"}
{"timestamp": "2026-02-14T10:19:44.866193", "operation": "flashcards", "duration_ms": 101620.33, "duration_s": 101.62, "input_chars": 14788, "output_chars": 1226, "chars_per_second": 12.06, "model": "llama-3-8b-q4", "gpu_layers": -1, "platform": "edge"}
{"timestamp": "2026-02-14T10:21:28.201851", "operation": "quiz", "duration_ms": 99227.64, "duration_s": 99.228, "input_chars": 14788, "output_chars": 1207, "chars_per_second": 12.16, "model": "llama-3-8b-q4", "gpu_layers": -1, "platform": "edge"}

4.3 engagement_metrics.jsonl

Fields per line:

timestamp, user_id, material_id, engagement_score, session_duration_s, signal_length, sampling_rate_hz,
mean, std, max, min, energy, power, zero_crossings, zcr, dominant_freq_hz, spectral_centroid,
reading_ratio, idle_ratio, skimming_ratio, base_score, energy_bonus, zcr_bonus, final_score,
filter_type, filter_order, scipy_available, scroll_engagement_score

Sample rows (real):

{"timestamp": "2026-02-14T10:17:55.080668", "user_id": 3, "material_id": 1, "engagement_score": 39.73, "session_duration_s": 196, "signal_length": 196, "sampling_rate_hz": 1.0, "mean": 606.59, "std": 2205.5, "max": 26235.0, "min": 0.0, "energy": 5232205.51, "zero_crossings": 24, "zcr": 0.1231, "reading_ratio": 0.1429, "idle_ratio": 0.2806, "skimming_ratio": 0.5765, "base_score": 8.57, "energy_bonus": 25.0, "zcr_bonus": 6.15, "final_score": 39.73, "scroll_engagement_score": 39.73, "filter_type": "FIR_moving_average", "filter_order": 5}
{"timestamp": "2026-02-14T10:21:35.259098", "user_id": 3, "material_id": 1, "engagement_score": 59.66, "session_duration_s": 39, "signal_length": 39, "mean": 130.31, "reading_ratio": 0.3846, "idle_ratio": 0.1026, "skimming_ratio": 0.5128, "scroll_engagement_score": 59.66}
{"timestamp": "2026-02-14T10:35:09.782521", "user_id": 6, "material_id": 1, "engagement_score": 64.32, "session_duration_s": 451, "reading_ratio": 0.4146, "scroll_engagement_score": 64.32}
{"timestamp": "2026-02-14T10:42:00.343996", "user_id": 8, "material_id": 1, "engagement_score": 37.83, "session_duration_s": 116, "reading_ratio": 0.069, "skimming_ratio": 0.6207, "scroll_engagement_score": 37.83}
{"timestamp": "2026-02-15T14:28:39.160779", "user_id": 10, "material_id": 1, "engagement_score": 25.0, "session_duration_s": 4, "reading_ratio": 0.0, "skimming_ratio": 0.75, "scroll_engagement_score": 25.0}

5. Performance Metrics

5.1 Inference Times (from inference_metrics.jsonl)

Operation duration_ms duration_s input_chars output_chars chars_per_second
ocr 167.3 0.167 1,769,772 14,788 88,392.74
summary 111,667.16 111.67 14,788 1,701 15.23
flashcards 101,620.33 101.62 14,788 1,226 12.06
quiz 99,227.64 99.23 14,788 1,207 12.16

Note: OCR input is raw bytes (char count ≈ file size); LLM operations use text. chars_per_second = output_chars / duration_s.

5.2 Latency Percentiles (from latency_percentiles.csv)

operation count avg_ms p50_ms p95_ms min_ms max_ms
ocr 1 167.3 167.3 167.3 167.3 167.3
summary 1 111667.2 111667.2 111667.2 111667.2 111667.2
flashcards 1 101620.3 101620.3 101620.3 101620.3 101620.3
quiz 1 99227.6 99227.6 99227.6 99227.6 99227.6

5.3 Model Load Time (from system_metrics.jsonl)

event model_load_time_ms
model_load ~1,454 – 5,992

Typical range observed: 2,300–6,000 ms depending on cold start.

5.4 CPU/Throughput Notes

  • OCR: EasyOCR runs on CPU; ~88k chars/s throughput for text extraction.
  • LLM: Throughput ~12–15 chars/s for quiz/summary/flashcards on edge GPU (RTX 3060 6GB class).
  • Vision: MediaPipe Face Mesh runs on CPU; ~0.5 s per frame (2 Hz capture). No explicit CPU % logged; all vision logic is CPU-only.