Labsco
asume21 logo

webear and web perception

β˜… 1

from asume21

Give your AI real senses (audio, visuals, performance, network, security, and console logs) in real time from any running web application.

πŸ”₯πŸ”₯πŸ”₯βœ“ VerifiedPaid serviceAdvanced setup

webear

npm version npm downloads License: MIT MCP Compatible

Give your AI real senses β€” hear, see, and feel any web app.

An MCP server + browser SDK that gives AI coding assistants direct sensory access to a live web application. Audio, visuals, performance, network, security, and console β€” captured from the browser, analyzed in real time, delivered via MCP.

"The beat sounds muddy" β†’ your AI captures 3 seconds, measures the spectral centroid at 580 Hz with 45% energy below 250 Hz, and tells you exactly why.


AI Web Perception Demo


What It Does

ToolDescription
capture_audioRecord a short clip (500ms–30s) of what your web app is outputting right now
analyze_audioSignal analysis: RMS, peak dB, clipping, spectral centroid, frequency bands, BPM, timing jitter
describe_audioPlain-English AI description β€” "the kick is boomy with heavy sub buildup around 80 Hz"
diff_audioCompare two captures and flag what changed β€” loudness, tone, timing, clipping

How It Works

Browser (Web Audio API)
    ↓ MediaRecorder taps the AudioContext output node
    ↓ Uploads WebM blob via HTTP POST
Express Middleware (your dev server)
    ↓ Stores captures in memory, dispatches commands via SSE
MCP Server (stdio β€” runs inside your IDE)
    ↓ Retrieves captures, sends to CodedSwitch analysis API
AI Coding Assistant
    β†’ "Your bass band is 42% of the mix (high), spectral centroid
       is 580 Hz (muddy), and timing jitter is 23ms β€” the scheduler
       is drifting under load."

The key difference from every other audio MCP: this taps the Web Audio graph directly, bypassing room acoustics, microphone hardware, and the need to export files.


Example Output

analyze_audio

── Audio Analysis Report ──────────────────────────────
Duration:          3.02s

── Loudness ─────────────────────────────────────────
RMS:               -12.4 dBFS
Peak:              -1.2 dBFS
Dynamic range:     11.2 dB
Crest factor:      3.63
Clipping:          none

── Tone ──────────────────────────────────────────────
Spectral centroid: 2847 Hz
DC offset:         0.00012 (ok)

── Frequency Bands ───────────────────────────────────
Sub  (20-80 Hz):   8.2%
Bass (80-250 Hz):  22.1%
Mid  (250-2k Hz):  38.4%
Hi-mid (2-6k Hz):  21.8%
High (6k+ Hz):     9.5%

── Rhythm ────────────────────────────────────────────
Estimated BPM:     92
Onset count:       12
Timing jitter:     4.2 ms std dev

── Summary ───────────────────────────────────────────
Loudness: -12.4 dBFS RMS, peak -1.2 dBFS. Tone: balanced (centroid 2847 Hz).
Band mix β€” sub: 8% | bass: 22% | mid: 38% | hi-mid: 22% | high: 10%.
Rhythm: estimated 92 BPM, 12 onsets detected. Timing: very tight (< 5 ms jitter).

diff_audio

── Audio Diff: a1b2c3d4… β†’ e5f6g7h8… ──

── Loudness ──────────────────────────────────────────
  RMS: -14.2 dBFS β†’ -12.4 dBFS  (+1.8 dBFS)
⚠ Peak: -3.1 dBFS β†’ -0.2 dBFS  (+2.9 dBFS)
⚠ CLIPPING INTRODUCED β€” gain staging regression

── Tone ──────────────────────────────────────────────
⚠ Spectral centroid: 2847.0 Hz β†’ 1920.0 Hz  (-927.0 Hz)

── Interpretation ────────────────────────────────────
A gain bug was introduced that causes clipping.
Tonal character changed noticeably β€” EQ or filter behaviour may have shifted.

Who Is This For?

  • Web Audio / Tone.js developers β€” debug beats, synths, effects, and mixing without leaving your IDE
  • Game audio developers β€” verify sound effects, spatial audio, and mixing in real-time
  • Music app builders β€” catch regressions between code changes with diff_audio
  • Podcast / streaming apps β€” validate audio quality, levels, and encoding
  • Anyone whose app makes sound β€” if it has a Web Audio graph, your AI can now hear it

Why Not Just Use the Microphone?

Microphone MCPs capture room sound β€” your fan noise, chair creaks, and room reverb are all in the recording. webear taps the Web Audio API before it hits the DAC, giving you a clean digital signal with no room artifacts.


Web Perception β€” Full Sensor Suite

WebEar started as audio-only. Web Perception expands it to 6 senses:

SensorWhat it perceives
WebEarAudio β€” mix quality, rhythm, instruments, clipping
WebEyeVisual β€” canvas, UI layout, animations, screenshots
WebSensePerformance β€” frame rate, memory, audio latency
WebNerveNetwork β€” API latencies, connection quality, storage
WebShieldSecurity β€” cookies, storage exposure, CSP, framing
WebLogConsole β€” logs, warnings, errors, uncaught exceptions

Install the full browser SDK

import { WebPerception } from 'webear/perception'

WebPerception.init({
  apiKey: 'wbr_YOUR_API_KEY',
  relayUrl: 'https://www.codedswitch.com',
  sensors: ['ear', 'eye', 'sense', 'nerve', 'shield', 'log'],
})

Or use a single sensor:

import { WebEar } from 'webear/perception'

WebEar.init({
  apiKey: 'wbr_YOUR_API_KEY',
  ear: { audioContext: myCtx, audioNode: masterGain },
})

Connect via MCP (hosted relay β€” no local server required)

{
  "mcpServers": {
    "webear": {
      "url": "https://www.codedswitch.com/api/webear/mcp/sse",
      "headers": {
        "Authorization": "Bearer wbr_YOUR_API_KEY"
      }
    }
  }
}

Available MCP Tools

SensorToolCreditsDescription
Earcapture_audioFreeRecord live tab audio
Earanalyze_audio1BPM, loudness, frequency bands, clipping, dynamic range
Eardescribe_audio2AI plain-English description β€” instruments, genre, mood, mix notes
Eardiff_audio1Compare two captures β€” loudness, tone, timing deltas
Eargroove_score2Grid alignment, swing factor, consistency (0–100%)
Earcapture_and_analyze1Capture + analysis in one call
Earmix_coach3Structured mixing feedback
Eyecapture_videoFreeRecord canvas/video from the tab
Eyedescribe_video2AI visual description β€” layout, colors, bugs
Eyediff_visuals2Compare two visual captures
Sensecapture_telemetryFreeFPS, memory, layout shifts, audio latency
Senseanalyze_telemetry1Frame drops, memory pressure, audio underruns
Nervecapture_nerveFreeAPI timings, connection quality, storage size
Nerveanalyze_nerve1Slow APIs, connection quality, storage bloat
Shieldcapture_shieldFreeCookies, CSP, storage exposure, framing
Shieldanalyze_shield1CORS issues, non-HttpOnly cookies, missing CSP
Logcapture_logsFreeConsole output + uncaught exceptions
Loganalyze_logs1Error patterns, stack traces, repeated warnings

Get an API Key

Sign up at codedswitch.com β†’ Settings β†’ WebEar. Keys start with wbr_.


Author

Built by @asume21 β€” CodedSwitch