A minimalist, music-focused mobile app that detects and visualizes musical notes in realtime from the microphone, with an elegant piano-roll UI and classical/Western notation options.

Built with Expo + React Native. Realtime pitch detection across web and native using optimized, low-latency paths.

• Realtime pitch detection
  • Web: WebAudio + time-domain autocorrelation (no server)
  • Native: Custom module with zero-crossing frequency estimation for accurate, low-latency detection
• Elegant piano-roll visualization
  • Fixed vertical playhead near the right edge (≈90% width)
  • Track scrolls right-to-left with time
  • Full-height sheet; vertical drag scrolling
  • Alternating lanes, hairline grid, subtle shadows
  • Thicker separators at natural boundaries (e.g., C) for visual grouping
• Notation & labels
  • Default Indian classical (Sa, Re, Ga, Ma, Pa, Dha, Ni)
  • Western (C, D, E, F, G, A, B) and Solfege (Do, Re, Mi, etc.) switchable in Settings
  • Octave watermarks centered horizontally per saptak range:
    • Indian: Mandra (low), Madhya (middle), Taar (high)
    • Western/Solfege: Lower/Middle/Higher Octave
• UI/UX & Aesthetic
  • Serif typography, black/white palette
  • Generous whitespace, minimal lines, refined motion
  • Floating bottom recording controls (shifted above system nav)
  • Small current-note badge at top-right (no background)
• Library & Settings
  • Recordings list with rename, favorite, delete
  • Settings for notation, noise reduction sensitivity (UI), auto-scroll, instrument placeholder

• Core realtime visualization and UX are implemented.
• Web realtime pitch is live and accurate for clean input.
• Native realtime now uses a custom Expo module with zero-crossing analysis for precise frequency detection.
• MIDI editing, smart contextual prediction (✨), export, synth playback are planned next.

• Framework: Expo + React Native
• Audio:
  • Web: getUserMedia + AudioContext + ScriptProcessorNode + autocorrelation
  • Native: Custom Expo module (expo-audio-frequency) with AudioRecord and zero-crossing pitch estimation
• Visualization:
  • Custom piano roll renderer in React Native Views
  • Playhead fixed; notes positioned relative to "now"
  • Vertical drag to scroll across pitch lanes; optional auto-scroll to center current pitch
• State:
  • Event stream of detected MIDI notes with start/end (sliding time window)
  • Notation state toggles UI labels and top-right badge

Key runtime dependencies (see package.json):

• expo ~54
• expo-audio ~1.0
• react-native 0.81, react 19
• fft.js (available; web uses autocorrelation instead of FFT for latency/perf)
• @tonejs/midi (future MIDI export/playback)
• expo-dev-client (recommended for native mic recording in dev)
• wavefile (future: waveform processing/export)

Note: expo-av is currently listed but the project uses expo-audio. You can safely remove expo-av if unused in your fork.

1. Install

• Node 18+ recommended
• Install packages:

  npm install
  

2. Web (quickest)

• Start:

  npm run web
  

• Open in a secure origin (https) for mic on browsers that require it.
• Grant mic permission.

3. Native (recommended for realtime mobile)

• Install Expo Dev Client (one-time per device):

  npx expo run:android
  # or
  npx expo run:ios
  

  This builds the app with config plugins and enables mic access in development.
• Start Metro and launch:

  npm run android
  # or
  npm run ios
  

• expo-audio config plugin adds microphone usage string on iOS.
• iOS background audio/recording (optional): add to app.json (if desired for future background features):

  {
    "expo": {
      "ios": {
        "infoPlist": {
          "UIBackgroundModes": ["audio"]
        }
      }
    }
  }

• Main Screen
  • Realtime piano-roll visualization (full height)
  • Vertical pitch axis on left; fixed playhead near right edge
  • Small current-note label at top-right
  • Floating recording controls at bottom (elevated above device nav)
• Library
  • Cards with name, date, duration
  • Long-press to rename; favorite star; delete button
• Settings
  • Notation: Western / Indian / Solfege
  • Auto-scroll: centers the current pitch vertically
  • Noise reduction sensitivity (UI placeholder for future DSP tuning)
  • Instrument (future synth path)

• Native realtime detection now uses zero-crossing analysis in the custom module for better accuracy.
• Expo Go generally does not support mic recording without a dev build. Use "Expo Dev Client" via npx expo run:[platform].
• Web requires HTTPS for mic on many browsers; local dev works in Chromium via special allowances but recommended to use secure origin if deploying.
• Autocorrelation and zero-crossing work best for monophonic vocals/humming; heavy noise or polyphonic inputs reduce accuracy.

• Enhanced pitch detection algorithms (e.g., YIN or AMDF) for even better accuracy
• Smart contextual note prediction (✨) with scale/interval heuristics and revert option
• MIDI export and playback (piano roll to MIDI events)
• Synth playback (Tone.js on web; native wavetable/FM strategy)
• Drag-and-drop note editing in the piano roll (duration, pitch)
• Upload processing with background progress and autosave to library
• Animation refinements, screen transitions, button micro-interactions

• No mic on device in dev:
  • Build Dev Client: npx expo run:android (or ios) and re-run.
  • Ensure permissions: accept mic prompts, check Settings app if denied.
• Web mic not working:
  • Use https origin; check browser permissions; try Chrome/Edge.
• Scroll not working:
  • The sheet uses a vertical ScrollView; on some simulators the gesture conflicts with the system. Test on a physical device or adjust simulator settings.
• Performance:
  • Close other audio-heavy apps; keep input monophonic for best results.

• npm start – Expo start
• npm run android – Start on Android device/emulator
• npm run ios – Start on iOS simulator/device
• npm run web – Start web build (mic requires permission, ideally https)

• App.tsx – Main app (realtime detection, piano-roll UI, library, settings)
• app.json – Expo config (plugins/permissions)
• index.ts – App entry
• modules/expo-audio-frequency/ – Custom native module for Android frequency detection
• assets/* – Icons and splash

• Audio never leaves the device in this implementation. All detection/visualization is local.
• Be mindful of platform mic permission prompts and user expectations.

Designed for a refined music experience—clean typography, minimal lines, and a piano-roll that “plays” as you sing.