International Csound Conference 2015

BRAINGAME – an  interactive sound-game for players with brainwave (and other) sensors and international Csound ensemble, participating over internet

Author: Tarmo Johannes tarmo@otsakool.edu.ee

Idea: to connect different distances - people being very far and neuro-cells working inside us; to unite the electricity in our bodies with the bits and bytes in computers to make music together.

How it works:

Three players on stage wear Neurosky headsets that measure brain activity and attention level, two have heart rate sensors. The data is sent to computer and forwarded to running Csound instance via channels.

Csounders from any part of the world can participate over internet – using a web based interface (built on Csound pnacl module of Google Chrome). They can send new Csound code to the Csound server on spot to be compiled live, start or stop their (previously) written instruments or redefine their constants.

The Csound instruments written for the performance must  use the channels of the sensor data. The players with the sensors can try to control the process (somewhat) with their mental or physical activity but they have no power  over what sounds come actually out.

Channel names and hints for instruments:

1) data from BRAIN SENSORS:

- attention1, attention2, attention3 (values are scaled to 0..1, updated every second);

From other data I use only the strength of high and low beta waves:

- hb1 , hb2, hb3; lb1,lb2,lb3 (scaled 0..1, updated once per second)

- raw data from the brain activity (constantly changing curve, mostly around -1000...1000, sometimes more, sent 512 times per second): raw1, raw2, raw2

Thus player 1 sends attention1, hb1,lb1 and raw1, player 2 attention2 etc.

Attention is the only parameter that player can control consciously, over 0.7 is relatively high attention. Beta waves vary pretty much and undulate up and down. Take them as a biological jspline :)

NB! these channels are sent only once every second. Use port opcode if you don't want a very stepped sound.

raw value is basically the electricity that the sensor picks up. It moves quickly back and forth, kind of ever-changing and quite chaotic line. Can sound cool.

2) HEART RATE.

One or two performers will be wearing heart-rate sensors. Their current pulse is sent via a smartphone application to the computer that settles channels heart1 and heart2. The values may probably fall into

a range approximately 80..200 beats per minute. The rate is updated after every couple of seconds.

Next to that there are two ticking metronome signals (gkHeartBeat1, gkHeartBeat2) according to the heart rates.

Very simple sample instrument:

gkLevelTarmo init 1 ; use one global variable for level

gkratio init 1.25 ; use some gk-variables that are easy to change (send init gkratio 1.234 for example)

; schedule "Tarmo",0,-1 ; compile it to turn on

; schedule -nstrnum("Tarmo"),0,0 ; turn off

instr Tarmo

aenv linenr 0.01,0.1,3,0.001 ; use an envelope with release

kattention chnget "attention1" ; 0..1

kamp port kattention, 0.25 ; make amplitude depend on attention

kfreq = 100+10 * chnget:k("heart1") ; heart rate channel

a1 poscil aenv*kamp, kfreq

a2 poscil aenv*kamp, kfreq*gkratio

aout = (a1+a2)/2*gkLevelTarmo

outs aout, aout

endin

Some suggestions:

- would be nice if the instrument included your name, but it does not have to

- use envelopes with release (like madsr, linenr etc) to make sure the endings are graceful

- start and stop the instrument with sending according schedule, event_i or scoreline_i command (see example)

- You can use the instruments with indefinite length (negative p3), but calls with definite duration are naturally fine as well.

- there should be a global variable gkLevelYourInstrument that enables to control the overall mix a bit.

- use global gkVariables to set some parameters of your sound. It is easiest to change something with sending just the new "gkVariable init <new value> line to server. The variable names should not be too common so you can be sure that other people will not override your, say, gkfreq

- read the sensors from according channels. use port wherever necessary

- you can write (or send in runtime) also several instruments.

User interface

… is web-based, using pnacl module for csound and websockets for communication. It is up now on:

http://tarmo.uuu.ee/varia/failid/cs/nacl/braingame.html

It is put together very quickly, it is unpolished but works.

It is based on Victor Lazzarini's example from the pnacl collection, interactive.html

There you can write your Csound code or load it from file. You can compile the code and listen to it and send selected text to the Csound server to be compiled.

Your sent code is displayed in the hall to big screen like "Tarmo says: gkLevel init 0.5" and the Csound's result "Csound says: OK" (or "Csound says: Error").

The messages are echoed also int the console down on the page.

You can send also messages to other players like "Are you ready?"  and messages to the screen in performance hall (non-csound messages like "Hi, Tarmo").

When the server program runs, the page connects automatically to the server via websocket protocol and the "Connect" button gets grayed out - it means you are connected and ready the server waits for your input (see also messages in console).

You need Google Chrome (in computer) to use the csound pnacl module. The websocket side of the UI works also in other browsers (so you can connect to the server and send messages), but you cannot make any local experiments, if you need.

Feedback

How do you know how it sounds all together?

Use live-jam software Jamulus  http://llcon.sourceforge.net/index.html  

The Csound server that makes the sound in performance sends its sound also to the central Jamulus server < to be found out still >. The latency is small, so you can hear the sound of the performance almost instantly.

Start Jamulus, enter your name in settings, connect to central server <the address>, you should see other internet players connected to that as well.  Turn down your input level (or everybody will hear your typing) and enjoy!

The source code is published on github https://github.com/tarmoj/braingame

Questions? tarmo@otsakool.edu.ee

Thank you!

tarmo