- 2013
- Feb
- 15
Announcing: ImageDB
ImageDB is corpus-based image retrieval for OpenFrameworks/OpenCV. It was developed on OS X, but there is no reason why it cannot be used for Linux or Windows.
At this point, I do not have time to do much with this software. I will attempt to make some documentation and tutorials/examples later this year. Get in touch with me if you have any questions, or need help getting up and running.
- 2012
- Nov
- 15
The Beginnings of ACTION
I am pleased to announce that the first bits of ACTION have been published here. Thus far, the code consists of two classes that analyze color histograms and optical flow for frames of video. Much more to come in the weeks and months ahead.
- 2012
- Nov
- 10
soundScape Toolkit: GPSON Guide
A crash course on JSON notation and its use in the RuebGPS project…
The files we are using are adapted from JSON files. JSON = JavaScript Object Notation. It is similar to xml, but a lot simpler. GPSON = GPS Object Notation. Both JSON and GPSON are plain text files (although your operating system will probably not recognize .gpson). They can be edited in any text editor; it is easiest to work with them using a mono-spaced font like Andale Mono, Courier, etc. I use a free program called TextWrangler for this and other coding, because the program does not try to format everything like Word or TextEdit sometimes do.
White space makes no difference to the computer. The parser can read the file no matter what spaces, tabs, and carriage-returns you use. There are 2 fundamental structures in JSON: the dictionary and the array. A dictionary is a key-value pair, where the key has to be a string, and the value can be anything. An array is an ordered list, where each slot can be anything. Since values and slots can contain anything, dictionaries and arrays may be infinitely nested.
There are only 5 types of punctuation that are used to structure the file:
{}
curly brackets denote dictionaries (a.k.a. hash tables)
[]
straight brackets denote arrays
,
commas separate elements in arrays
:
colons separate keys and values in dictionaries
""
quotation marks surround strings; ALL keys and ALL strings are surrounded by quotes, numbers that are not strings are not quoted
Every file starts and ends with curly brackets. Thus the whole file is one big dictionary. Inside this dictionary, there are only 2 keys. These keys divide the document into regions and sound files.
{
"bucket_name" : "my_awesome_bucket",
"regions" : {
}
}
There are 3 kinds of regions, circles, rectangles and polygons. Polygons are not implemented in the app yet, but should be soon. Every region, no matter the shape is a hash table within a hash table. The keys are integer strings, because keys always have to be strings.
The general pattern is this; note that the numbers are region numbers and are have no leading zeros:
"0" : { ... }
This is a shape with index 0, and all the information about the shape will go where the … is. To illustrate this, the following are three shapes with their parameters filled in. Thus, every region is a dictionary. You should now notice that the entire JSON/GPSON file is simply one big nested structure.
CIRCLES
"0" : {
"sides" : 0,
"centrad" : [ 48.0000, -78.0000, 0.0002 ],
"sfIDs" : [ 12, 1000, 1000 ]
}, // etc...
A shape with zero sides is considered a circle.
centrad stands for center + radius, [lat, long, radius], where radius is expressed in the same units as lat and long, namely fractions of a degree. sfIDs is an a array with the following format: sound file index that maps to this region, fade-in time, and fade-out time. (Note: For the time being, there is a limit of one sound file per region).
RECTANGLES
"1" : {
"sides" : 4,
"vertexes" : [[ 42.916100, -78.885000 ],
[ 42.916100, -78.875000 ],
[ 42.916500, -78.885000 ],
[ 42.916500, -78.875000 ]],
"sfIDs" : [ 1, 1000, 1000 ]
}, // etc...
A shape with exactly 4 sides is a rectangle (we will have to do things a bit different here to support 4-sided polygons…).
vertexes are the 4 vertex points. This is an array with 4 arrays inside it. Each internal array is a point and is laid out like this: [lat, long]. Pay close attention to the punctuation! sfIDs is the array with mapped sound file, the fade-in time, and the fade-out time.
POLYGONS (*not yet implemented)
"2" : {
"sides" : 5,
"vertexes" : [[ 42.916500, -78.885000 ],
[ 42.916500, -78.875000 ],
[ 42.916900, -78.885000 ],
[ 42.917100, -78.880000 ],
[ 42.916900, -78.875000 ]],
"sfIDs" : [ 2, 1000, 1000 ]
} // etc...
This polygon shape should look like a house. This should be straight-forward. The hardest part is dealing with the numbers as text. For polygons, the point ordering is important, but it does not matter which point is first. For rectangles, the ordering is not important.
Other attributes
All the regions must have sides, vertexes, and sfIDs. They can have several other attributes that are optional. If these optional attributes are missing, default values are used.
- loops: [ NUMBER-OF-LOOPS, END-ON-EXIT FLAG ] There should be at least one loop. If the end-on-exit flag is 1, this sound fades out as soon as the user leaves the region. Otherwise, if the flag is 0, then the sound continues to play until it completes. In the case where you have a number of loops specified with the end-on-exit flag set to 0, only the current playing loop is allowed to complete (a new loop iteration will not begin without re-entering the region).
- lives: [ NUMBER-OF-LIVES ] The max number of times that this region-sound file pair can be played in the composition. The default is 99 lives.
There are a few other attributes that will be added in the near future. These will be described here as well.
- in Programming+Patching, iOS, Other
- 2012
- Nov
- 10
soundScape Toolkit: Creating a soundScape Toolkit composition
Please note: this is a guide mainly intended for students in Teri Rueb’s graduate seminar at UB. I [Tom] will endeavor to keep this document as current as possible with respect to the latest version of the iOS library/app.
We are making steady progress making the soundScape Toolkit ready for public release. If you are interested in trying the software, let us know, and we will give you access to the GitHub code. The following is an overview of how one would develop a soundScape composition. (For now we will assume that you are using the ad hoc distribution, the one that you have to join our development team as a tester to get.)
You will need the following:
1. An iOS device with GPS capability: an iPhone or an iPad with 3/4G. You do not necessarily need a data plan for the iPad to test, though. The location-tracking hardware seems to work pretty well without an account. If you are interested in testing the private-beta version, email tms@kitefishlabs.com.
2. An Amazon AWS account. You will upload MP3 sound files to AWS that will then be downloaded into the app.
3. A text editor, preferably one that can edit plain text such as TextWrangler, Sublime, or TextEdit.
The steps:
(1A. Developers only: download the source code and get set up in xCode.)
1B. Non-developers: get a testflightapp.com account and get in touch with us (see above) to get access to the ad hoc app.
2. Establish your testflight account and download the ad hoc app. (Email us for directions.)
3. Create an account at aws.amazon.com and then create a bucket.
4. Upload sounds to that account. Name them using two-digit numbers: 00.mp3, 01.mp3, … 99.mp3.
(5. Developers only: edit your Constants.h file to include the correct keys that you got from your Security Credentials page in your Amazon account.)
6. Create your GPSON file, and include the correct bucket name. (See separate tutorial.)
7. Email the GPSON file to an account that you can access on your phone, then load that file into the soundScape app from Mail on the phone.
8. If everything is set up properly, then the app should download the sounds from your bucket (do this over WIFI if possible) and launch your composition. Test/experience your piece.
- in Other
- 2012
- Oct
- 28
Collaboration announcement/update
If you were at ISEA 2012, you may have already seen or experienced this. If you are in the Santa Fe area, you can still experience this (see the link)…
I programmed the app that underlies No Places with Names, a large soundscape composition composed by Teri Rueb et al. The underlying software allows a composer to map sound files to playback when a user with a smartphone is in a particular location. I will be porting the software to Android this winter and adding functionality to both the Android and iOS versions as I go.
An open source release of the underlying software will happen within the next few months. Stay tuned, and email me if you’re interested in finding out more!
- 2012
- Jun
- 18
New Job + New website (sort of)
Starting next week, I will be working at the Bregman Digital Music Studio at Dartmouth College. There I will be working on several interesting projects, including implementing cinematic information retrieval tools, my MIR composition and software, and other things. I imagine that I will have quite a bit to write about soon.
First up: by the end of June, version 1.0 of my corpus-based processing classes for SuperCollider, with documentation and the beginnings of a python parallel version.
Also note that my site is now called kitefishlabs.com. kitefishlabs was my company name for distributing (paid) iOS apps, but now it will be my general research/software site, which will have links for any and all iOS apps that I release. Blobsynth is being updated and rereleased as a free app.
- in Other, SuperCollider, Python
- 2012
- Jan
- 17
RAAR Residency Update
Just a few notes on the residency project that I am working on:
The premiere is set for March 10, 2012 7PM @ Squeaky Wheel!
I am developing software that is very similar to my corpus-based tools for music/audio, but now applied to images. This code uses Openframeworks and ofxOpenCV. I will be posting a lot of this to github as soon as I get it working completely.
I’m also looking forward to the works by my three co-residents. More on that closer to the time. For now you can see some information at Squeaky’s residencies page.
- in Events, CBPSC, SuperCollider, C/C++, OpenFrameworks
- 2012
- Jan
- 17
Documentation Round 2
Here are some more things that I have dug out of my hard drive…
ICMC 2009 poster I showed in Montreal
Presentation slides for TES 2010 in Toronto
- in
- 2011
- Oct
- 5
Documentation Installment 1
As promised, here is some documentation…well, sort of…in the form of academic papers.
These papers were published as part of the last three ICMCs and cover various aspects of what is now two intertwined, ongoing projects, CBP for MaxMSP and CBPSC for SuperCollider.
2009 poster on CBP for MaxMSP (Conceptual)
2011 paper introducing CBPSC for SuperCollider
- in