Michael Edgecumbe and I gave a presentation at the O’Reilly Strata Conference last month that takes stock of the myriad (and amazing) framework options available to creative coders and data visualization developers.
Both a video of the presentation and the slide deck are online. Thanks to everyone who attended and provided feedback. Also, the source code for Michael’s sample app in Processing is available on GitHub.
Lots of nice work at the ITP winter show this year. I still think a little more curation to prune the “hello world” projects wouldn’t hurt, but such is ITP.
Here are some personal favorites:
Zach Schwartz, Martín Bravo, Nara Kasbergen, Rachel Slaff, and Ruth Spencer
Brilliant mashup of Google Maps and the not-as-famous-as-it-should-be Your World of Text. Written world lets you leave persistent messages on a map in real time. I really think it needs an address search bar to make it easier to explore other parts of the world — and being able to zoom out and see the density of conversation around the globe would be amazing. Regardless, it’s a great idea and the execution is polished. Go try it!
Photo: Tom Igoe
A servo-mounted laser methodically draws text (tweets) on a phosphorescent screen. The glow-in-the-dark screen holds a record of the laser’s path. So nice to get away from flickering screens (even if we can’t get away from tweets).
Lia Martinez, William Jennings
A tiny catapult as an interface for cueing projected clips of bananas hitting a wall. Simple, but instantly gratifying.
Contextual Interface for Remote Gestural Input
Gavin Hackeling, Zena Koo
Luis Palacios, Kaitlin Till-Landry, and Peter Darche
An XY table that draws in a psuedo-zen garden sand pit. Instead of putting a pen or pencil on the business end of an XY table, SoZen’s creators use a strong magnet to drag a ferrous ball through the sand to build up a drawing. Making an XY table is a PComp rite of passage, but this is a great take on it and a nice use of magnetic indirection to keep the mechanism out of sight and mind.
A plaster sculpture slowly destroys itself from the inside out in the absence of attention. More art should behave like a suburban adolescent.
Finally, someone braves the ITP thesis archives. It’s the only real hope for deconstructing what, exactly, ITP is / was, while removing all excuses for creating the same projects over and over. Looking forward to seeing the evolution of Archived Visualizations next semester.
Animated Gif (Single Frame) – Eric Mika, 2011
It's a single frame animated gif.
The file was hand written, per the GIF file specification. Although the original specification accommodates basic animation, this particular file uses an extension of the original format created and first supported by Netscape 2.0. Netscape's addition remains out of the official spec — but seems to be the approach favored by most animated gifs around the web. While Animated Gif (Single Frame) may not follow CompuServe's canonical format to the byte, it's aligned instead with the vernacular of the real web. It renders, after all.
The above image is simply solid black, and won't appear to animate, but the file is structured in a way that identifies itself as a looping animation. Therefore, the parser rendering it in your browser is actually cycling between Frame 1 and Frame 1 every second, indefinitely.
It is intended as a “last work” for an exhausted medium.
47 49 46 38 39 61 00 02 56 01 80 00 00 00 00 00 FF FF FF 21 FF 0B 4E 45 54 53 43 41 50 45 32 2E 30 03 01 FF FF 00 21 F9 04 04 64 00 00 00 2C 00 00 00 00 00 02 56 01 00 02 FF 84 8F A9 CB ED 0F A3 9C B4 DA 8B B3 DE BC FB 0F 86 E2 48 96 E6 89 A6 EA CA B6 EE 0B C7 F2 4C D7 F6 8D E7 FA CE F7 FE 0F 0C 0A 87 C4 A2 F1 88 4C 2A 97 CC A6 F3 09 8D 4A A7 D4 AA F5 8A CD 6A B7 DC AE F7 0B 0E 8B C7 E4 B2 F9 8C 4E AB D7 EC B6 FB 0D 8F CB E7 F4 BA FD 8E CF EB F7 FC BE FF 0F 18 28 38 48 58 68 78 88 98 A8 B8 C8 D8 E8 F8 08 19 29 39 49 59 69 79 89 99 A9 B9 C9 D9 E9 F9 09 1A 2A 3A 4A 5A 6A 7A 8A 9A AA BA CA DA EA FA 0A 1B 2B 3B 4B 5B 6B 7B 8B 9B AB BB CB DB EB FB 0B 1C 2C 3C 4C 5C 6C 7C 8C 9C AC BC CC DC EC FC 0C 1D 2D 3D 4D 5D 6D 7D 8D 9D AD BD CD DD ED FD 0D 1E 2E 3E 4E 5E 6E 7E 8E 9E AE BE CE DE EE FE 0E 1F 2F 3F 4F 5F 6F 7F 8F 9F AF BF CF DF EF FF 0F 30 A0 C0 81 04 0B 1A 3C 88 30 A1 C2 85 0C 1B 3A 7C 08 31 A2 C4 89 14 2B 5A BC 88 31 A3 C6 8D FF 1C 3B 7A FC 08 32 A4 C8 91 24 4B 9A 3C 89 32 A5 CA 95 2C 5B BA 7C 09 33 A6 CC 99 34 6B DA BC 89 33 A7 CE 9D 3C 7B FA FC 09 34 A8 D0 A1 44 8B 1A 3D 8A 34 A9 D2 A5 4C 9B 3A 7D 0A 35 AA D4 A9 54 AB 5A BD 8A 35 AB D6 AD 5C BB 7A FD 0A 36 AC D8 B1 64 CB 9A 3D 8B 36 AD DA B5 6C DB BA 7D 0B 37 AE DC B9 74 EB DA BD 8B 37 AF DE BD 7C FB FA FD 0B 38 B0 E0 C1 84 0B 1B 3E 8C 38 B1 E2 C5 8C 1B 3B 7E 0C 39 B2 E4 C9 94 2B 5B BE 8C 39 B3 E6 CD 9C 3B 7B FE 0C 3A B4 E8 D1 A4 4B 9B 3E 8D 3A B5 EA D5 AC 5B BB 7E 0D 3B B6 EC D9 B4 6B DB BE 8D 3B B7 EE DD BC 7B FB FE 0D 3C B8 F0 E1 C4 8B 1B 3F 8E 3C B9 F2 E5 CC 9B 3B 7F 0E 3D BA F4 E9 D4 AB 5B BF 8E 3D BB F6 ED DC BB 7B FF 0E 3E BC F8 F1 E4 CB 9B 3F 8F 3E BD FA F5 EC DB BB 7F 0F 3F BE FC F9 F4 EB DB BF 8F 3F BF FE FD FC FB FB 6D FF 0F 60 80 02 0E 48 60 81 06 1E 88 60 82 0A 2E C8 60 83 0E 3E 08 61 84 12 4E 48 61 85 16 5E 88 61 86 1A 6E C8 61 87 1E 7E 08 62 88 22 8E 48 62 89 26 9E 88 62 8A 2A AE C8 62 8B 2E BE 08 63 8C 32 CE 48 63 8D 36 DE 88 63 8E 3A EE C8 63 8F 3E FE 08 64 90 42 0E 49 64 91 46 1E 89 64 92 4A 2E C9 64 93 4E 3E 09 65 94 52 4E C9 4A 01 00 21 FE 2E 41 4E 49 4D 41 54 45 44 20 47 49 46 20 28 53 49 4E 47 4C 45 20 46 52 41 4D 45 29 20 2D 20 45 52 49 43 20 4D 49 4B 41 20 32 30 31 31 00 3B
(Bytes responsible for animation in red.)
Flapping Toasters is an absurdist reconsideration of the classic screensaver Flying Toasters. You embody the toaster, flapping your arms to stay aloft, clapping to release toast, and tilting to roll away from danger. Created in collaboration with Michael Edgcumbe.
The screensaver is forbidden territory for those seeking to elevate the status of new media art. A screensaver is lowbrow, unserious, and fetishizes aesthetics over content. I’ve explored this hostility a bit further in another post. At the same time, early screensavers represent a raw, unpretentious play with the screen’s possibilities combined with a novel set of constraints. (E.g. a screensaver must have no static content, run for an unlimited duration, etc.)
Michael and I decided to embrace the kitsch, and compound the absurdity of the ever-criticized screen saver with the hype and widespread obsession with the Kinect. Beyond trying to show some love for the screensaver, we were conscious of how many camera + presence driven “interactive artworks” present an interaction model that reduces the participant to maniacal arm waving and overwrought gestures designed to satisfy a finicky computer vision algorithm. We wanted to poke some fun at this trope by creating an interaction model that explicitly demands that the participant is reduced to crazed arm-flapping.
Process / Implementation
We built the software in openFrameworks. Since we wanted skeleton tracking, we used the ofxOpenNI plugin instead of ofxKinect (since the latter only provides RGB-D data).
Detecting flaps was a bit trickier than expected, since there’s an issue of ranging, direction detection, correlating the animation to the arm position, direction reversal detection, etc. We ended up solving these issues with brute force by tracking velocity of the hand points, filtering heavily, and setting up a dynamic bounding system that would allow people with various degrees of enthusiasm / gesticulation to fly successfully.
We also added support for rolling in both directions. The player initiates a roll by tilting their arms in ether direction — we used an angle threshold on an imaginary line drawn between hand points to trigger it.
Life as a toaster would be unfulfilling without the opportunity to toast bread, so Michael implemented hand clap detection to trigger the release of golden-brown toast.
Ideally there would be some kind of system for recording sample motions, saving them, and then automatically firing an event when a similar motion is detected along with a confidence value. I’m sure this kind of thing exists on industrial-strength Kinect development platforms (and must have existed somewhere in the Wii’s development stack for ages). Would be great to have a similar library for OF or Processing… something for the to-do list.
OpenNI makes tracking multiple players relatively trivial, so we implemented support for up to 15 simultaneous flappers (OpenNI’s internal limit).
Tthe most recent version of the ofxOpenNI plugin includes support for OpenNI’s native hand tracking algorithms. They weren’t available during our initial development, so we used the raw skeleton data instead. These algorithms seem a bit more reliable for locating hands then the skeleton points alone, particularly when hands cross the torso, go out of frame, and so on — essentially, it uses a more generous set of inferences to give you the best guess about hand position.
Since flapping toasters is all about hand tracking, it sounds like this new feature would be great. So, I tried a quick reimplementation of Flapping Toasters using these points instead of the end of the lower arm joint. It turned out not to work with more than one player, since I couldn’t find a way to associate two hands with a single person. For example, if three hands are detected, I had no way of knowing which two belonged to one person, and to whom the third might belong. There might be a way, but it wasn’t immediately obvious from the API.
The graphics are scraped sprites from After Dark’s original Flying Toasters. We created PNGs from each frame of animation and re-implemented the basic design of the screensaver.
The final element, which was surprisingly the hardest to get right, was the physics simulation. It’s your basic gravity / acceleration / velocity simulation, but getting the power of a flap just right was tricky. It still doesn’t feel quite like I imagined. Flapping is a bit too much work, but it’s also too easy to flap all the way off the top of the screen. The solution to this probably involves spending a bit more times with the sliders to get the numbers just right, as well as taking into account the vigor of the flap. (E.g. use some combination of average arm velocity during the down-stroke and the relative size of the flap as a coefficient to the flap power value.)
After Dark’s version of flying toasters included two possible soundtracks — Wagner’s Ride of the Valkyries, and a slightly less compelling theme song with karaoke-style lyrics. Michael and I opted for Wagner, and rendered out a midi version of the song as a WAV for use in openFrameworks. I’d have preferred to make the recording using the midi synths from a Macintosh of the appropriate vintage, but we just couldn’t get access to one in time for the class deadline.
After flapping for a few minutes, most people ask what they’re trying to do. We haven’t implemented any kind of game logic — it’s just you, the toaster, and the joy of flap-induced flight. In one sense it’s nice to keep things minimal. But if we really wanted a game dynamic, there are a bunch of possibilities.
Some kind of collision detection between players could make things more interesting. You’re also given an unlimited supply of toast. Perhaps you could try to “catch” untoasted pieces of bread, carry them for an amount of time (toasting them in the process) before clapping to release. Carry them for too long, and you’ll burn the toast. (The horror!)
Download / Source code
The application was written in openFrameworks. The source code is available on GitHub. See the repository’s readme file for additional notes on development / compilation. A Mac OS binary is also included in the GitHub repo.
The remark I remember most from my undergraduate thesis critique came from a fellow student. A film major, I think.
“I’d like to have this as a screensaver!”
She meant it as a compliment, but at the time I took offense. I thought the screensaver was the epitome of digital banality. But in the last few years, I’ve come around a bit.
I think the late 80s / early 90s era screensavers now read as a refreshing double-negation of the recent 8 bit / animated gif fixation. (Although I’m not sure if there’s a way to revisit / reconsider the After Dark heyday without collapsing in a pile of insincerity.)
I can’t think of a course or context at ITP where calling something out as screensaver-esque would go over particularly well (although opportunities abound). The more general new media community seems divided on the aesthetic / contextual parallels between screen-based art and screensavers.
Some forsake the association, others mine the medium for nostalgic kitsch, and a few embrace the idea and use the screensaver as a means of distributing their work. I expected most to fall into the first category, but some Googling around on the topic suggests that NMSAS (New Media Screensaver Anxiety Syndrome) doesn’t run as deep as I initially expected.
Nevertheless, some choice quotes emerged from the sift. Sources are linked, and if anyone wants to comment / update their opinion — some of the quotes are years old — I’m game for a conversation.
Let’s open the floor with three quotes from Marius Watz:
“The curse of generative art: ‘So you make screensavers?’”
– Marius Watz
“To the frustration of many digital artists, screensavers have much in common with generative art. They often rely on some kind of ruleset to allow infinite animation and to avoid burning a single image into the screen because of repetition. Many classic screensavers use mathematical formulas like Beziers or Lissajous curves. But most screensavers are created by programmers, not designers, hence the bad reputation they have as cultural artifacts.”
– Marius Watz
“As for the complaint that generative art is simply decorative, fit only for screensavers or wallpaper patterns, it is hardly worth answering. Such a position would invalidate most of art history.”
– Marius Watz
“The other hurdle is the ‘screen saver’ comparison. Society has chosen to consider screen savers with very little regard — they are temporary visuals. Another challenge for the legitimacy of this type of art.”
– Steve Sacks
“Interactive art usually presents more abstract and complex concepts but has terrible interface / interaction. Ok so ‘that’s not the point of the work’ you say. But what irks me is that there is very little work that addresses this. As a result many people will dismiss much interactive art as just a screensaver or digital toy.”
– Tom Betts
“We don’t want it to look like an iTunes screensaver.”
– Matt Checkowski on the Opera of the Future
“This generative ‘art’ seemed better suited to screensavers or abstract desktops than canvases — a fact confirmed by Davis’s own Reflect app for the iPhone.”
– Daniel West on Joshua Davis (flamewar ensues)
“But for me it is just a screen saver, since there is no story.”
– Mauro Martino
“Neither [Casey Reas nor Sol Lewitt] creates interactive works per se, but they are touchstones for anyone interested in the algorithmic art as something other than a screen saver.”
– Joshua Noble
“The idea behind a work can sometimes be more compelling than what actually appears on the screen. And for viewers without a thorough grounding in technology — or advanced math — the most innovative visual programs can seem like little more than high-end screensavers.”
– Susan Delson
“Clouds also became a popular desktop and screen saver at some point. I read it on the Internet, like, “Here’s instructions on how to take this and make a screen saver.” I just surfed on it a while back. I was like, Wow, it probably would look nice on the desktop or whatever.”
– Cory Arcangel
My thesis project has undergone a major shift in the last week. I’m moving away from the post-apocalyptic pirate internet, and towards something completely different: A means of projecting content onto surfaces that makes the projection appear intrinsic to the surface.
Imagine a hand-held projector that you can sweep across a room, kind of like a flash light. As it moves, the projected content appears stuck to the wall, the floor, etc. For example, you could add something to the scene in a particular location — a bit of text, perhaps. After adding the text, you could sweep the projector to a different part of the wall. The text would appear to go out of view once it left the throw-area of the projector, but if you were to move the projector back towards the spot where you initially added the text, you would see the words come back into view. The words are stuck to their environment — the projection is just an incidental way of exploring the space and revealing its content. Two recent technologies make this a particularly ripe time for this project: The Kinect gives cheap 3D scene information, which can improve the quality of motion tracking and automate the projection mapping process. New pico-projectors that can run on battery power and weigh significantly less than their conference-table counterparts mean that carrying around and a projector and using it to explore a space is no longer an entirely ridiculous proposition. This whole idea, which I’m currently calling Thesis II (for personal reasons) will be written up in more detail soon.
Fronts of Inquiry
The creative challenge for the next twelve weeks is to conceive of and build an application that demonstrates the usefulness and creative possibilities of this tool.
The technical challenges are twofold. First, I need a way to track the relative motion between the projector and the projection surface (generally a wall) — I’ll refer to this as relative motion tracking. Second, I need a way to dynamically distort the projected image to match the geometry of the projection surface. This is similar in concept to projection mapping, except the projection surface isn’t static. I’ll call this dynamic projection mapping. The calculations for both of these steps need to happen in less than 20 milliseconds if the effect is going to work and feel fluid.
Other people are already working on dynamic projection mapping, and from a technical standpoint it’s both more familiar ground and less essential to the final project than relative motion tracking. Where projection mapping is “nice to have” and will contribute significantly to the quality of the project, the technology that the project depends on to work at all is dynamic motion tracking. So, this paper will focus on research into means of relative motion tracking, and which (if any) existing open-source projects could be adapted for this application.
At the most basic level, I need to find a way to take a camera feed and determine how content in the scene is moving. Traditionally, this is called camera tracking — a form of deriving structure from motion. The process goes something like this: First the software identifies feature points within each frame — these are generally areas of high contrast, which relatively easy to pick out algorithmically. On the next frame, the software finds another batch of feature points, and then does correspondence analysis between these feature points in the most recent frame and feature points in the last frame. From this information, the movement of the camera can be inferred. (e.g. if a feature point is at pixel [5, 100] in frame one, and then moves to pixel [10, 80] in frame two, we can guess that the camera shifted about [5, -20] between frames. It’s a bit more complicated than that, because of the parallax effect — points closer to the camera will appear to move more than points further away from the camera. The software can take this into account, and build a rough point cloud of the scene.
This process has applications in special effects and film / post-production. If you have a shot with a lot of camera movement, and you need to add an explosion to the scene, camera tracking gives exactly the information you need to position the explosion in a believable way from frame to frame. Because of this demand, there are a few über-expensive closed-source software packages designed to perform camera tracking reliably. Boujou, for example, sets you back about $10,000. There is, however, a free and open-source option called PTAM — Parallel Tracking and Mapping for Small AR Workspaces which can perform similar tracking.
The PTAM code seems like the right starting point for my own adaptation of this concept, but there are a few caveats that make me nervous about just how much of a head start the code will give me. First, PTAM and similar camera tracking software is designed for use on high-contrast two-dimensional RGB bitmaps — basic still film frames. In contrast, the grayscale depth map coming from the Kinect is relatively low contrast, and areas of high contrast are probably best avoided in the feature detection process, since they represent noisy edges between depths. I probably will not be able to use the Kinect’s RGB data, because it’s going to be filled with artifacts from the projection. Also, since the Kinect already gives us a point cloud, I don’t need any of the depth-calculation features from PTAM. Because of these issues, I will probably start work by skimming through the PTAM source code to get an idea of their approach to the implementation, and then seeing how PTAM behaves when fed the grayscale depth map from a Kinect. From there, I will probably start experiment a simpler feature extraction and tracking algorithms in Processing that make the most of the Kinect’s depth data. (This code would be destined for an eventual port to C++.)