Sketch 3 Updates

Hey ,

here are running updates for sketch 3.

Wednesday March 27,
Today we presented our idea to the class about our Sketch 3, "Where the Light Follows".  Here are some details about the project!

Title: 
- Where the Light Follows
Team Roles: 
- Anna & Kathy = dance / develop performance that complements the ideas of the show
- Doug and JJ  = build system to track and project color images onto Anna and Kathy
- Trish        =  make videos/textures/colors to be projected

Project Concept/ Scenario:
- The idea of the project is to watch the girls dance and play and spread their cheerful light to the world around them
- The girls will dance as colorful images will be projected onto them.
In the room will also be a large, solid box on the floor that will have dull colors
Every time the girls touch the box (touch, sit, stand, lay ect) the box will change to a slightly
brightet color
Each time the box will stay a little brighter for a little longer

Concept Research:
- past experience helped us decide what to add and what not to add


Technical Research/ Tech Diagram:
- http://www.youtube.com/watch?v=tAGnSrdOfyA
- http://www.youtube.com/watch?v=-wVq41Bi2yE
- http://www.youtube.com/watch?v=5MXwJpKToEY

- Tech we will be using
* multiple kinect sensors (use pics from jj's slides)
* these will extract the depth data to give us the silhouette (use pics from jj's slides)
* Use the silhouette data to map texture/color to bodies and box
* will use openGL/C++ and Ogre for the actual programming/ rendering
- has a more correct depth buffer (zbuffer) than processing
*

Schedule:
- 1st 2 weeks: Dancers work on their dance and tech peeps work on the system
- 3rd Week   : Start bringing the group together for final testing and final manipulation



Saturday March 30,
After several hours, JJ and I have successfully installed openCV and OGRE and are ready to begin working on the tech side of the project.

 Monday April 1,
JJ and I completed the a small step in the project.  We were able to correctly display a picture only on a silhouette and then used this progress to project the mapped visual into  the OGRE rendering system.  The progress can be seen in the videos below! The girls continued to work on their ideas for their dance and Trisha continued to work on the visual aspects of the project.  This progress can also be seen below!

 Wednesday April 3,
JJ and I began working on the actual projection of the images onto a human body.  This turned out much harder than we expected.  We are still working out the last few kinks but home to have it projecting correctly soon.  The videos below show some progress.  The girls continued to work on their ideas for their dance and Trisha continued to work on the visual aspects of the project. Looks like we are making some progress! Videos below.

Saturday April 6
JJ and I met in ETB today to work on the camera calibration between the Kinect and the projector.  As it turns out it is a much harder task to complete than we thought! But we have set up a plan for how we can determine the transformation matrix between the two fields of views so we are not giving up yet!

Sunday April 7,
Today I finished writing a small program to capture 2D points by a mouse click and writing those coordinates to a file for use in determining the transformation matrix.

Given two matrices, one that contains n number of 2D coordinates and another that contains n number of 3D coordinates, we can find their transform as such

                  x1  x2  ... xn                                               x1  x2  ... xn
Matrix P = y1  y2 ...  yn             and        Matrix K =   y1  y2 ...  yn
                  1    1  ...  1                                                z1  z2 ...  zn
                                                                                    1    1  ...  1
            P = Transform *K

Monday April 8,
Today JJ and I added our parts together to try and pull together the Kinect-Projector calibration.  We were very close to success but had a few little problems.  We will be addressing these more on Wednesday to work out the little kinks.  Here is a video explaining how the calibration works

Tuesday April 9,

Possible interactive art festivals to submit our work to!

1. Santa Fe International New Media Festival

More to be looked into and added later!

Wednesday April 10,

Today JJ and I made some pretty good progress on the technical aspect of the projection mapping.  However, due to our limited time on the project, we all made a team decision to change the project concept ever so slightly.  We decided rather than focusing on extreme accuracy and making the projection mapping only on the dancers, we decided to make the projection some what of a shadow of the dancer.  This way the dancer or audience user can play with their shadow while still having projection mapping on themselves.  

Thursday, April 11,

Today JJ and I came to ETB and continued to work on the shadowing effect of the program.  We also worked on making the video display only on the silhouette of the user.  We were able to figure out the proper method for calibration between the Kinect and the projector and now to finish implementing it!

Monday, April 15

Today JJ and I finished most of the calibration formatting and were able to test it on the dancer who finished making their white costumes for the performance! Everything is coming together quite nicely.  Only a few more tweaks to work on the calibration and we will be set.  Also need to finish up the server code for receiving the information from the two Kinect system.  A video of today's results is shown below.

Wednesday, April 17

Today, after a few hours of banging our heads against the wall, JJ and I finally figured out the trick to perfecting(or as close to perfecting that we will be able to get) the calibration system.  We figured out the we needed 3 tables to use for the calibration system: two at diagonals and the other perpendicular to it.  This allows us to get a more precise transformation matrix for the calibration! The results can be seen below.   

Sunday, April 21

Today Peizhao gave us the confirmation that his 2 Kinect System has been successfully modified to send depth data over a UDP network as apposed to just the skeleton data.  We will be testing it later today or tomorrow to mesh it with our code.

Monday, April 22:

This morning, JJ and I got together and set up the new computer that Peizhao built for the Kinect server/client system.  We also set up the client end to work with our code.  Here are a few pictures of the setup and us struggling with the calibration.

Tuesday April 23,
Well after a long night last night and a lot of work today we have gotten the 2 kinect system to work!!! Whoop!! Glad to be almost done!

Wednesday April 24,

Today JJ and I worked a little more on the final little kinks in the program.  JJ was able to make the silhouette of the dancers be slightly larger than the dancer.  This added a little more of a playful feel to the program.  The dancer were able to demo the system as well.  After the demo we decided we might need to add a third kinect.  This shouldn't be to hard to do.  We will meet in  ETB on Friday evening and get it done!

Friday April 26, 
And its DONE!!!! After several more hours in the ETB we are DOOONNNNNNNNNEEEEEE! we ended up not using the 3rd kinect as it would not connect to the system(for some unknown reason!!)  JJ and I finished putting the video in the proper order that the girls requested and also we found an optimal positioning for the two kinnects such that the dancers are able to use the entire stage space and still be detected by the system.  Here is a video of the practice run.

Monday April 29,
Well today was the day! We performed and thought it went quite well! Everything went smoothly after set up despite a few mishaps.  Here is a video of the final performance, enjoy!

Dance-Inspired Technology, Technology-Inspired Dance

In this article, Berto Gonzales, Erin Carroll, and Celine Latulipe combine their thoughts and research to approach the topic of the combination of dance and technology.  They describe how the technologists work a designers and how they must consider how all of the parts to the dance and media will be integrated to inform the overall gestalt of the piece.  They  describe 5 main design principles for making interactive dance

1.  Connected Kinetics: Audience members should be able to tell the dancers' movements control, manipulate, or influence the kinetics of the visualization.
   
2. Augmented Expression: The technology should enrich the mediums of expression available to the choreographer and dancers or provide entirely new mediums of expression.
   
3.  Aesthetic Harmony: A relationship should be established between the dancers and visualizations such that they complement each other to create an integrated aesthetic.
   
4. 4. Interactive Build: The variability and intensity of the visualizations and interaction should progress throughout the dance to match the build of the choreography.
   
5. Integrated Process: The technologist should be integrated into the choreographic design process to better inform the development of the technology, visualizations, and choreography to support their subsequent union.

          

Project 2 Updates

Here are a collection of updates for Project 2.

Monday, February 18
Today we met in class and discussed several ideas for project 2.  We are not entirely certain on what we would like to do but we are thinking about using the Vicon motion capture system in HRBB 505.

Wednesday, February 20
Today we discussed more about our ideas for Project 2 and after a few hours of debate, we decided to create a prerendered video using the Vicon motion capture system of the girls' dance, add aesthetics to the data using processing, and have the girls dance in front of the projections for the performance.  All the details have not been worked out yet but we will have more soon. Here is a simple picture of the setup of the performance.  The pre-rendered video will be displayed on the 2 large screens.

Friday, February 22
Today we all met in HRBB 505 to capture the dancers' motion on the Vicon system.  JJ worked hard and was able to get the data captured correctly.  JJ is now going to take the data he captured and turn it into joint data that Van and I can use to make the aesthetics in processing.  Van and I will also be responsible for writing a program to add more points to the skeleton.  More to be added to this.  Still not exactly sure what is needed.

Sunday, February 24
Today Van sent a little test demo of what she created with some sample data that JJ had from the Vicon system.  You can click this LINK and check out her simple results.  I am going to now create a program to reads in the text file full of joint positions that JJ creates and output a similar text file but with more joint locations.  This will remove all of the gaps in the skeleton that can be seen in Van's video.

Monday, February 25
Today was a class workday and we made a little more progress.  Van and I continued to work on the visual part of the project, deciding what looked better and what colors worked best.  JJ and the dancers had to go back to the HRBB lab and re-record some parts of the dance.

Wednesday, Febuary 27
Van and I continued to wok on the visualizations in processing.  I have finished the program to add extra joints to the text file.  Now we just have to work out the final kinks in the code.  We tested the project in class but the colors were a little to dark so we must re-render the video with brighter colors so the video is visible on the screens.

Saturday, March 2
Today Van and I finished up putting the final touches on the visualizations in Processing.  We rendered the video and now are getting ready to add the song to the video for the performance! And lastly to line up the video in madMapper to get the projection done properly.  Here is a picture for the video of the rendered data.

Monday, March 4
Today was the big day! We presented today.  Thought it went really well.  The girls changed the idea slightly of the performance and so their physical dance was only in sync with the virtual dance projected on the screen.  The idea was that what was shown on the screens was simply an alternative to what the dancers chose to perform in real life. Here is a video of the performance!

Project 1 Updates

January 31
Just an update, here are some of the objects Ive made for the scene so far! More to come

February 2
Ok so here are some more Objects I have made for our scene.  The plan is to export these files to an OGRE object file use OGRE rendering engine to make the scene.  Hope it all works!

February  5
Today JJ and I met to see if the objects I made will work in OGRE.  We had some trouble with the OGRE exporter for Maya so we are thinking about using Unity but we are not sure yet.  More to come.

February  6
Ok we have figured out what we need to do to get the objects working in the OGRE scene.  After several hours of struggling we have figured it all out!

February  7
Just an update! JJ got some of the objects put into an OGRE scene and has figured out how to get the OGRE exporter to work.  Here is a picture of his work so far.

February 10
Getting all of the final touches put on everything.  Finishing up the coding. Van has done a good job with the pose recognition.  Just working out the last few kinks with the entire system.

February 12
Ok got the powerpoint all finished up for the final presentation of our Project 1.  Cant wait to see the performance!  Here is a LINK to the PDF of the powerpoint.

February 13
ok so project 1 is all finished up and worked pretty great.  here is a photo of the virtual set and a link to the video of the performance!

Video of Performance

Project 1 Research and 1 Slide

Alrighty, here we have a really neat short film I found. Click the link and check it out!  Link to Video

Summary of Live Media by David Saltz

Saltz's Definition of Interactive Media

•     Saltz firs begins by defining interactive media as the sounds and images stored and created on a computer in response to a performers actions

•    Saltz continued to define three critical features that distinguish interactive media from linear media

1. Random access
2. An arbitrary link between trigger and output
3. Media manipulation                  

•  Saltz then explained the importance of making the live media respond to the performer as apposed to the actor responding to the media.  If it is done by the actor responding to the media, the performer is bound by the program and in the event of a slight error or a slight bit of improvisation, the dancer's actions will be clearly wrong as they will not match the preprogrammed media. 

3 Productions by Saltz Using Live Media

• Medicated Interactions: Hair
• In this production, the audience is joined with the performers from a unique atmosphere created when the actors were experiencing drug-related visualizations.  He did this by projecting images of the actors floating through the air along with a keyboard that added even more seemingly random media to the performances.

• If the Walls Could Talk: Kaspar
• In this production, an large, omnipotent eyeball was controlled by an offstage computer operator. This eye followed Kaspar around offering him advise and sometimes would close and display images or text that befitted Kaspar.  Also, voice sensors were used to activate and deactivate the voices that Kaspar was hearing, all at the sound of Kaspar's voice.  

• Real Time Motion Capture: The Tempest
• In this production, the role of Ariel was played by an actress in a motion capture suit.  Her actions were recorded, interpreted, and re-displayed on a large projection screen where the other actors interacted with the on-screen character, ignoring the actress in the suit.

The relationship between Performer and Media

• Saltz next discussed what he called a taxonomy of media that can be added to a performance
• Virtual Scenery
• Interactive Costumes
• Alternate Perspective
• Subjective Perspectives  - thoughts or dreams
• Illustrations
• Commentary
• Diegetic Media - media within the world of the narrative
• Affective Media - media used to produce an emotional effect on the audience
• Synesthesia - using media to stimulate one sense organ to simulate another
• Instrumental Media
• Virtual Puppetry - using media to create a performers double
• Dramatic Media - media that dramatically interacts with the character in the narrative

• Saltz made it clear that this list is not exhaustive or mutually exclusive.

A Project from My Portfolio

3D Insect Project

    - As the title hints this project was designed to challenge us to create a 3D insect with OpenGL 

    -Each body part has the capability of moving. The legs move up and down/ forward and back and the   

      forearms move forward and back.  The head moves up and down as well.

    - Below are a few screen shots of the bug.  The bug can be downloaded HERE

Lamp Animation Project

     - Here we were assigned to design a basic 2D lamp and have it perform some form of animation

     - Below you can see a picture of the lamp after it has completed the animation.  

     - To see the full animation click HERE and download the executable file.  Once it is open press the "A"       

        key and you will see the lamp work its magic.

  

      

Dance Related Video

Howdy!

Last year in our Algorithms class we were introduced to several interpretive dances helping us better understand certain sorting algorithms.  They were quite enjoyable and beneficial. Check it out.

http://www.youtube.com/watch?v=XaqR3G_NVoo