Well, not quite that bad, but it takes me back a bit.

For example, as the whole of PatchBox is LUA scripted, I bunged together a quick 50 line script that reads the public feed from twitter every 15 seconds and renders the first entry as vector text, seen here with a phong lighting CGFX shader.

I didn?t have to add any extra code as LUA happily loaded some external libraries (luaexpat and luasocket) after I?d copied them into the PatchBox directory, and then it was a simple case of making an HTTP request and parsing the XML.

Pretty basic, but a quick proof of concept and it works!

Thinking it might make quite a fine installation rendering out the real-time twitter feed onto a building using video mapping.

FUN.

Anyway, back to the cafe project?

While playing around with PatchBox’s projective texturing features (and listening to Radiohead) I caught this bit of video. It’s using the same model that we created for the "Out Of While" show we did at the GV Art gallery in London at the beginning of 2009 but this time projected onto a sphere, all in real time.

The music is being analysed also in real time using AudioBox (another bit of software that I’ve written) and the frequencies turned into MIDI that get sent to PatchBox.

The model was created in Blender, and the movement of the blocks is controlled inside PatchBox using a single LUA script.

• Re-wrote the task scheduler to be more extensible.? This has enabled a ?job? to be wrapped in a (sub)classed object with additional context parameters.
• Fixed up DirectShow loading.? It?s nice to see 1080p content streaming in with synchronised audio!
• Started adding in a particle system
• Added a 3D model loader.? Currently only loads MS3D files, and the animation system needs some work, but it?s getting there.
• Wrote a whole CGFX SAS parser to handle shaders made with FX Composer.? PatchBox can now have pre and post scene shaders!
• Added support for DDS texture files
• The LUA scripting system continues to evolve:
  • You can now add a LUA script as a render job so it?ll be called each frame
  • Added in LuaGL so it has complete control over OpenGL
  • Scripts can dynamically add their own parameters so they can be controlled by the user, MIDI, OSC, or other scripts

Quadratura is doing visuals for a record launch this Thursday that has required a few new features in PatchBox.

It?s been on my mind for a long time that the LUA interface was pretty messy ? more a proof of concept than anything.  So yesterday I re-wrote the whole thing.  I?d already added a large amount of support code while implementing OSC so the actual interface allowing control over the entire system was pretty trivial.

Now you can do things like ?screen = root.geometry:add();? to add a screen, and ?screen.OriginRotation.y = 45;? to set that screens rotation from the origin.  Obviously there are some uppercase/lowercase ?issues? to work out so the whole system will be a bit more consistent!

The image shows the work in progress for the installation.  Double-clicking the script (to execute it) adds (currently) 50 screens, randomises their origin Y rotation and Z depth.  On each frame the script increases the Y rotation a little, dependent on each screens Z depth, giving a fake ? but rather pleasing ? parallax effect.

In the output window you can see each screen has been assigned an alpha mask.  These will eventually become clouds.

I also added an onStop() handler in the script, which is called ? surprisingly ? when the script is about to finish.  This allows the script to clear-up and release any resources it?s been using.  In this case, the script deletes all the screens that it has added.

The other major addition is the light-blue ’space? view (which will probably be renamed to a more consistent ?world? view).  It shows all the screens positions as a wire-frame preview and allows the operator to fly about the world using the mouse.  At the moment it?s a nice bit of eye candy but it?s going to become vital for some of the complicated features I have in mind for the future!

Today I?m implementing stencil screens.  These are basically normal screens but get drawn first into the stencil buffer, masking out the normal screens behind.  It?ll all make sense?

There’s nothing quite like jumping in to a new event, in a new
venue, with a software program that’s never been tested publicly
before, and doing an hour of completely improvised visuals in front of
a room full of strangers (many of whom where film makers) to really get
the concentration focussed!

Technically it performed pretty
well. Lots of video clips, FreeFrame effects, LUA scripts, all being
fed into a GLSL shader. I kept vertical sync on, just for safety, and
it seemed quite happy purring along at 30fps @ 1024×768.

It did crash the once but fortunately just as I was fading out anyway!

What
rapidly became apparent was the need to be able to get rapid access to
key features. I had to spend too long dragging and dropping, switching
between property pages, and entering values via the keyboard.

It’ll
be a priority to get MIDI support implemented as soon as possible,
which will help alleviate the value control situation dramatically.

However I was pleased on the whole. There’s obviously a couple of stability issues that need ironing out ASAP, and a fresh set of entries on the to-do list, but altogether a pretty successful first live test!

And they’ve asked me back to do next month, too: Sunday, February 17th.

Sadly didn’t manage to get any filming done this weekend. Think I’ll prepare some clips tonight and try and film the thing in action on Wednesday (am in London tomorrow afternoon/evening for a meeting about this year’s InTransit festival).

For years the plan was to have the entire internal system of PB available for scripting. Previously I’ve messed about with various approaches but have settled on a hierarchical tree of named wxVariant objects.

This benefits me in three ways:

1. it’s instantly ready to be tied into OSC as the naming conventions are exactly the same
2. using wxVariant saves having lots of classes for each type
3. using the OSC addressing scheme in LUA reduces the need for a lot of class specific functions and provides a quick way to access new functionality without having to write a lot of supporting glue code

At the moment there are only global functions as I’m just using light user data so the code looks like this:

-- test_000.lua

root = pb.getroot();      -- returns the object that the script was dragged 'n' dropped on

var_alpha = pb.getchild( root, 'alpha' );

pb.setvalue( var_alpha, 0.5 );

Which is obviously pretty long winded and confusing. Have already partially written the code for using LUA objects instead but just need to make sure it’s going to be workable without introducing class dependence.

Bolstered by that success I also turned my hand to adding OpenGL shader support. I’m almost ashamed to admit I’ve not really gotten into shader programming as yet. Have been too busy pushing pixels around with the CPU, and besides I haven’t been using any visual software that supported them. Until now!

My original thinking was that shaders would act a bit like textures: you can drag and drop them onto geometry but reading up and refining my understanding of how they worked I realised there may be some interesting possibilities for not only shifting off workload to the GPU, but also expanding on the flexibility of the application.

At the moment PB supports multitexturing and the default blending modes – this is added in the code, but by moving that part into the shader, and just expanding the properties system to be able to drag and drop textures into the configuration of the shader, I can get rid of that hard coded functionality, reduce the amount of GL instructions I need to call per frame, and get much more flexibility. Sounds like a plan to me!

]]> http://www.bigfug.com/2007/12/17/patchbox-r161/feed/ 1 http://www.bigfug.com/2007/08/20/more-patchbox-musings/ http://www.bigfug.com/2007/08/20/more-patchbox-musings/#comments Mon, 20 Aug 2007 01:42:19 +0000 Alex May http://www.bigfug.com/test/?p=13 Finally got around to adding a few new features into PatchBox that I’m planning to utilise for the upcoming Belle Atmos gig on Wednesday:

- Ability to lock screens so they can’t be edited (useful when you’re dealing with more than about 10!)
- Fixed Z sorting so screens can be moved in front and behind others
- When dragging a texture onto a group you can now hold the shift key to duplicate the texture across all screens, rather than spreading it.

After the last gig a couple of months ago where I was using 25-30 screens with five live video streams (mostly from VJO), I got the feeling it was all getting a bit much, visually!  While it’s fun to push the technology, it comes to a point where there’s just so much going on that there is no focus for the viewer.  Time to scale it back!

My plan for the next one is to only display one or two video based clips, confined to a single screen, or perhaps spread in a partial mosaic type effect, and have the other screens pretty much be flat colour, though perhaps with a subtle corner shaded mask.  Should create a more ambient effect and provide the needed focus.

There is one trick that I want to try out towards the end of the set that will involve a screen that covers the entire stage with some alpha blended particle effects.  As the screens tend to be very defined, introducing images floating over and between them will radically alter the perception of the set up.  May save that for the last song!

As well as going to the Loop Festival yesterday, I also popped in to the Concorde 2 to see VJ Spank doing his thing.  He’s been working with the latest version of the streaming plugins and had two machines set up, one running vvvv and sending 800×300 video via ethernet to the other, which was running OpenTZT.  There seemed to be a few problems with loss of connection now and then, due to switching patches, but when active it was streaming very smoothly.  I’m adding a couple of minor features this week so look for a new release soon.

Today I’ve mostly been considering how to structure the internal data of PatchBox in order to allow the maximum flexibility and control from LUA.  Was looking at wxVarient from wxWidgets, but not sure that’s quite right since it actually goes further than I need i.e. the contained type can actually change at run time, and I really only need the RTTI functionality.  However, this poses and interesting problem that at the moment constructs like the virtual screens are defined as a nice class hierarchy that works very well in a C++ environment but not so well if you want another interface to be able to access the contained data.

It’s either a case of providing get/set functions throughout the classes to expose the data, which is probably going to introduce way too much overhead, or alternatively, move the data outside of the classes, though this does possibly mean the need for an observer interface.  For the screens it wouldn’t be too much of an issue since every frame they run through and read the data and render the screens with no need for dynamic event triggering.

The other consideration is whether to have the data contained purely within the screen library, as it is at present, or implement a more global system as I had in the original PatchBox design.  Both ways have their advantages.  More thought needed!

In the image – which is an actual screen grab - you can see the four screens in the top right have been grouped and an image applied across them.  On the bottom row, going from left to right:

- Coloured image with second texture applied that has both image and alpha data
- Coloured image with an alpha mask
- Alpha image with the image/alpha mask (alpha carries through)
- Shear transformed normal image with the image/alpha mask

Both the image and the mask can be animated, too.  I just need to rewrite the depth manipulation code (so you can move screens in front or behind each other) and this iteration will be done.  Not quite sure what the next step will be, but probably adding the virtual screen control functions to Lua so I can switch them via MIDI.  Also would like to be able to switch groups of screens on and off.