Position Pass and Camera Metadata Screengrab

The first is the embedding and extraction of a camera transform from the metadata of an EXR.   3ds max (and some other packages) write out the position, rotation, field of view, etc. to the metadata of EXR outputs.  Since we use this file format a lot, it’s a pretty easy thing to use in production, as it doesn’t require a special operation to export the data from the 3D scene.  It’s not as flexible as outputting an FBX, but it can be done without opening (or even finding) the 3D file.  The transform is stored in the metadata as a 4×4 float matrix, and using SimpleExpressions, you can apply the transform to a Fusion 3D Camera per-frame.  If you’ve ever used RPF files with Combustion, you’re probably already familiar with the workflow, we’re just doing that with EXR’s and Fusion now.

The second technique is the rendering of a worldspace intersection pass.  By rendering the location of the surface as an RGB vector, you can find the location, per pixel, of each sample in the render.  Combined with even a hardware rendered “beauty pass”, you can easily place your render into the “world”.  You’re limited to only seeing the first intersections, and only what the camera sees, but in many cases that’s all you need to see.  It’s smaller and faster than an animated FBX sequence, and you can play around with it with image tools.

The fun part is combining these techniques, getting both the shape of your scene and the location of your camera in one quickly rendered EXR sequence.  You can then use this information to try out some various camera separation and convergence setups using the stereoscopic rendering of Fusion, which can range from simple anaglyph to  full quad-buffer shutter setups.  The quality isn’t great, but it’s interactive, and until 3ds max gets better rendering of stereoscopy in the viewports, it’s a fast way to test out camera settings.

The pass and the camera can also be used in the composite as well, for making mattes, placing elements, etc.

(85MB .rar file)

ColorBlind fuse

It’s not a perfect simulation, since it only does a linear color transform, not accounting for local color frequency, and it doesn’t account for any output specific color shifts.  Nothing unusual when it comes to color, of course.  Typical things to consider, colors may look different on an LCD screen compared to web offset printing.

You can use this as a viewer macro, though I may at some point make a proper Cg viewshader.

Download ColorBlind Fuse 1.0

]]> http://www.anatomicaltravel.com/research/2010/02/color-blindness/feed/ 0 http://www.anatomicaltravel.com/research/2009/10/color-matrix-transform/ http://www.anatomicaltravel.com/research/2009/10/color-matrix-transform/#comments Fri, 23 Oct 2009 00:55:51 +0000 Chad http://www.anatomicaltravel.com/research/?p=1555 Fusion 6 added a Color Matrix tool that lets you enter your own matrix by hand, but the biggest problem with it is the lack of any methods to modify it with.  You can’t even assign controllers to it.

Fuses, however,  let you use handy methods to modify a matrix.  I’ve used some of them to create an RGB equivalent of the 3D Transform tool.  It has a similar UI, just as 3TT does, but this modifies RGB, not XYZ or UVW.

Color Matrix Transform fuse

Why would you want to do this?  Imagine that your RGB isn’t color, but UVW coordinates from a 3D render pass.  Now you can do texture transformations by using this tool to modify the mapping.  Also, if you have comps where you’re converting XYZ to RGB, you can use this to post-edit the tranformation, so you can apply spatial transformations in the form of XYZ -> RGB -> ColorMatrixTransform -> XYZ.

In the future I might add an input and output similar to the Color Matrix tool so you can insert a matrix found elsewhere, as well as generate an output matrix you can read off, but for the time being, input matrices can processed by the Color Matrix tool just prior to the ColorMatrixTranform tool, just remember to use 32bit float colors.

If you’re a bit confused as to what the tool is doing, try viewing the results in the 3D Histogram SubV.   Really helps you get your bearings.

Gel Sight

Gel Sight is a retrographic surface imaging technique that was wonderfully elegant in it’s simplicity and effectiveness.  They also gave out free samples…

Cuda raycasting 13GB of cryomacrotome goodness (in stereo)

Nvidia had a stereographic interactive realtime rendering of the full 13GB Visible Human dataset being rendered in CUDA on 3 Quadroplexi.  Very impressive.  The glasses used were the new Nvidia active shutter glasses, and were very effective.

Resistive multitouch in many form factors

A new startup out of NYU showed a novel resistive multitouch device.  Very effective, low cost, and suitable to many applications.

UPDATE: Sorry about the broken link, Touchco was bought up by Amazon, so pretty much all of the cool applications they had in mind are replaced by the Kindle 3.

VLC madness courtesy 2 Fusion-io cards

Fusion-io showed their new “budget” nonvolatile storage adapter, the ioXtreme.   $900 gets you 80GB, with a read speed o 700MB/s.  The IO’s aren’t very high, much less their enterprise solutions, but that doesn’t matter if you are reading sequential data.  The booth was pretty crazy, too, one of the better live hardware demos I’ve seen in a while.  I’ll get some pictures tomorrow.  VLC never looked so impressive…

This is example of a schema that came through last week.

I was about to whip out my favorite file renaming software, but I wanted to retain the original names for communication with the customer.  The solution is pretty easy so I thought I’d share it.  There might be a tool that does this already but its good to know how to do this on any machine without any special tool installed.  We’re going to fix this problem with CMD.exe. muahahaha!

What is an IFL?

IFL or “image file list” files are required for any file sequence work in 3ds Max.  Other programs, like After Effects implicitly work with sequences by name.  3ds Max creates IFLs every time you ask to use an image file and it detects a sequence.  This process happens so fast I bet most people click through the dialog without thinking about it. Here is a pic, in case you forgot.

If you open an IFL you’ll see it just contains text with a file name on each line.  They may have a full path, or a relative path, or just the filename.  Each file can be a completely different name or extension. ( There are other features of this simple format but I can’t seem to find any documentation on the spec.)  So how do we generate this list of files?

Solution

In the long long ago, you had to know some basic shell operations to do anything.  Otherwise you’d be staring at a blinking cursor until someone came over and did it for you.  Change directory, list files, change directory, list files, and then type the name of the program. Happy times! If this typing exercise got in the way of playing your games, you’d learn it real fast too.  And twenty plus years later, this inane knowledge saves the day.  I know command line isn’t for everyone but a few quick tips are worth knowing. Here is how to make the list with cmd.exe

c:\data\CustomerX\study01>dir /b *.jpg > study01.ifl

Easy, right? Let’s break it down.

c:\data\CustomerX\stud01> –this is the current path. If it’s not you’ll have to add this to the arguments of the “dir” command. So if the current path is c:\ and the files are on the network, it would be something like c:\>dir /b \\server\share\data\study01\*.jpg > \\server\share\data\study01\study01.ifl

dir – this is the command to list the contents of the directory/folder (type dir /? for help)

/b — switch to only list the bare name

*.jpg – this is the filter. in this case “all the files with extension jpg”. You could put a path here too. ie.(..\study02\new\test01\*.tiff)

> – the right bracket means redirect output in most command shells. This will send the output to a file instead of the screen.

study01.ifl  – This is the file that will be created.

Not so hard. You can also use /s to recursively list the files in subdirectories.  Now you can use this IFL in Fusion, Max, and many other packages.

Bonus: add a number after the filename. This will hold that frame that duration.

Ben Lipman
Tool Programmer/TD
Anatomical Travelogue R&D

You know, B&W footage.

I needed to find the parts of the data that were changing the most and compare them to the overall data and the maximum delta.

What I ended up with, once Ben pointed it out to me, was a simple example of calculus laid out in a couple tools.   The simplest case is just taking the frames I have and interpolating the same number of frames, so there’s no missing samples.  It’s silly, really.

But you can try it with other sampling, so there’s also an example of a Sobel filter, with a 1D kernel perpendicular to the normal 2D one.  Cute really.

If you checked out my interactive smoothing comp, you can see how I used a Sobel filter to make the forward facing laser pointer by looking at the differentiation of the R and G channels over time.  Same idea, just different way of expressing the temporal dimension.

I’m tossing in a Laplacian filter too, just for fun, it’s not useful for the calculus part, but it was easy to do, and shows how you can change the kernel to make different effects.  It’s possible to also evaluate 2D or 3D kernels this way, too.  The temporal offsets can be combined with spatial offsets so you could make a 3D blur filter, or a 3D sharpen.  Or a 3D Unsharp Mask, as I’ve also included.

Simple Definition

This is the standard illumination model, or how our eyes and cameras work. This is actually a tiny sub-set but you get the idea. Light sources emit energy that bounces off objects into a camera and absorbed by a sensor. This could be film or a CCD on a digital camera.  Actually the purpose of the camera isn’t so much collecting the light from the image, rather the camera is blocking the light that is not the image from exposing the film.  Light is coming from all directions and bouncing all over (not shown), but only the light that converges through the lens is captured.  This is not the case for x-rays. The whole point of x-ray imaging is to have the energy mostly penetrate the object and measure the how much reaches the sensor. This type of imaging is analogous to cast shadows.

The denser the object the more rays are deflected. Metal is denser than bone, and bone is denser than muscle.  It’s also important to note that the amount of object between the emitter and the sensor affects the amount of energy deflected. Also the characteristics of the sensor medium plays an important role.

Basic Preparation

Typically the much of iconic “x-ray look” is from the film.   X-ray film is a negative and has a bluish tint.  I find this hard to deal with directly.  It’s counter intuitive for the denser objects to be white.  If we invert the image and apply some other manipulations we can get a better idea of what the target render should look like. The target render will be what we render in the 3d package that will be flexible to tweak in the composite.   Afterwards it will be trivial to apply an invert and a tint to recreate the look of the film.

In this blow up we can observe a few of the characteristics close up.  First of all, the hand bones seem lighter at the bottom.  In fact, the fingers seem much lighter too.  This could be either from the emitter losing strength at the edges of the image. It may be that more stray x-rays are exposing the film . If you look back at the original, the thickest parts of the arm and wrist look like they are glowing.  What ever the cause, its hard to remove from the render target so we’ll leave it in. Adding a subtle vignette and glow will add to the realism of the render later on. But for now let’s move on the more important characteristics.  The bones have dark edges. This one is super important. This is the whole reason that falloff-shader technique became popular.  But if you look at the arm you might notice that the edges aren’t darker. In fact, it’s darkest at the thickest part but mostly its about the same level of gray. And at the edges it actually gets lighter.  Don’t confuse the blood vessels for dark edges on the arm.  But do notice that the blood vessels are darker than the arm and are darkest when passing behind or in front of a bone.

All these observations can be explained by what’s happening along the ray cast from the emitter through the medium to the sensor.  Don’t forget bones are hollow. Inside most bones is the less dense marrow.  And the skull is like a hollow sphere containing the brain. This takes care of the look of the bones and large vessels. The rest of the tissues behave like the solid examples.  Observe that at the edges the exposure is actually less. So the important factors are 1) entering and exiting an object or tissue 2) the thickness of the object or tissue.  If your renderer doesn’t support these two requirement it will be hard to create these images.

So it’s all about thickness and rayhits, not falloff.  Next time we’ll go into making some real renderings and maybe some shader writing.  I’m still deciding which renderer to use. Then we’ll discuss the problems involved in making models that will work with this technique.

Ben Lipman

Anatomical Travelogue R&D

Trying out some new datasets and new techniques…

EDIT:  Jim asked for some more details, and I already had some images that I intended to post, but forgot about.  So here’s a breakdown of the three layers used to make the above image…

Chameleon rendered layers

The left layer is an environment map lookup, the middle is a front lit with high opacity, and the right is a backlit with low opacity.   These were then additively composited together.

I also did some tests on this dataset with clipping.

Chameleon culled with spherical gizmo

Chameleon culled with box

The box culling was an accident, but I thought it looked like a cut of meat that had been chewed on by mice.

When you reduce the number of samples, you get a large increase in speed, but the downside is aliasing, which generally looks like noise, since it is stochastic.

The trick, then, is to manage the noise so that you get a clean render that is both temporally and spatially free of aliasing.

Ordinarily this can be done with some filtering, perhaps a 9×9 Gaussian blur or similar, perhaps even a 9x9x9 blur that can handle the temporal aliasing. The problem that you lose a lot of fine textural detail this way, to the point where the subtle surfaces of the cells or bacteria get lost, and you end with what looks like a low resolution or out of focus image. This can save you a lot of time when you do texturing and animation, but it isn’t professional.

We’ve been experimenting with using noise reduction method that uses optical flow to estimate the transformation of pixels over a number of frames, and only average between those pixels, even though they aren’t at the same XY coordinates in the image.

The results are fairly impressive.

And here are some crops showing the differences in noise levels.

a) Original render

b) Processed with both temporal and spatial noise reduction

c) Processed with spatial noise reduction only

The left column is modestly contrast enhanced to show the noise better, and the right one has a Laplacian filter applied to highlight the noise.

Notice that A and C are similar. There’s some noise reduction going on, but the whole image is softer too.

B is actually more detailed and sharper appearing than A. Because the samples are taken from 5 frames, you can actually get more accurate result that shows the texture better than the original.

If the optical flow doesn’t calculate the transforms correctly, there is a chance that the pixels will “jiggle” some. It’s actually a pleasant side effect when doing organic fluid motion. If the relative transforms of the surfaces can be recorded to a X and Y vector channels, as they often can with CG images, then you can eliminate the optical flow estimation and get a cleaner image. Unfortunately, that’s not feasible for transparent objects.

SEM shaders, before and after

When I started working at Anatomical, we had a compositor (of sorts) working here who we would pass shots off to. Because a variety of reasons it wasn’t very effective, and so I pushed to have the process modified so that the people doing the 3D rendering were actively involved in the compositing workflow, and vice versa. Fast forward some years, and at this point, I comp nearly all of my own shots, and our “compositor” is pretty darn good with rendering from 3ds max.

Turns out the rest of the industry pretty much agrees with me, and now we see a lot of people who are skilled at both ends of the rendering pipeline. The software is moving that way too. Just take a look at Fusion 6 to see what’s coming.

Here at Anatomical, we had a mix of causes and consequences for this transition that I’ll be discussing in the future, but one of the most interesting is that this makes for some fun renders, as the images that we design the shaders and lighting for are intended to be mutilated afterward.

We hope to share with you some of our more interesting images, those that show some of the process behind their creation. Not all of them are good renders, some are the experiments gone horribly wrong; but they are, at least on a technical level, interesting.

So here’s the first batch…

This is what came out of Brazil.

20 minutes later, Fusion spit this out.

Ok, the color is a lot different, which will throw your eye off making you miss the details, so I did a little mockup to let you compare just the luminance of the two.

Luminance only, Brazil on top left, composite on bottom right.

The watermarking was done just for this posting. I am not sure if I like it or not.