ColorBlind fuse

It’s not a perfect simulation, since it only does a linear color transform, not accounting for local color frequency, and it doesn’t account for any output specific color shifts.  Nothing unusual when it comes to color, of course.  Typical things to consider, colors may look different on an LCD screen compared to web offset printing.

You can use this as a viewer macro, though I may at some point make a proper Cg viewshader.

Download ColorBlind Fuse 1.0

]]> http://www.anatomicaltravel.com/research/2010/02/color-blindness/feed/ 0 http://www.anatomicaltravel.com/research/2009/10/color-matrix-transform/ http://www.anatomicaltravel.com/research/2009/10/color-matrix-transform/#comments Fri, 23 Oct 2009 00:55:51 +0000 Chad http://www.anatomicaltravel.com/research/?p=1555 Fusion 6 added a Color Matrix tool that lets you enter your own matrix by hand, but the biggest problem with it is the lack of any methods to modify it with.  You can’t even assign controllers to it.

Fuses, however,  let you use handy methods to modify a matrix.  I’ve used some of them to create an RGB equivalent of the 3D Transform tool.  It has a similar UI, just as 3TT does, but this modifies RGB, not XYZ or UVW.

Color Matrix Transform fuse

Why would you want to do this?  Imagine that your RGB isn’t color, but UVW coordinates from a 3D render pass.  Now you can do texture transformations by using this tool to modify the mapping.  Also, if you have comps where you’re converting XYZ to RGB, you can use this to post-edit the tranformation, so you can apply spatial transformations in the form of XYZ -> RGB -> ColorMatrixTransform -> XYZ.

In the future I might add an input and output similar to the Color Matrix tool so you can insert a matrix found elsewhere, as well as generate an output matrix you can read off, but for the time being, input matrices can processed by the Color Matrix tool just prior to the ColorMatrixTranform tool, just remember to use 32bit float colors.

If you’re a bit confused as to what the tool is doing, try viewing the results in the 3D Histogram SubV.   Really helps you get your bearings.

Color to Mono

Mono to Color

Why would you use this?  Memory savings, mostly.  An alpha-only image takes 1/4 the RAM of an RGBA image.   For scalar data operations, or masks, or any time where you don’t need vectors, you can store more data in cache and thus make everything fast and smooth.

The latest builds of Fusion also support mono-channel textures in 3D, so in cases where your video card doesn’t have a lot of memory, this can be even more useful.

Why not just use a Bmp to convert to mono?  Well, it’s a different class output, so it keeps trying to connect to mask inputs.  And you can’t set a Bmp to passthrough and get color data out.  You get nothing.  Furthermore, having Bmp’s littered around your flow is ugly.

Converting from mono back to color could be done by masking off a BG set to white, right?  But you’d have to keep making that BG be the same size, depth, etc as your incoming mask, and that’s not convenient.  And it uglies up your flow, too.

Download ColorToMono 1.0

Download MonoToColor 1.0

Gel Sight

Gel Sight is a retrographic surface imaging technique that was wonderfully elegant in it’s simplicity and effectiveness.  They also gave out free samples…

Cuda raycasting 13GB of cryomacrotome goodness (in stereo)

Nvidia had a stereographic interactive realtime rendering of the full 13GB Visible Human dataset being rendered in CUDA on 3 Quadroplexi.  Very impressive.  The glasses used were the new Nvidia active shutter glasses, and were very effective.

Resistive multitouch in many form factors

A new startup out of NYU showed a novel resistive multitouch device.  Very effective, low cost, and suitable to many applications.

UPDATE: Sorry about the broken link, Touchco was bought up by Amazon, so pretty much all of the cool applications they had in mind are replaced by the Kindle 3.

VLC madness courtesy 2 Fusion-io cards

Fusion-io showed their new “budget” nonvolatile storage adapter, the ioXtreme.   $900 gets you 80GB, with a read speed o 700MB/s.  The IO’s aren’t very high, much less their enterprise solutions, but that doesn’t matter if you are reading sequential data.  The booth was pretty crazy, too, one of the better live hardware demos I’ve seen in a while.  I’ll get some pictures tomorrow.  VLC never looked so impressive…

Download KrazyKey 1.0

A keying operation masks out a subset of the RGB color cube. This subset is usually a contiguous region within the cube, with the final masking weights related in some way to the parameters of the shape’s geometry.

As an example, the ChromaKeyer makes an octahedron, but not a regular one; think of it as a box aligned to the color cube, but with 2 corners truncated along the white/black “desaturated” axis. Primatte uses a a regular polyhedron but with the vertices repositioned to enclose the appropriate bounding region.  The main algorithm uses 128 faces in a closed shape. In both these cases, forcing the sampling of a contiguous region may include color coordinates that you don’t want, no matter how complex the shape within the RGB cube.

KrazyKey does not create a polyhedron at all, but uses a point cloud to define the sampling region.  It can thus enclose ANY color set at all, regardless of shape, and does not have to be contiguous.  This means that any keying operation can be performed with just one KrazyKey, whereas you might need a few Primattes, or dozens of ChromaKeyers.

Chad put together a demo comp that shows the subset of the color cube used by several common keying methods compared to what’s possible with KrazyKey.

Comparing keying geometries (red/cyan)

KrazyKey’s interface is pretty simple:

You input two images, the first (Input) is the image you want the key applied to, and the second (Selection) is the set of colors you want to compare against.  Any pixels with an alpha of zero are ignored, which reduces the number of compared, but also allows you to mask out which portions of the color set to use. The Selection image is collapsed to a 1D vector, as any spatial information in the Selection image is useless.

The Threshold specifies the maximum distance between the Input pixel and its closest color from the Selection image.  If the closest distance is greater than this value, the pixel returns black.  Remember, this is operating in float 3D space, so the maximum distance between 2 colors in a clamped 0-1 color cube is sqrt(3).  In unclamped colors, the distances could be much larger.

Rectify Output returns the absolute value of the distances.  Otherwise, the tool will output both positive and negative distances for each of the components of the vector.  Without this option, integer images would return black instead of the correct distances if negative.  We strongly recommend using KKey with float inputs.

The final option, Use cuda, is where the magic happens. If you don’t have CUDA, the tool will failover to a CPU only version, and will thus be slower, but that allows it to be more render-farm friendly.   This toggle is also good for benchmarking your CUDA hardware.

Download Comparing keyer geometry

Download KrazyKey keying example

So give the tool a try and leave some feedback in the comments section.  We’re also curious about the performance of various CUDA hardware, so let us know what kind of speed improvements you’re getting.

You know, B&W footage.

I needed to find the parts of the data that were changing the most and compare them to the overall data and the maximum delta.

What I ended up with, once Ben pointed it out to me, was a simple example of calculus laid out in a couple tools.   The simplest case is just taking the frames I have and interpolating the same number of frames, so there’s no missing samples.  It’s silly, really.

But you can try it with other sampling, so there’s also an example of a Sobel filter, with a 1D kernel perpendicular to the normal 2D one.  Cute really.

If you checked out my interactive smoothing comp, you can see how I used a Sobel filter to make the forward facing laser pointer by looking at the differentiation of the R and G channels over time.  Same idea, just different way of expressing the temporal dimension.

I’m tossing in a Laplacian filter too, just for fun, it’s not useful for the calculus part, but it was easy to do, and shows how you can change the kernel to make different effects.  It’s possible to also evaluate 2D or 3D kernels this way, too.  The temporal offsets can be combined with spatial offsets so you could make a 3D blur filter, or a 3D sharpen.  Or a 3D Unsharp Mask, as I’ve also included.

Screengrab from the realtime colonoscopy demo

The CT

Here we have an axial projection of the CT, (commonly known as CAT, for computed axial tomography).  The video shows a series moving through the Z axis from the anus to the top of the diaphragm / base of heart.  I’ve gone ahead and adjusted the brightness and gamma so it is easier to see on a normal computer monitor.

What we can see here is that the patient was lying supine, and the contrast enhanced fluids in the colon are collected on the bottom surfaces of the colon.  Sometimes the patient is scanned twice, once supine, once prone to make sure that all surfaces are imaged, but we found that the contrast between the walls of the colon and the fluid was sufficient to make out any “submerged” surface details to the same resolution as in the upper “air exposed” surfaces.

Segmentation

Ok, so the next steps are to remove the fluid from the colon, and to segment the colon lumen.  The values in the CT are sufficient for this, the lighter values are the fluid, the darker values are air, and the middle greys are the colon itself.  We have to isolate the colon from the bones, lungs, small intestines, etc. which share similar values, but they are spatially much different, so we can exclude them that way. The image below shows the result of this, a single axial slice of the CT on the left, the fluid removed in the center, and the segmentation mask of the colon lumen on the right.

Segmentation results for colon lumen. Left: Original CT; Center: Fluid removed; Right: Lumen segment

Normals

The CT dataset can now be processed to get some additional visualization data out of it.  We clean up some of the noise in the scan, and then extract gradients from the data which will be used as normals at the threshold interface of the colon lumen.   Essentially, we’re determining where a voxel is “facing”.  This vector is encoded as color, where red represents the X axis, green represents the Y axis, and blue represents the Z axis.

Encoding normal direction to color

The image below shows the normals we calculated, as well as those values multiplied by the CT values.

Left: Normals as RGB (not yet normalized); Right: Multiplied by the CT intensities

And this video shows these images stacked on top of one another with a little perspective added to show the depth.

And here’s a video with just the normals on the colon segment itself.

Isosurface

From the lumen segmentation, we extract a polygonal isosurface.  The resulting mesh is scaled to the correct real-world size according to the CT scan itself.

Marching cubes extraction of the lumen, scaled to real world units

Rationalization

The isosurface is then rationalized to a curve network.  Along the length of the colon, longitudinal lines are created as well as cross-sectional lines.

Rationalized curve network of colon lumen

Centerline Finding

The curves generated from the rationalization can then be averaged together to find the centerline of the colon, which will be used as a camera path and as the base of our unwrapping process.

Centerline path (thickened to show more clearly)

We normalized the spline and took some measurements off it.  In this case, the colon measured 1866 mm.

Unwrapping

So now we have all the information we need to do a basic unwrapping.  We create a new voxel array XYZ where X is the length along the centerline spline, Y is the complete rotation of the vector perpendicular to tangent of the centerline curve, and Z is the distance from the centerline curve.

How the voxels are sampled

Or put another way, it looks something like this:

Conceptually what is happening

This lookup was applied twice, once to the CT normal (gradient) volume and once to a unit cube of the dataset.   We also created a 1D array where the length along the spline intersected the dataset unit cube.

Here is a video showing the result from the CT normals.  The black voxels are outside the bounds of the dataset.  I’ve resized it such that each voxel is 3.5 mm in size.

And here is a still image where the density and normal values from the CT have been accumulated to generate a “surface” based on the unwrapping of the colon.  This isn’t an isosurface, we’re still looking at voxels at this point, but as a 2D projection, it creates a nice “map” of the colon.

Unwrapped normals, accumulated at the density of the CT. Each voxel is 1mm in X.

By applying a depth cue to the voxels based on distance from centerline, however, we make it a bit easier to read.   The colors show where the surface is facing, so that red is dextral (on the right), blue is superior, and green is posterior.   Something to notice… where the horizontal bands of color shift downward or upward, that is where the colon bends, or at a rectal valve.  Conversely, where you see the bands stretch out is the colon is relatively straight.  The darkening helps to differentiate protruding features, like polyps, from recessed features, like the appendix or diverticula.

With depth cue. Can you find anything suspicious?

By the same technique, we accumulate the samples from the unit cube.   This gives us a position lookup between the unwrapped dataset and the original CT.

Unwrapping samples from the unit cube of the dataset

It is now possible to create a vector between any point along the centerline and any unoccluded surface point of the colon.  But since we still have the voxel data, we can explore the occluded and subsurface details, if present.

Rendering

From here we can construct some renders that show the data relative to a point in space somewhere along the centerline.

Here is a volume rendering showing the path through the colon as a bright light.

We also created a raytraced image of the polygonal isosurface from inside, sampled along the length of the centerline.  This render covers a complete 360°
view from each millimeter along the centerline path.

This video shows the spherical fisheye view on top.  We can convert this to a latitude/longitude view, as shown on the bottom.

Because the view is a full 360°, you can see everything visible from that one point.   This lets us tumble around in a more normal “human eye perspective” to look around.

This is similar to how a traditional endoscope works, but a LOT easier to control.  If we wanted to be more “authentic” we could have put a vignette around the images.

Putting it all together

Once you have all the assets created, it becomes a question of packaging and delivery.  The unwrapped normal map with depth cueing can be printed out on a piece of paper, or viewed on a small cellphone screen.

Cut into strips, ready for printout to include in patient file

Another way to view the colon on paper

We can combine the other renders together into a linear video, where the number of frames is something fixed, like length in millimeters of the colon, and compare that to the unwrapped map X axis.   This could be burned to a DVD or posted on YouTube.

Or we can create a complete interactive system where the user can look for polyps in ANY of the views, the CT, the exterior view, the endoscope view, or the map, and the various views will all stay synced together.  This could be operated from a local workstation, or processed on a remote server and viewed on a webpage.   An automated version of this could be rendered out to a video as well.

A user interface for synchronizing the various views of the colon

Download Full size user interface movie (7.3MB x264 AVI)

You can see more of our colon explorations including other datasets at www.thevisualmd.com.

I’m smoothing animation data (or adding noise or offsets or whatever) using nothing more than some Probes.

In this video, the green dots represent the original animation, the blue dots the smoothed animation, and the orange is the original with some noise added.

The trick of course is to convert the XYT values to RGX, blur those, and convert back to XYT.

The advantage is that you don’t have to blur, you can resize, or add noise, or paint, or mask or whatever.  Heck, you can even disk cache it out WITHOUT modification, so you could, you know, open it up in some other package, like 3ds max or nuke or flash or whatever.

Here’s the green animation…  You might have to squint.

Animation of the green dot, expressed in XYT->RGX

Download Smoothing Animation with a Probe

This example video shows a yucky little roto problem…  Time lapse footage of a shape variant object. The colors flicker like mad, the shape is doesn’t interpolate well with few keyframes, there’s nothing for a point tracker to grab onto…  It’s just the sort of thing you need to use a spline on.

Many of the datasets we deal with are similarly shape variant, and many are essentially time lapse, or at least shot with a very, very slow framerate. In this case, we look at how you can check (and tweak) your polyline over time without A) having to select the points and B) having to SEE your point.  The burning edge of the cigarette is flying across the screen, but we don’t have to pan along to follow it.  The Polyline Unwrapper keeps the “edge” in one place in the viewer.  This makes it easier to find the shape, true, but more importantly, it lets you observe irregularities instantly.  Using the Filmstripper, you can arrange multple frames in a row to check for drift or any “peaks and troughs”.

Here’s a second example of Eyeon’s Fusion courseware showing the “overlay samples” mode, where the unwrapped sample points are plotted over the original image.

Unwrapped example from Eyeon's courseware

Download PolyLine Unwrap 1.0
Download Example files

There’s some other fun stuff in the example comp, like using it on a paint stroke, instancing the result to another input, and viewing the results in a Trails tool.   Have fun, and experiment!

Footage courtesy openfootage.net