Rendering foliage with highpass filter

If rendering foliage on has two options:

1. Render it in opaque pass, using alpha test to discard transparent pixels
2. Render it in transparent pass, using true alpha blending

Alpha blending gives the best results but kills your rendering performance very quickly. First - the blending itself is much more fillrate intensive operation than opaque rendering. Second - you cannot use deferred rendering. And third - you have to sort polygons.

Here is the transparent pass render of small test scene, as a reference:

A patch of Dactylis glomerata, rendered in transparent pass

Even small patch of forest or meadow will have tens of thousands of polygons. Something like realistic landscape simply is not doable using transparency on current generation hardware.


This leaves us with the other option - opaque rendering.


While writing foliage shaders for Shinya the biggest problem was, that scaled down textures (i.e. higher mipmap levels) looked totally crappy. Whatever was the original shape of plant - looking at it from distance it morphed into rounded blob.
The reason for it was the minimization algorithm, that averaged alpha. Thus the parts of foliage where there were only few "holes" - i.e. central parts - became completely opaque. And the parts, where there were only few leaves - i.e. outer regions - became fully transparent.

The "original" opaque render, using default mipmap generation (box filer):

The same image, rendered in opaque pass with box-filtered mipmaps

While messing with textures and trying to find the best alpha cutoff value, I finally got the right idea - we do not want to get the average image but want to preserve "features" of image in minimization. Features being both "holes" and "patches" (i.e. leaves). This can be done by increasing high frequencies of image. I.e. we want to replace the standard mipmap generation filter (normally box filter) with something better.
Fortunately I was already composing mipmaps by hand while creating packed textures. Thus I added very simple edge-detection filter with 3x3 kernel to the foliage mipmap generation (after the minimization with box filer). And the results were really good.

The "enhanced" opaque render with edge detection filter:

The same image, rendered in opaque pass with edge-detection filter

As you can see, edge detection successfully preserves the general features of the texture - stems and narrow leaves, while simple box filter tends to smear these out.

The kernel for edge-detection was:

-1 -1 -1
-1  9 -1
-1 -1 -1

And here is a screenshot of the foliage shader in action. The meadow size is about 40x40 meters, grass is about 50000 instanced polygons.

Sara in meadow

You can download the Dactylis glomerata texture from my DeviantArt gallery.

Have fun!