PDA

View Full Version : Heatmaps



Hai-Etlik
12-03-2012, 11:20 PM
So, one of the things I was working on recently at work was logging the bounding boxes of WMS requests made to GeoServer. This obviously made for a lot of overlapping boxes, which is hard to display with normal symbolization. Geoserver does however have some interesting alternative forms of symbolization, including something called a "heatmap" This is a surface made by drawing the centres of the features into the map and keeping track of how many times they overlap. Then the whole surface is blurred. Then the whole thing is normalized so the maximum ls 1.0.

Now that normalizing of overlapping levels is really useful, but I needed to preserve the shape. With a bit of work though, I was able to make two changes. Making it render the bounding box of a feature, and normalizing the minimum as well as the maximum to preserve contrast when zooming in.

The result looked like this

50371

Now, bounding boxes are nice and all I needed for the project I was working on, but it would be even better to render the feature properly. because of the way Heatmap works, it can't just leverage a stock graphics library like Java2D (which most of the rest of GeoServer uses for vector to raster.)

I didn't have time for polygons (at this point I was on my own time), but rendering a linestring is pretty straight forward. So I added that, took some GPS tracks from my time delivering pizza, and got a pretty cool looking map.

50372

Korash
12-04-2012, 01:14 PM
That is NEAT!!

I am thinking Star Lanes....or initial layout for a town, or city with enough samples, using the brightness for poulation usage...or a kobold warren....or trade routes......ad infinitum....

Hai-Etlik
12-04-2012, 08:17 PM
That is NEAT!!

I am thinking Star Lanes....or initial layout for a town, or city with enough samples, using the brightness for poulation usage...or a kobold warren....or trade routes......ad infinitum....

Well, really this is something where you need the data to begin with and what I've done is create a way to visualize it, although it could be used for some analysis of the data too. For instance, you could drape a road network over the heatmap surface to set its M ordinates. Then you would have the traffic levels on the roads.

Korash
12-04-2012, 08:43 PM
Yeah, I get it about having the data, but really, all we have to do is pick up a GPS and start gathering the info ;)

Jaxilon
12-06-2012, 12:34 PM
What is your goal in doing this Hai? You said it was work related so I'm wondering what the purpose is. That might help me to understand what bounding boxes are too :)

Hai-Etlik
12-06-2012, 04:54 PM
What is your goal in doing this Hai? You said it was work related so I'm wondering what the purpose is. That might help me to understand what bounding boxes are too :)

Heatmaps are a way of showing data that has a lot of overlap or clustering and where the density of that overlap/clustering is the important thing. In my case, I'm working with a GIS Server. It receives requests for sections of a map along the lines of: "Give me the map from N 40 to N 50 and E 130 to E 120". That range of coordinates defines a "box" that forms the "boundary" of the request and we call it a Bounding Box. I added the ability to log the bounding boxes of requests so the people running a server can figure out which parts of the map are under higher load and might benefit from more caching.

Obviously this produces a data set that is just a LOT of overlapping boxes which will cover each other up. So you can't just display them normally. So I took the Heatmap we already had, which just worked with points, and made it work with other geometries. But all my data was just boxes, and properly rasterizing geometries is rather more complex than just filling a box, so I cheated and made it render the bounding box of the geometry (The box that just fits the geometry) rather than the geometry itself. This was fast and easy. the result is the top image, you can see where more requests were made as the heatmap is brighter.

Adding support for Line Strings wasn't that hard either, and I thought it would look cool, so I did it on my own time. It's useful for showing things like traffic levels if you use a set of routes as the input features (as in the case of my pizza delivery map). When I have time, I plan to add proper support for Polygons/Multipolygons as well to round it all out.

Here's an illustration of Bounding Boxes:

50392

The red box is the bounding box for the grey polygon in the coordinate system shown by the grid. The green and blue shapes are the bounding boxes of that same polygon in other coordinate systems, transformed into the coordinate system of the image.

Jaxilon
12-06-2012, 07:02 PM
Great explaination, thanks. I kind of had an idea along those lines but it wasn't strongly defined.

amberroberts09
12-11-2012, 07:05 AM
Heat map is geographical representation of data in a form of colored matrix.

nameh
12-05-2014, 05:08 PM
...
I didn't have time for polygons (at this point I was on my own time), but rendering a linestring is pretty straight forward. So I added that, took some GPS tracks from my time delivering pizza, and got a pretty cool looking map.

50372

Hello,

I know it is an old post but this is exactly what I was looking for. Can you or someone tell me which program is needed for such a map?

Thank you

Azelor
12-05-2014, 05:23 PM
I believe you could achieve the same effect with any software like Gimp, Inkscape or Photoshop. Place dots representing the cities. Link them with large lines. Paint the lines with a different colour to represent the travel frequency.

Hai-Etlik
12-06-2014, 08:14 PM
I used GeoServer with a modified version of GeoTools. I never really finished the work so it's just been sitting in GitHub since then. https://github.com/smithkm/geotools/tree/non-point-heatmap

The same modification ought to be usable in any GeoTools based software that can use Processes. You might be able to use it with something like uDig.

nameh
12-07-2014, 04:58 PM
Can you please post some guidelines on how to set the programmes and how to draw such lines? I have not worked with GeoServer/Tools and I am not familiar with uDig either... unless it is very complicated and time consuming for you.

Thank you

Hai-Etlik
12-08-2014, 07:20 PM
Can you please post some guidelines on how to set the programmes and how to draw such lines? I have not worked with GeoServer/Tools and I am not familiar with uDig either... unless it is very complicated and time consuming for you.

If you're thinking of this as a graphics tool for "drawing lines" then this is not remotely the software for you. It's a spatial analysis tool which takes raw data like GPS traces and figures out how densely packed they are. Azelor's suggestion of using graphics software may be better. In essensce what the heatmap algorithm does is to draw shapes additively and then blur everything, then it's run through a colour map and optionally the contrast is stretched first. You can do much of that in graphics software. The benefit of this is that mostly that graphics software doesn't usually support high precision floating point channels or purely additive operations and doesn't support rasterizing GIS data, which it sounds like you don't have anyway. Also this is integrated with a full on GIS toolset.

nameh
12-09-2014, 05:20 PM
If you're thinking of this as ...

I was thinking about making a similar map of data gathered from my bike trips which were tracked by GPS. But if you say it is complicated I will believe you and will try making it with some other tools. Thank you anyway.

Midgardsormr
12-15-2014, 03:38 PM
If you can render the paths and overlay them meaningfully, then the heatmap approach still works, but I'm not sure how easy it will be to accomplish in typical graphics software. The process I am about to detail will work in any linear environment where you can use floating point colors and perform meaningful math on them.

Render each of your paths as a white line on black. Then put all of the lines in a layer stack with a simple add blend mode. This will push your color way above white, so now you need to divide the result by the number of layers in your stack. That will normalize the image to put your brightest colors back to no brighter than white. Once you have that, you can use the technique of your choice to make a gradient out of it, adjust the gamma for a pleasing look, or whatever.

Now, about that bit in the first paragraph: A linear, floating point environment is difficult to set up in the Gimp or Photoshop. I'm sure it can be done, but if I were performing this job, I'd probably use Blackmagic Fusion, which was conveniently just released for free. Fusion is a node-based image compositor used to make feature film visual effects. As such, it's ridiculously powerful, and a little difficult to wrap your brain around at first. If you'd like to go for it, though, I can set up a template for you and walk you through how to use it to achieve your goal. You can download the software here: https://www.blackmagicdesign.com/products/fusion

And yes, it really is free.

nameh
12-17-2014, 12:16 PM
If you can ... Fusion is a node-based image compositor used to make feature film visual effects. As such, it's ridiculously powerful, and a little difficult to wrap your brain around at first. If you'd like to go for it, though, I can set up a template for you and walk you through how to use it to achieve your goal. You can download the software here: https://www.blackmagicdesign.com/products/fusion

And yes, it really is free.


Man this is awesome piece of software! Unfortunately for my hobby map it would be a huge time-killer. Before I say "thank you, but no", let me summarize the knowledge I gathered so far.

A) I was thinking about a heat map of my rides. Road intersections have GPS coordinates
B) However not being able to set up (in fact I don't know how easy/difficult would it be and what kind of knowledge one should have) GeoServer and GeoTools, I took the approach which was suggested - to use some graphics software instead. I chose Inkscape which I used for other map.
C) I put my tracks on a paper map and divided them in sections (each being restricted by road intersection)
D) Each section I numbered and each number I summed so basically I received the number of times I passed the section in real life
E) These numbers I used to Normalize, to be able to calculate heat map colour (based on this formula: A_minimalist_heatmap_function (http://www.andrewnoske.com/wiki/Code_-_heatmaps_and_color_gradients#A_minimalist_heatmap _function))
F) Until now everything was fine and made on paper

1) I started drawing it in Inkscape so I made 1st layer which was a copy of a Google map road map
2) the uppermost layer would be left for captions
3) second layer would be used to draw lines = road sections as in D)
3a) first problem occurred: If I run through the same road section once or multiple times it does not need to be divided at each road intersection (for time saving reason). However if in future I used part of the road, I would need to divide it at the intersection where I stepped in. How would I number the sections so I knew where in the map it would be? (some of the roads do not have a number in real life so a naming method would have to be made. For those roads that have numbers, it would be rather difficult t use real life system because they may be more than 60 miles in length). The only option which I was able to figure out, was a number followed by a letter. Still the road section may be split (e.g. 1, divided in 1a and 1b. 1b can still be divided in 1b and 1c, however when I split 1b, I need to name 1b as 1ba and 1bb so that 1c can remain). The new numbers must be maintained and updated in a table for colour normalization (point E)
2b) There is one option how to resolve the issue in 2a) If I painted each track as a separate layer. But there would be problem with the fact that I would need to redraw each track individually but I don't know how it would work with colour merging so that the colours would make "heat map" effect.

I probably wrote more than you are keen to read, but I did it even for me to summarize my work so far, so I could come back and knew what I did and where the dead ends were.

Now comes the time to ask all of you and Midgardsormr especially. Have I put myself in a dead-end direction or am I going just through longer routes than necessary and would there be better tools or processes to use?

Thank you all. And thank you Midgardsormr for your will to give me a hand with Fusion.

BTW: sorry for typos and grammar mistakes as English is not my mother tongue

Midgardsormr
12-17-2014, 08:54 PM
It sounds to me as if you're doing an awful lot of computation by hand instead of allowing the computer to do what it's good at.
Have you seen this? http://brunosan.eu/2012/07/20/a-heatmap-for-all-your-runs-in-runkeeper/

You haven't said how you're getting your data out of your GPS, but from what I understand the .gpx format is fairly common.