GIS in Everyday Life

If you take a look, where you can find a kind of a GIS in companies, the readers that have to do with GIS, will easily find a lot of spots. After you successfully integrated a GIS into your IT-structure, it can provide it’s data to several other applications, from spreadsheets to external databases.

But how about your everyday life? I’m sure, everyone of you had to do with GIS there in different ways. Sometimes active, somtimes passive. Let’s have a look. You’ll find yourselfe in one or the other situation, maybe without beeing aware by now.

Over the last few months I noticed that I’ve been asked for my zip-code several times during paying at the check-out. One of the most simple way to get basic data for geomarketing. Anonymous data provides a basic information to evaluate the catchment area of a discounter, so they can more precisely address their advertising. The results for every single discounter may vary. They can check the data with the portion they wanted to achieve. By precise advertising in areas with a lower portion they can try to reach more customers than before. If the discounter has access to demographic data, they could use it to reach regions with more spending capacity. You can refine these analysis to nearly every level of detail. The last one is an example for the passive contact with GIS.

Advertisement

OSM-Data in PostGIS and QGIS

Some time ago I had the idea of importing some of the data of OpenStreetMap (OSM) into a database — as they’ve become much better quality over the last few years — and find special objects systematically. This information could populate a databasetable and could be used by our sales department. 

Requirements

As I use to work with Linux (Ubuntu), my description is related to such an environment. On my notebook I have a basic Ubuntu installed (with recent version Lucid, and a Gnome desktop).  I added PostgreSQL 8.4, PostGIS 1.4, OSM2PGSQL and QGIS 1.4 (from the standard repository of Ubuntu). To work in PostgreSQL with a GUI, I installed PGAdminIII in addition to this.  After installing the software you just have to tweak some things and soon you have a useable GIS with OSM-data. But let’s do it step by step …

Preparing the Database

After installing PostgreSQL and PostGIS, you have already activated the spatial extension for the databaseserver. In my case, I could use psql (textbased client for PostgreSQL) in a terminal immediately to connect to PostgreSQL, but not via PGAdminIII. In PGAdminIII they expect a password, which you have to set in psql at first.  So you go and open a terminal window in Linux (e.g. bash) and create a new databaseuser using createuser gisadmin (as far as you only test this case on a local PC, you can answer the question about the superuser with YES). After that you create a fresh database with the command createdb mygis. To avoid problems with PGAdminIII you should set the password for the user gisadmin right now. To do so, you connect to PostgreSQL by psql -d meingis and use the command \password gisadmin. Than you have to enter the new password twice (you’ll be asked to). Now you can disconnet from the server by \q.
Back in the terminal, you have to activate the procedural language in PostgreSQL by createlang plpgsql mygis. Now you have to find the files postgis.sql and spatial_ref_sys.sql in the filesystem. In my case they were found in /usr/share/postgresql/8.4/contrib. To really find them, it’s important not to have the packet postgis-common installed, but the one that is bundled by Ubuntu! The necessary tables for really using PostGIS in the database mygis will be created by pgsql -d mygis -f /usr/share/postgresql/8.4/contrib/postgis.sql and pgsql -d mygis -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql.

Detailed information about installing of PostGIS can be found here…

So, where’s the OSM-data …?

Well, at this point we have a fresh and clean GIS database, but without any data. Unfortunately you cannot use the webinterface of OpenStreetMap (OSM) (www.openstreettmap.org) to extract the data at once. You have to use the so called XAPI of OSM. In the beginning, I though about the XAPI as something very complicated just to get some data. But fortunately, this was completely wrong.  The only energy you have to invest is to define the area you want to extract and the objects you want to see in your database. Let me give you an example of an area in Oberhausen (Germany) where we first simply extract all data unfiltered. For a more complexe query you should have a look at this description.

I assume, that you don’t have the geographic coordinates (Lon/Lat) of the area you want to extract at hand. You need the lower left corner and and the upper right corner. To get them, you can open the webinterface of OpenStreetMap and navigate to the area you’d like to extract. Place the centre of the map to the lower left corner of the area and point the mouse to the right left corner of the map, wher you find the link ”Permalink”. At the bottomline of your browser you can see the link (permalink) with the value for Lon and Lat. Write down the complete number. Repeat this with the right upper corner.

Next step is to really get our data. You can either put this line into your browser and safe the returned data into data.osm http://xapi.openstreetmap.org/api/0.6/map?bbox=LON-LU,LAT-LU,LON-RO,LAT-RO. Or you can use the following command in your terminal window  wget http://xapi.openstreetmap.org/api/0.6/map?bbox=LON-LU,LAT-LU,LON-RO,LAT-RO -O data.osm. You certainly have to change the values after bbox= with the ones you have identyfied in your case. The resulting file can range from a few MB to a half GB, according to the area you’ve selected. If your file is empty — 0 kb — than you have to check the coordinates. This problem appears regulary …

Preparing the OSM-data

At this stage you still cannot use the OSM-data — but we will change this right now. We transfer the OSM-data (the file data.osm) into PostGIS using OSM2PGSQL. If you use the command for the first time you should use it like this:  osm2pgsql -c -d mygis -U gisadmin -W -H localhost data.osm. The parameter -c stands for creating the new tables (create) or to clean the old ones before adding the new data. This is followed by the name of the database, the name of the user, an option for a passwordrequest, the hostname for the databaseserver and the name of the OSM-datafile, which we want to transfer.
If this executes without an error — which usually are errors in the part in front of the filename — you can find your OSM-data in the mygis database. Besides OSM2PGSQL you could use OSMOSIS. I won’t describe this here, but you can find information about this via http://wiki.openstreetmap.org.

Connecting PostGIS with QGIS

One or the other reader may find it useful, to analyse the OSM-data directly in the database with SQL. But the real strength compared to textbased datacollections is the visualization of the data in a map. As PostGIS finally is just a datacollection with a huge package of geospatial functions you cannot use it directly to display the data in a map. Certainly there is a great number of GIS that you could connect to PostGIS, but I usually care about QGIS. And ou can easily connect PostGIS with PostGIS.
Within the buttons at the top of QGIS you can find a blue barrel-like symbol with a ”plus” to add a PostGIS-layer. After clicking the symbol, a dialogue pops up. In this dialogue you have to add a database-connection if you use it for the first time. Click on the button ”New” and enter the required information (mygis, gisadmin, hostname aso.). After entering the information you should go and click on ”test connection”.  If this test is successful you can leave the dialogue with the ”OK”-button. If this test fails, you have to check the entered data. So, if you successfully created the new connection, you have to activate it by clicking on ”Connect”. In the lower area of the dialogue you will see the available tables with their names. There should be at least four tables with names beginning with planet_osm_… You can find lines (…_line), points (…_point), areas (…polygon) and roads (…roads). In my case, most of the roads were inserted into the table containing the lines and not among the roads. That’s just a note I wanted to add. After this you should click on all the tables in the list, and finally click on ”Add” to add them to QGIS.

Nice lines …

… don’t make an appealing map. If you haven’t used QGIS before you should maybe go through a tutorial to learn about the basic functions and configuration of the objects. As a quick howto, I can tell you this: click on a layername with the right mouse-button (e.g. planet_osm_line) and the properties of the layer will open. Besides the appearence of the object you can change the caption. If you activate the checkbox ”Show caption” you should select the field, which contains the info you want to display. I guess it is named ”name”, if you haven’t put your info into an extraordinary field. You should play around with textheight and colour. If you want a special layer to be placed over another layer you can easily drag and drop it in the list.

What’s next?

If you successfully went through all this, there will surely come some questions about special usecases in your mind. Let me tell you this: the OSM-data is much better quality than you think. But please don’t fool me, if there’s no data in the area you wanted to analyse: why haven’t YOU already collected the data?  OpenStreetMap is open to everyone, who can enrich OSM with his data. You can also simply place your question via email in my contact form (contactform). Maybe I can add a new entry in this blog to describe and solve your problem or I can assist you personally.

Some explanations on technorati dot com

To spread my thoughts I have created a Technorati Profile. This is needed to do some WTF there …

Business Intelligence / Location Intelligence

Going forward in this discussion we are bound to talk about location intelligence and business intelligence.

Intelligence means in this terms collecting, classifiying and analysing the available information to get some conclusions.

Location intelligence is a term that is often used by MapInfo (offering a software with the same name). While the terms of the posting “Geomarketing / Business GIS” deal either more with the internal matters of the company or deal 100% with the customer, it seems to me that the location intelligence is a mix of these two terms. It deals with informations about a customer but not just for advertising purpose. More than this it takes the results to decide where to place a new shop or office. If they collect the right information, some IT-systems will give them an outlook into future or just give them a prediction. Therefor they have to georeference all the information.

Business intelligence will go a step further, which IMHO gathers all aspects mentioned until now. Internal information and information about the customer will be analysed similarily. The results will be used to improve internal aims and to optimize the customer relationship (advertising, aquire new customers).

On the following slide I have tried to sort the different aspects of a GIS looking at their aims and their main attention.

 

Please click on the thumb to enlarge.

The original GIS aims at graphical portfolio management. The ongoing development of the systems leads away from the classical cartography. If cartography is nice to have — but needed — and use of the map is very customer related I would call this activity Geomarketing (GM). Therefor it is is placed at the right top of the slide. If internal aspects are as important as aspects about customer relation and if economical information is collected as well as geographival information I would speak about a Business GIS (BG). Somewhere inbetween I would place Location Intelligence (LI). In this case maps are used to show information but internals and cartography stand more in the back. The term that gathers all the tools for internal activities and customer related information is IMHO Business Intelligence (BI). Because BI uses internal and customer related information in same values but uses a bit less maps, I would place it somewhat more at the economic side of the slide.

If you can find your aim, which you try to reach with this software, in this slide, you can easily find the category of software you will need. But this is just the main target of the software. Before your final decision you should think about what you can reach with your software additionally. By this I mean functions for the controlling (business administration). Once you have gathered all the information about your customers you can even plan and exactly optimize delivery tours or something like that. All of these additional aspects (i.e. controlling) I will discuss later.

Business GIS / Geomarketing

The term geomarketing is not really used in english language. They use the term business GIS. Both terms nearly mean the same but are nevertheless different. Geomarketing means customer-oriented marketing of any products. Business GIS includes even aspects that are relevant for production. The origin of both terms is also another: geomarketing is originally used by geographical oriented economists, business GIS is used in economic geography. The both meanings are slightly but obviously different. F. Schüssler defines the term geomarketing like this:

“‘geomarketing’ means the planning, coordination and controlling of custumer-oriented marketactivities with a GIS; so methods are used, which create, analyse and present the spatial relation of the examined information” (SCHÜSSLER 1997, translated into english by M. Blazejak, 2006)

Maybe it all began with maps at the wall, colored pins and some fibres. In this way they defined areas on the map which show e. g. the catchment area of companies and shops. By means of this method they could find raw areas very good and could turn their special attention to them. But the abilities of this method are pretty limited. External information could only be badly presented in the maps and has to be analysed, collected in listings.

With the still continuing fast development of computer hardware since the end of the 1970’ies and also enhanced software the gap could be filled faster and more efficient today. In the late 1970’ies the first research supported by the available technology in terms of geomarketing has been done in great britain.

To be adequate for geomarketing, a GIS has to have only rudimental cartographic functionality. More important are the functions to intersect spatial referenced information, trace functions and tools for interpolation. The following fields can be mentioned related to the implementation of geomarketing:

• retail industry
  • specific advertising with an optimised catchment area
• distribution
  • improvement and controlling of the field service
  • optimised franchising
• media business (e. g. newspapers)
  • improvement of the local news and ads
  • specific controlling of the inserts

The advance of computer aided geomarketing with some kind of GIS is clear to see. Only by collecting and spatial referencing information you can efficiently create complex maps with combined information in a short time (visualisation). The available information growth constantly in quality and quantity. Costs of computer hardware doesn’t play an important role anymore, because a desktop-GIS can be run on a nearly every standard computer. Real costs are produced by software licenses and external information. Because of the short times for calculating analysis and intersections you can go and find the best solution out of a lot of possible attemps (trial-and-error).

So, if you have most of your information spatially referenced and don’t use a GIS-like-tool to analyse them, you give away more potential benefit than you know by now. You should check all your lists, to find all information related to your objects and analyse them. You will be one step ahead of your competing company. Don’t waste them! 😉

QGIS (I)

Like I mentioned before, QGIS (Quantum GIS) is a free GIS. It’s a Qt/C++ application (Anno.: it is NOT a Java-application, as I mentioned before! Changed April 4th 2007) and with that it is not depending on your pc’s operating system (as long as you can get a Qt environment for it!). I cannot promise that this is true for all the plug-ins. I use it on Linux and on MS Windows . You can fetch the binaries or th e source of QGIS from the QGIS-homepage (sourcecode is not recommende). There you can also find an english documentation and the usal forums where you can find additional help.

The installatio of QGIS on a Windows-machine ends without errors and after a few minutes you can start with some sampledata. Ideally you have some shape-files of your own objects already. If not, you can fetch sampladata out of the internet. Googling helps! After that I would advice you to “play around” with your sampledata to become familiar with the gui and the functions. The gui of QGIS version 0.7.4 ‘Seamus’ is available in many languages –even in german and shouldn’t make you any great problems. A german translation of the documentation (Install-/Userguide) will be available for version 0.8. For a detailed view on all the functions you should have a look at the userguide. Maybe I have a limited view on th e software, because I personally don’t need all the functions included. And by the way: this is not a product test.

I start -like I always do- by saving an empty project. Then I open some shape-files with roads, buildings and rivers. They are displayed by QGIS with a random color. I could change the color and save this with the projectfile. If more than one should work with the data you can place the projectfile somewhere in the network where it is avaiable for all and then you can make it readonly. Everyone can now open this project but cannot save any changes. If people should also change data you have to talk about alternative ways. At the moment only the administrator has the possibility to insert, change or delete layers from the project.

After a short time I cannot resist to test the connection to PostGIS. But I still have no data in PostGIS! Where can I get them? Nothing easier than this! 🙂
QGIS has an build-in importtool for shape-files. Someone has thought about it! After a short time I have added my three shapefiles to the importfilter and with the same high speed my files where imported into PostGIS. And now? Now I can access them in PostGIS instead of the shapefile.

The advance lays on the hand: I use the data to view them in QGIS and maybe change the geometry if neccessary. Maybe there are other tables in their, which are related to my data by some kind of ID. Because of that I can connect this tables with my data and can give them a reference in my maps.

An example:
I have a table with spatial information about buildings. This table is normalised – better say: as simple as can be. It contains no more information then the geometry and the unique IDs of the buildings. These IDs originally came from a table with a lot more information. Without having tested it before, you can now create a view (a kind of dynamic table based on a SQL-statement) which connects the geometrydata and the additional alphanumeric data with the ID of the buildings in both tables. If you now use the view to show you data in QGIS, you cannot only see the buldings but access the additional data, too.
The other way round, I would like to get the information about other objects and poi (points of interest) in a maximum of 500metres around my object. All these data can be found in my database so I can use them for a spatial analysis and put them together in a report. I’m sure that you can extend the report to get the detailed distanc of each poi. In an external application it will be easy to add another column with the time you would need for this distance walking. 🙂

Possibillities without end — if you just catch the importance of these information! Later I will give you more specifications working with QGIS, PostGIS and PostgreSQL … I promise! 🙂

PostGIS (I) – central datapool

Well, I know from my GIS at work, that just viewing objects will not satisfy people for a long time. For that we should create our environment in a lasting way. If data should be available in a company network why not doing it with all the data for our objects and why not ‘really central’. PostGIS is a free extension for PostgreSQL which includes geospatial objecttypes and functions related to them. PostgreSQL is as free as PostGIS. At the same time it is very powerful database system that can keep up with the great commercial products (e. g. Oracle). That means, you can use PostgreSQL as a datapool for all listings and other kind of information in form of a listing, that different people should access at the same time. Centralising information is always a good idea. And if you didn’t knew it before: that is the idea behind a buisness warehouse!

You can install the PostgreSQL software on a Linux- or Windows-machine. I prefer Linux, because you can than reactivate a much more older machine, that you maybe have sort out before. Even for a good database performance you don’t have to use a dual-core pentium somewhat. A machine with 1GB or more memory and a processor with about 1GHz will do. Maybe in your special case less would also do. Try it! The harddisk can be chosen by the amount of information you will store in the database plus the operation system and some clearance.

Finally we start! First of allyou should fetch the actual PostgreSQL sourcecode from their homepage. Depending on your operation system you should follow the according installation instructions. The sourcecode is needed to install PostGIS. So please don’t use binary packages. They won’t do. If all requirements are fullfilled, compiling and installing PostgreSQL should be no problem. After that we do the same with PostGIS. During the compilation I had a problem with a shared object (‘something.so’) that couldn’t be found. I searched for it and copied it into the PostgreSQL-lib-directory. After that it worked … If PostGIS is compiled, the SQL-statements to extend PostgreSQL would also be ready and you can run than like the installationinstructions tell you. Before you should have created a database in PostgreSQL, because you don’t extend the whole databse system, but a single instance of it: your dataspace. Now there is nothing more standing in your way, holding you from using your fresh created geospatial extension of PostgreSQL.

After that I had done the tests of the PostGIS FAQs to be sure that my database has got the extension and was amazed how simple you can access the geospatial data and functions. As a ‘help programmer’ there are a lot of SQL-statements coming into my mind, which I will test next time with my demo data.

An advice at the end: if you’d like to comfortably access PostgreSQL from a Windows-based machine, you should install PgAdmin III . It’s a kind of gui and admintool in one. And for all that cannot be devorced from MS Access there should be an ODBC-driver installed on their machines which can establish the neede connection to the PostgreSQL-backend.

And now you should do some first steps in QGIS connecting with PostGIS.

Free GIS or The value of a GIS

Many free GIS have been developed and enhanced in the last years beside the commercial products. Normally they are available as OpenSource. I can hear you scream and shout: “Oh my God! No more OpenSource!”.

That a system like GRASS was developed by the US Army plays a dicisive role. You can see at first view, that it was not developed to look nice, but the functions underneeth for analyzing bitmaps are unbeatable. GRASS is successfully used for the simulation of flood and forest fire. With the sampledata you can get a good overview what you can do with this system.

For my purposes systems like Jump/OpenJump and Quantum GIS are better suitable because they are vector-based. The features of any objects are shown in listings for one or a group of objects. Both systems have a Java-based gui. They handle data in an own format or one can save a layer as an ESRI shapefile, to mentione a well-known format. Further on they can both be connected to PostGIS (this is even true for GRASS) to read and write data in special tables.The objectfeatures can than be listed in the application. The connection is established by the GIS. The layers will be displayed in the gui. Datasets can also be limited by SQL-statements, which you can use during connection-process, to filter the objects you really need out of the whole.

I will come back to the database connection later on. Connecting to an external database gives also benefit and flexibilty to the non-GIS user. This point is also true for the commercial products, which have software interfaces for different database systems, too. The user can access an datapool which is maintenanced and administrated by a central department. To do so, the user does not have to use GIS. Pure alphanumeric data can often be accessed in an easier and faster way using special reportingtools. With the PostgreSQL-extension PostGIS you can do a spatial query without using a GIS — if you only know the correct syntax. The thing about GIS is identification and showing of data which cannot be brought in this dependency by using listings and tables. Listings and tables are good use if you want to analyze alphanumeric data. But every time an object comes along with geospatial information you can only get full potential if you identify and analyze the spatial relation.

At the moment I’m working on real estate information, like land parcels, buildings and houses. I’m in charge of the GIS. Beside our SAP, which is surely first choice for housing industry when it comes to interests based on the needs of tenants and buildings, the GIS is of real great use. Nearly 95% of information in this buisiness is geospatial referenced, so all of these information can be viewed in the application related to a single object. Based on information about land parcels, buildings and maps more information dealing with special contracts, debits and special features can be displayed at their correct location. The user can see at once what to consider about the objects. A topic which has gotten weight over the last years — in germany — is the correct billing of e. g. cutting grass and trees in the additional expense for the tenant. That means billing of all the work, that is done in the backyard and in front of the house with the plants, grass and bushes, done by an external company and not the real estate company itselfe.

All this information are — as soon as they are entered into the GIS — available in the system and can be part of a report. Additional expenses could not be prooved by the companies for years. Now the GIS knows areas and lenghts precisely. These data can be processed in internal or external applications to get the amount of expenses for a whole town or just for a single tenant.

By connecting to an external database the user can collect some information in GIS and extend it by non-GIS information that are also in this database. This is the only way to create a qualified report for objects that have a spatial dependency. As long as our data is located in one street, we can handle their dependency with a normal database. But a GIS gives you more flexibility for reports that apply to an area which you cannot describe with alphanumerical information. Economic units, quarters, colonies or related information can be added to the baseinformation of an object without problems. But to get them you have to look at a map and if you try to get a subset of it or a spanning amount they may be incapable.

I stop my post at this point and will go on with some more details and examples on single topics of the things written above.

Your GIS – the unknown being

Well … many people doesn’t care about what a GIS is. Many of them do not even had any contact with that kind of software. Nevertheless there surely are a lot of people which could use it for their advantage.

But one after the other!

GIS is the abbreviation for Geographic Information System. Such a software is always used where information is related to a geographic point or area (geospatial data). These systems were originally used to represent pipelines or buildings in the industry or real estate sector. With the development of the systems they could also be used for planning projects and give some strategical evidence. Systems like GE Smallworld GIS were classic representatives for the group of asset management, which can today be used for strategical use or support as well. A system like Grass was born in research. Developed by the US Army, it has a lot of tools for working with bitmaps (e.g. air photographs). The main use of these systems points out that there are several GIS for different use. On one hand you have a verctor-based approach with bitmaps just for orientation. On the other hand you have high-resolution bitmaps with information for every pixel and vectors and new bitmaps only as an result of complex queries.

Smallworld as a commercial product comes along with a great expense factor. Licenses and support are not cheap. Instead of that youget an very professional system which accomplishes nearly every wish one could have. The rest can be programmed! 🙂 OpenSource and freeware can even be interesting for you, because of the great amount of mature functions and the possibility to connect the GIS with a free and capable database system. And that works even, if you have a lot of data and maps.

With one of the next posts I will go more in depth.

A GIS usually comes along with a grafical workbench, called a gui. The user or admin can use this to handle the systems and it’s functions. A lot of the systems have an AddOn or connection for a webbased desktop. Smallworld comes with a standalone webservice other systems can be connected to the UNM mapserver.
Every system works with an internal or external database (e.g. Oracle, PostGIS, MySQL) to manage the alphanumeric data. But it has not to be a real database. Some systems work with ESRI shapefiles or GML/XML-files. It’s difficult to connect to Smallworlds internal database system from outside the GIS. Other systems use external database systems like PostGIS and Oracle spatial. At the point where a number of users should use the same data, the system should be a client-server-application to devide data (backend) and application (frontend).

The next posts bring hints and approaches to the interested reader — hopefully you — to point out the differences between the systems.

For every company with geospatial data there is a great benefit to have a closer look at these systems for making the work with their data more easier. No existing database has to be thrown away and no data has to be kept redundant. With a clever integration the freind of MS Access can still use its queries and reports. And VBA programmed macros can also be used after it. The other users get a noticeable surplus in terms of increased efficiency. Even if I always use this term related to computers with an unpleasant feeling. (Quote: “Computers help us with problems, we would’nt have without them.”)

If you have a look at the so called geomarketing [in engl. it’s Business GIS but it doesn’t mean exactly the same] you should find no serious company which has no GIS — maybe it’s integrated into their reportingtool(s). But later more …

Setup

Meanwhlie I’m working with GIS for about ten years and have made a lot of experiences. So I would like to put some of my ideas, projects and maybe interesting thoughts in this blog.

When I write about commercial GIS products, I will write about GE Smallworld GIS (yet still version 3). That’s because I’m working with it on my job. Beside that I will write about free or open-source products like GRASS, Quantum GIS and Jump/OpenJump. In this context I have to write about databases like PostgreSQL/PostGIS and maybe some MySQL-stuff.

Surely there will be some writing about topics that don’t have to do with any of the above.

For I’m german, you will please excuse my errors and I hope you can get the sense of my posts. 😉
Don’t hesitate to write a comment. Your opinion is always welcome.

Important Note!

Talking that much about GE Smallworld GIS has nothing to do with some kind of advertisment and any profit for me, selling this product. I do not sell this product and I do not have any benefit from doing avdvertising. The same is with the other product.