Mapserver

From PyWPS
Jump to: navigation, search

PyWPS and UMN MapServer

PyWPS (SVN version) has UMM Mapserver support, for ComplexData outputs. If a client requests a ComplexData output as reference then PyWPS will not send the link to the standard wpsoutput folder, but a link to a Mapserver OGC web service where the data is stored. According to the data type the link will point to a WMS, WFS or WCS service.

Later the client, can parse the URL of resulting ComplexValue output e.g. instead of having GeoTIFF file (result of the calculation), obtained from the WCS, it can request PNG file, via WMS.

Requirements

UMN MapServer support in requires the following packages:

python-mapscript
python-gdal
python-pyproj
mapserver

Initial development was based on mapserver5, currentely PyWPS is being developed using mapserver6. Even using packages it is necessary to make the mapserv object available has a web resource (see: mapserv object).

Advice: due to the changes in pywps code and mapserver it's advisable to use the SVN version of pywps and mapserver 6.0.1, other versions of mapserver and pywps maybe functional but without warranties.

Source code compilation

If using mapserver source code, the WCS and WFS support have to be turned on in the configure command e.g:

CFLAGS="-march=native -mtune=generic -O3 -pipe" CXXFLAGS="${CFLAGS}" ./configure --with-postgis=/usr/local/pgsql/bin/pg_config 
--with-gd  --with-proj --without-threads --with-geos --with-ogr --with-gdal 
--with-wfs --with-wcs --with-wmsclient --with-wfsclient --with-sos 
--with-curl-config --with-xml-mapfile --with-cairo

The example above uses specif GCC optimization flags and extra mapserver functionalitis like CAIRO rendering support.

The make command will compile the code. The python module is not automatically built and the user has to acess the python folder and build it:

cd ./mapscript/python
python build
sudo python install

The module can be tested by doing a simple import:

python -c "import mapscript"

Note: The compilation exampled used GDAL/OGR 1.9.0, GEOS 3.3.2 and proj-lib 4.7.1.

Source code compilation (detailed)

The following compilation septs have been suggested by Luis de Sousa in the pywps mailing list and refer to an install in Ubuntu:

  • Download the MapServer 6.0.1 tarball and expand it to ~/temp/mapserver-6.0.1
  • Previous install have to be totally removed e.g:
$ sudo rm /usr/lib/cgi-bin/mapserv
$ sudo rm -R /usr/local/lib/python2.7/dist-packages/MapScript-6.1_dev-py2.7-linux-x86_64.egg
$ sudo rm /usr/lib/python2.7/dist-packages/mapscript.pyc
  • Acess the source code
cd ~/temp/mapserver-6.0.1
  • Compile mapserver
./configure --with-ogr=/usr/local/bin/gdal-config --with-gdal=/usr/local/bin/gdal-config --with-wfsclient --with-wmsclient 
--enable-debug --with-curl-config=/usr/bin/curl-config --with-proj=/usr/local --with-tiff --with-gd=/usr/local/ --with-jpeg 
--with-freetype=/usr/ --with-threads --with-wcs --with-libiconv=/usr --with-geos=/usr/local/bin/geos-config --with-libiconv=/usr 
--with-xml2-config=/usr/bin/xml2-config --with-sos 

Note: Missing PostGIS support. Check the install instructions for details.

$ make
$ sudo cp mapserv /usr/lib/cgi-bin
$ cd mapscript/python/
$ swig -version 
  • Swig must be installed before proceeding
$ python setup.py build
$ sudo python setup.py install
  • Check if Mapscript is operational
$ python
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mapscript
>>> map = mapscript.mapObj()
>>>

mapserv object

Mapserver output requires that the mapserv object is accessible through a URL call. The mapserv object will be executed based on the stored map file object and the specific WCS/WFS parameters indicated in the request.

mapserv object should be inside (or linked to) the server's executable folder (normally in cgi-bin). Apache executes scripts/objects based on their extension name, so it is advisable to include some an extension name to the mapserv object (normally .cgi please check apache document in case of doubt).

Assuming that mapserv object is in the system bin (or was copied there from the folder where the source code was compiled), and that Apache will execute any object with extension CGI in cgi-bin:

ln -s /usr/sbin/mapserv /var/www/cgi-bin/mapserv.cgi

Calling the mapserv object:

http://localhost/cgi-bin/mapserv.cgi

should report the following error:

loadMap(): Web application error. CGI variable "map" is not set.

PyWPS configuration

PyWPS configuration file contains a mapscript section that requires the correct URL to the mapserv object e.g:

:
[mapserver]
mapserveraddress=http://localhost/mapserv.cgi
:

PyWPS Usage

Mapserver support is set by passing a mapserver attribute (useMapscript) equal to true to the desired ComplexData Output inside I/O process definitions:

class Process(WPSProcess):
 def __init__(self):
 WPSProcess.__init__(self,
 identifier="mapserverOutput"

 <process definitions and metadata>

 self.imageOut=self.addComplexOutput(identifier="tiffOut",
      title="GeotTiff result",
      useMapscript=True,
      formats = [
           {"mimeType":"image/tiff"},
           {"mimeType":"image/png"}
           ])
 <more code>

The required format is used to decide what sort of WxS service will be generated, WCS if mimeType is GeoTIFF and similar, WFS for format "xml" or "text" or WMS for PNG, JPEG, and GIF formats.

Output URL

An complexOutput should be something like this:

http%3A//localhost/mapserv.cgi%3Fmap%3D/var/www/html/wpsoutputs/pywps-cd506dc8-5ef6-11e1-ba82-a4badbfc32f4.map
%26SERVICE%3DWCS%26REQUEST%3DGetCoverage%26VERSION%3D1.0.0%26COVERAGE%3Doutput%26CRS%3Depsg%3A4269
%26BBOX%3D630000.0%2C215000.0%2C645000.0%2C228500.0%26HEIGHT%3D450%26WIDTH%3D500%26FORMAT%3Dimage/tiff

It's necessary to unquote the string into something like:

http://localhost/mapserv.cgi?map=/var/www/html/wpsoutputs/pywps-cd506dc8-5ef6-11e1-ba82-a4badbfc32f4.map
&SERVICE=WCS&REQUEST=GetCoverage&VERSION=1.0.0&COVERAGE=output&
CRS=epsg:4269&BBOX=630000.0,215000.0,645000.0,228500.0&HEIGHT=450&WIDTH=500&FORMAT=image/tiff

The urllib module provides an unquote functionaly that can be very practical:

python -c "import urllib;print urllib.unquote('<your url>')"

The complete URL has to be unquoted, it's not enough to just unquote the http, this will raise a 404 not found error. If the section concerning mapserver is not properly unquoted the following error will be raised:

msLoadMap(): Regular expression error. MS_DEFAULT_MAPFILE_PATTERN validation failed. msEvalRegex(): 
Regular expression error. String failed expression test. 

Working with CRS

Until PyWPS SVN r1297 the default CRS was EPSG:4326 and it wasn't possible to apply CRS manipulation to the returned WCS/WFS service url. Currentely it is possible to set multiple EPSG values and make CRS transformations

In the new code, data CRS is determined from 3 points in the code:

  • PyWPS configuration file
  • From ComplexOutput's attribute projection
  • Determined automatically from data using GDAL/OGR

The choice of EPSG goes through a decision tree:

  1. Use of pywps.cfg
  2. Configuration file overwritten by projection attribute in ComplexOutput
  3. If there is no CRS information then try to get it from data....(it normally works...)

The EPSG codes are normally defined in the epsg file (/usr/share/proj/epsg), if a used code is not located in the epsg file the following error will rise:

<ows:Exception exceptionCode="NoApplicableCode" locator="None">
    <ows:ExceptionText>Process executed. Failed to build final response for output [buffer]: 
      msProcessProjection(): Projection library error. no options found in 'init' file
    </ows:ExceptionText>
</ows:Exception>

Configuration file

EPSG codes are set in the mapserv section using variable projs [mapserver]

projs=epsg:4326,epsg:102067,epsg:3059,epsg:900913

Using complexOutput

ComplexOutput object contains a "projection" property that accept a projection string identical to the one used in the PyWPS configuration file

e.g: in the buffer.py

:
 self.output =self.addComplexOutput(identifier="buffer", title="Buffered data",formats = [{'mimeType': 'text/xml', 'schema': 'http://schemas.opengis.net/gml/2.1.2/feature.xsd', 'encoding': 'UTF-8'}],useMapscript=True)
:
self.output.projection="epsg:4326,epsg:102067,epsg:3059,epsg:900913"

Automatic detection

Should the projs value in pywps config be empty and also the self.output.projection attribute (this is the standard)

[mapserver]
projs=

The code will determine the data's projection using GDAL/OGR (using the spatialReference object). This approach is not "bullet proof" and should be avoided

Getting data in a different CRS

WFS1.0.0 does not support CRS transformations, therefore the URL (ComplexOutput) has to be changed into WFS1.1.0. Doing a "getCapabilities" to the WFS1.1.0 service, a list of supported CRS will be outputed:

<OtherSRS>urn:ogc:def:crs:EPSG::900913</OtherSRS>

For example to obtain the data in EPSG:900913 (google projection) we have to use the srsName parameter:

http://localhost/mapserv.cgi?
map=/var/www/html/wpsoutputs/pywps-3af0c93a-645c-11e1-ad6c-a4badbfc32f4.map&
SERVICE=WFS&
REQUEST=GetFeature&
VERSION=1.1.0&
TYPENAME=buffer
&srsName=EPSG:900913

For WCS it's the same thing,considering the following output:

SERVICE=WCS&
REQUEST=GetCoverage&
VERSION=1.0.0&
COVERAGE=output&
CRS=epsg:32119
&BBOX=630000.0,215000.0,645000.0,228500.0
&HEIGHT=450&WIDTH=500&FORMAT=image/tiff

By changing the request to describeCoverage it is possible to obtain the available CRS.

   <requestResponseCRSs>EPSG:32119</requestResponseCRSs>
   <requestResponseCRSs>EPSG:4326</requestResponseCRSs>
   <nativeCRSs>EPSG:32119</nativeCRSs>

Also for each CRS there is a different BBOX numbers:

<gml:Envelope srsName="EPSG:4326">
<gml:pos>-78.7746204948163 35.6875072977337</gml:pos>
<gml:pos>-78.6083031766916 35.8096093834499</gml:pos>
</gml:Envelope>

To obtain the image as EPSG:4326 it is necessary to change the CRS and BBOX request, as follows:

SERVICE=WCS&
REQUEST=GetCoverage&
VERSION=1.0.0&
COVERAGE=output&
CRS=epsg:4326
&BBOX=-78.7746204948163,35.6875072977337,-78.6083031766916,35.8096093834499
&HEIGHT=450&WIDTH=500&FORMAT=image/tiff

Example

For example the ogrbuffer process acepts a GML2.1.2 (e.g [1]) content and will generate a buffer around vectorial structure, the final output is given as a WFS service that can be copy/past into QGIS (after being unquoted)


PywpsOutputWFS.png


Error message

This is a tentative compilation of common mapserver error messages and problems

msProcessProjection(): Projection library error. no system list, errno: 2

In 90% of the cases the problem is caused by:

  • Misconfiguration of the EPSG file location, or problems reading it
  • Problem in the projection string inside PROJECTION

These problems have been explained in the following posts [2][3]

what is not refered is that any problem with projection string using of web services will raise the same error, for example an incorrect "ows_srs", for example a white space at the begining of the string, for example:

  • "ows_srs" " EPSG:4326 EPSG:102067 EPSG:3059 EPSG:900913"

Will raise an error since mapserver will parse the string and start searching the projection list for a blank string " " --Wikiadmin 16:38, 10 January 2011 (UTC)