PyCallGraph

From PyWPS
Jump to: navigation, search

Introduction

PyCallGraph is used to generate graphic representations of code being run, indicating the number of calls to a function and the time spent. This is very practical to identify bottlenecks and to understand how code is run.

In a nutshell PyCallGraph uses a color ramp to indicate the major bottle necks, from light grey to dark pink. This makes it easy to identify major sections of the code and how its logic sequence. Also it is important to define the number of levels to be exposed, in the output graphic, the default option is to follow all the calls no mater what, this will end up with a massive graphic containing hundreds of calls to insignificant function. In this case a level 4 was used to generate the images

For information on how to use PyCallGraph please check the official page here

PyWPS methodology

Inside the wps.py, at the beginning it was included the following code:

import pycallgraph
filter_func = pycallgraph.GlobbingFilter(max_depth=4)
pycallgraph.start_trace(filter_func=filter_func)

and at the end of the script the stop trace call:

pycallgraph.make_dot_graph('<path_to_save_tile>')

PyWPS results (PyCallGraph)

PyWPS doesn't have major bottle necks and the majority of the time is spent running the classes that work together as an assemble factory, staring with parsing the request and then calling the specific classes according to the needs.

GetCapabilities

The following request was performed on the bash:

$ python wps.py "request=GetCapabilities&service=WPS"


GetCapabilities thumbnail PyWPS calls are mainly related to the GetCapabilities Class from the Wps package, as shown in detain in the image below


Pywps GetCapabilties 4 detail.png

The class GetCapabilities will continue and in the major graphic it indicates that the class is taking the majority of the time in the initProcess() method. This method is responsible for gathering process information, therefore more processes, more time to generate the WPS response.

DescribeProcess

The following request was performed on the bash for the ultimate question process (a method used to test async calls):

$ python wps.py "request=DescribeProcess&service=WPS&
version=1.0.0&
identifier=ultimatequestionprocess"

For those familiar with the "Hitchhiker's guide to the galaxy" will know that the answer is: a LiteralData with value 42


Pywps DescribeProcess Ultimate 4 tb.png In closer detail:


Pywps DescribeProcess Ultimate 4 detail.png

PyWPS will call the performRequest class then DescribeProcess that will initiate the process class from where it will gather the information necessary to describe the process. If we increase the level of analysis, it'ss possible to arrive to the method that is the delay.


Pywps DescribeProcess Ultimate 4 detail extra.png

The _initFromDirectory is a private method locate in the parent class of DescribeProcess . This method is responsible by search the system path and loading the processes, this method also logs any problems while loading the classes.

Execute

The UltimateQuestion process, will take more or less 100 seconds to give an answer, of course the process it self will be classified as the major bottleneck


Pywps Execute Sync Ultimate 4 tb.png

The execute continues the same class logic, starting with class pyWPS calling the method performRequest()


Pywps Execute Sync Ultimate 4 detail 1.png


and then following to the Execute class that will use the executeProcess() method to start the ultimate question process class


Pywps Execute Sync Ultimate 4 detail 2.png


If the the ultimatequestion process is called with a status=true, then the situation would be the same with a very small over head caused by the status update.

Exception report

Sometimes things go wrong....for example a request to a process that is not found in the process list:

python ./wps.py "request=DescribeProcess&service=wps&version=1.0.0&identifier=unknown"

PyWPS has implemented the OGC's Exception Report as an extension of the Exception native python class, therefore it is a bit hard to track things inside PyCallGraph, since the Exception classes are initiated when the script is run, independent to the fact that an error has been risen.


Pywps Exception tb.png

Since the DescribeProcess class is taking less time, the relative proportion of time taken to run the class changes (despite the absolute time value being the same) making the Exception classes more "colourful"


Pywps Exception detail.png


The advantage of using python Exceptions is that a OGC Exception can be raised anywhere in the code (as necessary) and they will back propagate to the major script (wps.py) and then outputted to the server.

Final remarks

PyWPS doesn't have major bottlenecks or problems with the current philosophy of "factory assemblage" according to the request being done. Module loading is a time consuming step and this is common in the "Python world"

Editor:{$revisioneditor} Date: {$revisiondate}