Integrating pycsw and GeoNode
Features of CSW used by GeoNode
- Creating and updating CSW records.
- Searching in free form, a set of words in any field.
- Searching by bounding box.
- Support for TC/211 metadata format.
pycsw notes
(including review of https://github.com/GFDRR/geonode/commit/4e8bc00202e5fd4dcc427156564ee46da87a9588)
- http://pycsw.org/
- supports transactions (Insert/Update/Delete) and Harvest
- must be enabled; (list of) IP restricted access
- supports federated search
- supports ISO (must be enabled)
- support INSPIRE (must be enabled)
- passes 103/103 OGC CITE tests
- search support
- full text (csw:AnyText)
- spatial and aspatial ( http://pycsw.org/docs/introduction.html#supported-filters)
- repository
- supports sqlite3, mysql and postgresql
- postgresql setup requires setting up plpythonu functions ( https://sourceforge.net/apps/trac/pycsw/browser/trunk/sbin/setup_db.py#L87)
- data model
- data model loosely based on ISO 19115, full model can be seen at https://sourceforge.net/apps/trac/pycsw/browser/trunk/sbin/setup_db.py#L75
- django integration: this would need to investigated/implemented. pycsw repository is self contained, gets setup with https://sourceforge.net/apps/trac/pycsw/browser/trunk/sbin/setup_db.py. Underlying transaction operations via sqlalchemy can be reviewed at https://sourceforge.net/apps/trac/pycsw/browser/trunk/server/repository.py
- supports sqlite3, mysql and postgresql
- privileges/groups/roles
- pycsw has no implementation of this. CSW, by nature, leaves this to the implementation. By default, everything published to pycsw is published to all
- UUIDs are required for Transaction Insert'ed records. We could extend by assigning a server generated UUID
The most pertinent change would be django integration. This would affect how the repository is setup and how it is interacted with. Options:
- create a django based pycsw plugin
- have a configuration option in pycsw to signal django based interaction
- update CRUD functions to be django based if requested (repository_model=internal|django|geonode)
- if most of the interaction is CSW request based, then just create db in django-site/settings.py, and ref db connection from pycsw configuration. pycsw would still use sqlalchemy as the ORM/API. Abstraction would be required between data model specifics (column names).
Metadata Links
Background
The OGC CSW 2.0.2 standard provides a discovery mechanism of geospatial metadata. For the HTTP protocol binding, the baseline information model is the Dublin Core (DC) Metadata Element Set. Additional profiles (such as APISO 1.0) exist to support specific information models (i.e. ISO 19139:2007) and output schemas.
Both ISO and DC allow for the advertisement of HTTP links. ISO's CI_OnlineResource construct allows for a certain level of granularity to express links (linkage, protocol, applicationProfile, name, description, function). Dublin Core allows for dct:references [0..*], which has a scheme attribute. There is no further granularity within dct:references.
Implementations have used dc:URI, however dc:URI is a non-standard element of DC record model in CSW 2.0.2
A finer model is required to communicate different types of links (opaque, binding parameters, etc.).
Motivation
Geonode, as a CSW client, requires the ability to identify various online linkages and be aware of their types/model to be able to process them accordingly (i.e. opaque, binding parameters, etc.).
This is possible given the ISO model, however it would be valuable to have an lightweight approach within DC that is inspired by ISO's CI_OnlineResource as well as fit within DC's dct:references element.
Proposal
Express ISO's CI_OnlineResource model in JSON using in dct:references/@scheme.
Example: <dct:references scheme="{'protocol': 'OGC:WMS-1.1.1-http-get-map', 'name': 'base%3Adistrits', 'description': 'distrits (JPEG Format)', 'linkage': 'http://localhost:8001/geoserver/wms'}">http://localhost:8001/geoserver/wms?layers=base%3Adistrits&width=620&bbox=-86.144%2C8.041%2C-82.555%2C11.221&service=WMS&format=image%2Fjpeg&srs=EPSG%3A4326&request=GetMap&height=550</dct:references>
Where:
- dct:references.text: the full URL to the resource
- dct:references/@scheme: a JSON representation of ISO's CI_OnlineResource:
- name: the name of the link. In the case of an OWS, this should be the resource (layer|featureType|coverage) name
- description: a human-readable description.
- protocol: the type assigned to the URL, from the GeoNetwork gmd:online/gmd:CI_OnlineResource/gmd:protocol enumeration [1]
- linkage: the URL of the resource. In the case of an OWS, this should be the service endpoint with query parameters
[1] protocol enumeration and description
| protocol | description |
| ESRI:AIMS--http--configuration | ArcIMS Map Service Configuration File (*.AXL) |
| ESRI:AIMS--http-get-feature | ArcIMS Internet Feature Map Service |
| ESRI:AIMS--http-get-image | ArcIMS Internet Image Map Service |
| GLG:KML-2.0-http-get-map | Google Earth KML service (ver 2.0) |
| OGC:CSW | OGC-CSW Catalogue Service for the Web |
| OGC:KML | OGC-KML Keyhole Markup Language |
| OGC:GML | OGC-GML Geography Markup Language |
| OGC:ODS | OGC-ODS OpenLS Directory Service |
| OGC:OGS | OGC-ODS OpenLS Gateway Service |
| OGC:OUS | OGC-ODS OpenLS Utility Service |
| OGC:OPS | OGC-ODS OpenLS Presentation Service |
| OGC:ORS | OGC-ODS OpenLS Route Service |
| OGC:SOS | OGC-SOS Sensor Observation Service |
| OGC:SPS | OGC-SPS Sensor Planning Service |
| OGC:SAS | OGC-SAS Sensor Alert Service |
| OGC:WCS | OGC-WCS Web Coverage Service |
| OGC:WCS-1.1.0-http-get-capabilities | OGC-WCS Web Coverage Service (ver 1.1.0) |
| OGC:WCTS | OGC-WCTS Web Coordinate Transformation Service |
| OGC:WFS | OGC-WFS Web Feature Service |
| OGC:WFS-1.0.0-http-get-capabilities | OGC-WFS Web Feature Service (ver 1.0.0) |
| OGC:WFS-G | OGC-WFS-G Gazzetteer Service |
| OGC:WMC-1.1.0-http-get-capabilities | OGC-WMC Web Map Context (ver 1.1) |
| OGC:WMS | OGC-WMS Web Map Service |
| OGC:WMS-1.1.1-http-get-capabilities | OGC-WMS Capabilities service (ver 1.1.1) |
| OGC:WMS-1.3.0-http-get-capabilities | OGC-WMS Capabilities service (ver 1.3.0) |
| OGC:WMS-1.1.1-http-get-map | OGC Web Map Service (ver 1.1.1) |
| OGC:WMS-1.3.0-http-get-map | OGC Web Map Service (ver 1.3.0) |
| OGC:SOS-1.0.0-http-get-observation | OGC-SOS Get Observation (ver 1.0.0) |
| OGC:SOS-1.0.0-http-post-observation | OGC-SOS Get Observation (POST) (ver 1.0.0) |
| OGC:WNS | OGC-WNS Web Notification Service |
| OGC:WPS | OGC-WPS Web Processing Service |
| WWW:DOWNLOAD-1.0-ftp--download | File for download through FTP |
| WWW:DOWNLOAD-1.0-http--download | File for download |
| FILE:GEO | GIS file |
| FILE:RASTER | GIS RASTER file |
| WWW:LINK-1.0-http--ical | iCalendar (URL) |
| WWW:LINK-1.0-http--link | Web address (URL) |
| WWW:LINK-1.0-http--partners | Partner web address (URL) |
| WWW:LINK-1.0-http--related | Related link (URL) |
| WWW:LINK-1.0-http--rss | RSS News feed (URL) |
| WWW:LINK-1.0-http--samples | Showcase product (URL) |
| DB:POSTGIS | PostGIS database table |
| DB:ORACLE | ORACLE database table |
| WWW:LINK-1.0-http--opendap | OPeNDAP URL |
| RBNB:DATATURBINE | Data Turbine |
| UKST | Unknown Service Type |
Extensions to protocol enumerations and descriptions
| WWW:LINK-1.0-http--image-thumbnail | Web image thumbnail (URL) |
Examples
WMS Layer with only binding information <dct:references scheme="{'protocol': 'OGC:WMS-1.1.1-http-get-map', 'name': 'base%3Adistrits', 'description': 'distrits (JPEG Format)', 'linkage': 'http://localhost:8001/geoserver/wms'}">http://localhost:8001/geoserver/wms?layers=base%3Adistrits&width=620&bbox=-86.144%2C8.041%2C-82.555%2C11.221&service=WMS&format=image%2Fjpeg&srs=EPSG%3A4326&request=GetMap&height=550</dct:references>
Thumbnail <dct:references scheme="{'protocol': 'WWW:LINK-1.0-http--image-thumbnail', 'name': 'base%3Adistrits', 'description': 'Web image thumbnail (URL)', 'linkage': 'http://localhost:8001/geoserver/wms'}">http://localhost:8001/geoserver/wms?layers=base%3Adistrits&width=20&bbox=-86.144%2C8.041%2C-82.555%2C11.221&service=WMS&format=image%2Fjpeg&srs=EPSG%3A4326&request=GetMap&height=20</dct:references>
