Re: CivicAccess-discuss Digest, Vol 54, Issue 34

[Joe Murray]

If I could echo a few points others have made in my own way:

1. For performance reasons I think I would like to get the shape files you have used and make them available for various groups to load them into CiviCRM. One benefit will be to do the calculation locally rather than over the wire via JSON. Another will be to reduce the load on your server when we need to calculate the wards and ridings for databases with tens and hundreds of thousands of records.
   
1. This will require converting them to MySQL's geo format. But since 5.1.26 or so I think, MySQL has provided support for accurate calculation of whether a point is within a polygon (before that it was an inaccurate but quick calculation of whether the point was within the minimum bounding rectangle).
2. So I'm looking forward to seeing you publish the sources of the shapefiles and their licenses.
   
2. In some cases the licenses will likely not permit you to share the shapefiles. I imagine that some of my clients who want this information for advocacy purposes would be happy to pay for a higher usage than 60 requests per minute during bulk encoding, since otherwise it will take a couple of days to deal with all of their data (e.g. I recently geocoded 175k addresses for federal riding and where appropriate Toronto ward). With a bit of advance coordination, this might allow you to bump up the capabilities of your vps for a day to handle the increased load, and heh, maybe even subsidize more development or support. ;) A common provision in licenses for this sort of service is to say that there is a cap but if you want to exceed the cap to contact the provider, ie you folks.
3. Without wanting to be persnickety, postal code centroids are not generally accurate enough. A few apps I have seen allow people to see the several ridings that their address may be a part of.
   
1. Several years ago, using the voters list data provided for the country to political parties there were a few hundred thousand postal codes that appear in more than one riding if I recall correctly, and the number of ridings could go up to 7 for a particular postal code (I think it was a rural one in the Atlantic provinces for a highway). When I tracked down the quality of the data in the PCFRF through Elections Canada to StatsCanada they gave me a long explanation for why this was so. I bet it is still the case.
   

Congratulations and thank you so much!!


Joe Murray, PhD
President, JMA Consulting
[hidden email]
skype JosephPMurray twitter JoeMurray
416.466.1281

On Tue, Jan 31, 2012 at 12:00 PM, <[hidden email]> wrote:

Send CivicAccess-discuss mailing list submissions to
       [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
       http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
or, via email, send a message with subject or body 'help' to
       [hidden email]

You can reach the person managing the list at
       [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of CivicAccess-discuss digest..."


Today's Topics:

  1. Re: Announcing Represent, an open database of political
     districts and representatives (Cory Horner)
  2. Re: Announcing Represent, an open database of political
     districts and representatives (Michael Mulley)


----------------------------------------------------------------------

Message: 1
Date: Mon, 30 Jan 2012 12:58:23 -0800
From: Cory Horner <[hidden email]>
To: civicaccess discuss <[hidden email]>
Subject: Re: [CivicAccess-discuss] Announcing Represent,        an open
       database of political districts and representatives
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=windows-1252

Since our APIs are overlapping, i'm considering shutting mine down since you guys have more coverage (I only have federal, bc, and ontario so far).

- can you add an XML output format?
- can you add an option to return nearby ridings within a certain radius (because the postal code centroid is inaccurate)?
- can people contact you to avoid the 60 request/second limit?

If you can add those... I can punt my users to you and spend my time elsewhere...

Thanks,
Cory.
----
http://howdtheyvote.ca/



On 2012-01-30, at 8:44 AM, James McKinney wrote:

> Open North is proud to announce Represent, an API (web service) to federal, provincial and municipal political districts and representatives across Canada. To see a demo and read the API docs, visit: http://represent.opennorth.ca/
>
> Represent establishes a foundation upon which other projects may be built - a necessary piece of infrastructure which we hope will enable useful and innovative features and products. It is free to use and open-source.
>
> This open database was built by James McKinney and Michael Mulley, with important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel, Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The project takes inspiration from mySociety?s MapIt and code from the Chicago Tribune?s Boundary Service.
>
> We currently have electoral districts and representatives for all provinces and Canada, and the ward boundaries of ten cities. We have postal code correspondences for Ontario (and soon Quebec) and postal code centroids from GeoCoder.ca.
>
> But we could have much more. To keep this project sustainable, we are asking for your help. Things you can help with:
>
> Add more representatives
> - Write scrapers (in PHP, Ruby, or Python) to get the elected officials for more cities
> - Ask provincial Municipal Affairs ministries for municipal elected officials in Excel format
>
> Get more boundaries
> - Create a list of all municipalities with ward boundaries
> - Ask them to provide their ward boundaries under an open license
>
> Free the postal code
> - Ask your province or city for a postal code to electoral district concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
>
> Maintain the database
> - Monitor scrapers to make sure we always provide the latest, most accurate data available
> - Write an app to track electoral calendars, so we can plan updates and prepare scrapers
> - Donate to Open North so a staff member can improve and maintain this service part-time
>
> Comments are very welcome, and please let us know if you've built something using the API.
>
> James & Michael
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss



------------------------------

Message: 2
Date: Mon, 30 Jan 2012 16:56:55 -0500
From: Michael Mulley <[hidden email]>
To: civicaccess discuss <[hidden email]>
Subject: Re: [CivicAccess-discuss] Announcing Represent, an open
       database of political districts and representatives
Message-ID:
       <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi Cory,

I think a lot of people use & are grateful for your API -- I know I am --
and I'd certainly like to see it stay around.

I've got no plans to add XML and James & I originally planned this a JSON
service, though we might change our minds if several people really want XML.

Nearby: played with this briefly but the range queries I was trying were
making PostGIS slow to a crawl, since we store everything in lat/lon
geometry columns. This is certainly worth adding to our issues list.

As to rate limits, this is a brand-new project & we don't have any kind of
policy yet, but we built this as a tool for people to build on top of, and
we're not gonna stand in the way of legitimate uses without a good reason.

On Mon, Jan 30, 2012 at 3:58 PM, Cory Horner <[hidden email]> wrote:

> Since our APIs are overlapping, i'm considering shutting mine down since
> you guys have more coverage (I only have federal, bc, and ontario so far).
>
> - can you add an XML output format?
> - can you add an option to return nearby ridings within a certain radius
> (because the postal code centroid is inaccurate)?
> - can people contact you to avoid the 60 request/second limit?
>
> If you can add those... I can punt my users to you and spend my time
> elsewhere...
>
> Thanks,
> Cory.
> ----
> http://howdtheyvote.ca/
>
>
>
> On 2012-01-30, at 8:44 AM, James McKinney wrote:
>
> > Open North is proud to announce Represent, an API (web service) to
> federal, provincial and municipal political districts and representatives
> across Canada. To see a demo and read the API docs, visit:
> http://represent.opennorth.ca/
> >
> > Represent establishes a foundation upon which other projects may be
> built - a necessary piece of infrastructure which we hope will enable
> useful and innovative features and products. It is free to use and
> open-source.
> >
> > This open database was built by James McKinney and Michael Mulley, with
> important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,
> Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The
> project takes inspiration from mySociety?s MapIt and code from the Chicago
> Tribune?s Boundary Service.
> >
> > We currently have electoral districts and representatives for all
> provinces and Canada, and the ward boundaries of ten cities. We have postal
> code correspondences for Ontario (and soon Quebec) and postal code
> centroids from GeoCoder.ca.
> >
> > But we could have much more. To keep this project sustainable, we are
> asking for your help. Things you can help with:
> >
> > Add more representatives
> > - Write scrapers (in PHP, Ruby, or Python) to get the elected officials
> for more cities
> > - Ask provincial Municipal Affairs ministries for municipal elected
> officials in Excel format
> >
> > Get more boundaries
> > - Create a list of all municipalities with ward boundaries
> > - Ask them to provide their ward boundaries under an open license
> >
> > Free the postal code
> > - Ask your province or city for a postal code to electoral district
> concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
> >
> > Maintain the database
> > - Monitor scrapers to make sure we always provide the latest, most
> accurate data available
> > - Write an app to track electoral calendars, so we can plan updates and
> prepare scrapers
> > - Donate to Open North so a staff member can improve and maintain this
> service part-time
> >
> > Comments are very welcome, and please let us know if you've built
> something using the API.
> >
> > James & Michael
> > _______________________________________________
> > CivicAccess-discuss mailing list
> > [hidden email]
> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pwd.ca/pipermail/civicaccess-discuss/attachments/20120130/4167f5ec/attachment-0001.html>

------------------------------

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

End of CivicAccess-discuss Digest, Vol 54, Issue 34
***************************************************

[Michael Mulley]

Hi Joe,

1. Licence conditions don't currently permit us to redistribute the shapefiles en masse. But we do provide, as part of the API, the source (including a URL when possible) for all the shapefiles, so it's not tremendously difficult to reassemble. And our code, which loads all the shapefiles into a PostGIS table (I believe MySQL's geo support is better than it used to be, but Postgres is still way better in that department), is open source and comes with install instructions.

2. Re the rate limit, yeah, certainly anyone who wants to avoid the cap should contact us. I should add a sentence to that effect to the docs. This is brand new and we haven't figured out any policies yet, but we want anyone who wants to use this to be able to.

3. Agreed about centroids. We mention the inaccuracy in the docs, but maybe it should be more strongly worded? (The first draft of the API had the 'boundaries_centroid' field called 'boundaries_inaccurate' -- maybe it should've stayed that way.) We do provide access to concordances when we have them: try an Ontario postal code, for example. If the PCFRF does end up part of Statscan's open data release -- last I heard this was still unclear -- we'll of course include it.

best

michael

On Tue, Jan 31, 2012 at 12:29 PM, Joe Murray <[hidden email]> wrote:

If I could echo a few points others have made in my own way:

1. For performance reasons I think I would like to get the shape files you have used and make them available for various groups to load them into CiviCRM. One benefit will be to do the calculation locally rather than over the wire via JSON. Another will be to reduce the load on your server when we need to calculate the wards and ridings for databases with tens and hundreds of thousands of records.
   
1. This will require converting them to MySQL's geo format. But since 5.1.26 or so I think, MySQL has provided support for accurate calculation of whether a point is within a polygon (before that it was an inaccurate but quick calculation of whether the point was within the minimum bounding rectangle).
2. So I'm looking forward to seeing you publish the sources of the shapefiles and their licenses.
   
2. In some cases the licenses will likely not permit you to share the shapefiles. I imagine that some of my clients who want this information for advocacy purposes would be happy to pay for a higher usage than 60 requests per minute during bulk encoding, since otherwise it will take a couple of days to deal with all of their data (e.g. I recently geocoded 175k addresses for federal riding and where appropriate Toronto ward). With a bit of advance coordination, this might allow you to bump up the capabilities of your vps for a day to handle the increased load, and heh, maybe even subsidize more development or support. ;) A common provision in licenses for this sort of service is to say that there is a cap but if you want to exceed the cap to contact the provider, ie you folks.
3. Without wanting to be persnickety, postal code centroids are not generally accurate enough. A few apps I have seen allow people to see the several ridings that their address may be a part of.
   
1. Several years ago, using the voters list data provided for the country to political parties there were a few hundred thousand postal codes that appear in more than one riding if I recall correctly, and the number of ridings could go up to 7 for a particular postal code (I think it was a rural one in the Atlantic provinces for a highway). When I tracked down the quality of the data in the PCFRF through Elections Canada to StatsCanada they gave me a long explanation for why this was so. I bet it is still the case.
   

Congratulations and thank you so much!!


Joe Murray, PhD
President, JMA Consulting
[hidden email]
skype JosephPMurray twitter JoeMurray
<a href="tel:[phone]" value="[phone]" target="_blank">416.466.1281

[James McKinney-2]

Thanks for the feedback, Joe!

We provide a link in the API to the page where you can download the shapefile, and a link to the license, e.g. http://represent.opennorth.ca/boundary-sets/federal-electoral-districts/?format=apibrowser The majority of the shapefiles are publicly available on the internet. The rest we got by email.

Can't you convert the shapefiles to MySQL yourself using a tool like shp2mysql? I think it's CiviCRM's responsibility to give you tools for loading geospatial data into its database.

I've added a clarification to contact us if you expect to exceed limits.

We admit the inaccuracy of centroids in the documentation. For provinces that post official concordances between postal codes and boundaries, we use that data instead, e.g. http://represent.opennorth.ca/postcodes/K0A1K0/?format=apibrowser

There are roughly a million postal codes, so I doubt that several hundred thousands appear in more than one riding. According Elections ON, of its lion's share of the postal codes, only 2724 map to multiple electoral districts. http://www.elections.on.ca/en-CA/Tools/ElectoralDistricts/PostalCodeFile.htm The PCCF and PCFRF are notorious for errors. GeoCoder.ca instead uses Canada Post's assignment of postal codes to road segments (a postal code is not a polygon but a mail route) to do its geocoding of postal codes.

Cheers,

James

On 2012-01-31, at 12:29 PM, Joe Murray wrote:

> If I could echo a few points others have made in my own way:
>
> • For performance reasons I think I would like to get the shape files you have used and make them available for various groups to load them into CiviCRM. One benefit will be to do the calculation locally rather than over the wire via JSON. Another will be to reduce the load on your server when we need to calculate the wards and ridings for databases with tens and hundreds of thousands of records.
> • This will require converting them to MySQL's geo format. But since 5.1.26 or so I think, MySQL has provided support for accurate calculation of whether a point is within a polygon (before that it was an inaccurate but quick calculation of whether the point was within the minimum bounding rectangle).
> • So I'm looking forward to seeing you publish the sources of the shapefiles and their licenses.
> • In some cases the licenses will likely not permit you to share the shapefiles. I imagine that some of my clients who want this information for advocacy purposes would be happy to pay for a higher usage than 60 requests per minute during bulk encoding, since otherwise it will take a couple of days to deal with all of their data (e.g. I recently geocoded 175k addresses for federal riding and where appropriate Toronto ward). With a bit of advance coordination, this might allow you to bump up the capabilities of your vps for a day to handle the increased load, and heh, maybe even subsidize more development or support. ;) A common provision in licenses for this sort of service is to say that there is a cap but if you want to exceed the cap to contact the provider, ie you folks.
> • Without wanting to be persnickety, postal code centroids are not generally accurate enough. A few apps I have seen allow people to see the several ridings that their address may be a part of.
> • Several years ago, using the voters list data provided for the country to political parties there were a few hundred thousand postal codes that appear in more than one riding if I recall correctly, and the number of ridings could go up to 7 for a particular postal code (I think it was a rural one in the Atlantic provinces for a highway). When I tracked down the quality of the data in the PCFRF through Elections Canada to StatsCanada they gave me a long explanation for why this was so. I bet it is still the case.
> Congratulations and thank you so much!!
>
>
> Joe Murray, PhD
> President, JMA Consulting
> [hidden email]
> skype JosephPMurray twitter JoeMurray
> 416.466.1281
>
>
>
> On Tue, Jan 31, 2012 at 12:00 PM, <[hidden email]> wrote:
> Send CivicAccess-discuss mailing list submissions to
>        [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> or, via email, send a message with subject or body 'help' to
>        [hidden email]
>
> You can reach the person managing the list at
>        [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of CivicAccess-discuss digest..."
>
>
> Today's Topics:
>
>   1. Re: Announcing Represent, an open database of political
>      districts and representatives (Cory Horner)
>   2. Re: Announcing Represent, an open database of political
>      districts and representatives (Michael Mulley)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 30 Jan 2012 12:58:23 -0800
> From: Cory Horner <[hidden email]>
> To: civicaccess discuss <[hidden email]>
> Subject: Re: [CivicAccess-discuss] Announcing Represent,        an open
>        database of political districts and representatives
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=windows-1252
>
> Since our APIs are overlapping, i'm considering shutting mine down since you guys have more coverage (I only have federal, bc, and ontario so far).
>
> - can you add an XML output format?
> - can you add an option to return nearby ridings within a certain radius (because the postal code centroid is inaccurate)?
> - can people contact you to avoid the 60 request/second limit?
>
> If you can add those... I can punt my users to you and spend my time elsewhere...
>
> Thanks,
> Cory.
> ----
> http://howdtheyvote.ca/
>
>
>
> On 2012-01-30, at 8:44 AM, James McKinney wrote:
>
> > Open North is proud to announce Represent, an API (web service) to federal, provincial and municipal political districts and representatives across Canada. To see a demo and read the API docs, visit: http://represent.opennorth.ca/
> >
> > Represent establishes a foundation upon which other projects may be built - a necessary piece of infrastructure which we hope will enable useful and innovative features and products. It is free to use and open-source.
> >
> > This open database was built by James McKinney and Michael Mulley, with important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel, Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The project takes inspiration from mySociety?s MapIt and code from the Chicago Tribune?s Boundary Service.
> >
> > We currently have electoral districts and representatives for all provinces and Canada, and the ward boundaries of ten cities. We have postal code correspondences for Ontario (and soon Quebec) and postal code centroids from GeoCoder.ca.
> >
> > But we could have much more. To keep this project sustainable, we are asking for your help. Things you can help with:
> >
> > Add more representatives
> > - Write scrapers (in PHP, Ruby, or Python) to get the elected officials for more cities
> > - Ask provincial Municipal Affairs ministries for municipal elected officials in Excel format
> >
> > Get more boundaries
> > - Create a list of all municipalities with ward boundaries
> > - Ask them to provide their ward boundaries under an open license
> >
> > Free the postal code
> > - Ask your province or city for a postal code to electoral district concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
> >
> > Maintain the database
> > - Monitor scrapers to make sure we always provide the latest, most accurate data available
> > - Write an app to track electoral calendars, so we can plan updates and prepare scrapers
> > - Donate to Open North so a staff member can improve and maintain this service part-time
> >
> > Comments are very welcome, and please let us know if you've built something using the API.
> >
> > James & Michael
> > _______________________________________________
> > CivicAccess-discuss mailing list
> > [hidden email]
> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 30 Jan 2012 16:56:55 -0500
> From: Michael Mulley <[hidden email]>
> To: civicaccess discuss <[hidden email]>
> Subject: Re: [CivicAccess-discuss] Announcing Represent, an open
>        database of political districts and representatives
> Message-ID:
>        <[hidden email]>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Cory,
>
> I think a lot of people use & are grateful for your API -- I know I am --
> and I'd certainly like to see it stay around.
>
> I've got no plans to add XML and James & I originally planned this a JSON
> service, though we might change our minds if several people really want XML.
>
> Nearby: played with this briefly but the range queries I was trying were
> making PostGIS slow to a crawl, since we store everything in lat/lon
> geometry columns. This is certainly worth adding to our issues list.
>
> As to rate limits, this is a brand-new project & we don't have any kind of
> policy yet, but we built this as a tool for people to build on top of, and
> we're not gonna stand in the way of legitimate uses without a good reason.
>
> On Mon, Jan 30, 2012 at 3:58 PM, Cory Horner <[hidden email]> wrote:
>
> > Since our APIs are overlapping, i'm considering shutting mine down since
> > you guys have more coverage (I only have federal, bc, and ontario so far).
> >
> > - can you add an XML output format?
> > - can you add an option to return nearby ridings within a certain radius
> > (because the postal code centroid is inaccurate)?
> > - can people contact you to avoid the 60 request/second limit?
> >
> > If you can add those... I can punt my users to you and spend my time
> > elsewhere...
> >
> > Thanks,
> > Cory.
> > ----
> > http://howdtheyvote.ca/
> >
> >
> >
> > On 2012-01-30, at 8:44 AM, James McKinney wrote:
> >
> > > Open North is proud to announce Represent, an API (web service) to
> > federal, provincial and municipal political districts and representatives
> > across Canada. To see a demo and read the API docs, visit:
> > http://represent.opennorth.ca/
> > >
> > > Represent establishes a foundation upon which other projects may be
> > built - a necessary piece of infrastructure which we hope will enable
> > useful and innovative features and products. It is free to use and
> > open-source.
> > >
> > > This open database was built by James McKinney and Michael Mulley, with
> > important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,
> > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The
> > project takes inspiration from mySociety?s MapIt and code from the Chicago
> > Tribune?s Boundary Service.
> > >
> > > We currently have electoral districts and representatives for all
> > provinces and Canada, and the ward boundaries of ten cities. We have postal
> > code correspondences for Ontario (and soon Quebec) and postal code
> > centroids from GeoCoder.ca.
> > >
> > > But we could have much more. To keep this project sustainable, we are
> > asking for your help. Things you can help with:
> > >
> > > Add more representatives
> > > - Write scrapers (in PHP, Ruby, or Python) to get the elected officials
> > for more cities
> > > - Ask provincial Municipal Affairs ministries for municipal elected
> > officials in Excel format
> > >
> > > Get more boundaries
> > > - Create a list of all municipalities with ward boundaries
> > > - Ask them to provide their ward boundaries under an open license
> > >
> > > Free the postal code
> > > - Ask your province or city for a postal code to electoral district
> > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
> > >
> > > Maintain the database
> > > - Monitor scrapers to make sure we always provide the latest, most
> > accurate data available
> > > - Write an app to track electoral calendars, so we can plan updates and
> > prepare scrapers
> > > - Donate to Open North so a staff member can improve and maintain this
> > service part-time
> > >
> > > Comments are very welcome, and please let us know if you've built
> > something using the API.
> > >
> > > James & Michael
> > > _______________________________________________
> > > CivicAccess-discuss mailing list
> > > [hidden email]
> > > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> >
> > _______________________________________________
> > CivicAccess-discuss mailing list
> > [hidden email]
> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.pwd.ca/pipermail/civicaccess-discuss/attachments/20120130/4167f5ec/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
> End of CivicAccess-discuss Digest, Vol 54, Issue 34
> ***************************************************
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

[Nik G]

James,

Great project, congrats!

Last year OpenHalton, OpenHamilton, OpenGuelph & London ON hacked WardRep.ca
to provide similar type functionality for Municipal Council representatives.
We scraped (and otherwise liberated data from pdfs, etc.) for Ward
Boundaries and Council / Mayor info for 4 cities:

Milton, ON
London, ON
Guelph, ON
Hamilton, ON

The data is in DataDOTgc's OGDI instance with APIs, which hopefully you
could plug into to easily feed into REPRESENT.

Ward Boundaries:
DATA -- http://datadotgc.cloudapp.net/DataBrowser/def/CityWards
API -- XML: http://datadotgcds.cloudapp.net/v1/def/CityWards  JSON (append
?format=json): http://datadotgcds.cloudapp.net/v1/def/CityWards?format=json

Ward Boundaries is pipe-separated polygon coordinates.

Representatives
DATA -- http://datadotgc.cloudapp.net/DataBrowser/def/WardRepRepresentatives
API --- XML/Atom:
http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives JSON:
http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives?format=json

Nik Garkusha
http://openhalton.ca

-----Original Message-----
From: James McKinney
Sent: Tuesday, January 31, 2012 1:16 PM
To: civicaccess discuss
Subject: Re: [CivicAccess-discuss] CivicAccess-discuss Digest, Vol 54,Issue
34

Thanks for the feedback, Joe!

We provide a link in the API to the page where you can download the
shapefile, and a link to the license, e.g.
http://represent.opennorth.ca/boundary-sets/federal-electoral-districts/?format=apibrowser 
The majority of the shapefiles are publicly available on the internet. The
rest we got by email.

Can't you convert the shapefiles to MySQL yourself using a tool like
shp2mysql? I think it's CiviCRM's responsibility to give you tools for
loading geospatial data into its database.

I've added a clarification to contact us if you expect to exceed limits.

We admit the inaccuracy of centroids in the documentation. For provinces
that post official concordances between postal codes and boundaries, we use
that data instead, e.g.
http://represent.opennorth.ca/postcodes/K0A1K0/?format=apibrowser

There are roughly a million postal codes, so I doubt that several hundred
thousands appear in more than one riding. According Elections ON, of its
lion's share of the postal codes, only 2724 map to multiple electoral
districts.
http://www.elections.on.ca/en-CA/Tools/ElectoralDistricts/PostalCodeFile.htm 
The PCCF and PCFRF are notorious for errors. GeoCoder.ca instead uses Canada
Post's assignment of postal codes to road segments (a postal code is not a
polygon but a mail route) to do its geocoding of postal codes.

Cheers,

James

On 2012-01-31, at 12:29 PM, Joe Murray wrote:

> If I could echo a few points others have made in my own way:
>
> • For performance reasons I think I would like to get the shape files you
> have used and make them available for various groups to load them into
> CiviCRM. One benefit will be to do the calculation locally rather than
> over the wire via JSON. Another will be to reduce the load on your server
> when we need to calculate the wards and ridings for databases with tens
> and hundreds of thousands of records.
> • This will require converting them to MySQL's geo format. But since
> 5.1.26 or so I think, MySQL has provided support for accurate calculation
> of whether a point is within a polygon (before that it was an inaccurate
> but quick calculation of whether the point was within the minimum bounding
> rectangle).
> • So I'm looking forward to seeing you publish the sources of the
> shapefiles and their licenses.
> • In some cases the licenses will likely not permit you to share the
> shapefiles. I imagine that some of my clients who want this information
> for advocacy purposes would be happy to pay for a higher usage than 60
> requests per minute during bulk encoding, since otherwise it will take a
> couple of days to deal with all of their data (e.g. I recently geocoded
> 175k addresses for federal riding and where appropriate Toronto ward).
> With a bit of advance coordination, this might allow you to bump up the
> capabilities of your vps for a day to handle the increased load, and heh,
> maybe even subsidize more development or support. ;) A common provision in
> licenses for this sort of service is to say that there is a cap but if you
> want to exceed the cap to contact the provider, ie you folks.
> • Without wanting to be persnickety, postal code centroids are not
> generally accurate enough. A few apps I have seen allow people to see the
> several ridings that their address may be a part of.
> • Several years ago, using the voters list data provided for the country
> to political parties there were a few hundred thousand postal codes that
> appear in more than one riding if I recall correctly, and the number of
> ridings could go up to 7 for a particular postal code (I think it was a
> rural one in the Atlantic provinces for a highway). When I tracked down
> the quality of the data in the PCFRF through Elections Canada to
> StatsCanada they gave me a long explanation for why this was so. I bet it
> is still the case.
> Congratulations and thank you so much!!
>
>
> Joe Murray, PhD
> President, JMA Consulting
> [hidden email]
> skype JosephPMurray twitter JoeMurray
> 416.466.1281
>
>
>
> On Tue, Jan 31, 2012 at 12:00 PM,
> <[hidden email]> wrote:
> Send CivicAccess-discuss mailing list submissions to
>        [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> or, via email, send a message with subject or body 'help' to
>        [hidden email]
>
> You can reach the person managing the list at
>        [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of CivicAccess-discuss digest..."
>
>
> Today's Topics:
>
>   1. Re: Announcing Represent, an open database of political
>      districts and representatives (Cory Horner)
>   2. Re: Announcing Represent, an open database of political
>      districts and representatives (Michael Mulley)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 30 Jan 2012 12:58:23 -0800
> From: Cory Horner <[hidden email]>
> To: civicaccess discuss <[hidden email]>
> Subject: Re: [CivicAccess-discuss] Announcing Represent,        an open
>        database of political districts and representatives
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=windows-1252
>
> Since our APIs are overlapping, i'm considering shutting mine down since
> you guys have more coverage (I only have federal, bc, and ontario so far).
>
> - can you add an XML output format?
> - can you add an option to return nearby ridings within a certain radius
> (because the postal code centroid is inaccurate)?
> - can people contact you to avoid the 60 request/second limit?
>
> If you can add those... I can punt my users to you and spend my time
> elsewhere...
>
> Thanks,
> Cory.
> ----
> http://howdtheyvote.ca/
>
>
>
> On 2012-01-30, at 8:44 AM, James McKinney wrote:
>
> > Open North is proud to announce Represent, an API (web service) to
> > federal, provincial and municipal political districts and
> > representatives across Canada. To see a demo and read the API docs,
> > visit: http://represent.opennorth.ca/
> >
> > Represent establishes a foundation upon which other projects may be
> > built - a necessary piece of infrastructure which we hope will enable
> > useful and innovative features and products. It is free to use and
> > open-source.
> >
> > This open database was built by James McKinney and Michael Mulley, with
> > important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,
> > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The
> > project takes inspiration from mySociety?s MapIt and code from the
> > Chicago Tribune?s Boundary Service.
> >
> > We currently have electoral districts and representatives for all
> > provinces and Canada, and the ward boundaries of ten cities. We have
> > postal code correspondences for Ontario (and soon Quebec) and postal
> > code centroids from GeoCoder.ca.
> >
> > But we could have much more. To keep this project sustainable, we are
> > asking for your help. Things you can help with:
> >
> > Add more representatives
> > - Write scrapers (in PHP, Ruby, or Python) to get the elected officials
> > for more cities
> > - Ask provincial Municipal Affairs ministries for municipal elected
> > officials in Excel format
> >
> > Get more boundaries
> > - Create a list of all municipalities with ward boundaries
> > - Ask them to provide their ward boundaries under an open license
> >
> > Free the postal code
> > - Ask your province or city for a postal code to electoral district
> > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
> >
> > Maintain the database
> > - Monitor scrapers to make sure we always provide the latest, most
> > accurate data available
> > - Write an app to track electoral calendars, so we can plan updates and
> > prepare scrapers
> > - Donate to Open North so a staff member can improve and maintain this
> > service part-time
> >
> > Comments are very welcome, and please let us know if you've built
> > something using the API.
> >
> > James & Michael
> > _______________________________________________
> > CivicAccess-discuss mailing list
> > [hidden email]
> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 30 Jan 2012 16:56:55 -0500
> From: Michael Mulley <[hidden email]>
> To: civicaccess discuss <[hidden email]>
> Subject: Re: [CivicAccess-discuss] Announcing Represent, an open
>        database of political districts and representatives
> Message-ID:
>
> <[hidden email]>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Cory,
>
> I think a lot of people use & are grateful for your API -- I know I am --
> and I'd certainly like to see it stay around.
>
> I've got no plans to add XML and James & I originally planned this a JSON
> service, though we might change our minds if several people really want
> XML.
>
> Nearby: played with this briefly but the range queries I was trying were
> making PostGIS slow to a crawl, since we store everything in lat/lon
> geometry columns. This is certainly worth adding to our issues list.
>
> As to rate limits, this is a brand-new project & we don't have any kind of
> policy yet, but we built this as a tool for people to build on top of, and
> we're not gonna stand in the way of legitimate uses without a good reason.
>
> On Mon, Jan 30, 2012 at 3:58 PM, Cory Horner <[hidden email]> wrote:
>
> > Since our APIs are overlapping, i'm considering shutting mine down since
> > you guys have more coverage (I only have federal, bc, and ontario so
> > far).
> >
> > - can you add an XML output format?
> > - can you add an option to return nearby ridings within a certain radius
> > (because the postal code centroid is inaccurate)?
> > - can people contact you to avoid the 60 request/second limit?
> >
> > If you can add those... I can punt my users to you and spend my time
> > elsewhere...
> >
> > Thanks,
> > Cory.
> > ----
> > http://howdtheyvote.ca/
> >
> >
> >
> > On 2012-01-30, at 8:44 AM, James McKinney wrote:
> >
> > > Open North is proud to announce Represent, an API (web service) to
> > federal, provincial and municipal political districts and
> > representatives
> > across Canada. To see a demo and read the API docs, visit:
> > http://represent.opennorth.ca/
> > >
> > > Represent establishes a foundation upon which other projects may be
> > built - a necessary piece of infrastructure which we hope will enable
> > useful and innovative features and products. It is free to use and
> > open-source.
> > >
> > > This open database was built by James McKinney and Michael Mulley,
> > > with
> > important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,
> > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The
> > project takes inspiration from mySociety?s MapIt and code from the
> > Chicago
> > Tribune?s Boundary Service.
> > >
> > > We currently have electoral districts and representatives for all
> > provinces and Canada, and the ward boundaries of ten cities. We have
> > postal
> > code correspondences for Ontario (and soon Quebec) and postal code
> > centroids from GeoCoder.ca.
> > >
> > > But we could have much more. To keep this project sustainable, we are
> > asking for your help. Things you can help with:
> > >
> > > Add more representatives
> > > - Write scrapers (in PHP, Ruby, or Python) to get the elected
> > > officials
> > for more cities
> > > - Ask provincial Municipal Affairs ministries for municipal elected
> > officials in Excel format
> > >
> > > Get more boundaries
> > > - Create a list of all municipalities with ward boundaries
> > > - Ask them to provide their ward boundaries under an open license
> > >
> > > Free the postal code
> > > - Ask your province or city for a postal code to electoral district
> > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
> > >
> > > Maintain the database
> > > - Monitor scrapers to make sure we always provide the latest, most
> > accurate data available
> > > - Write an app to track electoral calendars, so we can plan updates
> > > and
> > prepare scrapers
> > > - Donate to Open North so a staff member can improve and maintain this
> > service part-time
> > >
> > > Comments are very welcome, and please let us know if you've built
> > something using the API.
> > >
> > > James & Michael
> > > _______________________________________________
> > > CivicAccess-discuss mailing list
> > > [hidden email]
> > > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> >
> > _______________________________________________
> > CivicAccess-discuss mailing list
> > [hidden email]
> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.pwd.ca/pipermail/civicaccess-discuss/attachments/20120130/4167f5ec/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
> End of CivicAccess-discuss Digest, Vol 54, Issue 34
> ***************************************************
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss 

[James McKinney]

Thanks, Nik! I was planning on writing to you, as a core member of the wardrep.ca team.

In terms of integrating wardrep.ca data, our issue is that the representative data is static (it was manually entered). Manually entered data has a higher maintenance cost than programatically scraped data. A scraper's data will remain accurate for as long as the authority doesn't change its page layout. When they do change the page layout, the scraper is likely to fail, which will alert us to the need to update the scraper. So for representative data, we prefer scraper code to raw data.

With respect to the ward boundaries, as I understand, these were created using Google Earth and PDF references. It is difficult for us to verify the accuracy of such data. We prefer authoritative datasets. You can help contribute to Represent by using your contacts in those cities to request a shapefile of their boroughs. I'm certain they can do that for you.

Thanks,

James

On 2012-01-31, at 1:45 PM, Nik G wrote:

> James,
>
> Great project, congrats!
>
> Last year OpenHalton, OpenHamilton, OpenGuelph & London ON hacked WardRep.ca to provide similar type functionality for Municipal Council representatives.
> We scraped (and otherwise liberated data from pdfs, etc.) for Ward Boundaries and Council / Mayor info for 4 cities:
>
> Milton, ON
> London, ON
> Guelph, ON
> Hamilton, ON
>
> The data is in DataDOTgc's OGDI instance with APIs, which hopefully you could plug into to easily feed into REPRESENT.
>
> Ward Boundaries:
> DATA -- http://datadotgc.cloudapp.net/DataBrowser/def/CityWards
> API -- XML: http://datadotgcds.cloudapp.net/v1/def/CityWards  JSON (append ?format=json): http://datadotgcds.cloudapp.net/v1/def/CityWards?format=json
>
> Ward Boundaries is pipe-separated polygon coordinates.
>
> Representatives
> DATA -- http://datadotgc.cloudapp.net/DataBrowser/def/WardRepRepresentatives
> API --- XML/Atom: http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives JSON: http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives?format=json
>
> Nik Garkusha
> http://openhalton.ca
>
> -----Original Message----- From: James McKinney
> Sent: Tuesday, January 31, 2012 1:16 PM
> To: civicaccess discuss
> Subject: Re: [CivicAccess-discuss] CivicAccess-discuss Digest, Vol 54,Issue 34
>
> Thanks for the feedback, Joe!
>
> We provide a link in the API to the page where you can download the shapefile, and a link to the license, e.g. http://represent.opennorth.ca/boundary-sets/federal-electoral-districts/?format=apibrowser The majority of the shapefiles are publicly available on the internet. The rest we got by email.
>
> Can't you convert the shapefiles to MySQL yourself using a tool like shp2mysql? I think it's CiviCRM's responsibility to give you tools for loading geospatial data into its database.
>
> I've added a clarification to contact us if you expect to exceed limits.
>
> We admit the inaccuracy of centroids in the documentation. For provinces that post official concordances between postal codes and boundaries, we use that data instead, e.g. http://represent.opennorth.ca/postcodes/K0A1K0/?format=apibrowser
>
> There are roughly a million postal codes, so I doubt that several hundred thousands appear in more than one riding. According Elections ON, of its lion's share of the postal codes, only 2724 map to multiple electoral districts. http://www.elections.on.ca/en-CA/Tools/ElectoralDistricts/PostalCodeFile.htm The PCCF and PCFRF are notorious for errors. GeoCoder.ca instead uses Canada Post's assignment of postal codes to road segments (a postal code is not a polygon but a mail route) to do its geocoding of postal codes.
>
> Cheers,
>
> James
>
> On 2012-01-31, at 12:29 PM, Joe Murray wrote:
>
>> If I could echo a few points others have made in my own way:
>>
>> • For performance reasons I think I would like to get the shape files you have used and make them available for various groups to load them into CiviCRM. One benefit will be to do the calculation locally rather than over the wire via JSON. Another will be to reduce the load on your server when we need to calculate the wards and ridings for databases with tens and hundreds of thousands of records.
>> • This will require converting them to MySQL's geo format. But since 5.1.26 or so I think, MySQL has provided support for accurate calculation of whether a point is within a polygon (before that it was an inaccurate but quick calculation of whether the point was within the minimum bounding rectangle).
>> • So I'm looking forward to seeing you publish the sources of the shapefiles and their licenses.
>> • In some cases the licenses will likely not permit you to share the shapefiles. I imagine that some of my clients who want this information for advocacy purposes would be happy to pay for a higher usage than 60 requests per minute during bulk encoding, since otherwise it will take a couple of days to deal with all of their data (e.g. I recently geocoded 175k addresses for federal riding and where appropriate Toronto ward). With a bit of advance coordination, this might allow you to bump up the capabilities of your vps for a day to handle the increased load, and heh, maybe even subsidize more development or support. ;) A common provision in licenses for this sort of service is to say that there is a cap but if you want to exceed the cap to contact the provider, ie you folks.
>> • Without wanting to be persnickety, postal code centroids are not generally accurate enough. A few apps I have seen allow people to see the several ridings that their address may be a part of.
>> • Several years ago, using the voters list data provided for the country to political parties there were a few hundred thousand postal codes that appear in more than one riding if I recall correctly, and the number of ridings could go up to 7 for a particular postal code (I think it was a rural one in the Atlantic provinces for a highway). When I tracked down the quality of the data in the PCFRF through Elections Canada to StatsCanada they gave me a long explanation for why this was so. I bet it is still the case.
>> Congratulations and thank you so much!!
>>
>>
>> Joe Murray, PhD
>> President, JMA Consulting
>> [hidden email]
>> skype JosephPMurray twitter JoeMurray
>> 416.466.1281
>>
>>
>>
>> On Tue, Jan 31, 2012 at 12:00 PM, <[hidden email]> wrote:
>> Send CivicAccess-discuss mailing list submissions to
>>       [hidden email]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>       http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> or, via email, send a message with subject or body 'help' to
>>       [hidden email]
>>
>> You can reach the person managing the list at
>>       [hidden email]
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of CivicAccess-discuss digest..."
>>
>>
>> Today's Topics:
>>
>>  1. Re: Announcing Represent, an open database of political
>>     districts and representatives (Cory Horner)
>>  2. Re: Announcing Represent, an open database of political
>>     districts and representatives (Michael Mulley)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 30 Jan 2012 12:58:23 -0800
>> From: Cory Horner <[hidden email]>
>> To: civicaccess discuss <[hidden email]>
>> Subject: Re: [CivicAccess-discuss] Announcing Represent,        an open
>>       database of political districts and representatives
>> Message-ID: <[hidden email]>
>> Content-Type: text/plain; charset=windows-1252
>>
>> Since our APIs are overlapping, i'm considering shutting mine down since you guys have more coverage (I only have federal, bc, and ontario so far).
>>
>> - can you add an XML output format?
>> - can you add an option to return nearby ridings within a certain radius (because the postal code centroid is inaccurate)?
>> - can people contact you to avoid the 60 request/second limit?
>>
>> If you can add those... I can punt my users to you and spend my time elsewhere...
>>
>> Thanks,
>> Cory.
>> ----
>> http://howdtheyvote.ca/
>>
>>
>>
>> On 2012-01-30, at 8:44 AM, James McKinney wrote:
>>
>> > Open North is proud to announce Represent, an API (web service) to > federal, provincial and municipal political districts and > representatives across Canada. To see a demo and read the API docs, > visit: http://represent.opennorth.ca/
>> >
>> > Represent establishes a foundation upon which other projects may be > built - a necessary piece of infrastructure which we hope will enable > useful and innovative features and products. It is free to use and > open-source.
>> >
>> > This open database was built by James McKinney and Michael Mulley, with > important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel, > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The > project takes inspiration from mySociety?s MapIt and code from the > Chicago Tribune?s Boundary Service.
>> >
>> > We currently have electoral districts and representatives for all > provinces and Canada, and the ward boundaries of ten cities. We have > postal code correspondences for Ontario (and soon Quebec) and postal > code centroids from GeoCoder.ca.
>> >
>> > But we could have much more. To keep this project sustainable, we are > asking for your help. Things you can help with:
>> >
>> > Add more representatives
>> > - Write scrapers (in PHP, Ruby, or Python) to get the elected officials > for more cities
>> > - Ask provincial Municipal Affairs ministries for municipal elected > officials in Excel format
>> >
>> > Get more boundaries
>> > - Create a list of all municipalities with ward boundaries
>> > - Ask them to provide their ward boundaries under an open license
>> >
>> > Free the postal code
>> > - Ask your province or city for a postal code to electoral district > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
>> >
>> > Maintain the database
>> > - Monitor scrapers to make sure we always provide the latest, most > accurate data available
>> > - Write an app to track electoral calendars, so we can plan updates and > prepare scrapers
>> > - Donate to Open North so a staff member can improve and maintain this > service part-time
>> >
>> > Comments are very welcome, and please let us know if you've built > something using the API.
>> >
>> > James & Michael
>> > _______________________________________________
>> > CivicAccess-discuss mailing list
>> > [hidden email]
>> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 30 Jan 2012 16:56:55 -0500
>> From: Michael Mulley <[hidden email]>
>> To: civicaccess discuss <[hidden email]>
>> Subject: Re: [CivicAccess-discuss] Announcing Represent, an open
>>       database of political districts and representatives
>> Message-ID:
>> <[hidden email]>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi Cory,
>>
>> I think a lot of people use & are grateful for your API -- I know I am --
>> and I'd certainly like to see it stay around.
>>
>> I've got no plans to add XML and James & I originally planned this a JSON
>> service, though we might change our minds if several people really want XML.
>>
>> Nearby: played with this briefly but the range queries I was trying were
>> making PostGIS slow to a crawl, since we store everything in lat/lon
>> geometry columns. This is certainly worth adding to our issues list.
>>
>> As to rate limits, this is a brand-new project & we don't have any kind of
>> policy yet, but we built this as a tool for people to build on top of, and
>> we're not gonna stand in the way of legitimate uses without a good reason.
>>
>> On Mon, Jan 30, 2012 at 3:58 PM, Cory Horner <[hidden email]> wrote:
>>
>> > Since our APIs are overlapping, i'm considering shutting mine down since
>> > you guys have more coverage (I only have federal, bc, and ontario so > far).
>> >
>> > - can you add an XML output format?
>> > - can you add an option to return nearby ridings within a certain radius
>> > (because the postal code centroid is inaccurate)?
>> > - can people contact you to avoid the 60 request/second limit?
>> >
>> > If you can add those... I can punt my users to you and spend my time
>> > elsewhere...
>> >
>> > Thanks,
>> > Cory.
>> > ----
>> > http://howdtheyvote.ca/
>> >
>> >
>> >
>> > On 2012-01-30, at 8:44 AM, James McKinney wrote:
>> >
>> > > Open North is proud to announce Represent, an API (web service) to
>> > federal, provincial and municipal political districts and > representatives
>> > across Canada. To see a demo and read the API docs, visit:
>> > http://represent.opennorth.ca/
>> > >
>> > > Represent establishes a foundation upon which other projects may be
>> > built - a necessary piece of infrastructure which we hope will enable
>> > useful and innovative features and products. It is free to use and
>> > open-source.
>> > >
>> > > This open database was built by James McKinney and Michael Mulley, > > with
>> > important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,
>> > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The
>> > project takes inspiration from mySociety?s MapIt and code from the > Chicago
>> > Tribune?s Boundary Service.
>> > >
>> > > We currently have electoral districts and representatives for all
>> > provinces and Canada, and the ward boundaries of ten cities. We have > postal
>> > code correspondences for Ontario (and soon Quebec) and postal code
>> > centroids from GeoCoder.ca.
>> > >
>> > > But we could have much more. To keep this project sustainable, we are
>> > asking for your help. Things you can help with:
>> > >
>> > > Add more representatives
>> > > - Write scrapers (in PHP, Ruby, or Python) to get the elected > > officials
>> > for more cities
>> > > - Ask provincial Municipal Affairs ministries for municipal elected
>> > officials in Excel format
>> > >
>> > > Get more boundaries
>> > > - Create a list of all municipalities with ward boundaries
>> > > - Ask them to provide their ward boundaries under an open license
>> > >
>> > > Free the postal code
>> > > - Ask your province or city for a postal code to electoral district
>> > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
>> > >
>> > > Maintain the database
>> > > - Monitor scrapers to make sure we always provide the latest, most
>> > accurate data available
>> > > - Write an app to track electoral calendars, so we can plan updates > > and
>> > prepare scrapers
>> > > - Donate to Open North so a staff member can improve and maintain this
>> > service part-time
>> > >
>> > > Comments are very welcome, and please let us know if you've built
>> > something using the API.
>> > >
>> > > James & Michael
>> > > _______________________________________________
>> > > CivicAccess-discuss mailing list
>> > > [hidden email]
>> > > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> >
>> > _______________________________________________
>> > CivicAccess-discuss mailing list
>> > [hidden email]
>> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.pwd.ca/pipermail/civicaccess-discuss/attachments/20120130/4167f5ec/attachment-0001.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>> End of CivicAccess-discuss Digest, Vol 54, Issue 34
>> ***************************************************
>>
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss 
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

[Nik G]

Hi James,

It's great that you're taking a well thought-out approach RE: scrapers and
authoritative datasets.

Can you share what most of the Scrapers used for Represent are written in,
and is there any way to quickly identify which of scrapers are most
appropriate for different types of council pages?

Regarding ward boundaries, Guelph was from the city via OpenGuelph; London
I'm not sure about, and Milton and Hamilton were scraped. I wouldn't be so
certain about some muni's like Milton sharing shape/boundary data... in fact
there's very little chance we'll be getting anything from Milton at this
point, considering just yesterday the council expressed their concerns
around "risks" and "costs" associated with open data and the Open Data
motion was put on hold. Hamilton may be a possibility. Stay tuned.

Thanks
Nik


-----Original Message-----
From: James McKinney
Sent: Tuesday, January 31, 2012 4:41 PM
To: civicaccess discuss
Subject: Re: [CivicAccess-discuss] CivicAccess-discuss Digest, Vol 54,Issue
34

Thanks, Nik! I was planning on writing to you, as a core member of the
wardrep.ca team.

In terms of integrating wardrep.ca data, our issue is that the
representative data is static (it was manually entered). Manually entered
data has a higher maintenance cost than programatically scraped data. A
scraper's data will remain accurate for as long as the authority doesn't
change its page layout. When they do change the page layout, the scraper is
likely to fail, which will alert us to the need to update the scraper. So
for representative data, we prefer scraper code to raw data.

With respect to the ward boundaries, as I understand, these were created
using Google Earth and PDF references. It is difficult for us to verify the
accuracy of such data. We prefer authoritative datasets. You can help
contribute to Represent by using your contacts in those cities to request a
shapefile of their boroughs. I'm certain they can do that for you.

Thanks,

James

On 2012-01-31, at 1:45 PM, Nik G wrote:

> James,
>
> Great project, congrats!
>
> Last year OpenHalton, OpenHamilton, OpenGuelph & London ON hacked
> WardRep.ca to provide similar type functionality for Municipal Council
> representatives.
> We scraped (and otherwise liberated data from pdfs, etc.) for Ward
> Boundaries and Council / Mayor info for 4 cities:
>
> Milton, ON
> London, ON
> Guelph, ON
> Hamilton, ON
>
> The data is in DataDOTgc's OGDI instance with APIs, which hopefully you
> could plug into to easily feed into REPRESENT.
>
> Ward Boundaries:
> DATA -- http://datadotgc.cloudapp.net/DataBrowser/def/CityWards
> API -- XML: http://datadotgcds.cloudapp.net/v1/def/CityWards  JSON (append
> ?format=json):
> http://datadotgcds.cloudapp.net/v1/def/CityWards?format=json
>
> Ward Boundaries is pipe-separated polygon coordinates.
>
> Representatives
> DATA --  
> http://datadotgc.cloudapp.net/DataBrowser/def/WardRepRepresentatives
> API --- XML/Atom:
> http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives JSON:
> http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives?format=json
>
> Nik Garkusha
> http://openhalton.ca
>
> -----Original Message----- From: James McKinney
> Sent: Tuesday, January 31, 2012 1:16 PM
> To: civicaccess discuss
> Subject: Re: [CivicAccess-discuss] CivicAccess-discuss Digest, Vol
> 54,Issue 34
>
> Thanks for the feedback, Joe!
>
> We provide a link in the API to the page where you can download the
> shapefile, and a link to the license, e.g.
> http://represent.opennorth.ca/boundary-sets/federal-electoral-districts/?format=apibrowser 
> The majority of the shapefiles are publicly available on the internet. The
> rest we got by email.
>
> Can't you convert the shapefiles to MySQL yourself using a tool like
> shp2mysql? I think it's CiviCRM's responsibility to give you tools for
> loading geospatial data into its database.
>
> I've added a clarification to contact us if you expect to exceed limits.
>
> We admit the inaccuracy of centroids in the documentation. For provinces
> that post official concordances between postal codes and boundaries, we
> use that data instead, e.g.
> http://represent.opennorth.ca/postcodes/K0A1K0/?format=apibrowser
>
> There are roughly a million postal codes, so I doubt that several hundred
> thousands appear in more than one riding. According Elections ON, of its
> lion's share of the postal codes, only 2724 map to multiple electoral
> districts.
> http://www.elections.on.ca/en-CA/Tools/ElectoralDistricts/PostalCodeFile.htm 
> The PCCF and PCFRF are notorious for errors. GeoCoder.ca instead uses
> Canada Post's assignment of postal codes to road segments (a postal code
> is not a polygon but a mail route) to do its geocoding of postal codes.
>
> Cheers,
>
> James
>
> On 2012-01-31, at 12:29 PM, Joe Murray wrote:
>
>> If I could echo a few points others have made in my own way:
>>
>> • For performance reasons I think I would like to get the shape files you
>> have used and make them available for various groups to load them into
>> CiviCRM. One benefit will be to do the calculation locally rather than
>> over the wire via JSON. Another will be to reduce the load on your server
>> when we need to calculate the wards and ridings for databases with tens
>> and hundreds of thousands of records.
>> • This will require converting them to MySQL's geo format. But since
>> 5.1.26 or so I think, MySQL has provided support for accurate calculation
>> of whether a point is within a polygon (before that it was an inaccurate
>> but quick calculation of whether the point was within the minimum
>> bounding rectangle).
>> • So I'm looking forward to seeing you publish the sources of the
>> shapefiles and their licenses.
>> • In some cases the licenses will likely not permit you to share the
>> shapefiles. I imagine that some of my clients who want this information
>> for advocacy purposes would be happy to pay for a higher usage than 60
>> requests per minute during bulk encoding, since otherwise it will take a
>> couple of days to deal with all of their data (e.g. I recently geocoded
>> 175k addresses for federal riding and where appropriate Toronto ward).
>> With a bit of advance coordination, this might allow you to bump up the
>> capabilities of your vps for a day to handle the increased load, and heh,
>> maybe even subsidize more development or support. ;) A common provision
>> in licenses for this sort of service is to say that there is a cap but if
>> you want to exceed the cap to contact the provider, ie you folks.
>> • Without wanting to be persnickety, postal code centroids are not
>> generally accurate enough. A few apps I have seen allow people to see the
>> several ridings that their address may be a part of.
>> • Several years ago, using the voters list data provided for the country
>> to political parties there were a few hundred thousand postal codes that
>> appear in more than one riding if I recall correctly, and the number of
>> ridings could go up to 7 for a particular postal code (I think it was a
>> rural one in the Atlantic provinces for a highway). When I tracked down
>> the quality of the data in the PCFRF through Elections Canada to
>> StatsCanada they gave me a long explanation for why this was so. I bet it
>> is still the case.
>> Congratulations and thank you so much!!
>>
>>
>> Joe Murray, PhD
>> President, JMA Consulting
>> [hidden email]
>> skype JosephPMurray twitter JoeMurray
>> 416.466.1281
>>
>>
>>
>> On Tue, Jan 31, 2012 at 12:00 PM,
>> <[hidden email]> wrote:
>> Send CivicAccess-discuss mailing list submissions to
>>       [hidden email]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>       http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> or, via email, send a message with subject or body 'help' to
>>       [hidden email]
>>
>> You can reach the person managing the list at
>>       [hidden email]
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of CivicAccess-discuss digest..."
>>
>>
>> Today's Topics:
>>
>>  1. Re: Announcing Represent, an open database of political
>>     districts and representatives (Cory Horner)
>>  2. Re: Announcing Represent, an open database of political
>>     districts and representatives (Michael Mulley)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 30 Jan 2012 12:58:23 -0800
>> From: Cory Horner <[hidden email]>
>> To: civicaccess discuss <[hidden email]>
>> Subject: Re: [CivicAccess-discuss] Announcing Represent,        an open
>>       database of political districts and representatives
>> Message-ID: <[hidden email]>
>> Content-Type: text/plain; charset=windows-1252
>>
>> Since our APIs are overlapping, i'm considering shutting mine down since
>> you guys have more coverage (I only have federal, bc, and ontario so
>> far).
>>
>> - can you add an XML output format?
>> - can you add an option to return nearby ridings within a certain radius
>> (because the postal code centroid is inaccurate)?
>> - can people contact you to avoid the 60 request/second limit?
>>
>> If you can add those... I can punt my users to you and spend my time
>> elsewhere...
>>
>> Thanks,
>> Cory.
>> ----
>> http://howdtheyvote.ca/
>>
>>
>>
>> On 2012-01-30, at 8:44 AM, James McKinney wrote:
>>
>> > Open North is proud to announce Represent, an API (web service) to >
>> > federal, provincial and municipal political districts and >
>> > representatives across Canada. To see a demo and read the API docs, >
>> > visit: http://represent.opennorth.ca/
>> >
>> > Represent establishes a foundation upon which other projects may be >
>> > built - a necessary piece of infrastructure which we hope will enable >
>> > useful and innovative features and products. It is free to use and >
>> > open-source.
>> >
>> > This open database was built by James McKinney and Michael Mulley, with
>> >  > important contributions from Aran Rasmussen, Kent Mewhort, Amir
>> > Pakdel, > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine
>> > Casciato. The > project takes inspiration from mySociety?s MapIt and
>> > code from the > Chicago Tribune?s Boundary Service.
>> >
>> > We currently have electoral districts and representatives for all >
>> > provinces and Canada, and the ward boundaries of ten cities. We have >
>> > postal code correspondences for Ontario (and soon Quebec) and postal >
>> > code centroids from GeoCoder.ca.
>> >
>> > But we could have much more. To keep this project sustainable, we are >
>> > asking for your help. Things you can help with:
>> >
>> > Add more representatives
>> > - Write scrapers (in PHP, Ruby, or Python) to get the elected officials
>> >  > for more cities
>> > - Ask provincial Municipal Affairs ministries for municipal elected >
>> > officials in Excel format
>> >
>> > Get more boundaries
>> > - Create a list of all municipalities with ward boundaries
>> > - Ask them to provide their ward boundaries under an open license
>> >
>> > Free the postal code
>> > - Ask your province or city for a postal code to electoral district >
>> > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
>> >
>> > Maintain the database
>> > - Monitor scrapers to make sure we always provide the latest, most >
>> > accurate data available
>> > - Write an app to track electoral calendars, so we can plan updates and
>> >  > prepare scrapers
>> > - Donate to Open North so a staff member can improve and maintain this
>> >  > service part-time
>> >
>> > Comments are very welcome, and please let us know if you've built >
>> > something using the API.
>> >
>> > James & Michael
>> > _______________________________________________
>> > CivicAccess-discuss mailing list
>> > [hidden email]
>> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 30 Jan 2012 16:56:55 -0500
>> From: Michael Mulley <[hidden email]>
>> To: civicaccess discuss <[hidden email]>
>> Subject: Re: [CivicAccess-discuss] Announcing Represent, an open
>>       database of political districts and representatives
>> Message-ID:
>> <[hidden email]>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi Cory,
>>
>> I think a lot of people use & are grateful for your API -- I know I am --
>> and I'd certainly like to see it stay around.
>>
>> I've got no plans to add XML and James & I originally planned this a JSON
>> service, though we might change our minds if several people really want
>> XML.
>>
>> Nearby: played with this briefly but the range queries I was trying were
>> making PostGIS slow to a crawl, since we store everything in lat/lon
>> geometry columns. This is certainly worth adding to our issues list.
>>
>> As to rate limits, this is a brand-new project & we don't have any kind
>> of
>> policy yet, but we built this as a tool for people to build on top of,
>> and
>> we're not gonna stand in the way of legitimate uses without a good
>> reason.
>>
>> On Mon, Jan 30, 2012 at 3:58 PM, Cory Horner <[hidden email]> wrote:
>>
>> > Since our APIs are overlapping, i'm considering shutting mine down
>> > since
>> > you guys have more coverage (I only have federal, bc, and ontario so >
>> > far).
>> >
>> > - can you add an XML output format?
>> > - can you add an option to return nearby ridings within a certain
>> > radius
>> > (because the postal code centroid is inaccurate)?
>> > - can people contact you to avoid the 60 request/second limit?
>> >
>> > If you can add those... I can punt my users to you and spend my time
>> > elsewhere...
>> >
>> > Thanks,
>> > Cory.
>> > ----
>> > http://howdtheyvote.ca/
>> >
>> >
>> >
>> > On 2012-01-30, at 8:44 AM, James McKinney wrote:
>> >
>> > > Open North is proud to announce Represent, an API (web service) to
>> > federal, provincial and municipal political districts and >
>> > representatives
>> > across Canada. To see a demo and read the API docs, visit:
>> > http://represent.opennorth.ca/
>> > >
>> > > Represent establishes a foundation upon which other projects may be
>> > built - a necessary piece of infrastructure which we hope will enable
>> > useful and innovative features and products. It is free to use and
>> > open-source.
>> > >
>> > > This open database was built by James McKinney and Michael Mulley, >
>> > >  > with
>> > important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,
>> > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The
>> > project takes inspiration from mySociety?s MapIt and code from the >
>> > Chicago
>> > Tribune?s Boundary Service.
>> > >
>> > > We currently have electoral districts and representatives for all
>> > provinces and Canada, and the ward boundaries of ten cities. We have >
>> > postal
>> > code correspondences for Ontario (and soon Quebec) and postal code
>> > centroids from GeoCoder.ca.
>> > >
>> > > But we could have much more. To keep this project sustainable, we are
>> > asking for your help. Things you can help with:
>> > >
>> > > Add more representatives
>> > > - Write scrapers (in PHP, Ruby, or Python) to get the elected > >
>> > > officials
>> > for more cities
>> > > - Ask provincial Municipal Affairs ministries for municipal elected
>> > officials in Excel format
>> > >
>> > > Get more boundaries
>> > > - Create a list of all municipalities with ward boundaries
>> > > - Ask them to provide their ward boundaries under an open license
>> > >
>> > > Free the postal code
>> > > - Ask your province or city for a postal code to electoral district
>> > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)
>> > >
>> > > Maintain the database
>> > > - Monitor scrapers to make sure we always provide the latest, most
>> > accurate data available
>> > > - Write an app to track electoral calendars, so we can plan updates >
>> > >  > and
>> > prepare scrapers
>> > > - Donate to Open North so a staff member can improve and maintain
>> > > this
>> > service part-time
>> > >
>> > > Comments are very welcome, and please let us know if you've built
>> > something using the API.
>> > >
>> > > James & Michael
>> > > _______________________________________________
>> > > CivicAccess-discuss mailing list
>> > > [hidden email]
>> > > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> >
>> > _______________________________________________
>> > CivicAccess-discuss mailing list
>> > [hidden email]
>> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.pwd.ca/pipermail/civicaccess-discuss/attachments/20120130/4167f5ec/attachment-0001.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>> End of CivicAccess-discuss Digest, Vol 54, Issue 34
>> ***************************************************
>>
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss 

[James McKinney-2]

All our scrapers are at: https://scraperwiki.com/tags/cdnpoli ScraperWiki allows you to write in PHP, Python or Ruby. All the scrapers are under 100 lines, so it should be fairly easy to read one or two and get a grasp of the process. It's basically the same process for any government website. We have some documentation of what fields to scrape here: http://represent.opennorth.ca/api/#representative

It'd be great if you could share the contact at the City of Guelph who provided the geospatial data. We're not asking cities to commit to open data. We're just asking for a document, and cities are used to giving documents when requested (sometimes you need an FOI, but we try to avoid that). So far, I haven't had trouble getting this uncontroversial data, except from the territories (mostly because they have very small offices, not because they were unwilling).

On 2012-01-31, at 6:34 PM, Nik G wrote:

Hi James,

It's great that you're taking a well thought-out approach RE: scrapers and authoritative datasets.

Can you share what most of the Scrapers used for Represent are written in, and is there any way to quickly identify which of scrapers are most appropriate for different types of council pages?

Regarding ward boundaries, Guelph was from the city via OpenGuelph; London I'm not sure about, and Milton and Hamilton were scraped. I wouldn't be so certain about some muni's like Milton sharing shape/boundary data... in fact there's very little chance we'll be getting anything from Milton at this point, considering just yesterday the council expressed their concerns around "risks" and "costs" associated with open data and the Open Data motion was put on hold. Hamilton may be a possibility. Stay tuned.

Thanks
Nik


-----Original Message----- From: James McKinney
Sent: Tuesday, January 31, 2012 4:41 PM
To: civicaccess discuss
Subject: Re: [CivicAccess-discuss] CivicAccess-discuss Digest, Vol 54,Issue 34

Thanks, Nik! I was planning on writing to you, as a core member of the wardrep.ca team.

In terms of integrating wardrep.ca data, our issue is that the representative data is static (it was manually entered). Manually entered data has a higher maintenance cost than programatically scraped data. A scraper's data will remain accurate for as long as the authority doesn't change its page layout. When they do change the page layout, the scraper is likely to fail, which will alert us to the need to update the scraper. So for representative data, we prefer scraper code to raw data.

With respect to the ward boundaries, as I understand, these were created using Google Earth and PDF references. It is difficult for us to verify the accuracy of such data. We prefer authoritative datasets. You can help contribute to Represent by using your contacts in those cities to request a shapefile of their boroughs. I'm certain they can do that for you.

Thanks,

James

On 2012-01-31, at 1:45 PM, Nik G wrote:

James,

Great project, congrats!

Last year OpenHalton, OpenHamilton, OpenGuelph & London ON hacked WardRep.ca to provide similar type functionality for Municipal Council representatives.

We scraped (and otherwise liberated data from pdfs, etc.) for Ward Boundaries and Council / Mayor info for 4 cities:

Milton, ON

London, ON

Guelph, ON

Hamilton, ON

The data is in DataDOTgc's OGDI instance with APIs, which hopefully you could plug into to easily feed into REPRESENT.

Ward Boundaries:

DATA -- http://datadotgc.cloudapp.net/DataBrowser/def/CityWards

API -- XML: http://datadotgcds.cloudapp.net/v1/def/CityWards  JSON (append ?format=json): http://datadotgcds.cloudapp.net/v1/def/CityWards?format=json

Ward Boundaries is pipe-separated polygon coordinates.

Representatives

DATA --  http://datadotgc.cloudapp.net/DataBrowser/def/WardRepRepresentatives

API --- XML/Atom: http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives JSON: http://datadotgcds.cloudapp.net/v1/def/WardRepRepresentatives?format=json

Nik Garkusha

http://openhalton.ca

-----Original Message----- From: James McKinney

Sent: Tuesday, January 31, 2012 1:16 PM

To: civicaccess discuss

Subject: Re: [CivicAccess-discuss] CivicAccess-discuss Digest, Vol 54,Issue 34

Thanks for the feedback, Joe!

We provide a link in the API to the page where you can download the shapefile, and a link to the license, e.g. http://represent.opennorth.ca/boundary-sets/federal-electoral-districts/?format=apibrowser The majority of the shapefiles are publicly available on the internet. The rest we got by email.

Can't you convert the shapefiles to MySQL yourself using a tool like shp2mysql? I think it's CiviCRM's responsibility to give you tools for loading geospatial data into its database.

I've added a clarification to contact us if you expect to exceed limits.

We admit the inaccuracy of centroids in the documentation. For provinces that post official concordances between postal codes and boundaries, we use that data instead, e.g. http://represent.opennorth.ca/postcodes/K0A1K0/?format=apibrowser

There are roughly a million postal codes, so I doubt that several hundred thousands appear in more than one riding. According Elections ON, of its lion's share of the postal codes, only 2724 map to multiple electoral districts. http://www.elections.on.ca/en-CA/Tools/ElectoralDistricts/PostalCodeFile.htm The PCCF and PCFRF are notorious for errors. GeoCoder.ca instead uses Canada Post's assignment of postal codes to road segments (a postal code is not a polygon but a mail route) to do its geocoding of postal codes.

Cheers,

James

On 2012-01-31, at 12:29 PM, Joe Murray wrote:

If I could echo a few points others have made in my own way:

• For performance reasons I think I would like to get the shape files you have used and make them available for various groups to load them into CiviCRM. One benefit will be to do the calculation locally rather than over the wire via JSON. Another will be to reduce the load on your server when we need to calculate the wards and ridings for databases with tens and hundreds of thousands of records.

• This will require converting them to MySQL's geo format. But since 5.1.26 or so I think, MySQL has provided support for accurate calculation of whether a point is within a polygon (before that it was an inaccurate but quick calculation of whether the point was within the minimum bounding rectangle).

• So I'm looking forward to seeing you publish the sources of the shapefiles and their licenses.

• In some cases the licenses will likely not permit you to share the shapefiles. I imagine that some of my clients who want this information for advocacy purposes would be happy to pay for a higher usage than 60 requests per minute during bulk encoding, since otherwise it will take a couple of days to deal with all of their data (e.g. I recently geocoded 175k addresses for federal riding and where appropriate Toronto ward). With a bit of advance coordination, this might allow you to bump up the capabilities of your vps for a day to handle the increased load, and heh, maybe even subsidize more development or support. ;) A common provision in licenses for this sort of service is to say that there is a cap but if you want to exceed the cap to contact the provider, ie you folks.

• Without wanting to be persnickety, postal code centroids are not generally accurate enough. A few apps I have seen allow people to see the several ridings that their address may be a part of.

• Several years ago, using the voters list data provided for the country to political parties there were a few hundred thousand postal codes that appear in more than one riding if I recall correctly, and the number of ridings could go up to 7 for a particular postal code (I think it was a rural one in the Atlantic provinces for a highway). When I tracked down the quality of the data in the PCFRF through Elections Canada to StatsCanada they gave me a long explanation for why this was so. I bet it is still the case.

Congratulations and thank you so much!!

Joe Murray, PhD

President, JMA Consulting

[hidden email]

skype JosephPMurray twitter JoeMurray

416.466.1281

On Tue, Jan 31, 2012 at 12:00 PM, <[hidden email]> wrote:

Send CivicAccess-discuss mailing list submissions to

     [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit

     http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

or, via email, send a message with subject or body 'help' to

     [hidden email]

You can reach the person managing the list at

     [hidden email]

When replying, please edit your Subject line so it is more specific

than "Re: Contents of CivicAccess-discuss digest..."

Today's Topics:

1. Re: Announcing Represent, an open database of political

   districts and representatives (Cory Horner)

2. Re: Announcing Represent, an open database of political

   districts and representatives (Michael Mulley)

----------------------------------------------------------------------

Message: 1

Date: Mon, 30 Jan 2012 12:58:23 -0800

From: Cory Horner <[hidden email]>

To: civicaccess discuss <[hidden email]>

Subject: Re: [CivicAccess-discuss] Announcing Represent,        an open

     database of political districts and representatives

Message-ID: <[hidden email]>

Content-Type: text/plain; charset=windows-1252

Since our APIs are overlapping, i'm considering shutting mine down since you guys have more coverage (I only have federal, bc, and ontario so far).

- can you add an XML output format?

- can you add an option to return nearby ridings within a certain radius (because the postal code centroid is inaccurate)?

- can people contact you to avoid the 60 request/second limit?

If you can add those... I can punt my users to you and spend my time elsewhere...

Thanks,

Cory.

----

http://howdtheyvote.ca/

On 2012-01-30, at 8:44 AM, James McKinney wrote:

> Open North is proud to announce Represent, an API (web service) to > > federal, provincial and municipal political districts and > > representatives across Canada. To see a demo and read the API docs, > > visit: http://represent.opennorth.ca/

>

> Represent establishes a foundation upon which other projects may be > > built - a necessary piece of infrastructure which we hope will enable > > useful and innovative features and products. It is free to use and > > open-source.

>

> This open database was built by James McKinney and Michael Mulley, with >  > important contributions from Aran Rasmussen, Kent Mewhort, Amir > Pakdel, > Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine > Casciato. The > project takes inspiration from mySociety?s MapIt and > code from the > Chicago Tribune?s Boundary Service.

>

> We currently have electoral districts and representatives for all > > provinces and Canada, and the ward boundaries of ten cities. We have > > postal code correspondences for Ontario (and soon Quebec) and postal > > code centroids from GeoCoder.ca.

>

> But we could have much more. To keep this project sustainable, we are > > asking for your help. Things you can help with:

>

> Add more representatives

> - Write scrapers (in PHP, Ruby, or Python) to get the elected officials >  > for more cities

> - Ask provincial Municipal Affairs ministries for municipal elected > > officials in Excel format

>

> Get more boundaries

> - Create a list of all municipalities with ward boundaries

> - Ask them to provide their ward boundaries under an open license

>

> Free the postal code

> - Ask your province or city for a postal code to electoral district > > concordance file (we already have Ontario, Qu?bec, and Saskatchewan)

>

> Maintain the database

> - Monitor scrapers to make sure we always provide the latest, most > > accurate data available

> - Write an app to track electoral calendars, so we can plan updates and >  > prepare scrapers

> - Donate to Open North so a staff member can improve and maintain this >  > service part-time

>

> Comments are very welcome, and please let us know if you've built > > something using the API.

>

> James & Michael

> _______________________________________________

> CivicAccess-discuss mailing list

> [hidden email]

> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

------------------------------

Message: 2

Date: Mon, 30 Jan 2012 16:56:55 -0500

From: Michael Mulley <[hidden email]>

To: civicaccess discuss <[hidden email]>

Subject: Re: [CivicAccess-discuss] Announcing Represent, an open

     database of political districts and representatives

Message-ID:

<[hidden email]>

Content-Type: text/plain; charset="utf-8"

Hi Cory,

I think a lot of people use & are grateful for your API -- I know I am --

and I'd certainly like to see it stay around.

I've got no plans to add XML and James & I originally planned this a JSON

service, though we might change our minds if several people really want XML.

Nearby: played with this briefly but the range queries I was trying were

making PostGIS slow to a crawl, since we store everything in lat/lon

geometry columns. This is certainly worth adding to our issues list.

As to rate limits, this is a brand-new project & we don't have any kind of

policy yet, but we built this as a tool for people to build on top of, and

we're not gonna stand in the way of legitimate uses without a good reason.

On Mon, Jan 30, 2012 at 3:58 PM, Cory Horner <[hidden email]> wrote:

> Since our APIs are overlapping, i'm considering shutting mine down > since

> you guys have more coverage (I only have federal, bc, and ontario so > > far).

>

> - can you add an XML output format?

> - can you add an option to return nearby ridings within a certain > radius

> (because the postal code centroid is inaccurate)?

> - can people contact you to avoid the 60 request/second limit?

>

> If you can add those... I can punt my users to you and spend my time

> elsewhere...

>

> Thanks,

> Cory.

> ----

> http://howdtheyvote.ca/

>

>

>

> On 2012-01-30, at 8:44 AM, James McKinney wrote:

>

> > Open North is proud to announce Represent, an API (web service) to

> federal, provincial and municipal political districts and > > representatives

> across Canada. To see a demo and read the API docs, visit:

> http://represent.opennorth.ca/

> >

> > Represent establishes a foundation upon which other projects may be

> built - a necessary piece of infrastructure which we hope will enable

> useful and innovative features and products. It is free to use and

> open-source.

> >

> > This open database was built by James McKinney and Michael Mulley, > > >  > with

> important contributions from Aran Rasmussen, Kent Mewhort, Amir Pakdel,

> Camilo Lopez, Rory Geoghegan, Trevor Knight, and Carmine Casciato. The

> project takes inspiration from mySociety?s MapIt and code from the > > Chicago

> Tribune?s Boundary Service.

> >

> > We currently have electoral districts and representatives for all

> provinces and Canada, and the ward boundaries of ten cities. We have > > postal

> code correspondences for Ontario (and soon Quebec) and postal code

> centroids from GeoCoder.ca.

> >

> > But we could have much more. To keep this project sustainable, we are

> asking for your help. Things you can help with:

> >

> > Add more representatives

> > - Write scrapers (in PHP, Ruby, or Python) to get the elected > > > > officials

> for more cities

> > - Ask provincial Municipal Affairs ministries for municipal elected

> officials in Excel format

> >

> > Get more boundaries

> > - Create a list of all municipalities with ward boundaries

> > - Ask them to provide their ward boundaries under an open license

> >

> > Free the postal code

> > - Ask your province or city for a postal code to electoral district

> concordance file (we already have Ontario, Qu?bec, and Saskatchewan)

> >

> > Maintain the database

> > - Monitor scrapers to make sure we always provide the latest, most

> accurate data available

> > - Write an app to track electoral calendars, so we can plan updates > > >  > and

> prepare scrapers

> > - Donate to Open North so a staff member can improve and maintain > > this

> service part-time

> >

> > Comments are very welcome, and please let us know if you've built

> something using the API.

> >

> > James & Michael

> > _______________________________________________

> > CivicAccess-discuss mailing list

> > [hidden email]

> > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

>

> _______________________________________________

> CivicAccess-discuss mailing list

> [hidden email]

> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

>

-------------- next part --------------

An HTML attachment was scrubbed...

URL: <http://lists.pwd.ca/pipermail/civicaccess-discuss/attachments/20120130/4167f5ec/attachment-0001.html>

------------------------------

_______________________________________________

CivicAccess-discuss mailing list

[hidden email]

http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

End of CivicAccess-discuss Digest, Vol 54, Issue 34

***************************************************

_______________________________________________

CivicAccess-discuss mailing list

[hidden email]

http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________

CivicAccess-discuss mailing list

[hidden email]

http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________

CivicAccess-discuss mailing list

[hidden email]

http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss