Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
Locked
Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
 
|
Statistics Canada Consulting on 'data and products - What do users need?
http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a Begin forwarded message: > From: Kevin McArthur <[hidden email]> > Subject: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need? > Date: 18 June, 2012 4:27:09 PM EDT > To: [hidden email] > Reply-To: [hidden email] > > FYI, http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a |
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
I filled the thing in last week. I requested that we be able to search the data geographically, as in from a map, and also that data be aggregated into standardized geographies such as health districts, neighbourhoods, planning areas, and that those geo files also be made available.
Cheers t
On Mon, Jun 18, 2012 at 4:37 PM, James McKinney <[hidden email]> wrote: Statistics Canada Consulting on 'data and products - What do users need? Tracey P. Lauriault 613-234-2805 |
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
Tracey, can you publish your submission somewhere. This is really awesome feedback to give them and will have more weight if a bunch of us give it - I'd love to reemphasize your points! -- @daeaves Sent from my iPhone
|
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
Will do in the next couple of days!
On Mon, Jun 18, 2012 at 5:45 PM, David Eaves <[hidden email]> wrote:
Tracey P. Lauriault 613-234-2805 |
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Tracey P. Lauriault
Hi Tracy ...
While I can appreciate that some might like the data aggregated I find, for myself, this is more of a hindrance than a help. Most of the time when data is aggregated by the data provider it is not what I want or how I want it. I would rather have the data as raw as possible but organized such that I can do my own aggregating. I would also like the data providers spending more effort on just getting the data out and less time on building large and extravagant web portals. ... gerry tychon On 18/06/2012 3:42 PM, Tracey P.
Lauriault wrote:
I filled the thing in last week. I requested that we be able to search the data geographically, as in from a map, and also that data be aggregated into standardized geographies such as health districts, neighbourhoods, planning areas, and that those geo files also be made available. |
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
I don't think this is an either or debate. Great addition Gerry - good points. -- @daeaves Sent from my iPhone
|
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Gerry Tychon-2
In many cases, the raw data cannot be released for privacy concerns. Take the census. They have to aggregate it in order to release it.
Anyway, as David suggested, where possible, data providers should do both: offer "raw" data as well as data that's undergone some more processing. With respect to a similar debate, an FCC representative made an interesting comment at Transparency Camp. The FCC distributes some data through APIs, which its own apps consume. The rep described how some developers complain: "Government shouldn't make apps. Just provide the APIs. Let everyone else make apps." I've heard similar comments from Canadians, too. Anyway, the rep astutely pointed out that if you aren't using your own APIs ("eating your own dogfood"), you'll make really shitty APIs. Of course, some will respond that government should make APIs either, but there are cases where they cannot distribute the data any other way for various reasons. James On 2012-06-18, at 9:06 PM, Gerry Tychon wrote:
|
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
(forgot to put negation) ... some will respond that governments **shouldn't** make APIs either ... On 2012-06-18, at 9:26 PM, James McKinney wrote:
|
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by David Eaves
I feel the same way Gerry, however, in the context of StatCan, data and privacy are paramount, and data will only ever be shared publicly in an aggregated form. You can imagine the privacy issues related to the release of geo-referenced individual points. So in the case of surveys with individual subjects from state agencies, the best we can ask for is aggregation into geographies that make sense to communities, such as neighbourhoods, or wards, or postal regions and to communities of practices such as health districts or economic planning zones. At the moment most data are aggregated only into StatCan geographies which are not always intuitive (e.g., do you know in which census tract or dissemination you live in?), and these do not always line up with an electoral district, and the smaller the geography the more data are suppressed.
Also, what I was hoping for, as a user, was to be able to request all the data products aggregated into wards or health districts, that way I could see the range of products available to me at the geographic unit of my choice and from there I could decide upon some comparable sets for a national or sub-national analysis. Also, I would like to find data via a map, for instance, I would like to select a region and have all the data products associated to that selection region pop up.
On Mon, Jun 18, 2012 at 9:14 PM, David Eaves <[hidden email]> wrote:
Tracey P. Lauriault 613-234-2805 |
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by James McKinney-2
On the issue of APIs and APPs, I am not a developer, but I do analysis, and some policy work, and most citizens are not developers, and many people just want to play with data in a spreadsheet, and on the many topics for which I am not an expert, I want data and I want them along with information product that helps me understand them and know how to model them or question them. Sometimes an app is the best way, sometimes it is a PDF report and at other times it is a sweet visualization with some excellent text and a link to the raw or aggregated data. I want my government to share public data and to create public information products to meet different needs and to disseminate to citizens broadly and then specialist communities next. An open data, information and knowledge dissemination strategy solely centred on developers would be as adequate as the development of a transportation system solely for pavement management systems engineers and car manufacturers.
In my mind, government has a purpose, and that is to govern, and as part of the business of governing it produces data and related products, and that is what I want shared. I do not want it to produce data and products beyond what it needs to do to do its job, unless of course there is a better way for it to do its job (we can debate what that is all day!). In other words I am not sure we should be demanding data in the potentially non persistent apps and APIs for the cell phone or mobile app du jour, especially if those who produce the data as part of their business process do not use or produce them in that way. For instance does everything need to be available on mobile device or is it just nice for it to be so? I know that is probably not a popular view, but I appreciate the Geogratis approach, which was, here are the data in the way we use them (format, etc.) and if you want different format then you do the transformation, or here is a transformation service which you can use yourself, here are the metadata that parameterize what you can do with the data set. And as the Canadian Geoospatial Data Infrastructure evolved with interoperable standards and a more open architecture, data were then created in ways that were more reusable, more standards based, more framework data that others could build on and platform agnostic, and I think that was a good governance model and a good way for a system to evolve. There are over 140 national and regional geospatial data infrastructures in the world and they are talking to each other and are delivering big integratable datasets (e.g., GPS, in car navigation, satellite images, street networks, and more). Open data is nascent and teensy weensy in comparison but in no way insignificant.
I think demanding governments be them big or small to immediately deliver in a way that is not aligned with how they use data internally might not be very helpful, for instance, do I need my StatCan data on a mobile device? It is nice, but really do I? And if you talk to public officials in business units and public officials on the ITS side, you will hear the grumbling between them because ITS expects business units to conform to a technology standard that causes them more work, that is not necessarily attuned to the environments they work or or to the demands of the clients they serve. Instead, starting to share data in their formats and with existing metadata and eventually moving towards a more integrated system might be better. The geospatial world is moving along in a more integrated way as it has had 40+ years to do so and an explicit community with shared needs, open data is new and we are not even near infrastructural consensus let alone interoperability between jurisdictions, law, communities of practice, technology, standards, policies, and so on.
So, lets not ask the government to deliver a specific product in a specific way to meet the needs of only one subset of the citizenry and lets see what kind of system wide approach we can come up with. Ya, the OKNF standards are a good place to start, but even those principles miss inteoperability, so...Developers are clever, they can transform data, providing they have the breadcrumbs they need, and together lets build a robust platform agnostic, multi-jurisdictional interoperable open data ecology/system/infrastructure that can meet many needs.
On Mon, Jun 18, 2012 at 9:28 PM, James McKinney <[hidden email]> wrote:
Tracey P. Lauriault <a href="tel:613-234-2805" value="+16132342805" target="_blank">613-234-2805 |
Re: Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by James McKinney-2
They are also consulting on website. Central issue remains will they go forward with community and user feedback.
-----Original Message----- From: James McKinney <[hidden email]> To: <[hidden email]> Sent: 06/18/2012 4:37:34 PM Subject: [CivicAccess-discuss] Fwd: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need? Statistics Canada Consulting on 'data and products - What do users need? http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a Begin forwarded message: > From: Kevin McArthur <[hidden email]> > Subject: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need? > Date: 18 June, 2012 4:27:09 PM EDT > To: [hidden email] > Reply-To: [hidden email] > > FYI, http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a _______________________________________________ CivicAccess-discuss mailing list [hidden email] http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss |
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Tracey P. Lauriault
On this issue, the whole notion of privacy needs to be reviewed. While I am all for protecting a persons privacy, there are times where data is not released under the "guise" of "privacy". There may be times where a person's name is OK to release. For example, when they consent as a contact say, for community services or program information in service directories.
As for APIs, these are limited in that they only provide a specific view of data based on the programmer's perspective or person who develops the API. They are also very expensive and time consuming to maintain. But they offer a window to data that may take the guesswork out of the technical limits of the data (e.g., changes in geography over time series data such as BFI and DA and CT splits). APIs need to be carefully thought out and released in combination with "raw" data. They are not a silver bullet on their own.
Harvey Low
>>> "Tracey P. Lauriault" <[hidden email]> 6/18/2012 10:34 pm >>>
On the issue of APIs and APPs, I am not a developer, but I do analysis, and some policy work, and most citizens are not developers, and many people just want to play with data in a spreadsheet, and on the many topics for which I am not an expert, I want data and I want them along with information product that helps me understand them and know how to model them or question them. Sometimes an app is the best way, sometimes it is a PDF report and at other times it is a sweet visualization with some excellent text and a link to the raw or aggregated data. I want my government to share public data and to create public information products to meet different needs and to disseminate to citizens broadly and then specialist communities next. An open data, information and knowledge dissemination strategy solely centred on developers would be as adequate as the development of a transportation system solely for pavement management systems engineers and car manufacturers. In my mind, government has a purpose, and that is to govern, and as part of the business of governing it produces data and related products, and that is what I want shared. I do not want it to produce data and products beyond what it needs to do to do its job, unless of course there is a better way for it to do its job (we can debate what that is all day!). In other words I am not sure we should be demanding data in the potentially non persistent apps and APIs for the cell phone or mobile app du jour, especially if those who produce the data as part of their business process do not use or produce them in that way. For instance does everything need to be available on mobile device or is it just nice for it to be so? I know that is probably not a popular view, but I appreciate the Geogratis approach, which was, here are the data in the way we use them (format, etc.) and if you want different format then you do the transformation, or here is a transformation service which you can use yourself, here are the metadata that parameterize what you can do with the data set. And as the Canadian Geoospatial Data Infrastructure evolved with interoperable standards and a more open architecture, data were then created in ways that were more reusable, more standards based, more framework data that others could build on and platform agnostic, and I think that was a good governance model and a good way for a system to evolve. There are over 140 national and regional geospatial data infrastructures in the world and they are talking to each other and are delivering big integratable datasets (e.g., GPS, in car navigation, satellite images, street networks, and more). Open data is nascent and teensy weensy in comparison but in no way insignificant.
I think demanding governments be them big or small to immediately deliver in a way that is not aligned with how they use data internally might not be very helpful, for instance, do I need my StatCan data on a mobile device? It is nice, but really do I? And if you talk to public officials in business units and public officials on the ITS side, you will hear the grumbling between them because ITS expects business units to conform to a technology standard that causes them more work, that is not necessarily attuned to the environments they work or or to the demands of the clients they serve. Instead, starting to share data in their formats and with existing metadata and eventually moving towards a more integrated system might be better. The geospatial world is moving along in a more integrated way as it has had 40+ years to do so and an explicit community with shared needs, open data is new and we are not even near infrastructural consensus let alone interoperability between jurisdictions, law, communities of practice, technology, standards, policies, and so on.
So, lets not ask the government to deliver a specific product in a specific way to meet the needs of only one subset of the citizenry and lets see what kind of system wide approach we can come up with. Ya, the OKNF standards are a good place to start, but even those principles miss inteoperability, so...Developers are clever, they can transform data, providing they have the breadcrumbs they need, and together lets build a robust platform agnostic, multi-jurisdictional interoperable open data ecology/system/infrastructure that can meet many needs.
On Mon, Jun 18, 2012 at 9:28 PM, James McKinney <[hidden email]> wrote:
Tracey P. Lauriault <A href="tel:613-234-2805" target=_blank value="+16132342805">613-234-2805 |
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
You want data release and APIs.
They serve different purposes. They mostly complement each other. The problem with only APIs is that they are usually rate limited, so they will cut you off if your project needs to hit the APIs a lot (as well as the reasons others have referred to). The highest priority is to have a data dump. It is the most broadly usable and useful item for use. A nice-to-have is an API, which facilitates mashups but whose absence can be managed by developers looking after the data themselves. I deal with datasets that are 100s of GB to 10s of TB, so I want to bring them down once across the network. :-) While this is not generally the use case for most developers, it does overlap with many of them. Thanks, Glen On Tue, Jun 19, 2012 at 9:32 AM, Harvey Low <[hidden email]> wrote: > On this issue, the whole notion of privacy needs to be reviewed. While I am > all for protecting a persons privacy, there are times where data is not > released under the "guise" of "privacy". There may be times where a person's > name is OK to release. For example, when they consent as a contact say, for > community services or program information in service directories. > > As for APIs, these are limited in that they only provide a specific view of > data based on the programmer's perspective or person who develops the API. > They are also very expensive and time consuming to maintain. But they offer > a window to data that may take the guesswork out of the technical limits of > the data (e.g., changes in geography over time series data such as BFI and > DA and CT splits). APIs need to be carefully thought out and released in > combination with "raw" data. They are not a silver bullet on their own. > > Harvey Low > > >>>> "Tracey P. Lauriault" <[hidden email]> 6/18/2012 10:34 pm >>> > > On the issue of APIs and APPs, I am not a developer, but I do analysis, and > some policy work, and most citizens are not developers, and many people just > want to play with data in a spreadsheet, and on the many topics for which I > am not an expert, I want data and I want them along with information product > that helps me understand them and know how to model them or question them. > Sometimes an app is the best way, sometimes it is a PDF report and at other > times it is a sweet visualization with some excellent text and a link to the > raw or aggregated data. I want my government to share public data and to > create public information products to meet different needs and to > disseminate to citizens broadly and then specialist communities next. An > open data, information and knowledge dissemination strategy solely centred > on developers would be as adequate as the development of a transportation > system solely for pavement management systems engineers and car > manufacturers. > > In my mind, government has a purpose, and that is to govern, and as part of > the business of governing it produces data and related products, and that is > what I want shared. I do not want it to produce data and products beyond > what it needs to do to do its job, unless of course there is a better way > for it to do its job (we can debate what that is all day!). In other words I > am not sure we should be demanding data in the potentially non persistent > apps and APIs for the cell phone or mobile app du jour, especially if those > who produce the data as part of their business process do not use or produce > them in that way. For instance does everything need to be available on > mobile device or is it just nice for it to be so? I know that is probably > not a popular view, but I appreciate the Geogratis approach, which was, here > are the data in the way we use them (format, etc.) and if you want different > format then you do the transformation, or here is a transformation service > which you can use yourself, here are the metadata that parameterize what you > can do with the data set. And as the Canadian Geoospatial Data > Infrastructure evolved with interoperable standards and a more open > architecture, data were then created in ways that were more reusable, more > standards based, more framework data that others could build on and platform > agnostic, and I think that was a good governance model and a good way for a > system to evolve. There are over 140 national and regional geospatial data > infrastructures in the world and they are talking to each other and are > delivering big integratable datasets (e.g., GPS, in car navigation, > satellite images, street networks, and more). Open data is nascent and > teensy weensy in comparison but in no way insignificant. > > I think demanding governments be them big or small to immediately deliver in > a way that is not aligned with how they use data internally might not be > very helpful, for instance, do I need my StatCan data on a mobile device? It > is nice, but really do I? And if you talk to public officials in business > units and public officials on the ITS side, you will hear the grumbling > between them because ITS expects business units to conform to a technology > standard that causes them more work, that is not necessarily attuned to the > environments they work or or to the demands of the clients they serve. > Instead, starting to share data in their formats and with existing metadata > and eventually moving towards a more integrated system might be better. The > geospatial world is moving along in a more integrated way as it has had 40+ > years to do so and an explicit community with shared needs, open data is new > and we are not even near infrastructural consensus let alone > interoperability between jurisdictions, law, communities of practice, > technology, standards, policies, and so on. > > So, lets not ask the government to deliver a specific product in a specific > way to meet the needs of only one subset of the citizenry and lets see what > kind of system wide approach we can come up with. Ya, the OKNF standards are > a good place to start, but even those principles miss inteoperability, > so...Developers are clever, they can transform data, providing they have the > breadcrumbs they need, and together lets build a robust platform agnostic, > multi-jurisdictional interoperable open data ecology/system/infrastructure > that can meet many needs. > > On Mon, Jun 18, 2012 at 9:28 PM, James McKinney <[hidden email]> wrote: >> >> (forgot to put negation) ... some will respond that governments >> **shouldn't** make APIs either ... >> >> On 2012-06-18, at 9:26 PM, James McKinney wrote: >> >> In many cases, the raw data cannot be released for privacy concerns. Take >> the census. They have to aggregate it in order to release it. >> >> In any case, not everyone can manipulate the raw data that easily, but >> they can use data that's already aggregated into standard geographies. If >> data providers only offered "raw" data, only the most technically proficient >> would benefit. I think it's good to start with raw data, but I still >> recognize the need for some more processed data. >> >> Anyway, as David suggested, where possible, data providers should do both: >> offer "raw" data as well as data that's undergone some more processing. >> >> With respect to a similar debate, an FCC representative made an >> interesting comment at Transparency Camp. The FCC distributes some data >> through APIs, which its own apps consume. The rep described how some >> developers complain: "Government shouldn't make apps. Just provide the APIs. >> Let everyone else make apps." I've heard similar comments from Canadians, >> too. Anyway, the rep astutely pointed out that if you aren't using your own >> APIs ("eating your own dogfood"), you'll make really shitty APIs. >> >> Of course, some will respond that government should make APIs either, but >> there are cases where they cannot distribute the data any other way for >> various reasons. >> >> James >> >> On 2012-06-18, at 9:06 PM, Gerry Tychon wrote: >> >> Hi Tracy ... >> >> While I can appreciate that some might like the data aggregated I find, >> for myself, this is more of a hindrance than a help. Most of the time when >> data is aggregated by the data provider it is not what I want or how I want >> it. I would rather have the data as raw as possible but organized such that >> I can do my own aggregating. >> >> I would also like the data providers spending more effort on just getting >> the data out and less time on building large and extravagant web portals. >> >> ... gerry tychon >> >> >> On 18/06/2012 3:42 PM, Tracey P. Lauriault wrote: >> >> I filled the thing in last week. I requested that we be able to search the >> data geographically, as in from a map, and also that data be aggregated into >> standardized geographies such as health districts, neighbourhoods, planning >> areas, and that those geo files also be made available. >> >> Cheers >> t >> >> On Mon, Jun 18, 2012 at 4:37 PM, James McKinney <[hidden email]> >> wrote: >>> >>> Statistics Canada Consulting on 'data and products - What do users need? >>> >>> http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a >>> >>> >>> Begin forwarded message: >>> >>> > From: Kevin McArthur <[hidden email]> >>> > Subject: [OpenDataBC] Statistics Canada Consulting on 'data and >>> > products - What do users need? >>> > Date: 18 June, 2012 4:27:09 PM EDT >>> > To: [hidden email] >>> > Reply-To: [hidden email] >>> > >>> > FYI, >>> > http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a >>> >>> _______________________________________________ >>> CivicAccess-discuss mailing list >>> [hidden email] >>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >> >> >> >> >> -- >> Tracey P. Lauriault >> 613-234-2805 >> >> >> >> _______________________________________________ >> CivicAccess-discuss mailing list >> [hidden email] >> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >> >> >> >> _______________________________________________ >> CivicAccess-discuss mailing list >> [hidden email] >> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >> >> >> >> >> _______________________________________________ >> CivicAccess-discuss mailing list >> [hidden email] >> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss > > > > > -- > Tracey P. Lauriault > 613-234-2805 > > > _______________________________________________ > CivicAccess-discuss mailing list > [hidden email] > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss -- - http://zzzoot.blogspot.com/ - |
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Harvey Low
The question of privacy cannot be
avoided. In order to publish names and personnal information
(enough to identify someone), it means changing some major laws
(like the code civil in Québec which is the cornerstone of the
legal system).
One of my recent request for information was rejected because it would provide some names. I read the laws (code civil and protection of personnal information) and it's true it's really a factor that limites the capability of agencies. In my case, I appealed since I don't need to identify the people, I just want a list of events. But removing the personal information is a problem: needs data manipulation (cost) and might lose some information. Where I worked, we had a "data scrambler" tool that allowed us to test some software on some real data but with personal information randomly hashed. Obviously, it was none personal, but it removed the opportunity to test longitudinal feature (how a given client behave in the time). Steph Le 12-06-19 09:32, Harvey Low a écrit :
|
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
Yes we need to address legislation. Assessment Act is another one that is dated. In the end, names are what some people want and what serve providers want to give out! some people still prefer names and phone numbers when they want information about services - go figure :)
>>> Stéphane Guidoin<[hidden email]> 6/19/2012 9:56 am >>> The question of privacy cannot be avoided. In order to publish names and personnal information (enough to identify someone), it means changing some major laws (like the code civil in Québec which is the cornerstone of the legal system).
One of my recent request for information was rejected because it would provide some names. I read the laws (code civil and protection of personnal information) and it's true it's really a factor that limites the capability of agencies. In my case, I appealed since I don't need to identify the people, I just want a list of events. But removing the personal information is a problem: needs data manipulation (cost) and might lose some information. Where I worked, we had a "data scrambler" tool that allowed us to test some software on some real data but with personal information randomly hashed. Obviously, it was none personal, but it removed the opportunity to test longitudinal feature (how a given client behave in the time). Steph Le 12-06-19 09:32, Harvey Low a écrit :
|
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Harvey Low
A small note related to privacy but instead of
names it is location. In some cases, data is not released or the
location is aggregated/randomized to protect privacy. I am
specifically thinking of crime information. I can see this for
private individuals but when it comes to your local shopping centre,
I would like to know in a more precise way where those auto thefts
and similar occurred.
... gerry tychon On 19/06/2012 7:32 AM, Harvey Low
wrote:
|
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
Le 19 juin 2012 à 10:44, Gerry Tychon a écrit : > I would like to know in a more precise way where those auto thefts and similar occurred. why? -- Karl Dubost Montréal, QC, Canada http://www.la-grange.net/karl/ |
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Gerry Tychon-2
Indeed. Some inroads have been made in NYC on that front where they provide map data by location of homicide.
>>> Gerry Tychon <[hidden email]> 6/19/2012 10:44 am >>> A small note related to privacy but instead of names it is location. In some cases, data is not released or the location is aggregated/randomized to protect privacy. I am specifically thinking of crime information. I can see this for private individuals but when it comes to your local shopping centre, I would like to know in a more precise way where those auto thefts and similar occurred. ... gerry tychon On 19/06/2012 7:32 AM, Harvey Low wrote:
|
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Glen Newton
I assume it'd be faster to ship 10s of TBs than to offer a download, no? If there are uses cases for using small excerpts of such a database, a government-issue API is a way better option that getting hard drives delivered, installing the hardware, reading the data into a fairly powerful server, and finally querying the database, likely with your own custom made API... Of course, people who really do want it all should be able to request it.
On 2012-06-19, at 9:43 AM, Glen Newton wrote: > You want data release and APIs. > They serve different purposes. They mostly complement each other. > > The problem with only APIs is that they are usually rate limited, so > they will cut you off if your project needs to hit the APIs a lot (as > well as the reasons others have referred to). > > The highest priority is to have a data dump. It is the most broadly > usable and useful item for use. > A nice-to-have is an API, which facilitates mashups but whose absence > can be managed by developers looking after the data themselves. > > I deal with datasets that are 100s of GB to 10s of TB, so I want to > bring them down once across the network. :-) > While this is not generally the use case for most developers, it does > overlap with many of them. > > Thanks, > Glen > > > > On Tue, Jun 19, 2012 at 9:32 AM, Harvey Low <[hidden email]> wrote: >> On this issue, the whole notion of privacy needs to be reviewed. While I am >> all for protecting a persons privacy, there are times where data is not >> released under the "guise" of "privacy". There may be times where a person's >> name is OK to release. For example, when they consent as a contact say, for >> community services or program information in service directories. >> >> As for APIs, these are limited in that they only provide a specific view of >> data based on the programmer's perspective or person who develops the API. >> They are also very expensive and time consuming to maintain. But they offer >> a window to data that may take the guesswork out of the technical limits of >> the data (e.g., changes in geography over time series data such as BFI and >> DA and CT splits). APIs need to be carefully thought out and released in >> combination with "raw" data. They are not a silver bullet on their own. >> >> Harvey Low >> >> >>>>> "Tracey P. Lauriault" <[hidden email]> 6/18/2012 10:34 pm >>> >> >> On the issue of APIs and APPs, I am not a developer, but I do analysis, and >> some policy work, and most citizens are not developers, and many people just >> want to play with data in a spreadsheet, and on the many topics for which I >> am not an expert, I want data and I want them along with information product >> that helps me understand them and know how to model them or question them. >> Sometimes an app is the best way, sometimes it is a PDF report and at other >> times it is a sweet visualization with some excellent text and a link to the >> raw or aggregated data. I want my government to share public data and to >> create public information products to meet different needs and to >> disseminate to citizens broadly and then specialist communities next. An >> open data, information and knowledge dissemination strategy solely centred >> on developers would be as adequate as the development of a transportation >> system solely for pavement management systems engineers and car >> manufacturers. >> >> In my mind, government has a purpose, and that is to govern, and as part of >> the business of governing it produces data and related products, and that is >> what I want shared. I do not want it to produce data and products beyond >> what it needs to do to do its job, unless of course there is a better way >> for it to do its job (we can debate what that is all day!). In other words I >> am not sure we should be demanding data in the potentially non persistent >> apps and APIs for the cell phone or mobile app du jour, especially if those >> who produce the data as part of their business process do not use or produce >> them in that way. For instance does everything need to be available on >> mobile device or is it just nice for it to be so? I know that is probably >> not a popular view, but I appreciate the Geogratis approach, which was, here >> are the data in the way we use them (format, etc.) and if you want different >> format then you do the transformation, or here is a transformation service >> which you can use yourself, here are the metadata that parameterize what you >> can do with the data set. And as the Canadian Geoospatial Data >> Infrastructure evolved with interoperable standards and a more open >> architecture, data were then created in ways that were more reusable, more >> standards based, more framework data that others could build on and platform >> agnostic, and I think that was a good governance model and a good way for a >> system to evolve. There are over 140 national and regional geospatial data >> infrastructures in the world and they are talking to each other and are >> delivering big integratable datasets (e.g., GPS, in car navigation, >> satellite images, street networks, and more). Open data is nascent and >> teensy weensy in comparison but in no way insignificant. >> >> I think demanding governments be them big or small to immediately deliver in >> a way that is not aligned with how they use data internally might not be >> very helpful, for instance, do I need my StatCan data on a mobile device? It >> is nice, but really do I? And if you talk to public officials in business >> units and public officials on the ITS side, you will hear the grumbling >> between them because ITS expects business units to conform to a technology >> standard that causes them more work, that is not necessarily attuned to the >> environments they work or or to the demands of the clients they serve. >> Instead, starting to share data in their formats and with existing metadata >> and eventually moving towards a more integrated system might be better. The >> geospatial world is moving along in a more integrated way as it has had 40+ >> years to do so and an explicit community with shared needs, open data is new >> and we are not even near infrastructural consensus let alone >> interoperability between jurisdictions, law, communities of practice, >> technology, standards, policies, and so on. >> >> So, lets not ask the government to deliver a specific product in a specific >> way to meet the needs of only one subset of the citizenry and lets see what >> kind of system wide approach we can come up with. Ya, the OKNF standards are >> a good place to start, but even those principles miss inteoperability, >> so...Developers are clever, they can transform data, providing they have the >> breadcrumbs they need, and together lets build a robust platform agnostic, >> multi-jurisdictional interoperable open data ecology/system/infrastructure >> that can meet many needs. >> >> On Mon, Jun 18, 2012 at 9:28 PM, James McKinney <[hidden email]> wrote: >>> >>> (forgot to put negation) ... some will respond that governments >>> **shouldn't** make APIs either ... >>> >>> On 2012-06-18, at 9:26 PM, James McKinney wrote: >>> >>> In many cases, the raw data cannot be released for privacy concerns. Take >>> the census. They have to aggregate it in order to release it. >>> >>> In any case, not everyone can manipulate the raw data that easily, but >>> they can use data that's already aggregated into standard geographies. If >>> data providers only offered "raw" data, only the most technically proficient >>> would benefit. I think it's good to start with raw data, but I still >>> recognize the need for some more processed data. >>> >>> Anyway, as David suggested, where possible, data providers should do both: >>> offer "raw" data as well as data that's undergone some more processing. >>> >>> With respect to a similar debate, an FCC representative made an >>> interesting comment at Transparency Camp. The FCC distributes some data >>> through APIs, which its own apps consume. The rep described how some >>> developers complain: "Government shouldn't make apps. Just provide the APIs. >>> Let everyone else make apps." I've heard similar comments from Canadians, >>> too. Anyway, the rep astutely pointed out that if you aren't using your own >>> APIs ("eating your own dogfood"), you'll make really shitty APIs. >>> >>> Of course, some will respond that government should make APIs either, but >>> there are cases where they cannot distribute the data any other way for >>> various reasons. >>> >>> James >>> >>> On 2012-06-18, at 9:06 PM, Gerry Tychon wrote: >>> >>> Hi Tracy ... >>> >>> While I can appreciate that some might like the data aggregated I find, >>> for myself, this is more of a hindrance than a help. Most of the time when >>> data is aggregated by the data provider it is not what I want or how I want >>> it. I would rather have the data as raw as possible but organized such that >>> I can do my own aggregating. >>> >>> I would also like the data providers spending more effort on just getting >>> the data out and less time on building large and extravagant web portals. >>> >>> ... gerry tychon >>> >>> >>> On 18/06/2012 3:42 PM, Tracey P. Lauriault wrote: >>> >>> I filled the thing in last week. I requested that we be able to search the >>> data geographically, as in from a map, and also that data be aggregated into >>> standardized geographies such as health districts, neighbourhoods, planning >>> areas, and that those geo files also be made available. >>> >>> Cheers >>> t >>> >>> On Mon, Jun 18, 2012 at 4:37 PM, James McKinney <[hidden email]> >>> wrote: >>>> >>>> Statistics Canada Consulting on 'data and products - What do users need? >>>> >>>> http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a >>>> >>>> >>>> Begin forwarded message: >>>> >>>>> From: Kevin McArthur <[hidden email]> >>>>> Subject: [OpenDataBC] Statistics Canada Consulting on 'data and >>>>> products - What do users need? >>>>> Date: 18 June, 2012 4:27:09 PM EDT >>>>> To: [hidden email] >>>>> Reply-To: [hidden email] >>>>> >>>>> FYI, >>>>> http://www.statcan.gc.ca/consult/2012/access-acces-eng.htm?bsf=121303a >>>> >>>> _______________________________________________ >>>> CivicAccess-discuss mailing list >>>> [hidden email] >>>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >>> >>> >>> >>> >>> -- >>> Tracey P. Lauriault >>> 613-234-2805 >>> >>> >>> >>> _______________________________________________ >>> CivicAccess-discuss mailing list >>> [hidden email] >>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >>> >>> >>> >>> _______________________________________________ >>> CivicAccess-discuss mailing list >>> [hidden email] >>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >>> >>> >>> >>> >>> _______________________________________________ >>> CivicAccess-discuss mailing list >>> [hidden email] >>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss >> >> >> >> >> -- >> Tracey P. Lauriault >> 613-234-2805 >> >> >> _______________________________________________ >> CivicAccess-discuss mailing list >> [hidden email] >> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss > > > > -- > - > http://zzzoot.blogspot.com/ > - > _______________________________________________ > CivicAccess-discuss mailing list > [hidden email] > http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss |
Re: [OpenDataBC] Statistics Canada Consulting on 'data and products - What do users need?
|
|
In reply to this post by Harvey Low
On 2012-06-19, at 9:32 AM, Harvey Low wrote:
APIs take effort to get right, and nothing is a silver bullet. But I feel a lot of people are down on open data APIs, including people who make very effective, efficient use of various other APIs. It's actually extremely rare in my experience that I think to myself, "this API sucks." In the vast majority of cases, an API does exactly as much as I need it to do. Systems like CKAN and Socrata offer standard APIs on top of tabular data: http://opendata.socrata.com/api/docs/views Many APIs, such as these, are just fancy table readers (whether that table is a CSV file, a SQL table, or otherwise). If the source data is tabular, then an API that functions as a table reader is very close to your having the data file itself. A simple API like this is easy to get right and doesn't require much maintenance. Just to be clear, I'm all for bulk downloads whenever possible, and I don't think the government should delay a data release until it has an API ready. But I also see the value of APIs, and I think the issues around APIs have been exaggerated. |
«
Return to CivicAccess
|
1 view|%1 views
| Free forum by Nabble | Edit this page |