CivicAccess

Abolition of Cost Recovery - StatCan

Classic

List

Threaded

37 messages Options

David Eaves

Re: Government Expenses OPENING

Hi Drew,

Thank you for the update - where are you grabbing the info from?

Interesting stuff.

cheers,
dave

On 11-01-06 9:53 AM, Drew Mcpherson wrote:

Hi Everyone, I’m the guy who did the government expenses crawler / report site and I am just finishing up a run of the engine in the next few days to get the latest 2010 data. I just wanted to let you know that when it’s finished, I’ll open up the search tool to the public for the day in case you were interested to peruse it. Oh and speaking of StatsCan cost recovery, if anyone wants to help support my project through a donation or subscription, it wouldn’t take much to help me out and I could afford to stop couch surfing :) I guarantee my efficiency is far superior to any government department and probably well worth it.

Cheers,

Drew Mcpherson

Bine Consulting Corp.
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Drew Mcpherson

Re: Government Expenses OPENING

The information is published across hundreds of official government websites. Each site has a proactive disclosure area with oftentimes thousands of pages to sift through.

Regards,

Drew Mcpherson

Bine Consulting Corp.

From: [hidden email]

Sent: Saturday, January 08, 2011 11:05 PM

To: [hidden email]

Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

Thank you for the update - where are you grabbing the info from?

Interesting stuff.

cheers,
dave

On 11-01-06 9:53 AM, Drew Mcpherson wrote:

Hi Everyone, I’m the guy who did the government expenses crawler / report site and I am just finishing up a run of the engine in the next few days to get the latest 2010 data. I just wanted to let you know that when it’s finished, I’ll open up the search tool to the public for the day in case you were interested to peruse it. Oh and speaking of StatsCan cost recovery, if anyone wants to help support my project through a donation or subscription, it wouldn’t take much to help me out and I could afford to stop couch surfing :) I guarantee my efficiency is far superior to any government department and probably well worth it.

Cheers,

Drew Mcpherson

Bine Consulting Corp.
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Drew Mcpherson

Re: Government Expenses OPENING

In reply to this post by David Eaves

By the way, I’m going to create some new reports to show the spending changes between 2009 and 2010 across departments and individuals to illustrate whose spending increased the most and whose decreased the most. Commend those who are saving money and cutting costs and draw out those who are spending more to find out why. Does anyone have any requests for other reports that might be useful?

Thanks,

Drew Mcpherson

Bine Consulting Corp.

From: [hidden email]

Sent: Saturday, January 08, 2011 11:05 PM

To: [hidden email]

Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

Thank you for the update - where are you grabbing the info from?

Interesting stuff.

cheers,
dave

On 11-01-06 9:53 AM, Drew Mcpherson wrote:

Hi Everyone, I’m the guy who did the government expenses crawler / report site and I am just finishing up a run of the engine in the next few days to get the latest 2010 data. I just wanted to let you know that when it’s finished, I’ll open up the search tool to the public for the day in case you were interested to peruse it. Oh and speaking of StatsCan cost recovery, if anyone wants to help support my project through a donation or subscription, it wouldn’t take much to help me out and I could afford to stop couch surfing :) I guarantee my efficiency is far superior to any government department and probably well worth it.

Cheers,

Drew Mcpherson

Bine Consulting Corp.
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Glen Newton

Re: Government Expenses OPENING

In reply to this post by Drew Mcpherson

Hi Drew,

Do you have an API or data dump of the crawl/collection for others to
use/reuse/mashup?

Because if you don't have these, then it isn't an open data project.
If you don't have these, all you've done is changed the situation of
the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.

Glen

On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:

> Hi Everyone, I’m the guy who did the government expenses crawler / report
> site and I am just finishing up a run of the engine in the next few days to
> get the latest 2010 data. I just wanted to let you know that when it’s
> finished, I’ll open up the search tool to the public for the day in case you
> were interested to peruse it. Oh and speaking of StatsCan cost recovery, if
> anyone wants to help support my project through a donation or subscription,
> it wouldn’t take much to help me out and I could afford to stop couch
> surfing :) I guarantee my efficiency is far superior to any government
> department and probably well worth it.
>
> Cheers,
> Drew Mcpherson
> Bine Consulting Corp.
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>

--

-

Drew Mcpherson

Re: Government Expenses OPENING

Indeed you are right.

I would love nothing more than to be able to release all of my work for
free. When I can afford a place to live and don't have to use the food bank
I'll definitely give more away. It seems that's not the world we live in
however, and no I am not exaggerating. I left a get-rich-quick public
service project because I was so disgusted with the state of the world that
there were 30 engineers receiving six figure salaries for several years to
fail to complete a project I could complete personally single handed in a
few months.

The world is not going to change over night, so the only thing that me
releasing everything for free would do is bankrupt me in a delusional
attempt to force change too quickly. So until the world changes a little
with me, I will try to recoup my losses for my public service projects in
any way I can which doesn't violate my ethical principles.

Tell you what, I'll make a personal promise. For every 10 dollars donated,
I will release public access to the custom data search tool I built (far
more useful and functional than a bulk data dump), for a full day. This
tool gives complete access to the database, but with a UI that more people
than just hard core programmers will understand.

So far to date there has been exactly 10 dollars donated, so that accounts
for the 1 day that I will release public access to the data once it has been
acquired and sorted through (should be in the next few days if all goes
well). For just a few thousand dollars that would provide for complete
access to all Canadians for the entire year, which is a LOT less than any
other contractor on the planet would charge. If only 1 percent of all
Canadians donated a single penny it would pay for the entire year. Does
that sound fair? I doubt you'll find a better deal on anything, anywhere,
ever, lol :)

It is a lot easier to continue my work on this and other public service /
humanitarian projects if I have a place to live.

Cheers,
Drew

-----Original Message-----
From: Glen Newton
Sent: Sunday, January 09, 2011 12:00 PM
To: civicaccess discuss ; GOSLING members in Ottawa
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

Do you have an API or data dump of the crawl/collection for others to
use/reuse/mashup?

Because if you don't have these, then it isn't an open data project.
If you don't have these, all you've done is changed the situation of
the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.

Glen

On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:

> Hi Everyone, I’m the guy who did the government expenses crawler / report
> site and I am just finishing up a run of the engine in the next few days
> to
> get the latest 2010 data. I just wanted to let you know that when it’s
> finished, I’ll open up the search tool to the public for the day in case
> you
> were interested to peruse it. Oh and speaking of StatsCan cost recovery,
> if
> anyone wants to help support my project through a donation or
> subscription,
> it wouldn’t take much to help me out and I could afford to stop couch
> surfing :) I guarantee my efficiency is far superior to any government
> department and probably well worth it.
>
> Cheers,
> Drew Mcpherson
> Bine Consulting Corp.
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>

--

-
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Drew Mcpherson

Re: Government Expenses OPENING

In reply to this post by Glen Newton

Ok you convinced me. The data is now open, searchable and downloadable in
Excel format. By the way whoever put the site on stumble, thanks for that.
The site has been getting quite a bit of traffic from there all day today!

Regards,
Drew Mcpherson

-----Original Message-----
From: Glen Newton
Sent: Sunday, January 09, 2011 12:00 PM
To: civicaccess discuss ; GOSLING members in Ottawa
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

Do you have an API or data dump of the crawl/collection for others to
use/reuse/mashup?

Because if you don't have these, then it isn't an open data project.
If you don't have these, all you've done is changed the situation of
the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.

Glen

On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:

--

-
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Jonathan Brun-2

Re: Government Expenses OPENING

Hey Drew,

It's Jonathan from Montreal Ouvert - we spoke some time ago. Congrats on the excel dump, very cool. If I might jump in, I think one thing that would help "sell" your site and get donations would be a bit of a esthetic makeover and perhaps visualization à la http://ffctn.com/a/expensevisualizer/ - by the way (and sorry if this has already been said) how is your database different from theirs?

I will mention all this to Sebastien Pierre (who did the visualizer and our site http://www.montrealouvert.net) and see what he thinks.

Also we are doing an open-data hackathon on january 29th and perhaps someone could work on your site's front end a bit, let me know what you think.

JB

On 2011-01-11, at 6:51 PM, Drew Mcpherson wrote:

> Ok you convinced me. The data is now open, searchable and downloadable in Excel format. By the way whoever put the site on stumble, thanks for that. The site has been getting quite a bit of traffic from there all day today!
>
> Regards,
> Drew Mcpherson
>
>
> -----Original Message----- From: Glen Newton
> Sent: Sunday, January 09, 2011 12:00 PM
> To: civicaccess discuss ; GOSLING members in Ottawa
> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>
> Hi Drew,
>
> Do you have an API or data dump of the crawl/collection for others to
> use/reuse/mashup?
>
> Because if you don't have these, then it isn't an open data project.
> If you don't have these, all you've done is changed the situation of
> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>
> Glen
>
>
> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>> Hi Everyone, I’m the guy who did the government expenses crawler / report
>> site and I am just finishing up a run of the engine in the next few days to
>> get the latest 2010 data. I just wanted to let you know that when it’s
>> finished, I’ll open up the search tool to the public for the day in case you
>> were interested to peruse it. Oh and speaking of StatsCan cost recovery, if
>> anyone wants to help support my project through a donation or subscription,
>> it wouldn’t take much to help me out and I could afford to stop couch
>> surfing :) I guarantee my efficiency is far superior to any government
>> department and probably well worth it.
>>
>> Cheers,
>> Drew Mcpherson
>> Bine Consulting Corp.
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>
>
>
> --
>
> -
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

David Eaves

Re: Government Expenses OPENING

In reply to this post by Drew Mcpherson

Drew - this is great news.

To follow up on Jonathan's comment, another thing for someone to
consider is creating an API, I could imagine some sites like Open
Parliament using that to list expenses for a given member of parliament.

Congratulations to being stumbled, I linked to your site on my blog today.

cheers,
dave

On 11-01-11 3:51 PM, Drew Mcpherson wrote:

> Ok you convinced me. The data is now open, searchable and
> downloadable in Excel format. By the way whoever put the site on
> stumble, thanks for that. The site has been getting quite a bit of
> traffic from there all day today!
>
> Regards,
> Drew Mcpherson
>
>
> -----Original Message----- From: Glen Newton
> Sent: Sunday, January 09, 2011 12:00 PM
> To: civicaccess discuss ; GOSLING members in Ottawa
> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>
> Hi Drew,
>
> Do you have an API or data dump of the crawl/collection for others to
> use/reuse/mashup?
>
> Because if you don't have these, then it isn't an open data project.
> If you don't have these, all you've done is changed the situation of
> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>
> Glen
>
>
> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>> Hi Everyone, I’m the guy who did the government expenses crawler /
>> report
>> site and I am just finishing up a run of the engine in the next few
>> days to
>> get the latest 2010 data. I just wanted to let you know that when it’s
>> finished, I’ll open up the search tool to the public for the day in
>> case you
>> were interested to peruse it. Oh and speaking of StatsCan cost
>> recovery, if
>> anyone wants to help support my project through a donation or
>> subscription,
>> it wouldn’t take much to help me out and I could afford to stop couch
>> surfing :) I guarantee my efficiency is far superior to any government
>> department and probably well worth it.
>>
>> Cheers,
>> Drew Mcpherson
>> Bine Consulting Corp.
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>
>
>

Drew Mcpherson

Re: Government Expenses OPENING

In reply to this post by Jonathan Brun-2

Hi Jonathan,

Yes the site does indeed need a makeover to slick it up. That's definitely
not my strong suit, so any suggestions or offers would be most appreciated.
The database at the expense visualizer is a couple of years old and looks
like it only has 23 departments.

That'd be cool if somebody wanted to spiffy up the front end for a
hackathon! I've been pondering using open data in a geolocated chat
application I wrote to suggest cultural activities and other things to the
chat users.

Thanks,
Drew

-----Original Message-----
From: Jonathan Brun
Sent: Tuesday, January 11, 2011 8:00 PM
To: civicaccess discuss
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hey Drew,

It's Jonathan from Montreal Ouvert - we spoke some time ago. Congrats on the
excel dump, very cool. If I might jump in, I think one thing that would help
"sell" your site and get donations would be a bit of a esthetic makeover and
perhaps visualization à la http://ffctn.com/a/expensevisualizer/ - by the
way (and sorry if this has already been said) how is your database different
from theirs?

I will mention all this to Sebastien Pierre (who did the visualizer and our
site http://www.montrealouvert.net) and see what he thinks.

Also we are doing an open-data hackathon on january 29th and perhaps someone
could work on your site's front end a bit, let me know what you think.

JB

On 2011-01-11, at 6:51 PM, Drew Mcpherson wrote:

> Ok you convinced me. The data is now open, searchable and downloadable in
> Excel format. By the way whoever put the site on stumble, thanks for
> that. The site has been getting quite a bit of traffic from there all day
> today!
>
> Regards,
> Drew Mcpherson
>
>
> -----Original Message----- From: Glen Newton
> Sent: Sunday, January 09, 2011 12:00 PM
> To: civicaccess discuss ; GOSLING members in Ottawa
> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>
> Hi Drew,
>
> Do you have an API or data dump of the crawl/collection for others to
> use/reuse/mashup?
>
> Because if you don't have these, then it isn't an open data project.
> If you don't have these, all you've done is changed the situation of
> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>
> Glen
>
>
> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>> Hi Everyone, I’m the guy who did the government expenses crawler / report
>> site and I am just finishing up a run of the engine in the next few days
>> to
>> get the latest 2010 data. I just wanted to let you know that when it’s
>> finished, I’ll open up the search tool to the public for the day in case
>> you
>> were interested to peruse it. Oh and speaking of StatsCan cost recovery,
>> if
>> anyone wants to help support my project through a donation or
>> subscription,
>> it wouldn’t take much to help me out and I could afford to stop couch
>> surfing :) I guarantee my efficiency is far superior to any government
>> department and probably well worth it.
>>
>> Cheers,
>> Drew Mcpherson
>> Bine Consulting Corp.
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>
>
>
> --
>
> -
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Glen Newton

Re: Government Expenses OPENING

In reply to this post by Drew Mcpherson

Hi Drew,

Drew Mcpherson <[hidden email]> wrote:
> The data is now open, searchable and downloadable in Excel format

The site allows someone to get the first 1000 records of the data in
Excel format.
Go to: http://www.governmentexpenses.ca/custom_reports.aspx and just
click on "Generate custom report" on the bottom.
The page that generates this report says:
"There were 1000 results found.
Click here to download the report
PLEASE NOTE: Only a maximum of 1000 records are shown in the custom
search, for more records, please purchase the complete database.
Purchase the entire database now"

So the data is NOT open, is searchable, and is _partially_ downloadable.

And I am not so sure you can legally sell this collection.

Thanks,
Glen

On Tue, Jan 11, 2011 at 6:51 PM, Drew Mcpherson <[hidden email]> wrote:

> Ok you convinced me. The data is now open, searchable and downloadable in
> Excel format. By the way whoever put the site on stumble, thanks for that.
> The site has been getting quite a bit of traffic from there all day today!
>
> Regards,
> Drew Mcpherson
>
>
> -----Original Message----- From: Glen Newton
> Sent: Sunday, January 09, 2011 12:00 PM
> To: civicaccess discuss ; GOSLING members in Ottawa
> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>
> Hi Drew,
>
> Do you have an API or data dump of the crawl/collection for others to
> use/reuse/mashup?
>
> Because if you don't have these, then it isn't an open data project.
> If you don't have these, all you've done is changed the situation of
> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>
> Glen
>
>
> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>>
>> Hi Everyone, I’m the guy who did the government expenses crawler / report
>> site and I am just finishing up a run of the engine in the next few days
>> to
>> get the latest 2010 data. I just wanted to let you know that when it’s
>> finished, I’ll open up the search tool to the public for the day in case
>> you
>> were interested to peruse it. Oh and speaking of StatsCan cost recovery,
>> if
>> anyone wants to help support my project through a donation or
>> subscription,
>> it wouldn’t take much to help me out and I could afford to stop couch
>> surfing :) I guarantee my efficiency is far superior to any government
>> department and probably well worth it.
>>
>> Cheers,
>> Drew Mcpherson
>> Bine Consulting Corp.
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>
>
>
> --
>
> -
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

--

-

Drew Mcpherson

Re: Government Expenses OPENING

In reply to this post by David Eaves

I suppose I could provide some sort of API. Is there anyone here who would
want to use that API? If so, what sort of interface would you prefer,
something like a standard SOAP web service? Also, what kind of functions
would be most useful?

Thanks for linking.

Regards,
Drew

-----Original Message-----
From: David Eaves
Sent: Tuesday, January 11, 2011 8:03 PM
To: [hidden email]
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Drew - this is great news.

To follow up on Jonathan's comment, another thing for someone to
consider is creating an API, I could imagine some sites like Open
Parliament using that to list expenses for a given member of parliament.

Congratulations to being stumbled, I linked to your site on my blog today.

cheers,
dave

On 11-01-11 3:51 PM, Drew Mcpherson wrote:

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Drew Mcpherson

Re: Government Expenses OPENING

In reply to this post by Glen Newton

The custom searches are very resource intensive so it provides 1000 records
max for those, otherwise it would completely overload the dedicated server
it is on, which is fairly costly. It used to be available on a subscription
basis, but nobody was subscribing. I recently changed it to not require a
username and password, so those reports are free now, but haven't changed
the full database option. All of the data is freely available through the
custom reports, just 1000 records at a time so that the server doesn't
crash.

As for the legalities of selling data and copyright law as such, there is
plenty of precedent for selling and copyrighting compilations of public
information. Just look at infousa.com for instance. Anyway, I haven't
actually prepared any kind of licence agreement and nobody has purchased it
anyway.

It would be nice to be able to afford to have a roof over my head and stop
couch surfing, let alone pay off my massive debts, but it looks like I'm
probably not going to recoup any of my enormous costs to build this data
crawler and prepare the full database and all the reports, so I will likely
end up giving it away as well eventually when it becomes clear that nobody
is interested in purchasing it. I wonder if there's some sort of government
grant, program or contract that would help me out?

Cheers,
Drew

-----Original Message-----
From: Glen Newton
Sent: Tuesday, January 11, 2011 9:13 PM
To: civicaccess discuss ; GOSLING members in Ottawa
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

Drew Mcpherson <[hidden email]> wrote:
> The data is now open, searchable and downloadable in Excel format

The site allows someone to get the first 1000 records of the data in
Excel format.
Go to: http://www.governmentexpenses.ca/custom_reports.aspx and just
click on "Generate custom report" on the bottom.
The page that generates this report says:
"There were 1000 results found.
Click here to download the report
PLEASE NOTE: Only a maximum of 1000 records are shown in the custom
search, for more records, please purchase the complete database.
Purchase the entire database now"

So the data is NOT open, is searchable, and is _partially_ downloadable.

And I am not so sure you can legally sell this collection.

Thanks,
Glen

On Tue, Jan 11, 2011 at 6:51 PM, Drew Mcpherson <[hidden email]> wrote:

> Ok you convinced me. The data is now open, searchable and downloadable in
> Excel format. By the way whoever put the site on stumble, thanks for
> that.
> The site has been getting quite a bit of traffic from there all day today!
>
> Regards,
> Drew Mcpherson
>
>
> -----Original Message----- From: Glen Newton
> Sent: Sunday, January 09, 2011 12:00 PM
> To: civicaccess discuss ; GOSLING members in Ottawa
> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>
> Hi Drew,
>
> Do you have an API or data dump of the crawl/collection for others to
> use/reuse/mashup?
>
> Because if you don't have these, then it isn't an open data project.
> If you don't have these, all you've done is changed the situation of
> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>
> Glen
>
>
> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>>
>> Hi Everyone, I’m the guy who did the government expenses crawler / report
>> site and I am just finishing up a run of the engine in the next few days
>> to
>> get the latest 2010 data. I just wanted to let you know that when it’s
>> finished, I’ll open up the search tool to the public for the day in case
>> you
>> were interested to peruse it. Oh and speaking of StatsCan cost recovery,
>> if
>> anyone wants to help support my project through a donation or
>> subscription,
>> it wouldn’t take much to help me out and I could afford to stop couch
>> surfing :) I guarantee my efficiency is far superior to any government
>> department and probably well worth it.
>>
>> Cheers,
>> Drew Mcpherson
>> Bine Consulting Corp.
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>
>
>
> --
>
> -
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

--

-
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

Glen Newton

Re: Government Expenses OPENING

Hi Drew,

On Tue, Jan 11, 2011 at 8:25 PM, Drew Mcpherson <[hidden email]> wrote:
> The custom searches are very resource intensive so it provides 1000 records
> max for those, otherwise it would completely overload the dedicated server
> it is on, which is fairly costly. It used to be available on a subscription
> basis, but nobody was subscribing. I recently changed it to not require a
> username and password, so those reports are free now, but haven't changed
> the full database option. All of the data is freely available through the
> custom reports, just 1000 records at a time so that the server doesn't
> crash.

If you want to generate a complete export, I can arrange for it to be
hosted so not to tax/cost your system.

> As for the legalities of selling data and copyright law as such, there is
> plenty of precedent for selling and copyrighting compilations of public
> information. Just look at infousa.com for instance. Anyway, I haven't
> actually prepared any kind of licence agreement and nobody has purchased it
> anyway.

I am not sure. Depending on the jurisdiction, things can be quite different.
The original sources are copyright, so any subsequent use of them may
need permission.

> It would be nice to be able to afford to have a roof over my head and stop
> couch surfing, let alone pay off my massive debts, but it looks like I'm
> probably not going to recoup any of my enormous costs to build this data
> crawler and prepare the full database and all the reports, so I will likely
> end up giving it away as well eventually when it becomes clear that nobody
> is interested in purchasing it. I wonder if there's some sort of government
> grant, program or contract that would help me out?

Just curious: were the costs primarily in your labour that was not
paid for, or the cost of crawling/hosting?

Clearly you are skilled and creative: this is good work.
What about supporting the site with Google Adsense?

What about exposing this content in a facebook app?
Something like: Canadian FB users can find their MP and have ongoing
updates on their MPs increasing expenses?
Or make it a game: FB users can 'bet' their friends whether their MP
will spend more than their friend's MP (or some other arbitrary MP).

constructively,
Glen

> Cheers,
> Drew
>
> -----Original Message----- From: Glen Newton
> Sent: Tuesday, January 11, 2011 9:13 PM
> To: civicaccess discuss ; GOSLING members in Ottawa
> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>
> Hi Drew,
>
> Drew Mcpherson <[hidden email]> wrote:
>>
>> The data is now open, searchable and downloadable in Excel format
>
> The site allows someone to get the first 1000 records of the data in
> Excel format.
> Go to: http://www.governmentexpenses.ca/custom_reports.aspx and just
> click on "Generate custom report" on the bottom.
> The page that generates this report says:
> "There were 1000 results found.
> Click here to download the report
> PLEASE NOTE: Only a maximum of 1000 records are shown in the custom
> search, for more records, please purchase the complete database.
> Purchase the entire database now"
>
> So the data is NOT open, is searchable, and is _partially_ downloadable.
>
> And I am not so sure you can legally sell this collection.
>
> Thanks,
> Glen
>
>
>
> On Tue, Jan 11, 2011 at 6:51 PM, Drew Mcpherson <[hidden email]> wrote:
>>
>> Ok you convinced me. The data is now open, searchable and downloadable in
>> Excel format. By the way whoever put the site on stumble, thanks for
>> that.
>> The site has been getting quite a bit of traffic from there all day today!
>>
>> Regards,
>> Drew Mcpherson
>>
>>
>> -----Original Message----- From: Glen Newton
>> Sent: Sunday, January 09, 2011 12:00 PM
>> To: civicaccess discuss ; GOSLING members in Ottawa
>> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>>
>> Hi Drew,
>>
>> Do you have an API or data dump of the crawl/collection for others to
>> use/reuse/mashup?
>>
>> Because if you don't have these, then it isn't an open data project.
>> If you don't have these, all you've done is changed the situation of
>> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>>
>> Glen
>>
>>
>> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>>>
>>> Hi Everyone, I’m the guy who did the government expenses crawler / report
>>> site and I am just finishing up a run of the engine in the next few days
>>> to
>>> get the latest 2010 data. I just wanted to let you know that when it’s
>>> finished, I’ll open up the search tool to the public for the day in case
>>> you
>>> were interested to peruse it. Oh and speaking of StatsCan cost recovery,
>>> if
>>> anyone wants to help support my project through a donation or
>>> subscription,
>>> it wouldn’t take much to help me out and I could afford to stop couch
>>> surfing :) I guarantee my efficiency is far superior to any government
>>> department and probably well worth it.
>>>
>>> Cheers,
>>> Drew Mcpherson
>>> Bine Consulting Corp.
>>> _______________________________________________
>>> CivicAccess-discuss mailing list
>>> [hidden email]
>>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>>
>>
>>
>>
>> --
>>
>> -
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
>
>
> --
>
> -
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

--

-

Drew Mcpherson

Re: Government Expenses OPENING

Hi Glen,

-----Original Message-----
From: Glen Newton
Sent: Tuesday, January 11, 2011 9:41 PM
To: civicaccess discuss
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

On Tue, Jan 11, 2011 at 8:25 PM, Drew Mcpherson <[hidden email]> wrote:

> The custom searches are very resource intensive so it provides 1000
> records
> max for those, otherwise it would completely overload the dedicated server
> it is on, which is fairly costly. It used to be available on a
> subscription
> basis, but nobody was subscribing. I recently changed it to not require a
> username and password, so those reports are free now, but haven't changed
> the full database option. All of the data is freely available through the
> custom reports, just 1000 records at a time so that the server doesn't
> crash.

>If you want to generate a complete export, I can arrange for it to be
>hosted so not to tax/cost your system.

I'm not sure an Excel file would be an appropriate vehicle to deliver the
content since there are hundreds of thousands of rows. It is likely only
something that would be of interest to a programmer so if any of them get in
touch, I suppose I'll go from there.

> As for the legalities of selling data and copyright law as such, there is
> plenty of precedent for selling and copyrighting compilations of public
> information. Just look at infousa.com for instance. Anyway, I haven't
> actually prepared any kind of licence agreement and nobody has purchased
> it
> anyway.

>I am not sure. Depending on the jurisdiction, things can be quite
>different.
>The original sources are copyright, so any subsequent use of them may
>need permission.

Well in order for this to have any legal basis, the government would have to
claim that these numbers are creative content (which I'm pretty sure by
definition they are not), and that they intended to copyright it and not
release it into the public domain. So even if it were technically
copyrightable (likely not - the way infousa gets around it is by adding fake
records), and even if it weren't released into the public domain by the
Canadian government, they'd have to sue me for copyright infringement, which
would be weird because it would mean that they didn't want the data to be
proactively disclosed, plus it probably wouldn't make for very good PR to be
suing the guy who gave up his livelihood in order to bring the data to the
public in a useable format. I'm not a politician, but I suspect such an act
might be political suicide.

> It would be nice to be able to afford to have a roof over my head and stop
> couch surfing, let alone pay off my massive debts, but it looks like I'm
> probably not going to recoup any of my enormous costs to build this data
> crawler and prepare the full database and all the reports, so I will
> likely
> end up giving it away as well eventually when it becomes clear that nobody
> is interested in purchasing it. I wonder if there's some sort of
> government
> grant, program or contract that would help me out?

>Just curious: were the costs primarily in your labour that was not
>paid for, or the cost of crawling/hosting?

Most of the cost would be my labour since it did take quite a significant
effort to complete, as I'm sure the other team(s) of people who tried this
project can attest to.

>Clearly you are skilled and creative: this is good work.
>What about supporting the site with Google Adsense?

Thanks for the compliment. I had Google Adsense on there for a short while
but it didn't seem to be overly successful with the low amount of hits on
the site.

>What about exposing this content in a facebook app?
>Something like: Canadian FB users can find their MP and have ongoing
>updates on their MPs increasing expenses?
>Or make it a game: FB users can 'bet' their friends whether their MP
>will spend more than their friend's MP (or some other arbitrary MP).

Interesting ideas. I wonder if people would actually care that much on
facebook to sign up for that? There is actually a facebook group but there
are only a couple dozen people on it since it got deleted by facebook staff
and then re-created by a friend of mine. The betting idea is kind of funny.
Personally I'd be more concerned about getting nabbed for illegal gambling
doing that than copyright violation by selling a compilation of the data
set.

>constructively,
>Glen

Thanks,
Drew

Michael Mulley

Re: Government Expenses OPENING

In reply to this post by Drew Mcpherson

Hi Drew,

Glad to see this!

An API is certainly a nice thing, and I'll let others chime in if they'd be interested (though for the most part this isn't a super-technical list). But given that your dataset probably doesn't change particularly often, it's an even easier step to just share your database. I imagine you're using a relational database -- just provide a dump of the relevant table schema and SQL INSERT statements. I can't promise that people will use it, but there're certainly some potential users -- I know Sébastien Pierre's expense visualizer is currently running on fairly old and inexact data, I can certainly think of possible integration points with openparliament.ca -- and publishing a dump is about fifteen minutes' effort.

best,

michael

On Tue, Jan 11, 2011 at 8:15 PM, Drew Mcpherson <[hidden email]> wrote:

I suppose I could provide some sort of API. Is there anyone here who would want to use that API? If so, what sort of interface would you prefer, something like a standard SOAP web service? Also, what kind of functions would be most useful?

Thanks for linking.

Regards,
Drew

-----Original Message----- From: David Eaves
Sent: Tuesday, January 11, 2011 8:03 PM
To: [hidden email]

Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Drew - this is great news.

To follow up on Jonathan's comment, another thing for someone to
consider is creating an API, I could imagine some sites like Open
Parliament using that to list expenses for a given member of parliament.

Congratulations to being stumbled, I linked to your site on my blog today.

cheers,
dave

On 11-01-11 3:51 PM, Drew Mcpherson wrote:

Ok you convinced me. The data is now open, searchable and downloadable in Excel format. By the way whoever put the site on stumble, thanks for that. The site has been getting quite a bit of traffic from there all day today!

Regards,
Drew Mcpherson

-----Original Message----- From: Glen Newton
Sent: Sunday, January 09, 2011 12:00 PM
To: civicaccess discuss ; GOSLING members in Ottawa
Subject: Re: [CivicAccess-discuss] Government Expenses OPENING

Hi Drew,

Do you have an API or data dump of the crawl/collection for others to
use/reuse/mashup?

Because if you don't have these, then it isn't an open data project.
If you don't have these, all you've done is changed the situation of
the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.

Glen

On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:

Hi Everyone, I’m the guy who did the government expenses crawler / report
site and I am just finishing up a run of the engine in the next few days to
get the latest 2010 data. I just wanted to let you know that when it’s
finished, I’ll open up the search tool to the public for the day in case you
were interested to peruse it. Oh and speaking of StatsCan cost recovery, if
anyone wants to help support my project through a donation or subscription,
it wouldn’t take much to help me out and I could afford to stop couch
surfing :) I guarantee my efficiency is far superior to any government
department and probably well worth it.

Cheers,
Drew Mcpherson
Bine Consulting Corp.
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
_______________________________________________
CivicAccess-discuss mailing list
[hidden email]
http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss

James McKinney

Re: Government Expenses OPENING

Apathy is Boring would be interested in integrating this data into
http://citizenfactory.com/

Like Michael said, a SQL dump is fine.

On Tue, Jan 11, 2011 at 9:59 PM, Michael Mulley
<[hidden email]> wrote:

> Hi Drew,
> Glad to see this!
> An API is certainly a nice thing, and I'll let others chime in if they'd be
> interested (though for the most part this isn't a super-technical list). But
> given that your dataset probably doesn't change particularly often, it's an
> even easier step to just share your database. I imagine you're using a
> relational database -- just provide a dump of the relevant table schema and
> SQL INSERT statements. I can't promise that people will use it, but there're
> certainly some potential users -- I know Sébastien Pierre's expense
> visualizer is currently running on fairly old and inexact data, I can
> certainly think of possible integration points with openparliament.ca -- and
> publishing a dump is about fifteen minutes' effort.
> best,
> michael
>
> On Tue, Jan 11, 2011 at 8:15 PM, Drew Mcpherson <[hidden email]> wrote:
>>
>> I suppose I could provide some sort of API. Is there anyone here who
>> would want to use that API? If so, what sort of interface would you prefer,
>> something like a standard SOAP web service? Also, what kind of functions
>> would be most useful?
>>
>> Thanks for linking.
>>
>> Regards,
>> Drew
>>
>> -----Original Message----- From: David Eaves
>> Sent: Tuesday, January 11, 2011 8:03 PM
>> To: [hidden email]
>> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>>
>> Drew - this is great news.
>>
>> To follow up on Jonathan's comment, another thing for someone to
>> consider is creating an API, I could imagine some sites like Open
>> Parliament using that to list expenses for a given member of parliament.
>>
>> Congratulations to being stumbled, I linked to your site on my blog today.
>>
>> cheers,
>> dave
>>
>> On 11-01-11 3:51 PM, Drew Mcpherson wrote:
>>>
>>> Ok you convinced me. The data is now open, searchable and downloadable
>>> in Excel format. By the way whoever put the site on stumble, thanks for
>>> that. The site has been getting quite a bit of traffic from there all day
>>> today!
>>>
>>> Regards,
>>> Drew Mcpherson
>>>
>>>
>>> -----Original Message----- From: Glen Newton
>>> Sent: Sunday, January 09, 2011 12:00 PM
>>> To: civicaccess discuss ; GOSLING members in Ottawa
>>> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>>>
>>> Hi Drew,
>>>
>>> Do you have an API or data dump of the crawl/collection for others to
>>> use/reuse/mashup?
>>>
>>> Because if you don't have these, then it isn't an open data project.
>>> If you don't have these, all you've done is changed the situation of
>>> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>>>
>>> Glen
>>>
>>>
>>> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>>>>
>>>> Hi Everyone, I’m the guy who did the government expenses crawler /
>>>> report
>>>> site and I am just finishing up a run of the engine in the next few days
>>>> to
>>>> get the latest 2010 data. I just wanted to let you know that when it’s
>>>> finished, I’ll open up the search tool to the public for the day in case
>>>> you
>>>> were interested to peruse it. Oh and speaking of StatsCan cost
>>>> recovery, if
>>>> anyone wants to help support my project through a donation or
>>>> subscription,
>>>> it wouldn’t take much to help me out and I could afford to stop couch
>>>> surfing :) I guarantee my efficiency is far superior to any government
>>>> department and probably well worth it.
>>>>
>>>> Cheers,
>>>> Drew Mcpherson
>>>> Bine Consulting Corp.
>>>> _______________________________________________
>>>> CivicAccess-discuss mailing list
>>>> [hidden email]
>>>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>>>
>>>
>>>
>>>
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>

Glen Newton

Re: Government Expenses OPENING

Hi Drew,

If you do release it as a SQL dump, you should make the request that
anyone using it properly attribute you (as the person who did all the
work in collecting the data!), and link back to your site if possible.

-Glen

On Wed, Jan 12, 2011 at 12:49 PM, James McKinney
<[hidden email]> wrote:

> Apathy is Boring would be interested in integrating this data into
> http://citizenfactory.com/
>
> Like Michael said, a SQL dump is fine.
>
> On Tue, Jan 11, 2011 at 9:59 PM, Michael Mulley
> <[hidden email]> wrote:
>> Hi Drew,
>> Glad to see this!
>> An API is certainly a nice thing, and I'll let others chime in if they'd be
>> interested (though for the most part this isn't a super-technical list). But
>> given that your dataset probably doesn't change particularly often, it's an
>> even easier step to just share your database. I imagine you're using a
>> relational database -- just provide a dump of the relevant table schema and
>> SQL INSERT statements. I can't promise that people will use it, but there're
>> certainly some potential users -- I know Sébastien Pierre's expense
>> visualizer is currently running on fairly old and inexact data, I can
>> certainly think of possible integration points with openparliament.ca -- and
>> publishing a dump is about fifteen minutes' effort.
>> best,
>> michael
>>
>> On Tue, Jan 11, 2011 at 8:15 PM, Drew Mcpherson <[hidden email]> wrote:
>>>
>>> I suppose I could provide some sort of API. Is there anyone here who
>>> would want to use that API? If so, what sort of interface would you prefer,
>>> something like a standard SOAP web service? Also, what kind of functions
>>> would be most useful?
>>>
>>> Thanks for linking.
>>>
>>> Regards,
>>> Drew
>>>
>>> -----Original Message----- From: David Eaves
>>> Sent: Tuesday, January 11, 2011 8:03 PM
>>> To: [hidden email]
>>> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>>>
>>> Drew - this is great news.
>>>
>>> To follow up on Jonathan's comment, another thing for someone to
>>> consider is creating an API, I could imagine some sites like Open
>>> Parliament using that to list expenses for a given member of parliament.
>>>
>>> Congratulations to being stumbled, I linked to your site on my blog today.
>>>
>>> cheers,
>>> dave
>>>
>>> On 11-01-11 3:51 PM, Drew Mcpherson wrote:
>>>>
>>>> Ok you convinced me. The data is now open, searchable and downloadable
>>>> in Excel format. By the way whoever put the site on stumble, thanks for
>>>> that. The site has been getting quite a bit of traffic from there all day
>>>> today!
>>>>
>>>> Regards,
>>>> Drew Mcpherson
>>>>
>>>>
>>>> -----Original Message----- From: Glen Newton
>>>> Sent: Sunday, January 09, 2011 12:00 PM
>>>> To: civicaccess discuss ; GOSLING members in Ottawa
>>>> Subject: Re: [CivicAccess-discuss] Government Expenses OPENING
>>>>
>>>> Hi Drew,
>>>>
>>>> Do you have an API or data dump of the crawl/collection for others to
>>>> use/reuse/mashup?
>>>>
>>>> Because if you don't have these, then it isn't an open data project.
>>>> If you don't have these, all you've done is changed the situation of
>>>> the GoC hoarding GoC data to the GoC and _you_ hoarding GoC data.
>>>>
>>>> Glen
>>>>
>>>>
>>>> On Thu, Jan 6, 2011 at 12:53 PM, Drew Mcpherson <[hidden email]> wrote:
>>>>>
>>>>> Hi Everyone, I’m the guy who did the government expenses crawler /
>>>>> report
>>>>> site and I am just finishing up a run of the engine in the next few days
>>>>> to
>>>>> get the latest 2010 data. I just wanted to let you know that when it’s
>>>>> finished, I’ll open up the search tool to the public for the day in case
>>>>> you
>>>>> were interested to peruse it. Oh and speaking of StatsCan cost
>>>>> recovery, if
>>>>> anyone wants to help support my project through a donation or
>>>>> subscription,
>>>>> it wouldn’t take much to help me out and I could afford to stop couch
>>>>> surfing :) I guarantee my efficiency is far superior to any government
>>>>> department and probably well worth it.
>>>>>
>>>>> Cheers,
>>>>> Drew Mcpherson
>>>>> Bine Consulting Corp.
>>>>> _______________________________________________
>>>>> CivicAccess-discuss mailing list
>>>>> [hidden email]
>>>>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> CivicAccess-discuss mailing list
>>> [hidden email]
>>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>> _______________________________________________
>>> CivicAccess-discuss mailing list
>>> [hidden email]
>>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
>> _______________________________________________
>> CivicAccess-discuss mailing list
>> [hidden email]
>> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>>
> _______________________________________________
> CivicAccess-discuss mailing list
> [hidden email]
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss
>

--

-