# How to limit the number of web pages downloaded from a site?

#### Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

•  Subject
• Author
• Posted on
I have a very large site with valuable information.

Any hints or pointers would be appreciated.

## Re: How to limit the number of web pages downloaded from a site?

Password protect folders or pages, make users register to get the
passwords, that would slow them down a bit. But really, if you make
stuff available publicly...

--
dorayme

## Re: How to limit the number of web pages downloaded from a site?

the entire sites) allows you to specify login/passwd.
So, once they register, they can enter this info and boom...

Well, the site is 150 megs, over 20k articles.
And there are plenty of people who would LOVE to have
the entire site on their own box.
Then you have a problem. Providers usually charge for
the amount of traffic. In one month, you'd have to shell
out some bux, just to give the information to the
"gimme free Coce" zombies.
That does not make sense.

## Re: How to limit the number of web pages downloaded from a site?

I understand your concerns and it is natural to worry a bit. But
consider again.

That there is Teleport Pro does not actually show that my suggestion
would not work. Perhaps you are looking at every stage at worst case
or be prepared to get it. That is one thing. The other thing is that
granting passwords might be conditional on them agreeing not to do what
you fear. Is your site a serious site liable to attract serious people?
You might be surprised how decent most people are if you make things
clear.

How sure are you of the likelihood of a whole bunch of people wanting to
download the whole lot? Most people are wary of over exposing themselves
to information and will get what they are interested in. So I guess, you
need to do some guessing and some analysis. Perhaps you are worrying
excessively?

Presumably you would be hoping your site is used and is useful. If a
bunch of folk download a small bunch of articles each, this might well
be the biggest factor rather than a few who download the lot. You would
have to make some projections concerning this, you would be in the best
position to crunch some numbers as it is your field. If you are more
successful than you imagine via people doing reasoanable things rather
than unreasonable things, you perhaps ought to be preparing yourself for
the possibility of serious server charges. I understand your concern to
limit things, but a huge site carves out a certain territory and you may
need to consider charging for access?

The other suggestion I might make is that you provide for the odd
possibility of some people wanting the lot by employing compressed
archives and utilising other than your own server, there might be some
free servers or cheap servers for this express purpose.

--
dorayme

## Re: How to limit the number of web pages downloaded from a site?

Not really. There are quite a few programs out there.
Easiest thing in the world to find.
Do a search on Teleport Pro. A VERY nice program.

Well, if you use that site, your salary could go up quite a bit
just in a few months. Is that "serious"?

:--}

I wish I had your optimism. But I have seen plenty of evidence
otherwise.

Who knows. But there is an issue here. I have no doubts about that much.

That is not a problem.

But those few count hundred times more that fair users.

That is what we are doing here.

Yep. For about $100/mo. I can have my own virtual domain on a tier 1 network. But that bites. There is no income from this enterprise. Nobody is going to give you a dime for getting something. That much I have seen. Nope. That is not reasonable. It should be totally free. No registration, no charges of any kind. Just come and look at anything you want. But not be a bastard. Thanks for feedback. ## Re: How to limit the number of web pages downloaded from a site? It doesn't work. For example, Teleport Pro (a program to download the entire sites) allows you to specify login/passwd. So, once they register, they can enter this info and boom... Well, the site is 150 megs, over 20k articles. And there are plenty of people who would LOVE to have the entire site on their own box. Then you have a problem. Providers usually charge for the amount of traffic. In one month, you'd have to shell out some bux, just to give the information to the "gimme free Coce" zombies. That does not make sense. -- The most powerful Usenet tool you have ever heard of. NewsMaestro v. 4.0.8 has been released. * Several nice improvements and bug fixes. Note: In some previous releases some class files were missing. As a result, the program would not run. Sorry for the inconvenience. Web page: http://newsmaestro.sourceforge.net / Download page: http://newsmaestro.sourceforge.net/Download_Information.htm Send any feedback, ideas, suggestions, test results to newsmaestroinfo \at/ mail.ru. Your personal info will not be released and your privacy will be honored. ## Re: How to limit the number of web pages downloaded from a site? Gazing into my crystal ball I observed nad@invalid.com (Nad) writing in You could store their IP address in a session, and check to see the length of time between requests. -- Adrienne Boswell at Home Arbpen Web Site Design Services http://www.cavalcade-of-coding.info Please respond to the group so others can share ## Re: How to limit the number of web pages downloaded from a site? Well, something along those lines. The problem is the server side support. Some servers do not allow cgi, php, javascript, or even ssi executable commands, and I'd like it to work on ANY server. ## Re: How to limit the number of web pages downloaded from a site? Something along those lines. I was thinking of detecting automated downloads. When people simply look at the information manually, it is all fine and dandy. But when they start running a program to download hundreds if not thousands of articles, that is another issue. But there are issues with this. Some web hosting vendors do not allow the executables to run as it is a security risk. They may not allow cgi, php, even executable commands of ssi and javascripts. Now, in order to detect a page access by the client, you either add an ssi include statement to either cgi or javascript, or make your pages dynamically assemble with php. But what if you can not even run those because the provider does not allow it? Sure, you can shell out some serious bux to get yourself a premiere hosting facility where you can have your own virtual domain. But if you spent years developing tools to automatically build a 20k+ article site, and are willing to give the information for free, then financing the top notch provider on the top of it, is not something that excites my imagination. There is an excellent provider - by.ru. They are free. They are huge, several hundred thousand sites and free email users. But they do not allow ANY executables to run. They even disable the ssi executable statements. So, what do you do in that case? ## Re: How to limit the number of web pages downloaded from a site? Something along those lines. I was thinking of detecting automated downloads. When people simply look at the information manually, it is all fine and dandy. But when they start running a program to download hundreds if not thousands of articles, that is another issue. But there are issues with this. Some web hosting vendors do not allow the executables to run as it is a security risk. They may not allow cgi, php, even executable commands of ssi and javascripts. Now, in order to detect a page access by the client, you either add an ssi include statement to either cgi or javascript, or make your pages dynamically assemble with php. But what if you can not even run those because the provider does not allow it? Sure, you can shell out some serious bux to get yourself a premiere hosting facility where you can have your own virtual domain. But if you spent years developing tools to automatically build a 20k+ article site, and are willing to give the information for free, then financing the top notch provider on the top of it, is not something that excites my imagination. There is an excellent provider - by.ru. They are free. They are huge, several hundred thousand sites and free email users. But they do not allow ANY executables to run. They even disable the ssi executable statements. So, what do you do in that case? ## Re: How to limit the number of web pages downloaded from a site? Gazing into my crystal ball I observed nad@invalid.com (Nad) writing in That is transparent to the client, and you find a provider that allows scripting on their servers. There are plenty of free hosts that do so. Look through this group for some recently mentioned hosts. -- Adrienne Boswell at Home Arbpen Web Site Design Services http://www.cavalcade-of-coding.info Please respond to the group so others can share ## Re: How to limit the number of web pages downloaded from a site? Well, I'd like to know about those servers. I'll review the posts. Do you know of any off hand? But I suspect there is a limit on the site size, isn't there? The sites I'll be creating are in the range of 150-250 megs. The one I am using now has no limit on size. Btw, I was thinking if it is worth creating a site on HTML, using this group's archive, going back a couple of years. Unfortunately, I am not an HTML expert and the site has to be organized on two categories of information 1) Code examples 2) Expert opinions Now, to generate the site automatically, you have to process the archives with fancy, multi-stage filters to make sure you get only articles that ARE exactly on topic for some issue. In other words, find a needle in a haystack. The information is categorized by different issues and I have no idea what those are for HTML. You have to spend some time on it and create a category list of interesting or important issues. Secondly, you need to know who are the "experts" around here. I have never participated in this group. So, I'd have to review tons of posts and decide who are those people that really know what they are talking about. Things like that. But who knows, you may see a very nice site on HTML in the near future what would contain thousands of code examples. If you have any requests, ideas or suggestions, that would help. If you can recommend a list of "experts", that'd help. They don't have to be the HTML gods. They just have to know what they are talking about and be helpful in the followups. Also, if you can give me a list of HTML issues that are worth creating a chapter from, that would help. Just type one entry per line. ## Re: How to limit the number of web pages downloaded from a site? Nad wrote: Their is NO SUCH THING as Unlimited, especially when it come to web servers. The best you can get is a VPS or dedicated server. Check out http://www.servergrade.com.au They are not free but they do have very good deals on web hosting and domain names. Especially if you buy your Domain and web hosting as the one package, then the domain only costs$1 per year!

No it is not at all. Also if you create sites useing content from
other sites, you can be sued for being in breach of copyright laws in
many / most countries of the world, also Google will ban you from it's
listings for having duplicated content.
--

## Re: How to limit the number of web pages downloaded from a site?

Change the articles' text to Olde Englishe.

--
Neredbojias
http://www.neredbojias.net /
Public Website

## Re: How to limit the number of web pages downloaded from a site?

Sure, but tell me they do it without server-side techniques which you so
explicitly eschewed...

Don't think it would work but just 10-15 minute meta refresh in page head.

--
Neredbojias
http://www.neredbojias.net /
Public Website

## Re: How to limit the number of web pages downloaded from a site?

It depends upon what you mean by 'articles.'  If put you html documents on a
web server. you are pretty much inviting the public to view/download as much
of it as they want.  If it is 'valuable', why are you giving it away?  And
if you are giving it away valuable stuff, what did you expect?  What is your
real concern here?

If you only worried about server load, why not zip or tar and gzip it up
and put it on an FTP server?  This is most practical for related documents,
such as parts of a tutorial or parts of a spec.  If you are a philanthropist
who is giving away valuable stuff, you can give it away in big chunks so
the nickel and dime requests don't bug you.

is pretty much a courtesy thing, and it won't stop anyone who is manually
Likewise, you can block nice spiders which send a true user-agent ID, but
not so nice spiders can spoof their ID.  That's kind of pointless, because
most of the nice spiders will obey robots.txt anyway.

You can make pages available through php or cgi which keeps track of the
number of documents with hidden controls.  This is easily defeated by
anyone determined to do so, and like a cheap lock, will only keep the honest
people out.  Beyond that, you can go to various user account schemes up to
putting your documents on a secure server.

But I think what you are asking is 'Can I keep my documents public and still
limit public access?'  And the answer to that is, of course not because
there is a fundamental contradiction in what you want.

--
Lars Eighner <http://larseighner.com/ usenet@larseighner.com
War hath no fury like a noncombatant.
- Charles Edward Montague

## Re: How to limit the number of web pages downloaded from a site?

all sorts of things.

entire site. By the time you put him into robots.txt, it is too late.

That is not a problem. They can manually download as much as they want.

How?

Well, no account schemes, no user verification, no limits beyond

I do not see it at the moment.

## Re: How to limit the number of web pages downloaded from a site?

all sorts of things.

entire site. By the time you put him into robots.txt, it is too late.

That is not a problem. They can manually download as much as they want.

How?

Well, no account schemes, no user verification, no limits beyond

I do not see it at the moment. Can you expand on that?

## Re: How to limit the number of web pages downloaded from a site?

If I go to your website and view every one of your pages, guess what?

simple coded text block so that they have to type in a code to get the
page.

## Re: How to limit the number of web pages downloaded from a site?

No problem. You did it manually, and you just viewed the information,
which is exactly what this site is for.

Nope. That's a hassle for the user.
They should be able to move freely around the site
without any passwords, log-ins or any other crap.