@Edent robots.txt is the wrong approach. It can stop crawling but not indexing. You want the noindex meta tag in your HTML. In fact, if you have the noindex tag but your robots.txt prevents e.g. Google from crawling your page it will not pick it up and still serve your pages from the index.